NFSv4, anyone?

[Paul Dekkers]

Hi,

Simon Matter wrote:

>>Since the use of NFSv4 was mentioned again in the HA thread, I thought
>>I'd give this a shot by using the imaptest-utility from dovecot (the
>>only imap stress-testing tool I can think of, I've stressed replication
>>with it too) and NFSv4 mounts on RedHat.
>>Maybe it does not show any errors... maybe it does: better then no
>>testing at all. Probably a far too simple test and a lot to tweak.
>>
>>It would be very interesting to see how and if this works with NetApp
>>filers for instance, instead of a RH server. Or on Solaris. I have a
>>feeling that this might show different results.
>>
>>I exported the filesystem simply with:
>>   /usr/sbin/exportfs -o rw,fsid=0,no_root_squash haver:/data/vol2
>>and also tried with the secure_locks option added to it, and mounted on
>>the client with:
>>   mount -t nfs4 gerst:/ mnt
>>so nothing really special. (Suggestions for further tests welcome!)
>>
>>I tried insecure_locks too, eventually, and that seems to result in a
>>similar situation to NFSv3, and apparently secure_locks are default.
>>
>>The cyrus install is cyrus-imapd-2.3.7-3 from Simon/Invoca with mostly
>>default options: so skiplist is the default for all databases. I changed
> 
> 
> To be a bit more precise here: The rpm uses skiplist for all those
> databases that are berkeley by default.
> 
> 
>>to "flushseenstate: 0" though, because imaptest with a local filesystem
>>gave me errors if I didn't (!).
> 
> 
> That's a bit strange, at least I didn't see any difference running
> imaptest with "flushseenstate: on or off.

Hmm, I get a lot of "No matching messages" if I change this (and more of 
them with more concurrent clients).

    0   22   22   17   22   11   13   11   29   20    0
Error: FETCH BODY[] failed: f2 NO No matching messages (0.000 sec)
Auth Logi Sele Fetc Fet2 Stor Dele Expu Appe Logo Disc
    0   24   23   21   19   14   18   20   32   25    0
    0   18   18   22   25    8   22   18   25   20    0
Error: FETCH BODY[] failed: f2 NO No matching messages (0.000 sec)
    0   24   26   26   21    8   15   18   24   20    0
    0   24   21   22   24    7   17   17   25   22    0
    0   22   22   21   25   15   21   17   28   24    0

I also tried this on a install compiled from source both on Linux and 
FreeBSD: same result...

>>With an ext3 filesystem I just get a normal output from imaptest:
>>
>>[root at haver dovecot-1.0.rc7]# ./imaptest
>>Auth Logi Sele Fetc Fet2 Stor Dele Expu Appe Logo Disc
>>    0   43   41   40   39   23   30   26   28   25    0
>>    0   35   36   36   37   17   26   24   39   33    0
>>    0   33   33   34   33   21   26   25   45   37    0
>>    0   38   39   38   38   11   23   24   43   34    0
>>    0   34   33   32   33   21   28   28   43   36    0
>>
>>If I mv the spool to the NFSv4 mount, and start cyrus with that
>>partition I see a lot of errors, unfortunally:
>>
>>[root at haver dovecot-1.0.rc7]# ./imaptest
>>Auth Logi Sele Fetc Fet2 Stor Dele Expu Appe Logo Disc
>>    0   20   20   20   20    1    3    1    0    0    0
>>Error: STORE failed: s NO System I/O error
> 
> 
> I tried the same now with NFSv4 mounts between two RHEL4 XEN instances and
> I can confirm the same errors. Now I tried something really crazy which I
> expected to not work at all: I created a loop mounted filesystem on the
> NFSv4 volume and mounted it as cyrus spool. And guess what, it works fine.
> 
> That's how filesystems are mounted:
> /dev/sda1 on /var/spool/imap type ext3 (rw)
> client128:/ on /var/spool/imap2 type nfs4 (rw,noac,addr=192.168.10.128)
> /var/spool/imap2/fs1 on /var/spool/imap3 type ext3 (rw,loop=/dev/loop0)
> 
> Cyrus works fine on /var/spool/imap and /var/spool/imap3, but not on
> /var/spool/imap2.
> Of course performance is bad that way, it's just interesting that it works.

I tried GFS yesterday, that worked fine too, but I just mounted the 
volume just on one box (no real cluster) so it was a lousy test 
anyway... (I could try it later, if I can find some time.)

> Regarding NFSv4, would be nice if the problem turns out to be a
> configuration issue. Maybe someone with a current Solaris environment
> could try the same there?

I just tried the in-memory NFSv4 server from citi (in newpynfs); it is 
very very slow, and I had to lower the number of concurrent users in 
imaptest (to prevent 'stalling errors') - but it seems to work better! I 
don't get any locking problems in the logs.

So it could be just the server on Linux that is causing problems... and 
it might work on a Solaris server or NAS like NetApp...

Paul