NFSv4, anyone?

[Simon Matter]

> Hi,
>
> Andrew Laurence wrote:
>
>> Has anyone evaluated the new "fixed" locking in NFSv4 for use with
>> Cyrus?  I attended a talk by Sun's Spencer Shepler at USENIX '05 in
>> which he spent a good deal of time on NFSv4 (supposedly) fixes the
>> locking shortcomings and can be used as a native file system.
>> http://blogs.sun.com/shepler/date/20050407
>>
>> Just curious if anyone has taken it for a drive with Cyrus?
>
> Since the use of NFSv4 was mentioned again in the HA thread, I thought
> I'd give this a shot by using the imaptest-utility from dovecot (the
> only imap stress-testing tool I can think of, I've stressed replication
> with it too) and NFSv4 mounts on RedHat.
> Maybe it does not show any errors... maybe it does: better then no
> testing at all. Probably a far too simple test and a lot to tweak.
>
> It would be very interesting to see how and if this works with NetApp
> filers for instance, instead of a RH server. Or on Solaris. I have a
> feeling that this might show different results.
>
> I exported the filesystem simply with:
>    /usr/sbin/exportfs -o rw,fsid=0,no_root_squash haver:/data/vol2
> and also tried with the secure_locks option added to it, and mounted on
> the client with:
>    mount -t nfs4 gerst:/ mnt
> so nothing really special. (Suggestions for further tests welcome!)
>
> I tried insecure_locks too, eventually, and that seems to result in a
> similar situation to NFSv3, and apparently secure_locks are default.
>
> The cyrus install is cyrus-imapd-2.3.7-3 from Simon/Invoca with mostly
> default options: so skiplist is the default for all databases. I changed

To be a bit more precise here: The rpm uses skiplist for all those
databases that are berkeley by default.

> to "flushseenstate: 0" though, because imaptest with a local filesystem
> gave me errors if I didn't (!).

That's a bit strange, at least I didn't see any difference running
imaptest with "flushseenstate: on or off.

>
> With an ext3 filesystem I just get a normal output from imaptest:
>
> [root at haver dovecot-1.0.rc7]# ./imaptest
> Auth Logi Sele Fetc Fet2 Stor Dele Expu Appe Logo Disc
>     0   43   41   40   39   23   30   26   28   25    0
>     0   35   36   36   37   17   26   24   39   33    0
>     0   33   33   34   33   21   26   25   45   37    0
>     0   38   39   38   38   11   23   24   43   34    0
>     0   34   33   32   33   21   28   28   43   36    0
>
> If I mv the spool to the NFSv4 mount, and start cyrus with that
> partition I see a lot of errors, unfortunally:
>
> [root at haver dovecot-1.0.rc7]# ./imaptest
> Auth Logi Sele Fetc Fet2 Stor Dele Expu Appe Logo Disc
>     0   20   20   20   20    1    3    1    0    0    0
> Error: STORE failed: s NO System I/O error

I tried the same now with NFSv4 mounts between two RHEL4 XEN instances and
I can confirm the same errors. Now I tried something really crazy which I
expected to not work at all: I created a loop mounted filesystem on the
NFSv4 volume and mounted it as cyrus spool. And guess what, it works fine.

That's how filesystems are mounted:
/dev/sda1 on /var/spool/imap type ext3 (rw)
client128:/ on /var/spool/imap2 type nfs4 (rw,noac,addr=192.168.10.128)
/var/spool/imap2/fs1 on /var/spool/imap3 type ext3 (rw,loop=/dev/loop0)

Cyrus works fine on /var/spool/imap and /var/spool/imap3, but not on
/var/spool/imap2.
Of course performance is bad that way, it's just interesting that it works.

Regarding NFSv4, would be nice if the problem turns out to be a
configuration issue. Maybe someone with a current Solaris environment
could try the same there?

Simon