NFSv4, anyone?

[Paul Dekkers]

Hi,

Andrew Laurence wrote:

> Has anyone evaluated the new "fixed" locking in NFSv4 for use with 
> Cyrus?  I attended a talk by Sun's Spencer Shepler at USENIX '05 in 
> which he spent a good deal of time on NFSv4 (supposedly) fixes the 
> locking shortcomings and can be used as a native file system.
> http://blogs.sun.com/shepler/date/20050407
> 
> Just curious if anyone has taken it for a drive with Cyrus?

Since the use of NFSv4 was mentioned again in the HA thread, I thought 
I'd give this a shot by using the imaptest-utility from dovecot (the 
only imap stress-testing tool I can think of, I've stressed replication 
with it too) and NFSv4 mounts on RedHat.
Maybe it does not show any errors... maybe it does: better then no 
testing at all. Probably a far too simple test and a lot to tweak.

It would be very interesting to see how and if this works with NetApp 
filers for instance, instead of a RH server. Or on Solaris. I have a 
feeling that this might show different results.

I exported the filesystem simply with:
   /usr/sbin/exportfs -o rw,fsid=0,no_root_squash haver:/data/vol2
and also tried with the secure_locks option added to it, and mounted on 
the client with:
   mount -t nfs4 gerst:/ mnt
so nothing really special. (Suggestions for further tests welcome!)

I tried insecure_locks too, eventually, and that seems to result in a 
similar situation to NFSv3, and apparently secure_locks are default.

The cyrus install is cyrus-imapd-2.3.7-3 from Simon/Invoca with mostly 
default options: so skiplist is the default for all databases. I changed 
to "flushseenstate: 0" though, because imaptest with a local filesystem 
gave me errors if I didn't (!).

With an ext3 filesystem I just get a normal output from imaptest:

[root at haver dovecot-1.0.rc7]# ./imaptest
Auth Logi Sele Fetc Fet2 Stor Dele Expu Appe Logo Disc
    0   43   41   40   39   23   30   26   28   25    0
    0   35   36   36   37   17   26   24   39   33    0
    0   33   33   34   33   21   26   25   45   37    0
    0   38   39   38   38   11   23   24   43   34    0
    0   34   33   32   33   21   28   28   43   36    0

If I mv the spool to the NFSv4 mount, and start cyrus with that 
partition I see a lot of errors, unfortunally:

[root at haver dovecot-1.0.rc7]# ./imaptest
Auth Logi Sele Fetc Fet2 Stor Dele Expu Appe Logo Disc
    0   20   20   20   20    1    3    1    0    0    0
Error: STORE failed: s NO System I/O error
Error: APPEND failed: a NO System I/O error
Error: STORE failed: s NO System I/O error
Error: STORE failed: s NO System I/O error
Error: EXPUNGE failed: e NO System I/O error
Error: EXPUNGE failed: e NO System I/O error
Error: STORE (\Deleted) failed: sd NO System I/O error
Error: STORE failed: s NO System I/O error
Error: STORE failed: s NO System I/O error
Error: STORE failed: s NO System I/O error
Error: STORE (\Deleted) failed: sd NO System I/O error
Error: STORE (\Deleted) failed: sd NO System I/O error
Error: APPEND failed: a NO System I/O error
Error: APPEND failed: a NO System I/O error
Error: APPEND failed: a NO System I/O error
Error: APPEND failed: a NO System I/O error
    0   16   16   16   16    9    6    2    5    0    0
Error: STORE failed: s NO System I/O error
Error: STORE failed: s NO System I/O error
    0    2    2    2    2    5    1    0    0    0    0
Error: APPEND failed: a NO System I/O error
    0    1    1    1    1    1    0    0    1    0    0
    0    0    0    0    0    1    1    0    0    0    0
    0    0    0    0    0    0    1    0    0    0    0
    0    4    4    3    3    1    0    1    4    4    0
    0    3    3    4    4    2    2    3    3    3    0

and in the logs a couple of these:

Nov 24 23:00:37 haver imap[12315]: IOERROR: locking header for 
user.test: Input/output error
Nov 24 23:00:37 haver imap[12312]: IOERROR: locking index for user.test: 
Input/output error

Just a simple test so far, but I'm afraid this doesn't work "out of the 
box"...

To show that NFSv4 *is* different from NFSv3, NFSv3 (or NFSv4 with 
insecure_locking) gives me:

[root at haver dovecot-1.0.rc7]# ./imaptest
Auth Logi Sele Fetc Fet2 Stor Dele Expu Appe Logo Disc
    0   22   22   22   22    1    2    1    3    2    0
    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0
...
    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0
  - 0. stalled for 27 secs in STORE
  - 1. stalled for 27 secs in APPEND
  - 2. stalled for 27 secs in STORE
...

That isn't really working at all.

Curious if someone knows anything else to test, or has more options for 
testing than I have.

Paul