(mmap, munmap) X (msync, NFS) => (NFSv4, fio)

[Andre Felipe Machado]

Hello,
After the important Bron's explanation, I have been running some preliminary
tests using fio in order to raise errors.
I had to start on simple small local Debian Linux 6.x Squeeze VMs before trying
to stress a XenServer environment hosts, network and storage.
Chose NFSv4 because of, presumably, better locking and delegations.

Tried some of the fio ioengines.
Until now, there were no errors.

Please, verify if there are misconceptions or any errors.
If not, will try on Debian Linux 6.x Squeeze VMs at production environment
before setting up a cyrus.


==== server /etc/default/nfs-common

add the following

#AFM 20130326
NEED_IDMAPD=YES

==== server /etc/exports

Add the following

#AFM20130326
/exports
192.168.0.0/255.255.255.0(sync,rw,no_root_squash,no_subtree_check,crossmnt,fsid=0)

invoke-rc.d portmap restart
invoke-rc.d nfs-common restart
invoke-rc.d nfs-kernel-server restart

==== client

mkdir -p /mnt/nfsimports
mount -t nfs4 -o timeo=900,retrans=3,intr,proto=tcp,lock,vers=4,actimeo=10
192.168.0.110:/ /mnt/nfsimports

Only for tests, without security !!!!

chmod -R 777 /mnt/nfsimports

==== fio tests /home/andremachado/fio_randomrw_1kthreads_nfsv4.txt

; random read write of data

[random-rw]
iodepth=64
rw=randrw
#rw=randwrites
rwmixwrite=90 #90% writes
invalidate=1
#buffered=1
direct=1 #zfsfuse segfault
fsync_on_close=1
size=4000m
directory=/mnt/nfsimports
numjobs=50
filesize=1k,75k
runtime=1800
time_based
verify=md5
verify_fatal=1
verify_async=10
#write_lat_log

lockfile=readwrite
fallocate=0 #cyrus, nfs use this?
fadvise_hint=0 #cyrus, nfs use this?
##scramble_buffers=false #anti dedupe. not implemented in debian squeeze
#ioengine=mmap #cyrus imap uses writev, write, fdatasync, mmap/memcpy, fclose
ioengine=sync
#ioengine=vsync #takes hours for finish, despite a 3 min time limit
fdatasync=1 #cyrus number of writes before force fdatasync syscall
#iomem=mmap #cyrus reads from mmap files
do_verify=1
nrfiles=1
openfiles=1
file_service_type=random
nice=19
prio=7 #highest 0, lowest 7
prioclass=2 #best effort priority
#ioscheduler=deadline #squeeze linux does not support this at now
unlink=1 #unlink files at end of job

==== run the test
nice fio --output=/tmp/fio_output_nfs.log
/home/andremachado/fio_randomrw_1kthreads_nfsv4.txt




Some useful links:
http://lists.andrew.cmu.edu/pipermail/info-cyrus/2013-March/thread.html
************************************************************
http://www.cyberciti.biz/faq/nfs4-server-debian-ubuntu-linux/ ***********
http://www.cyberciti.biz/faq/howto-mount-nfs-file-system-over-busy-slow-network/
********
http://wiki.debian.org/NFSTroubleshooting ****
http://wiki.dovecot.org/NFS
http://serverfault.com/questions/127615/how-does-nfs-read-cache-work-on-debian *****
http://www.redhat.com/resourcelibrary/whitepapers/nfsv4wp
http://linux.derkeiler.com/Mailing-Lists/Debian/2012-12/msg01237.html
http://nfsv4.bullopensource.org/doc/nfsv4.admin.frs.v03.php *****
http://linux-nfs.org/wiki/index.php/Cluster_Coherent_NFSv4_and_Delegations
**************
http://storagegaga.wordpress.com/tag/nfsv4/ **********
http://lists.freebsd.org/pipermail/freebsd-stable/2006-March/023766.html use tcp
http://dboptimizer.com/2012/07/16/nfs-versus-dnfs/
http://www.spinics.net/lists/fio/index.html fio mailing list
http://www.spinics.net/lists/fio/msg01457.html
http://www.dbi-services.com/index.php/blog/entry/flexible-io-simulating-database-like-io-activity-without-an-installed-database
http://www.spinics.net/lists/fio/msg00813.html
http://linux.die.net/man/1/fio
http://heatware.net/linux-unix/create-large-many-number-files-thousands-millions/
http://stackoverflow.com/questions/13400312/linux-create-random-directory-file-hierarchy
http://lists.debian.org/debian-user/2012/12/msg01253.html




Regards

Andre Felipe Machado
http://www.techforce.com.br