Cyrus with a NFS storage. random DBERROR

[Paul Dekkers]

Hi,

Took me a while before I found the time to try the meta-partitions and
NFS backed (data-)partitions, but:

Dmitriy Kirhlarov wrote:
> On Thu, May 03, 2007 at 05:08:52PM +0200, Paul Dekkers wrote:
>   
>> I recently tried to use NFS (on a RedHat client, both to a NetApp filer
>> as well as a RedHat NFS server) and I'll share my experiences:
>>
>> Michael Menge wrote:
>>     
>>> Cyrus has 2 problems with NFS.
>>>
>>> 1. Cyrus depends on filesystem locking. NFS-4 should have solved this
>>> problem
>>> but i have not tested it.
>>>
>>> 2. BerkleyDB uses shared Memory which does not work accros multiple
>>> servers.
>>>       
>> I used skiplist in the tests (default with Simon's RPM), and initially
>> just used NFSv3 (and I also tested NFSv4): as long as I mounted with the
>> -o nolock option it actually worked quite well (also on NFSv3). The
>> performance was even better with the NetApp as target than with a local
>> filesystem (and NFSv3 was faster than v4).
>>
>> The nolock options does not disable locking (as I understand it) for the
>> filesystem, it just disables locking over NFS, so other nodes won't have
>> the same file locked. (Correct me if I'm wrong.) My intention was not to
>> have an active-active setup, so in that regard this might not be that
>> bad. Not sure what other catches there are though.
>>     
>
> Are you try metapartition* options? If you don't need active-active
> setup it can be useful.
>   

I didn't try metapartitions with my "-o nolock" experiment (which of
course doesn't work with active-active either), but now I did another
experiment with regular NFS locking (no special mount-options) and a
metapartition for every type of metadata (metapartition_files: header
index cache expunge squat).

I'm glad to say that this seems to work quite well! Similar to the "-o
nolock", actually, but it sounds more solid without the "tweaking".
We use NetApp as NFS filer, and it actually seems to perform a bit
better than our (this) internal RAID5, load is similar, ... and
fortunally no errors from the imaptest.

It sounds like this could work. But I'm not sure about the Cyrus
internals if there are any catches; Ken (or someone else), could this be
considered safe? If it is safe, I'd prefer to use NFS because
performance is similar (or better) and the filers are more reliable than
our RAID5 setup. (I won't go into details, but it's basically a
physically separated RAID-1 set of drives in RAID-6-ish.)

(Performance-wise I only tried small folders, but as soon as the
metadata is cached, I think there a not a lot of directory reads when a
folder is opened, so that doesn't really matter... right?)

>> I stressed the setup with the imaptest tool from Dovecot, I saw problems
>> with that in the past (also with NFSv3 and v4, but in combination with
>> Cyrus 2.2 and I'm not sure if I tried nolock), now it seemed to do just
>> fine. Only NFSv4 does not seem to be the answer, it seems that -o nolock
>> is (on Linux as client).
>>
>> I'm very hesitant to put this into production, I just wanted to do some
>> more tests and ask others after that if they think this is wise or
>> not... I couldn't find the time to do more tests... (like see how RedHat
>> 5 behaves instead of RedHat 4, if the tric also works on FreeBSD, if I
>> can make it fail one way or another... suggestions always welcome...)
>>     

I still have to try how RedHat 5 and FreeBSD behave,

Paul