Cyrus - GFS slow start and poor performace

[Bron Gondwana]

On Wed, 14 May 2008 10:59:33 +0200, "Maurizio Lo Bosco" <maurizio.lobosco at eng.it> said:
> Hi all,
> I know that the usage of the GFS has been discussed for long time on this 
> mailing list but I would like to know if it is normal to have a very slow 
> start (15 minutes) with just 4300 users (the cyrus db is composed of
> 20940 
> lines).
> It happens only with the GFS and the skiplist database; using the flat it 
> takes few seconds to start.
> The system is composed of 2 IBM x3550 with redhat enterprise linux 5.1 
> attached to a SAN IBM DS4700 with an dual fibre channel (4Gb/s multipath 
> active-backup).
> 
> The dump of the database takes 7 minutes but the disk usage is definitely
> low 
> (less than 5%)
> RedHat is saying that there is no way to optimise the performance on the
> GFS 
> locking architecture and they will now take a look to the cyrus code. 
> 
> Do you have any tips?

Skiplist mailboxes.db gets a "recovery" run on it at startup.  The recovery
visits all the records in the file.

That said, it does it all in order.

Can you post the syslog output of cyrus as it starts (slowly)?  I wonder if
it's also doing a checkpoint, which visits all the mailboxes.db records as
well, but does them in...


oh, indeed.

Recovery also writes back pointers all over the place.  It does LOTS of writes
to random locations within the file.  If GFS is doing something insane like
writing back the entire file to the server for every single update (generally
4 bytes at a time) then this could be a big problem!

That said, the file is locked with a fcntl (flock if fcntl isn't available) lock
over the entire file +append space.  This is an exclusive lock, and it's held
for the entire recovery run.  If GFS's locking can say "just do the updates and
save copying back until the fsync at the end" then that should speed it up.

Bron.
-- 
  Bron Gondwana
  brong at fastmail.fm