Running multiple sync_clients

[Michael D. Sofka]

Bron Gondwana wrote:
> On Thu, Nov 18, 2010 at 12:58:10PM -0500, Michael D. Sofka wrote:
>> Is it safe to run multiple sync_clients?  Is there an advantage to doing so?
>>
>> I had to restart sync_client once.  After later restarting cyrus two 
>> sync_clients were running, and appeared to do well together.  Still, I 
>> stopped the old process out of caution.
> 
> What version?  In 2.3.x, the sync_server will obtain a lock, so you can
> only run one sync_server at a time.  This means the sync_clients will
> take it in turns sending data.

2.3.16.  This description fits what I was seeing.


> In 2.4, there is no global locking - just locks on each mailbox.  So it's
> possible for two sync_clients to run at the same time.  There is a
> possibility this will lead to the same data being sent twice.  It won't
> cause corruption, but it will cause excess IO!
> 
> Are you sure it's actually two sync_client instances?  In every released
> version, sync_client forks immediately upon starting and then forks again
> to actually send data.

Yes.  There were a total of 4 sync processes.  Two parents, each with a 
child worker.  The logs showed two sync_client processes taking turns.

> Which is a problem because it opens the BDB environment once and closes it
> many times.  So I've now changed it on my development branch to not fork
> the second time, but just keep it all in once process.  We'll see if there
> are actually any memory leaks left!

Another question I have is what are /var/lib/imap/db/*?  File says they 
are Berkeley dbs, but I thought all databases were skiplist in this 
release.  I can't find the configuration options associated with this 
db.  I noticed them when looking at files open by sync_client.  And not 
that all cyrus process have these files open.

I also noticed that when the child process dies, it tends to take out 
the parent.  Shouldn't the parent catch the error and fork another 
child?  This has so far only happened twice, and in both cases appears 
to be associated with xfering accounts from 2.2.12 to 2.3.16.  In at 
least one of the cases there was a back-end directory that has been 
deleted by the most recent rsync, prior to migration.  A reconstruct 
fixed everything.

Mike
-- 
Michael D. Sofka               sofkam at rpi.edu
C&MT Sr. Systems Programmer,   Email, HPC, TeX, Epistemology
Rensselaer Polytechnic Institute, Troy, NY.  http://www.rpi.edu/~sofkam/