Running multiple reconstructs concurrently

[Bron Gondwana]

On Mon, Jul 27, 2009 at 09:39:10PM -0400, Jack Neely wrote:
> On Tue, Jul 28, 2009 at 07:40:08AM +1000, Bron Gondwana wrote:
> > On Mon, Jul 27, 2009 at 11:19:00AM -0400, Jack Neely wrote:
> > > Folks,
> > > 
> > > I'm writing a script to migrate all the cyrus imap data from version
> > > 2.2.10 running on 32 bit RHEL 3 to version 2.3.11 on brand new servers
> > > running RHEL 5 in 64 bit mode.  The data moves fairly quickly but
> > > running reconstruct over the entire server takes 3 or 4 times longer
> > > than scp'ing across the data. 
> > > 
> > > Is it safe to run multiple reconstructs concurrently?  Provided they are
> > > not working on the same mailboxes, of course.
> > 
> > Yes, that's fine.  There will be some lock synchronisation on the 
> > mailboxes.db, but the bulk of the time it will be fine.
> 
> Great!  This takes my test runs from 5 hours to 3.
> 
> > 
> > By the way, 2.3.11 is pretty old for a 2.3 series.  There have 
> > been bugs fixed since.
> 
> Indeed there are several relevant things in the changelog.  Thanks.
> 
> > 
> > By the way number two, I hope you're using -G to the reconstructs
> > so you get GUIDs calculated (that will add process use, but is
> > worth it for integrity checking purposes...)
> 
> I'm not currently using any replication so I had not planned to turn on
> the GUIDs.  Are there any other advantages to them?

They're the sha1 of the message file - it allows you to detect corruption.
Not that we have good tools for that yet (not in Cyrus anyway - I have
"audit_slot.pl" here at FastMail which can do some pretty clever stuff),
but I'm hoping to build better tools in future.

Of course, without replication you can't actually do much about it, but
at least you notice.  We've probably had about 10 cases of corrupted
files in the past 6 months, to give you an idea how rare it is (that's
across 12 machines and heaps of disk)

Bron.