IMAP sync tool (rsync for IMAP)

[Rich Graves]

Florin Andrei wrote:
> I'm currently using two IMAP accounts, one on Cyrus-2.2 the other on 
> Cyrus-2.3.
> 
> The one on Cyrus-2.3 will get decomissioned so I need to transfer all my 
> email, preserving the folders/subfolders tree, under a specific folder 
> (oldmail/foo/bar) on the 2.2 server.

You need imapsync or similar if you are in a multimaster situation 
(syncing changes made to *both* ends) or if the IMAP server software 
differs, but if you have shell access to cyrus on both ends, rsync 
(literally) should work fine. If the folder is quiescent and you're 
using platform-independent skiplist databases, then you shouldn't even 
need to run reconstruct.

A few months ago, I rsync'd 500GB of mail from Solaris/vxfs/Cyrus 2.2 to 
Linux/ext3/Cyrus 2.3 with 15 minutes' downtime, reorganizing the 
partitions a bit along the way.

Old server: partitions /var/spool/imap (partition-default), 
/var/spool/imap1 (partition-1), /var/spool/imap2 (partition-2), 
/var/spool/imap0 (partition-0).

"partition-0" caused various problems with cyradmin (0 is false to 
perl), so I renamed it to partition-3 on the new server. At the same 
time, I renamed partition-default to partition-4.

New server: /var/spool/imap{1,2,3,4} (partition-{1,2,3,4}). New server 
also has metapartition_files: squat with all squat indices stored on a 
low-cost SATA LUN.

After rsyncing the mail partitions, imap/user, and imap/sieve 
directories (several times over several days, with services stopped for 
a final incremental sync early one morning), I just needed two minor 
changes to mailboxes.db. Permissions s/lrswipcda/lrswipkxtea/g, and 
prepend "0 " to the the partition column (murder node, I guess).

#!/usr/bin/perl
# mbxlist.pl -- translate mailboxes list from cyrus 2.2 to 2.3 format
unlink "mailboxes.db";
open OLD, "ssh cyrus at old-server /opt/cyrus/bin/ctl_mboxlist -d |";
open NEW, "| /usr/lib/cyrus-imapd/ctl_mboxlist -u";
while (<OLD>) {
  ($mbox,$part,$acl) = $_ =~ m/^([^\t]+)\t([^\t]+)\t(.+)\n/;
  $part =~ s/ +//g;
  $part eq '0' and $part = 4;
  $part eq 'default' and $part = 3;
  $path = $mbox;
  $path =~ s#\.#/#g;
  @subdirs = split /\//, $path;
  $path = "/var/spool/imap${part}/$path";
  if (! -e "$path/cyrus.header") {
    # There were some empty directories/invalid mailboxes on the
    # old server, legacy of previous sysadmin errors. Skip those.
    warn "No $path for $mbox\n";
    next;
  }
  $dir = shift @subdirs; # drop leading /user/
  $start = "/var/spool/imap-meta/$part/user";
  while ($dir = shift @subdirs) {
    $start = "$start/$dir";
    mkdir $start;
  }
  print join("\t",$mbox,$part,$acl) . "\n";
  $acl =~ 's/lrswipcda/lrswipkxtea/g';
  print NEW join("\t",$mbox,"0 $part",$acl) . "\n";
  $i++;
}
if ($i < 32000) {
   die "FATAL: Too few mailboxes!\n";
} else {
   exit 0;
}
-- 
Rich Graves <rgraves at carleton.edu>
Sr UNIX and Security Administrator
Ofc 507-646-7079 Cell 952-292-6529