copying and renaming mailboxes to clean up directories

[Andrew Morgan]

On Wed, 2 May 2007, Thorsten B=FCker wrote:

> Dear all,
>
> For quite a few month I ran a backup-scenario set up according to [1]
> including synchronisation of the mailboxes using "rsync -vaR". (In the
> meantime I migrated to the usage of LVM snapshots.)
>
> As restoring a backup became necessary due to filesystem corruption I
> fetched the mailboxes from the r'synched folders as well as the mailbox =

> databases and further necessary stuff. The restore went smoothly.
>
> But wondering about the heavily increased amount of partition in use, I
> noticed that my call of rsync lacked of the "--delete" attribute. In =

> consequence the restored mailbox directories include not only the message=
s, =

> which should be in the inbox, but all files beeing in the directories dur=
ing =

> the backup cronjobs. (My mailbox sizes ~700 MB instead of ~100 MB.)
>
> With regard to backup performance I'd like to clean up the relevant
> directories by ensuring uniformity of the inbox according to the Cyrus
> mailbox database and the files in the mailbox directory.
>
> In the wiki I found Andrew's collection of scripts [2] and had a try. My =

> intention is/was to firstly manually select mailbox directories, which =

> contain much too much unneeded files -- mainly mailboxes used via imap. A=
fter =

> temporarily stopping mail delivery and mailbox access, each chosen mailbo=
x =

> might be renamed and its contents copied to a new mailbox of its initial =

> name. After starting mail delivery and access again, the mailboxes and =

> directories containing wrong fails might be deleted at all.
>
> But, as usual, the idea doesn't work yet. A first try to copy the
> existing mailbox "thorsten" to "thorsten_neu" leads to the following:
>
> ~~~~
>
> mail:/var/spool/cyrus/mail/t/user#
> /usr/local/sbin/cyrus/copy_user_mailbox.pl thorsten thorsten_neu
> Successfully created inbox for thorsten_neu
> Successfully set quota for thorsten_neu to 102400
> "user.thorsten"
> Copying folder user.thorsten to user.thorsten_neu
> "user.thorsten.Archiv"
> Copying folder user.thorsten.Archiv to user.thorsten_neu.Archiv
> "user.thorsten.Drafts"
> Copying folder user.thorsten.Drafts to user.thorsten_neu.Drafts
> "user.thorsten.Mailinglisten"
> Copying folder user.thorsten.Mailinglisten to
> user.thorsten_neu.Mailinglisten
> Error copying messages to user.thorsten_neu.Mailinglisten
> "user.thorsten.Mailinglisten.Announces"
> Not connected at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 104
> Error sending '24 SELECT user.thorsten.Mailinglisten.Announces' to IMAP:
> at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 104
> Copying folder user.thorsten.Mailinglisten.Announces to
> user.thorsten_neu.Mailinglisten.Announces
> Not connected at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 129
> Error sending '25 STATUS user.thorsten.Mailinglisten (MESSAGES)' to
> IMAP:  at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 129
> Use of uninitialized value in numeric eq (=3D=3D) at
> /usr/local/sbin/cyrus/copy_user_mailbox.pl line 129.
>
> [.. various errors at lines 104 and 129 ..]
>
> Not connected at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 152
> Error sending '84 LOGOUT' to IMAP:  at
> /usr/local/sbin/cyrus/copy_user_mailbox.pl line 152
>
> ~~~~
>
> Please note, the additional output on the folder name results from
> uncommenting line 92 of the script. The relevant sections look like
> below [6]. Evaluating, why the connection to Cyrus gets lost while loopin=
g =

> through the folders, a discovered "word too long" in /var/log/mail.log.
>
> Well, in this case (my personal inbox) erasing various mails inside =

> "user.thorsten.Mailinglisten" (~2200 mails) helped to make copying work. =
But =

> regarding customer's inboxes this might not be first choice ;-)
>
> In google I found [3], suggesting to recompile imapd. Alas, is there any =
way =

> to modify the script instead of recompiling imapd to permit a larger valu=
e? =

> Indeed I'd prefer to keep sarge's "2.1.18-1+sarge2".

Ack, someone using my scripts!  :)

I bet the problem is that on a large mailbox, $imap->messages will return =

a really big list of UIDs (longer than the allowed MAXWORD of 32768).  You =

may be able to fix this by calling "$imap->Ranges(1);" earlier in the =

script.  According to the Mail::IMAPClient docs, with that enabled it will =

try to generate a condensed list of UIDs when possible.

However, if that still does not work then you may have to add a loop to =

the code to operate on "chunks" of UIDs at a time, rather than the whole =

list at once.  Instead of:

         my $uidlist =3D $imap1->copy($targetf, $imap1->messages);

you'd do something like:

 	@alluids =3D $imap1->messages;
 	foreach $uid (@alluids) {
 		$uiddone =3D $imap1->copy($targetf, $uid);
 	}

There is probably some smarter way to use bigger chunks of uids rather =

than 1 at a time.

> Furthermore I tried to rename a mailbox via cyradm in advance, but this l=
eads =

> to the following:
>
> ~~~~
>
> mail.myhostname.de> renm user.thorsten user.thorsten_orig
> renamemailbox: Operation is not supported on mailbox
>
> ~~~~

Did you set "allowusermoves: 1" in imapd.conf?

> The amount of results by google is rather poor, honestly spoken one resul=
t =

> [4]. The presented workaround [5] contains quite a lot of work dealing =

> manually with each subfolder. Is there any other way to rename a mailbox?
>
> Or, going even further, is there any more clever attempt than the one abo=
ve =

> to solve the problem? Thanks in advance!

If I understand you correctly, you want to get a list of "valid" UIDs from =

the folder listing, and then delete (from the filesystem) anything which =

is not in that list of "valid" UIDs?

Or do you intend to just copy the mailbox to a new mailbox, delete the old =

mailbox, and copy it back?  That sounds pretty reasonable to me.  One =

problem you might have is the loss of the seen state for each mailbox =

because the mailbox unique ID will change with the copy.  The unique ID is =

stored in the cyrus.header file in each mailbox/folder, so you might be =

able to save those and copy them back as you go.

An ugly problem!

 	Andy