Old mail rescue - can anyone help ?

[Craig Ringer]

Robert Peckham wrote:
> I am a Cyrus user by proxy - it's the email server now built-in
> to Apple's Panther (OSXv10.3) Server, so I hope this posting
> is allowed !

They picked a good mail server then :-)

> Early on Monday morning, the Cyrus Mail folders became corrupted
> and would not be reconstructed. This was quickly followed by the server
> directories corrupting, then mapping errors - the result being a
> complete re-install of the entire server from scratch.

Nasty. Filesystem corruption maybe. It's probably an HFS+ filesystem ;-) 
*nudge*

> Is there a way to re-import this mail into the new mail stores ?
> Or at least turn it all into text files for use ?

Note: I'm no Cyrus expert, but can offer a few suggestions. I can also 
suggest more info that others might need to know to better help you. As 
this is an Apple-modified system, take all this carefully. The following 
instructions are a bit technical and I assume you have some familiarity 
with working in a UNIX shell (the "Terminal" program).

It's probably not a bad idea to just look for now, and touch only if you 
know you have to. Preferably once others have had a chance to offer a 
little advice (or call me on any errors I may have made).

Apple has decent, if rather limited, documentation on their Cyrus system 
(and Postfix) available here:
http://a1328.g.akamai.net/7/1328/51/6227ccedc088b7/www.apple.com/server/pdfs/Mail_Service.pdf
It's probably worth a read, as is
http://www.apple.com/server/documentation/
in general. If you haven't already, that is.

According to their docs, Apple have stored the cyrus config files in 
their normal locations. See if you can view "/etc/cyrus.conf" and 
"/etc/imapd.conf", but DO NOT MODIFY THESE FILES. If you can, reply with 
those files attached to an email as it could be useful.

 From their documentation, it appears that Apple has not modified Cyrus 
too heavily - at least in terms of where and how it stores things. This 
makes things easier.

In the Cyrus email system, the individual messages are already text 
files. Your backup should contain a folder called "user" (unless Apple 
have renamed it) with your mailboxes in it. It's probably 
'var/spool/imap/user' in the backup archive. If you unpack that 
somewhere like your /Users/Yourname folder, you should see a bunch of 
folders under it:

	user/username1
	user/username2

etc, where username1, username2 etc are the user part of email accounts 
on your system. Please DO NOT "restore" the backup of your mail spools 
over the current one - it'll make a nasty mess, stuffing up the 
numbering scheme, probably corrupting the mailbox and possibly 
overwriting messages. Restore the old mail somewhere totally separate, 
then copy back select folders.

Within the actual user folder, there should be a few files with  names 
beginning with 'cyrus.' and a lot of files of the form 'number.' eg '1.' 
, '10.', etc. Each numbered file is a message, and can simply be opened 
in a text editor like SimpleText/TextEdit.

Be aware that attachments will be in the body of the message as huge 
masses of alphanumeric gibberish, and some text editors have real 
problems opening multimegabyte text files. Other than causing issues 
with text editors they cause no issues. If you simply can't view a file, 
try doing this in the terminal:

	cd /var/imap/username/
	less 10.

(where 'username' is the username, and '10.' is the numbered message 
file you want to view) as it'll let you progressively view the message. 
You have to hit enter at the end of each line. Alternately,
	
	cd /var/imap/username
	head -500 10. > "/Users/Yourname/some-message-10"

will save the first 500 lines of a message into a separate file in your 
home folder.

If it complains about 'permission denied,' you probably have to be the 
'cyrus' user. 'su cyrus' will let you become the Cyrus user on OSX; if 
not, 'sudo su cyrus' may. If you have to, become root then run 'su 
cyrus' to become the cyrus user. BE AWARE that the wrong typo can cause 
damage to your mail system and possibly data loss - be careful.

It's possible to restore user mailboxes so they can be accessed using an 
email client. I'd restore their mail to an "oldmail" folder, so the old 
'user/craig' becomes 'user/craig/oldmail' ; 'user/craig/Sent' becomes 
'user/craig/oldmail/Sent', etc. This should be pretty easy with your 
backup program, but BACK UP YOUR CURRENT EMAIL SPOOLS FIRST - just in case.

Cyrus won't immediately see the new mailboxes once they've been 
restored. You need to run 'reconstruct -r -f user.craig' (for example) 
to force Cyrus to search for restored mailboxes and add them to the 
mailboxes.db . That's a Terminal command, by the way, and may need to be 
run as a special user - see the Apple documentation for details, as they 
describe the reconstruct command there. The user may then need to 
subscribe to the mailboxes using their email program before they're visible.

If you can't access the restored mailboxes - they appear, but don't work 
in the email program - try deleting the cyrus.* files out of the 
restored mailboxes ONLY. Then re-run the reconstruct command and see if 
you have any luck.

Do not ever delete mailboxes using the Finder, unless they're definitely 
not in the Cyrus mailboxes list. It will mess up your mailbox list and 
confuse Cyrus. Basically, unless you just put the mailbox there using 
the Finder (or Terminal) and haven't run reconstruct yet, it's not safe 
to remove. You must instead use an email program or the Cyrus admin 
interface to delete the mailbox - I don't know how Apple have done it 
with OSX, whether there's a GUI tool or if you use good 'ol cyradm.

Oh - if you have problems at any stage, there's a good chance the file 
permissions are wrong. In a normal Cyrus install all Cyrus files must be 
owned by user 'cyrus' - and it's likely Apple has stuck to that. If your 
restored mailboxes won't work you may have to change them so that 
they're owned by the 'cyrus' user.

> Under OSX Panther, Cyrus puts incoming user mail in a private
> (hidden) spool folder, which nothing can open (as far as I can tell)

I expect it's accessable using the Terminal app, so long as you know 
where it is. According to Apple documentation, the mail spools are 
stored in '/var/spool/imap' just like a normal Cyrus install. DO NOT 
MODIFY THIS FOLDER unless you know what you're doing.

Anyway, it's late and I need sleep. I hope I've been of some help.

Craig Ringer