Yet another mail-restore question...

[Earl Shannon]

Hello,

Sadly we've a little experience in IMAP server recovery.
Most of what I'm listing makes common sense but I'll say
it anyway.

How to quickly get back a once working server depends upon
whats wrong with it. We've not had any problems with software
failing but have had hardware bite us(me) in the butt.
The major concern we have is restoring a server in the event
it goes completely south. We were looking at this recently
from the wrong side of the event. The RAID controller in 
the server was not a redundant controller and its battery
was failing. When I attempted to replace the battery I only
made things worse. The machine lost the file system with
the mail store and configuration files on it. While 40 Gb
of data may not sound like much by todays standards anymore,
wait until you lose it and are looking at recovering it.
( And with a 10 Mb quota limit currently, you figure out
how many users were somewhat upset. And yes, we are very 
overbooked for quota on our machines. Believe it or not
the students can get by with the 10 Mb limit. Staff on the
other hand, who have 20 Mb, are still not satisfied. )
We took a two pronged approach to see which would come back
faster, rebuilding the filesystem on the current machine
using fsck and selected restorations, or simply building
a new machine and restoring to it.  We actually had the
fsck'ed machine back before the restoration was done, but
decided that a reconcstruct on each individual mailbox should
also be done. That took almost another twelve hours.

Building the new machine was pretty straight forward in this 
case since we would have been able to simply take the
system disk from the old machine and put it in the new machine
and just rebooted with the new RAID unit. But we also have
an IMAP install "kit" we can use to create a new IMAP server
when we need one. We would then have done the restoration to
the newly installed machine. I don't know from experience if
any of the Bare Metal recovery methods would have done the
restoration any faster. I do believe that the system disk if
lost can probably be recovered significantly faster via Bare
Metal methods ( Such as what Veritas markets ) but we do not
yet have the technology in house here.

This has made me painfully aware that large file systems,
while nice to store lots of stuff, have severe drawbacks when
it comes to replacing them from backup. We are wanting to
get transitioned to a journaling file system in order to help
reduce fsck times on the occasional/unscheduled reboot. 

The bottom line is you have to know your system and its requirements
for recovery. Making public a document about how to restore
service in the event of a disaster would only give you
an outline to use at your site since you would probably use
different backup methods, have different OS you were using,
etc. And you should really be able to come up with the outline
yourself.

Regards,
Earl Shannon
-- 
Systems Programmer, Computing Services, Information Technology
NC State University.
http://www.earl.ncsu.edu



Bryntez wrote:
> 
> Has anybody on the list made a detailed doc about
> howto restore the maildb in case of disaster ?
> I mean a short, quick note with step-by-step commands
> to execute, to quickly get back in business ?
> 
> If for some reason the system crashes, it sure would have
> been nice to have a doc at hand to quickly restore the
> imap-system, saving time, and not having to browse the whole
> documentation when the sweat are dripping and the stress-
> level are at a dangerous level :-)
> 
> A copy of such doc would have been reassuring....
> 
> (Running Cyrus imap/sasl 2.1.5 on RedHat 7.3)
> 
> Regards
> bryntez