How many people to admin a Cyrus system?

[Bron Gondwana]

On Fri, Nov 09, 2007 at 08:24:00AM -0500, Adam Tauno Williams wrote:
> > > How many
> > > and what sort of people does it take to maintain a system such as
> > > this?  I need a good argument for hiring a replacement for me.
> > At a minimum you want 1 qualified person and someone cross-trained
> > as a backup, so that person can reasonably enough have vacations.
> > Any decent sysadmin should be able to MAINTAIN such a service
> > I don't think actually programming skills should be primary.  
> 
> Agree.  I maintain a Cyrus system.  And on most days that doesn't even
> involve touching it.  Any reasonably proficient person with UNIX skills
> should be able to take over Cyrus administration given they are willing
> to do some reading.
 
I maintain a Cyrus system and it's taken over my life!  Yikes.

Summary for this week:

* crc32 for indexes hummning along in the background.
* getting more skiplist corruptions - going to have to post about this.
  We lost 4 mailboxes.db files this week, 3 during controlled failover,
  one during the night when nobody was working on it.  Suspect new ACL
  feature on the web interface which allows more frequent updates is
  causing issues.
* the one buggy skiplist I actually still have a copy of, the "logstart"
  value in the header is wrong, causing recovery to fail with only a few
  of the records still reachable because it hits another INORDER record
  rather than the ADD record and drops out.  I've got the monitoring
  system set up to let me know if it finds any other skiplist errors and
  take a copy of the offending file.

> > I have been
> > doing sysadmin work since 1989 and the actual programming work I've
> > done in that time has been maybe 2% of it.  If you have a lot of custom
> > interface stuff to your campus systems maybe you need more programmer
> > skills.   As a completely inappropriate generalization, former engineers
> > and mathematicians also make good sysadmins because they have the mindset
> > and the skills for problem decomposition and trouble-shooting.
> 
> Yep.

Agree there.  Sysadmin has always been a fraction of my work because I
tend to do a lot of "glue" programming to abstract away anything that's
sysadmin work.  My first really major project (after converting us from
CVS to Subversion) was making all the servers install automatically from
PXE boot and the configurations all set themselves up with "make
install" from the Subversion repository, so that most everyday sysadmin
is now automated - just update the master config file, roll it out,
restart affected services.

So day to day we need less than one sysadmin, but of course incident
response is unpredicatable, and having two good sysadmins (Rob and I
share sysadmin responsibility here) available is very handy.  Both
for the "you can let one of them have a holiday" point of view and
the "two heads are better than one" ability to work past the other's
mental blocks and avoid getting stuck in a rut trying to solve 
problems.

Bron.