OT: Re: How many people to admin a Cyrus system?

[Scott M. Likens]

Gary Mills wrote:
> Thanks everyone for your responses.  I don't want to clutter up this
> technical mailing list with more management issues, although I'd
> certainly be pleased to receive personal e-mail on this topic.
>
> There appear to be two types of outsourcing.  The Google example was
> one where all of the e-mail resided on an external site.  In addition
> to the issues mentioned above, there is authentication and
> backup/restore to consider.  For all of those reasons, I don't think
> that this type will be suitable here.
>
> The Zimbra example, however, was one where a contractor was hired to
> install a new e-mail system at the university, and to do the migration
> and management.  This one I could see happening here, so that people
> with programming and development skills would no longer need to be
> kept on staff.  That seems like a bizarre idea to me.  It's
> essentially outsourcing the employees.  Since there are no problems
> whatsoever with the existing Cyrus system, I suppose that contracting
> with a company to maintain and manage it might be better than just
> abandoning it.
>
>   

I know I sent out an email earlier I don't know if it got anywhere... 
but I thought I would finish what I was saying, as I had written it out 
to explain it better.

It is certainly OT, and for that you can hate me.

Background, the company that this was deployed at was a rather small 
company.  We had MAYBE 100 employee(s).  Simultaneous connections? 
roughly 50 to 60.  We would max normally at 20 Messages/Minute Incoming, 
and a good amount of that was spam.  Depending on the time of day, it 
would go up and down, but I would say the daily average was around 7-8 
Messages/minute.

I will begin with the Pro(s) and Con(s) of Zimbra.

Pros,
1. Calendar and Contact and Mail Solution rolled into 1 package
2. Webmail loads pretty fast over EVDO
3. It has this wonderful Outlook plugin to make it look like MAPI to 
Outlook (or is that a Con?)

Cons,
1. Commercial Support that sucks... During the day you get someone in 
the US that may fix your problem, but not tell you what they did to fix 
it, or anything.  Additionally at night you get India, and my only 
experience with that was hearing 'My internet is down'...
2. Lack of Code Review... why else would bugs like DROP if exists 
(http://bugzilla.zimbra.com/show_bug.cgi?id=16879)
As well as (http://bugzilla.zimbra.com/show_bug.cgi?id=21117)
3. Can't handle high load very well, in fact it handles load horribly.
4. It uses Java/JSP and Tomcat... (Hateful I know)
5. It uses Postfix.  I guess using Postfix is better then having to 
write SMTP Support in Java. 
6. Multiple MySQL Instances... couldn't that be rolled into 1?
7. It would flag users that sent email using SMTP Authentication as spam 
(amavis-new would check against the rbls and determine that it was spam 
because it came from a Dialup/Broadband IP).  (which is extremely 
stupid, so the only obvious choice was to whitelist the user(s) that had 
this problem)

Note those are just examples, from my experience there are a number of 
things that just don't work right to begin with.

1) Upgrading, is a pain.  It follows the mentality of Redhat, and 
whatnot.  To upgrade you have to shut down Zimbra (duh) and then remove 
all the old rpm's, and then install the new rpm's (or debs in my case).  
Depending on the speed of your server it might take 5 minutes or 30.

2) Handles heavy load horribly.  In my case, we had a Super Micro Server 
with Dual Xeon (P4) 2.2(2.4?)Gigahertz processors.  2 Gigabytes of Ram, 
and was doing a sub-fancy Software RAID-5 setup with about 700gb U160 
SCSI disk(s) that were either 10 or 15k rpm.  (I Forget) Using an XFS 
Formatted Filesystem (I wasn't daring enough to try ReiserFS on this), 
and DRBD+Heartbeat. (not my idea) However it wasn't the end of the 
world.  The server was more then several years old.  But it should have 
no problem handling 350-400messages/minute.  That's not really a large 
amount of email is it?

3) ClamAV.  Do note how much email I said we dealt with a minute.  We 
didn't get a great deal of email.  Maybe 2000 email a day?  Not overly 
much.  However as the ClamAV database would grow, if you restarted 
ClamAV or Zimbra eventually it would take too long for ClamAV to start 
and would not listen on the port assigned and would make mail fail to 
deliver.  (Ouch huh?)

4) If it shuts down uncleanly which it did in my case, (first call to 
support).  I got someone in the US, they logged in via a broadband IP.  
(PTR did not even get close to zimbra.com)  Took them maybe 60-90 
Minutes to find out that there was a lockfile that did not get erased, 
which is why Tomcat was not starting the LMTP Service.  Great, 
fantastic, what was that file? I never knew.  History for that user did 
not show. 

5) Calling at Midnight... _I_ Made an amatuer mistake trying to upgrade 
some switches and caused some mass breakage, and a flood of mail was 
coming in (over 700/minute) and pop!  ClamAV crashed.  Amazing, and my 
boss found a mention on the Zimbra.com forums to raise the timeout in 
some perl script.  He raised it, and it started working.  However, I 
called support, I got India.  Their Internet was up and down (or so he 
said) and couldn't stay logged in long enough to fix it.  Thank god we 
fixed it huh?

I eventually upgraded ClamAV following these instructions 
http://wiki.zimbra.com/index.php?title=Updating_CLAMAV_from_version_lower_than_0.90.0  
It worked, and I had gone for over 2 weeks without an issue, until...

6) Tomcat decided to stop on it's own... Memory Exhausted.  Geeze, don't 
I ever get a break?  The server was never into swap.. so I just 
restarted tomcat and it was flying again.

Now I freely admit things were done on this system that did not make it 
optimal, it was not a brand new server using Hardware Raid.  I did not 
really spend a month reviewing Zimbra's code in detail to find out how 
bad it really would be.  I Installed it and got it running on Ubuntu 
without too much of a problem, and then proceeded to imapsync over all 
the mail after I added the accounts. 

I admit it was a fairly easy migration, I hated the fact that 95% of our 
users were pop3, so the UIDL would not match what they had, so we had to 
make a folder and move all their email to another folder.  Which sadly 
is migration pain, there's no real way around that with the exception of 
sha1. 

Another pain point for us was, we had bought Blackberr(y|ies), and we 
wanted to have our Calendar, Contacts and everything pushed to the 
Blackberry in a sane and easy way.  We checked out NotifyLink, it was a 
bit overpriced, and they had no Linux Solution.  We had 0 Windows 
Servers, and had no desire to have any.  I understand now they have a 
BES Solution that integrates the BES Server into Zimbra.  (or I saw bugs 
pertaining to it).

Now I am certain that Zimbra will work for some people just perfectly.  
I was not one of them, perhaps it's because I want too much control? I 
don't really know.  I did certainly question if they had gotten 
anyone(s) permission to modify and use their programs in Zimbra.  
Because I know certainly it's easy to whip together a bunch of F/OSS 
Software to do exactly the same thing as Zimbra, without the 
fluff/hassle.  Just think OpenLDAP for Contacts, and then an iCal or 
CalDAV Server.  There's many out there that are pretty easy to setup 
including Apple's CalDAV Server.  So right there you just gave the 
functionality of Zimbra with just about anything without their 3rd party 
crap.  Truthfully do you need those ugly 3rd party things?  Nope.

With Outlook you can configure an LDAP Directory Server, and it's not 
impossible to make Outlook use an iCal server.  So in the end, I would 
have done things much differently the first mistake is to Lease Zimbra.  
(You don't ever buy it, as you have to pay them yearly).  Which is why 
they are alive, and Scalix is dying.

Zimbra does have some challengers, such as Oracle Collaboration Suite 
(http://www.oracle.com/collabsuite/index.html) Oracle Collaboration 
Suite, and Postpath (http://www.postpath.com) so quite frankly there is 
alternatives. 

Truthfully as I have looked at some of the stories in here (UCDavis, 
Fastmail.fm) and their success and size alone.  It makes me sad about 
Zimbra, and I do hope that they can turn it around.  But I just feel 
they are out of touch with what their users want and need.