Performance and cheap storage

[Robert Mueller]

> not sure if we qualify as big enough, but here goes:  we typically have
> 3000 concurrent TLS/SSL connections on each Perdition server during peak
> hours (although we occasionally see 5000), but the CPU impact is
> negligible[1].  at peak, 8% system and 12% user out of 400% CPU
> available (this is Dell PE2650 with dual Xeon 1.8 GHz HT) and just 15
> Mbps of network traffic in and out -- quite symmetric as you would
> expect for a proxy server.  Perdition's connection to Cyrus is
> unencrypted, of course.
>
> [1] the main issue is RAM for all those processes. with just 2 GiB RAM
> it ran sluggishly once in while during lunchtime, they now have 4 GiB
> each.

We used to use perdition. The problem we found was that even with Linux 2.6 
and the O(1) scheduler, when you get to 7000+ processes, the entire box and 
all the connections feel sluggish and the machine has a load of around 2 or 
so, even though it's mostly CPU idle. RAM usage is also huge (3G or so). It 
seems that just handling the context switching and management of all those 
processes is a bottleneck.

Now by coincidence, about a year ago we were in contact with a guy called 
Igor Sysoev (who developed a module called mod_accel for apache) and he had 
started building a new http proxy system based on a non-blocking 
architecture using epoll/kqueue called nginx. We spoke with him about 
extending the proxy to work with POP/IMAP. We negotiated, came up with a 
price, and he built the appropriate code for us. It's now incorporated into 
the public release of nginx itself (http://sysoev.ru/nginx/). As an aside, 
we've done a similar thing with cyrus, sponsoring Ken to write various 
extensions that made it into 2.3 before he moved to CMU. I really think this 
is a great way to support open source projects and the people involved with 
them.

Anyway the good news:
Before: 2 frontend servers with 7000+ connections (eg 14,000+ total) using 
6G of RAM with a load on each of about 2
After: 1 frontend server with 14,000+ connections, less than 1G of RAM 
usage, load of under 0.5

The only problem is that all the docs are in Russian. I should get around to 
writing up a document on setting it up, it's not that hard. The main issue 
is that you have to write your own authentication daemon. It doesn't connect 
to anything standard like mysql or LDAP, but it does use a simple protocol 
you can write your own adaptor for quickly in something like perl.

Rob