Performance and cheap storage

[Bron Gondwana]

On Fri, Aug 04, 2006 at 03:26:39PM +1000, Robert Mueller wrote:
> Anyway the good news:
> Before: 2 frontend servers with 7000+ connections (eg 14,000+ total) using 
> 6G of RAM with a load on each of about 2
> After: 1 frontend server with 14,000+ connections, less than 1G of RAM 
> usage, load of under 0.5
> 
> The only problem is that all the docs are in Russian. I should get around 
> to writing up a document on setting it up, it's not that hard. The main 
> issue is that you have to write your own authentication daemon. It doesn't 
> connect to anything standard like mysql or LDAP, but it does use a simple 
> protocol you can write your own adaptor for quickly in something like perl.

Actually, one other scalability issue you should be aware of (and we'll
have to actually look into again soon...)

Each backend imap server is only listening on one address:port, e.g. in
our case my imap server listens on 10.x.x.5:2143 internally.

Your frontend only can make connections out using any port it likes, but
there are only 65k of them, and at any one time, a fraction of those
will be tied up doing other things.

We actually had to set a TCP parameter on our backends 
(net.ipv4.tcp_rfc1337 = 0) so that when a busy frontend re-used a socket
pair for a previously closed connection it didn't just get dropped
immediately and return a "connection failed" back to nginx and hence the
client.

That's fine for now, but if we ever hit 65k connections from a frontend
to a single imap server (eek) then we suspect that it will either lock
up trying to bind the connection or nginx will fail back to the client,
not a good look.

(my just-this-second thought involves multiple listening ports on the
IMAP backends with multiple imapd lines in cyrus.conf and a randomiser
in the nginx authentication proxy, but I haven't talked to Rob about
that yet... since I only came up with it as I wrote this!)

Bron.