Cyrus IMAP 2.3.9 on Solaris 10 with ZFS and SAN

[Rob Mueller]

> http://cyrus.brong.fastmail.fm/patches/cyrus-statuscache-2.3.8.diff
>
> This cuts back on meta IO traffic considerably for repeated SELECT
> calls on mailboxes which are unchanged.  We are in a similar boat,
> and it makes a huge difference (also for some other clients)

That should be "cuts back on meta IO traffic considerably for repeated 
STATUS calls on mailboxes which are unchanged", not SELECT calls. That's ok 
though, because the reality is that most clients do many, many more STATUS 
calls than SELECT calls. Basically every client that has a "updated unread 
count on mailboxes every 5 minutes" is calling STATUS on every mailbox every 
5 minutes.

If you're getting the "unseen" count, that means stepping through the 
cyrus.index file of every mailbox every 5 minutes, which if you've got lots 
of users, with large mailboxes, each with an IMAP client doing a status 
every 5 minutes on every mailbox, means you could be forcing re-reads of 
many gigabytes of data every few minutes. If your cache memory isn't big 
enough, your IO will jump enormously.

Basically this patch significantly reduced the amount of read IO from our 
meta partitions.

> > of passing FDs around between processes.  I don't know how it works
> > precisely, but it uses Socket::Pool which had a sourceforge project
> > and doesn't appear to any more.  My guess is that I'd rather not
> > look at the code for fear of going totally insane.
>
> You can pass file descriptors over UNIX domain sockets, see SCM_RIGHTS.
> Here is some code Google found ->
> http://search.cpan.org/src/SAMPO/Socket-PassAccessRights-0.03/passfd.c

It does use this underneath. It's not that magic (eg fork() automatically 
copies file descriptors from one process to another, why shouldn't there be 
an explicit way to do that between non-parent/child processes), though it's 
not that commonly used either, and the usage api is a bit weird (strange 
messages via unix sockets). I haven't seen many programs that use it since 
you have to be quite careful with it (eg you can't read any data from a 
socket before you read the file descriptor or it breaks), and there have 
been bugs with many different OS's in the past (even linux 2.4 had a very 
intermittent bug for a long time in our testing, which is why we had a "try 
this 3 times" loop on a bunch of code).

In our case it is very useful, since our imappool daemon keeps track of the 
server + user + currently selected folder, we basically call the imappool 
daemon to get a socket for a server/user combination, and it returns us one 
(connects to the server if needed), and tells us the state (eg connected but 
not logged in, logged in, logged in and which folder is selected). It then 
"locks" that socket in it's pool so nothing else can use it (eg another web 
process). We then do whatever we need with the socket, and when done, we 
tell the imappool to "unlock" the socket, and we also tell it the new state 
it's in (eg which users/folder). That way, the imappool always has the 
socket (we don't need to pass it back), but ensures that only one process 
can use it at a time. The main issue is that the code has to be changed to 
explicitly contact the imappool to get a connection and free it when done, 
it's not transparent.

> In our installation it is not possible to log on as cyrus or other admin
> users from outside, I patched the imapd to check the caller's ip address -
> and the firewall does the rest so that no forged packet (with an inside
> sender ip coming from outside...) will trigger it.

We use nginx as a frontend proxy. For every IMAP/POP login, it "calls out" 
to a process that allows you to authenticate and/or authorise the login (you 
can return any error message you want, and it gives you information like 
login name, password and ip address - 
http://wiki.codemongers.com/NginxMailCoreModule), and what backend server + 
port to connect to. It uses a small process pool + epoll to easily handle 
10,000's of connections.

http://blog.fastmail.fm/2007/01/04/webimappop-frontend-proxies-changed-to-nginx/

Rob