UC Davis Cyrus Incident September 2007

[Scott Adkins]

--On Thursday, October 18, 2007 9:58 AM +0200 Pascal Gienger <Pascal.Gienger at uni-konstanz.de> wrote:

> Scott Adkins <adkinss at ohio.edu> wrote:
>
>> Meanwhile, we hacked around this in a very cool way.  We copied the imapd
>> process 60 times (assuming average of 12,000 processes, shooting for 200
>> processes per executable, that is 60 individual executables).  These were
>> named /usr/cyrus/bin/imapd_001 through /usr/cyrus/bin/imapd_060.  We then
>> symlinked the "imapd" binary to imapd_001.  We then wrote a cron job that
>> ran once a minute and relinked the imapd symlink to the next numbered
>> executable,
>
> And how did you solve the problem of the "deadlock" resulting in deleteing the symbolic link, setting the new one? Between these
> events an exec of imapd would break resulting in an error to the customer.
>
> Funny hack though.

Actually, I don't see a deadlock situation at all... I am guessing that
theorettically, it is possible... but the "ln -sf" option makes the
overwriting of the symlink an atomic action (as much as it can), which
is why it seems to work.  I stress tested this as much as I could prior
to putting it into production, simulating tons of users logging in and
out of the server while rotating the symlink as fast as I could.  There
was not one case of a connection failure at any point during the test.

There are other factors that are involved too... Process re-use keeps a
lot of problems to a minimum... We don't use preforking, but I imagine
that preforking would further reduce any possibilities of deadlocking.
Users coming into the system would be funneled to a socket associated
with an IMAP process that has already been running for a bit, so it is
clean from their perspective.

The only issue is actually between master and imapd, and I don't know what
master does if it tries to spawn an imapd and it fails... it may just try
again... I dounbt it just out-right refused the incoming connection.  I
need to look at the source code to understand it better.

Scott
-- 
 +-----------------------------------------------------------------------+
      Scott W. Adkins               Work (740)593-9478 Fax (740)593-1944
   UNIX Systems Engineer                 <mailto:adkinss at ohio.edu>
 +-----------------------------------------------------------------------+
     PGP Public Key <http://edirectory.ohio.edu/?$search?uid=adkinss>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 229 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20071018/25686fb0/attachment.bin