Skiplist / best practice for 2.1 branch

[Paul M Fleming]

I've just finished doing some benchmarking and general server abusing
using ZD Labs IMAP test tool
(http://www.etestinglabs.com/benchmarks/svrtools/email/t1intro.asp) and
lmtpd performance makes sense. Each inbound message goes through lmtpd
which does duplicate suppression (by default) meaning each deliver
requires a lookup and more than likely a write. Doing a write requires a
LOCK which costs -- on any os. I found 8-10 was an upper limit on the
number of lmtp processes otherwise they just sit and spin waiting for a
LOCK. With a limit of 10 lmtpds I was able to get 8-10 messages / second
inbound. Using an mta upfront helps manage the backlog (queuing). Bottom
line if I didn't limit lmtpd the whole thing went down the drain -
quickly, but with cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=5
maxchild=10 I was able to hammer the server at 5-7 messages/second for
hours. 

I'm using 2.1.10 w/ a a custom auth (regular expression based - another
note when I get time) module everything else vanilla and I could hammer
out 8-10 messages/second inbound (LMTP) and upwards of 20-30 outbound
(IMAP). This is on a RedHat 7.1 box. I'm going to write this mess up
when I'm done .. beating up murder tonight -- appropriate name cause
it's murder ;-)

name       : Cyrus IMAPD
version    : v2.1.10 2002/11/11 22:12:50
vendor     : Project Cyrus
support-url: http://asg.web.cmu.edu/cyrus
os         : Linux
os-version : 2.4.18-17.7.x
environment: Cyrus SASL 2.1.9
             Sleepycat Software: Berkeley DB 3.1.17: (July 31, 2000)
             OpenSSL 0.9.6 24 Sep 2000
             CMU Sieve 2.2
             TCP Wrappers
             mmap = shared
             lock = fcntl
             nonblock = fcntl
             auth = regexp
             idle = poll
             mboxlist.db = skiplist
             subs.db = flat
             seen.db = flat
             duplicate.db = db3-nosync
             tls.db = db3-nosync




Rob Siemborski wrote:
> 
> On Mon, 18 Nov 2002, Patrick Boutilier wrote:
> 
> > I tried those stability patches and they didn't help. The only thing
> > that stopped lmtp from hanging was switching from db3_nosync to
> > skiplist. What are the advantages of using db3_nosync over skiplist?
> 
> db3_nosync uses the Berkeley DB backend, which should be faster than
> skiplist for random access.  Also, since the duplicate delivery database
> doesn't *need* consistancy, the lack of syncing to disk (which skiplist
> probably does more than is strictly necessary) makes it much faster to
> update than skiplist ever could be.
> 
> Skiplist's main advantage is that it can enumerate the database much
> faster than the other backends (except for flat, possibly), and avoids the
> massive cost of updates present in flat.
> 
> -Rob
> 
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456
> Research Systems Programmer * /usr/contributed Gatekeeper