lmtpd: deliver.db checkpointing

[David Mayo]

On 15/11/2010 14:46, Patrick Boutilier wrote:
> On 11/15/2010 10:36 AM, David Mayo wrote:
>>
>> Having looked at the source code, we can see that ctl_cyrusdb does not
>> touch deliver.db and it is not possible to force checkpoints on or off
>> within lmtpd. Ideally we want to schedule the checkpointing to a "less
>> sociable" time.
>>
>> On our setup, the checkpoint runs every couple of days and usually takes
>> 4-6 minutes[1]. We run cyr_expire every night to remove entries older
>> than 3 days from the duplicate deliveries database but this is the only
>> maintenance we can schedule on this database.
>
> No solution, but I wonder why your checkpoint takes so long? With
> similar DB size it take only seconds here.
>
> Nov 8 21:18:04 student2 lmtpunix[6541]: skiplist: checkpointed
> /var/imap/deliver.db (650493 records, 67180160 bytes) in 5 seconds
>
> Nov 12 23:14:06 student2 lmtpunix[26425]: skiplist: checkpointed
> /var/imap/deliver.db (768262 records, 80010936 bytes) in 11 seconds

Looking back further through our logs, I found a checkpoint that was run 
overnight which completes somewhat faster, but nowhere near your times! 
Our IMAP back-end is running on a 4x Quad-Core AMD Opteron 2.3 GHz 
processor with 16 GB RAM:

Oct 14 00:17:47 imap.bath.ac.uk lmtp[29924]: [ID 301543 mail.info] 
skiplist: checkpointed /opt/etc/imapd/deliver.db (715978 records, 
68761076 bytes) in 76 seconds

This presumably doesn't cause anyone much trouble overnight, but it's a 
bit more noticeable during the day.

Regards,


Dave.

David Mayo
Networks/Systems Administrator
University of Bath Computing Services, UK

>> Do other people suffer from this? Is there a patch that anyone has
>> written and/or will be submitting upstream any time soon?
>>
>> We are running Cyrus 2.3.13 across a front end, a back end and a
>> replication host for around 25,000 users.
>>
>> Regards,
>>
>>
>> Dave.
>>
>> David Mayo
>> Networks/Systems Administrator
>> University of Bath Computing Services, UK
>>
>> [1]
>> Nov 4 15:32:33 imap.bath.ac.uk lmtp[29595]: [ID 301543 mail.info]
>> skiplist: checkpointed /opt/etc/imapd/deliver.db (787618 records,
>> 74775536 bytes) in 386 seconds
>> Nov 9 14:26:06 imap.bath.ac.uk lmtp[1676]: [ID 301543 mail.info]
>> skiplist: checkpointed /opt/etc/imapd/deliver.db (386167 records,
>> 36638408 bytes) in 232 seconds
>> Nov 11 12:45:35 imap.bath.ac.uk lmtp[2497]: [ID 301543 mail.info]
>> skiplist: checkpointed /opt/etc/imapd/deliver.db (657254 records,
>> 62540568 bytes) in 396 seconds