Delayed delete oddities

[Patrick Boutilier]

On 12/17/2009 02:33 PM, David Mayo wrote:
>
>
> David Mayo wrote:
>
>> We have just started transferring the first mailboxes onto our new Cyrus
>> IMAP 2.3.13 server running on Solaris 10. Delayed expunge seems to work
>> fine but I've noticed a strange phenomenon with delayed delete on mailboxes.
>>
>> We have set up cyrus to delete expunged emails and deleted mailboxes
>> after 7 days:
>>
>>     delprune      cmd="cyr_expire -E 3 -D 7 -X 7" at=0400
>>
>> It appears to be deleting emails 7 days after they have been expunged,
>> so this looks fine:
>>
>> ...
>>
>> It's just the first entry I'm concerned by:
>>
>> Dec 11 04:00:00 imap-1.bath.ac.uk cyr_expire[29491]: [ID 956378
>> mail.notice] Expunged 4662 messages from DELETED.user.usera.BUCS.IT
>> discussion list.4B2123D3
>>
>> This mailbox was only deleted yesterday:
>>
>> Dec 10 16:37:39 sauber.bath.ac.uk imap[11321]: [ID 630590 mail.notice]
>> Deleted mailbox user.usera.BUCS.IT discussion list
>>
>> The mailbox was moved to the right area of the IMAP tree but all the
>> messages were removed after just one day. The messages in the mailbox
>> were not marked for deletion after being moved. Worse still, it hasn't
>> even kept the messages on the filesystem:
>>
>> imap-1 $ ls -la /mail/02/DELETED/user/usera/BUCS/IT\ discussion\
>> list/4B2123D3/
>> total 21441
>> drwx------   2 cyrus    cyrus         12 Dec 11 04:00 .
>> drwx------   3 cyrus    cyrus          3 Dec 10 16:37 ..
>> -rw-------   1 cyrus    cyrus      27366 Feb 17  2009 4861.
>> -rw-------   1 cyrus    cyrus       3151 May 31  2009 5007.
>> -rw-------   1 cyrus    cyrus       2543 Jun 15 10:04 5025.
>> -rw-------   1 cyrus    cyrus       2396 Jun 15 11:45 5026.
>> -rw-------   1 cyrus    cyrus       2366 Jul 29 09:46 5084.
>> -rw-------   1 cyrus    cyrus          4 Dec 11 04:00 cyrus.cache
>> -rw-------   1 cyrus    cyrus        536 Dec 10 16:37 cyrus.expunge
>> -rw-------   1 cyrus    cyrus        175 Dec 10 16:37 cyrus.header
>> -rw-------   1 cyrus    cyrus         96 Dec 11 04:00 cyrus.index
>> -rw-------   1 cyrus    cyrus    10734041 Dec 11 02:00 cyrus.squat
>>
>> (squatter is run at 2am, cyr_expire at 4am, which explains the size of
>> cyrus.squat)
>>
>> There were emails newer and older than those ones there, so I'm not sure
>> why those 5 were kept. Running unexpunge on the mailbox resulted in this:
>>
>> imap-1 $ unexpunge -l "DELETED.user.usera.BUCS.IT discussion list.4B2123D3"
>> UID: 4861
>>           Size: 27366
>>           Sent: Tue Feb 17 12:00:00 2009
>>           Recv: Tue Feb 17 10:47:12 2009
>>           Expg: Thu Dec 10 16:37:39 2009
>> Segmentation Fault (core dumped)
>>
>> [imap-1][/] # pstack /var/tmp/core.unexpunge
>> core '/var/tmp/core.unexpunge' of 516:  unexpunge -l
>> DELETED.user.usera.BUCS.IT discussion list.4B2123D3
>>    08058317 list_expunged (8046734, 8134898, 5, fea50000) + 186
>>    080599df main     (3, 8047a5c, 8047a6c) + 941
>>    08057f84 _start   (3, 8047b58, 8047b62, 8047b65, 0, 8047b9a) + 80
>>
>> Is the deleted mailbox code considered reliable? Should we be pruning
>> these using a different method? I'm happy to supply more information if
>> it would lead to this being better understood and/or fixed.
>
> I have been doing more testing on this by deleting some of my own
> mailboxes. Each time, all the messages are removed from the mailbox and
> most or all of them are deleted from the filesystem too by cyr_expire
> the first time it runs after the mailbox has been deleted.
>
> I tried setting the 'expire' annotation to '99' on the DELETED hierarchy
> yesterday but the emails in the mailbox were still all deleted. Tonight
> I have removed the -D 7 switch from cyr_expire to see if that fixes things.
>
> Is anyone else using delayed delete for mailboxes?

Yes.

  Does cyr_expire leave
> the mailboxes untouched after a mailbox has been deleted?

I haven't looked that closely but cyr_expire does remove some messages 
from a DELETED mailbox, just not sure if it is all of them or not.

  Are other
> people running cyr_expire from within cyrus.conf ?

We run as a nightly cron job.


  It would be nice to
> know if other people are seeing this or if it's something unique to our
> setup.
>
> Regards,
>
>
> Dave.
>
> David Mayo
> Networks/Systems Administrator
> University of Bath Computing Services, UK
>
> Tel: +44 1225 38 6046
> Email: D.J.Mayo at bath.ac.uk
> ----
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html