cyrus-imap 2.5.3 - `lost' messages

[Michael Menge]

Quoting Dirk-Willem van Gulik <dirkx at webweaving.org>:

>> On 14 Aug 2018, at 09:38, Michael Menge  
>> <michael.menge at zdv.uni-tuebingen.de> wrote:
>>
>>
>> Quoting Dirk-Willem van Gulik <dirkx at webweaving.org  
>> <mailto:dirkx at webweaving.org>>:
>>
>>> On a smal cyrus imap 2.5.3 setup (freebsd ports, default settings,  
>>> 5k users, 300 Gb mboxes) we are seeing very sporadic `loss' of  
>>> messages (once every few months; 1 or 2 messages; out of a few  
>>> 100k to a million going through perfectly fine in the same period).
>>>
>>> They disappear; but are still present as a /var/spool/imap/user/X/12345.
>>> files.
>>>
>>> But cannot be found; even when searching through IMAP on  
>>> message-ID or otherwise unique elements. Searches in the same  
>>> veign for the messages before/after the lost one work and return  
>>> those.
>>>
>>> A grep through the cyrus.index shows its unique message ID.
>>>
>>> A full  reconstruct with /-r/-R/-G does not seem to bring it back  
>>> as visible.
>>>
>>> The mesaages lost are otherwise normal; fully valid mime; and when
>>> re-injected - pass through and surface.
>>>
>>> What is the right way to debug this ? We suspect it may have  
>>> something to do with either (al)pine and thunderbird; as it seems  
>>> to hit users that mix those two clients - during periods they use  
>>> both. All else seems normal.
>>
>> Do you use delayed expunge mode (man imapd.conf see expunge_mode)?
>> If yes, "unexpunge -l user/X" will show you the mails that have  
>> been deleted but are still on the disk.
>> You will also see the time the mail was deleted.
>
> Yes we do. And did not know about that command.
>
> Where is this recorded / what rebuilds this file (and no - in this  
> case - it was not deleted, we do use delayed expunge) ?
>

The idea behind delayed expunge is twofold, on one hand it postpones
the I/O to an more convenient time, where less interactive I/O is done,
and on the other hand it is very useful for backup/restore as you can
keep the messages for a few days so that your normal backup has a chance
to see the deleted mails even if the user (op a POP3 client) did delete
before the backup is run. As a additional benefit you can restore these
mails very easy with unexpunge (see the manpage for details)

We use the unix hierarchy separator "/" (imapd.conf unixhierarchysep: 1)
which is not the default for Cyrus 2.5. If you do use "." als  
hierarchy separator
you have to use "unexpunge -l user.X"

For more information about this take a look at  
https://www.cyrusimap.org/2.5/imap/features/delayed-expunge.html
as well as the imapd.conf manpage ( deletedprefix, delete_mode,  
expunge_mode, expunge_days) and cyr_expire manpage.

Regards,


    Michael




--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universität Tübingen                   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung          mail:  
michael.menge at zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen