cyrus-imap 2.5.3 - `lost' messages

Michael Menge michael.menge at
Tue Aug 14 05:21:40 EDT 2018

Quoting Dirk-Willem van Gulik <dirkx at>:

>> On 14 Aug 2018, at 09:38, Michael Menge  
>> <michael.menge at> wrote:
>> Quoting Dirk-Willem van Gulik <dirkx at  
>> <mailto:dirkx at>>:
>>> On a smal cyrus imap 2.5.3 setup (freebsd ports, default settings,  
>>> 5k users, 300 Gb mboxes) we are seeing very sporadic `loss' of  
>>> messages (once every few months; 1 or 2 messages; out of a few  
>>> 100k to a million going through perfectly fine in the same period).
>>> They disappear; but are still present as a /var/spool/imap/user/X/12345.
>>> files.
>>> But cannot be found; even when searching through IMAP on  
>>> message-ID or otherwise unique elements. Searches in the same  
>>> veign for the messages before/after the lost one work and return  
>>> those.
>>> A grep through the cyrus.index shows its unique message ID.
>>> A full  reconstruct with /-r/-R/-G does not seem to bring it back  
>>> as visible.
>>> The mesaages lost are otherwise normal; fully valid mime; and when
>>> re-injected - pass through and surface.
>>> What is the right way to debug this ? We suspect it may have  
>>> something to do with either (al)pine and thunderbird; as it seems  
>>> to hit users that mix those two clients - during periods they use  
>>> both. All else seems normal.
>> Do you use delayed expunge mode (man imapd.conf see expunge_mode)?
>> If yes, "unexpunge -l user/X" will show you the mails that have  
>> been deleted but are still on the disk.
>> You will also see the time the mail was deleted.
> Yes we do. And did not know about that command.
> Where is this recorded / what rebuilds this file (and no - in this  
> case - it was not deleted, we do use delayed expunge) ?

The idea behind delayed expunge is twofold, on one hand it postpones
the I/O to an more convenient time, where less interactive I/O is done,
and on the other hand it is very useful for backup/restore as you can
keep the messages for a few days so that your normal backup has a chance
to see the deleted mails even if the user (op a POP3 client) did delete
before the backup is run. As a additional benefit you can restore these
mails very easy with unexpunge (see the manpage for details)

We use the unix hierarchy separator "/" (imapd.conf unixhierarchysep: 1)
which is not the default for Cyrus 2.5. If you do use "." als  
hierarchy separator
you have to use "unexpunge -l user.X"

For more information about this take a look at
as well as the imapd.conf manpage ( deletedprefix, delete_mode,  
expunge_mode, expunge_days) and cyr_expire manpage.



M.Menge                                Tel.: (49) 7071/29-70316
Universität Tübingen                   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung          mail:  
michael.menge at
Wächterstraße 76
72074 Tübingen

More information about the Cyrus-devel mailing list