imap clients say i have 4K messages but spool has 12894 files

[Nic Bernstein]

On 5/26/20 8:33 AM, Brian J. Murrell wrote:
> Hi.
>
> Every IMAP client I query my cyrus imapd 2.4.17 server with says I have
> ~4K messages in my INBOX.  However when I do a listing of
> /var/spool/imap/b/user/brian/ it shows almost 13K files.
>
> None of these include messages which have been deleted but not
> expunged.  I manually expunge my mailbox many times per day.
>
> If I'm understanding mbexamine's output correctly, I have files on disk
> that are not being displayed by mbexmine.  My understanding of
> mbexamine's output is that on a line formatted as such:
>
> 000001> UID:00089183   INT_DATE:[redacted] SENTDATE:[redacted] SIZE:1537
>
> that the 00089183 is the reference to the file on the spool in
> /var/spool/imap/b/user/brian/89183.
>
> Is that correct?  If so, I definitely have files on the disk which are
> not found in any "000001> UID" line from mbexamine.  ~9600 of them.
> That seems to make up the difference between what an IMAP client sees
> and how many files are on disk.
>
> I also have multiple occurrences of the same "000001> UID:" and where
> there are no matching files on the disk.  Should that be possible?
>
> So how come the huge discrepancies and how do I reconcile them?
>
> Cheers,
> b.

Brian,
In a nutshell, RTFM.  Or, read this mailing list, since the answer to 
this is literally exactly the same as to the very last question in this 
list.  It's all about the setting of 'expunge_mode' in imapd.conf:

    |expunge_mode:| delayed

        The mode in which messages (and their corresponding cache
        entries) are expunged. “semidelayed” mode is the old behavior in
        which the message files are purged at the time of the EXPUNGE,
        but index and cache records are retained to facilitate QRESYNC.
        In “delayed” mode, which is the default since Cyrus 2.5.0, the
        message files are also retained, allowing unexpunge to rescue
        them. In “immediate” mode, both the message files and the index
        records are removed as soon as possible. In all cases, nothing
        will be finally purged until all other processes have closed the
        mailbox to ensure they never see data disappear under them. In
        “semidelayed” or “delayed” mode, a later run of “cyr_expire”
        will clean out the retained records (and possibly message
        files). This reduces the amount of I/O that takes place at the
        time of EXPUNGE and should result in greater responsiveness for
        the client, especially when expunging a large number of
        messages. Allowed values: /immediate/, /semidelayed/, /delayed/

As shown in this excerpt from the manpage 
<http://www.cyrusimap.org/imap/reference/manpages/configs/imapd.conf.html>, 
the default is now "delayed," since v2.5.0.  Those files on disk are 
expunged messages which have not yet been deleted.  They may be 
recovered via the 'unexpunge' command, as described on its manpage, here 
<http://www.cyrusimap.org/imap/reference/manpages/systemcommands/unexpunge.html?highlight=unexpunge>.  
To see a list of such messages, try 'sudo -u cyrus -c "unexpunge -l 
user/username at domain.tld'

Cheers,
     -nic

-- 
Nic Bernstein                               nic at nicbernstein.com
https://www.nicbernstein.com
https://www.linkedin.com/in/nic-b-26577a178/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200526/293c2e1b/attachment.html>