expunge_days & cyr_expire best practice

[Bron Gondwana]

On Mon, Sep 12, 2011 at 04:29:47PM +0200, Wolfgang Breyha wrote:
> Hi!
> 
> I'm currently preparing our "murder" migration from 2.3 to 2.4. Since we
> already use delayed expunge and delete I try to fully understand the impact of
> expunge_days: setting. Currently we use -X 2 with cyr_expire.

Fair enough.

> Since QRESYNC needs at least "expunge_days: 7" as stated in "man imapd.conf" I
> tried to figure out how to set it up right.

Nah, it doesn't really.  It can cope fine with instant expunge.
It's just a bit less efficient if it can't guarantee that it
knows about all the expunges since the last request, so it falls
back to telling about EVERY gap in the UID sequence, rather than
only telling about known removals.

If you're comfortable reading the code (which it looks like you
are!) you can see the algorithm working with highestmodseq and
deletedmoseq.  Deletedmodseq is the HIGHEST modseq of any record
that has ever been removed from the index.  If you request
changes since a modseq after that, you can guarantee that all
the required expunge information is available, so you can give
efficient responses.

This also applies to replication, if you're using it.  Replication
can be much more efficient if expunge data is available, because
it can send a full list of changes rather than having to sync
the whole mailbox.

> If I read the source right only mailbox.c:mailbox_close() and cyr_expire do
> cleanup jobs on cyrus cache and unlink files.

Yes, that's correct.  expunge_days is a nasty workaround for the
fact that otherwise mailboxes would just grow unbounded if people
didn't run a cyr_expire.  The default is now expunge_mode: default,
which deletes the message, but keeps the index metadata.

> mailbox_close() waits until expunge_days+8 after the index record was expired.
> What are these 8 days for? Is this the minimum needed for QRESYNC mentioned in
> "man imapd.conf"?

The basic idea is that if you run cyr_expire regularly enough, then
you never get a random other process having to do the expensive
expunge task.  On the other hand, if you FAIL to run cyr_expire,
your mess still gets cleaned up eventually.  Again, bogus heuristic.
I can only claim I wasn't thinking super clearly when I wrote that
section of the logic.  It's more "clever" than "good".

> cyr_expire doesn't care about expunge_days at all. So I guess -X should be at
> least >= expunge_days, right?

No, -X should be the same or less than expunge_days, then you get
to control when you expunge.

> Is
> expunge_days: 7
> and
> cyr_expire -X 7
> ok so far?

That's what we run, and it keeps us nice and happy.  We also only
expunge on Saturdays, which keeps the IO load down during the week.

Bron ( and we run checkreplication on Sundays - so the weekends are
       actually _higher_ IO load than the weeks... )