Expunged emails after upgrade from 2.3.16 to 2.4.5
Stephen Carr
sgcarr at civeng.adelaide.edu.au
Sun Dec 5 21:58:03 EST 2010
Dear All
Mea Culpa !!
I think I was assuming that replication would work the same in 2.4.5 as
it was in 2.3.16 - but it has had a major rewrite.
I think the problem is I had this in the replica's cyrus.conf
purgeall cmd="ipurge -f -d 150" at=2200
So it impacted on the client / master but the replica the logs report
Dec 5 01:00:02 proxy cyr_expire[25067]: Expunged 0 out of 640465
messages from 1479 mailboxes
Dec 6 01:01:27 proxy cyr_expire[29917]: Expunged 0 out of 588837
messages from 1431 mailboxes
The reason I suspect this has an impact is the -d 150 and I now suspect
what happens on the replica is reflected on the master.
I had this parameter as I did not want the amount of emails to get
massive - if an email was more than 150 days old it was lost.
Now I know I was not using the replica as a true "image" of the master
in that I have in the replica's imapd.conf the following
expunge_mode: delayed
delete_mode: delayed
The reason for this is that I have had instances of users misconfiguring
POP or deleting emails and then emptying Trash and asking where is the
email from John Citizen.
I have had to restore whole mail folders for a few users.
So I was using the replica as a emergency backup store of emails.
What do you recommend ?
I have disabled the purgeall command.
Thanks
Stephen Carr
Stephen Carr wrote:
> Dear Bron
>
> Yes ant = any - see comments below as well.
>
> Any chance the reconstruct runs could have had an impact ?
>
> I have restored all the emails on one account and have run reconstruct
> - r -f -G on that account that had about 70,000 emails rediscovered.
>
> Will watch what happens overnight.
>
> PS Most users POP theie emails off the server only a few use pure IMAP.
>
> Thanks again.
> Stephen Carr
>
>
> Bron Gondwana wrote:
>> On Mon, Dec 06, 2010 at 08:22:57AM +1030, Stephen Carr wrote:
>>
>>> After fixing sync problems with the help of Bron this morning I
>>> discovered that ant email older that 150 days has been expunged.
>>> What alerted me was the size of the Full backup done one day after
>>> the upgrade.
>>>
>>
>> "ant email" - do you mean "any email"?
>>
>>
>>> Here is part of the log
>>>
>>> Dec 5 04:00:02 brooks cyr_expire[23141]: Expunged 0 out of 208877
>>> messages from 1453 mailboxes
>>>
>>
>> No messages expired.
>>
>
> I did an ls -lR of the users directory and this morning got 77,179
> where as before it was around 230,000
>
> I restored the maildata from the backup prior to the upgrade to
> another system as a check and ran the command below result was 235,362
>
> The command is
>
> ls -lR /maildata/cyrus/user/ | grep " [1-9]*\." | wc -l
>>
>>
>>> Dec 5 21:00:00 brooks ipurge[4410]: Expunged 2 messages from
>>> user.user1.Trash
>>> Dec 5 21:00:00 brooks ipurge[4410]: Expunged 1 messages from
>>> user.user2.Trash
>>> Dec 5 21:00:01 brooks ipurge[4410]: Expunged 25 messages from
>>> user.user3.Trash
>>> ...
>>> Dec 6 03:00:00 brooks ipurge[18145]: Expunged 1 messages from
>>> user.user4.quarantine
>>> Dec 6 03:00:00 brooks ipurge[18145]: Expunged 5 messages from
>>> user.user5.quarantine
>>> ....
>>> Dec 6 04:00:00 brooks ipurge[20370]: Expunged 95 messages from
>>> user.support.Bacula
>>>
>>
>> Lots of messages cleaned up by ipurge.
>>
>
> Some of these emails are temporary especially from Bacula where I get
> 150+ emails a day as I backup clients and servers.
>>
>>> Dec 6 04:00:02 brooks cyr_expire[20368]: Expunged 0 out of 113053
>>> messages from 1453 mailboxes
>>>
>>
>> Still no messages expired.
>>
>>
>>> Notice the 208877 is now 113053 and I suspect it now more like 77000
>>> since it is 2 days after the upgrade.
>>>
>>> Is there a default Expunge ?
>>>
>>
>> It's looking to me more like ipurge used to be broken, and now it's
>> working!
>>
>>
>>> expunge_mode: immediate
>>> delete_mode: immediate
>>>
>>
>> I strongly recommend removing the "expunge_mode: immediate" and either
>> setting it explicitly to "expunge_mode: default" or even just not
>> specifying it. "expunge_mode: immediate" doesn't work nicely with
>> replication - you wind up having to run inefficient sync all the time
>> because it cleaned out the expunge information too quickly.
>>
>>
> Change ran kill -HUP of master process ID.
>
>>> sync_password: 30cyrus51
>>>
>>
>>
>
> Opps will change
>
>> In general I wouldn't recommend posting this bit to a public
>> mailing list...
>>
>>
>>> # this is only necessary if using duplicate delivery suppression,
>>> # Sieve or NNTP
>>> delprune cmd="cyr_expire -E 3" at=0400
>>>
>>
>> You don't have any "-X" in there, which means that delayed expunge
>> will be running immediately for all mailboxes. Again, -X 1 along
>> with the expunge_mode changes above will make your replication
>> experience nicer.
>>
> Updated cyrus config
>>
>>
>>> trashprune1 cmd="ipurge -f -d 21 user.%.Trash" at=2100
>>> trashprune2 cmd="ipurge -f -d 21 user.%.trash" at=2130
>>>
>>> # Purge quarantine mailboxes older than 14 days at 3 am
>>> purgespam cmd="ipurge -f -d 14 user.%.quarantine" at=0300
>>> purgespam2 cmd="ipurge -f -d 2 user.%.spam" at=0330
>>> purgebacula cmd="ipurge -f -d 28 user.support.Bacula" at=0400
>>>
>>
>> Yeah, it's those ipurge commands that are doing it. I suspect
>> there used to be bugs in ipurge with immediate expunge, and
>> you just didn't notice that it wasn't actually being cleaned up
>> because you didn't have a -X specified in you cyr_expire...
>>
>> Bron.
>>
>>
>
> --
> Stephen Carr
> School of Civil, Environmental & Mining Engineering
> The University of Adelaide
> Adelaide, South Australia, 5005
> Australia
> Tel +618-8303-4313
> Fax +618-8303-4359
>
> Email sgcarr at civeng.adelaide.edu.au
>
> CRICOS Provider Number 00123M
> -----------------------------------------------------------
> This email message is intended only for the addressee(s) and
> contains information that may be confidential and/or copyright.
> If you are not the intended recipient please notify the sender
> by reply email and immediately delete this email. Use, disclosure
> or reproduction of this email by anyone other than the intended
> recipient(s) is strictly prohibited. No representation is made
> that this email or any attachments are free of viruses. Virus
> scanning is recommended and is the responsibility of the recipient.
>
>
> ------------------------------------------------------------------------
>
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
--
Stephen Carr
School of Civil, Environmental & Mining Engineering
The University of Adelaide
Adelaide, South Australia, 5005
Australia
Tel +618-8303-4313
Fax +618-8303-4359
Email sgcarr at civeng.adelaide.edu.au
CRICOS Provider Number 00123M
-----------------------------------------------------------
This email message is intended only for the addressee(s) and
contains information that may be confidential and/or copyright.
If you are not the intended recipient please notify the sender
by reply email and immediately delete this email. Use, disclosure
or reproduction of this email by anyone other than the intended
recipient(s) is strictly prohibited. No representation is made
that this email or any attachments are free of viruses. Virus
scanning is recommended and is the responsibility of the recipient.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20101206/168004ad/attachment-0001.html
More information about the Info-cyrus
mailing list