From brian at interlinx.bc.ca  Mon Jun  1 21:45:42 2020
From: brian at interlinx.bc.ca (Brian J. Murrell)
Date: Mon, 01 Jun 2020 21:45:42 -0400
Subject: imap clients say i have 4K messages but spool has 12894 files
In-Reply-To: <7077b241561e36be6f3aa7696ca5d11b0494ef45.camel@interlinx.bc.ca>
References: <7077b241561e36be6f3aa7696ca5d11b0494ef45.camel@interlinx.bc.ca>
Message-ID: <9cc7e8defc69f7981b886ee786bb8144d626c3b8.camel@interlinx.bc.ca>

On Tue, 2020-05-26 at 09:33 -0400, Brian J. Murrell wrote:
> Hi.
> 
> Every IMAP client I query my cyrus imapd 2.4.17 server with says I
> have
> ~4K messages in my INBOX.  However when I do a listing of
> /var/spool/imap/b/user/brian/ it shows almost 13K files.
> 
> None of these include messages which have been deleted but not
> expunged.  I manually expunge my mailbox many times per day.
> 
> If I'm understanding mbexamine's output correctly, I have files on
> disk
> that are not being displayed by mbexmine.  My understanding of
> mbexamine's output is that on a line formatted as such:
> 
> 000001> UID:00089183   INT_DATE:[redacted] SENTDATE:[redacted]
> SIZE:1537  
> 
> that the 00089183 is the reference to the file on the spool in
> /var/spool/imap/b/user/brian/89183.
> 
> Is that correct?  If so, I definitely have files on the disk which
> are
> not found in any "000001> UID" line from mbexamine.  ~9600 of them. 
> That seems to make up the difference between what an IMAP client sees
> and how many files are on disk.
> 
> I also have multiple occurrences of the same "000001> UID:" and where
> there are no matching files on the disk.  Should that be possible?
> 
> So how come the huge discrepancies and how do I reconcile them?

No other thoughts on how I can reconcile this gross discrepancy?

Ultimately I have an IMAP spool that is growing without bound due to
messages continuing to live on the spool beyond their life in the index
and getting orphaned.

Cheers,
b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200601/03fb15eb/attachment.sig>

From info-cyrus at jelmail.com  Wed Jun  3 10:09:07 2020
From: info-cyrus at jelmail.com (David Moyes)
Date: Wed, 3 Jun 2020 15:09:07 +0100
Subject: Sieve EditHeaders and Logging
Message-ID: <946d462d-4437-6c01-b2cf-388f2cd89633@lane.uk.net>

Couple of questions about sieve....

Is it possible to enable the "editheaders" sieve extension? if so, how?

Are sieve actions logged anywhere, e.g. to aid with debugging?

this is on Cyus-imap 3.0.13

thanks,

David.


From ianw at checksum.net.au  Wed Jun  3 19:30:50 2020
From: ianw at checksum.net.au (Ian Willis)
Date: Thu, 04 Jun 2020 09:30:50 +1000
Subject: imap clients say i have 4K messages but spool has 12894 files
In-Reply-To: <9cc7e8defc69f7981b886ee786bb8144d626c3b8.camel@interlinx.bc.ca>
References: <7077b241561e36be6f3aa7696ca5d11b0494ef45.camel@interlinx.bc.ca>
 <9cc7e8defc69f7981b886ee786bb8144d626c3b8.camel@interlinx.bc.ca>
Message-ID: <5eeb0c23bb04092bbdd3fa6468946fa417fa9428.camel@checksum.net.au>

Hi Brian,

The answer to your question is that yes, UID appears to correlate with
the message file name. 
At a guess something appears significantly awry.

To resolve the immediate issue.
Have you tried create a separate mail user. Copy your existing message
over via imap to the new folder. 
Delete and expunge the original mailbox and recreate, recopy.

In the longer term I would be tempted to move to a newer version of
cyrus or if you have the patience closely monitor the file-system to
debug how this is occurring.

Kind Regards 
Ian

-----Original Message-----
From: Brian J. Murrell <brian at interlinx.bc.ca>
To: info-cyrus at lists.andrew.cmu.edu
Subject: Re: imap clients say i have 4K messages but spool has 12894
files
Date: Mon, 01 Jun 2020 21:45:42 -0400

On Tue, 2020-05-26 at 09:33 -0400, Brian J. Murrell wrote:
Hi.
Every IMAP client I query my cyrus imapd 2.4.17 server with says
Ihave~4K messages in my INBOX.  However when I do a listing
of/var/spool/imap/b/user/brian/ it shows almost 13K files.
None of these include messages which have been deleted but
notexpunged.  I manually expunge my mailbox many times per day.
If I'm understanding mbexamine's output correctly, I have files
ondiskthat are not being displayed by mbexmine.  My understanding
ofmbexamine's output is that on a line formatted as such:
000001> UID:00089183   INT_DATE:[redacted]
SENTDATE:[redacted]SIZE:1537  
that the 00089183 is the reference to the file on the spool
in/var/spool/imap/b/user/brian/89183.

Is that correct?  If so, I definitely have files on the disk
whicharenot found in any "000001> UID" line from mbexamine.  ~9600 of
them. That seems to make up the difference between what an IMAP client
seesand how many files are on disk.
I also have multiple occurrences of the same "000001> UID:" and
wherethere are no matching files on the disk.  Should that be possible?
So how come the huge discrepancies and how do I reconcile them?
No other thoughts on how I can reconcile this gross discrepancy?
Ultimately I have an IMAP spool that is growing without bound due
tomessages continuing to live on the spool beyond their life in the
indexand getting orphaned.
Cheers,b.

----Cyrus Home Page: http://www.cyrusimap.org/List Archives/Info: 
http://lists.andrew.cmu.edu/pipermail/info-cyrus/To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200604/8068b73c/attachment.html>

From murch at fastmail.com  Wed Jun  3 19:35:36 2020
From: murch at fastmail.com (Ken Murchison)
Date: Wed, 3 Jun 2020 19:35:36 -0400
Subject: imap clients say i have 4K messages but spool has 12894 files
In-Reply-To: <5eeb0c23bb04092bbdd3fa6468946fa417fa9428.camel@checksum.net.au>
References: <7077b241561e36be6f3aa7696ca5d11b0494ef45.camel@interlinx.bc.ca>
 <9cc7e8defc69f7981b886ee786bb8144d626c3b8.camel@interlinx.bc.ca>
 <5eeb0c23bb04092bbdd3fa6468946fa417fa9428.camel@checksum.net.au>
Message-ID: <85d287a5-6f2d-9cfe-4d9c-d5d693309ef0@fastmail.com>

Brian,

Trying running 'unexpunge -l' on the mailbox in question.? This will 
list expunged, but not-yet-removed-from-disk messages.

cyr_expire is used to removed expunged messages from disk.



On 6/3/20 7:30 PM, Ian Willis wrote:
> Hi Brian,
>
> The answer to your question is that yes, UID appears to correlate with 
> the message file name.
> At a guess something appears significantly awry.
>
> To resolve the immediate issue.
> Have you tried create a separate mail user. Copy your existing message 
> over via imap to the new folder.
> Delete and expunge the original mailbox and recreate, recopy.
>
> In the longer term I would be tempted to move to a newer version of 
> cyrus or if you have the patience closely monitor the file-system to 
> debug how this is occurring.
>
> Kind Regards
> Ian
>
> -----Original Message-----
> *From*: Brian J. Murrell <brian at interlinx.bc.ca 
> <mailto:%22Brian%20J.%20Murrell%22%20%3cbrian at interlinx.bc.ca%3e>>
> *To*: info-cyrus at lists.andrew.cmu.edu 
> <mailto:info-cyrus at lists.andrew.cmu.edu>
> *Subject*: Re: imap clients say i have 4K messages but spool has 12894 
> files
> *Date*: Mon, 01 Jun 2020 21:45:42 -0400
>
> On Tue, 2020-05-26 at 09:33 -0400, Brian J. Murrell wrote:
> Hi.
> Every IMAP client I query my cyrus imapd 2.4.17 server with says I
> have
> ~4K messages in my INBOX.  However when I do a listing of
> /var/spool/imap/b/user/brian/ it shows almost 13K files.
> None of these include messages which have been deleted but not
> expunged.  I manually expunge my mailbox many times per day.
> If I'm understanding mbexamine's output correctly, I have files on
> disk
> that are not being displayed by mbexmine.  My understanding of
> mbexamine's output is that on a line formatted as such:
> 000001> UID:00089183   INT_DATE:[redacted] SENTDATE:[redacted]
> SIZE:1537
> that the 00089183 is the reference to the file on the spool in
> /var/spool/imap/b/user/brian/89183.
> Is that correct?  If so, I definitely have files on the disk which
> are
> not found in any "000001> UID" line from mbexamine.  ~9600 of them.
> That seems to make up the difference between what an IMAP client sees
> and how many files are on disk.
> I also have multiple occurrences of the same "000001> UID:" and where
> there are no matching files on the disk.  Should that be possible?
> So how come the huge discrepancies and how do I reconcile them?
> No other thoughts on how I can reconcile this gross discrepancy?
> Ultimately I have an IMAP spool that is growing without bound due to
> messages continuing to live on the spool beyond their life in the index
> and getting orphaned.
> Cheers,
> b.
> ----
> Cyrus Home Page:http://www.cyrusimap.org/
> List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
Kenneth Murchison
Senior Software Developer
Fastmail US LLC

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200603/536b8743/attachment.html>

From falon at ruparpiemonte.it  Thu Jun  4 04:23:12 2020
From: falon at ruparpiemonte.it (Marco)
Date: Thu, 4 Jun 2020 10:23:12 +0200
Subject: Object Storage and Cyrus IMAP
Message-ID: <f49d6bf9-d4ec-9044-fa98-1476a5c9df05@ruparpiemonte.it>

Hello,

    I see that Cyrus IMAP 3 can interface with some Object Storage such 
as Caringo or OpenIO.

Is anyone using these solutions?

I would like to know how I can find more details about these deployment, 
other than the brief description in imapd.conf man page.

In particular I would like to know if these interfaces are stable and 
supported in future releases of Cyrus IMAP.

Do you plan to add wider support to object storage, maybe by adopting 
some standard vendor independent?

I thank you very much for every info you could provide.

Warm Regards
--
Marco

From brian at interlinx.bc.ca  Thu Jun  4 06:45:47 2020
From: brian at interlinx.bc.ca (Brian J. Murrell)
Date: Thu, 04 Jun 2020 06:45:47 -0400
Subject: imap clients say i have 4K messages but spool has 12894 files
In-Reply-To: <85d287a5-6f2d-9cfe-4d9c-d5d693309ef0@fastmail.com>
References: <7077b241561e36be6f3aa7696ca5d11b0494ef45.camel@interlinx.bc.ca>
 <9cc7e8defc69f7981b886ee786bb8144d626c3b8.camel@interlinx.bc.ca>
 <5eeb0c23bb04092bbdd3fa6468946fa417fa9428.camel@checksum.net.au>
 <85d287a5-6f2d-9cfe-4d9c-d5d693309ef0@fastmail.com>
Message-ID: <df15ed3d1e909811fb206c4917a4a10cbca24fa3.camel@interlinx.bc.ca>

On Wed, 2020-06-03 at 19:35 -0400, Ken Murchison wrote:
> Brian,
> 
> Trying running 'unexpunge -l' on the mailbox in question.

This avenue has already been explored earlier in this thread:

https://lists.andrew.cmu.edu/pipermail/info-cyrus/2020-May/041258.html

To save the effort of re-reading the message:

# sudo -u cyrus bash -c "/usr/lib/cyrus-imapd/unexpunge -l user.brian"
[nothing returned]

So this is looking more like a "bad accounting" problem than something
typically operational.

But how to reconcile it?

It seems to me that a process of comparing what's in the index to
what's on disk to account for the orphans is needed.  I just don't know
what that process is.  I probably just don't know the toolset well
enough to know which tools to apply and how.  mbexamine seems a
candidate but I'm not sure how to interpret it's output to this task. 
Or maybe there other/better tools?

Any suggestions?

Cheers,
b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200604/8fc6b1e4/attachment-0001.sig>

From brian at interlinx.bc.ca  Thu Jun  4 06:52:16 2020
From: brian at interlinx.bc.ca (Brian J. Murrell)
Date: Thu, 04 Jun 2020 06:52:16 -0400
Subject: imap clients say i have 4K messages but spool has 12894 files
In-Reply-To: <5eeb0c23bb04092bbdd3fa6468946fa417fa9428.camel@checksum.net.au>
References: <7077b241561e36be6f3aa7696ca5d11b0494ef45.camel@interlinx.bc.ca>
 <9cc7e8defc69f7981b886ee786bb8144d626c3b8.camel@interlinx.bc.ca>
 <5eeb0c23bb04092bbdd3fa6468946fa417fa9428.camel@checksum.net.au>
Message-ID: <259f309a2a46dac48a096f5f0831850056bdd627.camel@interlinx.bc.ca>

On Thu, 2020-06-04 at 09:30 +1000, Ian Willis wrote:
> Hi Brian,

Hi Ian,

> The answer to your question is that yes, UID appears to correlate
> with
> the message file name. 

Thanks.

> At a guess something appears significantly awry.

Indeed.

> Have you tried create a separate mail user. Copy your existing
> message
> over via imap to the new folder. 
> Delete and expunge the original mailbox and recreate, recopy.

I have considered that.  But I suspect that is going to cause the
message numbers on the disk to be recreated.  Since this seems to
affect many (all?) folders for many (again, all?) users, that would
result in trashing the efficiency of my incremental backup.

> In the longer term I would be tempted to move to a newer version of
> cyrus

I'm stuck with what my distro vendor supplies.  That said, once the
cause of this problem, or it's reproducibiilty can be confirmed, distro
vendor will be getting a ticket to resolve this in some way.  But the
first step is identifying the issue, resolving it and seeing if it
reproduces.

> or if you have the patience closely monitor the file-system to
> debug how this is occurring.

Right.  I think the first step is to figure out how to get things back
to normal so that it can be monitored more closely and the discrepancy
is no longer a haystack and will be more incrementally obvious when it
does happen.  Figuring out how to get back to normal will also provide
the tools/process to monitor on an ongoing basis to see why it's
happening.

I'm just not sure how to get back to normal at this point.

Cheers,
b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200604/d0514015/attachment.sig>

From murch at fastmail.com  Thu Jun  4 07:49:30 2020
From: murch at fastmail.com (Ken Murchison)
Date: Thu, 4 Jun 2020 07:49:30 -0400
Subject: imap clients say i have 4K messages but spool has 12894 files
In-Reply-To: <df15ed3d1e909811fb206c4917a4a10cbca24fa3.camel@interlinx.bc.ca>
References: <7077b241561e36be6f3aa7696ca5d11b0494ef45.camel@interlinx.bc.ca>
 <9cc7e8defc69f7981b886ee786bb8144d626c3b8.camel@interlinx.bc.ca>
 <5eeb0c23bb04092bbdd3fa6468946fa417fa9428.camel@checksum.net.au>
 <85d287a5-6f2d-9cfe-4d9c-d5d693309ef0@fastmail.com>
 <df15ed3d1e909811fb206c4917a4a10cbca24fa3.camel@interlinx.bc.ca>
Message-ID: <5a323d52-22b1-1d62-ecf7-31b4241c6dee@fastmail.com>

You could try 'reconstruct -R' which should force a re-parsing of all 
message files in the mailboxes directory.? Note that if this works, you 
will have 8k new messages show up in your mailbox. Adding -n may just 
report what reconstruct will do rather than actually doing it.


On 6/4/20 6:45 AM, Brian J. Murrell wrote:
> On Wed, 2020-06-03 at 19:35 -0400, Ken Murchison wrote:
>> Brian,
>>
>> Trying running 'unexpunge -l' on the mailbox in question.
> This avenue has already been explored earlier in this thread:
>
> https://lists.andrew.cmu.edu/pipermail/info-cyrus/2020-May/041258.html
>
> To save the effort of re-reading the message:
>
> # sudo -u cyrus bash -c "/usr/lib/cyrus-imapd/unexpunge -l user.brian"
> [nothing returned]
>
> So this is looking more like a "bad accounting" problem than something
> typically operational.
>
> But how to reconcile it?
>
> It seems to me that a process of comparing what's in the index to
> what's on disk to account for the orphans is needed.  I just don't know
> what that process is.  I probably just don't know the toolset well
> enough to know which tools to apply and how.  mbexamine seems a
> candidate but I'm not sure how to interpret it's output to this task.
> Or maybe there other/better tools?
>
> Any suggestions?
>
> Cheers,
> b.
>
>
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
Kenneth Murchison
Senior Software Developer
Fastmail US LLC

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200604/20446fcc/attachment.html>

From boutilpj at ednet.ns.ca  Thu Jun  4 07:56:00 2020
From: boutilpj at ednet.ns.ca (Patrick Boutilier)
Date: Thu, 4 Jun 2020 08:56:00 -0300
Subject: imap clients say i have 4K messages but spool has 12894 files
In-Reply-To: <df15ed3d1e909811fb206c4917a4a10cbca24fa3.camel@interlinx.bc.ca>
References: <7077b241561e36be6f3aa7696ca5d11b0494ef45.camel@interlinx.bc.ca>
 <9cc7e8defc69f7981b886ee786bb8144d626c3b8.camel@interlinx.bc.ca>
 <5eeb0c23bb04092bbdd3fa6468946fa417fa9428.camel@checksum.net.au>
 <85d287a5-6f2d-9cfe-4d9c-d5d693309ef0@fastmail.com>
 <df15ed3d1e909811fb206c4917a4a10cbca24fa3.camel@interlinx.bc.ca>
Message-ID: <039485ec-f339-a8ff-5ac1-4d27520b7196@ednet.ns.ca>



On 6/4/20 7:45 AM, Brian J. Murrell wrote:
> On Wed, 2020-06-03 at 19:35 -0400, Ken Murchison wrote:
>> Brian,
>>
>> Trying running 'unexpunge -l' on the mailbox in question.
> 
> This avenue has already been explored earlier in this thread:
> 
> https://lists.andrew.cmu.edu/pipermail/info-cyrus/2020-May/041258.html
> 
> To save the effort of re-reading the message:
> 
> # sudo -u cyrus bash -c "/usr/lib/cyrus-imapd/unexpunge -l user.brian"
> [nothing returned]
> 
> So this is looking more like a "bad accounting" problem than something
> typically operational.
> 
> But how to reconcile it?
> 
> It seems to me that a process of comparing what's in the index to
> what's on disk to account for the orphans is needed.  I just don't know
> what that process is.  I probably just don't know the toolset well
> enough to know which tools to apply and how.  mbexamine seems a
> candidate but I'm not sure how to interpret it's output to this task.
> Or maybe there other/better tools?
> 
> Any suggestions?
> 

Have you looked in some of the orphaned messages to see if they are 
emails you have deleted before? My thought would be to move these 
orphaned messages out of /var/spool/imap/b/user/brian . Then delete and 
expunge a few messages using your mail client and see if they are also 
removed from /var/spool/imap/b/user/brian





> Cheers,
> b.
> 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: boutilpj.vcf
Type: text/x-vcard
Size: 286 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200604/202c275b/attachment.vcf>

From falon at ruparpiemonte.it  Thu Jun  4 09:04:49 2020
From: falon at ruparpiemonte.it (Marco)
Date: Thu, 4 Jun 2020 15:04:49 +0200
Subject: Cyrus IMAP 3.2.1 released
In-Reply-To: <1c317fea-46b4-4487-a975-81b4be5287d8@www.fastmail.com>
References: <1c317fea-46b4-4487-a975-81b4be5287d8@www.fastmail.com>
Message-ID: <921ec398-d58e-3ac1-6169-ffd646df2ff7@ruparpiemonte.it>

On 29/05/2020 06:20, ellie timoney has written:
> The Cyrus team is proud to announce the immediate availability of a new version of Cyrus IMAP: 3.2.1

Hello,

  in Redhat EL8 I still fail these tests:

ERRORS:
Rename.rename_inbox
  Perl exception: Errors found in syslog
  at Cassandane/Instance.pm line 1322, <GEN3> line 91.

FAILS:

1) Admin.imap_admins
  expected 'ok', got 'no' at Cassandane/Cyrus/Admin.pm line 90.
2) Caldav.supports_event
  Boolean assertion failed at 
/usr/share/perl5/vendor_perl/Test/Unit/Exception.pm line 13.
3) ImapTest.append-binary
  Boolean assertion failed at 
/usr/share/perl5/vendor_perl/Test/Unit/Exception.pm line 13.
4) ImapTest.fetch-binary-mime
  Boolean assertion failed at 
/usr/share/perl5/vendor_perl/Test/Unit/Exception.pm line 13.
5) ImapTest.urlauth-binary
  Boolean assertion failed at 
/usr/share/perl5/vendor_perl/Test/Unit/Exception.pm line 13.
10) Cassandane::Test::Core.core_files_*
   didn't match /(?^:Core files found)/ at Cassandane/Test/Core.pm line 96.

The details are attached, if it could be of interest.
The Cassandane commit is very recent: 
68e4890346edeb68463d20f394a37f0862bef2ae

I think all these failure are somewhere related to Redhat environment... 
I'll skip these tests in Cyrus IMAP 3.2.1 build.

Regards
Marco

From igb at batten.eu.org  Thu Jun  4 12:38:11 2020
From: igb at batten.eu.org (Ian Batten)
Date: Thu, 4 Jun 2020 17:38:11 +0100
Subject: Replication and Deleted Files
Message-ID: <etPan.5ed92378.5de34120.17cb2@batten.eu.org>



Hi, long-time Cyrus user (25 years, I think), but stumped on this one?

I have an ancient Cyrus 2.5.11 on Solaris 11 installation I am trying to migrate off. ?The strategy is to run rolling replication onto the new server (3.0.8-6+deb10u4 on Debian 10.4), and then point the DNS record at the new server. ?With Covid, this has become more protracted than I would like, as I don?t want to accidentally mess up users who are isolating, so the replication has been running for some weeks.

The replication structure is old-server -> new-server -> (backup1, backup2) where backup1 and backup2 are configured as separate channels on new-server. ?This has been running seemingly correctly for about three months now.

Today I decided to check all was well by using rsync -an to confirm that the replicas have everything that is on the master. ?They do, in that using?

rsync -anvO --size-only? --exclude='cyrus.*' root at mail:/var/imap/partition1/user/ /var/imap/partition1/user?

where ?mail? is the old server shows that there are no messages missing (?size-only because there?s some time slew in a few places, usually only of a few seconds, but up to a day in others).

However, reversing it:

rsync -anvO --size-only? --exclude='cyrus.*' /var/imap/partition1/user/ root at mail:/var/imap/partition1/user

Shows that there are a _lot_ of files on the replicas which are not on the master, some of them relating to recent deletions, but some of them seemingly quite old. ?I am using:

delete_mode: delayed
expunge_mode: delayed

everywhere, running cyr_expire on the master but not on the replicas. ?I have enough bandwidth that sync_reset and re-sync is realistic, but I?d rather not have to do that immediately prior to a cut-over. ? These old files are a worry because if I ever had to reconstruct one of the mailboxes, presumably the deleted (I think) messages would all reappear. ?Does anyone have any suggestions?

Thanks

ian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200604/51b99619/attachment.html>

From michael.menge at zdv.uni-tuebingen.de  Thu Jun  4 13:56:35 2020
From: michael.menge at zdv.uni-tuebingen.de (Michael Menge)
Date: Thu, 04 Jun 2020 19:56:35 +0200
Subject: Replication and Deleted Files
In-Reply-To: <etPan.5ed92378.5de34120.17cb2@batten.eu.org>
Message-ID: <20200604195635.Horde.RJWby91suvYv4IniOiWuN2t@webmail.uni-tuebingen.de>

Hi,

Quoting Ian Batten via Info-cyrus <info-cyrus at lists.andrew.cmu.edu>:

> Hi, long-time Cyrus user (25 years, I think), but stumped on this one?
>
> I have an ancient Cyrus 2.5.11 on Solaris 11 installation I am  
> trying to migrate off. ?The strategy is to run rolling replication  
> onto the new server (3.0.8-6+deb10u4 on Debian 10.4), and then point  
> the DNS record at the new server. ?With Covid, this has become more  
> protracted than I would like, as I don?t want to accidentally mess  
> up users who are isolating, so the replication has been running for  
> some weeks.
>
> The replication structure is old-server -> new-server -> (backup1,  
> backup2) where backup1 and backup2 are configured as separate  
> channels on new-server. ?This has been running seemingly correctly  
> for about three months now.
>
> Today I decided to check all was well by using rsync -an to confirm  
> that the replicas have everything that is on the master. ?They do,  
> in that using?
>
> rsync -anvO --size-only? --exclude='cyrus.*'  
> root at mail:/var/imap/partition1/user/ /var/imap/partition1/user?
>
> where ?mail? is the old server shows that there are no messages  
> missing (?size-only because there?s some time slew in a few places,  
> usually only of a few seconds, but up to a day in others).
>
> However, reversing it:
>
> rsync -anvO --size-only? --exclude='cyrus.*'  
> /var/imap/partition1/user/ root at mail:/var/imap/partition1/user
>
> Shows that there are a _lot_ of files on the replicas which are not  
> on the master, some of them relating to recent deletions, but some  
> of them seemingly quite old. ?I am using:
>
> delete_mode: delayed
> expunge_mode: delayed
>
> everywhere, running cyr_expire on the master but not on the  
> replicas. ?I have enough bandwidth that sync_reset and re-sync is  
> realistic, but I?d rather not have to do that immediately prior to a  
> cut-over. ? These old files are a worry because if I ever had to  
> reconstruct one of the mailboxes, presumably the deleted (I think)  
> messages would all reappear. ?Does anyone have any suggestions?
>

you also need to run cyr_expire on the "new_server" to remove the old
expunged mails and deleted folders.

I haven't used replication for backup but I suspect that cyr_expire is
also required your backup servers





--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universit?t T?bingen                   Fax.: (49) 7071/29-5912
Zentrum f?r Datenverarbeitung          mail:  
michael.menge at zdv.uni-tuebingen.de
W?chterstra?e 76
72074 T?bingen


From brian at interlinx.bc.ca  Thu Jun  4 14:16:18 2020
From: brian at interlinx.bc.ca (Brian J. Murrell)
Date: Thu, 04 Jun 2020 14:16:18 -0400
Subject: imap clients say i have 4K messages but spool has 12894 files
In-Reply-To: <5a323d52-22b1-1d62-ecf7-31b4241c6dee@fastmail.com>
References: <7077b241561e36be6f3aa7696ca5d11b0494ef45.camel@interlinx.bc.ca>
 <9cc7e8defc69f7981b886ee786bb8144d626c3b8.camel@interlinx.bc.ca>
 <5eeb0c23bb04092bbdd3fa6468946fa417fa9428.camel@checksum.net.au>
 <85d287a5-6f2d-9cfe-4d9c-d5d693309ef0@fastmail.com>
 <df15ed3d1e909811fb206c4917a4a10cbca24fa3.camel@interlinx.bc.ca>
 <5a323d52-22b1-1d62-ecf7-31b4241c6dee@fastmail.com>
Message-ID: <238d3959e849014a9e1da07856a8845d9c602567.camel@interlinx.bc.ca>

Interestingly, through no action on my (as admin) part, this problem
seems to have resolved itself on May 31.  According to my backup, on
May 29 for my main inbox, user.brian, there were 13339 files on the
disk but on May 31's backup there are only 4136.

IMAP has always reported in the neighborhood of the 4K messages.

Strange.

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200604/7948f369/attachment.sig>

From Albert.Shih at obspm.fr  Thu Jun  4 14:48:13 2020
From: Albert.Shih at obspm.fr (Albert Shih)
Date: Thu, 4 Jun 2020 20:48:13 +0200
Subject: Object Storage and Cyrus IMAP
In-Reply-To: <f49d6bf9-d4ec-9044-fa98-1476a5c9df05@ruparpiemonte.it>
References: <f49d6bf9-d4ec-9044-fa98-1476a5c9df05@ruparpiemonte.it>
Message-ID: <20200604184813.GF45779@io.chezmoi.fr>

Le 04/06/2020 ? 10:23:12+0200, Marco a ?crit
> Hello,
>
>    I see that Cyrus IMAP 3 can interface with some Object Storage such as
> Caringo or OpenIO.
>
> Is anyone using these solutions?
>
> I would like to know how I can find more details about these deployment,
> other than the brief description in imapd.conf man page.
>
> In particular I would like to know if these interfaces are stable and
> supported in future releases of Cyrus IMAP.
>
> Do you plan to add wider support to object storage, maybe by adopting some
> standard vendor independent?
>
> I thank you very much for every info you could provide.

No sure if my informations are still correct, but long time ago, I ask
openio about that. They say the connector between openIO and cyrus are
maintained by openIO, it's not opensource and you need to pay a licence.

And when Cyrus make a new release openio would make adjustment to make it
work.

I not sure who use that, but as I ear they(openIO) got few customer use
this solution on a very large scale > 1Po.

Regards
--
Albert SHIH
DIO b?timent 15
xmpp: jas at obspm.fr
Heure local/Local time:
Thu 04 Jun 2020 08:44:56 PM CEST

From igb at batten.eu.org  Thu Jun  4 17:46:40 2020
From: igb at batten.eu.org (Ian Batten)
Date: Thu, 4 Jun 2020 22:46:40 +0100
Subject: Replication and Deleted Files
In-Reply-To: <20200604195635.Horde.RJWby91suvYv4IniOiWuN2t@webmail.uni-tuebingen.de>
References: <20200604195635.Horde.RJWby91suvYv4IniOiWuN2t@webmail.uni-tuebingen.de>
Message-ID: <etPan.5ed96bc6.5e4f6396.17cb2@batten.eu.org>




On Thu 04 Jun 2020 at 18:57:37, Michael Menge (michael.menge at zdv.uni-tuebingen.de) wrote:


>

you also need to run cyr_expire on the "new_server" to remove the old
expunged mails and deleted folders.


Obvious when you try it! ? ?Thanks so much. ?

Expired 23 and expunged 7617 out of 289060 messages from 268 mailboxes

For some reason I had decided that you only ran cyr_expire on the master, and I was quite emphatic about it some years ago:

? # expire old stuff: dups 7 days, keep deletions for 3 days
? # XXX XXX XXX expire does not run on replica, does run on master XXX XXX XXX
? # expire ? ? ? ?cmd="cyr_expire -E 7 -X 3 -D 3" at=0100

Thank you again,.,.I shall be back in another 25 years with another query :-)

ian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200604/e1e76157/attachment-0001.html>

From ellie at fastmail.com  Fri Jun  5 00:12:05 2020
From: ellie at fastmail.com (ellie timoney)
Date: Fri, 05 Jun 2020 14:12:05 +1000
Subject: Object Storage and Cyrus IMAP
In-Reply-To: <20200604184813.GF45779@io.chezmoi.fr>
References: <f49d6bf9-d4ec-9044-fa98-1476a5c9df05@ruparpiemonte.it>
 <20200604184813.GF45779@io.chezmoi.fr>
Message-ID: <c116fd36-b3ac-4537-9409-9fbbd0513219@www.fastmail.com>

On Fri, Jun 5, 2020, at 4:48 AM, Albert Shih wrote:
> Le 04/06/2020 ? 10:23:12+0200, Marco a ?crit
> > Hello,
> >
> >    I see that Cyrus IMAP 3 can interface with some Object Storage such as
> > Caringo or OpenIO.
> >
> > Is anyone using these solutions?
> >
> > I would like to know how I can find more details about these deployment,
> > other than the brief description in imapd.conf man page.
> >
> > In particular I would like to know if these interfaces are stable and
> > supported in future releases of Cyrus IMAP.
> >
> > Do you plan to add wider support to object storage, maybe by adopting some
> > standard vendor independent?
> >
> > I thank you very much for every info you could provide.
> 
> No sure if my informations are still correct, but long time ago, I ask
> openio about that. They say the connector between openIO and cyrus are
> maintained by openIO, it's not opensource and you need to pay a licence.
> 
> And when Cyrus make a new release openio would make adjustment to make it
> work.
> 
> I not sure who use that, but as I ear they(openIO) got few customer use
> this solution on a very large scale > 1Po.

Our object storage support was contributed by one of those two vendors (i.e Caringo or OpenIO, though I don't remember which).  As I understand it, they implemented support in Cyrus for both backends, to ensure that the object storage support was generalised, not specific to their own product.  This might have been a condition we imposed for accepting their patches?

Anyway, in theory, some third backend (whether closed- or open-source) could be similarly integrated, now that the abstract support is there.  I don't know if such a third backend even exists.

Looking through commit history, the last commits from the vendor developers to the object storage code were just before the release of 3.0.0.  So I would expect this works okay for 3.0 deployments.  I'm not sure about 3.2 -- we haven't had any specific updates, so maybe it works okay and doesn't need any?  Or maybe it won't work until it's updated, and their customers are simply staying on 3.0 for the time being.

I guess this is kind of a long-winded way of saying, you probably need to speak to those vendors about this.  Whether Cyrus supports their backend is almost incidental compared to the larger question of whether (and how) they support their backend being used from Cyrus.

Cheers,

ellie

From ellie at fastmail.com  Fri Jun  5 00:29:28 2020
From: ellie at fastmail.com (ellie timoney)
Date: Fri, 05 Jun 2020 14:29:28 +1000
Subject: Sieve EditHeaders and Logging
In-Reply-To: <946d462d-4437-6c01-b2cf-388f2cd89633@lane.uk.net>
References: <946d462d-4437-6c01-b2cf-388f2cd89633@lane.uk.net>
Message-ID: <2b765944-9898-4a29-bde5-da9580d6c44d@www.fastmail.com>

Hi David,

> Is it possible to enable the "editheaders" sieve extension? if so, how?

Not in 3.0, but it's available in 3.2

> Are sieve actions logged anywhere, e.g. to aid with debugging?

Generally? I don't know.  Maybe if you increase your syslog log level to "debug" and add "debug: yes" to your imapd.conf?

3.2 has an experimental "x-cyrus-log" extension, which adds an action that produces a log line.  You probably don't want to allow this for user-supplied sieve scripts, but if you have some sort of "sieve builder" system that your users use (rather than uploading their own handcrafted sieve scripts), then, if you were running 3.2, this extension could be useful.  On the master branch, and therefore in the next major release (2021), this extension will be formalised as "vnd.cyrus.log".

Cheers,

ellie

From ellie at fastmail.com  Fri Jun  5 02:22:06 2020
From: ellie at fastmail.com (ellie timoney)
Date: Fri, 05 Jun 2020 16:22:06 +1000
Subject: Cyrus IMAP 3.2.1 released
In-Reply-To: <921ec398-d58e-3ac1-6169-ffd646df2ff7@ruparpiemonte.it>
References: <1c317fea-46b4-4487-a975-81b4be5287d8@www.fastmail.com>
 <921ec398-d58e-3ac1-6169-ffd646df2ff7@ruparpiemonte.it>
Message-ID: <fe695d97-e33e-4953-8490-e53efed42fd9@www.fastmail.com>

Hi Marco,

On Thu, Jun 4, 2020, at 11:04 PM, Marco wrote:
> On 29/05/2020 06:20, ellie timoney has written:
> > The Cyrus team is proud to announce the immediate availability of a new version of Cyrus IMAP: 3.2.1
> 
> Hello,
> 
>   in Redhat EL8 I still fail these tests:
> 
> ERRORS:
> Rename.rename_inbox
>   Perl exception: Errors found in syslog
>   at Cassandane/Instance.pm line 1322, <GEN3> line 91.

You probably saw on github -- I've just pushed a fix to Cassandane which I think should fix up the remaining syslog-related failures...

Though, Rename.rename_inbox is a special exception.  This one will probably always error on 3.2.  There's a real (niche) bug, which this test is detecting.  It's existed for a very long time, but we're not sure how to fix it yet, so you might as well just ignore it.

Though... it's very interesting that it FOUND errors in syslog, when the usual RedHat problem is not being able to find the syslog output.  Curious?

> FAILS:
> 
> 1) Admin.imap_admins
>   expected 'ok', got 'no' at Cassandane/Cyrus/Admin.pm line 90.

This is interesting.  I'd like to see the full output please!

> 2) Caldav.supports_event
>   Boolean assertion failed at 
> /usr/share/perl5/vendor_perl/Test/Unit/Exception.pm line 13.

I think we discussed Caldav.supports_event before -- it passes here, still no idea what the difference is.  Maybe it's a libical thing?  What version is your libical?

> 3) ImapTest.append-binary
>   Boolean assertion failed at 
> /usr/share/perl5/vendor_perl/Test/Unit/Exception.pm line 13.
> 4) ImapTest.fetch-binary-mime
>   Boolean assertion failed at 
> /usr/share/perl5/vendor_perl/Test/Unit/Exception.pm line 13.
> 5) ImapTest.urlauth-binary
>   Boolean assertion failed at 
> /usr/share/perl5/vendor_perl/Test/Unit/Exception.pm line 13.

The "ImapTest.*-binary*" tests were known to be not-quite-right, and I believe have been removed from ImapTest-- my build is from 20190509 and doesn't contain them.  If you can't update ImapTest, it's reasonable to just suppress these three instead.

> 10) Cassandane::Test::Core.core_files_*
>    didn't match /(?^:Core files found)/ at Cassandane/Test/Core.pm line 96.

The RedHat environment is probably preventing core files from being created, in which case the Test::Core tests will always fail.  That being the case, I'd suggest just suppressing them.

> The details are attached, if it could be of interest.

There's no attachment, did the mailing list eat it?  Feel free to send to me directly

If you update to the latest Cassandane (for the syslog fixes), then you'll start seeing failures on 3.2.1 from these tests:

JMAPEmail.blob_get
JMAPEmail.email_query_guidsearch_mixedfilter

These tests have been updated to detect bugs that will be fixed in 3.2.2, but of course they fail for 3.2.1 cause the bugs still exist in that version.

Cheers,

ellie

From falon at ruparpiemonte.it  Fri Jun  5 05:53:00 2020
From: falon at ruparpiemonte.it (Marco)
Date: Fri, 5 Jun 2020 11:53:00 +0200
Subject: Renaming mailboxes
In-Reply-To: <67733b36-b987-27ad-e499-3f3a4b0088d3@lane.uk.net>
References: <67733b36-b987-27ad-e499-3f3a4b0088d3@lane.uk.net>
Message-ID: <a60fba56-ad70-1492-f222-83b13f3a47e3@ruparpiemonte.it>


Hello David,

Il 27/05/2020 16:48, David Moyes ha scritto:
> 
> I want to rename a mailbox (and all of its descendants), however I am
> struggling with what, on the surface, would appear to be a simple task.
> 
> I first tried to use cyradm rename command to rename user.foo to
> user.foo at example.com, like this:
> 
>      localhost> rename user.foo user.foo at example.com
> 
> However the rename command does not appear to be recursive, and I could
> not find any argument to tell it to recurse. Do I have this right or am
> I doing something wrong?

from what I remember, the rename is "recursive". If you login as an 
admin you can simply rename all users mailboxes from a name to a new 
name in a single step.

If you login as the user, then you have to move all folders as you 
describe below. Pay attention to specialuse folders and to the xapian 
indexes which could increase a lot.

I didn't remember on cyradm, but I currently use Cyrus::IMAP::Admin for 
almost all my scripts. I don't know if something of this could be useful 
for you:

https://github.com/falon/cyr_scripts
https://cloudsmith.io/~csi/repos/cyrus-scripts/packages/

(see at cyr_moveMailboxPart.pl, cyr_moveINBOX.pl)

Greetings
Marco

> I decided to try it in a Perl script using Cyrus::IMAP::Admin. By doing
> a `list` I could get the mailbox and its descendants like this:
> 
>      @mailboxes = $client->list("user/$oldname");
>      @mailboxes_sub = $client->list("user/$oldname/*");
>      push (@mailboxes , @mailboxes_sub);
> 
> (I found that listing "user/$oldname*" got me user.foo but also
> user.foobar so I had to combine two calls to prevent that.)
> 
> Then I could iterate over the mailboxes doing
> 
>      $client->rename( $oldmailbox, $mailbox );
> 
> However, nothing works after the first rename; the program just exits,
> not even "or die" error messages are displayed. In case it means
> anything, the exit status is 141 which I think indicates a SIGPIPE (as
> confirmed using `kill -l "$?"` which returns `PIPE`).
> 
> I did also notice a similar thing in cyradm, that after doing a rename
> that subsequent action caused cyradm to exit:
> 
> localhost> rename user/testuser/Trash user/test at example.net/Trash
> renamemailbox:
> localhost> lm          # this curiously returns nothing!
> 
> localhost> echo $?
> 0
> localhost> lm
> [cyrus at fee7fffd9be7 ~]$ echo $?
> 141
> 
> 
> Any help would be greatly appreciated. I am currently using cyrus-imapd
> 3.0.13-3 on Arch Linux.
> 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 


From ianw at checksum.net.au  Sat Jun  6 00:21:42 2020
From: ianw at checksum.net.au (Ian)
Date: Sat, 06 Jun 2020 14:21:42 +1000
Subject: Path to shared Calendars and task lists
Message-ID: <c330d52e12c3c542ca6b86734be5e39837788ada.camel@checksum.net.au>

Hi All,

One thing that I wasn't sure from the documentation was where shared
calendars and tasklists are on the CalDav server.

If joe has permission to fred's calendar 

https://servername/dav/calendars/user/joe/Default

Attempting access to the following by joe fails even when joe has full
access to all of fred's folders. What URL should be used?
https://servername/dav/calendars/user/fred/Default

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200606/c162265c/attachment.html>

From info-cyrus at jelmail.com  Mon Jun  8 04:51:51 2020
From: info-cyrus at jelmail.com (David Moyes)
Date: Mon, 8 Jun 2020 09:51:51 +0100
Subject: Sieve EditHeaders and Logging
In-Reply-To: <2b765944-9898-4a29-bde5-da9580d6c44d@www.fastmail.com>
References: <946d462d-4437-6c01-b2cf-388f2cd89633@lane.uk.net>
 <2b765944-9898-4a29-bde5-da9580d6c44d@www.fastmail.com>
Message-ID: <e5f86d97-0a7f-d523-5e3a-4443254ae826@lane.uk.net>



On 05/06/2020 05:29, ellie timoney wrote:
> Hi David,
> 
>> Is it possible to enable the "editheaders" sieve extension? if so, how?
> 
> Not in 3.0, but it's available in 3.2

I plan to update to 3.2 once I have 3.0 working in my environment (I'm
migrating an existing legacy 2.5.3 server).

> 
>> Are sieve actions logged anywhere, e.g. to aid with debugging?
> 
> Generally? I don't know.  Maybe if you increase your syslog log level to "debug" and add "debug: yes" to your imapd.conf?
> 
> 3.2 has an experimental "x-cyrus-log" extension, which adds an action that produces a log line.  You probably don't want to allow this for user-supplied sieve scripts, but if you have some sort of "sieve builder" system that your users use (rather than uploading their own handcrafted sieve scripts), then, if you were running 3.2, this extension could be useful.  On the master branch, and therefore in the next major release (2021), this extension will be formalised as "vnd.cyrus.log".

thanks. I have a number of questions around logging; I'll post another
question about that.


From miguel at prodemge.gov.br  Mon Jun  8 11:37:43 2020
From: miguel at prodemge.gov.br (Miguel Mucio Santos Moreira)
Date: Mon, 08 Jun 2020 12:37:43 -0300
Subject: Cyrus Murder Environment Upgrade
Message-ID: <600-5ede5b80-19-7a296480@233731763>


Hello guys!

We have a cyrus murder environment with cyrus 2.4 version, according to cyrus documentation we should upgrade backend servers, mupdate master and for last frontend servers.
Following this recommendantions we've installed new backend servers, with 2.5 version, joined them to the cluster and migrated mailboxes from old servers to new servers throught xfer.
Now we're in doubt about how is the best solution to replace the mupdate master server for a new one.
Nowadays we have around 16K mailboxes.

Any hint?

Thanks in advance

Cheers!

--

Miguel Moreira
DTE/SRE/GRE - Ger?ncia de Redes
+55(31)3339-1401
PRODEMGE - Companhia de Tecnologia da Informa??o do Estado de Minas Gerais


Aviso: Esta mensagem ? destinada exclusivamente para a(s) pessoa(s) a quem ? dirigida, podendo conter informa??o sigilosa e legalmente protegida. O uso impr?prio ser? tratado conforme as normas da empresa e a legisla??o em vigor. Caso n?o seja o destinat?rio, favor notificar o remetente, ficando proibidas a utiliza??o, divulga??o, c?pia e distribui??o.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200608/eafaa948/attachment.html>

From wbreyha at gmx.net  Tue Jun  9 04:31:33 2020
From: wbreyha at gmx.net (Wolfgang Breyha)
Date: Tue, 9 Jun 2020 10:31:33 +0200
Subject: Cyrus Murder Environment Upgrade
In-Reply-To: <600-5ede5b80-19-7a296480@233731763>
References: <600-5ede5b80-19-7a296480@233731763>
Message-ID: <3aa713ef-f140-0ed4-7750-1bd6d4d1ecc1@gmx.net>

On 08/06/2020 17:37, Miguel Mucio Santos Moreira wrote:
> Now we're in doubt about how is the best solution to replace the mupdate
> master server for a new one.
> Nowadays we have around 16K mailboxes.

IIRC we simply replaced the mupdate server and did a "ctl_mboxlist -m" on
all backends to fill it. Shouldn't take that long since we did it with 130k
on 8 backends in under 20 minutes (or even shorter).

Greetings, Wolfgang
--
Wolfgang Breyha <wbreyha at gmx.net> | https://www.blafasel.at/
Vienna University Computer Center | Austria

From miguel at prodemge.gov.br  Tue Jun  9 08:56:11 2020
From: miguel at prodemge.gov.br (Miguel Mucio Santos Moreira)
Date: Tue, 09 Jun 2020 09:56:11 -0300
Subject: =?utf-8?q?Re=3A?= Cyrus Murder Environment Upgrade
In-Reply-To: <3aa713ef-f140-0ed4-7750-1bd6d4d1ecc1@gmx.net>
Message-ID: <3f88-5edf8700-5-668e1400@141699943>


Dear Wolfgang,

Firstly thanks for your answer, secondly I have one more doubt, during this time where the new Mupdate Master is receiving mailboxes information from backend servers, is necessary stopping comunications between frontend servers and mupdate master or none action is necessary besides that one you've mentioned before?
We're concerned if frontend servers will connect to Mupdate Master and receive from it an information which there's no mailboxes anymore until the backend push entirely mailboxes information and this situation to cause any trouble.

One more time, thanks

Cheers!

--

Miguel Moreira
DTE/SRE/GRE - Ger?ncia de Redes
+55(31)3339-1401
PRODEMGE - Companhia de Tecnologia da Informa??o do Estado de Minas Gerais


Aviso: Esta mensagem ? destinada exclusivamente para a(s) pessoa(s) a quem ? dirigida, podendo conter informa??o sigilosa e legalmente protegida. O uso impr?prio ser? tratado conforme as normas da empresa e a legisla??o em vigor. Caso n?o seja o destinat?rio, favor notificar o remetente, ficando proibidas a utiliza??o, divulga??o, c?pia e distribui??o. Em Ter?a, Junho 09, 2020 05:31 -03, Wolfgang Breyha <wbreyha at gmx.net> escreveu:On 08/06/2020 17:37, Miguel Mucio Santos Moreira wrote:
> Now we're in doubt about how is the best solution to replace the mupdate
> master server for a new one.
> Nowadays we have around 16K mailboxes.

IIRC we simply replaced the mupdate server and did a "ctl_mboxlist -m" on
all backends to fill it. Shouldn't take that long since we did it with 130k
on 8 backends in under 20 minutes (or even shorter).

Greetings, Wolfgang
--
Wolfgang Breyha <wbreyha at gmx.net> | https://www.blafasel.at/
Vienna University Computer Center | Austria
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200609/29d60871/attachment.html>

From wbreyha at gmx.net  Tue Jun  9 09:39:11 2020
From: wbreyha at gmx.net (Wolfgang Breyha)
Date: Tue, 9 Jun 2020 15:39:11 +0200
Subject: Cyrus Murder Environment Upgrade
In-Reply-To: <3f88-5edf8700-5-668e1400@141699943>
References: <3f88-5edf8700-5-668e1400@141699943>
Message-ID: <40e951d9-111a-e811-2a51-073413633d52@gmx.net>

Hi!

On 09/06/2020 14:56, Miguel Mucio Santos Moreira wrote:
> Dear Wolfgang,
>
> Firstly thanks for your answer, secondly I have one more doubt, during this
> time where the new Mupdate Master is receiving mailboxes information from
> backend servers, is necessary stopping comunications between frontend
> servers and mupdate master or none action is necessary besides that one
> you've mentioned before?
> We're concerned if frontend servers will connect to Mupdate Master and
> receive from it an information which there's no mailboxes anymore until the
> backend push entirely mailboxes information and this situation to cause any
> trouble.

I recommend the follow steps:
*) check that your currently running setup has no conflicts in mailboxes.db
   by running "ctl_mboxlist -mw". It is possible that you see output if
   somebody changes his mailboxes while you run the command, but it should
   not appear again if you run the command a second time. If everything is
   fine...

*) shut down all running cyrus
   frontends first, then backends and mupdate last
*) backup your mailboxes.db on mupdate server
*) replace/update mupdate and start it with empty/removed mailboxes.db.
*) backup your mailboxes.db on the backends
*) do the ctl_mboxlist -m by hand on the backends (only one at the same
   time) you can check if everything went ok with "-mw" at any time
   afterwards.
*) start cyrus on backends

In our setup the frontends have mupdate running as well. I can't currently
remember if this is mandatory. If this is true in your setup then:
*) remove or rename the mailboxes.db database on the frontends and start
   them. They will fetch the database immediately from the mupdate server.
   This is visible in syslog as well and takes about 30 seconds in our
   setup (~250MB mailboxes.db).
otherwise
*) start the frontends

At this point everything should be up and running again.

Watching syslog output all the time usually helps.

IIRC we updated the mupdate server to 2.5 first. Then we had a mix of 2.4
and 2.5 backends for some time. The 2.5 backends had some capabilities
suppressed. Frontends had 2.4 until all backends had 2.5. Last but not
least we upgraded to 2.5 on the frontends.

If you need more info/details feel free to ask.

Greetings, Wolfgang
--
Wolfgang Breyha <wbreyha at gmx.net> | https://www.blafasel.at/
Vienna University Computer Center | Austria

From Albert.Shih at obspm.fr  Tue Jun  9 09:51:42 2020
From: Albert.Shih at obspm.fr (Albert Shih)
Date: Tue, 9 Jun 2020 15:51:42 +0200
Subject: What you do with old account
Message-ID: <20200609135142.GI4484@io.chezmoi.fr>

Hi

I like to know what you do in time with the olds accounts. Long time ago
(well 2 years) we using dovecot as imap server. And when some accounts are
closed we create a archive of the person mailbox (with tar), put that in
some backup server where the file going to stay about 2 years. After that
the data are destroy

The reason we keep that is some time (30% of the time) the person going to
come back and event it's not mandatory for us, the person are very happy to
go it old mailbox.

After switching to cyrus imap, I think about how to do that.

If I'm correct I cannot just copy the file somewhere else, because cyrus
database would keep the information about the existance of the mailbox, so
what will the ?state of the art? way to remove a mail account and all the
mail.

And how what would be the ?state of the art? way to put it back ?

Regards.

JAS

--
Albert SHIH
DIO b?timent 15
Observatoire de Paris
Heure local/Local time:
Tue 09 Jun 2020 03:44:13 PM CEST

From miguel at prodemge.gov.br  Tue Jun  9 10:01:40 2020
From: miguel at prodemge.gov.br (Miguel Mucio Santos Moreira)
Date: Tue, 09 Jun 2020 11:01:40 -0300
Subject: =?utf-8?q?Re=3A?= Cyrus Murder Environment Upgrade
In-Reply-To: <40e951d9-111a-e811-2a51-073413633d52@gmx.net>
Message-ID: <4d9b-5edf9680-5-2188c540@104545673>


Wolfgang,

I'm sure your help and experience with Cyrus Murder upgrading will save a lot of time and reduce the possibility of an eventual problem during the upgrade.

Thankful

--

Miguel Moreira
DTE/SRE/GRE - Ger?ncia de Redes
+55(31)3339-1401
PRODEMGE - Companhia de Tecnologia da Informa??o do Estado de Minas Gerais


Aviso: Esta mensagem ? destinada exclusivamente para a(s) pessoa(s) a quem ? dirigida, podendo conter informa??o sigilosa e legalmente protegida. O uso impr?prio ser? tratado conforme as normas da empresa e a legisla??o em vigor. Caso n?o seja o destinat?rio, favor notificar o remetente, ficando proibidas a utiliza??o, divulga??o, c?pia e distribui??o. Em Ter?a, Junho 09, 2020 10:39 -03, Wolfgang Breyha <wbreyha at gmx.net> escreveu:Hi!

On 09/06/2020 14:56, Miguel Mucio Santos Moreira wrote:
> Dear Wolfgang,
>
> Firstly thanks for your answer, secondly I have one more doubt, during this
> time where the new Mupdate Master is receiving mailboxes information from
> backend servers, is necessary stopping comunications between frontend
> servers and mupdate master or none action is necessary besides that one
> you've mentioned before?
> We're concerned if frontend servers will connect to Mupdate Master and
> receive from it an information which there's no mailboxes anymore until the
> backend push entirely mailboxes information and this situation to cause any
> trouble.

I recommend the follow steps:
*) check that your currently running setup has no conflicts in mailboxes.db
by running "ctl_mboxlist -mw". It is possible that you see output if
somebody changes his mailboxes while you run the command, but it should
not appear again if you run the command a second time. If everything is
fine...

*) shut down all running cyrus
frontends first, then backends and mupdate last
*) backup your mailboxes.db on mupdate server
*) replace/update mupdate and start it with empty/removed mailboxes.db.
*) backup your mailboxes.db on the backends
*) do the ctl_mboxlist -m by hand on the backends (only one at the same
time) you can check if everything went ok with "-mw" at any time
afterwards.
*) start cyrus on backends

In our setup the frontends have mupdate running as well. I can't currently
remember if this is mandatory. If this is true in your setup then:
*) remove or rename the mailboxes.db database on the frontends and start
them. They will fetch the database immediately from the mupdate server.
This is visible in syslog as well and takes about 30 seconds in our
setup (~250MB mailboxes.db).
otherwise
*) start the frontends

At this point everything should be up and running again.

Watching syslog output all the time usually helps.

IIRC we updated the mupdate server to 2.5 first. Then we had a mix of 2.4
and 2.5 backends for some time. The 2.5 backends had some capabilities
suppressed. Frontends had 2.4 until all backends had 2.5. Last but not
least we upgraded to 2.5 on the frontends.

If you need more info/details feel free to ask.

Greetings, Wolfgang
--
Wolfgang Breyha <wbreyha at gmx.net> | https://www.blafasel.at/
Vienna University Computer Center | Austria
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200609/e4ac14e2/attachment-0001.html>

From awilliam at whitemice.org  Tue Jun  9 10:32:03 2020
From: awilliam at whitemice.org (Adam Tauno Williams)
Date: Tue, 09 Jun 2020 10:32:03 -0400
Subject: What you do with old account
In-Reply-To: <20200609135142.GI4484@io.chezmoi.fr>
References: <20200609135142.GI4484@io.chezmoi.fr>
Message-ID: <1591713123.5805.8.camel@whitemice.org>

On Tue, 2020-06-09 at 15:51 +0200, Albert Shih wrote:> 
> After switching to cyrus imap, I think about how to do that.
> If I'm correct I cannot just copy the file somewhere else, because cyrus
> database would keep the information about the existance of the mailbox, so
> what will the ?state of the art? way to remove a mail account and all the
> mail.
> And how what would be the ?state of the art? way to put it back ?

I create a calendar event [task] to delete the mailbox and otherwise
just leave it. If the account itself is disabled it cannot be accessed.

Putting things back-into a mailstore is too much of a pain with current
storage prices.

-- 
Adam Tauno Williams, awilliam at whitemice.org
Multi-Modal Activists Against Auto Dependent Development
resisting the unAmerican socialists of the Motorist hegemony
http://www.mmaaadd.org 

From miguel at prodemge.gov.br  Wed Jun 10 14:47:38 2020
From: miguel at prodemge.gov.br (Miguel Mucio Santos Moreira)
Date: Wed, 10 Jun 2020 15:47:38 -0300
Subject: =?utf-8?q?Re=3A?= Cyrus Murder Environment Upgrade
In-Reply-To: <40e951d9-111a-e811-2a51-073413633d52@gmx.net>
Message-ID: <930-5ee12b00-3-6abd6400@207816469>


Wolfgang,

If you don't mind, I'd like to know if when you upgraded backend servers from 2.4 version to 2.5 version you have an increase, in metadata size.Here we had 30% around of increase

Thanks one more time

Greetings
--

Miguel Moreira
DTE/SRE/GRE - Ger?ncia de Redes
+55(31)3339-1401
PRODEMGE - Companhia de Tecnologia da Informa??o do Estado de Minas Gerais


Aviso: Esta mensagem ? destinada exclusivamente para a(s) pessoa(s) a quem ? dirigida, podendo conter informa??o sigilosa e legalmente protegida. O uso impr?prio ser? tratado conforme as normas da empresa e a legisla??o em vigor. Caso n?o seja o destinat?rio, favor notificar o remetente, ficando proibidas a utiliza??o, divulga??o, c?pia e distribui??o. Em Ter?a, Junho 09, 2020 10:39 -03, Wolfgang Breyha <wbreyha at gmx.net> escreveu:Hi!

On 09/06/2020 14:56, Miguel Mucio Santos Moreira wrote:
> Dear Wolfgang,
>
> Firstly thanks for your answer, secondly I have one more doubt, during this
> time where the new Mupdate Master is receiving mailboxes information from
> backend servers, is necessary stopping comunications between frontend
> servers and mupdate master or none action is necessary besides that one
> you've mentioned before?
> We're concerned if frontend servers will connect to Mupdate Master and
> receive from it an information which there's no mailboxes anymore until the
> backend push entirely mailboxes information and this situation to cause any
> trouble.

I recommend the follow steps:
*) check that your currently running setup has no conflicts in mailboxes.db
by running "ctl_mboxlist -mw". It is possible that you see output if
somebody changes his mailboxes while you run the command, but it should
not appear again if you run the command a second time. If everything is
fine...

*) shut down all running cyrus
frontends first, then backends and mupdate last
*) backup your mailboxes.db on mupdate server
*) replace/update mupdate and start it with empty/removed mailboxes.db.
*) backup your mailboxes.db on the backends
*) do the ctl_mboxlist -m by hand on the backends (only one at the same
time) you can check if everything went ok with "-mw" at any time
afterwards.
*) start cyrus on backends

In our setup the frontends have mupdate running as well. I can't currently
remember if this is mandatory. If this is true in your setup then:
*) remove or rename the mailboxes.db database on the frontends and start
them. They will fetch the database immediately from the mupdate server.
This is visible in syslog as well and takes about 30 seconds in our
setup (~250MB mailboxes.db).
otherwise
*) start the frontends

At this point everything should be up and running again.

Watching syslog output all the time usually helps.

IIRC we updated the mupdate server to 2.5 first. Then we had a mix of 2.4
and 2.5 backends for some time. The 2.5 backends had some capabilities
suppressed. Frontends had 2.4 until all backends had 2.5. Last but not
least we upgraded to 2.5 on the frontends.

If you need more info/details feel free to ask.

Greetings, Wolfgang
--
Wolfgang Breyha <wbreyha at gmx.net> | https://www.blafasel.at/
Vienna University Computer Center | Austria
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200610/07b8e282/attachment.html>

From nic at nicbernstein.com  Wed Jun 10 14:56:41 2020
From: nic at nicbernstein.com (Nic Bernstein)
Date: Wed, 10 Jun 2020 13:56:41 -0500
Subject: Cyrus Murder Environment Upgrade
In-Reply-To: <930-5ee12b00-3-6abd6400@207816469>
References: <930-5ee12b00-3-6abd6400@207816469>
Message-ID: <72e6f335-77aa-02a9-1bca-f5e3cb7f8f21@nicbernstein.com>

Miguel,
That's perfectly normal, and I think it's even covered in the release 
notes... Nope, I'm wrong, the release notes mention the larger Memory 
footprint, due to more data and metadata being cached in memory.? But 
the on-disk size increases, too, as there is more information being held 
in indexes, etc.

The same is true, again, when going from 2.5 to 3.X

Cheers,
 ??? -nic

On 6/10/20 1:47 PM, Miguel Mucio Santos Moreira wrote:
> Wolfgang,
>
> If you don't mind, I'd like to know if when you upgraded backend 
> servers from 2.4 version to 2.5 version you have an increase, in 
> metadata size.Here we had 30% around of increase
>
> Thanks one more time
>
> Greetings
> --
>
> *Miguel Moreira*
> DTE/SRE/GRE - Ger?ncia de Redes
> +55(31)3339-1401
> PRODEMGE - Companhia de Tecnologia da Informa??o do Estado de Minas Gerais
>
>
> Aviso: Esta mensagem ? destinada exclusivamente para a(s) pessoa(s) a 
> quem ? dirigida, podendo conter informa??o sigilosa e legalmente 
> protegida. O uso impr?prio ser? tratado conforme as normas da empresa 
> e a legisla??o em vigor. Caso n?o seja o destinat?rio, favor notificar 
> o remetente, ficando proibidas a utiliza??o, divulga??o, c?pia e 
> distribui??o. Em Ter?a, Junho 09, 2020 10:39 -03, Wolfgang Breyha 
> <wbreyha at gmx.net> escreveu:
>> Hi!
>>
>> On 09/06/2020 14:56, Miguel Mucio Santos Moreira wrote:
>> > Dear Wolfgang,
>> >
>> > Firstly thanks for your answer, secondly I have one more doubt, 
>> during this
>> > time where the new Mupdate Master is receiving mailboxes 
>> information from
>> > backend servers, is necessary stopping comunications between frontend
>> > servers and mupdate master or none action is necessary besides that one
>> > you've mentioned before?
>> > We're concerned if frontend servers will connect to Mupdate Master and
>> > receive from it an information which there's no mailboxes anymore 
>> until the
>> > backend push entirely mailboxes information and this situation to 
>> cause any
>> > trouble.
>>
>> I recommend the follow steps:
>> *) check that your currently running setup has no conflicts in 
>> mailboxes.db
>> by running "ctl_mboxlist -mw". It is possible that you see output if
>> somebody changes his mailboxes while you run the command, but it should
>> not appear again if you run the command a second time. If everything is
>> fine...
>>
>> *) shut down all running cyrus
>> frontends first, then backends and mupdate last
>> *) backup your mailboxes.db on mupdate server
>> *) replace/update mupdate and start it with empty/removed mailboxes.db.
>> *) backup your mailboxes.db on the backends
>> *) do the ctl_mboxlist -m by hand on the backends (only one at the same
>> time) you can check if everything went ok with "-mw" at any time
>> afterwards.
>> *) start cyrus on backends
>>
>> In our setup the frontends have mupdate running as well. I can't 
>> currently
>> remember if this is mandatory. If this is true in your setup then:
>> *) remove or rename the mailboxes.db database on the frontends and start
>> them. They will fetch the database immediately from the mupdate server.
>> This is visible in syslog as well and takes about 30 seconds in our
>> setup (~250MB mailboxes.db).
>> otherwise
>> *) start the frontends
>>
>> At this point everything should be up and running again.
>>
>> Watching syslog output all the time usually helps.
>>
>> IIRC we updated the mupdate server to 2.5 first. Then we had a mix of 2.4
>> and 2.5 backends for some time. The 2.5 backends had some capabilities
>> suppressed. Frontends had 2.4 until all backends had 2.5. Last but not
>> least we upgraded to 2.5 on the frontends.
>>
>> If you need more info/details feel free to ask.
>>
>> Greetings, Wolfgang
>> --
>> Wolfgang Breyha <wbreyha at gmx.net> | https://www.blafasel.at/
>> Vienna University Computer Center | Austria
>
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
Nic Bernstein                               nic at nicbernstein.com
https://www.nicbernstein.com
https://www.linkedin.com/in/nic-b-26577a178/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200610/01ee46cd/attachment.html>

From wbreyha at gmx.net  Thu Jun 11 05:25:19 2020
From: wbreyha at gmx.net (Wolfgang Breyha)
Date: Thu, 11 Jun 2020 11:25:19 +0200
Subject: Cyrus Murder Environment Upgrade
In-Reply-To: <930-5ee12b00-3-6abd6400@207816469>
References: <930-5ee12b00-3-6abd6400@207816469>
Message-ID: <d40ebc05-b277-3e32-c006-201b7d55b785@gmx.net>

On 10/06/2020 20:47, Miguel Mucio Santos Moreira wrote:
> Wolfgang,
>
> If you don't mind, I'd like to know if when you upgraded backend servers
> from 2.4 version to 2.5 version you have an increase, in metadata size.Here
> we had 30% around of increase

Yes, cyrus.* are larger. Not sure about .cache, but .index for sure. And
the databases will grow as well if you change to twoskip.

We had to go back to skiplist on some hosts after the database reached
~200MB due to long locking times. The frontends started to have troubles
with long locks leading to timeouts on connect.

Greetings, Wolfgang
--
Wolfgang Breyha <wbreyha at gmx.net> | https://www.blafasel.at/
Vienna University Computer Center | Austria

From falon at ruparpiemonte.it  Fri Jun 12 03:35:30 2020
From: falon at ruparpiemonte.it (Marco)
Date: Fri, 12 Jun 2020 09:35:30 +0200
Subject: CalDav CardDav webmail client ?
Message-ID: <1f4dbf85-b9ac-8ac5-8525-b0467e132eaa@ruparpiemonte.it>

Hello,

  I'm interesting in Cyrus Caldav/Carddav server. I never used it 
before, because most webmail solutions implement their own calendar and 
addressbook.

I wonder if I can interface Cyrus CalDav/CardDav in some opensource 
webmail, such as SOGo, Roundcube, or Horde Webmail.

Could you help me or tell me about your experience about these integrations?

Thank you very much
Greetings
Marco

From nd at syndicat.com  Fri Jun 12 05:14:23 2020
From: nd at syndicat.com (Niels Dettenbach)
Date: Fri, 12 Jun 2020 11:14:23 +0200
Subject: CalDav CardDav webmail client ?
In-Reply-To: <1f4dbf85-b9ac-8ac5-8525-b0467e132eaa@ruparpiemonte.it>
References: <1f4dbf85-b9ac-8ac5-8525-b0467e132eaa@ruparpiemonte.it>
Message-ID: <3253660.QJadu78ljV@gongo>

Am Freitag, 12. Juni 2020, 09:35:30 CEST schrieb Marco:

from my last experiences / knowledge:
> Roundcube, 
is not able to use "external" CardDav/CalDav server

But there seem to exist external (commercial?) plugins which allow "manual 
configurable" client mode (see bottom):
https://roundcubeplus.com/tutorials/caldav/creating-caldav-connection

some third party developments:
https://github.com/christian-putzke/Roundcube-CardDAV
https://packagist.org/packages/roundcube/carddav


> or Horde Webmail.
Horde 4 has it's own CardDAV/CalDAV server implementation wwhich is default. 
However, it provides usage / connectivity to "special" external IMAP based 
CardDAV/CalDAV within the Kolab project which is developed on/with cyrus, but 
still does not provide the new HTTP standards mechs..

"manual client mode":
=================
But Horde is able to "use" CalDAV" als client by "external calendars" over 
HTTP ressources, but must be manually added / configured to a users account 
(with ugly manual auth). So this is probably not what you looking for.

Would be cool if Horde 4 get Cyrus HTTP support too in the future.


cheers,


niels.


-- 
 ---
 Niels Dettenbach
 Syndicat IT & Internet
 http://www.syndicat.com
 PGP: https://syndicat.com/pub_key.asc
 ---
 







From xavier.bestel at free.fr  Fri Jun 12 08:01:27 2020
From: xavier.bestel at free.fr (Xavier Bestel)
Date: Fri, 12 Jun 2020 14:01:27 +0200
Subject: CalDav CardDav webmail client ?
In-Reply-To: <3253660.QJadu78ljV@gongo>
References: <1f4dbf85-b9ac-8ac5-8525-b0467e132eaa@ruparpiemonte.it>
 <3253660.QJadu78ljV@gongo>
Message-ID: <437215efa46088b2c8c8edc72e608f2a351f42e9.camel@free.fr>

A bit on the heavy side, but Nextcloud's agenda component is CalDAV and
can access external CalDAV/ICS calendars. However its mail client will
only access its own address book (which is CardDAV).

	Xav


From igb at batten.eu.org  Sat Jun 13 08:31:50 2020
From: igb at batten.eu.org (Ian Batten)
Date: Sat, 13 Jun 2020 13:31:50 +0100
Subject: Correct use of cyr_expire on replicas when expire annotations
 are in use
Message-ID: <etPan.5ee4c73b.4f519066.a05@batten.eu.org>

What flags should I supply to cyr_expire in a replication configuration when I have expire annotations in use?

I have a replication relationship from a 2.5.12 machine (in the process of being replaced, but currently the master) to a?3.0.8-6+deb10u4 ?(as replica).

I have had problems with old messages not being deleted on the replica, and found I was not running cyr_expire on the replica. ?I have configured it, so I have:

master:?cmd="cyr_expire -E 7 -X 3 -D 3" at=0100

slave: cmd="cyr_expire -E 7 -X 3 -D 3" at=0300

Since I installed this, I have had periodic replication failures. ?What seems to happen is this:

?MAILBOX received NO response: IMAP_SYNC_CHECKSUM Checksum Failure
?CRC failure on sync for user.igb.routine, trying full update
?SYNCNOTICE: highestmodseq higher on replica user.igb.routine, updating 49317 => 49321
?SYNCERROR: guid mismatch user.igb.routine 46708 (890a6db331aaccd8c423bbff598bb81bc3c19146 87fb316dea9c9dbbed691d11759b7b1e000d5487)
?FETCH received NO response: IMAP_PROTOCOL_BAD_PARAMETERS
?SYNCNOTICE: failed to prepare update for user.igb.routine: The remote Server(s) denied the operation
?do_folders(): update failed: user.igb.routine 'The remote Server(s) denied the operation'
?MAILBOX received NO response: IMAP_SYNC_CHECKSUM Checksum Failure
?CRC failure on sync for user.igb.routine, trying full update
?SYNCERROR: guid mismatch user.igb.routine 46708 (890a6db331aaccd8c423bbff598bb81bc3c19146 87fb316dea9c9dbbed691d11759b7b1e000d5487)
?FETCH received NO response: IMAP_PROTOCOL_BAD_PARAMETERS
?SYNCNOTICE: failed to prepare update for user.igb.routine: The remote Server(s) denied the operation
?do_folders(): update failed: user.igb.routine 'The remote Server(s) denied the operation'
?IOERROR: The remote Server(s) denied the operation
?Error in do_sync(): bailing out! The remote Server(s) denied the operation


The synchronisation continues to do this, making no progress. ?The clean-up is straightforward: I delete all the files in user.igb.routine on the replica, reconstruct -G -U user.igb.routine on replica, sync_client -m user.igb.routine on the master. ? ?It then runs for a few days, and then goes bang again: suggestively, always in the early but not too early hours of the morning. ?What?s going wrong, and what?s needed to fix it properly?

user.igb.routine is the only mailbox I have which both (a) is relatively high traffic and (b) has the expire annotation.

My hypothesis (I only have 8 days of searchable logs, so this is n=small) ?is that the failure happens as we replicate the first message to be delivered to user.igb.routine after 0300. ? My supposition is that a trap is set by the delivery of a message into user.igb.routine between 0100 and 0300. ?3 (the value of the ?expire? annotation) days later at 0300, that message is deleted on the replica (as it is >(86400*3) seconds old) but was not deleted on the master (because when expire ran on the master, at 0100, the message was _not_ >(86400*3) seconds old). ?Something bad happens during the replication run, and bad things then continue to ensue.

To test this theory my original thought was to run expire on the slave at the same time it runs on the master, making the race condition much smaller. ?But I see that in 3.0.8 there is a ?-a? flag to cyr_expire which suppresses processing of the expire annotation, which I assume deals with this case.

Am I thinking along the right lines?

ian


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200613/a2a82db8/attachment.html>

From Hagedorn at uni-koeln.de  Sun Jun 14 15:51:49 2020
From: Hagedorn at uni-koeln.de (Sebastian Hagedorn)
Date: Sun, 14 Jun 2020 21:51:49 +0200
Subject: What you do with old account
In-Reply-To: <1591713123.5805.8.camel@whitemice.org>
References: <20200609135142.GI4484@io.chezmoi.fr>
 <1591713123.5805.8.camel@whitemice.org>
Message-ID: <b6b753ee-f16c-b86d-5241-38b6bb3b3a17@uni-koeln.de>

Am 09.06.20 um 16:32 schrieb Adam Tauno Williams:
> On Tue, 2020-06-09 at 15:51 +0200, Albert Shih wrote:> 
>> After switching to cyrus imap, I think about how to do that.
>> If I'm correct I cannot just copy the file somewhere else, because cyrus
>> database would keep the information about the existance of the mailbox, so
>> what will the ?state of the art? way to remove a mail account and all the
>> mail.
>> And how what would be the ?state of the art? way to put it back ?
> I create a calendar event [task] to delete the mailbox and otherwise
> just leave it. If the account itself is disabled it cannot be accessed.
> 
> Putting things back-into a mailstore is too much of a pain with current
> storage prices.

We have about 90,000 accounts, and our current model is that we leave
expired accounts around for a year. The user can't login, and we don't
accept new mails, but it's still there in case the account is
re-activated. After one year the mailbox hierarchy is put into a .tgz
and written to tape. When that is done the account is permanently
deleted. If the user should come back, they get a completely new
account. I can't recall a single instance where the .tgz was ever
needed, but that's not my problem. After one more year the .tgz is
deleted from tape as well.
-- 
   .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:.
                .:.Regionales Rechenzentrum (RRZK).:.
  .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Hagedorn.vcf
Type: text/x-vcard
Size: 333 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200614/8f24a778/attachment.vcf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5367 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200614/8f24a778/attachment.p7s>

From egoitz at sarenet.es  Wed Jun 17 07:26:49 2020
From: egoitz at sarenet.es (egoitz at sarenet.es)
Date: Wed, 17 Jun 2020 13:26:49 +0200
Subject: Question about replication, split-brains and unclean failover
Message-ID: <2837BC2C-01A3-44E0-A3B9-C7F3851595D7@sarenet.es>

Hi!,


I was writing some code for automating the server fail-over and was trying to see, how should or could I handle not run log files from sync_client. Well when in a clean shutdown, it?s pretty easy to know how to manage because the replication is up-to-date? so almost no problem there, it?s pretty fast... The problem comes in an unclean shutdown where some delay exists.

I sometimes suffer about unclean shutdowns, the way are described here https://www.fastmail.com/help/technical/architecture.html "Unclean failover?. There sais too, some improvements where going to be done (or that were committed perhaps) to the replication in order to avoid them. But, by the way replication is handled (as I have seen in the source code about locks, modseq checks and so) and the way Cyrus writes replication logs for rolling replication later, the possibility of replying the logs from an actual slave (just failed over), for covering a split-brain I assume is not carried out at least nowadays?. Perhaps am I wrong?. I ask this just, for confirming and avoid having wrong ideas? 

If it?s undone or at least partially undone, I would love really doing something? although unfortunately due to our high work load... I can?t say when I could have some time for it? but I?ll try my bests...



Cheers!

From egoitz at sarenet.es  Wed Jun 17 08:39:36 2020
From: egoitz at sarenet.es (egoitz at sarenet.es)
Date: Wed, 17 Jun 2020 14:39:36 +0200
Subject: CalDav CardDav webmail client ?
In-Reply-To: <1f4dbf85-b9ac-8ac5-8525-b0467e132eaa@ruparpiemonte.it>
References: <1f4dbf85-b9ac-8ac5-8525-b0467e132eaa@ruparpiemonte.it>
Message-ID: <7314AE56-4566-4844-A5BA-0E4898FB6242@sarenet.es>

Hi,

Although we at present, are not running Cyrus Caldav/Carddav, Davical instead (it comes from long time ago), we are running Roundcube. I adapted the Caldav Kolab plugin in order to even support Free/Busy and for fixing some bug? I should have uploaded it to Github or wherever, but I have not have time for getting it ready for that. By the way, it?s not done for Roundcube 1.4. It?s for Roundcube 1.3. We even use Caldavzap (with a lot of js work too) with Roundcube as tasks plugin too.

The only important thing in the combination of all of them, is that you need to use a specific version of Sabre for the contacts plugin and there?s a packaged version of the Contacts plugin with the needed Sabredav version which allows you to run it combined with the Kolab caldav plugin, although it?s slightly limited and slightly buggy (Kolab caldav)?. I hope to have some time for uploading our patches next months? I have not tested them and don?t expect them to work in Roundcube 1.4. by the way? sadly...

Cheers,


> El 12 jun 2020, a las 9:35, Marco <falon at ruparpiemonte.it> escribi?:
> 
> Hello,
> 
> I'm interesting in Cyrus Caldav/Carddav server. I never used it before, because most webmail solutions implement their own calendar and addressbook.
> 
> I wonder if I can interface Cyrus CalDav/CardDav in some opensource webmail, such as SOGo, Roundcube, or Horde Webmail.
> 
> Could you help me or tell me about your experience about these integrations?
> 
> Thank you very much
> Greetings
> Marco
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200617/93820eee/attachment.html>

From nd at syndicat.com  Wed Jun 17 09:09:17 2020
From: nd at syndicat.com (Niels Dettenbach)
Date: Wed, 17 Jun 2020 15:09:17 +0200
Subject: CalDav CardDav webmail client ?
In-Reply-To: <7314AE56-4566-4844-A5BA-0E4898FB6242@sarenet.es>
References: <1f4dbf85-b9ac-8ac5-8525-b0467e132eaa@ruparpiemonte.it>
 <7314AE56-4566-4844-A5BA-0E4898FB6242@sarenet.es>
Message-ID: <2979540.fEcJ0Lxnt5@gongo>

Am Mittwoch, 17. Juni 2020, 14:39:36 CEST schrieb egoitz at sarenet.es:
> Although we at present, are not running Cyrus Caldav/Carddav, Davical
> instead (it comes from long time ago), we are running Roundcube. I adapted
> the Caldav Kolab plugin in order to even support Free/Busy and for fixing
> some bug? I should have uploaded it to Github or wherever, but I have not
> have time for getting it ready for that. By the way, it?s not done for
> Roundcube 1.4. It?s for Roundcube 1.3. We even use Caldavzap (with a lot
> of js work too) with Roundcube as tasks plugin too.
Thanks for that tip,

but it seems requiring PHP5 (which is obsolete today) and don't get any 
secuity updates anymore.

It seems that even many other famous open source projects for CalDAV/CardDAV 
servers stacks are still more maintained - including i.e. Apples Darwin 
Calendar Servers (DCS):
https://www.calendarserver.org/

We use Cyrus IMAP with Horde5 to provide CalDAV/CardDAV as "MS Exchange Sync" 
emulation ("newer ActiveSync") "out of one box" plus a "nice" (responsive) 
Groupware web GUI, but the Calendar / Adressbook server stack is still used 
from Horde (on SQL) while Cyrus serves the Emails for Horde only. Horde5 now 
works on PHP7.x and is still maintained.

It would be nice to get that Horde5 stack completely run with Cyrus as this 
would offer even more compatibility / protocols as the very known "Cyrus 
robustness"...?) ...and on the longer run Horde out of the path for 
compatible clients / users.

just my .02$



niels.

-- 
 ---
 Niels Dettenbach
 Syndicat IT & Internet
 http://www.syndicat.com
 PGP: https://syndicat.com/pub_key.asc
 ---
 







From egoitz at sarenet.es  Wed Jun 17 09:45:23 2020
From: egoitz at sarenet.es (egoitz at sarenet.es)
Date: Wed, 17 Jun 2020 15:45:23 +0200
Subject: CalDav CardDav webmail client ?
In-Reply-To: <2979540.fEcJ0Lxnt5@gongo>
References: <1f4dbf85-b9ac-8ac5-8525-b0467e132eaa@ruparpiemonte.it>
 <7314AE56-4566-4844-A5BA-0E4898FB6242@sarenet.es> <2979540.fEcJ0Lxnt5@gongo>
Message-ID: <7C598D48-2656-4F15-8F30-8825F8508681@sarenet.es>

Hi!

Answering below!!


> El 17 jun 2020, a las 15:09, Niels Dettenbach <nd at syndicat.com> escribi?:
> 
> Am Mittwoch, 17. Juni 2020, 14:39:36 CEST schrieb egoitz at sarenet.es:
>> Although we at present, are not running Cyrus Caldav/Carddav, Davical
>> instead (it comes from long time ago), we are running Roundcube. I adapted
>> the Caldav Kolab plugin in order to even support Free/Busy and for fixing
>> some bug? I should have uploaded it to Github or wherever, but I have not
>> have time for getting it ready for that. By the way, it?s not done for
>> Roundcube 1.4. It?s for Roundcube 1.3. We even use Caldavzap (with a lot
>> of js work too) with Roundcube as tasks plugin too.
> Thanks for that tip,
> 
> but it seems requiring PHP5 (which is obsolete today) and don't get any 
> secuity updates anymore.
> 

You could get it working with PHP7?. It has some work? but you can?.


Cheers!!


> It seems that even many other famous open source projects for CalDAV/CardDAV 
> servers stacks are still more maintained - including i.e. Apples Darwin 
> Calendar Servers (DCS):
> https://www.calendarserver.org/
> 
> We use Cyrus IMAP with Horde5 to provide CalDAV/CardDAV as "MS Exchange Sync" 
> emulation ("newer ActiveSync") "out of one box" plus a "nice" (responsive) 
> Groupware web GUI, but the Calendar / Adressbook server stack is still used 
> from Horde (on SQL) while Cyrus serves the Emails for Horde only. Horde5 now 
> works on PHP7.x and is still maintained.
> 
> It would be nice to get that Horde5 stack completely run with Cyrus as this 
> would offer even more compatibility / protocols as the very known "Cyrus 
> robustness"...?) ...and on the longer run Horde out of the path for 
> compatible clients / users.
> 
> just my .02$
> 
> 
> 
> niels.
> 
> -- 
> ---
> Niels Dettenbach
> Syndicat IT & Internet
> http://www.syndicat.com
> PGP: https://syndicat.com/pub_key.asc
> ---
> 
> 
> 
> 
> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200617/69100c6f/attachment-0001.html>

From falon at ruparpiemonte.it  Thu Jun 18 08:19:43 2020
From: falon at ruparpiemonte.it (Marco)
Date: Thu, 18 Jun 2020 14:19:43 +0200
Subject: backupd and sync_client IOERROR
Message-ID: <7d2c31db-a3b3-fca3-9e25-574c373e7f23@ruparpiemonte.it>

Hello,

  I'm trying to configure backupd in rolling mode as a final setup. 
Running a first backup on few users

sync_client -A -n bck -z -v -v

after a while the process die with:

cyrus/sync_client[9540]: MESSAGE received NO response: 
IMAP_PROTOCOL_ERROR Protocol error
cyrus/sync_client[9540]: do_folders(): update failed: 
example.com!user.utente^archivista 'Bad protocol'
cyrus/sync_client[9540]: IOERROR: do_user_main: Bad protocol for 
utente.archivista at example.com to [no channel] (tst-msg03-bck.example.com)
Error from do_user(utente.archivista at example.com): bailing out!
cyrus/sync_client[9540]: Error in 
do_user(utente.archivista at example.com): bailing out!

The backupd host doesn't complain:

2020-06-18T11:50:30.528584+02:00 tst-msg03-bck backupd[1308172]: 
creating sql_db 
/var/spool/cyr_backup/u/utente.archivista at example.com_cxlsmX.index
2020-06-18T11:56:44.971042+02:00 tst-msg03-bck backupd[1308172]: login: 
tst-msg03.example.com [10.102.42.168] cyr_backup LOGIN User logged in

So I repeated the same command, but with less verbosity:

sync_client -A -n bck -z -v

and it worked well, without errors. Uhm...

Maybe, before to run the first initial backup (sync_client -A), do I 
have to stop the sync_client already working in rolling mode?

http://www.cyrusimap.org/imap/reference/admin/backups.html says "Update 
cyrus.conf(5) to add a sync_client(8) invocation to the SERVICES section 
specifying (at least) the -r and -n channel options.". Really maybe you 
intend START. In SERVICE I need to specify a listen too, this is not 
very clear to me.


Around of backupd I would ask two other questions:

1) The backupd host always claims this:

backupd[1309970]: DBERROR: reading /var/lib/imap/db/skipstamp, assuming 
the worst: No such file or directory

Could you tell me how to fix that DBERROR too?

2) Is there a way to delete a user from backupd host?

Thank you very much
Cheers

Marco



master imapd.conf contains:

bck_sync_host: tst-msg03-bck.example.com
bck_sync_port: csync
bck_sync_authname: cyr_backup
bck_sync_password: *********
bck_sync_log: 1
bck_sync_repeat_interval: 1m
bck_sync_try_imap: no
force_sasl_client_mech: PLAIN, LOGIN
sync_log: on
sync_log_channels: squatter, bck
========================================

master cyrus.conf:
START {
   # do not delete this entry!
   recover       cmd="ctl_cyrusdb -r"

   # backup
   backup        cmd="/sbin/sync_client -r -n bck -z"
}

==========================================

backupd full imapd.conf is:

# Configuration directory
configdirectory: /var/lib/imap
temp_path: /run/cyrus

# Directories for proc and lock files
proc_path: /run/cyrus/proc
mboxname_lockpath: /run/cyrus/lock


# One year backup
backup_retention: 365

backuppartition-name: /var/spool/cyr_backup

# Maybe to improve:
backup_compact_minsize: 0
backup_compact_maxsize: 0
backup_compact_work_threshold: 1

admins: cyr_backup

###################################################################
## User Authentication settings
###################################################################

# Allow plaintext logins by default (SASL PLAIN)
allowplaintext: yes

###################################################################
## SASL library options (these are handled directly by the SASL
## libraries, refer to SASL documentation for an up-to-date list of
## these)
###################################################################

# The mechanism(s) used by the server to verify plaintext passwords.
# Possible values are "saslauthd", "auxprop", "pwcheck" and
# "alwaystrue".  They are tried in order, you can specify more than one,
# separated by spaces.
sasl_pwcheck_method: saslauthd

# If enabled, the SASL library will automatically create authentication
# secrets when given a plaintext password. Refer to SASL documentation
sasl_auto_transition: no

=======================================================================

backupd full cyrus.conf is:

SERVICES {
     # backupd is probably the only service entry your backup server needs
     backupd cmd="backupd" listen="csync" proto="tcp4" prefork=0
}

EVENTS {
     # this is required
     checkpoint cmd="ctl_cyrusdb -c" period=30

     # arrange for compact to run at some interval
     compact cmd="ctl_backups compact -A" at=0400
}
===========================================================================

From ellie at fastmail.com  Thu Jun 18 21:01:02 2020
From: ellie at fastmail.com (ellie timoney)
Date: Fri, 19 Jun 2020 11:01:02 +1000
Subject: backupd and sync_client IOERROR
In-Reply-To: <7d2c31db-a3b3-fca3-9e25-574c373e7f23@ruparpiemonte.it>
References: <7d2c31db-a3b3-fca3-9e25-574c373e7f23@ruparpiemonte.it>
Message-ID: <ba49a89c-04de-4960-b280-b4136715d640@www.fastmail.com>

Hi Marco,

On Thu, Jun 18, 2020, at 10:19 PM, Marco wrote:
> Hello,
> 
>   I'm trying to configure backupd in rolling mode as a final setup. 
> Running a first backup on few users
> 
> sync_client -A -n bck -z -v -v
> 
> after a while the process die with:
> 
> cyrus/sync_client[9540]: MESSAGE received NO response: 
> IMAP_PROTOCOL_ERROR Protocol error
> cyrus/sync_client[9540]: do_folders(): update failed: 
> example.com!user.utente^archivista 'Bad protocol'
> cyrus/sync_client[9540]: IOERROR: do_user_main: Bad protocol for 
> utente.archivista at example.com to [no channel] (tst-msg03-bck.example.com)
> Error from do_user(utente.archivista at example.com): bailing out!
> cyrus/sync_client[9540]: Error in 
> do_user(utente.archivista at example.com): bailing out!
> 
> The backupd host doesn't complain:
> 
> 2020-06-18T11:50:30.528584+02:00 tst-msg03-bck backupd[1308172]: 
> creating sql_db 
> /var/spool/cyr_backup/u/utente.archivista at example.com_cxlsmX.index
> 2020-06-18T11:56:44.971042+02:00 tst-msg03-bck backupd[1308172]: login: 
> tst-msg03.example.com [10.102.42.168] cyr_backup LOGIN User logged in
> 
> So I repeated the same command, but with less verbosity:
> 
> sync_client -A -n bck -z -v
> 
> and it worked well, without errors. Uhm...
> 
> Maybe, before to run the first initial backup (sync_client -A), do I 
> have to stop the sync_client already working in rolling mode?

Rolling mode only makes incremental updates, so if you're starting from a server that already has existing data, you should do the first manual initial backups before enabling the rolling mode.

It's generally safe to run a manual sync_client side-by-side with a rolling one (they will lock things properly and keep out of each other's way).  But the exception to this is that a rolling mode replication can't coherently update an uninitialised replica (since it only sends new changes recorded in the sync log, but not the pre-existing data).  You need to do a full manual replication first, to ensure both sides are the same, before rolling mode can work sensibly.  This is the same regardless of whether your replica is a normal replica or a backupd one.

As you saw, replicating again manually will probably fix you up, at least. :)

> http://www.cyrusimap.org/imap/reference/admin/backups.html says "Update 
> cyrus.conf(5) to add a sync_client(8) invocation to the SERVICES section 
> specifying (at least) the -r and -n channel options.". Really maybe you 
> intend START. In SERVICE I need to specify a listen too, this is not 
> very clear to me.

This is wrong.  It should be the DAEMON section, not the SERVICES section.  I'm fixing this now, so the website should be updated shortly.

You can also run it from the START section, but note that if the sync_client exits early for some reason, master will not restart it.  If you run it from the START section, you will need to monitor/restart it yourself.

> Around of backupd I would ask two other questions:
> 
> 1) The backupd host always claims this:
> 
> backupd[1309970]: DBERROR: reading /var/lib/imap/db/skipstamp, assuming 
> the worst: No such file or directory
> 
> Could you tell me how to fix that DBERROR too?

I think you might need to add the usual recover entry to the START section:
    
recover       cmd="ctl_cyrusdb -r"

I notice this is missing from the backups documentation -- please let me know if it this sorts it out, and I'll fix it up!
 
> 2) Is there a way to delete a user from backupd host?

That's an interesting question...

Under normal use, if a user is deleted from their imap server, then, after the deletion has replicated to the backup server, and after the "backup_retention" period has elapsed, and after a subsequent "ctl_backups compact ..." has been run, then the backup file should be approximately empty.  But I think it will still exist; nothing actually deletes it yet.

To get rid of it manually, you can use "ctl_backups list ..." to find the filename where it exists on disk.  You can also find it by using cyr_dbtool to look up the user's key in backups.db.  You will then need to:

a) use cyr_dbtool to delete that user's key from backups.db
b) delete the named file (it will be like "/some/path/userid_XXXXXX")
c) also delete the corresponding "/some/path/userid_XXXXXX.index" file

Note that if the user still really exists, and a rolling sync_client replicates them to the same backup server again, then the backup file will be recreated (with a new XXXXXX) -- with the same problems as above wrt no-initial-sync.

Cheers,

ellie

From falon at ruparpiemonte.it  Fri Jun 19 04:44:56 2020
From: falon at ruparpiemonte.it (Marco)
Date: Fri, 19 Jun 2020 10:44:56 +0200
Subject: backupd and sync_client IOERROR
In-Reply-To: <ba49a89c-04de-4960-b280-b4136715d640@www.fastmail.com>
References: <7d2c31db-a3b3-fca3-9e25-574c373e7f23@ruparpiemonte.it>
 <ba49a89c-04de-4960-b280-b4136715d640@www.fastmail.com>
Message-ID: <a53e5a30-3264-b696-23ff-1c141ad4a900@ruparpiemonte.it>

Hi Ellie,

Il 19/06/2020 03:01, ellie timoney ha scritto:
> I think you might need to add the usual recover entry to the START section:
>      
> recover       cmd="ctl_cyrusdb -r"
> 
> I notice this is missing from the backups documentation -- please let me know if it this sorts it out, and I'll fix it up!

wow, yes, it works. With this config, after the Cyrus restart the 
mailboxes.db and the skipstamps dbs are created and the error disappears 
from syslog.

>> 2) Is there a way to delete a user from backupd host?
> That's an interesting question...
> 
> Under normal use, if a user is deleted from their imap server, then, after the deletion has replicated to the backup server, and after the "backup_retention" period has elapsed, and after a subsequent "ctl_backups compact ..." has been run, then the backup file should be approximately empty.  But I think it will still exist; nothing actually deletes it yet.
> 
> To get rid of it manually, you can use "ctl_backups list ..." to find the filename where it exists on disk.  You can also find it by using cyr_dbtool to look up the user's key in backups.db.  You will then need to:
> 
> a) use cyr_dbtool to delete that user's key from backups.db
> b) delete the named file (it will be like "/some/path/userid_XXXXXX")
> c) also delete the corresponding "/some/path/userid_XXXXXX.index" file
> 
> Note that if the user still really exists, and a rolling sync_client replicates them to the same backup server again, then the backup file will be recreated (with a new XXXXXX) -- with the same problems as above wrt no-initial-sync.

It works!
Making some tests and the initial sync, it's easy with sync_client 
insert a wrong non-existent userid (I added "/user" before the userid). 
And that wrong userid is immediately created on the backupd server :)
With the above procedure I can delete the wrong entry.


Definitively I think that this backup feature is a very good idea.

I suggested the possibility to choose the recovery date (ie: all 
messages and mailboxes deleted or expunged from <dateA> to <dateB>). I 
hope you could add this feature.

Thank you very very much for these detailed answers!!


I now tried a restore, but I still have some problems. I very appreciate 
if you could read the following issue.

My goal is to recover a message that was removed (expunged) from the 
IMAP server, ignoring already existing messages.

It seems the "-x" flag could help. It seems it allows to restore and 
unexpunge a previously expunged messages in the IMAP server.

Yesterday I expunged the message 16ceead9802286784d7a54c5bc782891f76f2f2e:

2020-06-18T15:44:37.374556+02:00 tst-msg03 cyrus/imap[9461]: auditlog: 
expunge sessionid=<cyrus-1592487850-9461-1-3828781086337945617> 
mailbox=<example.com!user.utente^archivista> 
uniqueid=<nvf96uirjgfs6xaekwwc7g44> uid=<32092> modseq=<46828> 
sysflags=<DE|SE|AR|EX> guid=<16ceead9802286784d7a54c5bc782891f76f2f2e>


I tried today to restore it:
cyr_restore -v -U -x tst-msg03.example.com -u 
utente.archivista at example.com 16ceead9802286784d7a54c5bc782891f76f2f2e
example.com!user.utente^archivista: trying to keep 
uidvalidity(1550228105), uniqueid(nvf96uirjgfs6xaekwwc7g44), 
highestmodseq(46828)

and I see:
2020-06-19T09:29:24.030624+02:00 tst-msg03 cyrus/imap[32742]: login: 
tst-msg03-bck.example.com [10.102.42.169] cyr_restore LOGIN User logged 
in SESSIONID=<cyrus-1592551763-32742-1-6306147161275335691>

2020-06-19T09:29:24.162909+02:00 tst-msg03 cyrus/imap[32742]: auditlog: 
append sessionid=<cyrus-1592551763-32742-1-6306147161275335691> 
mailbox=<example.com!user.utente^archivista> 
uniqueid=<nvf96uirjgfs6xaekwwc7g44> uid=<32096> modseq=<46829> 
sysflags=<DE|SE|AR|EX> guid=<16ceead9802286784d7a54c5bc782891f76f2f2e> 
messageid=<ab818c62-5f3a-089b-89d5-55059fd7265c at example.com>

2020-06-19T09:29:24.163058+02:00 tst-msg03 cyrus/imap[32742]: auditlog: 
expunge sessionid=<cyrus-1592551763-32742-1-6306147161275335691> 
mailbox=<example.com!user.utente^archivista> 
uniqueid=<nvf96uirjgfs6xaekwwc7g44> uid=<32096> modseq=<46829> 
sysflags=<DE|SE|AR|EX> guid=<16ceead9802286784d7a54c5bc782891f76f2f2e>

2020-06-19T09:29:24.167561+02:00 tst-msg03 cyrus/imap[32742]: auditlog: 
modseq sessionid=<cyrus-1592551763-32742-1-6306147161275335691> 
mailbox=<example.com!user.utente^archivista> 
uniqueid=<nvf96uirjgfs6xaekwwc7g44> highestmodseq=<46829>

2020-06-19T09:29:24.215036+02:00 tst-msg03 cyrus/imap[32742]: USAGE 
cyr_restore user: 0.042492 sys: 0.015435

2020-06-19T09:29:24.215191+02:00 tst-msg03 cyrus/imap[32742]: auditlog: 
traffic sessionid=<cyrus-1592551763-32742-1-6306147161275335691> 
bytes_in=<2412> bytes_out=<1039>

2020-06-19T09:29:32.855379+02:00 tst-msg03 cyrus/sync_client[32566]: 
IOERROR: zero length response to MAILBOXES (end of file reached)

2020-06-19T09:29:32.855682+02:00 tst-msg03 cyrus/sync_client[32566]: 
IOERROR: zero length response to RESTART (end of file reached)

2020-06-19T09:29:32.855713+02:00 tst-msg03 cyrus/sync_client[32566]: 
Error in do_sync(): bailing out! Bad protocol

2020-06-19T09:29:32.855738+02:00 tst-msg03 cyrus/sync_client[32566]: 
Processing sync log file /var/lib/imap/sync/bck/log-run failed: Bad protocol

I tried again and I see the same result, but without the IOERROR:

2020-06-19T09:36:51.153939+02:00 tst-msg03 cyrus/imap[32762]: login: 
tst-msg03-bck.example.com [10.102.42.169] cyr_restore LOGIN User logged 
in SESSIONID=<cyrus-1592552211-32762-1-17619865152399331525>

2020-06-19T09:36:51.199742+02:00 tst-msg03 cyrus/imap[32762]: 
example.com!user.utente^archivista: same message appears twice 32096 32097

(why twice? It expunges 32096 just above)

2020-06-19T09:36:51.225525+02:00 tst-msg03 cyrus/imap[32762]: auditlog: 
append sessionid=<cyrus-1592552211-32762-1-17619865152399331525> 
mailbox=<example.com!user.utente^archivista> 
uniqueid=<nvf96uirjgfs6xaekwwc7g44> uid=<32097> modseq=<46834> 
sysflags=<DE|SE|AR|EX> guid=<16ceead9802286784d7a54c5bc782891f76f2f2e> 
messageid=<ab818c62-5f3a-089b-89d5-55059fd7265c at example.com>

2020-06-19T09:36:51.225714+02:00 tst-msg03 cyrus/imap[32762]: auditlog: 
expunge sessionid=<cyrus-1592552211-32762-1-17619865152399331525> 
mailbox=<example.com!user.utente^archivista> 
uniqueid=<nvf96uirjgfs6xaekwwc7g44> uid=<32097> modseq=<46834> 
sysflags=<DE|SE|AR|EX> guid=<16ceead9802286784d7a54c5bc782891f76f2f2e>

So the message is restored and immediately expunged again.
Finally I expunged a new message and after few seconds I tried to 
recover it:

cyr_restore -v -U -x tst-msg03.example.com -u 
utente.archivista at example.com 3cfd9e7dff35bcb38667ed5896db3cb82b05fd94
example.com!user.utente^archivista: trying to keep 
uidvalidity(1550228105), uniqueid(nvf96uirjgfs6xaekwwc7g44), 
highestmodseq(46828)
example.com!user.utente^archivista: excluding unexpunged message: 
3cfd9e7dff35bcb38667ed5896db3cb82b05fd94

A different behavior occurs. But the message is expunged, is not unexpunged:

# grep 3cfd9e7dff35bcb38667ed5896db3cb82b05fd94 cyruslog | tail
2020-06-19T09:35:11.349226+02:00 tst-msg03 cyrus/imap[32571]: auditlog: 
expunge sessionid=<cyrus-1592551351-32571-1-6833532399537986033> 
mailbox=<example.com!user.utente^archivista> 
uniqueid=<nvf96uirjgfs6xaekwwc7g44> uid=<32087> modseq=<46833> 
sysflags=<AN|DE|SE|AR|EX> guid=<3cfd9e7dff35bcb38667ed5896db3cb82b05fd94>


Uhm... removing the "-x" I finally obtain the result of restoring 
expunged messages.
The same message still remains in the unexpunged list of the IMAP 
server. If we use delayed expunge on the IMAP server and backupd some 
confusion can occurs.

I'm confused about "-x". If a message is expunged in the IMAP server, 
then isn't it flagged as expunged in the Backups server too?

I checked the backup, is ok:
$ ctl_backups verify -u utente.archivista at example.com
verify utente.archivista at example.com: ok


Thank you very much
Cheers
marco

From pongracz.istvan at gmail.com  Sat Jun 20 15:04:52 2020
From: pongracz.istvan at gmail.com (=?ISO-8859-1?Q?Pongr=E1cz_Istv=E1n?=)
Date: Sat, 20 Jun 2020 21:04:52 +0200
Subject: cyrus-imapd exporting databases failed on shutdown -
 deliver.db.skiplist 2048M
Message-ID: <f9f332dbc90170d193738331c5d2d7cd6fa27721.camel@gmail.com>

Hi,

I run into a problem on an old clearos server, where the cyrus shutdown
always failed at step exporting databases.
As I checked the situation using ps ax on an other console, I found
that, it was exporting deliver.db.skiplist file, which failed after a
loooong time (some minutes).
I checked that file on the filesystem, I saw the file size is 2048MB,
which seems a limit for me and I suspect the problem should be that,
the 32 bit cyrus cannot write more data to that file and caused the
problem.
As I read the db_export.log, that confirmed my theory, file size limit
exceeded.

This kind of issue causes long hangup at system shutdown and I guess
causes other potential problems, as this file could be truncated and
useless.

My question, do you have any idea/hint, what to do to get it work?
For example, changing export.db format to something else? (How?)
Thank you!

I copied some important information below, please check it.

db_export.log:
cvt_cyrusdb_all version: 1.2.1db_checkpoint: open: No such file or
directory/usr/lib/cyrus-imapd/cvt_cyrusdb_all: line 199:  9150 File
size limit exceeded$cvt_cyrusdb "$target" "$old_db"
"${target}.skiplist" skiplistERROR: unable to convert
/var/lib/imap/deliver.db from berkeley to skiplistremoved
`/var/lib/imap/db/log.0000000001'removed
`/var/lib/imap/db/__db.001'removed `/var/lib/imap/db/__db.002'removed
`/var/lib/imap/db/__db.003'removed `/var/lib/imap/db/__db.004'removed
`/var/lib/imap/db/__db.005'removed `/var/lib/imap/db/__db.006'removed
`/var/lib/imap/db/__db.007'removed `/var/lib/imap/db/__db.008'removed
`/var/lib/imap/db/__db.009'removed `/var/lib/imap/db/__db.010'removed
`/var/lib/imap/db/__db.011'removed `/var/lib/imap/db/__db.012'

The system details:
 * Clearos 5.2 32 bit
 * Cyrus is 2.3.11
 * kernel 2.6.18-308.1.1.v5
 * filesystem is ext3 (no problem to write 200-300GB into a single
file)
 * size of email folder (/var/spool/imap/*): 1007 GB
 * size of /var/lib/imap/deliver.db 48MB

Other interesting config files:


db.cfg.cache content
annotation_db=skiplistduplicate_db=berkeleymboxkey_db=skiplistmboxlist_
db=skiplistptscache_db=berkeleyquota_db=quotalegacyseenstate_db=skiplis
tsieve_version=2.2.3subscription_db=flattlscache_db=berkeley


/var/lib/imap/db/DB_CONFIG
set_cachesize 0 8388608 8set_lg_regionmax 5242880set_lg_bsize 8097152



Thank you!

Istv?n (Steve)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200620/6ffd1a40/attachment.html>

From simon.matter at invoca.ch  Sat Jun 20 15:31:38 2020
From: simon.matter at invoca.ch (Simon Matter)
Date: Sat, 20 Jun 2020 21:31:38 +0200
Subject: cyrus-imapd exporting databases failed on shutdown -
 deliver.db.skiplist 2048M
In-Reply-To: <f9f332dbc90170d193738331c5d2d7cd6fa27721.camel@gmail.com>
References: <f9f332dbc90170d193738331c5d2d7cd6fa27721.camel@gmail.com>
Message-ID: <a90cde02343de1996682dec7f8640d7c.squirrel@webmail.bi.corp.invoca.ch>

> Hi,
>
> I run into a problem on an old clearos server, where the cyrus shutdown
> always failed at step exporting databases.
> As I checked the situation using ps ax on an other console, I found
> that, it was exporting deliver.db.skiplist file, which failed after a
> loooong time (some minutes).
> I checked that file on the filesystem, I saw the file size is 2048MB,
> which seems a limit for me and I suspect the problem should be that,
> the 32 bit cyrus cannot write more data to that file and caused the
> problem.
> As I read the db_export.log, that confirmed my theory, file size limit
> exceeded.

Hi,

The question is why is the deliver db > 2GB in skiplist format? Is it
normal or do you have a corrupt BDB db or does your db pruning not work
for deliverdb. I think that should be something like 'delprune     
cmd="cyr_expire -D 7 -E 3 -X 7" at=0400' in cyrus.conf.

I think the easiest way would be to make sure you have pruning configured
correctly, then change config of deliver db to skiplist, and start without
a db so a new, empty deliver db is created.

Then have an eye on the db file to see if it grows again to almost 2GB. If
it doesn't grow so much, you should be fine.

Regards,
Simon


From pongracz.istvan at gmail.com  Sat Jun 20 16:25:37 2020
From: pongracz.istvan at gmail.com (=?ISO-8859-1?Q?Pongr=E1cz_Istv=E1n?=)
Date: Sat, 20 Jun 2020 22:25:37 +0200
Subject: cyrus-imapd exporting databases failed on shutdown -
 deliver.db.skiplist 2048M
In-Reply-To: <a90cde02343de1996682dec7f8640d7c.squirrel@webmail.bi.corp.invoca.ch>
References: <f9f332dbc90170d193738331c5d2d7cd6fa27721.camel@gmail.com>
 <a90cde02343de1996682dec7f8640d7c.squirrel@webmail.bi.corp.invoca.ch>
Message-ID: <b5bebdb5d234cc16e3fda7a0d757823e4cc8fea9.camel@gmail.com>

2020. 06.  20, szombat keltez?ssel 21.31-kor Simon Matter ezt ?rta:
> 
> Hi,
> 
> The question is why is the deliver db > 2GB in skiplist format? Is it
> normal or do you have a corrupt BDB db or does your db pruning not work
> for deliverdb. I think that should be something like 'delprune     
> cmd="cyr_expire -D 7 -E 3 -X 7" at=0400' in cyrus.conf.
> 
> I think the easiest way would be to make sure you have pruning configured
> correctly, then change config of deliver db to skiplist, and start without
> a db so a new, empty deliver db is created.
> 
> Then have an eye on the db file to see if it grows again to almost 2GB. If
> it doesn't grow so much, you should be fine.
> 
> Regards,
> Simon
> 
> 

Hi,
Thank you for your answer!
I have this in my config at EVENTS section:

EVENTS {
  # this is required
  checkpoint    cmd="ctl_cyrusdb -c" period=30

  # this is only necessary if using duplicate delivery suppression
  delprune      cmd="ctl_deliver -E 3" period=1440

  # this is only necessary if caching TLS sessions
  tlsprune      cmd="tls_prune" period=1440
}

Regarding to users, we do not see any anomalies on daily use or restarting the server sometimes, only this exporting failed.

I issued the mentioned command on command line (cyr_expire -D 7 -E 3 -X 7) it finished in around 1-2 seconds. (At this moment there are no online users as this is weekend).

Additional information I forgot to mention:
there are a lot of shared email folder of the catchall account to share common mailboxes.

I got this error after some lines, when I run /usr/lib/cyrus-imapd/ctl_deliver -d

fatal error: Internal error: assertion failed: duplicate.c: 344: (datalen == sizeof(time_t)) || (datalen == sizeof(time_t) + sizeof(unsigned long))

Probably deliver.db itself is corrupted?

Cheers,
Istv?n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200620/3bd1fad6/attachment.html>

From pongracz.istvan at gmail.com  Sat Jun 20 16:37:53 2020
From: pongracz.istvan at gmail.com (=?ISO-8859-1?Q?Pongr=E1cz_Istv=E1n?=)
Date: Sat, 20 Jun 2020 22:37:53 +0200
Subject: cyrus-imapd exporting databases failed on shutdown -
 deliver.db.skiplist 2048M
In-Reply-To: <a90cde02343de1996682dec7f8640d7c.squirrel@webmail.bi.corp.invoca.ch>
References: <f9f332dbc90170d193738331c5d2d7cd6fa27721.camel@gmail.com>
 <a90cde02343de1996682dec7f8640d7c.squirrel@webmail.bi.corp.invoca.ch>
Message-ID: <5b589849fd0eab9073395e8428fcd53f128167bd.camel@gmail.com>

2020. 06.  20, szombat keltez?ssel 21.31-kor Simon Matter ezt ?rta:
> Hi,
> 
> The question is why is the deliver db > 2GB in skiplist format? Is it
> normal or do you have a corrupt BDB db or does your db pruning not work
> for deliverdb. I think that should be something like 'delprune     
> cmd="cyr_expire -D 7 -E 3 -X 7" at=0400' in cyrus.conf.
> 
> I think the easiest way would be to make sure you have pruning configured
> correctly, then change config of deliver db to skiplist, and start without
> a db so a new, empty deliver db is created.
> 
> Then have an eye on the db file to see if it grows again to almost 2GB. If
> it doesn't grow so much, you should be fine.
> 
> Regards,
> Simon

Hi,

Something definitely not seems fine:

-bash-3.2$ /usr/lib/cyrus-imapd/cyr_expire -E 3 -D 7 -X 7 -v

expunged 0 out of 0 messages from 0 mailboxes

The deliver.db still about 48MB.

Tomorrow I will continue.

Thanks,
Istv?n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200620/b28597ff/attachment-0001.html>

From simon.matter at invoca.ch  Sun Jun 21 04:53:26 2020
From: simon.matter at invoca.ch (Simon Matter)
Date: Sun, 21 Jun 2020 10:53:26 +0200
Subject: cyrus-imapd exporting databases failed on shutdown -
 deliver.db.skiplist 2048M
In-Reply-To: <5b589849fd0eab9073395e8428fcd53f128167bd.camel@gmail.com>
References: <f9f332dbc90170d193738331c5d2d7cd6fa27721.camel@gmail.com>
 <a90cde02343de1996682dec7f8640d7c.squirrel@webmail.bi.corp.invoca.ch>
 <5b589849fd0eab9073395e8428fcd53f128167bd.camel@gmail.com>
Message-ID: <aab34bd2720a05fcdd4291f72f11a34a.squirrel@webmail.bi.corp.invoca.ch>

> 2020. 06.  20, szombat keltez?ssel 21.31-kor Simon Matter ezt ?rta:
>> Hi,
>>
>> The question is why is the deliver db > 2GB in skiplist format? Is it
>> normal or do you have a corrupt BDB db or does your db pruning not work
>> for deliverdb. I think that should be something like 'delprune
>> cmd="cyr_expire -D 7 -E 3 -X 7" at=0400' in cyrus.conf.
>>
>> I think the easiest way would be to make sure you have pruning
>> configured
>> correctly, then change config of deliver db to skiplist, and start
>> without
>> a db so a new, empty deliver db is created.
>>
>> Then have an eye on the db file to see if it grows again to almost 2GB.
>> If
>> it doesn't grow so much, you should be fine.
>>
>> Regards,
>> Simon
>
> Hi,
>
> Something definitely not seems fine:
>
> -bash-3.2$ /usr/lib/cyrus-imapd/cyr_expire -E 3 -D 7 -X 7 -v

Please make sure the options here are also valid for your cyrus version.
However, I also guess your deliver.db is corrupted somehow. From my own
experience skiplist dbs are easier to handle than bdb and using skiplist
only has not shown any issues.

Regards,
Simon

>
> expunged 0 out of 0 messages from 0 mailboxes
>
> The deliver.db still about 48MB.
>
> Tomorrow I will continue.
>
> Thanks,
> Istv?n
>



From pongracz.istvan at gmail.com  Sun Jun 21 05:01:13 2020
From: pongracz.istvan at gmail.com (=?ISO-8859-1?Q?Pongr=E1cz_Istv=E1n?=)
Date: Sun, 21 Jun 2020 11:01:13 +0200
Subject: cyrus-imapd exporting databases failed on shutdown -
 deliver.db.skiplist 2048M
In-Reply-To: <aab34bd2720a05fcdd4291f72f11a34a.squirrel@webmail.bi.corp.invoca.ch>
References: <f9f332dbc90170d193738331c5d2d7cd6fa27721.camel@gmail.com>
 <a90cde02343de1996682dec7f8640d7c.squirrel@webmail.bi.corp.invoca.ch>
 <5b589849fd0eab9073395e8428fcd53f128167bd.camel@gmail.com>
 <aab34bd2720a05fcdd4291f72f11a34a.squirrel@webmail.bi.corp.invoca.ch>
Message-ID: <745f4da3d63dd8fa70ee4655217a453f5faa0d9a.camel@gmail.com>



2020. 06.  21, vas?rnap keltez?ssel 10.53-kor Simon Matter ezt ?rta:
> 2020. 06.  20, szombat keltez?ssel 21.31-kor Simon Matter ezt ?rta:
> Please make sure the options here are also valid for your cyrus version.
> However, I also guess your deliver.db is corrupted somehow. From my own
> experience skiplist dbs are easier to handle than bdb and using skiplist
> only has not shown any issues.
> 
> Regards,
> Simon

Hi,

Thank you the hint Simon.
They are valid.

What do you think, should I just delete deliver.db after I stopped cyrus-imap and restart again?
What can I loose?

Cheers,
Istv?n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200621/3990b001/attachment.html>

From pongracz.istvan at gmail.com  Sun Jun 21 05:45:21 2020
From: pongracz.istvan at gmail.com (=?ISO-8859-1?Q?Pongr=E1cz_Istv=E1n?=)
Date: Sun, 21 Jun 2020 11:45:21 +0200
Subject: SOLVED Re: cyrus-imapd exporting databases failed on shutdown -
 deliver.db.skiplist 2048M
In-Reply-To: <aab34bd2720a05fcdd4291f72f11a34a.squirrel@webmail.bi.corp.invoca.ch>
References: <f9f332dbc90170d193738331c5d2d7cd6fa27721.camel@gmail.com>
 <a90cde02343de1996682dec7f8640d7c.squirrel@webmail.bi.corp.invoca.ch>
 <5b589849fd0eab9073395e8428fcd53f128167bd.camel@gmail.com>
 <aab34bd2720a05fcdd4291f72f11a34a.squirrel@webmail.bi.corp.invoca.ch>
Message-ID: <0bb121a7ff1cdf327fe617ca369d91404b9d8aac.camel@gmail.com>

2020. 06.  21, vas?rnap keltez?ssel 10.53-kor Simon Matter ezt ?rta:
> 2020. 06.  20, szombat keltez?ssel 21.31-kor Simon Matter ezt ?rta:
> 
> The deliver.db still about 48MB.
> 

I just deleted deliver.db and restart cyrus.
Restarting was pretty quick, takes only some seconds.
New deliver.db created, its size is 8KByte.

So far, so good.

Thank you!

Istv?n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200621/478d0f38/attachment.html>

From ellie at fastmail.com  Sun Jun 21 21:29:18 2020
From: ellie at fastmail.com (ellie timoney)
Date: Mon, 22 Jun 2020 11:29:18 +1000
Subject: backupd and sync_client IOERROR
In-Reply-To: <a53e5a30-3264-b696-23ff-1c141ad4a900@ruparpiemonte.it>
References: <7d2c31db-a3b3-fca3-9e25-574c373e7f23@ruparpiemonte.it>
 <ba49a89c-04de-4960-b280-b4136715d640@www.fastmail.com>
 <a53e5a30-3264-b696-23ff-1c141ad4a900@ruparpiemonte.it>
Message-ID: <d20e84d8-5764-4beb-b3b0-ed63fbb408f2@www.fastmail.com>

Hi Marco,

On Fri, Jun 19, 2020, at 6:44 PM, Marco wrote:
> wow, yes, it works. With this config, after the Cyrus restart the 
> mailboxes.db and the skipstamps dbs are created and the error disappears 
> from syslog.

Great! I've updated the documentation, and the website should update shortly.

> I suggested the possibility to choose the recovery date (ie: all 
> messages and mailboxes deleted or expunged from <dateA> to <dateB>). I 
> hope you could add this feature.

Yep, I saw that, thanks for the request :)

> I now tried a restore, but I still have some problems. I very appreciate 
> if you could read the following issue.
> 
> My goal is to recover a message that was removed (expunged) from the 
> IMAP server, ignoring already existing messages.
> 
> It seems the "-x" flag could help. It seems it allows to restore and 
> unexpunge a previously expunged messages in the IMAP server.

No, you've misunderstood.  The "-x" flag is to ONLY restore expunged messages.  If you've specified a list of mailboxes or messages on the command line, and some of them are not expunged, the "-x" option will cause it to ignore the ones that are not expunged, and only restore the expunged ones.  The "-X" option does the inverse -- they're filters.  Without one of these filters, it will restore everything you asked for, regardless of its expunged status.

So like, maybe a user has deleted some stuff, and you don't want to mess around figuring out which individual messages they need restored, so you just want to restore everything, and let the user figure it out.  This is what -x is for -- so you can say "restore the contents of mailbox foo, but only the expunged stuff" or "restore every mailbox (-a), but only the expunged stuff".

Think about it: if the messages _weren't_ expunged, there would usually be no need to restore them from backup!  You would simply remove the \Deleted flag over IMAP.  So, you are almost always using restore to bring back expunged messages, thus there is no special flag needed to enable this functionality.

(But, if you had lost an entire server to some disaster, and restoring to its replacement, then you would need to also restore the unexpunged stuff.)
 
> Yesterday I expunged the message 16ceead9802286784d7a54c5bc782891f76f2f2e:
> 
> 2020-06-18T15:44:37.374556+02:00 tst-msg03 cyrus/imap[9461]: auditlog: 
> expunge sessionid=<cyrus-1592487850-9461-1-3828781086337945617> 
> mailbox=<example.com!user.utente^archivista> 
> uniqueid=<nvf96uirjgfs6xaekwwc7g44> uid=<32092> modseq=<46828> 
> sysflags=<DE|SE|AR|EX> guid=<16ceead9802286784d7a54c5bc782891f76f2f2e>
> 
> 
> I tried today to restore it:
> cyr_restore -v -U -x tst-msg03.example.com -u 
> utente.archivista at example.com 16ceead9802286784d7a54c5bc782891f76f2f2e
> example.com!user.utente^archivista: trying to keep 
> uidvalidity(1550228105), uniqueid(nvf96uirjgfs6xaekwwc7g44), 
> highestmodseq(46828)
> 
> and I see:
> 2020-06-19T09:29:24.030624+02:00 tst-msg03 cyrus/imap[32742]: login: 
> tst-msg03-bck.example.com [10.102.42.169] cyr_restore LOGIN User logged 
> in SESSIONID=<cyrus-1592551763-32742-1-6306147161275335691>
> 
> 2020-06-19T09:29:24.162909+02:00 tst-msg03 cyrus/imap[32742]: auditlog: 
> append sessionid=<cyrus-1592551763-32742-1-6306147161275335691> 
> mailbox=<example.com!user.utente^archivista> 
> uniqueid=<nvf96uirjgfs6xaekwwc7g44> uid=<32096> modseq=<46829> 
> sysflags=<DE|SE|AR|EX> guid=<16ceead9802286784d7a54c5bc782891f76f2f2e> 
> messageid=<ab818c62-5f3a-089b-89d5-55059fd7265c at example.com>
> 
> 2020-06-19T09:29:24.163058+02:00 tst-msg03 cyrus/imap[32742]: auditlog: 
> expunge sessionid=<cyrus-1592551763-32742-1-6306147161275335691> 
> mailbox=<example.com!user.utente^archivista> 
> uniqueid=<nvf96uirjgfs6xaekwwc7g44> uid=<32096> modseq=<46829> 
> sysflags=<DE|SE|AR|EX> guid=<16ceead9802286784d7a54c5bc782891f76f2f2e>
> 
> 2020-06-19T09:29:24.167561+02:00 tst-msg03 cyrus/imap[32742]: auditlog: 
> modseq sessionid=<cyrus-1592551763-32742-1-6306147161275335691> 
> mailbox=<example.com!user.utente^archivista> 
> uniqueid=<nvf96uirjgfs6xaekwwc7g44> highestmodseq=<46829>
> 
> 2020-06-19T09:29:24.215036+02:00 tst-msg03 cyrus/imap[32742]: USAGE 
> cyr_restore user: 0.042492 sys: 0.015435
> 
> 2020-06-19T09:29:24.215191+02:00 tst-msg03 cyrus/imap[32742]: auditlog: 
> traffic sessionid=<cyrus-1592551763-32742-1-6306147161275335691> 
> bytes_in=<2412> bytes_out=<1039>
> 
> 2020-06-19T09:29:32.855379+02:00 tst-msg03 cyrus/sync_client[32566]: 
> IOERROR: zero length response to MAILBOXES (end of file reached)
> 
> 2020-06-19T09:29:32.855682+02:00 tst-msg03 cyrus/sync_client[32566]: 
> IOERROR: zero length response to RESTART (end of file reached)
> 
> 2020-06-19T09:29:32.855713+02:00 tst-msg03 cyrus/sync_client[32566]: 
> Error in do_sync(): bailing out! Bad protocol
> 
> 2020-06-19T09:29:32.855738+02:00 tst-msg03 cyrus/sync_client[32566]: 
> Processing sync log file /var/lib/imap/sync/bck/log-run failed: Bad protocol
> 
> I tried again and I see the same result, but without the IOERROR:
> 
> 2020-06-19T09:36:51.153939+02:00 tst-msg03 cyrus/imap[32762]: login: 
> tst-msg03-bck.example.com [10.102.42.169] cyr_restore LOGIN User logged 
> in SESSIONID=<cyrus-1592552211-32762-1-17619865152399331525>
> 
> 2020-06-19T09:36:51.199742+02:00 tst-msg03 cyrus/imap[32762]: 
> example.com!user.utente^archivista: same message appears twice 32096 32097
> 
> (why twice? It expunges 32096 just above)
> 
> 2020-06-19T09:36:51.225525+02:00 tst-msg03 cyrus/imap[32762]: auditlog: 
> append sessionid=<cyrus-1592552211-32762-1-17619865152399331525> 
> mailbox=<example.com!user.utente^archivista> 
> uniqueid=<nvf96uirjgfs6xaekwwc7g44> uid=<32097> modseq=<46834> 
> sysflags=<DE|SE|AR|EX> guid=<16ceead9802286784d7a54c5bc782891f76f2f2e> 
> messageid=<ab818c62-5f3a-089b-89d5-55059fd7265c at example.com>
> 
> 2020-06-19T09:36:51.225714+02:00 tst-msg03 cyrus/imap[32762]: auditlog: 
> expunge sessionid=<cyrus-1592552211-32762-1-17619865152399331525> 
> mailbox=<example.com!user.utente^archivista> 
> uniqueid=<nvf96uirjgfs6xaekwwc7g44> uid=<32097> modseq=<46834> 
> sysflags=<DE|SE|AR|EX> guid=<16ceead9802286784d7a54c5bc782891f76f2f2e>
> 
> So the message is restored and immediately expunged again.

I think this is happening because you're specifying the "-U" option, which tries very hard to put the message back into the same state as it was last in.  Its last state was having been expunged, and therefore the restored copy needs to be expunged...

The point of -U is so that you can recover from data loss, while preserving the IMAP state so the client doesn't need to completely resync.  For the "user accidentally deleted and expunged, and now wants it back" use case, it's not useful, because in that case you explicitly want the message back with a new state (i.e. not expunged).

> Finally I expunged a new message and after few seconds I tried to 
> recover it:
> 
> cyr_restore -v -U -x tst-msg03.example.com -u 
> utente.archivista at example.com 3cfd9e7dff35bcb38667ed5896db3cb82b05fd94
> example.com!user.utente^archivista: trying to keep 
> uidvalidity(1550228105), uniqueid(nvf96uirjgfs6xaekwwc7g44), 
> highestmodseq(46828)
> example.com!user.utente^archivista: excluding unexpunged message: 
> 3cfd9e7dff35bcb38667ed5896db3cb82b05fd94
> 
> A different behavior occurs. But the message is expunged, is not unexpunged:
> 
> # grep 3cfd9e7dff35bcb38667ed5896db3cb82b05fd94 cyruslog | tail
> 2020-06-19T09:35:11.349226+02:00 tst-msg03 cyrus/imap[32571]: auditlog: 
> expunge sessionid=<cyrus-1592551351-32571-1-6833532399537986033> 
> mailbox=<example.com!user.utente^archivista> 
> uniqueid=<nvf96uirjgfs6xaekwwc7g44> uid=<32087> modseq=<46833> 
> sysflags=<AN|DE|SE|AR|EX> guid=<3cfd9e7dff35bcb38667ed5896db3cb82b05fd94>
> 
> Uhm... removing the "-x" I finally obtain the result of restoring 
> expunged messages.

I'm not really sure why this would make a difference.  Maybe in this case the expunge hadn't replicated to the backup server yet?

> The same message still remains in the unexpunged list of the IMAP 
> server. If we use delayed expunge on the IMAP server and backupd some 
> confusion can occurs.
> 
> I'm confused about "-x". If a message is expunged in the IMAP server, 
> then isn't it flagged as expunged in the Backups server too?

That depends on whether the expunge has been replicated yet or not.  But generally, yes.

I think most of the confusion here is because you're using "-U", and therefore restoring an expunged message as an expunged message.

Cheers,

ellie

From ellie at fastmail.com  Mon Jun 22 00:35:30 2020
From: ellie at fastmail.com (ellie timoney)
Date: Mon, 22 Jun 2020 14:35:30 +1000
Subject: Cyrus IMAP 3.2.2 released
Message-ID: <36d4a79e-cde3-4d0f-8adb-9bd47cd16c7c@www.fastmail.com>

The Cyrus team is proud to announce the immediate availability of a new version of Cyrus IMAP: 3.2.2

Download URLs:

    https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.2/cyrus-imapd-3.2.2.tar.gz
    https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.2/cyrus-imapd-3.2.2.tar.gz.sig    

Please consult the release notes and upgrade documentation before upgrading to 3.2.2:

    https://www.cyrusimap.org/imap/download/release-notes/3.2/x/3.2.2.html
    https://www.cyrusimap.org/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report issues, join in the deliberations of new features for the next Cyrus IMAP release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

From falon at ruparpiemonte.it  Mon Jun 22 06:11:36 2020
From: falon at ruparpiemonte.it (Marco)
Date: Mon, 22 Jun 2020 12:11:36 +0200
Subject: backupd and sync_client IOERROR
In-Reply-To: <d20e84d8-5764-4beb-b3b0-ed63fbb408f2@www.fastmail.com>
References: <7d2c31db-a3b3-fca3-9e25-574c373e7f23@ruparpiemonte.it>
 <ba49a89c-04de-4960-b280-b4136715d640@www.fastmail.com>
 <a53e5a30-3264-b696-23ff-1c141ad4a900@ruparpiemonte.it>
 <d20e84d8-5764-4beb-b3b0-ed63fbb408f2@www.fastmail.com>
Message-ID: <ec0c4c88-32f7-e903-9b25-b2ded3f6bb88@ruparpiemonte.it>

Hello,

On 22/06/2020 03:29, ellie timoney has written:
[...]
> So like, maybe a user has deleted some stuff, and you don't want to mess around figuring out which individual messages they need restored, so you just want to restore everything, and let the user figure it out.  This is what -x is for -- so you can say "restore the contents of mailbox foo, but only the expunged stuff" or "restore every mailbox (-a), but only the expunged stuff".
[...]


  thank you for all these explanations, I think I understand now. My use 
case is "the user accidentally deleted and expunged and now he ask for 
recovery". I have to unexpunge all messages after I recovered it with 
"restore -x".

I think there isn't a all-in-one command for this use case: a user 
expunged some messages and deleted some folders somewhere. I want to 
recover all expunged messages and all the deleted folders which are no 
more present in the original IMAP server (because they were expired from 
cyr_expire).

I have to set "-x" to avoid duplication of messages.
With "-a -x" I recover all expunged messages and all deleted mailboxes. 
But messages inside deleted mailboxes are not marked as expunged, so 
these mailboxes are recovered empty in the IMAP server.

I have to examine the output of the restore command and repeat a restore 
without the "-x" on each DELETED recovered mailboxes at the previous step.


Another idea is to recover all in another empty IMAP server, without 
"-x" at all, and the user can look at the mailbox recovered there...

Thank you very much!!

Cheers
Marco

From zorg at probesys.com  Tue Jun 23 08:19:47 2020
From: zorg at probesys.com (Zorg)
Date: Tue, 23 Jun 2020 14:19:47 +0200
Subject: httpd server signature off
Message-ID: <bab3ff57-9a31-1008-ac0a-334601e139f1@probesys.com>

Hi

for security reason i want to get rid off

Cyrus-HTTP/3.0.6-Debian-3.0.6-6+deb1u1 Cyrus-SASL/2.1.23 OpenSSL/1.1 
Zlib/1.2.10 LibXML2.9.5 SQLite/3.21.1 LibiCal/3.0 ICU4C/63.1 
Jansson/2.12 Server at cyrus.domain.com Port 9443

like apache using

ServerTokens Prod or ServerSignature Off


Have I try serverinfo: off in imapd.conf but it don't work

What should i do

Cyril


From murch at fastmail.com  Tue Jun 23 08:44:45 2020
From: murch at fastmail.com (Ken Murchison)
Date: Tue, 23 Jun 2020 08:44:45 -0400
Subject: httpd server signature off
In-Reply-To: <bab3ff57-9a31-1008-ac0a-334601e139f1@probesys.com>
References: <bab3ff57-9a31-1008-ac0a-334601e139f1@probesys.com>
Message-ID: <0d0c1ea9-fa22-4364-b9cc-5c8c8906ea1c@fastmail.com>

Are you talking about removing this from the body of error responses?

Currently you can't, but I will patch master so that it obeys the 
serverinfo option.


On 6/23/20 8:19 AM, Zorg wrote:
> Hi
>
> for security reason i want to get rid off
>
> Cyrus-HTTP/3.0.6-Debian-3.0.6-6+deb1u1 Cyrus-SASL/2.1.23 OpenSSL/1.1 
> Zlib/1.2.10 LibXML2.9.5 SQLite/3.21.1 LibiCal/3.0 ICU4C/63.1 
> Jansson/2.12 Server at cyrus.domain.com Port 9443
>
> like apache using
>
> ServerTokens Prod or ServerSignature Off
>
>
> Have I try serverinfo: off in imapd.conf but it don't work
>
> What should i do
>
> Cyril
>
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


-- 
Kenneth Murchison
Cyrus Development Team
Fastmail US LLC


From Albert.Shih at obspm.fr  Tue Jun 23 09:41:59 2020
From: Albert.Shih at obspm.fr (Albert Shih)
Date: Tue, 23 Jun 2020 15:41:59 +0200
Subject: Maintenance of cyrus
Message-ID: <20200623134159.GA76707@io.chezmoi.fr>

Hi everyone.

Just like to know if some can tell me until when the cyrus 3.0.x will be
maintained ?

The mail are super top critical for me, so I don't like to change major
version during the life of the hardware.

I see they are still update on 2.x so...maybe I'm lucky and the 3.0.x will
ben maintained until end of time ;-)

Of course I can understand the cyrus team don't want to do something like
that, so no misunderstanding I don't want to blame anyone if you tell me
tomorrow it's the end of cyrus 3.0.x (just I rush to the supermarket to buy
a bottle of whisky if it is ;-) ).

Regards

--
Albert SHIH
Observatoire de Paris
xmpp: jas at obspm.fr
Heure local/Local time:
Tue 23 Jun 2020 03:37:50 PM CEST

From tim+kernel.org at coote.org  Tue Jun 23 09:42:03 2020
From: tim+kernel.org at coote.org (Tim Coote)
Date: Tue, 23 Jun 2020 14:42:03 +0100
Subject: reconstructing mailboxes from backup
Message-ID: <12F9FC39-70B9-41E0-8813-876A898E917C@coote.org>

Hullo

I have a cyrus implementation on Fedora for a small (~10) users that?s been migrated through many versions of the various components, including several different of IMAP clients.

Realising the fragility of the setup, I thought I?d restore from a backup. However, I?m finding that several of the mailboxes are not being recovered. I feel that I am missing somethign obvious, but I cannot spot it. 

The restoring version of cyrus-imap is: cyrus-imapd-3.0.13-2.fc32.x86_64,

The restored filesystem layout can be summarised thus:

`sudo find /var/spool/imap/g | grep cyrus.header`:

/var/spool/imap/g/user/george/Notes/cyrus.header
/var/spool/imap/g/user/george/cyrus.header
/var/spool/imap/g/user/george/Sent Messages/cyrus.header
/var/spool/imap/g/user/george/Deleted Messages/cyrus.header
/var/spool/imap/g/user/george/Sent/cyrus.header
/var/spool/imap/g/user/george/Trash/cyrus.header
/var/spool/imap/g/user/george/INBOX/Sent Messages/cyrus.header
/var/spool/imap/g/user/george/INBOX/Deleted Messages/cyrus.header
/var/spool/imap/g/user/george/INBOX/Drafts/cyrus.header
/var/spool/imap/g/user/george/INBOX^Deleted Messages/cyrus.header
/var/spool/imap/g/user/george/Drafts/cyrus.header

[so I would expect all of the subdirectories to be reconstructed as mailboxes]

however, using:
`sudo -u cyrus reconstruct -r -f user/george`

I only get:
user/george
user/george/Deleted Messages
user/george/Drafts
user/george/INBOX.Deleted Messages
user/george/Notes
user/george/Sent
user/george/Sent Messages
user/george/Trash

ie no subdirectories below the top level, but excluding those directories below INBOX.
Should there be a file: `/var/spool/imap/g/user/george/INBOX/cyrus.header`? 

Is there anything that I should be doing/how can I recover the other mailboxes?


From Albert.Shih at obspm.fr  Tue Jun 23 10:10:56 2020
From: Albert.Shih at obspm.fr (Albert Shih)
Date: Tue, 23 Jun 2020 16:10:56 +0200
Subject: What you do with old account
In-Reply-To: <b6b753ee-f16c-b86d-5241-38b6bb3b3a17@uni-koeln.de>
References: <20200609135142.GI4484@io.chezmoi.fr>
 <1591713123.5805.8.camel@whitemice.org>
 <b6b753ee-f16c-b86d-5241-38b6bb3b3a17@uni-koeln.de>
Message-ID: <20200623141056.GE76707@io.chezmoi.fr>

Le 14/06/2020 ? 21:51:49+0200, Sebastian Hagedorn a ?crit

Hi,


Thanks for your answer.

> Am 09.06.20 um 16:32 schrieb Adam Tauno Williams:
> > On Tue, 2020-06-09 at 15:51 +0200, Albert Shih wrote:>
> >> After switching to cyrus imap, I think about how to do that.
> >> If I'm correct I cannot just copy the file somewhere else, because cyrus
> >> database would keep the information about the existance of the mailbox, so
> >> what will the ?state of the art? way to remove a mail account and all the
> >> mail.
> >> And how what would be the ?state of the art? way to put it back ?
> > I create a calendar event [task] to delete the mailbox and otherwise
> > just leave it. If the account itself is disabled it cannot be accessed.
> >
> > Putting things back-into a mailstore is too much of a pain with current
> > storage prices.
>
> We have about 90,000 accounts, and our current model is that we leave
> expired accounts around for a year. The user can't login, and we don't
> accept new mails, but it's still there in case the account is
> re-activated. After one year the mailbox hierarchy is put into a .tgz
> and written to tape. When that is done the account is permanently
> deleted. If the user should come back, they get a completely new

In fact I just notice, I've no idea...how to remove a mailbox in cyrus....

With dovecot it's rm -rf ;-) Something I famillar with.

> account. I can't recall a single instance where the .tgz was ever
> needed, but that's not my problem. After one more year the .tgz is

Absolutly.

> deleted from tape as well.

Ok.

Thanks

Regards.


--
Albert SHIH
Observatoire de Paris
xmpp: jas at obspm.fr
Heure local/Local time:
Tue 23 Jun 2020 04:09:05 PM CEST

From ktm at rice.edu  Tue Jun 23 10:16:51 2020
From: ktm at rice.edu (Kenneth Marshall)
Date: Tue, 23 Jun 2020 09:16:51 -0500
Subject: What you do with old account
In-Reply-To: <20200623141056.GE76707@io.chezmoi.fr>
References: <20200609135142.GI4484@io.chezmoi.fr>
 <1591713123.5805.8.camel@whitemice.org>
 <b6b753ee-f16c-b86d-5241-38b6bb3b3a17@uni-koeln.de>
 <20200623141056.GE76707@io.chezmoi.fr>
Message-ID: <20200623141651.GI1497@aart.rice.edu>

On Tue, Jun 23, 2020 at 04:10:56PM +0200, Albert Shih wrote:
> 
> In fact I just notice, I've no idea...how to remove a mailbox in cyrus....
> 
> With dovecot it's rm -rf ;-) Something I famillar with.

cyradm deletemailbox user/xxx

Regards,
Ken

From ellie at fastmail.com  Tue Jun 23 20:41:26 2020
From: ellie at fastmail.com (ellie timoney)
Date: Wed, 24 Jun 2020 10:41:26 +1000
Subject: reconstructing mailboxes from backup
In-Reply-To: <12F9FC39-70B9-41E0-8813-876A898E917C@coote.org>
References: <12F9FC39-70B9-41E0-8813-876A898E917C@coote.org>
Message-ID: <99e17480-3c2a-4b2f-ab40-689acd5fb4bd@www.fastmail.com>

Hi Tim,

It's worth observing that, in Cyrus, the user "george"'s IMAP inbox is the "user/george" folder.  Which means, on disk, this user has another folder called "INBOX" within their inbox.  Depending on the Cyrus version, and maybe depending on your server's value of "altnamespace", this is invalid -- and it looks like your reconstruct has skipped it and everything under it, unsurprisingly.

It's also worth observing that there is both a "Deleted Messages" folder as a subdirectory of the bad "INBOX", and an "INBOX^Deleted Messages" directory that looks like maybe the result of unixhierarchysep being changed out from under the client, or something like that.  Looks like reconstruct has pulled the latter one in, cause it's not technically bad (but it will be very weird/confusing for the user to have a folder called "INBOX.Deleted Messages" in their client that is neither their inbox, nor their Deleted Messages folder, nor a directory hierarchy of the two).

So, it looks like reconstruct has found all the valid folders, and skipped the invalid INBOX and everything in it.  That seems coherent.

If you can start again from scratch: then I'd suggest renaming, on disk, that "INBOX" folder to something like "old inbox", and optionally renaming the "INBOX^Deleted Messages" folder to something like "old deleted messages", before you run the reconstruct.  Then the reconstruct will be able to find everything, and the user can then move the messages from the "old..." folders back into wherever they want them to be just over IMAP.

If you can't start again from scratch: then you should only rename the bad "INBOX" folder on disk, and then reconstruct. The previous reconstruct already found and repaired the "INBOX^Deleted Messages" folder, so renaming it on disk now might make a new mess.  But it can be renamed over IMAP, either by an admin session or the user.

Hope this helps :)

ellie

On Tue, Jun 23, 2020, at 11:42 PM, Tim Coote wrote:
> Hullo
> 
> I have a cyrus implementation on Fedora for a small (~10) users that?s 
> been migrated through many versions of the various components, 
> including several different of IMAP clients.
> 
> Realising the fragility of the setup, I thought I?d restore from a 
> backup. However, I?m finding that several of the mailboxes are not 
> being recovered. I feel that I am missing somethign obvious, but I 
> cannot spot it. 
> 
> The restoring version of cyrus-imap is: cyrus-imapd-3.0.13-2.fc32.x86_64,
> 
> The restored filesystem layout can be summarised thus:
> 
> `sudo find /var/spool/imap/g | grep cyrus.header`:
> 
> /var/spool/imap/g/user/george/Notes/cyrus.header
> /var/spool/imap/g/user/george/cyrus.header
> /var/spool/imap/g/user/george/Sent Messages/cyrus.header
> /var/spool/imap/g/user/george/Deleted Messages/cyrus.header
> /var/spool/imap/g/user/george/Sent/cyrus.header
> /var/spool/imap/g/user/george/Trash/cyrus.header
> /var/spool/imap/g/user/george/INBOX/Sent Messages/cyrus.header
> /var/spool/imap/g/user/george/INBOX/Deleted Messages/cyrus.header
> /var/spool/imap/g/user/george/INBOX/Drafts/cyrus.header
> /var/spool/imap/g/user/george/INBOX^Deleted Messages/cyrus.header
> /var/spool/imap/g/user/george/Drafts/cyrus.header
> 
> [so I would expect all of the subdirectories to be reconstructed as mailboxes]
> 
> however, using:
> `sudo -u cyrus reconstruct -r -f user/george`
> 
> I only get:
> user/george
> user/george/Deleted Messages
> user/george/Drafts
> user/george/INBOX.Deleted Messages
> user/george/Notes
> user/george/Sent
> user/george/Sent Messages
> user/george/Trash
> 
> ie no subdirectories below the top level, but excluding those 
> directories below INBOX.
> Should there be a file: 
> `/var/spool/imap/g/user/george/INBOX/cyrus.header`? 
> 
> Is there anything that I should be doing/how can I recover the other mailboxes?
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

From ellie at fastmail.com  Wed Jun 24 00:52:13 2020
From: ellie at fastmail.com (ellie timoney)
Date: Wed, 24 Jun 2020 14:52:13 +1000
Subject: backupd and sync_client IOERROR
In-Reply-To: <ec0c4c88-32f7-e903-9b25-b2ded3f6bb88@ruparpiemonte.it>
References: <7d2c31db-a3b3-fca3-9e25-574c373e7f23@ruparpiemonte.it>
 <ba49a89c-04de-4960-b280-b4136715d640@www.fastmail.com>
 <a53e5a30-3264-b696-23ff-1c141ad4a900@ruparpiemonte.it>
 <d20e84d8-5764-4beb-b3b0-ed63fbb408f2@www.fastmail.com>
 <ec0c4c88-32f7-e903-9b25-b2ded3f6bb88@ruparpiemonte.it>
Message-ID: <db1366db-838f-4d5f-9f41-0461a26e1d97@www.fastmail.com>

> I think there isn't a all-in-one command for this use case: a user 
> expunged some messages and deleted some folders somewhere. I want to 
> recover all expunged messages and all the deleted folders which are no 
> more present in the original IMAP server (because they were expired from 
> cyr_expire).
> 
> I have to set "-x" to avoid duplication of messages.
> With "-a -x" I recover all expunged messages and all deleted mailboxes. 
> But messages inside deleted mailboxes are not marked as expunged, so 
> these mailboxes are recovered empty in the IMAP server.

I would run restore twice in this case:  once without -x, specifying just the deleted mailboxes (you can use "cyr_backup list mailboxes ..." to get a list of the mailboxes in the user's backup).  And once with -x to get all the expunged stuff.  For the -x invocation only, I would probably also use the -M option to dump all the recovered stuff into a new folder, so they can easily tell it apart from any new mail that might have arrived coincidentally at the same time.

> Another idea is to recover all in another empty IMAP server, without 
> "-x" at all, and the user can look at the mailbox recovered there...

We kinda had a similar idea!  Putting it all into a folder with -M is much easier than setting up a separate server, but it'll lose the folder structure.  Recovering to a separate server means the folder structure can be preserved.  I guess it depends on the specific recovery situation, and how hard it is to spin up a server in your environment.

Cheers,

ellie

From javier.sanchez at coam.org  Wed Jun 24 10:30:30 2020
From: javier.sanchez at coam.org (javier.sanchez at coam.org)
Date: Wed, 24 Jun 2020 16:30:30 +0200
Subject: MESSAGES UNSEEN RECENT and IMAP server closes connection.
Message-ID: <80fe9aca-ca50-2b9a-5653-de267a3e7c3d@coam.org>


 ??? Hi


 ??? I'm dealing with a very old cyrus server (cyrus-imapd-2.2.12-27.2).


 ??? Two days ago we had a problem that made "/var" full (/var/lib/imap/ 
is the cyrus imap directory).


 ??? After cleaning the postfix queue and restarting cyrus daemon 
everything looked to be fine. But today we have found two user mailboxes 
that are unable to login using IMAP (pop3 looks to be working for these 
users).


 ??? When the user tries to login IMAP the connection with the server is 
closed with the message "MESSAGES UNSEEN RECENT".


 ??? This is the trace:

 ??? [...]

---------- user Wed Jun 24 16:15:48 2020

 >1593008148>A001 OK User logged in
<1593008148<A002 CAPABILITY
 >1593008148>* CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ 
MAILBOX-REFERRALS NAMESPACE UIDPLUS ID NO_ATOMIC_RENAME UNSELECT 
CHILDREN MULTIAPPEND BINARY SORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES 
ANNOTATEMORE IDLE LOGINDISABLED X-NETSCAPE
A002 OK Completed
<1593008148<A003 NAMESPACE
 >1593008148>* NAMESPACE (("INBOX." ".")) (("user." ".")) (("" "."))
A003 OK Completed
<1593008148<A004 LOGOUT
 >1593008148>* BYE LOGOUT received
A004 OK Completed

---------- user Wed Jun 24 16:15:49 2020

 >1593008149>A001 OK User logged in
<1593008149<A002 STATUS "INBOX" (MESSAGES UNSEEN RECENT)

 ??? [...]



 ??? We have tried to reconstruct the mailbox (-r -f), the reconstruct 
command finishes without errors but the problem persists.


 ??? Any idea about what is happening and how to try to solve It.


 ??? Kind regards.



-- 


Javier S?nchez.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200624/a5af8f18/attachment.html>

From steffo76 at gmx.de  Fri Jun 26 07:06:30 2020
From: steffo76 at gmx.de (Stephan)
Date: Fri, 26 Jun 2020 13:06:30 +0200
Subject: Upgrade to 3.2.2 and sieve
Message-ID: <a12acd02-2304-f04d-7536-b96eec6070cb@gmx.de>

Hi all,

I'm having trouble upgrading version 3.0.13 to 3.2.2. I am running two
systems, one of them is a replica. I upgraded on the replica first and
had issues with "intermediate" mailboxes being created which seems to be
expected if I read the release notes right.

Now I'm seeing errors on the replica when users update their script on
the master, they do this via horde's ingo module. On the replica I get
errors like

sieve_rebuild: [path to script] parse failed: script errors:#015#012line
5: header 'resent-from': not a valid header for an address test

These lines are generated by ingo when entering whitelist addresses and
3.2.2 doesn't like them

if address :all :comparator "i;ascii-casemap" :is ["From", "Sender",
"Resent-From"] "thisaddress at ourdomain.xyz"  {

I'm not familiar with the sieve specification, is this a problem in
cyrus or should ingo handle things differently ?

Regards,
Stephan

From delepine at u-picardie.fr  Fri Jun 26 08:18:51 2020
From: delepine at u-picardie.fr (Jean Charles =?utf-8?B?RGVsw6lwaW5l?=)
Date: Fri, 26 Jun 2020 14:18:51 +0200
Subject: Upgrade to 3.2.2 and sieve
In-Reply-To: <a12acd02-2304-f04d-7536-b96eec6070cb@gmx.de>
References: <a12acd02-2304-f04d-7536-b96eec6070cb@gmx.de>
Message-ID: <20200626121851.GB149556@u-picardie.fr>

Stephan <steffo76 at gmx.de> ?crivait (wrote) :

> sieve_rebuild: [path to script] parse failed: script errors:#015#012line
> 5: header 'resent-from': not a valid header for an address test

I had the same error with a 2.5 to 3.0 migration.

Corrected with 'rfc3028_strict: 0' :

rfc3028_strict: 1
  If enabled, Sieve will be strict (per RFC 3028) with regards to which
  headers are allowed to be used in address and envelope tests.  This means that
  only those headers which are defined to contain addresses will be allowed in
  address tests and  only  "to"  and "from" will be allowed in envelope tests.
  When disabled, ANY grammatically correct header will be allowed.


From cjc at cjc.org  Fri Jun 26 08:43:41 2020
From: cjc at cjc.org (Cheng-Jih Chen)
Date: Fri, 26 Jun 2020 08:43:41 -0400
Subject: Newly arrived mail is marked as "read"
Message-ID: <86a4aaec-486c-c359-f81c-4c79a6dbd9f7@cjc.org>

I'm in the process of testing the upgrade from a CentOS 6 box running 
2.3.16 to a CentOS 8 box running 3.0.7.

The upgrade has been less troublesome than expected, but I'm seeing a 
strange problem with newly received mail being marked as "read" even 
though they have not been opened by the client.  This is the case for 
both mail sent locally as well as mail received externally (Gmail).

Can anyone point me in the right direction on where to look to fix this?

Thanks.

From steffo76 at gmx.de  Fri Jun 26 09:01:12 2020
From: steffo76 at gmx.de (Stephan)
Date: Fri, 26 Jun 2020 15:01:12 +0200
Subject: Upgrade to 3.2.2 and sieve
In-Reply-To: <20200626121851.GB149556@u-picardie.fr>
References: <a12acd02-2304-f04d-7536-b96eec6070cb@gmx.de>
 <20200626121851.GB149556@u-picardie.fr>
Message-ID: <8c4ad4ad-ee77-c29f-f59f-d4118d4dac0c@gmx.de>

Am 26.06.20 um 14:18 schrieb Jean Charles Del?pine:
> Stephan <steffo76 at gmx.de> ?crivait (wrote) :
>
>> sieve_rebuild: [path to script] parse failed: script errors:#015#012line
>> 5: header 'resent-from': not a valid header for an address test
>
> I had the same error with a 2.5 to 3.0 migration.
>
> Corrected with 'rfc3028_strict: 0' :
>
> rfc3028_strict: 1
>    If enabled, Sieve will be strict (per RFC 3028) with regards to which
>    headers are allowed to be used in address and envelope tests.  This means that
>    only those headers which are defined to contain addresses will be allowed in
>    address tests and  only  "to"  and "from" will be allowed in envelope tests.
>    When disabled, ANY grammatically correct header will be allowed.
>

Thank you ! rfc3028_strict was enabled here in 3.0.13 and the scripts
were accepted so I guess it must be one of the sieve bug fixes and
features mentioned in the 3.2 release notes.

From a_s_y at sama.ru  Sun Jun 28 08:28:25 2020
From: a_s_y at sama.ru (Sergey)
Date: Sun, 28 Jun 2020 16:28:25 +0400
Subject: cyrus-imap buildin: sse extention
Message-ID: <202006281628.25789.a_s_y@sama.ru>

Hello.

I saw what Cyrus-IMAP use SSE:

Hardware support:
   SSE4.2:             yes

Can Cyrus-IMAP be running on systems without SSE4 at this case?
If no, can I set limit to SSE2 ?

-- 
Regards,
Sergey

From ellie at fastmail.com  Sun Jun 28 20:29:19 2020
From: ellie at fastmail.com (ellie timoney)
Date: Mon, 29 Jun 2020 10:29:19 +1000
Subject: cyrus-imap buildin: sse extention
In-Reply-To: <202006281628.25789.a_s_y@sama.ru>
References: <202006281628.25789.a_s_y@sama.ru>
Message-ID: <2ca23805-7343-4e7d-8925-3d1bef44e59e@www.fastmail.com>

Hi Sergey,

> Hardware support:
>    SSE4.2:             yes

This is detected for a hardware implementation of the CRC32c algorithm.  Cyrus doesn't actually use it though, because it's not compatible with the existing CRC32 algorithm: i.e. for the same input, it produces a different checksum, which would break everything on a system with pre-existing data.

If you _want_ to use the hardware CRC32c algorithm on a brand new deployment (and know what you are doing), I believe at this stage you would need to patch Cyrus to use it -- the code is there, but it is never called.

> Can Cyrus-IMAP be running on systems without SSE4 at this case?

Yep, it'll work just fine.  The hardware CRC32c code will simply not be compiled, which, since it isn't used anyway, will have no effect.

> If no, can I set limit to SSE2 ?

There's currently no configurability for this at all.  I don't know if it's even possible to implement the same algorithm with SSE2.  At the moment I assume that, since it's looking for SSE42 specifically, then that means that the hardware feature it needs is probably only available in SSE42.

Cheers,

ellie

From me at anatoli.ws  Sun Jun 28 23:15:57 2020
From: me at anatoli.ws (Anatoli)
Date: Mon, 29 Jun 2020 00:15:57 -0300
Subject: cyrus-imap buildin: sse extention
In-Reply-To: <2ca23805-7343-4e7d-8925-3d1bef44e59e@www.fastmail.com>
References: <202006281628.25789.a_s_y@sama.ru>
 <2ca23805-7343-4e7d-8925-3d1bef44e59e@www.fastmail.com>
Message-ID: <4108dd82-db6a-b652-b397-c7753f55604b@anatoli.ws>

Ellie,

I also had the doubt about this feature, though I'd already seen a mention that the result of the hw implementation is incompatible (and before your last mail completely forgot about it).

Maybe it makes sense to remove its mention (and detection) from configure altogether, until it becomes useful? Just to not confuse those building it from sources.

Regards,
Anatoli

On 28/6/20 21:29, ellie timoney wrote:
> Hi Sergey,
> 
>> Hardware support:
>>    SSE4.2:             yes
> 
> This is detected for a hardware implementation of the CRC32c algorithm.  Cyrus doesn't actually use it though, because it's not compatible with the existing CRC32 algorithm: i.e. for the same input, it produces a different checksum, which would break everything on a system with pre-existing data.
> 
> If you _want_ to use the hardware CRC32c algorithm on a brand new deployment (and know what you are doing), I believe at this stage you would need to patch Cyrus to use it -- the code is there, but it is never called.
> 
>> Can Cyrus-IMAP be running on systems without SSE4 at this case?
> 
> Yep, it'll work just fine.  The hardware CRC32c code will simply not be compiled, which, since it isn't used anyway, will have no effect.
> 
>> If no, can I set limit to SSE2 ?
> 
> There's currently no configurability for this at all.  I don't know if it's even possible to implement the same algorithm with SSE2.  At the moment I assume that, since it's looking for SSE42 specifically, then that means that the hardware feature it needs is probably only available in SSE42.
> 
> Cheers,
> 
> ellie
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 

From ellie at fastmail.com  Sun Jun 28 23:31:09 2020
From: ellie at fastmail.com (ellie timoney)
Date: Mon, 29 Jun 2020 13:31:09 +1000
Subject: Newly arrived mail is marked as "read"
In-Reply-To: <86a4aaec-486c-c359-f81c-4c79a6dbd9f7@cjc.org>
References: <86a4aaec-486c-c359-f81c-4c79a6dbd9f7@cjc.org>
Message-ID: <eafd721f-8c42-465b-bebc-57e1f94f31f5@www.fastmail.com>

Hi,

I'm not sure, but it kind of sounds like your mailbox's index version is too old?

Around 2.4, the storage of the mailbox owner's seen state was moved from the seen databases to the cyrus.index.  (i.e. nowadays the seen database only stores the seen state for _other_ users who have been given access to a mailbox -- which is often nobody).

If your mailbox indexes have not been updated yet, then they won't have a field to record the owner's seen state, which might result in the behaviour you're seeing?

To update your mailbox indexes to the latest version, you can use the "-V max" option with the reconstruct utility.  The upgrade documentation for 3.0 (https://www.cyrusimap.org/3.0/imap/download/upgrade.html#reconstruct-databases-and-cache) mentions that this may take a long time, so maybe you already saw this but haven't gotten to that stage yet.

You should definitely take note of the warning on that page though, and consider upgrading to at least 3.0.11 rather than 3.0.7.

Hope this is helpful!

Cheers,

ellie

On Fri, Jun 26, 2020, at 10:43 PM, Cheng-Jih Chen wrote:
> I'm in the process of testing the upgrade from a CentOS 6 box running 
> 2.3.16 to a CentOS 8 box running 3.0.7.
> 
> The upgrade has been less troublesome than expected, but I'm seeing a 
> strange problem with newly received mail being marked as "read" even 
> though they have not been opened by the client.  This is the case for 
> both mail sent locally as well as mail received externally (Gmail).
> 
> Can anyone point me in the right direction on where to look to fix this?
> 
> Thanks.
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>

From a_s_y at sama.ru  Mon Jun 29 10:35:06 2020
From: a_s_y at sama.ru (Sergey)
Date: Mon, 29 Jun 2020 18:35:06 +0400
Subject: cyrus-imap building: sse extension
In-Reply-To: <2ca23805-7343-4e7d-8925-3d1bef44e59e@www.fastmail.com>
References: <202006281628.25789.a_s_y@sama.ru>
 <2ca23805-7343-4e7d-8925-3d1bef44e59e@www.fastmail.com>
Message-ID: <202006291835.06937.a_s_y@sama.ru>

On Monday 29 June 2020, ellie timoney wrote:

> If you _want_ to use the hardware CRC32c algorithm

No. I want for Cyrus-IMAP works on systems without SSE4 when it built
on system with SSE4.

> > Can Cyrus-IMAP be running on systems without SSE4 at this case?
> 
> Yep, it'll work just fine.  The hardware CRC32c code will simply not
> be compiled, which, since it isn't used anyway, will have no effect.

Question about running precompiled binary.

> I don't know if it's even possible to implement the same algorithm
> with SSE2

--disable-hardware-CRC32c would be enough In my case. :-)

> Cyrus doesn't actually use it though

--disable-hardware-CRC32c may be required in the future I think.

-- 
Regards,
Sergey

From a_s_y at sama.ru  Mon Jun 29 15:14:27 2020
From: a_s_y at sama.ru (Sergey)
Date: Mon, 29 Jun 2020 23:14:27 +0400
Subject: Cyrus IMAP 3.2.2 released
In-Reply-To: <36d4a79e-cde3-4d0f-8adb-9bd47cd16c7c@www.fastmail.com>
References: <36d4a79e-cde3-4d0f-8adb-9bd47cd16c7c@www.fastmail.com>
Message-ID: <202006292314.27972.a_s_y@sama.ru>

On Monday 22 June 2020, ellie timoney wrote:

> The Cyrus team is proud to announce the immediate availability of a new version of Cyrus IMAP: 3.2.2

Tests have issued a new warning compared to 3.0.x (building in Linux):

verify-elf: WARNING: ./usr/lib64/libcyrus.so.0.0.0: found executable STACK entry:
  GNU_STACK  0x000000 0x000000000000  0000 0x0000000000000000 0x000000 0x000000 RWE 0x10

Is executable STACK really needed?

-- 
Regards,
Sergey

From a_s_y at sama.ru  Mon Jun 29 17:53:15 2020
From: a_s_y at sama.ru (Sergey)
Date: Tue, 30 Jun 2020 01:53:15 +0400
Subject: Cyrus IMAP 3.2.2 (Re: cyrus imap 2.5.5 build in Linux: problem with
 pcre)
In-Reply-To: <201510162249.29940.a_s_y@sama.ru>
References: <201510162249.29940.a_s_y@sama.ru>
Message-ID: <202006300153.15551.a_s_y@sama.ru>

On Friday 16 October 2015, Sergey wrote:

> I wanted to build Cyrus-IMAP with libpcre but it did not success.
> System libpcre-devel package install headers to /usr/include/pcre.
 
I returned to this question. 3.2.2 still does not find /usr/include/pcre/pcre.h

$ pkg-config --cflags libpcre
-I/usr/include/pcre

Build is successful with CFLAGS="-I/usr/include/pcre" ./configure ...
Except perl/imap: CFLAGS is not used by Makefile.PL.

And there is another problem that is not obvious. -lpcreposix is needed
in perl/imap/Makefile.PL and in perl/sieve/managesieve/Makefile.PL I seems.

-- 
Regards,
Sergey

From ellie at fastmail.com  Mon Jun 29 19:07:21 2020
From: ellie at fastmail.com (ellie timoney)
Date: Tue, 30 Jun 2020 09:07:21 +1000
Subject: Cyrus IMAP 3.2.2 released
In-Reply-To: <202006292314.27972.a_s_y@sama.ru>
References: <36d4a79e-cde3-4d0f-8adb-9bd47cd16c7c@www.fastmail.com>
 <202006292314.27972.a_s_y@sama.ru>
Message-ID: <ed271a8e-0508-4739-a92d-66d1af4ab502@www.fastmail.com>

I have no idea what this refers to or where it comes from.  Any further information you could provide would be greatly appreciated!

Thanks

On Tue, Jun 30, 2020, at 5:14 AM, Sergey wrote:
> On Monday 22 June 2020, ellie timoney wrote:
> 
> > The Cyrus team is proud to announce the immediate availability of a new version of Cyrus IMAP: 3.2.2
> 
> Tests have issued a new warning compared to 3.0.x (building in Linux):
> 
> verify-elf: WARNING: ./usr/lib64/libcyrus.so.0.0.0: found executable 
> STACK entry:
>   GNU_STACK  0x000000 0x000000000000  0000 0x0000000000000000 0x000000 
> 0x000000 RWE 0x10
> 
> Is executable STACK really needed?
> 
> -- 
> Regards,
> Sergey
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>

From a_s_y at sama.ru  Tue Jun 30 06:36:52 2020
From: a_s_y at sama.ru (Sergey)
Date: Tue, 30 Jun 2020 14:36:52 +0400
Subject: Cyrus IMAP 3.2.2 released
In-Reply-To: <ed271a8e-0508-4739-a92d-66d1af4ab502@www.fastmail.com>
References: <36d4a79e-cde3-4d0f-8adb-9bd47cd16c7c@www.fastmail.com>
 <202006292314.27972.a_s_y@sama.ru>
 <ed271a8e-0508-4739-a92d-66d1af4ab502@www.fastmail.com>
Message-ID: <202006301436.52187.a_s_y@sama.ru>

On Tuesday 30 June 2020, ellie timoney wrote:

> > > The Cyrus team is proud to announce the immediate availability of
> > > a new version of Cyrus IMAP: 3.2.2 
> > 
> > Tests have issued a new warning compared to 3.0.x (building in Linux):
 
> > verify-elf: WARNING: ./usr/lib64/libcyrus.so.0.0.0: found executable 
> > STACK entry:
> >   GNU_STACK  0x000000 0x000000000000  0000 0x0000000000000000 0x000000 
> > 0x000000 RWE 0x10
> > 
> > Is executable STACK really needed?

> I have no idea what this refers to or where it comes from.  Any
> further information you could provide would be greatly appreciated! 
 
I didn't go investigate detail yet and remove it from binary by
"execstack -c libcyrus.so.0.0.0". Everything seems to be working.
The problem is probably wrong flags for gcc. I can find out it
later maybe.

-- 
Regards, Sergey

From a_s_y at sama.ru  Tue Jun 30 08:10:52 2020
From: a_s_y at sama.ru (Sergey)
Date: Tue, 30 Jun 2020 16:10:52 +0400
Subject: Cyrus IMAP 3.2.2 released
In-Reply-To: <36d4a79e-cde3-4d0f-8adb-9bd47cd16c7c@www.fastmail.com>
References: <36d4a79e-cde3-4d0f-8adb-9bd47cd16c7c@www.fastmail.com>
Message-ID: <202006301610.52081.a_s_y@sama.ru>

On Monday 22 June 2020, ellie timoney wrote:

> The Cyrus team is proud to announce the immediate availability
> of a new version of Cyrus IMAP: 3.2.2 
 
There was a problem with AC_SYS_LARGEFILE. Most likely it was
there before, but I was no need to use AC_SYS_LARGEFILE.
Programs for x32 cannot work with large partitions without
building with _FILE_OFFSET_BITS 64. AC_SYS_LARGEFILE in configure.ac
usually helps. This mostly helps when building Cyrus IMAP, but not
for all components.

without AC_SYS_LARGEFILE:

$ cat cyrus-imapd-3.2.2-alt0-SS.log | grep "uses non-LFS functions"| wc -l
42

with AC_SYS_LARGEFILE:

$ cat cyrus-imapd-3.2.2-alt0-SS.log | grep "uses non-LFS functions"| wc -l
7

verify-elf: WARNING: ./usr/lib/libcyrus_imap.so.0.0.0: uses non-LFS functions: statvfs
verify-elf: WARNING: ./usr/lib/cyrus/restore: uses non-LFS functions: lseek open
verify-elf: WARNING: ./usr/lib/cyrus/httpd: uses non-LFS functions: __fxstat __xstat
verify-elf: WARNING: ./usr/lib/cyrus/cyr_backup: uses non-LFS functions: lseek open
verify-elf: WARNING: ./usr/lib/cyrus/ctl_backups: uses non-LFS functions: lseek open
verify-elf: WARNING: ./usr/lib/cyrus/backupd: uses non-LFS functions: lseek open
verify-elf: WARNING: ./usr/lib/libcyrus.so.0.0.0: uses non-LFS functions: fopen

Put it in a bugtracker on github?

-- 
Regards, Sergey

From ellie at fastmail.com  Tue Jun 30 19:21:23 2020
From: ellie at fastmail.com (ellie timoney)
Date: Wed, 01 Jul 2020 09:21:23 +1000
Subject: Cyrus IMAP 3.2.2 released
In-Reply-To: <202006301610.52081.a_s_y@sama.ru>
References: <36d4a79e-cde3-4d0f-8adb-9bd47cd16c7c@www.fastmail.com>
 <202006301610.52081.a_s_y@sama.ru>
Message-ID: <81e56b4e-4ba6-43ae-a5b0-0af1d304fc0e@www.fastmail.com>

On Tue, Jun 30, 2020, at 10:10 PM, Sergey wrote:
> On Monday 22 June 2020, ellie timoney wrote:
> 
> > The Cyrus team is proud to announce the immediate availability
> > of a new version of Cyrus IMAP: 3.2.2 
>  
> There was a problem with AC_SYS_LARGEFILE. Most likely it was
> there before, but I was no need to use AC_SYS_LARGEFILE.
> Programs for x32 cannot work with large partitions without
> building with _FILE_OFFSET_BITS 64. AC_SYS_LARGEFILE in configure.ac
> usually helps. This mostly helps when building Cyrus IMAP, but not
> for all components.
> 
> without AC_SYS_LARGEFILE:
> 
> $ cat cyrus-imapd-3.2.2-alt0-SS.log | grep "uses non-LFS functions"| wc -l
> 42
> 
> with AC_SYS_LARGEFILE:
> 
> $ cat cyrus-imapd-3.2.2-alt0-SS.log | grep "uses non-LFS functions"| wc -l
> 7
> 
> verify-elf: WARNING: ./usr/lib/libcyrus_imap.so.0.0.0: uses non-LFS 
> functions: statvfs
> verify-elf: WARNING: ./usr/lib/cyrus/restore: uses non-LFS functions: 
> lseek open
> verify-elf: WARNING: ./usr/lib/cyrus/httpd: uses non-LFS functions: 
> __fxstat __xstat
> verify-elf: WARNING: ./usr/lib/cyrus/cyr_backup: uses non-LFS 
> functions: lseek open
> verify-elf: WARNING: ./usr/lib/cyrus/ctl_backups: uses non-LFS 
> functions: lseek open
> verify-elf: WARNING: ./usr/lib/cyrus/backupd: uses non-LFS functions: 
> lseek open
> verify-elf: WARNING: ./usr/lib/libcyrus.so.0.0.0: uses non-LFS 
> functions: fopen
> 
> Put it in a bugtracker on github?

Yes, please.  If you don't, I will, but then you won't get automatic notifications of updates.

Where do you see these warnings?  Are they appearing in a log somewhere, or is this output from an analysis tool you're running?  I tried searching for "verify-elf" but most of the results seem to be about signing binaries, which I don't think is related.

Cheers,

ellie

From ellie at fastmail.com  Tue Jun 30 20:19:26 2020
From: ellie at fastmail.com (ellie timoney)
Date: Wed, 01 Jul 2020 10:19:26 +1000
Subject: =?UTF-8?Q?Re:_Cyrus_IMAP_3.2.2_(Re:_cyrus_imap_2.5.5_build_in_Linux:_pro?=
 =?UTF-8?Q?blem_with_pcre)?=
In-Reply-To: <202006300153.15551.a_s_y@sama.ru>
References: <201510162249.29940.a_s_y@sama.ru>
 <202006300153.15551.a_s_y@sama.ru>
Message-ID: <58bf2d56-0c3c-46d9-a62c-7d82efa9ffba@www.fastmail.com>

Hi Sergey,

Quoting out of order, because it's a bit easier to explain that way:

> And there is another problem that is not obvious. -lpcreposix is needed
> in perl/imap/Makefile.PL and in perl/sieve/managesieve/Makefile.PL I seems.

This bit sounds a lot like https://github.com/cyrusimap/cyrus-imapd/issues/2629

There's an experimental patch from me in that issue, which seems to have worked around the issue for Fedora package maintainers.  But I can't reproduce the problem on Debian, so I can't confirm myself that it fixes things correctly.  More feedback would be great.

> I returned to this question. 3.2.2 still does not find /usr/include/pcre/pcre.h
> 
> $ pkg-config --cflags libpcre
> -I/usr/include/pcre
> 
> Build is successful with CFLAGS="-I/usr/include/pcre" ./configure ...

Cyrus's configure script does not currently use pkg-config to locate libpcre.  Instead it uses bespoke detection built from AC_CHECK_HEADER, which I guess doesn't know to look in /usr/include/pcre.  There might be a generic way for a sysadmin to add this path to the paths that AC_CHECK_HEADER checks, but I don't know it offhand.  Otherwise, specifying it in CFLAGS like you have done seems like the right thing to do.

The issue linked above also touches on the possibility of changing the libpcre detection to use pkg-config instead of bespoke detection, in which case this wouldn't be necessary, but it has its own complications.

> Except perl/imap: CFLAGS is not used by Makefile.PL.

Aaand this sounds like its own whole separate bug!  I'm not sure offhand what the right way to pass this down will be, but it clearly needs doing.  New issue here: https://github.com/cyrusimap/cyrus-imapd/issues/3087

Cheers,

ellie