From me at anatoli.ws  Tue Oct  1 18:24:59 2019
From: me at anatoli.ws (Anatoli)
Date: Tue, 1 Oct 2019 19:24:59 -0300
Subject: sieve filter based on utf-8 encoded text part of address
In-Reply-To: <3404478.SVmyWBeI9G@xrated>
References: <3404478.SVmyWBeI9G@xrated>
Message-ID: <3094de9c-37ab-1f7c-ad88-d5fcaba71232@anatoli.ws>

Hi Pete,

I guess the 'address' test command matches only the actual address, not
the description. In your example it would match "my at address".

I suggest you check
https://www.cyrusimap.org/imap/reference/admin/sieve.html and
https://thsmi.github.io/sieve-reference/en/index.html.

Regards,
Anatoli

On 30/9/19 08:12, Hans-Peter Jansen wrote:
> Hi,
> 
> I try to filter based on the text part of an utf-8 encoded address, but the 
> string matches neither decoded nor encoded.
> 
> This is a periodic mail from one of a couple of different routers, where only 
> the text part defines a useful origin:
> 
> From: "=?UTF-8?B?RlJJVFohQm94IDc0OTAgU0ZK?=" <my at address>
> 
> which decodes to:
> 
> FRITZ!Box 7490 SFJ <my at address>
> 
> sieve filter script except:
> 
> ; my requirements
> require ["fileinto", "reject", "regex", "vacation"];
> 
> ; the filter rule
> if address :contains "From" [
> 	"FRITZ!Box 7490 SFJ",
> 	"=?UTF-8?B?RlJJVFohQm94IDc0OTAgU0ZK?="
> ] {
>     fileinto "INBOX.some.folder";
>     stop;
> }
> 
> How do I do this correctly?
> 
> Thanks in advance,
> Pete
> 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 

From me at anatoli.ws  Tue Oct  1 18:28:40 2019
From: me at anatoli.ws (Anatoli)
Date: Tue, 1 Oct 2019 19:28:40 -0300
Subject: strange sort order of these emails
In-Reply-To: <502130238.636.1569509530124@www>
References: <502130238.636.1569509530124@www>
Message-ID: <8e394301-08dd-d62b-a7e2-2fee8e6cedad@anatoli.ws>

Hi Gabriele,

I suggest you send the actual headers of both emails here as plain text
in the body of the email, as people may not want to open attachments
from unknown sources. Also it's easier to analyze and reply this way.

Regards,
Anatoli

On 26/9/19 11:52, Gabriele Bulfon wrote:
> Hello,
> someone received these two emails in the inbox, but strangely?sorting by
> date descending was showing them swapped (they have same date but
> different hour).
> I did not beleive my eyes, so I downloaded the two eml files and
> uploaded them into a cyrus folder of mine, on another server.
> Same effect: the SORT command with descending DATE, will always returns
> indexes in swapped order, so the one at 3am is before the one at 9am.
> ?
> I looked at all the headers but didn't find anything strange.
> Cyrus is version 2.5.11.
> Any idea?!
> Thanks!
> Gabriele
> 
> ?
> *Sonicle S.r.l.?*:?http://www.sonicle.com <http://www.sonicle.com/>
> *Music:?*http://www.gabrielebulfon.com <http://www.gabrielebulfon.com/>
> *Quantum Mechanics :?*http://www.cdbaby.com/cd/gabrielebulfon
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 

From mp at petermann-it.de  Mon Oct  7 01:45:50 2019
From: mp at petermann-it.de (Matthias Petermann)
Date: Mon, 7 Oct 2019 07:45:50 +0200
Subject: Caching related security issue? (Cyrus httpd CalDAV server behind
 Apache mod_proxy)
Message-ID: <d7137493-ddcd-e8bf-9b48-4c0e004f0ed6@petermann-it.de>

Hello,

since Cyrus is able to share calendars, I try to set up a CalDAV server 
using Cyrus httpd behind an Apache reverse proxy (mod_proxy). During 
this I made a security-related observation which makes me a bit 
concerned. It seems to be about caching private data, probably more an 
issue of my individual setup and not a bug in Cyrus. Anyway ? I am a bit 
clueless and would be happy to discuss this to find a mitigation.

...

Test setup / precondtions:

- Cyrus 3.0.11 server (imapd, httpd) freshly restarted
- Apache 2.4 with mod_proxy (ssl vhost, no explicit cache configuration, 
only mod_socache_shmcb)
- Browser A with clean cache
- Browser B with clean cache

Test steps:

1) Request CalDAV URL from Browser A

- Expected: Browser presents HTTP Auth, delivers content after 
successful login
- Observed: Browser presents HTTP Auth, delivers content after 
successful login
- Result: PASS

2) Request same CalDAV URL from Browser B

- Expected: Browser presents HTTP Auth
- Observed: Browser delivers content without presenting HTTP Auth
- Result: FAIL

Test observations:

- The result of step 2) seems to differ depending of which Cyrus httpd 
process is hit by the request
- The cache-control header delivered by Cyrus httpd seems to not contain 
?private? which seems to allow intermediate caches to cache private data

...

My first configuration of Apache did allow mod_proxy to reuse the 
backend-connections to Cyrus httpd. After I added disablereuse=On (which 
causes Apache to close the backend-connection immediately after 
processing a single request), it seems to mitigate the observed issue. 
How could this be possible? These two questions could help me to 
investigate further on my system:

- When receiving a HTTP Request with an Etag included, when exactly does 
Cyrus httpd decide whether to request HTTP Authentication? In case the 
Etag is valid (content unchanged), will it return the 304 without 
requesting a HTTP Auth?

- How do the Cyrus httpd workers handle HTTP Auth in general? In case of 
re-using a worker by a permanent connection from a reverse proxy, does 
it check Authentication on any request, or only at the very first?

I am happy about suggestions or pointers to best practises in regards of 
using Cyrus httpd behind a reverse proxy.

Kind regards
Matthias



-- 
Matthias Petermann <mp at petermann-it.de> | www.petermann-it.de
Innovative IT-L?sungen, Systemintegration, Linux/FreeBSD/Unix-Support
Wildparkring 13, 01458 Ottendorf-Okrilla | Tel.: +49 (0)35205 597 991

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4001 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20191007/a9660bbf/attachment.p7s>

From nros at netbsd.org  Tue Oct  8 13:50:48 2019
From: nros at netbsd.org (Niclas Rosenvik)
Date: Tue, 8 Oct 2019 17:50:48 +0000
Subject: ftp seems to be down
Message-ID: <20191008175048.49769644@pkgsrc>

Hi, I just want to point out that the ftp server at ftp.cyrusimap.org
seems to be down. I have tried to connect to it the last couple of days
and it did not succeed. It is listed on the homepage as one of the
download sources for cyrus tarballs. 

Regards
Niclas

From debbiep at polyfoam.com.au  Tue Oct  8 22:46:43 2019
From: debbiep at polyfoam.com.au (Deborah Pickett)
Date: Wed, 9 Oct 2019 13:46:43 +1100
Subject: XBACKUP and backupd not backing up public folders (3.0.8)
Message-ID: <a04f8634-f9b3-2167-a36d-5130c6bc69a8@polyfoam.com.au>

Hi everyone,

I'm deploying Cyrus 3.0.8 (Debian buster 3.0.8-6) at $dayjob to replace
an Exchange server.? That part is going well, but I'm hitting a hurdle
pulling backups of public folders (shared mailboxes, calendars and
address books, anything outside the user/ hierarchy) using XBACKUP and
backupd.

Steps to reproduce:

1. On master server (mail-3175-1), run imtest and authenticate as admin.
2. Issue XBACKUP to backup normal user.? This succeeds.
3. Issue XBACKUP to backup public shared mailbox.? This produces error
BAD PROTOCOL.

Expected behaviour is that the backup server backs up this mailbox with
an OK response.

I'm seeing what looks like a segfault in the backup server logs.? Don't
know if this is significant.

Help?

Main server (mail-3175-1): Debian buster, cyrus 3.0.8-6
Backup server (rsync): Debian buster, cyrus 3.0.8-6

--- imtest session ---

mail-3175-1$ /usr/lib/cyrus/bin/imtest -a cyrus
WARNING: no hostname supplied, assuming localhost

S: * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=PLAIN
SASL-IR] mail-3175-1 Cyrus IMAP 3.0.8-Debian-3.0.8-6 server ready
Please enter your password:
C: A01 AUTHENTICATE PLAIN ***DELETED***
S: A01 OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxten
QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT
CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SEARCH=FUZZY SORT
SORT=MODSEQ SORT=DISPLAY SORT=UID THREAD=ORDEREDSUBJECT
THREAD=REFERENCES THREAD=REFS ANNOTATEMORE ANNOTATE-EXPERIMENT-1
METADATA LIST-EXTENDED LIST-STATUS LIST-MYRIGHTS LIST-METADATA WITHIN
QRESYNC SCAN XLIST XMOVE MOVE SPECIAL-USE CREATE-SPECIAL-USE DIGEST=SHA1
X-REPLICATION URLAUTH URLAUTH=BINARY LOGINDISABLED COMPRESS=DEFLATE
X-QUOTA=STORAGE X-QUOTA=MESSAGE X-QUOTA=X-ANNOTATION-STORAGE
X-QUOTA=X-NUM-FOLDERS IDLE] Success (no protection)
SESSIONID=<cyrus-***DELETED***>
Authenticated.
Security strength factor: 0
AAA XBACKUP user/debbiep at polyfoam.com.au rsync
* OK USER debbiep at polyfoam.com.au
AAA OK Completed
BBB XBACKUP support at polyfoam.com.au rsync
* NO MAILBOX polyfoam.com.au!support (Bad protocol)
BBB NO Bad protocol

--- log on master ---

Oct? 9 13:31:37 mail-3175-1 cyrus/imap[189353]: login: localhost [::1]
cyrus PLAIN User logged in
SESSIONID=<cyrus-189353-1570588288-1-17156687401313158221>
Oct? 9 13:31:55 mail-3175-1 cyrus/imap[189353]: XBACKUP: connecting to
server 'rsync.polyfoam.com.au' for channel 'rsync'
Oct? 9 13:32:01 mail-3175-1 cyrus/imap[189353]: XBACKUP: replicating
user debbiep at polyfoam.com.au
Oct? 9 13:32:15 mail-3175-1 cyrus/imap[189353]: XBACKUP: connecting to
server 'rsync.polyfoam.com.au' for channel 'rsync'
Oct? 9 13:32:21 mail-3175-1 cyrus/imap[189353]: XBACKUP: replicating
mailbox polyfoam.com.au!support
Oct? 9 13:32:21 mail-3175-1 cyrus/imap[189353]: IOERROR: zero length
response to MAILBOXES (end of file reached)
Oct? 9 13:32:21 mail-3175-1 cyrus/imap[189353]: IOERROR: zero length
response to RESTART (end of file reached)
Oct? 9 13:32:22 mail-3175-1 cyrus/imap[189353]: USAGE cyrus user:
0.059546 sys: 0.024519

--- log on backup server ---

***successful backup of user debbiep not shown***
Oct? 9 13:32:15 rsync cyrus/backupd[21340]: telling master 2
Oct? 9 13:32:15 rsync cyrus/backupd[21340]: accepted connection
Oct? 9 13:32:15 rsync cyrus/backupd[21340]: telling master 3
Oct? 9 13:32:15 rsync cyrus/master[16697]: service backupd/ipv4 pid
21340 in READY state: now unavailable and in BUSY state
Oct? 9 13:32:15 rsync cyrus/master[16697]: service backupd/ipv4 now has
0 ready workers
Oct? 9 13:32:15 rsync cyrus/master[16697]: service backupd/ipv4 pid
21340 in BUSY state: now serving connection
Oct? 9 13:32:15 rsync cyrus/master[16697]: service backupd/ipv4 now has
0 ready workers
Oct? 9 13:32:21 rsync cyrus/backupd[21340]: login:
mail-3175-1.polyfoam.com.au [10.3.244.125] rsync-mail-3175-1 DIGEST-MD5
User logged in
Oct? 9 13:32:21 rsync cyrus/master[16697]: process type:SERVICE
name:backupd path:/usr/lib/cyrus/bin/backupd age:25.007s pid:21340
signaled to death by signal 11 (Segmentation fault)
Oct? 9 13:32:21 rsync cyrus/master[16697]: service backupd/ipv4 pid
21340 in BUSY state: terminated abnormally
Oct? 9 13:32:21 rsync cyrus/master[16697]: service backupd/ipv4 now has
0 ready workers

---master config---

admins: cyrus
allowanonymouslogin: no
allowplaintext: yes
altnamespace: yes
autocreate_inbox_folders: Junk|Trash|Archive|Drafts|Important|Sent
autocreate_post: yes
autocreate_quota: 0
autocreate_subscribe_folders: Junk|Trash|Archive|Drafts|Sent
calendarprefix: #calendars
configdirectory: /var/lib/cyrus
defaultdomain: ad.polyfoam.com.au
defaultpartition: default
hashimapspool: true
httpmodules: caldav carddav
idlesocket: /run/cyrus/socket/idle
lmtp_downcase_rcpt: yes
lmtpsocket: /run/cyrus/socket/lmtp
loginrealms:? ad.polyfoam.com.au polyfoam.com.au flexifoam.com.au
mboxname_lockpath: /run/cyrus/lock
newsspool: /var/spool/news
notifysocket: /run/cyrus/socket/notify
partition-default: /var/spool/cyrus/mail
partition-news: /var/spool/cyrus/news
popminpoll: 1
proc_path: /run/cyrus/proc
rsync_sync_authname: rsync-mail-3175-1 at rsync
rsync_sync_host: rsync.polyfoam.com.au
rsync_sync_password: ***DELETED***
rsync_sync_port: csync
sasl_auto_transition: no
sasl_auxprop_plugin: sasldb
sasl_mech_list: PLAIN
sasl_pwcheck_method: auxprop saslauthd
sievedir: /var/spool/sieve
sieveusehomedir: false
syslog_prefix: cyrus
tls_client_ca_dir: /etc/ssl/certs
tls_server_cert: /etc/cert/mail-3175-1.polyfoam.com.au.crt
tls_server_key: /etc/cert/mail-3175-1.polyfoam.com.au.key
tls_session_timeout: 1440
umask: 077
unixhierarchysep: yes
virtdomains: on
xbackup_enabled: yes
xlist-Archive: Archive
xlist-Drafts: Drafts
xlist-Important: Important
xlist-Junk: Junk
xlist-Sent: Drafts
xlist-Trash: Trash

---backup config---

admins: rsync-mail-3175-1
allowanonymouslogin: no
altnamespace: yes
backup_compact_maxsize: 0
backup_compact_minsize: 0
backup_compact_work_threshold: 1
backup_db: twoskip
backuppartition-name: /home/mail-3175-1/cyrus-backup/partitions/default
backup_retention_days: 7
configdirectory: /var/lib/cyrus
debug: 1
defaultdomain: rsync
mboxname_lockpath: /run/cyrus/lock
proc_path: /run/cyrus/proc
sasl_auxprop_plugin: sasldb
sasl_mech_list: DIGEST-MD5
sasl_pwcheck_method: auxprop
syslog_prefix: cyrus
unixhierarchysep: yes
virtdomains: on


From ellie at fastmail.com  Wed Oct  9 01:17:45 2019
From: ellie at fastmail.com (ellie timoney)
Date: Wed, 09 Oct 2019 16:17:45 +1100
Subject: XBACKUP and backupd not backing up public folders (3.0.8)
In-Reply-To: <a04f8634-f9b3-2167-a36d-5130c6bc69a8@polyfoam.com.au>
References: <a04f8634-f9b3-2167-a36d-5130c6bc69a8@polyfoam.com.au>
Message-ID: <90b6d2f0-924d-4530-93fd-3eeaf95b0e18@www.fastmail.com>

Hi Deborah,

Does the same problem occur if you use sync_client (on the master server, as the cyrus user) to replicate the shared mailbox to the backup server (rather than using XBACKUP over IMAP)?  Something like "sync_client -n rsync  -m support at polyfoam.com.au" I think?  What about if you use "sync_client -n rsync -u support at polyfoam.com.au" instead (i.e. with -u treating the shared mailbox as a USER rather than as a -m MAILBOX)?

On the backup server, what does the "ctl_backups verify -vvv -m polyfoam.com.au!support" command say about the shared mailbox?  There might be personally-identifying information in this output, I can't remember -- please check/censor carefully before pasting into email.

> I'm seeing what looks like a segfault in the backup server logs.? Don't
> know if this is significant.

It's almost certainly significant.  Are you able to enable core dumps and get a backtrace from it?  That'll probably be the fastest path toward a solution.  Let me know if you need help with this :)

Cheers,

ellie

On Wed, Oct 9, 2019, at 1:46 PM, Deborah Pickett wrote:
> Hi everyone,
> 
> I'm deploying Cyrus 3.0.8 (Debian buster 3.0.8-6) at $dayjob to replace
> an Exchange server.? That part is going well, but I'm hitting a hurdle
> pulling backups of public folders (shared mailboxes, calendars and
> address books, anything outside the user/ hierarchy) using XBACKUP and
> backupd.
> 
> Steps to reproduce:
> 
> 1. On master server (mail-3175-1), run imtest and authenticate as admin.
> 2. Issue XBACKUP to backup normal user.? This succeeds.
> 3. Issue XBACKUP to backup public shared mailbox.? This produces error
> BAD PROTOCOL.
> 
> Expected behaviour is that the backup server backs up this mailbox with
> an OK response.
> 
> I'm seeing what looks like a segfault in the backup server logs.? Don't
> know if this is significant.
> 
> Help?
> 
> Main server (mail-3175-1): Debian buster, cyrus 3.0.8-6
> Backup server (rsync): Debian buster, cyrus 3.0.8-6
> 
> --- imtest session ---
> 
> mail-3175-1$ /usr/lib/cyrus/bin/imtest -a cyrus
> WARNING: no hostname supplied, assuming localhost
> 
> S: * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=PLAIN
> SASL-IR] mail-3175-1 Cyrus IMAP 3.0.8-Debian-3.0.8-6 server ready
> Please enter your password:
> C: A01 AUTHENTICATE PLAIN ***DELETED***
> S: A01 OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxten
> QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT
> CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SEARCH=FUZZY SORT
> SORT=MODSEQ SORT=DISPLAY SORT=UID THREAD=ORDEREDSUBJECT
> THREAD=REFERENCES THREAD=REFS ANNOTATEMORE ANNOTATE-EXPERIMENT-1
> METADATA LIST-EXTENDED LIST-STATUS LIST-MYRIGHTS LIST-METADATA WITHIN
> QRESYNC SCAN XLIST XMOVE MOVE SPECIAL-USE CREATE-SPECIAL-USE DIGEST=SHA1
> X-REPLICATION URLAUTH URLAUTH=BINARY LOGINDISABLED COMPRESS=DEFLATE
> X-QUOTA=STORAGE X-QUOTA=MESSAGE X-QUOTA=X-ANNOTATION-STORAGE
> X-QUOTA=X-NUM-FOLDERS IDLE] Success (no protection)
> SESSIONID=<cyrus-***DELETED***>
> Authenticated.
> Security strength factor: 0
> AAA XBACKUP user/debbiep at polyfoam.com.au rsync
> * OK USER debbiep at polyfoam.com.au
> AAA OK Completed
> BBB XBACKUP support at polyfoam.com.au rsync
> * NO MAILBOX polyfoam.com.au!support (Bad protocol)
> BBB NO Bad protocol
> 
> --- log on master ---
> 
> Oct? 9 13:31:37 mail-3175-1 cyrus/imap[189353]: login: localhost [::1]
> cyrus PLAIN User logged in
> SESSIONID=<cyrus-189353-1570588288-1-17156687401313158221>
> Oct? 9 13:31:55 mail-3175-1 cyrus/imap[189353]: XBACKUP: connecting to
> server 'rsync.polyfoam.com.au' for channel 'rsync'
> Oct? 9 13:32:01 mail-3175-1 cyrus/imap[189353]: XBACKUP: replicating
> user debbiep at polyfoam.com.au
> Oct? 9 13:32:15 mail-3175-1 cyrus/imap[189353]: XBACKUP: connecting to
> server 'rsync.polyfoam.com.au' for channel 'rsync'
> Oct? 9 13:32:21 mail-3175-1 cyrus/imap[189353]: XBACKUP: replicating
> mailbox polyfoam.com.au!support
> Oct? 9 13:32:21 mail-3175-1 cyrus/imap[189353]: IOERROR: zero length
> response to MAILBOXES (end of file reached)
> Oct? 9 13:32:21 mail-3175-1 cyrus/imap[189353]: IOERROR: zero length
> response to RESTART (end of file reached)
> Oct? 9 13:32:22 mail-3175-1 cyrus/imap[189353]: USAGE cyrus user:
> 0.059546 sys: 0.024519
> 
> --- log on backup server ---
> 
> ***successful backup of user debbiep not shown***
> Oct? 9 13:32:15 rsync cyrus/backupd[21340]: telling master 2
> Oct? 9 13:32:15 rsync cyrus/backupd[21340]: accepted connection
> Oct? 9 13:32:15 rsync cyrus/backupd[21340]: telling master 3
> Oct? 9 13:32:15 rsync cyrus/master[16697]: service backupd/ipv4 pid
> 21340 in READY state: now unavailable and in BUSY state
> Oct? 9 13:32:15 rsync cyrus/master[16697]: service backupd/ipv4 now has
> 0 ready workers
> Oct? 9 13:32:15 rsync cyrus/master[16697]: service backupd/ipv4 pid
> 21340 in BUSY state: now serving connection
> Oct? 9 13:32:15 rsync cyrus/master[16697]: service backupd/ipv4 now has
> 0 ready workers
> Oct? 9 13:32:21 rsync cyrus/backupd[21340]: login:
> mail-3175-1.polyfoam.com.au [10.3.244.125] rsync-mail-3175-1 DIGEST-MD5
> User logged in
> Oct? 9 13:32:21 rsync cyrus/master[16697]: process type:SERVICE
> name:backupd path:/usr/lib/cyrus/bin/backupd age:25.007s pid:21340
> signaled to death by signal 11 (Segmentation fault)
> Oct? 9 13:32:21 rsync cyrus/master[16697]: service backupd/ipv4 pid
> 21340 in BUSY state: terminated abnormally
> Oct? 9 13:32:21 rsync cyrus/master[16697]: service backupd/ipv4 now has
> 0 ready workers
> 
> ---master config---
> 
> admins: cyrus
> allowanonymouslogin: no
> allowplaintext: yes
> altnamespace: yes
> autocreate_inbox_folders: Junk|Trash|Archive|Drafts|Important|Sent
> autocreate_post: yes
> autocreate_quota: 0
> autocreate_subscribe_folders: Junk|Trash|Archive|Drafts|Sent
> calendarprefix: #calendars
> configdirectory: /var/lib/cyrus
> defaultdomain: ad.polyfoam.com.au
> defaultpartition: default
> hashimapspool: true
> httpmodules: caldav carddav
> idlesocket: /run/cyrus/socket/idle
> lmtp_downcase_rcpt: yes
> lmtpsocket: /run/cyrus/socket/lmtp
> loginrealms:? ad.polyfoam.com.au polyfoam.com.au flexifoam.com.au
> mboxname_lockpath: /run/cyrus/lock
> newsspool: /var/spool/news
> notifysocket: /run/cyrus/socket/notify
> partition-default: /var/spool/cyrus/mail
> partition-news: /var/spool/cyrus/news
> popminpoll: 1
> proc_path: /run/cyrus/proc
> rsync_sync_authname: rsync-mail-3175-1 at rsync
> rsync_sync_host: rsync.polyfoam.com.au
> rsync_sync_password: ***DELETED***
> rsync_sync_port: csync
> sasl_auto_transition: no
> sasl_auxprop_plugin: sasldb
> sasl_mech_list: PLAIN
> sasl_pwcheck_method: auxprop saslauthd
> sievedir: /var/spool/sieve
> sieveusehomedir: false
> syslog_prefix: cyrus
> tls_client_ca_dir: /etc/ssl/certs
> tls_server_cert: /etc/cert/mail-3175-1.polyfoam.com.au.crt
> tls_server_key: /etc/cert/mail-3175-1.polyfoam.com.au.key
> tls_session_timeout: 1440
> umask: 077
> unixhierarchysep: yes
> virtdomains: on
> xbackup_enabled: yes
> xlist-Archive: Archive
> xlist-Drafts: Drafts
> xlist-Important: Important
> xlist-Junk: Junk
> xlist-Sent: Drafts
> xlist-Trash: Trash
> 
> ---backup config---
> 
> admins: rsync-mail-3175-1
> allowanonymouslogin: no
> altnamespace: yes
> backup_compact_maxsize: 0
> backup_compact_minsize: 0
> backup_compact_work_threshold: 1
> backup_db: twoskip
> backuppartition-name: /home/mail-3175-1/cyrus-backup/partitions/default
> backup_retention_days: 7
> configdirectory: /var/lib/cyrus
> debug: 1
> defaultdomain: rsync
> mboxname_lockpath: /run/cyrus/lock
> proc_path: /run/cyrus/proc
> sasl_auxprop_plugin: sasldb
> sasl_mech_list: DIGEST-MD5
> sasl_pwcheck_method: auxprop
> syslog_prefix: cyrus
> unixhierarchysep: yes
> virtdomains: on
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

From byrnejb at harte-lyne.ca  Wed Oct  9 11:41:49 2019
From: byrnejb at harte-lyne.ca (James B. Byrne)
Date: Wed, 9 Oct 2019 11:41:49 -0400
Subject: Cyrus-3.0 getting rid of old message files
Message-ID: <f1fea4be6bc63b84ce21178dcc7a46a5.squirrel@webmail.harte-lyne.ca>

What does one do to permanently remove from the file system old
messages.  In cyrus.conf I have this:

postmaster    cmd="ipurge -X -f -d 8 user/postmaster/delivery" at=0420
delprune      cmd="cyr_expire -D 180d -E 3d -X 180d" at=0400

cyradm info shows this:

info
{Server Wide}
  private:
    admin: NIL
    comment: NIL
    expire: NIL
    squat: NIL
    usercounters: 0 0 0 0 0 0 0 0 0 0 0
    usermodseq: 0
  shared:
    admin: NIL
    comment: NIL
    motd: NIL
    expire: NIL
    freespace: 68182048
    most: 68182048;192015872
    total: 68182048;192015872
    shutdown: NIL
    squat: NIL

When I go to the user mailbox
(/var/spool/imap/p/user/postmaster/delivery) I see messages from
December 2018 and earlier.

When I check the maillog I see this:

# grep cyr_expire /var/log/maillog | grep postmaster
Oct  9 05:46:14 inet17 CYRUS/cyr_expire[21667]: mailbox: longlock
user.postmaster.investigate for 2.3 seconds
Oct  9 05:46:16 inet17 CYRUS/cyr_expire[21667]: mailbox: longlock
user.postmaster.mailinglists for 1.2 seconds
Oct  9 05:46:19 inet17 CYRUS/cyr_expire[21667]: mailbox: longlock
user.postmaster.spamno for 1.3 seconds

What I do not see is anything being expunged for the postmaster. 
cyr_expire is running as I see this in the maillog:

inet17 CYRUS/cyr_expire[21667]: Expired 0 and expunged 2055 out of
744420 messages from 2500 mailboxes


I want messages that are deleted and that are older than 180 days gone
from the file system entirely.  I would like to know why is this not
happening?  And, I would like to know how do I make it happen?  What
setting or settings am I missing?



-- 
***          e-Mail is NOT a SECURE channel          ***
        Do NOT transmit sensitive data via e-Mail
 Do NOT open attachments nor follow links sent by e-Mail

James B. Byrne                mailto:ByrneJB at Harte-Lyne.ca
Harte & Lyne Limited          http://www.harte-lyne.ca
9 Brockley Drive              vox: +1 905 561 1241
Hamilton, Ontario             fax: +1 905 561 0757
Canada  L8E 3C3


From debbiep at polyfoam.com.au  Fri Oct 11 03:02:17 2019
From: debbiep at polyfoam.com.au (Deborah Pickett)
Date: Fri, 11 Oct 2019 18:02:17 +1100
Subject: XBACKUP and backupd not backing up public folders (3.0.8)
In-Reply-To: <90b6d2f0-924d-4530-93fd-3eeaf95b0e18@www.fastmail.com>
References: <a04f8634-f9b3-2167-a36d-5130c6bc69a8@polyfoam.com.au>
 <90b6d2f0-924d-4530-93fd-3eeaf95b0e18@www.fastmail.com>
Message-ID: <d9f4f951-baaf-68e1-60d1-be2768621349@polyfoam.com.au>

Hi Ellie,

Thanks for helping me look at this.

On 2019-10-09 16:17, ellie timoney wrote:
> Does the same problem occur if you use sync_client (on the master server, as the cyrus user) to replicate the shared mailbox to the backup server (rather than using XBACKUP over IMAP)?  Something like "sync_client -n rsync  -m support at polyfoam.com.au" I think? 

With -m I get this familiar output on the master:

# /usr/lib/cyrus/bin/sync_client -v -n rsync -m support at polyfoam.com.au
MAILBOXES polyfoam.com.au!support
Error from sync_do_mailboxes(): bailing out!

and this is seen in the log on the backup server:

Oct 11 17:39:58 rsync cyrus/backupd[3969]: login:
mail-3175-1.polyfoam.com.au [10.3.244.125] rsync-mail-3175-1 DIGEST-MD5
User logged in
Oct 11 17:39:58 rsync cyrus/backupd[3969]: created decompress buffer of
4102 bytes
Oct 11 17:39:58 rsync cyrus/backupd[3969]: created compress buffer of
4073 bytes
Oct 11 17:39:58 rsync cyrus/backupd[3969]: decompressed 47 -> 41 bytes
Oct 11 17:39:58 rsync cyrus/master[458]: process type:SERVICE
name:backupd path:/usr/lib/cyrus/bin/backupd age:201.603s pid:3969
signaled to death by signal 11 (Segmentation fault, core dumped)
Oct 11 17:39:58 rsync cyrus/master[458]: service backupd/ipv4 pid 3969
in BUSY state: terminated abnormally
Oct 11 17:39:58 rsync cyrus/master[458]: service backupd/ipv4 now has 0
ready workers

There, a core dump.? Here is what I get from a backtrace:

# coredumpctl gdb -1
?????????? PID: 3969 (backupd)
?????????? UID: 103 (cyrus)
?????????? GID: 8 (mail)
??????? Signal: 11 (SEGV)
???? Timestamp: Fri 2019-10-11 17:39:58 AEDT (2min 16s ago)
? Command Line: /usr/lib/cyrus/bin/backupd
??? Executable: /usr/lib/cyrus/bin/backupd
?Control Group: /system.slice/cyrus-imapd.service
????????? Unit: cyrus-imapd.service
???????? Slice: system.slice
?????? Boot ID: c887b7eb1d734962b8bddb745df21e8f
??? Machine ID: facebc4e2dcd47a68a097acc9077814e
????? Hostname: rsync
?????? Storage:
/var/lib/systemd/coredump/core.backupd.103.c887b7eb1d734962b8bddb745df21e8f.3969.1570775998000000.lz4
?????? Message: Process 3969 (backupd) of user 103 dumped core.
???????????????
??????????????? Stack trace of thread 3969:
??????????????? #0? 0x00007f95c17e3206 __GI___strlen_sse2 (libc.so.6)
??????????????? #1? 0x00007f95c2080119 xstrdup (libcyrus_min.so.0)
??????????????? #2? 0x00005557e89fe440 is_mailboxes_single_user (backupd)
??????????????? #3? 0x00005557e89eec88 main (backupd)
??????????????? #4? 0x00007f95c176f09b __libc_start_main (libc.so.6)
??????????????? #5? 0x00005557e89ef34a _start (backupd)

GNU gdb (Debian 8.2.1-2+b1) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
??? <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/cyrus/bin/backupd...Reading symbols from
/usr/lib/debug/.build-id/e3/b2619440ce57c6ae7db282266976a826059cf2.debug...done.
done.
[New LWP 3969]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/cyrus/bin/backupd'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0? __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
120??? ../sysdeps/x86_64/multiarch/../strlen.S: No such file or directory.
(gdb) bt
#0? __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
#1? 0x00007f95c2080119 in xstrdup (str=0x0) at lib/xmalloc.c:95
#2? 0x00005557e89fe440 in is_mailboxes_single_user (dl=0x5557e9472900)
at backup/backupd.c:1438
#3? cmd_get (dl=0x5557e9472900) at backup/backupd.c:1489
#4? cmdloop () at backup/backupd.c:688
#5? service_main (argc=<optimized out>, argv=<optimized out>,
envp=<optimized out>) at backup/backupd.c:282
#6? 0x00005557e89eec88 in main (argc=<optimized out>, argv=<optimized
out>, envp=0x7ffc01307b38)
??? at master/service.c:638
(gdb) up
#1? 0x00007f95c2080119 in xstrdup (str=0x0) at lib/xmalloc.c:95
95??? lib/xmalloc.c: No such file or directory.
(gdb) up
#2? 0x00005557e89fe440 in is_mailboxes_single_user (dl=0x5557e9472900)
at backup/backupd.c:1438
1438??? backup/backupd.c: No such file or directory.
(gdb) p *dl
$2 = {name = 0x5557e9415590 "MAILBOXES", head = 0x5557e947c0b0, tail =
0x5557e947c0b0, next = 0x0, type = 10,
? sval = 0x0, nval = 0, gval = 0x0, part = 0x0}
(gdb)?

> What about if you use "sync_client -n rsync -u support at polyfoam.com.au" instead (i.e. with -u treating the shared mailbox as a USER rather than as a -m MAILBOX)?

This doesn't crash, but it instead transfers a since-deleted full user support at polyfoam.com.au (from earlier dross in my database before I made support at polyfoam.com.au a public mailbox) to the backup server:

# /usr/lib/cyrus/bin/sync_client -v -v -n rsync -u support at polyfoam.com.au
cyrus/sync_client[201108]: couldn't authenticate to backend server: no
mechanism available
>1570776887>COMPRESS DEFLATE
<1570776887<OK DEFLATE active
USER support at polyfoam.com.au
>1570776887>GET USER support at polyfoam.com.au
<1570776887<* MAILBOX %(UNIQUEID jkt2bnhgvdgxbotfkmo7jz9j MBOXNAME
polyfoam.com.au!user.support MBOXTYPE NIL LAST_UID 3 HIGHESTMODSEQ 11
RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 1570530398 POP3_LAST_LOGIN 0
POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION default ACL
"support at polyfoam.com.au??? lrswipkxtecdan??? debbiep at polyfoam.com.au???
lrswipkxtecda??? " OPTIONS P SYNC_CRC 0 SYNC_CRC_ANNOT 0 QUOTAROOT NIL
XCONVMODSEQ 0 ANNOTATIONS (%(ENTRY /comment USERID "" VALUE "Support
(Polyfoam group)") %(ENTRY /comment USERID admdebbiep VALUE Support)))
* MAILBOX %(UNIQUEID 33oidc614u2i31hqutvkywu8 MBOXNAME
polyfoam.com.au!user.support.Archive MBOXTYPE NIL LAST_UID 0
HIGHESTMODSEQ 1 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0
POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION
default ACL "support at polyfoam.com.au??? lrswipkxtecdan??? " OPTIONS P
SYNC_CRC 0 SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0 ANNOTATIONS
(%(ENTRY /specialuse USERID support at polyfoam.com.au VALUE {8+}
\archive)))
* MAILBOX %(UNIQUEID 5wwsybv1ftur0vwaeeuvb0pk MBOXNAME
polyfoam.com.au!user.support.Drafts MBOXTYPE NIL LAST_UID 0
HIGHESTMODSEQ 2 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0
POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION
default ACL "support at polyfoam.com.au??? lrswipkxtecdan??? " OPTIONS P
SYNC_CRC 0 SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0 ANNOTATIONS
(%(ENTRY /specialuse USERID support at polyfoam.com.au VALUE {5+}
\sent)))
* MAILBOX %(UNIQUEID 7k6a536yhh9zc3t3j5f9z93u MBOXNAME
polyfoam.com.au!user.support.Important MBOXTYPE NIL LAST_UID 0
HIGHESTMODSEQ 1 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0
POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION
default ACL "support at polyfoam.com.au??? lrswipkxtecdan??? " OPTIONS P
SYNC_CRC 0 SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0 ANNOTATIONS
(%(ENTRY /specialuse USERID support at polyfoam.com.au VALUE {10+}
\important)))
* MAILBOX %(UNIQUEID zdl5xlyfx1wtcpp9gd3ce50l MBOXNAME
polyfoam.com.au!user.support.Junk MBOXTYPE NIL LAST_UID 0 HIGHESTMODSEQ
1 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0
POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION default ACL
"support at polyfoam.com.au??? lrswipkxtecdan??? " OPTIONS P SYNC_CRC 0
SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0 ANNOTATIONS (%(ENTRY
/specialuse USERID support at polyfoam.com.au VALUE {5+}
\junk)))
* MAILBOX %(UNIQUEID pi3ppdusz8pgio8lyrkj7tiw MBOXNAME
polyfoam.com.au!user.support.Sent MBOXTYPE NIL LAST_UID 0 HIGHESTMODSEQ
1 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0
POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION default ACL
"support at polyfoam.com.au??? lrswipkxtecdan??? " OPTIONS P SYNC_CRC 0
SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0)
* MAILBOX %(UNIQUEID a888coldtyz7dwop51uuqbot MBOXNAME
polyfoam.com.au!user.support.Trash MBOXTYPE NIL LAST_UID 0 HIGHESTMODSEQ
1 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0
POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION default ACL
"support at polyfoam.com.au??? lrswipkxtecdan??? " OPTIONS P SYNC_CRC 0
SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0 ANNOTATIONS (%(ENTRY
/specialuse USERID support at polyfoam.com.au VALUE {6+}
\trash)))
* LSUB (polyfoam.com.au!user.support
polyfoam.com.au!user.support.Archive polyfoam.com.au!user.support.Drafts
polyfoam.com.au!user.support.Junk polyfoam.com.au!user.support.Sent
polyfoam.com.au!user.support.Trash)
OK Success
cyrus/sync_client[201108]: Inbox missing on master for
support at polyfoam.com.au
UNUSER support at polyfoam.com.au
>1570776887>APPLY UNUSER support at polyfoam.com.au
<1570776887<OK Success
>1570776887>EXIT
<1570776887<OK Finished

But I think that's a red herring.? If I do it for a public mailbox that
has never had a matching user, I get this:

# /usr/lib/cyrus/bin/sync_client -v -v -n rsync -u info2 at polyfoam.com.au
cyrus/sync_client[201121]: couldn't authenticate to backend server: no
mechanism available
>1570777011>COMPRESS DEFLATE
<1570777011<OK DEFLATE active
USER info2 at polyfoam.com.au
>1570777011>GET USER info2 at polyfoam.com.au
<1570777011<OK Success

and running with the -m option on info2 produces the same segfault as above.

> On the backup server, what does the "ctl_backups verify -vvv -m polyfoam.com.au!support" command say about the shared mailbox?  

It segfaults, not the same as above.

# /usr/lib/cyrus/bin/ctl_backups verify -vvv -m 'polyfoam.com.au!support'
Segmentation fault (core dumped)
# coredumpctl gdb -1
?????????? PID: 3927 (ctl_backups)
?????????? UID: 103 (cyrus)
?????????? GID: 8 (mail)
??????? Signal: 11 (SEGV)
???? Timestamp: Fri 2019-10-11 17:26:10 AEDT (50s ago)
? Command Line: /usr/lib/cyrus/bin/ctl_backups verify -vvv -m
polyfoam.com.au!support
??? Executable: /usr/lib/cyrus/bin/ctl_backups
?Control Group: /user.slice/user-1000.slice/session-1.scope
????????? Unit: session-1.scope
???????? Slice: user-1000.slice
?????? Session: 1
???? Owner UID: 1000 (localadmin)
?????? Boot ID: c887b7eb1d734962b8bddb745df21e8f
??? Machine ID: facebc4e2dcd47a68a097acc9077814e
????? Hostname: rsync
?????? Storage:
/var/lib/systemd/coredump/core.ctl_backups.103.c887b7eb1d734962b8bddb745df21e8f.3927.1570775170000000.lz4
?????? Message: Process 3927 (ctl_backups) of user 103 dumped core.
???????????????
??????????????? Stack trace of thread 3927:
??????????????? #0? 0x00007fc24cfdb206 __GI___strlen_sse2 (libc.so.6)
??????????????? #1? 0x0000557f0137c495 backup_get_paths (ctl_backups)
??????????????? #2? 0x0000557f0136d678 main (ctl_backups)
??????????????? #3? 0x00007fc24cf6709b __libc_start_main (libc.so.6)
??????????????? #4? 0x0000557f0136db6a _start (ctl_backups)

GNU gdb (Debian 8.2.1-2+b1) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
??? <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/cyrus/bin/ctl_backups...Reading symbols
from
/usr/lib/debug/.build-id/11/23ce6d2f413e1384c144165f996813ad4924c0.debug...done.
done.
[New LWP 3927]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/cyrus/bin/ctl_backups verify -vvv -m
polyfoam.com.au!support'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0? __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
120??? ../sysdeps/x86_64/multiarch/../strlen.S: No such file or directory.
(gdb) bt
#0? __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
#1? 0x0000557f0137c495 in backup_get_paths (mbname=0x557f0300bfd0,
data_fname=0x7ffcbe2daf90, index_fname=0x0,
??? create=BACKUP_OPEN_NOCREATE) at backup/lcb.c:373
#2? 0x0000557f0136d678 in main (argc=5, argv=0x7ffcbe2db108) at
backup/ctl_backups.c:525
(gdb) up
#1? 0x0000557f0137c495 in backup_get_paths (mbname=0x557f0300bfd0,
data_fname=0x7ffcbe2daf90, index_fname=0x0,
??? create=BACKUP_OPEN_NOCREATE) at backup/lcb.c:373
373??? backup/lcb.c: No such file or directory.
(gdb) p *mbname
$2 = {boxes = 0x557f0300c090, is_deleted = 0, localpart = 0x0, domain =
0x557f0300c050 "polyfoam.com.au",
? extns = 0x0, extuserid = 0x0, userid = 0x0, intname = 0x557f0300c030
"polyfoam.com.au!support", extname = 0x0,
? recipient = 0x0}
(gdb) p userid
$4 = 0x0
(gdb)

Calling strlen() on the null userid would be the immediate cause of the
crash.? I'm not familiar enough with the code to know what leads to
userid being null, or if that's also the cause of backupd crashing.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20191011/ae17d03e/attachment-0001.html>

From ellie at fastmail.com  Sun Oct 13 19:29:17 2019
From: ellie at fastmail.com (ellie timoney)
Date: Mon, 14 Oct 2019 10:29:17 +1100
Subject: XBACKUP and backupd not backing up public folders (3.0.8)
In-Reply-To: <d9f4f951-baaf-68e1-60d1-be2768621349@polyfoam.com.au>
References: <a04f8634-f9b3-2167-a36d-5130c6bc69a8@polyfoam.com.au>
 <90b6d2f0-924d-4530-93fd-3eeaf95b0e18@www.fastmail.com>
 <d9f4f951-baaf-68e1-60d1-be2768621349@polyfoam.com.au>
Message-ID: <d0e5c248-17e9-4677-8c45-47cb91d1d22e@www.fastmail.com>

Hi Deborah,

Thanks, that's all useful! Looks like in both places it's struggling with lack of a userid, which makes some sense because it's a shared mailbox, and shared mailboxes don't have userids.

I guess this means that in its current state, the backup system can't handle shared mailboxes. :( Which is weird, because the docs say it can, and I wouldn't have written that if I didn't think it worked, but maybe I didn't understand shared mailboxes as well as I thought way back then.

I've created a github issue (https://github.com/cyrusimap/cyrus-imapd/issues/2893), and am about to make a test case to reproduce the problem, so I can get on with fixing it. :)

Cheers,

ellie

On Fri, Oct 11, 2019, at 6:02 PM, Deborah Pickett wrote:
> Hi Ellie,

> Thanks for helping me look at this.

> On 2019-10-09 16:17, ellie timoney wrote:
>> Does the same problem occur if you use sync_client (on the master server, as the cyrus user) to replicate the shared mailbox to the backup server (rather than using XBACKUP over IMAP)?  Something like "sync_client -n rsync  -m support at polyfoam.com.au" I think? 
>> 
> With -m I get this familiar output on the master:


> # /usr/lib/cyrus/bin/sync_client -v -n rsync -m support at polyfoam.com.au
>  MAILBOXES polyfoam.com.au!support
>  Error from sync_do_mailboxes(): bailing out!

> and this is seen in the log on the backup server:


> Oct 11 17:39:58 rsync cyrus/backupd[3969]: login: mail-3175-1.polyfoam.com.au [10.3.244.125] rsync-mail-3175-1 DIGEST-MD5 User logged in
>  Oct 11 17:39:58 rsync cyrus/backupd[3969]: created decompress buffer of 4102 bytes
>  Oct 11 17:39:58 rsync cyrus/backupd[3969]: created compress buffer of 4073 bytes
>  Oct 11 17:39:58 rsync cyrus/backupd[3969]: decompressed 47 -> 41 bytes
>  Oct 11 17:39:58 rsync cyrus/master[458]: process type:SERVICE name:backupd path:/usr/lib/cyrus/bin/backupd age:201.603s pid:3969 signaled to death by signal 11 (Segmentation fault, core dumped)
>  Oct 11 17:39:58 rsync cyrus/master[458]: service backupd/ipv4 pid 3969 in BUSY state: terminated abnormally
>  Oct 11 17:39:58 rsync cyrus/master[458]: service backupd/ipv4 now has 0 ready workers

> There, a core dump. Here is what I get from a backtrace:


> # coredumpctl gdb -1 
>  PID: 3969 (backupd)
>  UID: 103 (cyrus)
>  GID: 8 (mail)
>  Signal: 11 (SEGV)
>  Timestamp: Fri 2019-10-11 17:39:58 AEDT (2min 16s ago)
>  Command Line: /usr/lib/cyrus/bin/backupd
>  Executable: /usr/lib/cyrus/bin/backupd
>  Control Group: /system.slice/cyrus-imapd.service
>  Unit: cyrus-imapd.service
>  Slice: system.slice
>  Boot ID: c887b7eb1d734962b8bddb745df21e8f
>  Machine ID: facebc4e2dcd47a68a097acc9077814e
>  Hostname: rsync
>  Storage: /var/lib/systemd/coredump/core.backupd.103.c887b7eb1d734962b8bddb745df21e8f.3969.1570775998000000.lz4
>  Message: Process 3969 (backupd) of user 103 dumped core.
> 
>  Stack trace of thread 3969:
>  #0 0x00007f95c17e3206 __GI___strlen_sse2 (libc.so.6)
>  #1 0x00007f95c2080119 xstrdup (libcyrus_min.so.0)
>  #2 0x00005557e89fe440 is_mailboxes_single_user (backupd)
>  #3 0x00005557e89eec88 main (backupd)
>  #4 0x00007f95c176f09b __libc_start_main (libc.so.6)
>  #5 0x00005557e89ef34a _start (backupd)
> 
>  GNU gdb (Debian 8.2.1-2+b1) 8.2.1
>  Copyright (C) 2018 Free Software Foundation, Inc.
>  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>  This is free software: you are free to change and redistribute it.
>  There is NO WARRANTY, to the extent permitted by law.
>  Type "show copying" and "show warranty" for details.
>  This GDB was configured as "x86_64-linux-gnu".
>  Type "show configuration" for configuration details.
>  For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
>  Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> 
>  For help, type "help".
>  Type "apropos word" to search for commands related to "word"...
>  Reading symbols from /usr/lib/cyrus/bin/backupd...Reading symbols from /usr/lib/debug/.build-id/e3/b2619440ce57c6ae7db282266976a826059cf2.debug...done.
>  done.
>  [New LWP 3969]
>  [Thread debugging using libthread_db enabled]
>  Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>  Core was generated by `/usr/lib/cyrus/bin/backupd'.
>  Program terminated with signal SIGSEGV, Segmentation fault.
>  #0 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
>  120 ../sysdeps/x86_64/multiarch/../strlen.S: No such file or directory.
>  (gdb) bt
>  #0 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
>  #1 0x00007f95c2080119 in xstrdup (str=0x0) at lib/xmalloc.c:95
>  #2 0x00005557e89fe440 in is_mailboxes_single_user (dl=0x5557e9472900) at backup/backupd.c:1438
>  #3 cmd_get (dl=0x5557e9472900) at backup/backupd.c:1489
>  #4 cmdloop () at backup/backupd.c:688
>  #5 service_main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at backup/backupd.c:282
>  #6 0x00005557e89eec88 in main (argc=<optimized out>, argv=<optimized out>, envp=0x7ffc01307b38)
>  at master/service.c:638
>  (gdb) up
>  #1 0x00007f95c2080119 in xstrdup (str=0x0) at lib/xmalloc.c:95
>  95 lib/xmalloc.c: No such file or directory.
>  (gdb) up
>  #2 0x00005557e89fe440 in is_mailboxes_single_user (dl=0x5557e9472900) at backup/backupd.c:1438
>  1438 backup/backupd.c: No such file or directory.
>  (gdb) p *dl
>  $2 = {name = 0x5557e9415590 "MAILBOXES", head = 0x5557e947c0b0, tail = 0x5557e947c0b0, next = 0x0, type = 10, 
>  sval = 0x0, nval = 0, gval = 0x0, part = 0x0}
>  (gdb) 

> > What about if you use "sync_client -n rsync -u support at polyfoam.com.au" instead (i.e. with -u treating the shared mailbox as a USER rather than as a -m MAILBOX)?

This doesn't crash, but it instead transfers a since-deleted full user support at polyfoam.com.au (from earlier dross in my database before I made support at polyfoam.com.au a public mailbox) to the backup server:
> 

> # /usr/lib/cyrus/bin/sync_client -v -v -n rsync -u support at polyfoam.com.au
>  cyrus/sync_client[201108]: couldn't authenticate to backend server: no mechanism available
>  >1570776887>COMPRESS DEFLATE
>  <1570776887<OK DEFLATE active
>  USER support at polyfoam.com.au
>  >1570776887>GET USER support at polyfoam.com.au
>  <1570776887<* MAILBOX %(UNIQUEID jkt2bnhgvdgxbotfkmo7jz9j MBOXNAME polyfoam.com.au!user.support MBOXTYPE NIL LAST_UID 3 HIGHESTMODSEQ 11 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 1570530398 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION default ACL "support at polyfoam.com.au lrswipkxtecdan debbiep at polyfoam.com.au lrswipkxtecda " <mailto:support at polyfoam.com.au%C2%A0%C2%A0%C2%A0lrswipkxtecdan%C2%A0%C2%A0%C2%A0debbiep at polyfoam.com.au%C2%A0%C2%A0%C2%A0lrswipkxtecda> OPTIONS P SYNC_CRC 0 SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0 ANNOTATIONS (%(ENTRY /comment USERID "" VALUE "Support (Polyfoam group)") %(ENTRY /comment USERID admdebbiep VALUE Support)))
>  * MAILBOX %(UNIQUEID 33oidc614u2i31hqutvkywu8 MBOXNAME polyfoam.com.au!user.support.Archive MBOXTYPE NIL LAST_UID 0 HIGHESTMODSEQ 1 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION default ACL "support at polyfoam.com.au lrswipkxtecdan " <mailto:support at polyfoam.com.au%C2%A0%C2%A0%C2%A0lrswipkxtecdan> OPTIONS P SYNC_CRC 0 SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0 ANNOTATIONS (%(ENTRY /specialuse USERID support at polyfoam.com.au VALUE {8+}
>  \archive)))
>  * MAILBOX %(UNIQUEID 5wwsybv1ftur0vwaeeuvb0pk MBOXNAME polyfoam.com.au!user.support.Drafts MBOXTYPE NIL LAST_UID 0 HIGHESTMODSEQ 2 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION default ACL "support at polyfoam.com.au lrswipkxtecdan " <mailto:support at polyfoam.com.au%C2%A0%C2%A0%C2%A0lrswipkxtecdan> OPTIONS P SYNC_CRC 0 SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0 ANNOTATIONS (%(ENTRY /specialuse USERID support at polyfoam.com.au VALUE {5+}
>  \sent)))
>  * MAILBOX %(UNIQUEID 7k6a536yhh9zc3t3j5f9z93u MBOXNAME polyfoam.com.au!user.support.Important MBOXTYPE NIL LAST_UID 0 HIGHESTMODSEQ 1 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION default ACL "support at polyfoam.com.au lrswipkxtecdan " <mailto:support at polyfoam.com.au%C2%A0%C2%A0%C2%A0lrswipkxtecdan> OPTIONS P SYNC_CRC 0 SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0 ANNOTATIONS (%(ENTRY /specialuse USERID support at polyfoam.com.au VALUE {10+}
>  \important)))
>  * MAILBOX %(UNIQUEID zdl5xlyfx1wtcpp9gd3ce50l MBOXNAME polyfoam.com.au!user.support.Junk MBOXTYPE NIL LAST_UID 0 HIGHESTMODSEQ 1 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION default ACL "support at polyfoam.com.au lrswipkxtecdan " <mailto:support at polyfoam.com.au%C2%A0%C2%A0%C2%A0lrswipkxtecdan> OPTIONS P SYNC_CRC 0 SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0 ANNOTATIONS (%(ENTRY /specialuse USERID support at polyfoam.com.au VALUE {5+}
>  \junk)))
>  * MAILBOX %(UNIQUEID pi3ppdusz8pgio8lyrkj7tiw MBOXNAME polyfoam.com.au!user.support.Sent MBOXTYPE NIL LAST_UID 0 HIGHESTMODSEQ 1 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION default ACL "support at polyfoam.com.au lrswipkxtecdan " <mailto:support at polyfoam.com.au%C2%A0%C2%A0%C2%A0lrswipkxtecdan> OPTIONS P SYNC_CRC 0 SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0)
>  * MAILBOX %(UNIQUEID a888coldtyz7dwop51uuqbot MBOXNAME polyfoam.com.au!user.support.Trash MBOXTYPE NIL LAST_UID 0 HIGHESTMODSEQ 1 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION default ACL "support at polyfoam.com.au lrswipkxtecdan " <mailto:support at polyfoam.com.au%C2%A0%C2%A0%C2%A0lrswipkxtecdan> OPTIONS P SYNC_CRC 0 SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0 ANNOTATIONS (%(ENTRY /specialuse USERID support at polyfoam.com.au VALUE {6+}
>  \trash)))
>  * LSUB (polyfoam.com.au!user.support polyfoam.com.au!user.support.Archive polyfoam.com.au!user.support.Drafts polyfoam.com.au!user.support.Junk polyfoam.com.au!user.support.Sent polyfoam.com.au!user.support.Trash)
>  OK Success
>  cyrus/sync_client[201108]: Inbox missing on master for support at polyfoam.com.au
>  UNUSER support at polyfoam.com.au
>  >1570776887>APPLY UNUSER support at polyfoam.com.au
>  <1570776887<OK Success
>  >1570776887>EXIT
>  <1570776887<OK Finished

> But I think that's a red herring. If I do it for a public mailbox that has never had a matching user, I get this:


> # /usr/lib/cyrus/bin/sync_client -v -v -n rsync -u info2 at polyfoam.com.au
>  cyrus/sync_client[201121]: couldn't authenticate to backend server: no mechanism available
>  >1570777011>COMPRESS DEFLATE
>  <1570777011<OK DEFLATE active
>  USER info2 at polyfoam.com.au
>  >1570777011>GET USER info2 at polyfoam.com.au
>  <1570777011<OK Success

> and running with the -m option on info2 produces the same segfault as above.

> 

>> On the backup server, what does the "ctl_backups verify -vvv -m polyfoam.com.au!support" command say about the shared mailbox?  
> It segfaults, not the same as above.


> # /usr/lib/cyrus/bin/ctl_backups verify -vvv -m 'polyfoam.com.au!support'
>  Segmentation fault (core dumped)
>  # coredumpctl gdb -1 
>  PID: 3927 (ctl_backups)
>  UID: 103 (cyrus)
>  GID: 8 (mail)
>  Signal: 11 (SEGV)
>  Timestamp: Fri 2019-10-11 17:26:10 AEDT (50s ago)
>  Command Line: /usr/lib/cyrus/bin/ctl_backups verify -vvv -m polyfoam.com.au!support
>  Executable: /usr/lib/cyrus/bin/ctl_backups
>  Control Group: /user.slice/user-1000.slice/session-1.scope
>  Unit: session-1.scope
>  Slice: user-1000.slice
>  Session: 1
>  Owner UID: 1000 (localadmin)
>  Boot ID: c887b7eb1d734962b8bddb745df21e8f
>  Machine ID: facebc4e2dcd47a68a097acc9077814e
>  Hostname: rsync
>  Storage: /var/lib/systemd/coredump/core.ctl_backups.103.c887b7eb1d734962b8bddb745df21e8f.3927.1570775170000000.lz4
>  Message: Process 3927 (ctl_backups) of user 103 dumped core.
> 
>  Stack trace of thread 3927:
>  #0 0x00007fc24cfdb206 __GI___strlen_sse2 (libc.so.6)
>  #1 0x0000557f0137c495 backup_get_paths (ctl_backups)
>  #2 0x0000557f0136d678 main (ctl_backups)
>  #3 0x00007fc24cf6709b __libc_start_main (libc.so.6)
>  #4 0x0000557f0136db6a _start (ctl_backups)
> 
>  GNU gdb (Debian 8.2.1-2+b1) 8.2.1
>  Copyright (C) 2018 Free Software Foundation, Inc.
>  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>  This is free software: you are free to change and redistribute it.
>  There is NO WARRANTY, to the extent permitted by law.
>  Type "show copying" and "show warranty" for details.
>  This GDB was configured as "x86_64-linux-gnu".
>  Type "show configuration" for configuration details.
>  For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
>  Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> 
>  For help, type "help".
>  Type "apropos word" to search for commands related to "word"...
>  Reading symbols from /usr/lib/cyrus/bin/ctl_backups...Reading symbols from /usr/lib/debug/.build-id/11/23ce6d2f413e1384c144165f996813ad4924c0.debug...done.
>  done.
>  [New LWP 3927]
>  [Thread debugging using libthread_db enabled]
>  Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>  Core was generated by `/usr/lib/cyrus/bin/ctl_backups verify -vvv -m polyfoam.com.au!support'.
>  Program terminated with signal SIGSEGV, Segmentation fault.
>  #0 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
>  120 ../sysdeps/x86_64/multiarch/../strlen.S: No such file or directory.
>  (gdb) bt
>  #0 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
>  #1 0x0000557f0137c495 in backup_get_paths (mbname=0x557f0300bfd0, data_fname=0x7ffcbe2daf90, index_fname=0x0, 
>  create=BACKUP_OPEN_NOCREATE) at backup/lcb.c:373
>  #2 0x0000557f0136d678 in main (argc=5, argv=0x7ffcbe2db108) at backup/ctl_backups.c:525
>  (gdb) up
>  #1 0x0000557f0137c495 in backup_get_paths (mbname=0x557f0300bfd0, data_fname=0x7ffcbe2daf90, index_fname=0x0, 
>  create=BACKUP_OPEN_NOCREATE) at backup/lcb.c:373
>  373 backup/lcb.c: No such file or directory.
>  (gdb) p *mbname
>  $2 = {boxes = 0x557f0300c090, is_deleted = 0, localpart = 0x0, domain = 0x557f0300c050 "polyfoam.com.au", 
>  extns = 0x0, extuserid = 0x0, userid = 0x0, intname = 0x557f0300c030 "polyfoam.com.au!support", extname = 0x0, 
>  recipient = 0x0}
>  (gdb) p userid
>  $4 = 0x0
>  (gdb) 

> Calling strlen() on the null userid would be the immediate cause of the crash. I'm not familiar enough with the code to know what leads to userid being null, or if that's also the cause of backupd crashing.

> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20191014/695f4f1f/attachment-0001.html>

From debbiep at polyfoam.com.au  Sun Oct 13 21:00:06 2019
From: debbiep at polyfoam.com.au (Deborah Pickett)
Date: Mon, 14 Oct 2019 12:00:06 +1100
Subject: XBACKUP and backupd not backing up public folders (3.0.8)
In-Reply-To: <d0e5c248-17e9-4677-8c45-47cb91d1d22e@www.fastmail.com>
References: <a04f8634-f9b3-2167-a36d-5130c6bc69a8@polyfoam.com.au>
 <90b6d2f0-924d-4530-93fd-3eeaf95b0e18@www.fastmail.com>
 <d9f4f951-baaf-68e1-60d1-be2768621349@polyfoam.com.au>
 <d0e5c248-17e9-4677-8c45-47cb91d1d22e@www.fastmail.com>
Message-ID: <013b01d5822a$b7f51ff0$27df5fd0$@polyfoam.com.au>

Thanks for confirming my suspicions, Ellie.

 

So how are people currently backing up shared folders, if they?re not using Cyrus backupd? I looked at replication, but that doesn?t seem to help me to reach the point where I can dump a coherent filesystem structure to tape or NAS or some other external storage.

 

Deborah Pickett

System Administrator

Polyfoam Australia Pty Ltd

T: +61 (3) 9794 8320 | F: +61 (3) 9791 1222 | M: +61 408 962 109

E:  <mailto:debbiep at polyfoam.com.au> debbiep at polyfoam.com.au | W:  <http://www.polyfoam.com.au/> www.polyfoam.com.au

Proudly Australian owned and operated for over 30 years

 

From: ellie timoney <ellie at fastmail.com> 
Sent: Monday, 14 October 2019 10:29
To: Deborah Pickett <debbiep at polyfoam.com.au>; info-cyrus at lists.andrew.cmu.edu
Subject: Re: XBACKUP and backupd not backing up public folders (3.0.8)

 

Hi Deborah,

 

Thanks, that's all useful!  Looks like in both places it's struggling with lack of a userid, which makes some sense because it's a shared mailbox, and shared mailboxes don't have userids.

 

I guess this means that in its current state, the backup system can't handle shared mailboxes. :(  Which is weird, because the docs say it can, and I wouldn't have written that if I didn't think it worked, but maybe I didn't understand shared mailboxes as well as I thought way back then.

 

I've created a github issue (https://github.com/cyrusimap/cyrus-imapd/issues/2893), and am about to make a test case to reproduce the problem, so I can get on with fixing it. :)

 

Cheers,

 

ellie

 

On Fri, Oct 11, 2019, at 6:02 PM, Deborah Pickett wrote:

Hi Ellie,

Thanks for helping me look at this.

On 2019-10-09 16:17, ellie timoney wrote:

Does the same problem occur if you use sync_client (on the master server, as the cyrus user) to replicate the shared mailbox to the backup server (rather than using XBACKUP over IMAP)?  Something like "sync_client -n rsync  -m  <mailto:support at polyfoam.com.au> support at polyfoam.com.au" I think? 
 

With -m I get this familiar output on the master:

# /usr/lib/cyrus/bin/sync_client -v -n rsync -m  <mailto:support at polyfoam.com.au> support at polyfoam.com.au

MAILBOXES polyfoam.com.au!support

Error from sync_do_mailboxes(): bailing out!

and this is seen in the log on the backup server:

Oct 11 17:39:58 rsync cyrus/backupd[3969]: login: mail-3175-1.polyfoam.com.au [10.3.244.125] rsync-mail-3175-1 DIGEST-MD5 User logged in

Oct 11 17:39:58 rsync cyrus/backupd[3969]: created decompress buffer of 4102 bytes

Oct 11 17:39:58 rsync cyrus/backupd[3969]: created compress buffer of 4073 bytes

Oct 11 17:39:58 rsync cyrus/backupd[3969]: decompressed 47 -> 41 bytes

Oct 11 17:39:58 rsync cyrus/master[458]: process type:SERVICE name:backupd path:/usr/lib/cyrus/bin/backupd age:201.603s pid:3969 signaled to death by signal 11 (Segmentation fault, core dumped)

Oct 11 17:39:58 rsync cyrus/master[458]: service backupd/ipv4 pid 3969 in BUSY state: terminated abnormally

Oct 11 17:39:58 rsync cyrus/master[458]: service backupd/ipv4 now has 0 ready workers

There, a core dump.  Here is what I get from a backtrace:

# coredumpctl gdb -1 

           PID: 3969 (backupd)

           UID: 103 (cyrus)

           GID: 8 (mail)

        Signal: 11 (SEGV)

     Timestamp: Fri 2019-10-11 17:39:58 AEDT (2min 16s ago)

  Command Line: /usr/lib/cyrus/bin/backupd

    Executable: /usr/lib/cyrus/bin/backupd

 Control Group: /system.slice/cyrus-imapd.service

          Unit: cyrus-imapd.service

         Slice: system.slice

       Boot ID: c887b7eb1d734962b8bddb745df21e8f

    Machine ID: facebc4e2dcd47a68a097acc9077814e

      Hostname: rsync

       Storage: /var/lib/systemd/coredump/core.backupd.103.c887b7eb1d734962b8bddb745df21e8f.3969.1570775998000000.lz4

       Message: Process 3969 (backupd) of user 103 dumped core.

                

                Stack trace of thread 3969:

                #0  0x00007f95c17e3206 __GI___strlen_sse2 (libc.so.6)

                #1  0x00007f95c2080119 xstrdup (libcyrus_min.so.0)

                #2  0x00005557e89fe440 is_mailboxes_single_user (backupd)

                #3  0x00005557e89eec88 main (backupd)

                #4  0x00007f95c176f09b __libc_start_main (libc.so.6)

                #5  0x00005557e89ef34a _start (backupd)

 

GNU gdb (Debian 8.2.1-2+b1) 8.2.1

Copyright (C) 2018 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later  <http://gnu.org/licenses/gpl.html> <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.

Type "show copying" and "show warranty" for details.

This GDB was configured as "x86_64-linux-gnu".

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

 <http://www.gnu.org/software/gdb/bugs/> <http://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at:

     <http://www.gnu.org/software/gdb/documentation/> <http://www.gnu.org/software/gdb/documentation/>.

 

For help, type "help".

Type "apropos word" to search for commands related to "word"...

Reading symbols from /usr/lib/cyrus/bin/backupd...Reading symbols from /usr/lib/debug/.build-id/e3/b2619440ce57c6ae7db282266976a826059cf2.debug...done.

done.

[New LWP 3969]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Core was generated by `/usr/lib/cyrus/bin/backupd'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120

120    ../sysdeps/x86_64/multiarch/../strlen.S: No such file or directory.

(gdb) bt

#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120

#1  0x00007f95c2080119 in xstrdup (str=0x0) at lib/xmalloc.c:95

#2  0x00005557e89fe440 in is_mailboxes_single_user (dl=0x5557e9472900) at backup/backupd.c:1438

#3  cmd_get (dl=0x5557e9472900) at backup/backupd.c:1489

#4  cmdloop () at backup/backupd.c:688

#5  service_main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at backup/backupd.c:282

#6  0x00005557e89eec88 in main (argc=<optimized out>, argv=<optimized out>, envp=0x7ffc01307b38)

    at master/service.c:638

(gdb) up

#1  0x00007f95c2080119 in xstrdup (str=0x0) at lib/xmalloc.c:95

95    lib/xmalloc.c: No such file or directory.

(gdb) up

#2  0x00005557e89fe440 in is_mailboxes_single_user (dl=0x5557e9472900) at backup/backupd.c:1438

1438    backup/backupd.c: No such file or directory.

(gdb) p *dl

$2 = {name = 0x5557e9415590 "MAILBOXES", head = 0x5557e947c0b0, tail = 0x5557e947c0b0, next = 0x0, type = 10, 

  sval = 0x0, nval = 0, gval = 0x0, part = 0x0}

(gdb) 

> What about if you use "sync_client -n rsync -u  <mailto:support at polyfoam.com.au> support at polyfoam.com.au" instead (i.e. with -u treating the shared mailbox as a USER rather than as a -m MAILBOX)?
 
This doesn't crash, but it instead transfers a since-deleted full user  <mailto:support at polyfoam.com.au> support at polyfoam.com.au (from earlier dross in my database before I made  <mailto:support at polyfoam.com.au> support at polyfoam.com.au a public mailbox) to the backup server:
 

# /usr/lib/cyrus/bin/sync_client -v -v -n rsync -u  <mailto:support at polyfoam.com.au> support at polyfoam.com.au

cyrus/sync_client[201108]: couldn't authenticate to backend server: no mechanism available

>1570776887>COMPRESS DEFLATE

<1570776887<OK DEFLATE active

USER  <mailto:support at polyfoam.com.au> support at polyfoam.com.au

>1570776887>GET USER  <mailto:support at polyfoam.com.au> support at polyfoam.com.au

<1570776887<* MAILBOX %(UNIQUEID jkt2bnhgvdgxbotfkmo7jz9j MBOXNAME polyfoam.com.au!user.support MBOXTYPE NIL LAST_UID 3 HIGHESTMODSEQ 11 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 1570530398 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION default ACL  <mailto:support at polyfoam.com.au   lrswipkxtecdan   debbiep at polyfoam.com.au   lrswipkxtecda> "support at polyfoam.com.au    lrswipkxtecdan    debbiep at polyfoam.com.au    lrswipkxtecda    " OPTIONS P SYNC_CRC 0 SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0 ANNOTATIONS (%(ENTRY /comment USERID "" VALUE "Support (Polyfoam group)") %(ENTRY /comment USERID admdebbiep VALUE Support)))

* MAILBOX %(UNIQUEID 33oidc614u2i31hqutvkywu8 MBOXNAME polyfoam.com.au!user.support.Archive MBOXTYPE NIL LAST_UID 0 HIGHESTMODSEQ 1 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION default ACL  <mailto:support at polyfoam.com.au   lrswipkxtecdan> "support at polyfoam.com.au    lrswipkxtecdan    " OPTIONS P SYNC_CRC 0 SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0 ANNOTATIONS (%(ENTRY /specialuse USERID  <mailto:support at polyfoam.com.au> support at polyfoam.com.au VALUE {8+}

\archive)))

* MAILBOX %(UNIQUEID 5wwsybv1ftur0vwaeeuvb0pk MBOXNAME polyfoam.com.au!user.support.Drafts MBOXTYPE NIL LAST_UID 0 HIGHESTMODSEQ 2 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION default ACL  <mailto:support at polyfoam.com.au   lrswipkxtecdan> "support at polyfoam.com.au    lrswipkxtecdan    " OPTIONS P SYNC_CRC 0 SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0 ANNOTATIONS (%(ENTRY /specialuse USERID  <mailto:support at polyfoam.com.au> support at polyfoam.com.au VALUE {5+}

\sent)))

* MAILBOX %(UNIQUEID 7k6a536yhh9zc3t3j5f9z93u MBOXNAME polyfoam.com.au!user.support.Important MBOXTYPE NIL LAST_UID 0 HIGHESTMODSEQ 1 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION default ACL  <mailto:support at polyfoam.com.au   lrswipkxtecdan> "support at polyfoam.com.au    lrswipkxtecdan    " OPTIONS P SYNC_CRC 0 SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0 ANNOTATIONS (%(ENTRY /specialuse USERID  <mailto:support at polyfoam.com.au> support at polyfoam.com.au VALUE {10+}

\important)))

* MAILBOX %(UNIQUEID zdl5xlyfx1wtcpp9gd3ce50l MBOXNAME polyfoam.com.au!user.support.Junk MBOXTYPE NIL LAST_UID 0 HIGHESTMODSEQ 1 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION default ACL  <mailto:support at polyfoam.com.au   lrswipkxtecdan> "support at polyfoam.com.au    lrswipkxtecdan    " OPTIONS P SYNC_CRC 0 SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0 ANNOTATIONS (%(ENTRY /specialuse USERID  <mailto:support at polyfoam.com.au> support at polyfoam.com.au VALUE {5+}

\junk)))

* MAILBOX %(UNIQUEID pi3ppdusz8pgio8lyrkj7tiw MBOXNAME polyfoam.com.au!user.support.Sent MBOXTYPE NIL LAST_UID 0 HIGHESTMODSEQ 1 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION default ACL  <mailto:support at polyfoam.com.au   lrswipkxtecdan> "support at polyfoam.com.au    lrswipkxtecdan    " OPTIONS P SYNC_CRC 0 SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0)

* MAILBOX %(UNIQUEID a888coldtyz7dwop51uuqbot MBOXNAME polyfoam.com.au!user.support.Trash MBOXTYPE NIL LAST_UID 0 HIGHESTMODSEQ 1 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1570528039 PARTITION default ACL  <mailto:support at polyfoam.com.au   lrswipkxtecdan> "support at polyfoam.com.au    lrswipkxtecdan    " OPTIONS P SYNC_CRC 0 SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0 ANNOTATIONS (%(ENTRY /specialuse USERID  <mailto:support at polyfoam.com.au> support at polyfoam.com.au VALUE {6+}

\trash)))

* LSUB (polyfoam.com.au!user.support polyfoam.com.au!user.support.Archive polyfoam.com.au!user.support.Drafts polyfoam.com.au!user.support.Junk polyfoam.com.au!user.support.Sent polyfoam.com.au!user.support.Trash)

OK Success

cyrus/sync_client[201108]: Inbox missing on master for  <mailto:support at polyfoam.com.au> support at polyfoam.com.au

UNUSER  <mailto:support at polyfoam.com.au> support at polyfoam.com.au

>1570776887>APPLY UNUSER  <mailto:support at polyfoam.com.au> support at polyfoam.com.au

<1570776887<OK Success

>1570776887>EXIT

<1570776887<OK Finished

But I think that's a red herring.  If I do it for a public mailbox that has never had a matching user, I get this:

# /usr/lib/cyrus/bin/sync_client -v -v -n rsync -u  <mailto:info2 at polyfoam.com.au> info2 at polyfoam.com.au

cyrus/sync_client[201121]: couldn't authenticate to backend server: no mechanism available

>1570777011>COMPRESS DEFLATE

<1570777011<OK DEFLATE active

USER  <mailto:info2 at polyfoam.com.au> info2 at polyfoam.com.au

>1570777011>GET USER  <mailto:info2 at polyfoam.com.au> info2 at polyfoam.com.au

<1570777011<OK Success

and running with the -m option on info2 produces the same segfault as above.

 

On the backup server, what does the "ctl_backups verify -vvv -m polyfoam.com.au!support" command say about the shared mailbox?  

It segfaults, not the same as above.

# /usr/lib/cyrus/bin/ctl_backups verify -vvv -m 'polyfoam.com.au!support'

Segmentation fault (core dumped)

# coredumpctl gdb -1 

           PID: 3927 (ctl_backups)

           UID: 103 (cyrus)

           GID: 8 (mail)

        Signal: 11 (SEGV)

     Timestamp: Fri 2019-10-11 17:26:10 AEDT (50s ago)

  Command Line: /usr/lib/cyrus/bin/ctl_backups verify -vvv -m polyfoam.com.au!support

    Executable: /usr/lib/cyrus/bin/ctl_backups

 Control Group: /user.slice/user-1000.slice/session-1.scope

          Unit: session-1.scope

         Slice: user-1000.slice

       Session: 1

     Owner UID: 1000 (localadmin)

       Boot ID: c887b7eb1d734962b8bddb745df21e8f

    Machine ID: facebc4e2dcd47a68a097acc9077814e

      Hostname: rsync

       Storage: /var/lib/systemd/coredump/core.ctl_backups.103.c887b7eb1d734962b8bddb745df21e8f.3927.1570775170000000.lz4

       Message: Process 3927 (ctl_backups) of user 103 dumped core.

                

                Stack trace of thread 3927:

                #0  0x00007fc24cfdb206 __GI___strlen_sse2 (libc.so.6)

                #1  0x0000557f0137c495 backup_get_paths (ctl_backups)

                #2  0x0000557f0136d678 main (ctl_backups)

                #3  0x00007fc24cf6709b __libc_start_main (libc.so.6)

                #4  0x0000557f0136db6a _start (ctl_backups)

 

GNU gdb (Debian 8.2.1-2+b1) 8.2.1

Copyright (C) 2018 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later  <http://gnu.org/licenses/gpl.html> <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.

Type "show copying" and "show warranty" for details.

This GDB was configured as "x86_64-linux-gnu".

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

 <http://www.gnu.org/software/gdb/bugs/> <http://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at:

     <http://www.gnu.org/software/gdb/documentation/> <http://www.gnu.org/software/gdb/documentation/>.

 

For help, type "help".

Type "apropos word" to search for commands related to "word"...

Reading symbols from /usr/lib/cyrus/bin/ctl_backups...Reading symbols from /usr/lib/debug/.build-id/11/23ce6d2f413e1384c144165f996813ad4924c0.debug...done.

done.

[New LWP 3927]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Core was generated by `/usr/lib/cyrus/bin/ctl_backups verify -vvv -m polyfoam.com.au!support'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120

120    ../sysdeps/x86_64/multiarch/../strlen.S: No such file or directory.

(gdb) bt

#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120

#1  0x0000557f0137c495 in backup_get_paths (mbname=0x557f0300bfd0, data_fname=0x7ffcbe2daf90, index_fname=0x0, 

    create=BACKUP_OPEN_NOCREATE) at backup/lcb.c:373

#2  0x0000557f0136d678 in main (argc=5, argv=0x7ffcbe2db108) at backup/ctl_backups.c:525

(gdb) up

#1  0x0000557f0137c495 in backup_get_paths (mbname=0x557f0300bfd0, data_fname=0x7ffcbe2daf90, index_fname=0x0, 

    create=BACKUP_OPEN_NOCREATE) at backup/lcb.c:373

373    backup/lcb.c: No such file or directory.

(gdb) p *mbname

$2 = {boxes = 0x557f0300c090, is_deleted = 0, localpart = 0x0, domain = 0x557f0300c050 "polyfoam.com.au", 

  extns = 0x0, extuserid = 0x0, userid = 0x0, intname = 0x557f0300c030 "polyfoam.com.au!support", extname = 0x0, 

  recipient = 0x0}

(gdb) p userid

$4 = 0x0

(gdb) 

Calling strlen() on the null userid would be the immediate cause of the crash.  I'm not familiar enough with the code to know what leads to userid being null, or if that's also the cause of backupd crashing.

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20191014/31db3383/attachment-0001.html>

From Hagedorn at uni-koeln.de  Mon Oct 14 02:45:10 2019
From: Hagedorn at uni-koeln.de (Sebastian Hagedorn)
Date: Mon, 14 Oct 2019 08:45:10 +0200
Subject: XBACKUP and backupd not backing up public folders (3.0.8)
In-Reply-To: <013b01d5822a$b7f51ff0$27df5fd0$@polyfoam.com.au>
References: <a04f8634-f9b3-2167-a36d-5130c6bc69a8@polyfoam.com.au>
 <90b6d2f0-924d-4530-93fd-3eeaf95b0e18@www.fastmail.com>
 <d9f4f951-baaf-68e1-60d1-be2768621349@polyfoam.com.au>
 <d0e5c248-17e9-4677-8c45-47cb91d1d22e@www.fastmail.com>
 <013b01d5822a$b7f51ff0$27df5fd0$@polyfoam.com.au>
Message-ID: <293f20c0-487f-a7b9-7a72-8f32e0eba3f7@uni-koeln.de>

Am 14.10.19 um 03:00 schrieb Deborah Pickett:
> So how are people currently backing up shared folders, if they?re not
> using Cyrus backupd?

FWIW, we just take snapshots of the file systems and backup those. Mail
files aren't databases, so I don't think you have to worry about
corrupted files as much.

Cheers
Sebastian
-- 
   .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:.
                .:.Regionales Rechenzentrum (RRZK).:.
  .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Hagedorn.vcf
Type: text/x-vcard
Size: 333 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20191014/aa2abd81/attachment.vcf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5367 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20191014/aa2abd81/attachment.p7s>

From jwidera at oakton.edu  Mon Oct 14 16:43:11 2019
From: jwidera at oakton.edu (John Widera)
Date: Mon, 14 Oct 2019 15:43:11 -0500
Subject: cyradm and TLS 1.2
In-Reply-To: <a7338dc7ede34a4a0b3c460f61f91ce3@oakton.edu>
References: <a7338dc7ede34a4a0b3c460f61f91ce3@oakton.edu>
Message-ID: <1626d5f24ec0de0fcb1a1f96ae3dfc9a@oakton.edu>

Turns out imclient (at least in the latest RHEL7 pkg) is hardcoded to
use TLSv1.  Since we're building binary RPMs from Source RPMs anyway we
modified imclient.c, rebuilt the RPMs, reinstalled the cyrus-imapd-utils
package:  Here's the patch we used: 

---------------------------------------------------- 

--- IMCLIENT.C.ORIG 2012-12-01 13:57:54.000000000 -0600
+++ IMCLIENT.C 2019-10-03 14:40:11.254566297 -0500
@@ -1695,7 +1695,7 @@
RETURN -1;
} 

- IMCLIENT->TLS_CTX = SSL_CTX_NEW(TLSV1_CLIENT_METHOD());
+ IMCLIENT->TLS_CTX = SSL_CTX_NEW(TLSV1_2_CLIENT_METHOD());
IF (IMCLIENT->TLS_CTX == NULL) {
RETURN -1;
}; 

------------------------------------------- 

Maybe this helps someone else. 

Regards,

> Hi All, 
> 
> We're hoping to find some help on the list... 
> 
> We are running Cyrus-IMAP on RHEL7, using an RPM pkg (CYRUS-IMAPD-2.4.17-13.EL7) built from the Red Hat SRC RPM.  We also have SASL, Utils, devel etc pkgs all from RH. 
> 
> Now we're looking to finally move Cyrus completely off insecure TLS versions.  But now there is a lingering issue... 
> 
> We removed tls1_0 from impad.conf, and the CYRADM shell stopped working.  We can no longer connect at all: 
> 
> CYRADM -U CYRUS <SERVER>
> [ SSL_CONNECT ERROR -1 ]
> [ SSL SESSION REMOVED ]
> [ TLS NEGOTIATION DID NOT SUCCEED ]
> CYRADM: CANNOT AUTHENTICATE TO SERVER WITH AS CYRUS 
> 
> CYRADM -U CYRUS --NOTLS <SERVER>
> [ SSL_CONNECT ERROR -1 ]
> [ SSL SESSION REMOVED ]
> [ TLS NEGOTIATION DID NOT SUCCEED ]
> CYRADM: CANNOT AUTHENTICATE TO SERVER WITH AS CYRUS 
> 
> The presumption is (as cyradm is just a wrapper script) any PERL scripts calling Cyrus::IMAP::Admin over a STARTTLS connection could likewise be broken (?) if we block TLS 1.0.  
> 
> cyradm is using TLSv1 per maillog: 
> 
> IMAPS[14096]: STARTTLS: TLSV1 WITH CIPHER <SNIP> 
> 
> Our MAN page for cyradm shows a "--notls" option, which does not work/changes nothing.  Oddly, the cyradm HELP FLAG does NOT show this option, yet cyradm doesn't bark when it's passed: 
> 
> USAGE: CYRADM [ARGS] SERVER
> --USER <USER> CONNECT AS <USER> (AUTHENTICATION NAME)
> --AUTHZ <USER> AUTHORIZE AS <USER>
> --[NO]RC (DO NOT) LOAD THE CONFIGURATION FILES
> --SYSTEMRC <FILE> USE SYSTEM-WIDE CONFIGURATION <FILE>
> --USERRC <FILE> USE USER CONFIGURATION <FILE>
> --PORT <PORT> CONNECT TO SERVER ON <PORT>
> --AUTH <MECHANISM> AUTHENTICATE WITH <MECHANISM> 
> 
> A web search reveals the MAN page for cyradm in Cyrus v.3, and it shows NOTLS as an option to AUTHENTICATE, after a server connection is made, so its unclear to me what's going on... 
> 
> Does anyone have cyradm working with TLS1.2? 
> 
> Regards & THANKS in advance for any assistance or suggestions offered. 
> 
> -- 
> John 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20191014/d845e142/attachment.html>

From ellie at fastmail.com  Tue Oct 15 19:04:58 2019
From: ellie at fastmail.com (ellie timoney)
Date: Wed, 16 Oct 2019 10:04:58 +1100
Subject: cyradm and TLS 1.2
In-Reply-To: <1626d5f24ec0de0fcb1a1f96ae3dfc9a@oakton.edu>
References: <a7338dc7ede34a4a0b3c460f61f91ce3@oakton.edu>
 <1626d5f24ec0de0fcb1a1f96ae3dfc9a@oakton.edu>
Message-ID: <4eed8a30-25d1-4be7-a6b3-bc625b33fefe@www.fastmail.com>

Thanks for reporting back. For whatever its worth, the equivalent fix on 2.5+ uses "TLS_client_method()", not "TLSv1_2_client_method()". I'm not sure what difference it makes, but maybe it requires a newer OpenSSL than you have?

Here's the commit to master, fyi: https://github.com/cyrusimap/cyrus-imapd/commit/78f79ea53238c8596e2f8602b7b1e29a16863ae9

On Tue, Oct 15, 2019, at 7:43 AM, John Widera wrote:
> Turns out imclient (at least in the latest RHEL7 pkg) is hardcoded to use TLSv1. Since we're building binary RPMs from Source RPMs anyway we modified imclient.c, rebuilt the RPMs, reinstalled the cyrus-imapd-utils package: Here's the patch we used:

> *----------------------------------------------------*


> *--- imclient.c.orig 2012-12-01 13:57:54.000000000 -0600*
> *+++ imclient.c 2019-10-03 14:40:11.254566297 -0500*
> *@@ -1695,7 +1695,7 @@*
> *return -1;*
> *}*


> *- imclient->tls_ctx = SSL_CTX_new(TLSv1_client_method());*
> *+ imclient->tls_ctx = SSL_CTX_new(TLSv1_2_client_method());*
> *if (imclient->tls_ctx == NULL) {*
> *return -1;*
> *};*

> -------------------------------------------

> Maybe this helps someone else.

> Regards,

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20191016/c2bae383/attachment.html>

From jwade at oakton.edu  Tue Oct 15 19:37:47 2019
From: jwade at oakton.edu (John Wade)
Date: Tue, 15 Oct 2019 18:37:47 -0500
Subject: cyradm and TLS 1.2
In-Reply-To: <4eed8a30-25d1-4be7-a6b3-bc625b33fefe@www.fastmail.com>
References: <a7338dc7ede34a4a0b3c460f61f91ce3@oakton.edu>
 <1626d5f24ec0de0fcb1a1f96ae3dfc9a@oakton.edu>
 <4eed8a30-25d1-4be7-a6b3-bc625b33fefe@www.fastmail.com>
Message-ID: <7f04a0a9-51da-8ab6-38b2-e228f183cbff@oakton.edu>

Thanks!? You have the more correct fix:

From: https://www.openssl.org/docs/man1.1.0/man3/TLSv1_client_method.html

"TLS_method(), TLS_server_method(), TLS_client_method()
These are the general-purpose version-flexible SSL/TLS methods. The 
actual protocol version used will be negotiated to the highest version 
mutually supported by the client and the server. The supported protocols 
are SSLv3, TLSv1, TLSv1.1 and TLSv1.2. Applications should use these 
methods, and avoid the version-specific methods described below."

Thanks,
John


On 10/15/2019 6:04 PM, ellie timoney wrote:
>
> **********************
> CAUTION: EXTERNAL MAIL
> **********************
>
> Thanks for reporting back. ?For whatever its worth, the equivalent fix 
> on 2.5+ uses "TLS_client_method()", not "TLSv1_2_client_method()". 
> ?I'm not sure what difference it makes, but maybe it requires a newer 
> OpenSSL than you have?
>
> Here's the commit to master, fyi: 
> https://github.com/cyrusimap/cyrus-imapd/commit/78f79ea53238c8596e2f8602b7b1e29a16863ae9
>
> On Tue, Oct 15, 2019, at 7:43 AM, John Widera wrote:
>>
>> Turns out imclient (at least in the latest RHEL7 pkg) is hardcoded to 
>> use TLSv1.? Since we're building binary RPMs from Source RPMs anyway 
>> we modified imclient.c, rebuilt the RPMs, reinstalled the 
>> cyrus-imapd-utils package:? Here's the patch we used:
>>
>> *----------------------------------------------------*
>>
>> *--- imclient.c.orig 2012-12-01 13:57:54.000000000 -0600*
>> *+++ imclient.c 2019-10-03 14:40:11.254566297 -0500*
>> *@@ -1695,7 +1695,7 @@*
>> *return -1;*
>> *}*
>> *- imclient->tls_ctx = SSL_CTX_new(TLSv1_client_method());*
>> *+ imclient->tls_ctx = SSL_CTX_new(TLSv1_2_client_method());*
>> *if (imclient->tls_ctx == NULL) {*
>> *return -1;*
>> *};*
>>
>> -------------------------------------------
>>
>> Maybe this helps someone else.
>>
>> Regards,
>>
>
>
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20191015/2174b1dc/attachment.html>

From ellie at fastmail.com  Tue Oct 15 22:53:21 2019
From: ellie at fastmail.com (ellie timoney)
Date: Wed, 16 Oct 2019 13:53:21 +1100
Subject: XBACKUP and backupd not backing up public folders (3.0.8)
In-Reply-To: <d0e5c248-17e9-4677-8c45-47cb91d1d22e@www.fastmail.com>
References: <a04f8634-f9b3-2167-a36d-5130c6bc69a8@polyfoam.com.au>
 <90b6d2f0-924d-4530-93fd-3eeaf95b0e18@www.fastmail.com>
 <d9f4f951-baaf-68e1-60d1-be2768621349@polyfoam.com.au>
 <d0e5c248-17e9-4677-8c45-47cb91d1d22e@www.fastmail.com>
Message-ID: <4688d545-5bb5-4b6e-99c8-ef1f9813c708@www.fastmail.com>

On Mon, Oct 14, 2019, at 10:29 AM, ellie timoney wrote:
> I've created a github issue (https://github.com/cyrusimap/cyrus-imapd/issues/2893), and am about to make a test case to reproduce the problem, so I can get on with fixing it. :)

I think this is fixed now, on both master and 3.0 branches. Testing and feedback appreciated, of course!

Cheers,

ellie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20191016/7a005e50/attachment.html>

From debbiep at polyfoam.com.au  Wed Oct 16 07:28:27 2019
From: debbiep at polyfoam.com.au (Deborah Pickett)
Date: Wed, 16 Oct 2019 22:28:27 +1100
Subject: XBACKUP and backupd not backing up public folders (3.0.8)
In-Reply-To: <4688d545-5bb5-4b6e-99c8-ef1f9813c708@www.fastmail.com>
References: <a04f8634-f9b3-2167-a36d-5130c6bc69a8@polyfoam.com.au>
 <90b6d2f0-924d-4530-93fd-3eeaf95b0e18@www.fastmail.com>
 <d9f4f951-baaf-68e1-60d1-be2768621349@polyfoam.com.au>
 <d0e5c248-17e9-4677-8c45-47cb91d1d22e@www.fastmail.com>
 <4688d545-5bb5-4b6e-99c8-ef1f9813c708@www.fastmail.com>
Message-ID: <70dc99be-3a41-0d1f-1fad-7399e1153064@polyfoam.com.au>

Hi Ellie,

On 2019-10-16 13:53, ellie timoney wrote:
> On Mon, Oct 14, 2019, at 10:29 AM, ellie timoney wrote:
>
> I think this is fixed now, on both master and 3.0 branches. ?Testing
> and feedback appreciated, of course!

This is looking a lot better.

$ imtest -a cyrus localhost

[...]
BBB xbackup support at polyfoam.com.au rsync
* OK MAILBOX polyfoam.com.au!support
BBB OK Completed
[...]

backupd logs (sqldb_exec and _index_mailbox lines excluded):

Oct 16 22:06:55 rsync cyrus/backupd[244584]: login:
mail-3175-1.polyfoam.com.au [10.3.244.125] rsync-mail-3175-1 DIGEST-MD5
User logged in
Oct 16 22:06:55 rsync cyrus/backupd[244584]: %SHARED not found in
backups.db, creating new record
Oct 16 22:06:55 rsync cyrus/backupd[244584]: sending response to
MAILBOXES: 0 (Success)
Oct 16 22:06:55 rsync cyrus/backupd[244584]: received unreserved
messages, applying to all (1) open backups...
Oct 16 22:06:55 rsync cyrus/backupd[244584]: applying unreserved
messages to (null)
Oct 16 22:06:55 rsync cyrus/backupd[244584]: indexing MESSAGE at 29
(400443 bytes)...
Oct 16 22:06:55 rsync cyrus/backupd[244584]: sending response to
MESSAGE: 0 (Success)
Oct 16 22:06:55 rsync cyrus/backupd[244584]: indexing MAILBOX at 400472...
Oct 16 22:06:55 rsync cyrus/backupd[244584]: sending response to
MAILBOX: 0 (Success)
Oct 16 22:06:55 rsync cyrus/backupd[244584]: telling master 1
Oct 16 22:06:55 rsync cyrus/master[244397]: service backupd/ipv4 pid
244584 in BUSY state: now available and in READY state
Oct 16 22:06:55 rsync cyrus/master[244397]: service backupd/ipv4 now has
1 ready workers

Even a generic "XBACKUP * channel_name" does what I expect, doing all
USERs and then all MAILBOXes sequentially.

On the backup target:

# /usr/lib/cyrus/bin/ctl_backups list
2019-10-16 22:18:56??? 315180365??? %SHARED???
/home/mail-3175-1/cyrus-backup/partitions/default/q/%SHARED_vh91AO
[... and user mailboxes...]

# /usr/lib/cyrus/bin/cyr_backup -f
/home/mail-3175-1/cyrus-backup/partitions/default/q/%SHARED_vh91AO list
mailboxes | grep support
bguhtv8zjmd8za1a55pzmoks? 2019-10-16 16:36:39? polyfoam.com.au!support
6cayvdkso3pa37lotxz2780i? 2019-10-16 14:59:45?
polyfoam.com.au!support.completed
b07jfqamvne56ypieo4tchwa? 2019-10-15 16:39:02?
polyfoam.com.au!support.in-progress-debbiep
[... some other expected dross removed ...]

A few spot checks show that I can dump messages from the backup using
cyr_backup, so I'm confident that this'll work as a backup solution for me.

Many thanks for the rapid fix, Ellie.

Deborah


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20191016/75e369f8/attachment-0001.html>

From david.phan at bluemind.net  Tue Oct 29 05:04:27 2019
From: david.phan at bluemind.net (David Phan)
Date: Tue, 29 Oct 2019 10:04:27 +0100
Subject: Which imap command to rename a root mailbox while maintaining its
 partition
Message-ID: <13F3250B-9B46-4C37-81DF-2F68A5D93F0B@bluemind.net>

Hi,

Any update on this topic?

Regards,


David Phan
 

06.73.38.77.00

BlueMind
+33 (0)5 81 91 55 60
Hotel des T?l?coms, 40 rue du village d'entreprises
31670 Lab?ge, France
www.bluemind.net / https://blog.bluemind.net/fr/

From murch at fastmail.com  Tue Oct 29 08:13:53 2019
From: murch at fastmail.com (Ken Murchison)
Date: Tue, 29 Oct 2019 08:13:53 -0400
Subject: Which imap command to rename a root mailbox while maintaining its
 partition
In-Reply-To: <13F3250B-9B46-4C37-81DF-2F68A5D93F0B@bluemind.net>
References: <13F3250B-9B46-4C37-81DF-2F68A5D93F0B@bluemind.net>
Message-ID: <6d9174c8-a1f2-9268-571f-e2459fd08cc5@fastmail.com>

x RENAME <old> <new> <partition>


should work


On 10/29/19 5:04 AM, David Phan wrote:
> Hi,
>
> Any update on this topic?
>
> Regards,
>
>
> David Phan
>   
>
> 06.73.38.77.00
>
> BlueMind
> +33 (0)5 81 91 55 60
> Hotel des T?l?coms, 40 rue du village d'entreprises
> 31670 Lab?ge, France
> www.bluemind.net / https://blog.bluemind.net/fr/
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
Ken Murchison
Cyrus Development Team
Fastmail US LLC


From mp at petermann-it.de  Thu Oct 31 19:35:10 2019
From: mp at petermann-it.de (Matthias Petermann)
Date: Fri, 1 Nov 2019 00:35:10 +0100
Subject: Cyrus httpd inappropriate Cache-Control header leads to exposition of
 private data to unauthorized clients (was: Caching related security
 issue? (Cyrus httpd CalDAV server behind Apache mod_proxy)
In-Reply-To: <d7137493-ddcd-e8bf-9b48-4c0e004f0ed6@petermann-it.de>
References: <d7137493-ddcd-e8bf-9b48-4c0e004f0ed6@petermann-it.de>
Message-ID: <558da858-3b6b-bbd7-9940-9c2854cf4f59@petermann-it.de>

Hello,

it seams that Cyrus httpd does not set the Cache-Control header 
appropriate when serving private data. When operated behind a reverse 
proxy with caching (like for example Apache httpd with mod_proxy), there 
is a possibility that an unauthenticated anonymous client gets access to 
private information. I can reproduce this issue by having one client 
access the caldav URL with BASIC HTTP authentication. After this I use 
another client and request the same caldav URL without providing BASIC 
HTTP credentials. As long as the backend data did not change between the 
requests (304 Not Modified), the second client gets the private data 
without authentication from mod_proxy (which is unaware of the fact that 
this is private data).

Therefore I think it would be required to set the Cache-Control header 
for all httpd responses to "private".

As this is not an uncommon setup - having a reverse proxy doing the SSL 
offloading and URL routing - there might be a lot of hosts affected.

What is the best way to address this issue?

Kind regards
Matthias



Oct 31 23:18:33 mail http[14179]: read_body(flags=0x10, framing=2)
Oct 31 23:18:35 mail http[14167]: read & parse request-line
Oct 31 23:18:35 mail http[14167]: read & parse headers
Oct 31 23:18:35 mail http[14167]: conn flags: 0  upgrade flags: 0  tls 
req: 0
Oct 31 23:18:35 mail http[14167]: write_body(code = -1964266973, 
flags.te = 0, len = 0)
Oct 31 23:18:35 mail http[14167]: simple_hdr(Date: Thu, 31 Oct 2019 
22:18:35 GMT)
Oct 31 23:18:35 mail http[14167]: simple_hdr(Connection: Upgrade)
Oct 31 23:18:35 mail http[14167]: simple_hdr(Upgrade: h2c)
Oct 31 23:18:35 mail http[14167]: simple_hdr(Cache-Control: must-revalidate)
Oct 31 23:18:35 mail http[14167]: simple_hdr(Vary: Accept-Encoding)
Oct 31 23:18:35 mail http[14167]: simple_hdr(ETag: 
"1569656021-1572492600-17592-1569921236-602")
Oct 31 23:18:35 mail http[14167]: simple_hdr(Last-Modified: Thu, 31 Oct 
2019 03:30:00 GMT)
Oct 31 23:18:35 mail http[14167]: [10.0.3.2] as "mpeterma" with 
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, 
like Gecko) Iridium/2018.11 Safari/537.36 Chrome/71.0.0.0"; "GET 
/dav/calendars/user/mpeterma/ HTTP/1.1" 
(if-none-match=W/"1569656021-1572492600-17592-1569921236-602") => 
"HTTP/1.1 304 Not Modified"




On 10/7/19 7:45 AM, Matthias Petermann wrote:
> Hello,
> 
> since Cyrus is able to share calendars, I try to set up a CalDAV server 
> using Cyrus httpd behind an Apache reverse proxy (mod_proxy). During 
> this I made a security-related observation which makes me a bit 
> concerned. It seems to be about caching private data, probably more an 
> issue of my individual setup and not a bug in Cyrus. Anyway ? I am a bit 
> clueless and would be happy to discuss this to find a mitigation.
> 
> ...
> 
> Test setup / precondtions:
> 
> - Cyrus 3.0.11 server (imapd, httpd) freshly restarted
> - Apache 2.4 with mod_proxy (ssl vhost, no explicit cache configuration, 
> only mod_socache_shmcb)
> - Browser A with clean cache
> - Browser B with clean cache
> 
> Test steps:
> 
> 1) Request CalDAV URL from Browser A
> 
> - Expected: Browser presents HTTP Auth, delivers content after 
> successful login
> - Observed: Browser presents HTTP Auth, delivers content after 
> successful login
> - Result: PASS
> 
> 2) Request same CalDAV URL from Browser B
> 
> - Expected: Browser presents HTTP Auth
> - Observed: Browser delivers content without presenting HTTP Auth
> - Result: FAIL
> 
> Test observations:
> 
> - The result of step 2) seems to differ depending of which Cyrus httpd 
> process is hit by the request
> - The cache-control header delivered by Cyrus httpd seems to not contain 
> ?private? which seems to allow intermediate caches to cache private data
> 
> ...
> 
> My first configuration of Apache did allow mod_proxy to reuse the 
> backend-connections to Cyrus httpd. After I added disablereuse=On (which 
> causes Apache to close the backend-connection immediately after 
> processing a single request), it seems to mitigate the observed issue. 
> How could this be possible? These two questions could help me to 
> investigate further on my system:
> 
> - When receiving a HTTP Request with an Etag included, when exactly does 
> Cyrus httpd decide whether to request HTTP Authentication? In case the 
> Etag is valid (content unchanged), will it return the 304 without 
> requesting a HTTP Auth?
> 
> - How do the Cyrus httpd workers handle HTTP Auth in general? In case of 
> re-using a worker by a permanent connection from a reverse proxy, does 
> it check Authentication on any request, or only at the very first?
> 
> I am happy about suggestions or pointers to best practises in regards of 
> using Cyrus httpd behind a reverse proxy.
> 
> Kind regards
> Matthias
> 
> 
> 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>