From Albert.Shih at obspm.fr Tue Nov 5 05:06:22 2019 From: Albert.Shih at obspm.fr (Albert Shih) Date: Tue, 5 Nov 2019 11:06:22 +0100 Subject: sieve authentication. Message-ID: <20191105100622.GD5121@io.chezmoi.fr> Hi everyone I'm using SOGo as webui for the mail. SOGo as the capability to manage sieve filters and it's work well with cyrus. The only problem, is SOGo as the capability to disable automatically the sieve filter, but for that it need a account who has the right to modify sieve filter for everyone. I didn't find how to do that with cyrus. If someone have a idea... Regards -- Albert SHIH Observatoire de Paris xmpp: jas at obspm.fr Heure local/Local time: Tue 05 Nov 2019 11:04:19 AM CET From debbiep at polyfoam.com.au Wed Nov 6 04:20:53 2019 From: debbiep at polyfoam.com.au (Deborah Pickett) Date: Wed, 6 Nov 2019 20:20:53 +1100 Subject: Backup compaction optimization in a block-level replication environment Message-ID: <2ef90a7b-db81-59ff-c6fd-9d54cadd9889@polyfoam.com.au> (Sorry, that's a lot of big words.? I'll try explaining what I want to do.) On my LAN I have a Cyrus IMAP server (3.0.11), and a dedicated Cyrus backup server (patched with Ellie's shared-mailbox and 64-bit fseek fixes).? These are connected by a nice fat link so backups happen fast and often.? A scheduled compaction occurs each morning thanks to an event in cyrus.conf. I now want to back up the backups to an off-site server over a much slower link.? The off-site server doesn't speak the Cyrus sync protocol.? What it does do well is block-level backups: if only a part of a file has changed, only that part needs to be transferred over the slow link.? [I haven't decided whether my technology will be the rsync --checksum protocol, or Synology NAS XFS replication, or Microsoft Server VFS snapshots.? They all do block-level backups well.] Since Cyrus backup files are append-only, they should behave well with block-level backups. But?correct me if I'm wrong?compaction is going to ruin my day because a reduction in the size of chunk (say) 5 moves the start offset of chunk 6 (and so on).? Even if chunk 6 doesn't change it'll have to be retransmitted in its entirety. Right now I've set backup_compact_minsize and backup_compact_maxsize to zero but I'm not sure if even that is sufficient to prevent chunk offsets moving.? Perhaps I need to disable the compaction event in cyrus.conf entirely. I really want compaction, though, or else my backups are going to get very, very big. Which leads me to my idea.? What if compaction could be friendlier towards block-level backups, by deliberately avoiding changing chunk offsets in the backup file, even if that means gaps of unused bytes when (the aforementioned) chunk 5 shrinks?? It won't always work out, for instance when a chunk grows in size. A compromise would need to be struck between keeping chunk offsets fixed and wasted fragmented space between chunks as they shrink. I haven't collected enough data to know if I am making the right assumptions about how chunk size evolves over time and how effective compaction is at removing cruft from a backup file.? Has anyone thought about doing something like this with Cyrus backups? -- Deborah Pickett System Administrator Polyfoam Australia Pty Ltd From ellie at fastmail.com Thu Nov 7 17:13:14 2019 From: ellie at fastmail.com (ellie timoney) Date: Fri, 08 Nov 2019 09:13:14 +1100 Subject: =?UTF-8?Q?Re:_Backup_compaction_optimization_in_a_block-level_replicatio?= =?UTF-8?Q?n_environment?= In-Reply-To: <2ef90a7b-db81-59ff-c6fd-9d54cadd9889@polyfoam.com.au> References: <2ef90a7b-db81-59ff-c6fd-9d54cadd9889@polyfoam.com.au> Message-ID: <95f69ac4-00a6-4ec0-95dd-e449ec3b97cd@www.fastmail.com> I'm not sure if I'm just not understanding, but if the chunk offsets were to remain the same, then there's no benefit to compaction? A (say) 2gb file full of zeroes between small chunks is still the same 2gb on disk as one that's never been compacted at all! And if you don't use the compaction feature, you might as well skip the backups system entirely, and have your backup server just be a normal replica that doesn't accept client traffic (maybe with a very long cyr_expire -D time?), and then you shut it down on schedule for safe block/file system backups to your offsite location. > Right now I've set backup_compact_minsize and backup_compact_maxsize to > zero but I'm not sure if even that is sufficient to prevent chunk > offsets moving.? Perhaps I need to disable the compaction event in > cyrus.conf entirely. I don't have this system entirely in my head at the moment so I'm kinda just reading documentation here, but these settings are about optimising the gzip algorithm. Each chunk is compressed separately, and the tradeoff here is that bigger chunks compress better, but if the file becomes corrupted somehow you lose entire chunks, so smaller chunks are safer. > A compromise would need to be > struck between keeping chunk offsets fixed and wasted fragmented space > between chunks as they shrink. This setting might be helpful: > backup_compact_work_threshold: 1 > The number of chunks that must obviously need compaction before the com? > pact tool will go ahead with the compaction. If set to less than one, > the value is treated as being one. If you set your backup_compact_min/max_sizes to a size that's comfortable/practical for your block backup algorithm, but then set a very lax backup_compact_work_threshold, you might be able to find a sweet spot where you're getting the benefits of compaction eventually, but are not constantly changing every block in the file (until you do). The default (1) is basically for compaction to occur as soon as there's something to compact out, just because the default had to be something, and without experiential data any other value would just be a hat rabbit. But this sounds like a case where a big number would play nicer. I guess I'd try to target a minimum size of 1 disk block per chunk, and a maximum of (fair dice roll) 4 disk blocks? But you'd need some experimentation to figure out ballpark numbers, and won't be able to tune it to exact block sizes, because the configured thresholds are the uncompressed data size, not the compressed chunk size on disk. On Wed, Nov 6, 2019, at 8:20 PM, Deborah Pickett wrote: > (Sorry, that's a lot of big words.? I'll try explaining what I want to do.) > > On my LAN I have a Cyrus IMAP server (3.0.11), and a dedicated Cyrus > backup server (patched with Ellie's shared-mailbox and 64-bit fseek > fixes).? These are connected by a nice fat link so backups happen fast > and often.? A scheduled compaction occurs each morning thanks to an > event in cyrus.conf. > > I now want to back up the backups to an off-site server over a much > slower link.? The off-site server doesn't speak the Cyrus sync > protocol.? What it does do well is block-level backups: if only a part > of a file has changed, only that part needs to be transferred over the > slow link.? [I haven't decided whether my technology will be the rsync > --checksum protocol, or Synology NAS XFS replication, or Microsoft > Server VFS snapshots.? They all do block-level backups well.] > > Since Cyrus backup files are append-only, they should behave well with > block-level backups. But?correct me if I'm wrong?compaction is going to > ruin my day because a reduction in the size of chunk (say) 5 moves the > start offset of chunk 6 (and so on).? Even if chunk 6 doesn't change > it'll have to be retransmitted in its entirety. > > Right now I've set backup_compact_minsize and backup_compact_maxsize to > zero but I'm not sure if even that is sufficient to prevent chunk > offsets moving.? Perhaps I need to disable the compaction event in > cyrus.conf entirely. > > I really want compaction, though, or else my backups are going to get > very, very big. > > Which leads me to my idea.? What if compaction could be friendlier > towards block-level backups, by deliberately avoiding changing chunk > offsets in the backup file, even if that means gaps of unused bytes when > (the aforementioned) chunk 5 shrinks?? It won't always work out, for > instance when a chunk grows in size. A compromise would need to be > struck between keeping chunk offsets fixed and wasted fragmented space > between chunks as they shrink. > > I haven't collected enough data to know if I am making the right > assumptions about how chunk size evolves over time and how effective > compaction is at removing cruft from a backup file.? Has anyone thought > about doing something like this with Cyrus backups? > > -- > Deborah Pickett > System Administrator > Polyfoam Australia Pty Ltd > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From debbiep at polyfoam.com.au Thu Nov 7 21:35:40 2019 From: debbiep at polyfoam.com.au (Deborah Pickett) Date: Fri, 8 Nov 2019 13:35:40 +1100 Subject: Backup compaction optimization in a block-level replication environment In-Reply-To: <95f69ac4-00a6-4ec0-95dd-e449ec3b97cd@www.fastmail.com> References: <2ef90a7b-db81-59ff-c6fd-9d54cadd9889@polyfoam.com.au> <95f69ac4-00a6-4ec0-95dd-e449ec3b97cd@www.fastmail.com> Message-ID: <8461a83f-a986-a164-1790-331e62888075@polyfoam.com.au> On 2019-11-08 09:13, ellie timoney wrote: > I'm not sure if I'm just not understanding, but if the chunk offsets were to remain the same, then there's no benefit to compaction? A (say) 2gb file full of zeroes between small chunks is still the same 2gb on disk as one that's never been compacted at all! That's true.? I suppose I'm imagining a threshold, where if the file hits, say, 20% wasted space, then I can "defrag" the file and recover the lost space, on the understanding that the next sync will have to copy the entire file again. But you mentioned: > And if you don't use the compaction feature, you might as well skip the backups system entirely, and have your backup server just be a normal replica that doesn't accept client traffic (maybe with a very long cyr_expire -D time?), and then you shut it down on schedule for safe block/file system backups to your offsite location. ... and that seems a more reasonable approach.? I didn't know if copying the filesystem of a (paused) Cyrus replica was a supported way of backing up, but now I do.? Is there a list of which database and index files I need to copy apart from the files inside the partition structure? > This setting might be helpful: > >> backup_compact_work_threshold: 1 >> The number of chunks that must obviously need compaction before the com? >> pact tool will go ahead with the compaction. If set to less than one, >> the value is treated as being one. > If you set your backup_compact_min/max_sizes to a size that's comfortable/practical for your block backup algorithm, but then set a very lax backup_compact_work_threshold, you might be able to find a sweet spot where you're getting the benefits of compaction eventually, but are not constantly changing every block in the file (until you do). The default (1) is basically for compaction to occur as soon as there's something to compact out, just because the default had to be something, and without experiential data any other value would just be a hat rabbit. But this sounds like a case where a big number would play nicer. > > I guess I'd try to target a minimum size of 1 disk block per chunk, and a maximum of (fair dice roll) 4 disk blocks? But you'd need some experimentation to figure out ballpark numbers, and won't be able to tune it to exact block sizes, because the configured thresholds are the uncompressed data size, not the compressed chunk size on disk. Thanks, I saw that setting but didn't really think through how it would help me.? I'll experiment with it and report back. -- *Deborah Pickett* System Administrator *Polyfoam Australia Pty Ltd* From debbiep at polyfoam.com.au Fri Nov 8 01:12:37 2019 From: debbiep at polyfoam.com.au (Deborah Pickett) Date: Fri, 8 Nov 2019 17:12:37 +1100 Subject: Cyrus backup: is traffic from master to backup server encrypted? Message-ID: <14cb43b0-645c-0412-007f-f1bd2b5d1439@polyfoam.com.au> ... or do I need to establish my own SSH tunnel from master to backup server? I've set up my dedicated Cyrus backup server with tls_server_cert and tls_server_key, and when I connect to port 2005 I see that STARTTLS is offered: # nc localhost 2005 * SASL PLAIN LOGIN DIGEST-MD5 * STARTTLS * COMPRESS DEFLATE * OK rsync Cyrus backup server 3.0.11-Debian-3.0.11-1~bpo10+1 STARTTLS NO command not implemented But as shown, the STARTTLS command from the client is rejected. I believe that DIGEST-MD5 gives me some level of privacy (sync_test reports a security strength factor of 128) even without TLS? -- *Deborah Pickett* System Administrator *Polyfoam Australia Pty Ltd* -------------- next part -------------- An HTML attachment was scrubbed... URL: From boutilpj at ednet.ns.ca Fri Nov 8 06:35:59 2019 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Fri, 8 Nov 2019 07:35:59 -0400 Subject: Cyrus backup: is traffic from master to backup server encrypted? In-Reply-To: <14cb43b0-645c-0412-007f-f1bd2b5d1439@polyfoam.com.au> References: <14cb43b0-645c-0412-007f-f1bd2b5d1439@polyfoam.com.au> Message-ID: Odd, works here. telnet localhost 2005 Trying ::1... Connected to localhost. Escape character is '^]'. * SASL PLAIN * STARTTLS * COMPRESS DEFLATE * OK domain Cyrus sync server v2.4.20 STARTTLS OK Begin TLS negotiation now On 11/8/19 2:12 AM, Deborah Pickett wrote: > ... or do I need to establish my own SSH tunnel from master to backup > server? > > I've set up my dedicated Cyrus backup server with tls_server_cert and > tls_server_key, and when I connect to port 2005 I see that STARTTLS is > offered: > > # nc localhost 2005 > * SASL PLAIN LOGIN DIGEST-MD5 > * STARTTLS > * COMPRESS DEFLATE > * OK rsync Cyrus backup server 3.0.11-Debian-3.0.11-1~bpo10+1 > STARTTLS > NO command not implemented > > But as shown, the STARTTLS command from the client is rejected. > > I believe that DIGEST-MD5 gives me some level of privacy (sync_test > reports a security strength factor of 128) even without TLS? > > -- > *Deborah Pickett* > System Administrator > *Polyfoam Australia Pty Ltd* > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > -------------- next part -------------- A non-text attachment was scrubbed... Name: boutilpj.vcf Type: text/x-vcard Size: 286 bytes Desc: not available URL: From boutilpj at ednet.ns.ca Fri Nov 8 06:54:54 2019 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Fri, 8 Nov 2019 07:54:54 -0400 Subject: Cyrus backup: is traffic from master to backup server encrypted? In-Reply-To: References: <14cb43b0-645c-0412-007f-f1bd2b5d1439@polyfoam.com.au> Message-ID: <717053b3-6c08-b228-40d7-bfd58b357258@ednet.ns.ca> Just noticed that I am running an older version of Cyrus though. On 11/8/19 7:35 AM, Patrick Boutilier wrote: > Odd, works here. > > > telnet localhost 2005 > Trying ::1... > Connected to localhost. > Escape character is '^]'. > * SASL PLAIN > * STARTTLS > * COMPRESS DEFLATE > * OK domain Cyrus sync server v2.4.20 > STARTTLS > OK Begin TLS negotiation now > > > > > On 11/8/19 2:12 AM, Deborah Pickett wrote: >> ... or do I need to establish my own SSH tunnel from master to backup >> server? >> >> I've set up my dedicated Cyrus backup server with tls_server_cert and >> tls_server_key, and when I connect to port 2005 I see that STARTTLS is >> offered: >> >> # nc localhost 2005 >> * SASL PLAIN LOGIN DIGEST-MD5 >> * STARTTLS >> * COMPRESS DEFLATE >> * OK rsync Cyrus backup server 3.0.11-Debian-3.0.11-1~bpo10+1 >> STARTTLS >> NO command not implemented >> >> But as shown, the STARTTLS command from the client is rejected. >> >> I believe that DIGEST-MD5 gives me some level of privacy (sync_test >> reports a security strength factor of 128) even without TLS? >> >> -- >> *Deborah Pickett* >> System Administrator >> *Polyfoam Australia Pty Ltd* >> >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >> > > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > -------------- next part -------------- A non-text attachment was scrubbed... Name: boutilpj.vcf Type: text/x-vcard Size: 286 bytes Desc: not available URL: From debbiep at polyfoam.com.au Fri Nov 8 15:07:03 2019 From: debbiep at polyfoam.com.au (Deborah Pickett) Date: Sat, 9 Nov 2019 07:07:03 +1100 Subject: Cyrus backup: is traffic from master to backup server encrypted? In-Reply-To: References: Message-ID: <27111889-71A6-499D-89B1-D62924A3EF39@polyfoam.com.au> Thanks Patrick. I wonder if I inadvertently compiled out support for TLS when I built my binaries. I?ll try it again with the stock binaries rather than my patched ones. > On 8 Nov 2019, at 22:42, Patrick Boutilier wrote: > > ?Odd, works here. > > > telnet localhost 2005 > Trying ::1... > Connected to localhost. > Escape character is '^]'. > * SASL PLAIN > * STARTTLS > * COMPRESS DEFLATE > * OK domain Cyrus sync server v2.4.20 > STARTTLS > OK Begin TLS negotiation now > > > > >> On 11/8/19 2:12 AM, Deborah Pickett wrote: >> ... or do I need to establish my own SSH tunnel from master to backup server? >> I've set up my dedicated Cyrus backup server with tls_server_cert and tls_server_key, and when I connect to port 2005 I see that STARTTLS is offered: >> # nc localhost 2005 >> * SASL PLAIN LOGIN DIGEST-MD5 >> * STARTTLS >> * COMPRESS DEFLATE >> * OK rsync Cyrus backup server 3.0.11-Debian-3.0.11-1~bpo10+1 >> STARTTLS >> NO command not implemented >> But as shown, the STARTTLS command from the client is rejected. >> I believe that DIGEST-MD5 gives me some level of privacy (sync_test reports a security strength factor of 128) even without TLS? From helder at paxjulia.org Sat Nov 9 18:16:16 2019 From: helder at paxjulia.org (Helder Guerreiro) Date: Sat, 9 Nov 2019 23:16:16 +0000 Subject: IMAP over SSL (only) handshake hangs In-Reply-To: <2615668.8Df7ofhZoJ@gongo> References: <2615668.8Df7ofhZoJ@gongo> Message-ID: Hi all I'm having this exact same problem. Once the daemon is up it takes a while (a random while) to get to this state. I'm on Debian 9.11 (stretch) which still is on Cyrus imap 2.5.10. Any help would be very much appreciated. /Helder On 13/01/2015 10.22, Niels Dettenbach wrote: > Hi all, > > today i've runned into a very suspicious problem never seen before: > > While any other IMAP and POP3 ports with and without SSL / TLS are working - > connects to imaps (993) just hangs, there is nothing in the logs and a > > openssl s_client -connect mail.myhost.abc:993 > > just brings out: > > CONNECTED(00000003) > > what times out after minutes. Connection to 995 (POP3s) works perfectly. > > A imtest -v -s against the IP of the machine hangs on: > > starting TLS engine > setting up TLS connection > SSL_connect:before/connect initialization > write to 7F185DDB6480 [7F185DDC48F3] (216 bytes => 216 (0xD8)) > 0000 16 03 01 00 d3 01 00 00|cf 03 01 da 39 78 63 50 > 0010 b3 95 c8 e9 2f 11 4c 6c|de 39 e2 01 d1 e5 da 34 > 0020 61 e7 8d a5 85 68 6d 7a|14 e0 59 00 00 5c c0 14 > 0030 c0 0a 00 39 00 38 00 88|00 87 c0 0f c0 05 00 35 > 0040 00 84 c0 13 c0 09 00 33|00 32 00 9a 00 99 00 45 > 0050 00 44 c0 0e c0 04 00 2f|00 96 00 41 00 07 c0 11 > 0060 c0 07 c0 0c c0 02 00 05|00 04 c0 12 c0 08 00 16 > 0070 00 13 c0 0d c0 03 00 0a|00 15 00 12 00 09 00 14 > 0080 00 11 00 08 00 06 00 03|00 ff 02 01 00 00 49 00 > 0090 0b 00 04 03 00 01 02 00|0a 00 34 00 32 00 0e 00 > 00a0 0d 00 19 00 0b 00 0c 00|18 00 09 00 0a 00 16 00 > 00b0 17 00 08 00 06 00 07 00|14 00 15 00 04 00 05 00 > 00c0 12 00 13 00 01 00 02 00|03 00 0f 00 10 00 11 00 > 00d0 23 00 00 00 0f 00 01 01| > SSL_connect:SSLv3 write client hello A > > I tried to delete tls_sessions and even connecting to localhost (where it is > bound too). netstat shows ESTABLISHED on such connections too. > > The service is configured (and worked until tonight!): > > imaps cmd="imapd -s" listen="imaps" prefork=0 maxchild=150 > pop3s cmd="pop3d -s" listen="pop3s" prefork=0 maxchild=50 > > A crazy thing is, that connections to "localhost" seems to work as soon as it > uses the IPv6 adress of the localhost (::): > > imtest -v -s localhost > > while the IPv4 variant doesnt seem to work: > > imtest -v -s 127.0.0.1 > > Because we did not use any IPv6 on that Gentoo machine i've disabled any IPv6 > stuff now which doesnt seem to help. > > cyrus-imap is compiled > > with: > berkdb nntp pam sieve snmp sqlite ssl tcpd > > without: > -afs -kerberos -mysql -postgres -replication > > dev-libs/openssl is 1.0.1k compiled > > with: > sse2 tls-heartbeat zlib > > without: > -bindist -gmp -kerberos -rfc3779 -static-libs -test -vanilla > > anything under Intel Xeon (bare metal). > > > > many thanks for any help or ideas where to look further? > > > Some logs: > > startup: > Jan 13 11:06:41 blade4 master[12565]: about to exec > /usr/lib64/cyrus/ctl_cyrusdb > Jan 13 11:06:41 blade4 ctl_cyrusdb[12565]: SQL backend defaulting to engine > 'sqlite' > Jan 13 11:06:41 blade4 ctl_cyrusdb[12565]: recovering cyrus databases > Jan 13 11:06:41 blade4 ctl_cyrusdb[12565]: skiplist: checkpointed > /email/lib/cyrus/mailboxes.db (477 records, 60868 bytes) in 0 seconds > Jan 13 11:06:41 blade4 ctl_cyrusdb[12565]: skiplist: checkpointed > /email/lib/cyrus/annotations.db (0 records, 144 bytes) in 0 seconds > Jan 13 11:06:42 blade4 ctl_cyrusdb[12565]: done recovering cyrus databases > Jan 13 11:06:42 blade4 master[12595]: about to exec /usr/lib64/cyrus/idled > Jan 13 11:06:42 blade4 master[12598]: about to exec > /usr/lib64/cyrus/ctl_deliver > Jan 13 11:06:42 blade4 master[12599]: about to exec > /usr/lib64/cyrus/ctl_cyrusdb > Jan 13 11:06:42 blade4 master[12597]: about to exec /usr/lib64/cyrus/tls_prune > Jan 13 11:06:42 blade4 ctl_cyrusdb[12599]: SQL backend defaulting to engine > 'sqlite' > Jan 13 11:06:42 blade4 ctl_cyrusdb[12599]: checkpointing cyrus databases > Jan 13 11:06:42 blade4 ctl_cyrusdb[12599]: archiving database file: > /email/lib/cyrus/mailboxes.db > Jan 13 11:06:42 blade4 ctl_cyrusdb[12599]: archiving database file: > /email/lib/cyrus/annotations.db > Jan 13 11:06:42 blade4 ctl_cyrusdb[12599]: done checkpointing cyrus databases > Jan 13 11:06:42 blade4 tls_prune[12597]: skiplist: checkpointed > /email/lib/cyrus/tls_sessions.db (1 record, 324 bytes) in 0 seconds > Jan 13 11:06:42 blade4 cyr_expire[12598]: skiplist: checkpointed > /email/lib/cyrus/deliver.db (804 records, 121348 bytes) in 0 seconds > > and: > > Jan 13 11:07:54 blade4 master[12559]: exiting on SIGTERM/SIGINT > Jan 13 11:07:54 blade4 master[25695]: setrlimit: Unable to set file > descriptors limit to -1: Operation not permitted > Jan 13 11:07:54 blade4 master[25695]: retrying with 4096 (current max) > Jan 13 11:07:54 blade4 master[25695]: process started > Jan 13 11:07:54 blade4 master[25699]: about to exec > /usr/lib64/cyrus/ctl_cyrusdb > Jan 13 11:07:55 blade4 ctl_cyrusdb[25699]: SQL backend defaulting to engine > 'sqlite' > Jan 13 11:07:55 blade4 ctl_cyrusdb[25699]: recovering cyrus databases > Jan 13 11:07:55 blade4 ctl_cyrusdb[25699]: skiplist: checkpointed > /email/lib/cyrus/mailboxes.db (477 records, 60868 bytes) in 0 seconds > Jan 13 11:07:55 blade4 ctl_cyrusdb[25699]: skiplist: checkpointed > /email/lib/cyrus/annotations.db (0 records, 144 bytes) in 0 seconds > Jan 13 11:07:55 blade4 ctl_cyrusdb[25699]: done recovering cyrus databases > Jan 13 11:07:55 blade4 master[26043]: about to exec /usr/lib64/cyrus/idled > Jan 13 11:07:55 blade4 master[25695]: unable to setsocketopt(IP_TOS): > Operation not supported > Jan 13 11:07:55 blade4 master[25695]: ready for work > Jan 13 11:07:55 blade4 master[26046]: about to exec > /usr/lib64/cyrus/ctl_deliver > Jan 13 11:07:55 blade4 master[26047]: about to exec > /usr/lib64/cyrus/ctl_cyrusdb > Jan 13 11:07:55 blade4 master[26045]: about to exec /usr/lib64/cyrus/tls_prune > Jan 13 11:07:56 blade4 cyr_expire[26046]: SQL backend defaulting to engine > 'sqlite' > Jan 13 11:07:56 blade4 ctl_cyrusdb[26047]: SQL backend defaulting to engine > 'sqlite' > Jan 13 11:07:56 blade4 ctl_cyrusdb[26047]: checkpointing cyrus databases > Jan 13 11:07:56 blade4 ctl_cyrusdb[26047]: archiving database file: > /email/lib/cyrus/mailboxes.db > Jan 13 11:07:56 blade4 tls_prune[26045]: SQL backend defaulting to engine > 'sqlite' > Jan 13 11:07:56 blade4 idled[26044]: SQL backend defaulting to engine 'sqlite' > Jan 13 11:07:56 blade4 ctl_cyrusdb[26047]: archiving database file: > /email/lib/cyrus/annotations.db > Jan 13 11:07:56 blade4 ctl_cyrusdb[26047]: done checkpointing cyrus databases > Jan 13 11:07:56 blade4 master[25695]: process 26047 exited, status 0 > Jan 13 11:07:56 blade4 tls_prune[26045]: skiplist: checkpointed > /email/lib/cyrus/tls_sessions.db (1 record, 324 bytes) in 0 seconds > Jan 13 11:07:56 blade4 tls_prune[26045]: tls_prune: purged 0 out of 1 entries > Jan 13 11:07:56 blade4 master[25695]: process 26045 exited, status 0 > Jan 13 11:07:56 blade4 cyr_expire[26046]: skiplist: checkpointed > /email/lib/cyrus/deliver.db (800 records, 120752 bytes) in 0 seconds > Jan 13 11:07:59 blade4 cyr_expire[26046]: Expunged 0 out of 475379 messages > from 477 mailboxes > Jan 13 11:07:59 blade4 cyr_expire[26046]: duplicate_prune: pruning back 3.00 > days > Jan 13 11:07:59 blade4 cyr_expire[26046]: duplicate_prune: purged 0 out of 800 > entries > Jan 13 11:07:59 blade4 cyr_expire[26046]: DIGEST-MD5 common mech free > Jan 13 11:07:59 blade4 master[25695]: process 26046 exited, status 0 From boutilpj at ednet.ns.ca Sat Nov 9 18:34:13 2019 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Sat, 9 Nov 2019 19:34:13 -0400 Subject: IMAP over SSL (only) handshake hangs In-Reply-To: References: <2615668.8Df7ofhZoJ@gongo> Message-ID: <10df14d8-47ef-6c6f-3cdd-ee27d69f2ea9@ednet.ns.ca> Almost sounds like you are running out of entropy. What does this show? cat /proc/sys/kernel/random/entropy_avail On 11/9/19 7:16 PM, Helder Guerreiro via Info-cyrus wrote: > Hi all > > I'm having this exact same problem. Once the daemon is up it takes a > while (a random while) to get to this state. > > I'm on Debian 9.11 (stretch) which still is on Cyrus imap 2.5.10. > > Any help would be very much appreciated. > > /Helder > > On 13/01/2015 10.22, Niels Dettenbach wrote: >> Hi all, >> >> today i've runned into a very suspicious problem never seen before: >> >> While any other IMAP and POP3 ports with and without SSL / TLS are >> working - >> connects to imaps (993) just hangs, there is nothing in the logs and a >> >> ????openssl s_client -connect mail.myhost.abc:993 >> >> just brings out: >> >> ????CONNECTED(00000003) >> >> what times out after minutes. Connection to 995 (POP3s) works perfectly. >> >> A imtest -v -s against the IP of the machine hangs on: >> >> starting TLS engine >> setting up TLS connection >> SSL_connect:before/connect initialization >> write to 7F185DDB6480 [7F185DDC48F3] (216 bytes => 216 (0xD8)) >> 0000 16 03 01 00 d3 01 00 00|cf 03 01 da 39 78 63 50 >> 0010 b3 95 c8 e9 2f 11 4c 6c|de 39 e2 01 d1 e5 da 34 >> 0020 61 e7 8d a5 85 68 6d 7a|14 e0 59 00 00 5c c0 14 >> 0030 c0 0a 00 39 00 38 00 88|00 87 c0 0f c0 05 00 35 >> 0040 00 84 c0 13 c0 09 00 33|00 32 00 9a 00 99 00 45 >> 0050 00 44 c0 0e c0 04 00 2f|00 96 00 41 00 07 c0 11 >> 0060 c0 07 c0 0c c0 02 00 05|00 04 c0 12 c0 08 00 16 >> 0070 00 13 c0 0d c0 03 00 0a|00 15 00 12 00 09 00 14 >> 0080 00 11 00 08 00 06 00 03|00 ff 02 01 00 00 49 00 >> 0090 0b 00 04 03 00 01 02 00|0a 00 34 00 32 00 0e 00 >> 00a0 0d 00 19 00 0b 00 0c 00|18 00 09 00 0a 00 16 00 >> 00b0 17 00 08 00 06 00 07 00|14 00 15 00 04 00 05 00 >> 00c0 12 00 13 00 01 00 02 00|03 00 0f 00 10 00 11 00 >> 00d0 23 00 00 00 0f 00 01 01| >> SSL_connect:SSLv3 write client hello A >> >> I tried to delete tls_sessions and even connecting to localhost (where >> it is >> bound too). netstat shows ESTABLISHED on such connections too. >> >> The service is configured (and worked until tonight!): >> >> ?? imaps???????? cmd="imapd -s" listen="imaps" prefork=0 maxchild=150 >> ?? pop3s???????? cmd="pop3d -s" listen="pop3s" prefork=0 maxchild=50 >> >> A crazy thing is, that connections to "localhost" seems to work as >> soon as it >> uses the IPv6 adress of the localhost (::): >> >> ????imtest -v -s localhost >> >> while the IPv4 variant doesnt seem to work: >> >> ????imtest -v -s 127.0.0.1 >> >> Because we did not use any IPv6 on that Gentoo machine i've disabled >> any IPv6 >> stuff now which doesnt seem to help. >> >> cyrus-imap is compiled >> >> ????with: >> ????berkdb nntp pam sieve snmp sqlite ssl tcpd >> >> ????without: >> ????-afs -kerberos -mysql -postgres -replication >> >> dev-libs/openssl is 1.0.1k compiled >> >> ????with: >> ????sse2 tls-heartbeat zlib >> >> ????without: >> ????-bindist -gmp -kerberos -rfc3779 -static-libs -test -vanilla >> >> anything under Intel Xeon (bare metal). >> >> >> >> many thanks for any help or ideas where to look further? >> >> >> Some logs: >> >> startup: >> Jan 13 11:06:41 blade4 master[12565]: about to exec >> /usr/lib64/cyrus/ctl_cyrusdb >> Jan 13 11:06:41 blade4 ctl_cyrusdb[12565]: SQL backend defaulting to >> engine >> 'sqlite' >> Jan 13 11:06:41 blade4 ctl_cyrusdb[12565]: recovering cyrus databases >> Jan 13 11:06:41 blade4 ctl_cyrusdb[12565]: skiplist: checkpointed >> /email/lib/cyrus/mailboxes.db (477 records, 60868 bytes) in 0 seconds >> Jan 13 11:06:41 blade4 ctl_cyrusdb[12565]: skiplist: checkpointed >> /email/lib/cyrus/annotations.db (0 records, 144 bytes) in 0 seconds >> Jan 13 11:06:42 blade4 ctl_cyrusdb[12565]: done recovering cyrus >> databases >> Jan 13 11:06:42 blade4 master[12595]: about to exec >> /usr/lib64/cyrus/idled >> Jan 13 11:06:42 blade4 master[12598]: about to exec >> /usr/lib64/cyrus/ctl_deliver >> Jan 13 11:06:42 blade4 master[12599]: about to exec >> /usr/lib64/cyrus/ctl_cyrusdb >> Jan 13 11:06:42 blade4 master[12597]: about to exec >> /usr/lib64/cyrus/tls_prune >> Jan 13 11:06:42 blade4 ctl_cyrusdb[12599]: SQL backend defaulting to >> engine >> 'sqlite' >> Jan 13 11:06:42 blade4 ctl_cyrusdb[12599]: checkpointing cyrus databases >> Jan 13 11:06:42 blade4 ctl_cyrusdb[12599]: archiving database file: >> /email/lib/cyrus/mailboxes.db >> Jan 13 11:06:42 blade4 ctl_cyrusdb[12599]: archiving database file: >> /email/lib/cyrus/annotations.db >> Jan 13 11:06:42 blade4 ctl_cyrusdb[12599]: done checkpointing cyrus >> databases >> Jan 13 11:06:42 blade4 tls_prune[12597]: skiplist: checkpointed >> /email/lib/cyrus/tls_sessions.db (1 record, 324 bytes) in 0 seconds >> Jan 13 11:06:42 blade4 cyr_expire[12598]: skiplist: checkpointed >> /email/lib/cyrus/deliver.db (804 records, 121348 bytes) in 0 seconds >> >> and: >> >> Jan 13 11:07:54 blade4 master[12559]: exiting on SIGTERM/SIGINT >> Jan 13 11:07:54 blade4 master[25695]: setrlimit: Unable to set file >> descriptors limit to -1: Operation not permitted >> Jan 13 11:07:54 blade4 master[25695]: retrying with 4096 (current max) >> Jan 13 11:07:54 blade4 master[25695]: process started >> Jan 13 11:07:54 blade4 master[25699]: about to exec >> /usr/lib64/cyrus/ctl_cyrusdb >> Jan 13 11:07:55 blade4 ctl_cyrusdb[25699]: SQL backend defaulting to >> engine >> 'sqlite' >> Jan 13 11:07:55 blade4 ctl_cyrusdb[25699]: recovering cyrus databases >> Jan 13 11:07:55 blade4 ctl_cyrusdb[25699]: skiplist: checkpointed >> /email/lib/cyrus/mailboxes.db (477 records, 60868 bytes) in 0 seconds >> Jan 13 11:07:55 blade4 ctl_cyrusdb[25699]: skiplist: checkpointed >> /email/lib/cyrus/annotations.db (0 records, 144 bytes) in 0 seconds >> Jan 13 11:07:55 blade4 ctl_cyrusdb[25699]: done recovering cyrus >> databases >> Jan 13 11:07:55 blade4 master[26043]: about to exec >> /usr/lib64/cyrus/idled >> Jan 13 11:07:55 blade4 master[25695]: unable to setsocketopt(IP_TOS): >> Operation not supported >> Jan 13 11:07:55 blade4 master[25695]: ready for work >> Jan 13 11:07:55 blade4 master[26046]: about to exec >> /usr/lib64/cyrus/ctl_deliver >> Jan 13 11:07:55 blade4 master[26047]: about to exec >> /usr/lib64/cyrus/ctl_cyrusdb >> Jan 13 11:07:55 blade4 master[26045]: about to exec >> /usr/lib64/cyrus/tls_prune >> Jan 13 11:07:56 blade4 cyr_expire[26046]: SQL backend defaulting to >> engine >> 'sqlite' >> Jan 13 11:07:56 blade4 ctl_cyrusdb[26047]: SQL backend defaulting to >> engine >> 'sqlite' >> Jan 13 11:07:56 blade4 ctl_cyrusdb[26047]: checkpointing cyrus databases >> Jan 13 11:07:56 blade4 ctl_cyrusdb[26047]: archiving database file: >> /email/lib/cyrus/mailboxes.db >> Jan 13 11:07:56 blade4 tls_prune[26045]: SQL backend defaulting to engine >> 'sqlite' >> Jan 13 11:07:56 blade4 idled[26044]: SQL backend defaulting to engine >> 'sqlite' >> Jan 13 11:07:56 blade4 ctl_cyrusdb[26047]: archiving database file: >> /email/lib/cyrus/annotations.db >> Jan 13 11:07:56 blade4 ctl_cyrusdb[26047]: done checkpointing cyrus >> databases >> Jan 13 11:07:56 blade4 master[25695]: process 26047 exited, status 0 >> Jan 13 11:07:56 blade4 tls_prune[26045]: skiplist: checkpointed >> /email/lib/cyrus/tls_sessions.db (1 record, 324 bytes) in 0 seconds >> Jan 13 11:07:56 blade4 tls_prune[26045]: tls_prune: purged 0 out of 1 >> entries >> Jan 13 11:07:56 blade4 master[25695]: process 26045 exited, status 0 >> Jan 13 11:07:56 blade4 cyr_expire[26046]: skiplist: checkpointed >> /email/lib/cyrus/deliver.db (800 records, 120752 bytes) in 0 seconds >> Jan 13 11:07:59 blade4 cyr_expire[26046]: Expunged 0 out of 475379 >> messages >> from 477 mailboxes >> Jan 13 11:07:59 blade4 cyr_expire[26046]: duplicate_prune: pruning >> back 3.00 >> days >> Jan 13 11:07:59 blade4 cyr_expire[26046]: duplicate_prune: purged 0 >> out of 800 >> entries >> Jan 13 11:07:59 blade4 cyr_expire[26046]: DIGEST-MD5 common mech free >> Jan 13 11:07:59 blade4 master[25695]: process 26046 exited, status 0 > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------- next part -------------- A non-text attachment was scrubbed... Name: boutilpj.vcf Type: text/x-vcard Size: 286 bytes Desc: not available URL: From helder at paxjulia.org Sat Nov 9 19:19:52 2019 From: helder at paxjulia.org (Helder Guerreiro) Date: Sun, 10 Nov 2019 00:19:52 +0000 Subject: IMAP over SSL (only) handshake hangs In-Reply-To: <10df14d8-47ef-6c6f-3cdd-ee27d69f2ea9@ednet.ns.ca> References: <2615668.8Df7ofhZoJ@gongo> <10df14d8-47ef-6c6f-3cdd-ee27d69f2ea9@ednet.ns.ca> Message-ID: On 09/11/2019 23.34, Patrick Boutilier wrote: > Almost sounds like you are running out of entropy. What does this show? > > cat /proc/sys/kernel/random/entropy_avail Right now it's at 3769 bytes. I'll monitor this. Note that STARTTLS works fine. From zorg at probesys.com Sun Nov 10 04:22:21 2019 From: zorg at probesys.com (Zorg) Date: Sun, 10 Nov 2019 10:22:21 +0100 Subject: caldav and invitation Message-ID: <365d0029-1c61-0b25-7ccb-559c928f2d3f@probesys.com> Hello I try to configure my cyrus to use caldav correctly Caldav working nice but when I try to invite someone nothing happen Let me explain I have to user "toto at test.domain.com" and "titi at test.domain.com" if toto at test.domain.com create a event and invite titi at test.domain.com i ve got this in the log imip_send_sendmail(titi at test.domain.com): Sendmail process terminated normally, exit status 0 But if i look in the calendar of titi or in the mailbox I see nothing (I ve have been testing with thunderbird and evolution) There is nothing explaining well how it is supposed to work in the? cyrus documentation so maybe i'm missing something does someone could explain what to expect when i invite someone in a event Hope I'm clear Thanks From awilliam at whitemice.org Sun Nov 10 09:13:14 2019 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Sun, 10 Nov 2019 09:13:14 -0500 Subject: caldav and invitation In-Reply-To: <365d0029-1c61-0b25-7ccb-559c928f2d3f@probesys.com> References: <365d0029-1c61-0b25-7ccb-559c928f2d3f@probesys.com> Message-ID: <1573395194.5247.0.camel@whitemice.org> > i ve got this in the log > imip_send_sendmail(titi at test.domain.com): Sendmail process terminated > normally, exit status 0 > But if i look in the calendar of titi or in the mailbox I see nothing (I > ve have been testing with thunderbird and evolution) > There is nothing explaining well how it is supposed to work in the > cyrus documentation so maybe i'm missing something > does someone could explain what to expect when i invite someone in a even Do you have "server handles invitations" in the account setup in Evolution? -- Adam Tauno Williams, awilliam at whitemice.org Multi-Modal Activists Against Auto Dependent Development resisting the unAmerican socialists of the Motorist hegemony http://www.mmaaadd.org From ellie at fastmail.com Sun Nov 10 19:10:38 2019 From: ellie at fastmail.com (ellie timoney) Date: Mon, 11 Nov 2019 11:10:38 +1100 Subject: =?UTF-8?Q?Re:_Backup_compaction_optimization_in_a_block-level_replicatio?= =?UTF-8?Q?n_environment?= In-Reply-To: <8461a83f-a986-a164-1790-331e62888075@polyfoam.com.au> References: <2ef90a7b-db81-59ff-c6fd-9d54cadd9889@polyfoam.com.au> <95f69ac4-00a6-4ec0-95dd-e449ec3b97cd@www.fastmail.com> <8461a83f-a986-a164-1790-331e62888075@polyfoam.com.au> Message-ID: <8b0b8c9c-7bb8-4397-af47-a592996ae3b2@www.fastmail.com> On Fri, Nov 8, 2019, at 1:35 PM, Deborah Pickett wrote: > I didn't know if copying > the filesystem of a (paused) Cyrus replica was a supported way of > backing up, but now I do. Yeah, as long as there are no cyrus processes running, the database/index files can just be copied about and won't be corrupted along the way. This kind of backup is useful for a full system restore, but the "help I deleted an important email and then emptied my trash and then expunged, and now I need it back" type of recoveries... I guess you copy the message file back to the partition, reconstruct, and it comes back as a new message (unread, no flags, etc)? You would need to be careful of the window between delivery of a message, replication to the replica, and deletion of the message (and replication of the deletion), to ensure you get a backup of the state where the message existed. I *think* delayed_delete and a long cyr_expire -D time takes care of this, but I'm not certain, so please test it before you rely on it. Also (maybe obviously) the need to keep point in time snapshots for as long as your recovery policy dictates, and not just delete stuff from backup as soon as its deleted from source. I'm not sure what others are doing in this space really. There's a few threads on the list archive about various backup strategies, but my focus has mainly been the backupd-based system. >? Is there a list of which database and index > files I need to copy apart from the files inside the partition structure? This kind of covers it, I think: https://www.cyrusimap.org/imap/reference/admin/locations/configuration-state.html It would be quite useful to have a "this is what you need to back up" document, but at the moment there's a certain amount of reading between the lines of adjacent documentation :( > > This setting might be helpful: > > Thanks, I saw that setting but didn't really think through how it would > help me.? I'll experiment with it and report back. That would be great, thanks! Cheers, ellie From debbiep at polyfoam.com.au Sun Nov 10 22:25:31 2019 From: debbiep at polyfoam.com.au (Deborah Pickett) Date: Mon, 11 Nov 2019 14:25:31 +1100 Subject: Cyrus backup: is traffic from master to backup server encrypted? In-Reply-To: <14cb43b0-645c-0412-007f-f1bd2b5d1439@polyfoam.com.au> References: <14cb43b0-645c-0412-007f-f1bd2b5d1439@polyfoam.com.au> Message-ID: <9774dd21-c9b4-f2e9-9f50-a3bf81c913aa@polyfoam.com.au> Answering my own question: > > ... or do I need to establish my own SSH tunnel from master to backup > server? > I do have to supply my own tunnel.? The Cyrus backup daemon currently (3.1.7) doesn't support the STARTTLS command. ??????????? if (!strcmp(cmd.s, "Starttls") && tls_enabled()) { ??????????????? prot_printf(backupd_out, "NO command not implemented\r\n"); ??????????????? eatline(backupd_in, c); ??????????????? continue; ??????????? } Source: https://github.com/cyrusimap/cyrus-imapd/blob/master/backup/backupd.c#L715 Patrick: you are running the Cyrus sync server on port 2005, which does support TLS.? That's not the same as the backup server (at least in the 3.x.x tree).? Note the difference in the OK line from the server. -- *Deborah Pickett* System Administrator *Polyfoam Australia Pty Ltd* From zorg at probesys.com Mon Nov 11 04:26:51 2019 From: zorg at probesys.com (Zorg) Date: Mon, 11 Nov 2019 10:26:51 +0100 Subject: caldav and invitation In-Reply-To: <1573395194.5247.0.camel@whitemice.org> References: <365d0029-1c61-0b25-7ccb-559c928f2d3f@probesys.com> <1573395194.5247.0.camel@whitemice.org> Message-ID: Yes i have server handles invitations and same thing with thunderbird Le 10/11/2019 ? 15:13, Adam Tauno Williams a ?crit?: >> i ve got this in the log >> imip_send_sendmail(titi at test.domain.com): Sendmail process terminated >> normally, exit status 0 >> But if i look in the calendar of titi or in the mailbox I see nothing (I >> ve have been testing with thunderbird and evolution) >> There is nothing explaining well how it is supposed to work in the >> cyrus documentation so maybe i'm missing something >> does someone could explain what to expect when i invite someone in a even > Do you have "server handles invitations" in the account setup in > Evolution? > From helder at paxjulia.org Mon Nov 11 07:53:08 2019 From: helder at paxjulia.org (Helder Guerreiro) Date: Mon, 11 Nov 2019 12:53:08 +0000 Subject: IMAP over SSL (only) handshake hangs In-Reply-To: References: <2615668.8Df7ofhZoJ@gongo> <10df14d8-47ef-6c6f-3cdd-ee27d69f2ea9@ednet.ns.ca> Message-ID: On 10/11/19 00:19, Helder Guerreiro via Info-cyrus wrote: > On 09/11/2019 23.34, Patrick Boutilier wrote: >> Almost sounds like you are running out of entropy. What does this show? >> >> cat /proc/sys/kernel/random/entropy_avail > > Right now it's at 3769 bytes. I'll monitor this. It happened again, the entropy available never got bellow 3600 (logged it every minute or so). From falon at ruparpiemonte.it Mon Nov 11 10:51:54 2019 From: falon at ruparpiemonte.it (Marco) Date: Mon, 11 Nov 2019 16:51:54 +0100 Subject: [Help] Cyrus 2.4.17 segfault Message-ID: <65974aa5-1adb-23dc-16d5-7d5b405a2d87@ruparpiemonte.it> Hello, we are experiencing a strange problem on cyrus-imapd 2.4.17. An user user/a has full ACL to another mailbox user/b. When the user/a SELECT a folder on user/b where he has access the imap process crashes. $ telnet cyrus.example.com 143 Trying 10.10.10.10 ... Connected to cyrus.example.com. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE AUTH=PLAIN SASL-IR] cyrus.example.com Cyrus IMAP v2.4.17-Invoca-RPM-2.4.17-6.el6 server ready a authenticate plain dsfrrFwaWNlQGNzaS5pdABveGN5cnVzAENdfsdfMx a OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SORT SORT=MODSEQ SORT=DISPLAY THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE LIST-EXTENDED WITHIN QRESYNC SCAN XLIST URLAUTH URLAUTH=BINARY X-NETSCAPE LOGINDISABLED COMPRESS=DEFLATE IDLE] Success (no protection) SESSIONID= a SELECT "user/b/SOME THINGS/WATER" Connection closed by foreign host. Could you suggest me some way to resolve this problem? I have already tried "reconstruct -rf" on user/a and user/b and ctl_cyrusdb -r without success. Thank you very much for every hints Marco From simon.matter at invoca.ch Tue Nov 12 01:44:57 2019 From: simon.matter at invoca.ch (Simon Matter) Date: Tue, 12 Nov 2019 07:44:57 +0100 Subject: [Help] Cyrus 2.4.17 segfault In-Reply-To: <65974aa5-1adb-23dc-16d5-7d5b405a2d87@ruparpiemonte.it> References: <65974aa5-1adb-23dc-16d5-7d5b405a2d87@ruparpiemonte.it> Message-ID: <5c425a8ca8accc80e7c5fe0ce5982ee9.squirrel@webmail.bi.corp.invoca.ch> > Hello, > > we are experiencing a strange problem on cyrus-imapd 2.4.17. > > An user user/a has full ACL to another mailbox user/b. When the user/a > SELECT a folder on user/b where he has access the imap process crashes. > > $ telnet cyrus.example.com 143 > Trying 10.10.10.10 ... > Connected to cyrus.example.com. > Escape character is '^]'. > * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE AUTH=PLAIN SASL-IR] > cyrus.example.com Cyrus IMAP v2.4.17-Invoca-RPM-2.4.17-6.el6 server ready > a authenticate plain dsfrrFwaWNlQGNzaS5pdABveGN5cnVzAENdfsdfMx > a OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxte QUOTA > MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN > MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SORT SORT=MODSEQ > SORT=DISPLAY THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE > LIST-EXTENDED WITHIN QRESYNC SCAN XLIST URLAUTH URLAUTH=BINARY > X-NETSCAPE LOGINDISABLED COMPRESS=DEFLATE IDLE] Success (no protection) > SESSIONID= > a SELECT "user/b/SOME THINGS/WATER" > Connection closed by foreign host. > > > Could you suggest me some way to resolve this problem? I don't know but if it's a bug then it's probably solved in current version 2.4.20. Simon From zhivko_vasilev at mailtemi.com Tue Nov 12 06:48:44 2019 From: zhivko_vasilev at mailtemi.com (Zhivko Vasilev) Date: Tue, 12 Nov 2019 13:48:44 +0200 Subject: IMAP over SSL (only) handshake hangs In-Reply-To: References: <2615668.8Df7ofhZoJ@gongo> <10df14d8-47ef-6c6f-3cdd-ee27d69f2ea9@ednet.ns.ca> Message-ID: <2e4ffece-47c5-4745-9ad5-67794eba4483@www.fastmail.com> Hello Helder, If there is no enough random seed OpenSSL will not hang, just return an error. Even /dev/random is opened in non-blocking mode. This is true only if OpenSSL is built with non-blocking mode. https://github.com/openssl/openssl/blob/master/crypto/rand/rand_unix.c If it hangs after some time could it be a handle leak? In the past, I had a similar problem and used "lsof" and "ulimit" commands to see is it reached. This error could be a possible track to a handle leak. setrlimit: Unable to set file descriptors limit to -1: Operation not permitted Regards, Zhivko On Mon, Nov 11, 2019, at 2:53 PM, Helder Guerreiro via Info-cyrus wrote: > On 10/11/19 00:19, Helder Guerreiro via Info-cyrus wrote: > > On 09/11/2019 23.34, Patrick Boutilier wrote: > >> Almost sounds like you are running out of entropy. What does this show? > >> > >> cat /proc/sys/kernel/random/entropy_avail > > > > Right now it's at 3769 bytes. I'll monitor this. > > It happened again, the entropy available never got bellow 3600 (logged > it every minute or so). > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellie at fastmail.com Tue Nov 12 16:58:11 2019 From: ellie at fastmail.com (ellie timoney) Date: Wed, 13 Nov 2019 08:58:11 +1100 Subject: [Help] Cyrus 2.4.17 segfault In-Reply-To: <65974aa5-1adb-23dc-16d5-7d5b405a2d87@ruparpiemonte.it> References: <65974aa5-1adb-23dc-16d5-7d5b405a2d87@ruparpiemonte.it> Message-ID: <7c3be149-87ad-45fb-b4d6-e062970a5350@www.fastmail.com> On Tue, Nov 12, 2019, at 2:51 AM, Marco wrote: > An user user/a has full ACL to another mailbox user/b. When the user/a > SELECT a folder on user/b where he has access the imap process crashes. If you set up a couple of test accounts with the same sharing arrangement, do those crash in the same way? Or is it specific to the two original accounts? From falon at ruparpiemonte.it Wed Nov 13 02:08:50 2019 From: falon at ruparpiemonte.it (Marco) Date: Wed, 13 Nov 2019 08:08:50 +0100 Subject: [Help] Cyrus 2.4.17 segfault In-Reply-To: <7c3be149-87ad-45fb-b4d6-e062970a5350@www.fastmail.com> References: <65974aa5-1adb-23dc-16d5-7d5b405a2d87@ruparpiemonte.it> <7c3be149-87ad-45fb-b4d6-e062970a5350@www.fastmail.com> Message-ID: <5d970028-0faf-06e2-2142-8c2495f767f1@ruparpiemonte.it> Il 12/11/2019 22:58, ellie timoney ha scritto: > On Tue, Nov 12, 2019, at 2:51 AM, Marco wrote: >> An user user/a has full ACL to another mailbox user/b. When the user/a >> SELECT a folder on user/b where he has access the imap process crashes. > > If you set up a couple of test accounts with the same sharing arrangement, do those crash in the same way? Or is it specific to the two original accounts? Hi Ellie, I made other tests... It's specific to the user/a account. When user/a SELECT a shared folder where it has access, the imap process crashes. I updated to cyrus-imapd-2.4.20 and the issue still happens. I also tried to rebuild mailboxes.db with ctl_mboxlist -d -u, I tried to rebuild mailboxes.db with an old backup... I tried to delete the statuscache and to delete the skipstamp db... the problem still happens. Thank you Marco From michael.menge at zdv.uni-tuebingen.de Wed Nov 13 03:51:42 2019 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Wed, 13 Nov 2019 09:51:42 +0100 Subject: [Help] Cyrus 2.4.17 segfault In-Reply-To: <5d970028-0faf-06e2-2142-8c2495f767f1@ruparpiemonte.it> References: <65974aa5-1adb-23dc-16d5-7d5b405a2d87@ruparpiemonte.it> <7c3be149-87ad-45fb-b4d6-e062970a5350@www.fastmail.com> <5d970028-0faf-06e2-2142-8c2495f767f1@ruparpiemonte.it> Message-ID: <20191113095142.Horde.Iz1_-uN9oKZv_3OinSJRQP5@webmail.uni-tuebingen.de> Hi Marco, can you generate a coredump and generate a backtrace (e.g. with gdb https://stackoverflow.com/questions/5115613/core-dump-file-analysis ) You might need to install some debug packages. Quoting Marco : > Il 12/11/2019 22:58, ellie timoney ha scritto: >> On Tue, Nov 12, 2019, at 2:51 AM, Marco wrote: >>> An user user/a has full ACL to another mailbox user/b. When the user/a >>> SELECT a folder on user/b where he has access the imap process crashes. >> >> If you set up a couple of test accounts with the same sharing >> arrangement, do those crash in the same way? Or is it specific to >> the two original accounts? > > Hi Ellie, > > I made other tests... It's specific to the user/a account. > > When user/a SELECT a shared folder where it has access, the imap > process crashes. > > I updated to cyrus-imapd-2.4.20 and the issue still happens. > > I also tried to rebuild mailboxes.db with ctl_mboxlist -d -u, I > tried to rebuild mailboxes.db with an old backup... I tried to > delete the statuscache and to delete the skipstamp db... the problem > still happens. > > Thank you > Marco > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From falon at ruparpiemonte.it Thu Nov 14 03:10:58 2019 From: falon at ruparpiemonte.it (Marco) Date: Thu, 14 Nov 2019 09:10:58 +0100 Subject: [Help] Cyrus 2.4.17 segfault In-Reply-To: <20191113095142.Horde.Iz1_-uN9oKZv_3OinSJRQP5@webmail.uni-tuebingen.de> References: <65974aa5-1adb-23dc-16d5-7d5b405a2d87@ruparpiemonte.it> <7c3be149-87ad-45fb-b4d6-e062970a5350@www.fastmail.com> <5d970028-0faf-06e2-2142-8c2495f767f1@ruparpiemonte.it> <20191113095142.Horde.Iz1_-uN9oKZv_3OinSJRQP5@webmail.uni-tuebingen.de> Message-ID: <7fd0af44-c821-c8ce-7d2d-aad22d2baa4e@ruparpiemonte.it> Hello Michael, I've generated the dump, but it isn't easy for me to understand the content... [root at cyrus ccpp-2019-11-13-07:04:30-19177]# gdb coredump GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Missing separate debuginfo for the main executable file Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/59/d016b26d3d62fcd00eb1400ff1be9c0142ee7c [New Thread 19177] Core was generated by `imapd:'. Program terminated with signal 7, Bus error. #0 0x00007f3af70cd122 in ?? () "/var/spool/abrt/ccpp-2019-11-13-07:04:30-19177/coredump" is a core file. Please specify an executable to debug. (gdb) bt full #0 0x00007f3af70cd122 in ?? () No symbol table info available. #1 0x00007f3af757fcb0 in ?? () No symbol table info available. #2 0x00007f3af70cf008 in ?? () No symbol table info available. #3 0x00007fff00000000 in ?? () No symbol table info available. #4 0x00007f3af757fcb0 in ?? () No symbol table info available. #5 0x00007f3af757fcc0 in ?? () No symbol table info available. #6 0x00007f3af757fcc8 in ?? () No symbol table info available. #7 0x00007fff6dec5770 in ?? () No symbol table info available. #8 0x00007f3af757c8b0 in ?? () No symbol table info available. #9 0x0000000000000001 in ?? () No symbol table info available. #10 0x00007f3a00000fbc in ?? () No symbol table info available. #11 0x00007fff6dec5820 in ?? () No symbol table info available. #12 0x0000000000000000 in ?? () No symbol table info available. All reconstruct on user/a was useless. Now we have resolved deleting user/a at all. Thank you Marco Il 13/11/2019 09:51, Michael Menge ha scritto: > Hi Marco, > > can you generate a coredump and generate a backtrace (e.g. with gdb > https://stackoverflow.com/questions/5115613/core-dump-file-analysis ) > You might need to install some debug packages. > > > > > Quoting Marco : > >> Il 12/11/2019 22:58, ellie timoney ha scritto: >>> On Tue, Nov 12, 2019, at 2:51 AM, Marco wrote: >>>> An user user/a has full ACL to another mailbox user/b. When the user/a >>>> SELECT a folder on user/b where he has access the imap process crashes. >>> >>> If you set up a couple of test accounts with the same sharing >>> arrangement, do those crash in the same way?? Or is it specific to >>> the two original accounts? >> >> Hi Ellie, >> >> ?I made other tests... It's specific to the user/a account. >> >> When user/a SELECT a shared folder where it has access, the imap >> process crashes. >> >> I updated to cyrus-imapd-2.4.20 and the issue still happens. >> >> I also tried to rebuild mailboxes.db with ctl_mboxlist -d -u, I tried >> to rebuild mailboxes.db with an old backup... I tried to delete the >> statuscache and to delete the skipstamp db... the problem still happens. >> >> Thank you >> Marco >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > > > -------------------------------------------------------------------------------- > > M.Menge??????????????????????????????? Tel.: (49) 7071/29-70316 > Universit?t T?bingen?????????????????? Fax.: (49) 7071/29-5912 > Zentrum f?r Datenverarbeitung????????? mail: > michael.menge at zdv.uni-tuebingen.de > W?chterstra?e 76 > 72074 T?bingen > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From michael.menge at zdv.uni-tuebingen.de Thu Nov 14 03:44:08 2019 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Thu, 14 Nov 2019 09:44:08 +0100 Subject: [Help] Cyrus 2.4.17 segfault In-Reply-To: <7fd0af44-c821-c8ce-7d2d-aad22d2baa4e@ruparpiemonte.it> References: <65974aa5-1adb-23dc-16d5-7d5b405a2d87@ruparpiemonte.it> <7c3be149-87ad-45fb-b4d6-e062970a5350@www.fastmail.com> <5d970028-0faf-06e2-2142-8c2495f767f1@ruparpiemonte.it> <20191113095142.Horde.Iz1_-uN9oKZv_3OinSJRQP5@webmail.uni-tuebingen.de> <7fd0af44-c821-c8ce-7d2d-aad22d2baa4e@ruparpiemonte.it> Message-ID: <20191114094408.Horde.y1NbS8-Rp9VxR7Jc7VLMCxk@webmail.uni-tuebingen.de> Quoting Marco : > Hello Michael, > I've generated the dump, but it isn't easy for me to understand > the content... > > [root at cyrus ccpp-2019-11-13-07:04:30-19177]# gdb coredump > GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6) > Copyright (C) 2010 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu". > For bug reporting instructions, please see: > ... > Missing separate debuginfo for the main executable file here is the why only addresses are show and not function names you need the debug package for your cyrus imapd > Try: yum --disablerepo='*' --enablerepo='*-debug*' install > /usr/lib/debug/.build-id/59/d016b26d3d62fcd00eb1400ff1be9c0142ee7c > [New Thread 19177] > Core was generated by `imapd:'. > Program terminated with signal 7, Bus error. > #0 0x00007f3af70cd122 in ?? () > "/var/spool/abrt/ccpp-2019-11-13-07:04:30-19177/coredump" is a core file. > Please specify an executable to debug. > (gdb) bt full > #0 0x00007f3af70cd122 in ?? () > No symbol table info available. > #1 0x00007f3af757fcb0 in ?? () > No symbol table info available. > #2 0x00007f3af70cf008 in ?? () > No symbol table info available. > #3 0x00007fff00000000 in ?? () > No symbol table info available. > #4 0x00007f3af757fcb0 in ?? () > No symbol table info available. > #5 0x00007f3af757fcc0 in ?? () > No symbol table info available. > #6 0x00007f3af757fcc8 in ?? () > No symbol table info available. > #7 0x00007fff6dec5770 in ?? () > No symbol table info available. > #8 0x00007f3af757c8b0 in ?? () > No symbol table info available. > #9 0x0000000000000001 in ?? () > No symbol table info available. > #10 0x00007f3a00000fbc in ?? () > No symbol table info available. > #11 0x00007fff6dec5820 in ?? () > No symbol table info available. > #12 0x0000000000000000 in ?? () > No symbol table info available. > -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From cyrus at geoff.dj Thu Nov 14 06:26:25 2019 From: cyrus at geoff.dj (Geoff Winkless) Date: Thu, 14 Nov 2019 11:26:25 +0000 Subject: [Help] Cyrus 2.4.17 segfault In-Reply-To: <20191114094408.Horde.y1NbS8-Rp9VxR7Jc7VLMCxk@webmail.uni-tuebingen.de> References: <65974aa5-1adb-23dc-16d5-7d5b405a2d87@ruparpiemonte.it> <7c3be149-87ad-45fb-b4d6-e062970a5350@www.fastmail.com> <5d970028-0faf-06e2-2142-8c2495f767f1@ruparpiemonte.it> <20191113095142.Horde.Iz1_-uN9oKZv_3OinSJRQP5@webmail.uni-tuebingen.de> <7fd0af44-c821-c8ce-7d2d-aad22d2baa4e@ruparpiemonte.it> <20191114094408.Horde.y1NbS8-Rp9VxR7Jc7VLMCxk@webmail.uni-tuebingen.de> Message-ID: On Thu, 14 Nov 2019 at 08:44, Michael Menge wrote: > here is the why only addresses are show and not function names > you need the debug package for your cyrus imapd To be fair, you also need to run gdb /path/to/imapd coredump as per > "/var/spool/abrt/ccpp-2019-11-13-07:04:30-19177/coredump" is a core file. > Please specify an executable to debug. But yes, a debug version of imapd will make the dump more useful. Geoff From ngatsis at qbit.gr Thu Nov 14 06:38:03 2019 From: ngatsis at qbit.gr (Nikos Gatsis - Qbit) Date: Thu, 14 Nov 2019 13:38:03 +0200 Subject: Centos 8 Message-ID: <1a0a7b6e-f83c-f9f9-982a-e54730ced4bd@qbit.gr> Hello list. Does somebody knows witch version on cyrus-imapd centos 8 install? Thank you in advance. Nikos. From boutilpj at ednet.ns.ca Thu Nov 14 06:49:00 2019 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Thu, 14 Nov 2019 07:49:00 -0400 Subject: Centos 8 In-Reply-To: <1a0a7b6e-f83c-f9f9-982a-e54730ced4bd@qbit.gr> References: <1a0a7b6e-f83c-f9f9-982a-e54730ced4bd@qbit.gr> Message-ID: On 11/14/19 7:38 AM, Nikos Gatsis - Qbit wrote: > Hello list. > > Does somebody knows witch version on cyrus-imapd centos 8 install? > > Thank you in advance. yum list|grep cyrus-imapd cyrus-imapd.i686 3.0.7-15.el8_0.1 AppStream cyrus-imapd.x86_64 3.0.7-15.el8_0.1 AppStream cyrus-imapd-utils.x86_64 3.0.7-15.el8_0.1 AppStream cyrus-imapd-vzic.x86_64 3.0.7-15.el8_0.1 AppStream > > Nikos. > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------- next part -------------- A non-text attachment was scrubbed... Name: boutilpj.vcf Type: text/x-vcard Size: 286 bytes Desc: not available URL: From ngatsis at qbit.gr Thu Nov 14 06:58:12 2019 From: ngatsis at qbit.gr (Nikos Gatsis - Qbit) Date: Thu, 14 Nov 2019 13:58:12 +0200 Subject: Centos 8 In-Reply-To: References: <1a0a7b6e-f83c-f9f9-982a-e54730ced4bd@qbit.gr> Message-ID: <1a3e9994-c12a-9a81-0ed4-9f801c7cf62f@qbit.gr> An HTML attachment was scrubbed... URL: From ellie at fastmail.com Thu Nov 14 19:11:56 2019 From: ellie at fastmail.com (ellie timoney) Date: Fri, 15 Nov 2019 11:11:56 +1100 Subject: Cyrus IMAP 2.5.14 released Message-ID: The Cyrus team is proud to announce the immediate availability of a new version of Cyrus IMAP: 2.5.14 This release contains a fix for CVE-2019-18928, a session hijacking vulnerability in the httpd daemon. If you compile cyrus with HTTP support enabled, your cyrus.conf contains SERVICES entries that run the httpd daemon, and you provide a proxy frontend service that reuses connections to the backend, you will need this upgrade. I'm trialling hosting the release files using Github's releases feature. Please use the Github download links if possible, and advise if you have any problems! (It may even download faster due to Github's content delivery network.) Download URLs: https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.14/cyrus-imapd-2.5.14.tar.gz https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.14/cyrus-imapd-2.5.14.tar.gz.sig https://www.cyrusimap.org/releases/cyrus-imapd-2.5.14.tar.gz https://www.cyrusimap.org/releases/cyrus-imapd-2.5.14.tar.gz.sig Please consult the release notes before upgrading to 2.5.14: https://www.cyrusimap.org/imap/download/release-notes/2.5/x/2.5.14.html And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report issues, join in the deliberations of new features for the next Cyrus IMAP release, and to contribute to the documentation. On behalf of the Cyrus team, Kind regards, ellie timoney From ellie at fastmail.com Thu Nov 14 19:13:01 2019 From: ellie at fastmail.com (ellie timoney) Date: Fri, 15 Nov 2019 11:13:01 +1100 Subject: Cyrus IMAP 3.0.12 released Message-ID: The Cyrus team is proud to announce the immediate availability of a new version of Cyrus IMAP: 3.0.12 I'm trialling hosting the release files using Github's releases feature. Please use the Github download links if possible, and advise if you have any problems! (It may even download faster due to Github's content delivery network.) This release contains a fix for CVE-2019-18928, a session hijacking vulnerability in the httpd daemon. If you compile cyrus with HTTP support enabled, your cyrus.conf contains SERVICES entries that run the httpd daemon, and you provide a proxy frontend service that reuses connections to the backend, you will need this upgrade. Download URLs: https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.12/cyrus-imapd-3.0.12.tar.gz https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.12/cyrus-imapd-3.0.12.tar.gz.sig https://www.cyrusimap.org/releases/cyrus-imapd-3.0.12.tar.gz https://www.cyrusimap.org/releases/cyrus-imapd-3.0.12.tar.gz.sig Please consult the release notes and upgrade documentation before upgrading to 3.0.12: https://www.cyrusimap.org/imap/download/release-notes/3.0/x/3.0.12.html https://www.cyrusimap.org/imap/download/upgrade.html And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report issues, join in the deliberations of new features for the next Cyrus IMAP release, and to contribute to the documentation. On behalf of the Cyrus team, Kind regards, ellie timoney From debbiep at polyfoam.com.au Fri Nov 15 00:12:21 2019 From: debbiep at polyfoam.com.au (Deborah Pickett) Date: Fri, 15 Nov 2019 16:12:21 +1100 Subject: Backup compaction optimization in a block-level replication environment In-Reply-To: <8b0b8c9c-7bb8-4397-af47-a592996ae3b2@www.fastmail.com> References: <2ef90a7b-db81-59ff-c6fd-9d54cadd9889@polyfoam.com.au> <95f69ac4-00a6-4ec0-95dd-e449ec3b97cd@www.fastmail.com> <8461a83f-a986-a164-1790-331e62888075@polyfoam.com.au> <8b0b8c9c-7bb8-4397-af47-a592996ae3b2@www.fastmail.com> Message-ID: On 2019-11-11 11:10, ellie timoney wrote: >>> This setting might be helpful: >> Thanks, I saw that setting but didn't really think through how it would >> help me.? I'll experiment with it and report back. > That would be great, thanks! Progress report: I started with very large chunks (minimum 64 MB, maximum 1024 MB) and a threshold of 8 chunks but I found that compaction was running every time, even on a backup file that hardly changed.? Not certain why this would be; my current theory is that in chunks that size there is almost always some benefit to compacting, so the threshold is passed easily.? There were 41 chunks in my %SHARED backup. I'm now trying very small chunks (no minimum size, maximum 128 kB) with varying thresholds.? This is probably _too_ small (smaller than even some messages).? I'll bisect the difference and see if there is a sweet spot. I've settled on rsync as the transport protocol for sending the backups off site.? Its rolling-checksum algorithm means that even if a chunk moves within the file it can still be transmitted efficiently, saving me from needing all that fragmentation guff I started this thread with. Related: I had to apply the patch described in (https://www.mail-archive.com/info-cyrus at lists.andrew.cmu.edu/msg47320.html), "backupd IOERROR reading backup files larger than 2GB", because during initial population of my backup, chunks tended to by multiple GB in size (my %SHARED user backup is 23 GB, compressed).? Will this patch be merged to a main line? -- *Deborah Pickett* System Administrator *Polyfoam Australia Pty Ltd* From debbiep at polyfoam.com.au Fri Nov 15 01:12:12 2019 From: debbiep at polyfoam.com.au (Deborah Pickett) Date: Fri, 15 Nov 2019 17:12:12 +1100 Subject: Cyrus backups and deleted (users, mailboxes) Message-ID: <58a0a2ef-4320-2cb0-1e99-befe25246ace@polyfoam.com.au> Semi-related questions about how Cyrus backup servers deal with deleted stuff. I have the following settings on the main mail server: deletedprefix: DELETED delete_mode: delayed expunge_mode: delayed My first question is about deleted folders.? Scenario: I delete a folder on the server, and it is renamed into the DELETED namespace as expected.? I cannot back these folders up to the backup server.? They are not enumerated during "XBACKUP * channel", and if I name the DELETED namespace explicitly I get the following output: x xbackup DELETED/* rsync * NO MAILBOX polyfoam.com.au!DELETED.rawmaterials.Trash.Downer EDI.5DAD45A0 (Mailbox is locked) * NO MAILBOX polyfoam.com.au!DELETED.support.Has Spaces.5DCA22F4 (Mailbox is locked) x NO Mailbox is locked (Which is not nearly the whole list either: it should have listed hundreds of deleted folders.) Is this expected behaviour?? I suppose that my backup has a copy of this folder in its original location before it was deleted, so no one will miss these folders, (though I can imagine scenarios where a message arrives and is quickly put into a folder which is then deleted, and this message will never touch the backup server). My second question is about deleted users.? Scenario: A user "alice" departs the company, and I delete their root mailbox on the main server.? It is no longer enumerated during "XBACKUP * channel". What will happen to the backup file a/alice_XXXXXX on the backup server?? Will it stay there forever?? Do I need to manually delete it?? Do I need to manually remove alice from /var/lib/cyrus/backups.db?? I suppose I can use "ctl_backups list -t" to detect such files.? Any other things I should be aware of? -- *Deborah Pickett* System Administrator *Polyfoam Australia Pty Ltd* From debbiep at polyfoam.com.au Fri Nov 15 20:10:45 2019 From: debbiep at polyfoam.com.au (Deborah Pickett) Date: Sat, 16 Nov 2019 12:10:45 +1100 Subject: Backup compaction optimization in a block-level replication environment In-Reply-To: References: <2ef90a7b-db81-59ff-c6fd-9d54cadd9889@polyfoam.com.au> <95f69ac4-00a6-4ec0-95dd-e449ec3b97cd@www.fastmail.com> <8461a83f-a986-a164-1790-331e62888075@polyfoam.com.au> <8b0b8c9c-7bb8-4397-af47-a592996ae3b2@www.fastmail.com> Message-ID: Further progress report: with small chunks, compaction takes about 15 times longer.? It's almost as if there is an O(n^2) complexity somewhere, looking at the rate that the disk file grows.? (Running perf on a compaction suggests that 90% of the time ctl_backups is doing compression, decompression, or calculating SHA1 hashes.) So I'm going back to large-ish chunks again.? Current values: backup_compact_minsize: 1024 backup_compact_maxsize: 65536 backup_compact_work_threshold: 10 The compression ratio was hardly any different (less than 1%) with many small chunks compared with huge chunks. Setting the work threshold to a number greater than 1 is only helping a bit.? I think that the huge disparity between my smaller and larger user backups is hurting me here.? Whatever I set the threshold to, it is going to be simultaneously too large for most users, and too small for the huge %SHARED user. Confession time: having inspected the source of ctl_backups, I admit to misunderstanding what happens to chunks when compaction is triggered.? I thought that each chunk was examined, and either the chunk is compacted, or it is not (and the bytes in the chunk are copied from old to new unchanged).? But compaction happens to the entire file: every chunk in turn is inflated to /tmp and then deflated again from /tmp, minus any messages that may have expired, so the likelihood of the compressed byte stream being the same is slim.? That will confound the rsync rolling checksum algorithm and the entire backup file will likely have to be transmitted again. With that in mind I've decided that I'll make compaction a weekend-only task, take it out of cyrus.conf EVENTS and put a weekly cron/systemd job in place.? During the week backups will be append-only, to keep rsync happy.? At weekends, compaction will combine the last week of small chunks, and I've got all weekend to transmit the hundred GB of backup files offsite. -- Deborah Pickett System Administrator Polyfoam Australia Pty Ltd From adrien.remillieux at gmail.com Sun Nov 17 07:54:11 2019 From: adrien.remillieux at gmail.com (Adrien Remillieux) Date: Sun, 17 Nov 2019 13:54:11 +0100 Subject: Cyrus doesn't preserve hard-links on replication Message-ID: Hello, I set up replication between two cyrus servers (master runs 2.5.10 and slave 3.0.8) with plans to decommission the old server once everything is working. I noticed that the mail spool takes 950GB instead of ~300GB on the old server. I suspected the hardlinks for message deduplication weren't recreated so I ran rdfind on the mail spool and the tool found many identical files. Is there a cyrus tool to recreate the hardlinks ? I looked at the admin tools but I didn't find anything. Rdfind should work but it also matched metadata such as cyrus.annotations for example. So I need to go through the 600 mB dry-run log file to exclude unwanted files. 99% of the log file look like that : Are those safe to hardlink ? # duptype id depth size device inode priority name DUPTYPE_FIRST_OCCURRENCE 1030757 3 842 2065 330967752 3 /var/spool/cyrus/mail/c/user/user1/15384. DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 317750405 3 /var/spool/cyrus/mail/m/user/user2/19262. DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 323550741 3 /var/spool/cyrus/mail/r/user/user3/96106. DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 316733834 3 /var/spool/cyrus/mail/m/user/user4/41168. DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 314623377 3 /var/spool/cyrus/mail/m/user/user5/25377. DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 316201219 3 /var/spool/cyrus/mail/m/user/user6/49119. DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 321991878 3 /var/spool/cyrus/mail/q/user/user7/46487. Cheers, Adrien -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellie at fastmail.com Sun Nov 17 23:52:13 2019 From: ellie at fastmail.com (ellie timoney) Date: Mon, 18 Nov 2019 15:52:13 +1100 Subject: Cyrus backups and deleted (users, mailboxes) In-Reply-To: <58a0a2ef-4320-2cb0-1e99-befe25246ace@polyfoam.com.au> References: <58a0a2ef-4320-2cb0-1e99-befe25246ace@polyfoam.com.au> Message-ID: <0a06a33f-47fe-4206-bcc4-d7e3b142a6a5@www.fastmail.com> On Fri, Nov 15, 2019, at 5:12 PM, Deborah Pickett wrote: > Semi-related questions about how Cyrus backup servers deal with deleted > stuff. > > I have the following settings on the main mail server: > > deletedprefix: DELETED > delete_mode: delayed > expunge_mode: delayed > > My first question is about deleted folders.? Scenario: I delete a folder > on the server, and it is renamed into the DELETED namespace as > expected.? I cannot back these folders up to the backup server.? They > are not enumerated during "XBACKUP * channel", and if I name the DELETED > namespace explicitly I get the following output: > > x xbackup DELETED/* rsync > * NO MAILBOX polyfoam.com.au!DELETED.rawmaterials.Trash.Downer > EDI.5DAD45A0 (Mailbox is locked) > * NO MAILBOX polyfoam.com.au!DELETED.support.Has Spaces.5DCA22F4 > (Mailbox is locked) > x NO Mailbox is locked > > (Which is not nearly the whole list either: it should have listed > hundreds of deleted folders.) > > Is this expected behaviour?? I suppose that my backup has a copy of this > folder in its original location before it was deleted, so no one will > miss these folders, (though I can imagine scenarios where a message > arrives and is quickly put into a folder which is then deleted, and this > message will never touch the backup server). Hmm. With rolling replication to the backup server, these will be replicated as RENAMEs from "original" to "DELETED.original.hextimestamp", while remaining in the original user's backup. After "cyr_expire -D" deletes these folders from the mail server, then they will replicate as UNMAILBOXes, and the backup server will mark them in the index as having been deleted at the time the UNMAILBOX is replicated. Eventually, after a further backup_duration_days has passed, the messages in them will become eligible for compacting out of the backup (providing they're not also in some other folder where they are still valid), which will take effect according to the usual compact rules when ctl_backups compact runs. If it was a quick arrive-file-delete, it would eventually replicate as a new mailbox (already called DELETED.*) with all the messages, so nothing would be missed. But without rolling replication, or with mailboxes that were already DELETED.* before the backup service was configured, I'm not sure what to expect. I'm not sure if this is a case that I considered and did something for, but have since forgotten about, or if it's a case that I hadn't considered yet. What happens if you xbackup the user(s) they belong(ed) to, rather than the DELETED.* mailboxes directly? Does sync_client correctly detect them as renames, and replicate accordingly? If not, what if you sync_client -u them manually? If that doesn't work, what about sync_client -m with the DELETED.* names? If none of these work, I probably need to implement something... > My second question is about deleted users.? Scenario: A user "alice" > departs the company, and I delete their root mailbox on the main > server.? It is no longer enumerated during "XBACKUP * channel". > > What will happen to the backup file a/alice_XXXXXX on the backup > server?? Will it stay there forever?? Do I need to manually delete it?? > Do I need to manually remove alice from /var/lib/cyrus/backups.db?? I > suppose I can use "ctl_backups list -t" to detect such files. With rolling replication, the deleted user would eventually replicate as an "UNUSER", to which backupd would report success and ignore... Basically, the implementation hasn't gotten as far as removing backups -- and I want to see things working correctly and safely before I even think about starting to implement it. In the meantime, "ctl_backups list -t" is a pretty good way of identifying candidates to (thoroughly check and then) delete manually. You're right, you'd also want to remove their entry from backups.db, which you can do safely using "cyr_dbtool delete". >? Any other things I should be aware of? I guess at this point you're deeper into the woods on this than anyone else has been (which is why it's called "experimental"!). For me, it's great that you're trying stuff and shaking out the issues, but you're probably gonna find more, not less, as we go along Cheers, ellie From ellie at fastmail.com Mon Nov 18 00:36:00 2019 From: ellie at fastmail.com (ellie timoney) Date: Mon, 18 Nov 2019 16:36:00 +1100 Subject: =?UTF-8?Q?Re:_Backup_compaction_optimization_in_a_block-level_replicatio?= =?UTF-8?Q?n_environment?= In-Reply-To: References: <2ef90a7b-db81-59ff-c6fd-9d54cadd9889@polyfoam.com.au> <95f69ac4-00a6-4ec0-95dd-e449ec3b97cd@www.fastmail.com> <8461a83f-a986-a164-1790-331e62888075@polyfoam.com.au> <8b0b8c9c-7bb8-4397-af47-a592996ae3b2@www.fastmail.com> Message-ID: <0f9d2e8f-0737-4301-9c78-622ca3710fee@www.fastmail.com> > Related: I had to apply the patch described in > (https://www.mail-archive.com/info-cyrus at lists.andrew.cmu.edu/msg47320.html), > "backupd IOERROR reading backup files larger than 2GB", because during > initial population of my backup, chunks tended to by multiple GB in size > (my %SHARED user backup is 23 GB, compressed). Will this patch be > merged to a main line? Those were on master but I'm not sure why I didn't cherry-pick them back to 3.0.... Anyway, I've done that now, they'll be in the next release. > Progress report: I started with very large chunks (minimum 64 MB, > maximum 1024 MB) and a threshold of 8 chunks but I found that compaction > was running every time, even on a backup file that hardly changed. Not > certain why this would be; my current theory is that in chunks that size > there is almost always some benefit to compacting, so the threshold is > passed easily. There were 41 chunks in my %SHARED backup. Hmm. Yeah, the threshold is "number of chunks that would benefit from compaction", so the larger the chunks, the more likely any given chunk is to benefit from compaction, and the more likely you are to trip that threshold. On Sat, Nov 16, 2019, at 12:10 PM, Deborah Pickett wrote: > Further progress report: with small chunks, compaction takes about 15 > times longer.? It's almost as if there is an O(n^2) complexity > somewhere, looking at the rate that the disk file grows.? (Running perf > on a compaction suggests that 90% of the time ctl_backups is doing > compression, decompression, or calculating SHA1 hashes.) So I'm going > back to large-ish chunks again.? Current values: > > backup_compact_minsize: 1024 > backup_compact_maxsize: 65536 > backup_compact_work_threshold: 10 > > The compression ratio was hardly any different (less than 1%) with many > small chunks compared with huge chunks. That's really interesting to hear. It sounds like maybe the startup and cleanup of a gzip stream are more expensive than the compression/decompression parts, so it's cheaper to aim for fewer larger chunks than many smaller ones. zlib provides a range of 0-9 (default: 6) for whether to prioritise speed (0) or size (9) in its compression algorithm, but the backup system isn't using it in a way that exposes this as a tunable option (it's just letting it use the default by default). With enough data it might be interesting to make it tunable and see what impact it has, but I don't think we're at a stage of needing to care this much yet. > Setting the work threshold to a number greater than 1 is only helping a > bit.? I think that the huge disparity between my smaller and larger user > backups is hurting me here.? Whatever I set the threshold to, it is > going to be simultaneously too large for most users, and too small for > the huge %SHARED user. Food for thought. Maybe instead of having one "%SHARED" backup, having one "%SHARED.foo" backup per top-level shared folder would be a better implementation? I haven't seen shared folders used much in practice, so it's interesting to hear about it. Looking at your own data, if you had one "%SHARED.foo" backup per top level shared folder, would they be roughly user-sized pieces, or still too big? If too big, how deep would you need to go down the tree until the worst offenders are a manageable size? (If I make it split shared folders like this, maybe "how-deep-to-split-shared-folders" needs to be a configuration parameter, because I guess it'll vary from installation to installation.) > Confession time: having inspected the source of ctl_backups, I admit to > misunderstanding what happens to chunks when compaction is triggered.? I > thought that each chunk was examined, and either the chunk is compacted, > or it is not (and the bytes in the chunk are copied from old to new > unchanged).? But compaction happens to the entire file: every chunk in > turn is inflated to /tmp and then deflated again from /tmp, minus any > messages that may have expired, so the likelihood of the compressed byte > stream being the same is slim.? That will confound the rsync rolling > checksum algorithm and the entire backup file will likely have to be > transmitted again. Yeah, these files are append-only even within the backup system's own tooling. Compacting a backup file to be smaller is literally re-streaming it to a new file, minus bits that aren't needed anymore, and then (if all goes well) renaming it back over the original. It's meant to be atomic -- either it works, and you get the updated file, or something goes wrong, and the file is unchanged. It's never modified in place! (There's a note about this somewhere in the documentation, with regard to needing enough free disk space to write the second file in order to compact the first.) > With that in mind I've decided that I'll make compaction a weekend-only > task, take it out of cyrus.conf EVENTS and put a weekly cron/systemd job > in place.? During the week backups will be append-only, to keep rsync > happy.? At weekends, compaction will combine the last week of small > chunks, and I've got all weekend to transmit the hundred GB of backup > files offsite. That sounds like a pretty sensible approach Cheers, ellie From adrien.remillieux at gmail.com Mon Nov 18 15:47:49 2019 From: adrien.remillieux at gmail.com (Adrien Remillieux) Date: Mon, 18 Nov 2019 21:47:49 +0100 Subject: Cyrus doesn't preserve hard-links on replication In-Reply-To: References: Message-ID: By shuffling the keywords on my google searches I was able to find this: https://lists.andrew.cmu.edu/pipermail/info-cyrus/2006-March/021405.html Apparently there is a few settings to set to avoid copying the same message multiple times. This would be a nice addition to the cyrus docs on replication ! Le dim. 17 nov. 2019 ? 18:00, a ?crit : > Send Info-cyrus mailing list submissions to > info-cyrus at lists.andrew.cmu.edu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > or, via email, send a message with subject or body 'help' to > info-cyrus-request at lists.andrew.cmu.edu > > You can reach the person managing the list at > info-cyrus-owner at lists.andrew.cmu.edu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Info-cyrus digest..." > > > Today's Topics: > > 1. Cyrus doesn't preserve hard-links on replication > (Adrien Remillieux) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 17 Nov 2019 13:54:11 +0100 > From: Adrien Remillieux > To: info-cyrus at lists.andrew.cmu.edu > Subject: Cyrus doesn't preserve hard-links on replication > Message-ID: > ak+k8Ez8LiSz+cw83G1SMzU9qFFw at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hello, > > I set up replication between two cyrus servers (master runs 2.5.10 and > slave 3.0.8) with plans to decommission the old server once everything is > working. I noticed that the mail spool takes 950GB instead of ~300GB on the > old server. I suspected the hardlinks for message deduplication weren't > recreated so I ran rdfind on the mail spool and the tool found many > identical files. Is there a cyrus tool to recreate the hardlinks ? I looked > at the admin tools but I didn't find anything. Rdfind should work but it > also matched metadata such as cyrus.annotations for example. So I need to > go through the 600 mB dry-run log file to exclude unwanted files. > > 99% of the log file look like that : Are those safe to hardlink ? > > # duptype id depth size device inode priority name > DUPTYPE_FIRST_OCCURRENCE 1030757 3 842 2065 330967752 3 > /var/spool/cyrus/mail/c/user/user1/15384. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 317750405 3 > /var/spool/cyrus/mail/m/user/user2/19262. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 323550741 3 > /var/spool/cyrus/mail/r/user/user3/96106. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 316733834 3 > /var/spool/cyrus/mail/m/user/user4/41168. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 314623377 3 > /var/spool/cyrus/mail/m/user/user5/25377. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 316201219 3 > /var/spool/cyrus/mail/m/user/user6/49119. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 321991878 3 > /var/spool/cyrus/mail/q/user/user7/46487. > > Cheers, > Adrien > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20191117/60161ca7/attachment-0001.html > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Info-cyrus mailing list > Info-cyrus at lists.andrew.cmu.edu > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > > ------------------------------ > > End of Info-cyrus Digest, Vol 172, Issue 14 > ******************************************* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From debbiep at polyfoam.com.au Mon Nov 18 17:30:29 2019 From: debbiep at polyfoam.com.au (Deborah Pickett) Date: Tue, 19 Nov 2019 09:30:29 +1100 Subject: Cyrus backups and deleted (users, mailboxes) In-Reply-To: <0a06a33f-47fe-4206-bcc4-d7e3b142a6a5@www.fastmail.com> References: <58a0a2ef-4320-2cb0-1e99-befe25246ace@polyfoam.com.au> <0a06a33f-47fe-4206-bcc4-d7e3b142a6a5@www.fastmail.com> Message-ID: <18321f90-eb19-17dc-61ec-4563df18115b@polyfoam.com.au> > But without rolling replication, or with mailboxes that were already DELETED.* before the backup service was configured, I'm not sure what to expect. I'm not sure if this is a case that I considered and did something for, but have since forgotten about, or if it's a case that I hadn't considered yet. > > What happens if you xbackup the user(s) they belong(ed) to, rather than the DELETED.* mailboxes directly? Does sync_client correctly detect them as renames, and replicate accordingly? If not, what if you sync_client -u them manually? If that doesn't work, what about sync_client -m with the DELETED.* names? > You're right, the DELETED mailboxes do turn up in user backups automatically, so I think those renames are occurring as expected: # /usr/lib/cyrus/bin/cyr_backup -f cyrus-backup/partitions/default/d/debbiep\@polyfoam.com.au_* list mailboxes | grep DELETED r7vyz5wuyw3mfswj0xntumv2? 1970-01-01 10:00:00 polyfoam.com.au!DELETED.user.debbiep.Trash.Test.5D8B4FED [...] There are some DELETED entries in the %SHARED backup too, so I _think_ that renames are working there too, but I don't have enough data yet to know.? I'll go back and check when some more shared folder subfolders have been deleted. It may not be a problem that the DELETED mailboxes aren't getting enumerated in XBACKUP.? By definition, a deleted mailbox will have no further transactions on it, so there shouldn't be any need for it to be transmitted during XBACKUP.? The only unknown at this stage is what happens after the deleted mailboxes hit their expiry age: will they vanish from the backup during compaction or will they need to be removed manually?? My server isn't old enough for me to tell. > I guess at this point you're deeper into the woods on this than anyone else has been (which is why it's called "experimental"!). For me, it's great that you're trying stuff and shaking out the issues, but you're probably gonna find more, not less, as we go along Hurrah, I think. Sincere thanks for the info so far. -- Deborah Pickett System Administrator Polyfoam Australia Pty Ltd From adrien.remillieux at gmail.com Mon Nov 18 17:34:31 2019 From: adrien.remillieux at gmail.com (Adrien Remillieux) Date: Mon, 18 Nov 2019 23:34:31 +0100 Subject: Cyrus doesn't preserve hard-links on replication In-Reply-To: References: Message-ID: Sorry for the multiples emails... The option "provide_uuid=1" I found in my last message seems to be unrecognized by cyrus now (It's probably on by default). It's probably only useful for new messages anyway. So I'm back to square one and my google-fu failed me. If someone know how to solve that problem any help would be greatly appreciated. Cheers, Adrien Le dim. 17 nov. 2019 ? 18:00, a ?crit : > Send Info-cyrus mailing list submissions to > info-cyrus at lists.andrew.cmu.edu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > or, via email, send a message with subject or body 'help' to > info-cyrus-request at lists.andrew.cmu.edu > > You can reach the person managing the list at > info-cyrus-owner at lists.andrew.cmu.edu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Info-cyrus digest..." > > > Today's Topics: > > 1. Cyrus doesn't preserve hard-links on replication > (Adrien Remillieux) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 17 Nov 2019 13:54:11 +0100 > From: Adrien Remillieux > To: info-cyrus at lists.andrew.cmu.edu > Subject: Cyrus doesn't preserve hard-links on replication > Message-ID: > ak+k8Ez8LiSz+cw83G1SMzU9qFFw at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hello, > > I set up replication between two cyrus servers (master runs 2.5.10 and > slave 3.0.8) with plans to decommission the old server once everything is > working. I noticed that the mail spool takes 950GB instead of ~300GB on the > old server. I suspected the hardlinks for message deduplication weren't > recreated so I ran rdfind on the mail spool and the tool found many > identical files. Is there a cyrus tool to recreate the hardlinks ? I looked > at the admin tools but I didn't find anything. Rdfind should work but it > also matched metadata such as cyrus.annotations for example. So I need to > go through the 600 mB dry-run log file to exclude unwanted files. > > 99% of the log file look like that : Are those safe to hardlink ? > > # duptype id depth size device inode priority name > DUPTYPE_FIRST_OCCURRENCE 1030757 3 842 2065 330967752 3 > /var/spool/cyrus/mail/c/user/user1/15384. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 317750405 3 > /var/spool/cyrus/mail/m/user/user2/19262. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 323550741 3 > /var/spool/cyrus/mail/r/user/user3/96106. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 316733834 3 > /var/spool/cyrus/mail/m/user/user4/41168. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 314623377 3 > /var/spool/cyrus/mail/m/user/user5/25377. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 316201219 3 > /var/spool/cyrus/mail/m/user/user6/49119. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 321991878 3 > /var/spool/cyrus/mail/q/user/user7/46487. > > Cheers, > Adrien > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20191117/60161ca7/attachment-0001.html > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Info-cyrus mailing list > Info-cyrus at lists.andrew.cmu.edu > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > > ------------------------------ > > End of Info-cyrus Digest, Vol 172, Issue 14 > ******************************************* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From debbiep at polyfoam.com.au Mon Nov 18 17:38:28 2019 From: debbiep at polyfoam.com.au (Deborah Pickett) Date: Tue, 19 Nov 2019 09:38:28 +1100 Subject: Backup compaction optimization in a block-level replication environment In-Reply-To: <0f9d2e8f-0737-4301-9c78-622ca3710fee@www.fastmail.com> References: <2ef90a7b-db81-59ff-c6fd-9d54cadd9889@polyfoam.com.au> <95f69ac4-00a6-4ec0-95dd-e449ec3b97cd@www.fastmail.com> <8461a83f-a986-a164-1790-331e62888075@polyfoam.com.au> <8b0b8c9c-7bb8-4397-af47-a592996ae3b2@www.fastmail.com> <0f9d2e8f-0737-4301-9c78-622ca3710fee@www.fastmail.com> Message-ID: > Food for thought. Maybe instead of having one "%SHARED" backup, having one "%SHARED.foo" backup per top-level shared folder would be a better implementation? I haven't seen shared folders used much in practice, so it's interesting to hear about it. > > Looking at your own data, if you had one "%SHARED.foo" backup per top level shared folder, would they be roughly user-sized pieces, or still too big? If too big, how deep would you need to go down the tree until the worst offenders are a manageable size? (If I make it split shared folders like this, maybe "how-deep-to-split-shared-folders" needs to be a configuration parameter, because I guess it'll vary from installation to installation.) > For my data, %SHARED.foo would be the perfect granularity level. Each foo is a shared email address like "sales" or "accounts" and it gets about as much traffic as a user account does.? (Two months ago when we were on Exchange, they _were_ user accounts.) foo also includes "#calendars" and "#addressbooks" on my server so there are weird characters to deal with. -- Deborah Pickett System Administrator Polyfoam Australia Pty Ltd From egoitz at ramattack.net Tue Nov 19 04:40:13 2019 From: egoitz at ramattack.net (Egoitz Aurrekoetxea) Date: Tue, 19 Nov 2019 10:40:13 +0100 Subject: Cyrus 3, automation and master-master replication and mailbox movement Message-ID: <2FB1BA17-4521-4E5F-B06F-0BA617FB221C@ramattack.net> Good morning, I have been checking how could we manage for automating master -> slave and slave->master transition. I though one possibility could be having both servers configured in master mode with each being replicating to the other one. I know this was some time ago unsupported and have tried if it worked now in a testing env but it seems it fails too? Could any Cyrus guru confirm that really this does not work (just for avoiding driving myself crazy trying to find the config issue)?. By the way, I?m in progress too of automating mailbox movements? from partition, from server? I had a question about renaming mailboxes for moving from cyrus partition? as a safety measure, prior to launch a renm operation (renm user/aa at bb.es user/aa at bb.es different-partition) we block any kind of access to that mailbox (even mail delivering)? and I was wondering if that is really necessary nowadays?. or does Cyrus hold that locks by it?s own?. I mean does Cyrus take care by it?s own, of avoiding mailbox corruption due to a renm mailbox to a different partition?. Just one more question? when we move a mailbox from a partition (a renm to different partition) to another one? we usually do : - stop replication between master/slave (as a safety measure for having a very last ?fall back? if the renm goes wrong). You know, promoting the slave to master would have the mailbox of the failed renaming operation properly... - renm in the master - after successful rename, delete from the slave the mailboxes - sync each of the master mailboxes to the slave? this way among other things, the removed mailboxes in the slave (the dm is done in the slave for causing mailboxes to be resynced again from the master to the slave to it?s new location in the slave) - start replication again? Are all this steps really necessary?. What do you think about it?. Best regards, -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellie at fastmail.com Tue Nov 19 18:03:13 2019 From: ellie at fastmail.com (ellie timoney) Date: Wed, 20 Nov 2019 10:03:13 +1100 Subject: =?UTF-8?Q?Re:_Backup_compaction_optimization_in_a_block-level_replicatio?= =?UTF-8?Q?n_environment?= In-Reply-To: References: <2ef90a7b-db81-59ff-c6fd-9d54cadd9889@polyfoam.com.au> <95f69ac4-00a6-4ec0-95dd-e449ec3b97cd@www.fastmail.com> <8461a83f-a986-a164-1790-331e62888075@polyfoam.com.au> <8b0b8c9c-7bb8-4397-af47-a592996ae3b2@www.fastmail.com> <0f9d2e8f-0737-4301-9c78-622ca3710fee@www.fastmail.com> Message-ID: <5f8afb4d-e9cc-4135-a2b8-18bd362d8a3c@www.fastmail.com> On Tue, Nov 19, 2019, at 9:38 AM, Deborah Pickett wrote: > > Food for thought. Maybe instead of having one "%SHARED" backup, having one "%SHARED.foo" backup per top-level shared folder would be a better implementation? I haven't seen shared folders used much in practice, so it's interesting to hear about it. > > > > Looking at your own data, if you had one "%SHARED.foo" backup per top level shared folder, would they be roughly user-sized pieces, or still too big? If too big, how deep would you need to go down the tree until the worst offenders are a manageable size? (If I make it split shared folders like this, maybe "how-deep-to-split-shared-folders" needs to be a configuration parameter, because I guess it'll vary from installation to installation.) > > > For my data, %SHARED.foo would be the perfect granularity level. Each > foo is a shared email address like "sales" or "accounts" and it gets > about as much traffic as a user account does.? (Two months ago when we > were on Exchange, they _were_ user accounts.) Ah yep, that makes sense! > foo also includes "#calendars" and "#addressbooks" on my server so there > are weird characters to deal with. Now that's an interesting detail to consider. I think, with a hypothetical depth setting, I would treat any level that contains '#directories' as being "too deep" for splitting, regardless of the depth setting, because at that point we're looking at things that I guess we expect to belong together. Like, if hypothetical_depth is 3, but foo.#calendars exists, then I think we'd want to treat the entirety of foo as a single backup (as if hypothetical_depth were 1), regardless of what else is deeper in there. I need to think about this more. I'm gonna have a go at implementing this (I've opened https://github.com/cyrusimap/cyrus-imapd/issues/2915) but I'll step through it one level of complication at a time. From debbiep at polyfoam.com.au Tue Nov 19 19:06:58 2019 From: debbiep at polyfoam.com.au (Deborah Pickett) Date: Wed, 20 Nov 2019 11:06:58 +1100 Subject: Backup compaction optimization in a block-level replication environment In-Reply-To: <5f8afb4d-e9cc-4135-a2b8-18bd362d8a3c@www.fastmail.com> References: <2ef90a7b-db81-59ff-c6fd-9d54cadd9889@polyfoam.com.au> <95f69ac4-00a6-4ec0-95dd-e449ec3b97cd@www.fastmail.com> <8461a83f-a986-a164-1790-331e62888075@polyfoam.com.au> <8b0b8c9c-7bb8-4397-af47-a592996ae3b2@www.fastmail.com> <0f9d2e8f-0737-4301-9c78-622ca3710fee@www.fastmail.com> <5f8afb4d-e9cc-4135-a2b8-18bd362d8a3c@www.fastmail.com> Message-ID: <68739b6f-27f5-5b5c-5af3-7b36bb57f27e@polyfoam.com.au> On 2019-11-20 10:03, ellie timoney wrote: >> foo also includes "#calendars" and "#addressbooks" on my server so there >> are weird characters to deal with. > Now that's an interesting detail to consider. > I should restate my original message because I'm being fast and loose with the meaning of "contains": two of the values for foo on my server are "#calendars" and "#addressbooks".? In other words, there are top-level public mailboxes #calendars and #addressbooks which themselves contain sub-calendars and sub-addressbooks.? It never occurred to me to have calendar or contacts folders deeper in the normal shared folder namespace, though it has evidently occurred to you. In any case, I would only use hypothetical_depth = 1 so this wouldn't be an issue for my backups. -- *Deborah Pickett* System Administrator Polyfoam *Polyfoam Australia Pty Ltd* T: +61 (3) 9794 8320 | F: +61 (3) 9791 1222 | M: +61 408 962 109 E: debbiep at polyfoam.com.au | W: www.polyfoam.com.au /Proudly Australian owned and operated for over 30 years/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: nakkkcncappnhokg.png Type: image/png Size: 3525 bytes Desc: not available URL: From ellie at fastmail.com Tue Nov 19 19:29:18 2019 From: ellie at fastmail.com (ellie timoney) Date: Wed, 20 Nov 2019 11:29:18 +1100 Subject: =?UTF-8?Q?Re:_Backup_compaction_optimization_in_a_block-level_replicatio?= =?UTF-8?Q?n_environment?= In-Reply-To: <68739b6f-27f5-5b5c-5af3-7b36bb57f27e@polyfoam.com.au> References: <2ef90a7b-db81-59ff-c6fd-9d54cadd9889@polyfoam.com.au> <95f69ac4-00a6-4ec0-95dd-e449ec3b97cd@www.fastmail.com> <8461a83f-a986-a164-1790-331e62888075@polyfoam.com.au> <8b0b8c9c-7bb8-4397-af47-a592996ae3b2@www.fastmail.com> <0f9d2e8f-0737-4301-9c78-622ca3710fee@www.fastmail.com> <5f8afb4d-e9cc-4135-a2b8-18bd362d8a3c@www.fastmail.com> <68739b6f-27f5-5b5c-5af3-7b36bb57f27e@polyfoam.com.au> Message-ID: <8d2fd555-9fd1-459c-9c74-b974a489e9e8@www.fastmail.com> On Wed, Nov 20, 2019, at 11:06 AM, Deborah Pickett wrote: > On 2019-11-20 10:03, ellie timoney wrote: >>> foo also includes "#calendars" and "#addressbooks" on my server so there are weird characters to deal with. >>> >> Now that's an interesting detail to consider. >> > I should restate my original message because I'm being fast and loose with the meaning of "contains": two of the values for foo on my server are "#calendars" and "#addressbooks". In other words, there are top-level public mailboxes #calendars and #addressbooks which themselves contain sub-calendars and sub-addressbooks. It never occurred to me to have calendar or contacts folders deeper in the normal shared folder namespace, though it has evidently occurred to you. Oh, I see how I misread that! And... that also complicates things for me, I think (well, it's a possibility I hadn't even considered). I'm curious how these are working for you, or what sort of configuration and workflows leads to having #calendars and #addressbooks as top-level shared mailboxes? I've only very recently started learning how our DAV bits work (they have previously been black-boxes for me), and so far have only seen these existing in user accounts. Maybe this is a separate thread though. -------------- next part -------------- An HTML attachment was scrubbed... URL: From debbiep at polyfoam.com.au Wed Nov 20 00:41:14 2019 From: debbiep at polyfoam.com.au (Deborah Pickett) Date: Wed, 20 Nov 2019 16:41:14 +1100 Subject: Public calendars and addressbooks (was RE: Backup compaction optimization in a block-level replication environment) Message-ID: <00c401d59f65$1fb4bef0$5f1e3cd0$@polyfoam.com.au> > I'm curious how these are working for you, or what sort of configuration and workflows leads to having #calendars and #addressbooks as top-level shared mailboxes? ?I've only very recently started learning how our DAV bits work (they have previously been black-boxes for me), and so far have only seen these existing in user accounts. ?Maybe this is a separate thread though. We used to use public calendars in Exchange (they call them Public Folders) for, among other things, a read-only catalogue of who in the office is on leave on any given day. Some of our branch offices also had shared contact lists for phone numbers likely to be needed by all people at the local site (the local takeaway, the local hardware store, the local clinic...). Exchange public folders are almost an exact analogue to shared-namespace mailboxes in Cyrus. Once I learned the undocumented magic for creating public calendars and address books in Cyrus (@karagian on Github posted it: https://github.com/cyrusimap/cyrus-imapd/issues/2373#issuecomment-415738943) it's worked very well. My Outlook users use the free Caldav Synchronizer plugin (https://caldavsynchronizer.org/) to sync selected address books and calendars to their clients. I have a Perl script that queries our Active Directory server over LDAP to set ACLs on the folders. -- Deborah Pickett System Administrator Polyfoam Australia Pty Ltd From ellie at fastmail.com Thu Nov 21 17:54:38 2019 From: ellie at fastmail.com (ellie timoney) Date: Fri, 22 Nov 2019 09:54:38 +1100 Subject: =?UTF-8?Q?Re:_Public_calendars_and_addressbooks_(was_RE:_Backup_compacti?= =?UTF-8?Q?on_optimization_in_a_block-level_replication_environment)?= In-Reply-To: <00c401d59f65$1fb4bef0$5f1e3cd0$@polyfoam.com.au> References: <00c401d59f65$1fb4bef0$5f1e3cd0$@polyfoam.com.au> Message-ID: <2f3f957e-174a-4d28-ac7e-64c49828c174@www.fastmail.com> On Wed, Nov 20, 2019, at 4:41 PM, Deborah Pickett wrote: > > I'm curious how these are working for you, or what sort of configuration > > and workflows leads to having #calendars and #addressbooks as top-level > > shared mailboxes? ?I've only very recently started learning how our DAV bits > > work (they have previously been black-boxes for me), and so far have only > > seen these existing in user accounts. ?Maybe this is a separate thread > > though. > > We used to use public calendars in Exchange (they call them Public Folders) > for, among other things, a read-only catalogue of who in the office is on > leave on any given day. Some of our branch offices also had shared contact > lists for phone numbers likely to be needed by all people at the local site > (the local takeaway, the local hardware store, the local clinic...). > Exchange public folders are almost an exact analogue to shared-namespace > mailboxes in Cyrus. > > Once I learned the undocumented magic for creating public calendars and > address books in Cyrus (@karagian on Github posted it: > https://github.com/cyrusimap/cyrus-imapd/issues/2373#issuecomment-415738943) > it's worked very well. My Outlook users use the free Caldav Synchronizer > plugin (https://caldavsynchronizer.org/) to sync selected address books and > calendars to their clients. I have a Perl script that queries our Active > Directory server over LDAP to set ACLs on the folders. That's interesting, thanks! From johan at hattne.se Thu Nov 28 02:40:52 2019 From: johan at hattne.se (Johan Hattne) Date: Wed, 27 Nov 2019 23:40:52 -0800 Subject: Cyrus webdav with Joplin Message-ID: Dear all; I?m trying to get Joplin (https://joplinapp.org) to work with Cyrus?s webdav module, and I?ve run into two issues: (1) When attempting to MKCOL a collection that already exists, Cyrus is responding with a 403, rather than a 405, which is what Joplin expects. (2) Cyrus returns an error if the Content-type isn?t set where additional XML-formatted information is required in a POST to complete a request. My skimming of the relevant RFC:s now lead me to believe that Cyrus is right on both counts; however, I don?t know enough about this to say for sure. Can anyone here confirm, or are these genuine Cyrus bugs? // Best wishes; Johan From javier at jangulo.net Thu Nov 28 07:11:05 2019 From: javier at jangulo.net (Javier Angulo) Date: Thu, 28 Nov 2019 13:11:05 +0100 Subject: plus-addressing to mailbox with special-use Message-ID: Hello all, I would like to have spam messages delivered to special-use \Junk folder. As for my tests, this is possible using plus-addressing + the 'anyone p' permission on Junk + special-use "\Junk" attribute set on Junk mailbox; but in our setup, the special-use attribute "\Junk" is already set into many different mailboxes (apart from Junk) by the diversity of imap clients. And cyrus does not allow to have more than one folder with the same special-use attribute - the rfc leaves that part of the implementation deliberately open: Servers MAY allow the creation of a special-use mailbox even if one so designated already exists. This might have the effect of moving the special use from the old mailbox to the new one, or might create multiple mailboxes with the same special use. Alternatively, servers MAY refuse the creation, considering the designation to be a conflict. So, my question is how is people on the list handling this? My idea right now is to create the user Junk mailbox if there is not already one, and set the special-use attribute "\Junk" there if it is not set on another mailbox. Spam will end on the Junk mailbox, but this will not always be the special-use "\junk" maailbox. This is not perfect but I guess cyrus refuses to have 2 folders with the same special-use atttribute for a reason ... Any ideas or advice would be much appreciated. Javier. -------------- next part -------------- An HTML attachment was scrubbed... URL: