From brong at fastmail.fm Thu Jan 1 00:56:04 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 1 Jan 2009 16:56:04 +1100 Subject: choosing a file system In-Reply-To: <9843C43C-2CE4-4882-8C6C-FCABF4EE9809@bu.edu> References: <1617f8010812300849k1c7c878bl2f17e8d4287c10be@mail.gmail.com> <495A95F2.1010003@email.arizona.edu> <9843C43C-2CE4-4882-8C6C-FCABF4EE9809@bu.edu> Message-ID: <20090101055604.GB15669@brong.net> On Wed, Dec 31, 2008 at 07:47:31AM -0500, Nik Conwell wrote: > > On Dec 30, 2008, at 4:43 PM, Shawn Nock wrote: > > [...] > > > a scripted rename of mailboxes to balance partition utilization when > > we > > add another partition. > > Just curious - how do stop people from accessing their mailboxes > during the time they are being renamed and moved to another partition? All access goes via an nginx proxy - we use the proc directory contents to detect currently active connections and termintate them after blocking all new logins in the authentication daemon. Once they're fully moved, logins are enabled again. Bron. From robm at fastmail.fm Fri Jan 2 00:19:52 2009 From: robm at fastmail.fm (Rob Mueller) Date: Fri, 2 Jan 2009 16:19:52 +1100 Subject: choosing a file system References: <1617f8010812300849k1c7c878bl2f17e8d4287c10be@mail.gmail.com><495A95F2.1010003@email.arizona.edu><20081231005129.GA15299@brong.net><1617f8010812310247l4c94e47fr296efceb30951d8a@mail.gmail.com><8B28597E-F7D7-4303-B45E-156A61EA626C@yazzy.org> <20081231142023.GB32353@helsinki.fi> Message-ID: <8765283395DA4A1AB912FC5D3DED2F89@robmhp> > There are /lots/ of (comparative) tests done: The most recent I could > find with a quick Google is here: > > http://www.phoronix.com/scan.php?page=article&item=ext4_benchmarks Almost every filesystem benchmark I've ever seen is effectively useless for comparing what's best for a cyrus mail server. They try and show the maximums/minimums of a bunch of discrete operation types (eg streaming IO, creating files, deleting files, lots of small random reads, etc) running on near empty volumes. What none of them show is what happens to a filesystem when it's a real world cyrus mail spool/index: * 100,000's of directories * 10,000,000's of files * 1-1,000,000 files per directory * files continuously being created and deleted (emails) * data being appended to existing files (cyrus.* files) * lots of fsync calls all over the place (every lmtp append has multiple fsyncs, as well as various imap actions) * run over the course of multiple years of continuous operations * with a filesystem that's 60-90% full depending on your usage levels There's serious fragmentation issues going on here that no benchmark even comes close to simulating. Now from our experience, I can tell you that ext3 really does poorly on this workload compared to reiserfs. We had two exact same servers, one all reiserfs and one all ext3. The ext3 one started out ok, but over the course of a few weeks/months, it started getting worse and worse and was eventually being completely crushed by IO load. The machine running reiserfs had no problems at all even though it had more users on it as well and was growing at the same rate as the other machine. Yes we did have directory indexing enabled (we had it turned on from the start), and we tried different data modes like data=writeback and data=ordered but that didn't help either. To be honest, I don't know why exactly, and working out what's causing IO bottlenecks is not easy. We just went back to reiserfs. Some previous comments I've made. http://www.irbs.net/internet/info-cyrus/0412/0042.html http://lists.andrew.cmu.edu/pipermail/info-cyrus/2006-October/024119.html > The problem with reiserfs is... well. The developers have explicitely > stated that the development of v3 has come to its end, and there was the In this particular case, I'm really almost happy with this! Reiserfs has been very stable for us for at least 5 years, and I'm almost glad no-one is touching it because invariably people working on something will introduce new weird edge case bugs. This was a while back, but it demonstrates how apparently just adding 'some "sparse" endian annotations' caused a bug. http://oss.sgi.com/projects/xfs/faq.html#dir2 That one was really nasty, even the xfs_repair tool couldn't fix it for a while! Having said that, there have been some bugs over the last few years with reiserfs, however the kernel developers will still help with bug fixes if you find them and can trace them down. http://blog.fastmail.fm/2007/09/21/reiserfs-bugs-32-bit-vs-64-bit-kernels-cache-vs-inode-memory/ http://lkml.org/lkml/2005/7/12/396 http://lkml.org/lkml/2008/6/17/9 Rob From brong at fastmail.fm Fri Jan 2 01:14:26 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 2 Jan 2009 17:14:26 +1100 Subject: choosing a file system In-Reply-To: <8765283395DA4A1AB912FC5D3DED2F89@robmhp> References: <20081231142023.GB32353@helsinki.fi> <8765283395DA4A1AB912FC5D3DED2F89@robmhp> Message-ID: <20090102061426.GA31740@brong.net> On Fri, Jan 02, 2009 at 04:19:52PM +1100, Rob Mueller wrote: > http://lkml.org/lkml/2008/6/17/9 Ahh, that week. *sigh*. Not strictly a reiserfs problem of course, that would have affected everyone. Speaking of which, Linus did point out in that thread that the way Cyrus does IO (mmap for reads, fseek/fwrite for writes) is totally insane and guaranteed to hit every bug in existance. Normal people just use mmap for both. Does anyone actually run Cyrus on anything that doesn't support writable mmap these days? Bron. From jmadden at ivytech.edu Fri Jan 2 08:50:22 2009 From: jmadden at ivytech.edu (John Madden) Date: Fri, 2 Jan 2009 08:50:22 -0500 Subject: choosing a file system In-Reply-To: <8765283395DA4A1AB912FC5D3DED2F89@robmhp> References: <1617f8010812300849k1c7c878bl2f17e8d4287c10be@mail.gmail.com> <20081231142023.GB32353@helsinki.fi> <8765283395DA4A1AB912FC5D3DED2F89@robmhp> Message-ID: <200901020850.22355.jmadden@ivytech.edu> > Now from our experience, I can tell you that ext3 really does poorly on > this workload compared to reiserfs. We had two exact same servers, one all > reiserfs and one all ext3. The ext3 one started out ok, but over the course > of a few weeks/months, it started getting worse and worse and was > eventually being completely crushed by IO load. The machine running > reiserfs had no problems at all even though it had more users on it as well > and was growing at the same rate as the other machine. Now see, I've had almost exactly the opposite experience. Reiserfs seemed to start out well and work consistently until the filesystem reached a certain size (around 160GB, ~30m files) at which point backing it up would start to take too long and at around 180GB would take nearly a week. This forced us to move to ext3 and it doesn't seem to be degrade that way. We did, however, also move from a single partition to 8 of them, so that obviously has some effect as well. John -- John Madden Sr. UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden at ivytech.edu From lars at lhanke.de Fri Jan 2 11:19:41 2009 From: lars at lhanke.de (Lars Hanke) Date: Fri, 02 Jan 2009 17:19:41 +0100 Subject: GSSAPI authentication ceased working Message-ID: <495E3E9D.10907@lhanke.de> I'm currently setting up a new imap server to replace my old one. Yesterday I had GSSAPI authentication running, today it ceased working. I did quite some configuration in the meantime mostly on the LDAP server, but nothing I'd readily associate with cyrus-imap authentication. I appreciate any ideas for more systematic troubleshooting. Regards, - lars. The setup: KDC and LDAP is a sever called hel. The KDC uses LDAP as backend. Cyrus-Imap (v2.2.13-Debian-2.2.13-14+b3) runs on hermod. What worked yesterday: kinit cyrus imtest -v -u cyrus -a cyrus -p imap -r MGR hermod.mgr cyradm --user cyrus --auth GSSAPI --server hermod.mgr What still works today: kinit cyrus Diagnostics: # kinit cyrus hermod:~# klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: cyrus at MGR Valid starting Expires Service principal 01/02/09 16:41:41 01/03/09 02:41:41 krbtgt/MGR at MGR renew until 01/03/09 16:41:41 Kerberos 4 ticket cache: /tmp/tkt0 klist: You have no tickets cached hermod:~# imtest -v -u cyrus -a cyrus -p imap -r MGR hermod.mgr S: * OK hermod.mgr Cyrus IMAP4 v2.2.13-Debian-2.2.13-14+b3 server ready C: C01 CAPABILITY S: * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS ID NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE IDLE STARTTLS AUTH=GSSAPI AUTH=NTLM AUTH=DIGEST-MD5 AUTH=CRAM-MD5 SASL-IR S: C01 OK Completed Authentication failed. generic failure Security strength factor: 0 C: Q01 LOGOUT * BYE LOGOUT received Q01 OK Completed Connection closed. hermod: /var/log/auth.log Jan 2 17:07:54 hermod imtest: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Decrypt integrity check failed) hel: /var/log/syslog Jan 2 16:07:54 hel krb5kdc[1652]: TGS_REQ (7 etypes {18 17 16 23 1 3 2}) 172.16.6.5: PROCESS_TGS: authtime 0, for imap/hermod.mgr at MGR, Decrypt integrity check failed Jan 2 16:07:54 hel last message repeated 3 times What I tried: Since "Decrypt integrity check failed" means "wrong password" I recreated the principal "imap/hermod.mgr" and replaced the keytab file with the new key. I also removed the ldapdb auxprop, which I had installed in the meantime, but nothing helped. If I remove the ticket for cyrus, I receive: Jan 2 17:13:36 hermod imtest: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No credentials cache found) as I would expect. From lars at lhanke.de Fri Jan 2 14:06:59 2009 From: lars at lhanke.de (Lars Hanke) Date: Fri, 02 Jan 2009 20:06:59 +0100 Subject: ldapdb auxprop configuration Message-ID: <495E65D3.7050805@lhanke.de> I'm trying set up cyrus-imap using the ldapdb auxprop. I guess I've the LDAP part up and running, but somehow imap does not really request for authentication. So probably I still have something messed in the configuration, which apparently has changed with respect to my last install a couple of years ago. Any ideas for systematic troubleshooting are welcome. Regards, - lars. This is the sasl related part of the imap configuration: hermod:~# grep sasl /etc/imapd.conf | grep -v '^#' | grep -v '^\s*$' sasl_mech_list: PLAIN DIGEST-MD5 CRAM-MD5 sasl_pwcheck_method: auxprop sasl_auxprop_plugin: ldapdb sasl_ldapdb_uri: ldaps://hel.mgr sasl_ldapdb_id: mailadmin sasl_ldapdb_pw: ********* sasl_ldapdb_mech: DIGEST-MD5 sasl_auto_transition: no The following is running as expected: hermod:~# ldapwhoami -U mailadmin -X u:cyrus -W -Y DIGEST-MD5 -H ldaps://hel.mgr Enter LDAP Password: SASL/DIGEST-MD5 authentication started SASL username: u:cyrus SASL SSF: 128 SASL data security layer installed. dn:uid=cyrus,ou=mailbox,dc=mgr and of course: ldapsearch -U mailadmin -X u:cyrus -W -Y DIGEST-MD5 -b "ou=mailbox,dc=mgr" "(uid=cyrus)" returns the password of cyrus, which is kept as plaintext inside the LDAP repositiory. ldapsearch returns the base64 encoded plain password. However using this same password the following happens: hermod:~# imtest -v -u cyrus -a cyrus -p imap -m DIGEST-MD5 hermod.mgr S: * OK hermod.mgr Cyrus IMAP4 v2.2.13-Debian-2.2.13-14+b3 server ready C: C01 CAPABILITY S: * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS ID NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE IDLE STARTTLS AUTH=DIGEST-MD5 AUTH=CRAM-MD5 SASL-IR S: C01 OK Completed C: A01 AUTHENTICATE DIGEST-MD5 S: + bm9uY2U9IlBMREhNY0JjbG1XOUt2dk5FQWQrb0R5cmZ3YjY3cHcyb1VIWHhacDE0dXc9IixyZWFsbT0iaGVybW9kLm1nciIscW9wPSJhdXRoLGF1dGgtaW50LGF1dGgtY29uZiIsY2lwaGVyPSJyYzQtNDAscmM0LTU2LHJjNCxkZXMsM2RlcyIsbWF4YnVmPTQwOTYsY2hhcnNldD11dGYtOCxhbGdvcml0aG09bWQ1LXNlc3M= Please enter your password: C: dXNlcm5hbWU9ImN5cnVzIixyZWFsbT0iaGVybW9kLm1nciIsbm9uY2U9IlBMREhNY0JjbG1XOUt2dk5FQWQrb0R5cmZ3YjY3cHcyb1VIWHhacDE0dXc9Iixjbm9uY2U9IkVZR2hkY1UvZy9vU0J5VkNsMkhSVWt3NWVuMTlOR3puWU9PQjZuSUpPams9IixuYz0wMDAwMDAwMSxxb3A9YXV0aC1jb25mLGNpcGhlcj1yYzQsbWF4YnVmPTEwMjQsZGlnZXN0LXVyaT0iaW1hcC9oZXJtb2QubWdyIixyZXNwb25zZT00Yjk3OWJhMTU0NWUzZDBkMTJiYWNlNjY4NTk4YjhjZA== failure: prot layer failure The detailed log of slapd has the following for this request: slap_listener_activate(10): >>> slap_listener(ldaps:///) conn=15 fd=24 ACCEPT from IP=172.16.6.5:53956 (IP=0.0.0.0:636) connection_get(24): got connid=15 connection_read(24): checking for input on id=15 connection_get(24): got connid=15 connection_read(24): checking for input on id=15 connection_get(24): got connid=15 connection_read(24): checking for input on id=15 connection_get(24): got connid=15 connection_read(24): checking for input on id=15 connection_read(24): unable to get TLS client DN, error=49 id=15 conn=15 fd=24 TLS established tls_ssf=128 ssf=128 connection_get(24): got connid=15 connection_read(24): checking for input on id=15 ber_get_next ber_get_next on fd 24 failed errno=0 (Success) connection_closing: readying conn=15 sd=24 for close connection_close: conn=15 sd=24 conn=15 fd=24 closed (connection lost) So apparently imapd-ldapdb connects and establishes SSL. For the rest I'm unsure, but it seems like it does not talk to LDAP anymore and terminates, i.e. there is no authentication happening. The result is the same for trying telnet localhost imap2 and a login for cyrus. From dwhite at olp.net Fri Jan 2 14:58:50 2009 From: dwhite at olp.net (Dan White) Date: Fri, 02 Jan 2009 13:58:50 -0600 Subject: ldapdb auxprop configuration In-Reply-To: <495E65D3.7050805@lhanke.de> References: <495E65D3.7050805@lhanke.de> Message-ID: <495E71FA.1070708@olp.net> Lars Hanke wrote: > I'm trying set up cyrus-imap using the ldapdb auxprop. I guess I've the > LDAP part up and running, but somehow imap does not really request for > authentication. So probably I still have something messed in the > configuration, which apparently has changed with respect to my last > install a couple of years ago. > > Any ideas for systematic troubleshooting are welcome. > Regards, > - lars. > > This is the sasl related part of the imap configuration: > hermod:~# grep sasl /etc/imapd.conf | grep -v '^#' | grep -v '^\s*$' > sasl_mech_list: PLAIN DIGEST-MD5 CRAM-MD5 > sasl_pwcheck_method: auxprop > sasl_auxprop_plugin: ldapdb > sasl_ldapdb_uri: ldaps://hel.mgr > sasl_ldapdb_id: mailadmin > sasl_ldapdb_pw: ********* > sasl_ldapdb_mech: DIGEST-MD5 > sasl_auto_transition: no > > I don't see anything that sticks out. You may want to experiment with the ldapdb_starttls and ldapdb_rc options (see sasl's options.html doc). See 'man ldap.conf' for options that you can place in ldaprc. If you do choose to use starttls, you'll need to replace ldaps://hel.mgr with ldap://hel.mgr. To make sure that the ldapdb plugin is installed correctly: # cat > /usr/lib/sasl2/pluginview.conf mech_list: PLAIN DIGEST-MD5 CRAM-MD5 pwcheck_method: auxprop auxprop_plugin: ldapdb ldapdb_uri: ldaps://hel.mgr ldapdb_id: mailadmin ldapdb_pw: ********* ldapdb_mech: DIGEST-MD5 auto_transition: no # pluginviewer | grep ldapdb > The following is running as expected: > hermod:~# ldapwhoami -U mailadmin -X u:cyrus -W -Y DIGEST-MD5 -H > ldaps://hel.mgr > Enter LDAP Password: > SASL/DIGEST-MD5 authentication started > SASL username: u:cyrus > SASL SSF: 128 > SASL data security layer installed. > dn:uid=cyrus,ou=mailbox,dc=mgr > > and of course: > ldapsearch -U mailadmin -X u:cyrus -W -Y DIGEST-MD5 -b > "ou=mailbox,dc=mgr" "(uid=cyrus)" > returns the password of cyrus, which is kept as plaintext inside the > LDAP repositiory. ldapsearch returns the base64 encoded plain password. > > So apparently imapd-ldapdb connects and establishes SSL. For the rest > I'm unsure, but it seems like it does not talk to LDAP anymore and > terminates, i.e. there is no authentication happening. The result is the > same for trying telnet localhost imap2 and a login for cyrus. > Does your /var/log/auth.log or /var/log/syslog give you anything useful? - Dan From dick at nagual.nl Fri Jan 2 15:04:37 2009 From: dick at nagual.nl (dick hoogendijk) Date: Fri, 2 Jan 2009 21:04:37 +0100 Subject: login issues Message-ID: <20090102210437.63fdea7d.dick@nagual.nl> All mail users are in sasldb. But now the problem is that I get more and more outlook (MS) clients that can't do CRAM-MD5. So, I guess the imapd.conf I use needs some tweaking. What do I have to change so that outlook express users can login? The users are just -mail- users and have NO UNIX account on the system. ======imapd.conf======== configdirectory: /opt/csw/var/cyrus/config partition-default: /opt/csw/var/cyrus/mail sievedir: /opt/csw/var/cyrus/sieve admins: cyrus unixhierarchysep: no altnamespace: no munge8bit: yes sasl_pwcheck_method: saslauthd # sieve_allowplaintext: yes # sasl_minimum_layer: 0 # sasl_mech_list: PLAIN LOGIN # autocreatequota: -1 # createonpost: yes tls_ca_file: /opt/csw/ssl/private/imap/server.pem tls_cert_file: /opt/csw/ssl/private/imap/server.pem tls_key_file: /opt/csw/ssl/private/imap/server.pem ======================= -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS sxce snv104 ++ + All that's really worth doing is what we do for others (Lewis Carrol) From Hagedorn at uni-koeln.de Fri Jan 2 15:15:34 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Fri, 02 Jan 2009 21:15:34 +0100 Subject: login issues In-Reply-To: <20090102210437.63fdea7d.dick@nagual.nl> References: <20090102210437.63fdea7d.dick@nagual.nl> Message-ID: -- dick hoogendijk is rumored to have mumbled on 2. Januar 2009 21:04:37 +0100 regarding login issues: > All mail users are in sasldb. But now the problem is that I get more > and more outlook (MS) clients that can't do CRAM-MD5. > So, I guess the imapd.conf I use needs some tweaking. > What do I have to change so that outlook express users can login? We use the following for maximum compatibility, but it requires that all those SASL modules are actually installed: allowanonymouslogin: no allowplaintext: no sasl_pwcheck_method: auxprop sasl_mech_list: DIGEST-MD5 CRAM-MD5 PLAIN NTLM LOGIN That way Outlook can either use NTLM or some SSL/TLS secured connection. -- Sebastian Hagedorn - RZKR-R1 (Flachbau), Zi. 18, Robert-Koch-Str. 10 Zentrum f?r angewandte Informatik - Universit?tsweiter Service RRZK Universit?t zu K?ln / Cologne University - Tel. +49-221-478-5587 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090102/ded439b8/attachment.bin From lars at lhanke.de Fri Jan 2 16:37:08 2009 From: lars at lhanke.de (Lars Hanke) Date: Fri, 02 Jan 2009 22:37:08 +0100 Subject: ldapdb auxprop configuration In-Reply-To: <495E71FA.1070708@olp.net> References: <495E65D3.7050805@lhanke.de> <495E71FA.1070708@olp.net> Message-ID: <495E8904.6020506@lhanke.de> Thanks Dan, > To make sure that the ldapdb plugin is installed correctly: > # cat > /usr/lib/sasl2/pluginview.conf > # pluginviewer | grep ldapdb hermod:/# grep sasl /etc/imapd.conf | grep -v '^#' | grep -v '^\s*$' | sed 's/^sasl_//' > /usr/lib/sasl2/pluginviewer.conf hermod:/# saslpluginviewer -a Installed auxprop mechanisms are: ldapdb sasldb List of auxprop plugins follows Plugin "ldapdb" , API version: 4 supports store: yes Plugin "sasldb" , API version: 4 supports store: yes Didn't know this tool so far. Should it say something different? > Does your /var/log/auth.log or /var/log/syslog give you anything useful? At least it's not too useful to me ... (after setting sasl_log_level: 7) /var/log/auth.log: Jan 2 22:31:15 hermod cyrus/imap[3432]: DIGEST-MD5 server step 1 Jan 2 22:31:15 hermod imtest: DIGEST-MD5 client step 2 Jan 2 22:31:17 hermod imtest: DIGEST-MD5 client step 2 Jan 2 22:31:17 hermod cyrus/imap[3432]: DIGEST-MD5 server step 2 /var/log/syslog: Jan 2 22:31:15 hermod cyrus/master[3432]: about to exec /usr/lib/cyrus/bin/imapd Jan 2 22:31:15 hermod cyrus/imap[3432]: executed Jan 2 22:31:15 hermod cyrus/imap[3432]: accepted connection Jan 2 22:31:17 hermod cyrus/master[3425]: process 3432 exited, signaled to death by 11 Jan 2 22:31:17 hermod cyrus/master[3425]: service imap pid 3432 in BUSY state: terminated abnormally > You may want to experiment with the ldapdb_starttls and ldapdb_rc options (see sasl's options.html doc). See 'man ldap.conf' for options that you can place in ldaprc. If you do choose to use starttls, you'll need to replace ldaps://hel.mgr with ldap://hel.mgr. I tried sasl_ldapdb_uri: ldap://hel.mgr sasl_ldapdb_starttls: try and it comes out the same; slapd logs a successful STARTTLS. I tried: sasl_ldapdb_rc: /etc/ldap/ldap.conf which yields sending short packages in both cases. This slapd debug output is from a STARTTLS variant: TLS: can't accept: A TLS packet with unexpected length was received.. connection_read(16): TLS accept failure error=-1 id=8, closing connection_closing: readying conn=8 sd=16 for close connection_close: conn=8 sd=16 conn=8 fd=16 closed (TLS negotiation failure) But still imtest fails with "failure: prot layer failure". There is no activity in slapd before the password is entered in imtest. From dwhite at olp.net Fri Jan 2 17:32:36 2009 From: dwhite at olp.net (Dan White) Date: Fri, 02 Jan 2009 16:32:36 -0600 Subject: ldapdb auxprop configuration In-Reply-To: <495E8904.6020506@lhanke.de> References: <495E65D3.7050805@lhanke.de> <495E71FA.1070708@olp.net> <495E8904.6020506@lhanke.de> Message-ID: <495E9604.7030201@olp.net> Lars Hanke wrote: > hermod:/# saslpluginviewer -a > Installed auxprop mechanisms are: > ldapdb sasldb > List of auxprop plugins follows > Plugin "ldapdb" , API version: 4 > supports store: yes > > Plugin "sasldb" , API version: 4 > supports store: yes > > Didn't know this tool so far. Should it say something different? No, that confirms that ldapdb is installed. > > Does your /var/log/auth.log or /var/log/syslog give you anything > useful? > /var/log/syslog: > Jan 2 22:31:15 hermod cyrus/master[3432]: about to exec > /usr/lib/cyrus/bin/imapd > Jan 2 22:31:15 hermod cyrus/imap[3432]: executed > Jan 2 22:31:15 hermod cyrus/imap[3432]: accepted connection > Jan 2 22:31:17 hermod cyrus/master[3425]: process 3432 exited, > signaled to death by 11 > Jan 2 22:31:17 hermod cyrus/master[3425]: service imap pid 3432 in > BUSY state: terminated abnormally 'signaled to death by 11' is a big red flag... your imapd process is seg faulting. It's possibly caused by an old SASL/OpenLDAP reentrant bug (are you running an old version of libldap?). You can specify a debug_command in your imapd.conf to generate a back trace. See: https://langhorst.com/cgi-bin/dwww//usr/share/doc/cyrus21-common/README.Debian.debug.gz - Dan From robm at fastmail.fm Fri Jan 2 21:21:15 2009 From: robm at fastmail.fm (Rob Mueller) Date: Sat, 3 Jan 2009 13:21:15 +1100 Subject: choosing a file system References: <1617f8010812300849k1c7c878bl2f17e8d4287c10be@mail.gmail.com><20081231142023.GB32353@helsinki.fi><8765283395DA4A1AB912FC5D3DED2F89@robmhp> <200901020850.22355.jmadden@ivytech.edu> Message-ID: > Now see, I've had almost exactly the opposite experience. Reiserfs seemed > to > start out well and work consistently until the filesystem reached a > certain > size (around 160GB, ~30m files) at which point backing it up would start > to > take too long and at around 180GB would take nearly a week. This forced > us > to move to ext3 and it doesn't seem to be degrade that way. We did, > however, > also move from a single partition to 8 of them, so that obviously has some > effect as well. As you noted, changing two variables at once doesn't help you determine which was the problem! Multiple partitions will definitely allow more parallelism, which definitely helps speed things up, which is one of the other things we have done over time. Basically we went from a few large volumes to hundreds of 300G(data)/15G(meta) volumes. One of our machines has 40 data volumes + 40 meta data volumes + the standard FS mounts. $ mount | wc -l 92 We've found that splitting the data up into more volumes + more cyrus instances seems to help as well because it seems to reduce overall contention points in the kernel + software (eg filesystem locks spread across multiple mounts, db locks are spread across multiple dbs, etc) Also one thing I did fail to mention, was that for the data volumes, you should definitely be using the "notail" mount option. Unfortunately that's not the default, and I think it probably should be. Tails packing is neat for saving space, but it reduces the average meta-data density, which makes "stating" lots of files in a directory a lot slower. I think that's what you might have been seeing. Of course you also mounted "noatime,nodiratime" on both? I think that's another problem with a lot of filesystem benchmarks, not finding out what the right mount "tuning" options are for your benchmark. Arguing that "the default should be fine" is clearly wrong, because every sane person uses "noatime", so you're already doing some tuning, so you should find out what's best for the filesystem you are trying. For the record, we use: noatime,nodiratime,notail,data=ordered On all our reiserfs volumes. Rob From ram at netcore.co.in Fri Jan 2 23:40:17 2009 From: ram at netcore.co.in (ram) Date: Sat, 03 Jan 2009 10:10:17 +0530 Subject: Storage Sizing: IOPS per mailbox Message-ID: <1230957617.20171.17.camel@darkstar.netcore.co.in> When sizing a storage device for a large cyrus server, the typical question asked by storage vendors is what is the IOPS required per mailbox M$$ Exchange has this concept of IOPS. and they suggest 1.5 IOPS per mailbox ( heavy users ) If I use postfix and cyrus , on my imap server ( pure IMAP server .. All spam filtering , outgoing mails , authentication etc happens on different servers ) If the storage is used only for imap storage , what is the typical "IOPS" requirement per user We will probably assume 30-50 mails a day of average 100k , and an email client checking for new mail every 5minutes From ram at netcore.co.in Sat Jan 3 01:16:41 2009 From: ram at netcore.co.in (ram) Date: Sat, 03 Jan 2009 11:46:41 +0530 Subject: choosing a file system In-Reply-To: References: <1617f8010812300849k1c7c878bl2f17e8d4287c10be@mail.gmail.com> <20081231142023.GB32353@helsinki.fi> <8765283395DA4A1AB912FC5D3DED2F89@robmhp> <200901020850.22355.jmadden@ivytech.edu> Message-ID: <1230963401.20171.95.camel@darkstar.netcore.co.in> On Sat, 2009-01-03 at 13:21 +1100, Rob Mueller wrote: > > Now see, I've had almost exactly the opposite experience. Reiserfs seemed > > to > > start out well and work consistently until the filesystem reached a > > certain > > size (around 160GB, ~30m files) at which point backing it up would start > > to > > take too long and at around 180GB would take nearly a week. This forced > > us > > to move to ext3 and it doesn't seem to be degrade that way. We did, > > however, > > also move from a single partition to 8 of them, so that obviously has some > > effect as well. > > As you noted, changing two variables at once doesn't help you determine > which was the problem! > > Multiple partitions will definitely allow more parallelism, which definitely > helps speed things up, which is one of the other things we have done over > time. Basically we went from a few large volumes to hundreds of > 300G(data)/15G(meta) volumes. One of our machines has 40 data volumes + 40 > meta data volumes + the standard FS mounts. > > $ mount | wc -l > 92 > > We've found that splitting the data up into more volumes + more cyrus > instances seems to help as well because it seems to reduce overall > contention points in the kernel + software (eg filesystem locks spread > across multiple mounts, db locks are spread across multiple dbs, etc) > Running multiple cyrus instances with different dbs ? How do we do that. I have seen the ultimate io-contention point is the mailboxes.db file. And that has to be single. Do you mean dividing the users to different cyrus instances. That is a maintenance issue IMHO. I had the feeling whatever optimizations done at the FS level would give us a max of 5-10% benefit. We migrated from ext3 to reiserfs on our cyrus servers with 30k mailboxes. I am not sure I saw a great benefit in terms of the iowait. At peak times I always see a iowait of 40-60% But the new Solid-State-Disks seem very promising. They are claimed to give 30x the throughput of a 15k rpm disk. If IO improves by 30 times that should make all these optimizations unnecessary. As my boss used to tell me ... Good hardware always compensates for not-so-good software. > Also one thing I did fail to mention, was that for the data volumes, you > should definitely be using the "notail" mount option. Unfortunately that's > not the default, and I think it probably should be. Tails packing is neat > for saving space, but it reduces the average meta-data density, which makes > "stating" lots of files in a directory a lot slower. I think that's what you > might have been seeing. Of course you also mounted "noatime,nodiratime" on > both? > > I think that's another problem with a lot of filesystem benchmarks, not > finding out what the right mount "tuning" options are for your benchmark. > Arguing that "the default should be fine" is clearly wrong, because every > sane person uses "noatime", so you're already doing some tuning, so you > should find out what's best for the filesystem you are trying. > > For the record, we use: > > noatime,nodiratime,notail,data=ordered > > On all our reiserfs volumes. > > Rob > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html From brong at fastmail.fm Sat Jan 3 05:45:47 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sat, 3 Jan 2009 21:45:47 +1100 Subject: choosing a file system In-Reply-To: <1230963401.20171.95.camel@darkstar.netcore.co.in> References: <1617f8010812300849k1c7c878bl2f17e8d4287c10be@mail.gmail.com> <20081231142023.GB32353@helsinki.fi> <8765283395DA4A1AB912FC5D3DED2F89@robmhp> <200901020850.22355.jmadden@ivytech.edu> <1230963401.20171.95.camel@darkstar.netcore.co.in> Message-ID: <20090103104547.GA13718@brong.net> On Sat, Jan 03, 2009 at 11:46:41AM +0530, ram wrote: > Running multiple cyrus instances with different dbs ? How do we do that. > I have seen the ultimate io-contention point is the mailboxes.db file. > And that has to be single. Yeah, mailboxes.db access kinda sucks like that. If you're making any changes then it locks the entire DB with a single writelock. I did consider fine-grained mailboxes.db locking at one point. It's definitely doable with fcntl locking, which is what Cyrus is using on our machines. It would require some small format changes to skiplist though. Somewhere in a checkout I have cyrusdb_skiplist2.c which contains a bunch of checksumming code and the start of the new format. I got sidetracked and never finished it though. All our cyrus instances are installed on completely different drives. Entirely self-contained on those external units so we can plug then into a new machine and go. The init scripts are in /etc/init.d/, but they are generated from templates which pull their configuration from a central file. We can create a new pair of cyrus instances by adding a single line that looks like this in a config file: store$n slot$s1 slot$s2 where $n, $s1 and $s2 are just numbers. Slots are numbered as %d%02d with server and partition numbers (it will break if we ever have over 100 slots on a machine, but I'm happy to renumber at that point. Our biggest so far is 40. When I set this up the biggest was 8. Future proofing something so easily reconfigurable would have just meant more typing in the meanwhile. > Do you mean dividing the users to different cyrus instances. That is a > maintenance issue IMHO. It's amazing what you can do with good tools - besides, if your site is already bigger than any one single machine then you already have the issue. Might as well be smart about it. As I said upthread somewhere - moving a user is pretty easy for us: use ME::User; my $UserName = shift; my $TargetServer = shift; my $User = ME::User->new_find($UserName); $User->MoveUser($TargetServer); > But the new Solid-State-Disks seem very promising. They are claimed to > give 30x the throughput of a 15k rpm disk. If IO improves by 30 times > that should make all these optimizations unnecessary. > As my boss used to tell me ... Good hardware always compensates for > not-so-good software. Yeah, that would be nice. Modulo the rewrite cost of course. Note that mailboxes.db is a skiplist file. They make a lot of random updates to 4 bytes at a time when you append a record. Imagine what that costs if your minimum rewrite block is larger than the size of the whole file. You'd be better off going to flatfile DB. I'm not kidding you here. Running "recovery" at startup time would take days on a reasonable sized DB. Check out the seeks and rewrites that baby does. (ok, so if your filesystem isn't mounted writeback it would probably only rewrite twice when you actually did the fsyncs. So much for rhetorical devices) Bron ( rambling again ) From hmh at debian.org Sat Jan 3 09:45:19 2009 From: hmh at debian.org (Henrique de Moraes Holschuh) Date: Sat, 3 Jan 2009 12:45:19 -0200 Subject: choosing a file system In-Reply-To: <1230743048.6328.11.camel@linux-nnci.site> References: <1617f8010812300849k1c7c878bl2f17e8d4287c10be@mail.gmail.com> <495A95F2.1010003@email.arizona.edu> <20081231005129.GA15299@brong.net> <1617f8010812310247l4c94e47fr296efceb30951d8a@mail.gmail.com> <1230727101.5300.6.camel@linux-nnci.site> <20081231134616.GA32353@helsinki.fi> <1230743048.6328.11.camel@linux-nnci.site> Message-ID: <20090103144519.GB7152@khazad-dum.debian.net> On Wed, 31 Dec 2008, Adam Tauno Williams wrote: > > I never really got the point of the data=writeback mode. Sure, it > > increases throughput, but so does disabling the journal completely, and > > seems to me the end result as concerns data integrity is exactly the > > same. > > The *filesystem* is recoverable as the meta-data is journaled. > *Contents* of files may be lost/corrupted. I'm fine with that since a > serious abend usually leaves the state of the data in a questionable > state anyway for reasons other than the filesystem; I want something I > can safely (and quickly) remount and investigate/restore. It is a > trade-off. Err... you guys better read the recent threads in LKML where Pavel goes really hard on the data safety holes in ext3 and Linux VFS (and POSIX). Short answer: ext3 without barriers (you can also disable disk write cache, in that case barriers are not needed) is not deserving of the name "safe". At which point *I* personally prefer XFS, which is just as adverse to the lack of barriers on a disk with an enabled write cache, but performs better than ext3 on most workloads AND has delayed write allocation. Ext4, I never tried. Nor reiser3. I may have to, we will build a brand new Cyrus spool (small, just 5K users) next month, and the XFS unlink [lack of] performance worries me. -- "One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh From lars at lhanke.de Sat Jan 3 10:43:59 2009 From: lars at lhanke.de (Lars Hanke) Date: Sat, 03 Jan 2009 16:43:59 +0100 Subject: ldapdb auxprop configuration In-Reply-To: <495E9604.7030201@olp.net> References: <495E65D3.7050805@lhanke.de> <495E71FA.1070708@olp.net> <495E8904.6020506@lhanke.de> <495E9604.7030201@olp.net> Message-ID: <495F87BF.8000700@lhanke.de> _S_eems we're coming closer ... > 'signaled to death by 11' is a big red flag... your imapd process is > seg faulting. It's possibly caused by an old SASL/OpenLDAP reentrant > bug (are you running an old version of libldap?). Looking at SASL Bug 3032 and Debian Bug #409495 it looks quite like that. Both bugs are not resolved according to the bugtrackers. The install is Lenny current as per today. These are the relevant packages: hermod:/# dpkg -l '*cyrus*' | grep '^ii' ii cyrus-admin-2.2 2.2.13-14 Cyrus mail system (administration tools) ii cyrus-clients-2.2 2.2.13-14+b3 Cyrus mail system (test clients) ii cyrus-common-2.2 2.2.13-14+b3 Cyrus mail system (common files) ii cyrus-imapd-2.2 2.2.13-14+b3 Cyrus mail system (IMAP support) ii libcyrus-imap-perl22 2.2.13-14+b3 Interface to Cyrus imap client imclient libr hermod:/# dpkg -l '*ldap*' | grep '^ii' ii ldap-utils 2.4.11-1 OpenLDAP utilities ii libldap-2.4-2 2.4.11-1 OpenLDAP libraries hermod:/# dpkg -l '*sasl*' | grep '^ii' ii libsasl2-2 2.1.22.dfsg1-23 Cyrus SASL - authentication abstraction libr ii libsasl2-modules 2.1.22.dfsg1-23 Cyrus SASL - pluggable authentication module ii libsasl2-modules-gssapi-mit 2.1.22.dfsg1-23 Cyrus SASL - pluggable authentication module ii libsasl2-modules-ldap 2.1.22.dfsg1-23 Cyrus SASL - pluggable authentication module ii sasl2-bin 2.1.22.dfsg1-23 Cyrus SASL - administration programs for SASL The Debian Bugtracker has it that avoiding DIGEST-MD5 would work around the bug. So I removed that MECH and eventually replaced it with CRAM-MD5, but the SEGFAULT persists. > You can specify a debug_command in your imapd.conf to generate a back > trace. See: > https://langhorst.com/cgi-bin/dwww//usr/share/doc/cyrus21-common/README.Debian.debug.gz > Okay, if the error is known and open, it's worth the while to create a debugging version of probably all the packages. Does anyone know more about the current state of the bug? I could supply an OpenVZ Container, which readily produces it. ;) The strace ends in: 15:42:19.940818 poll([{fd=16, events=POLLIN}], 1, 5000) = 1 ([{fd=16, revents=POLLIN}]) 15:42:19.940883 ioctl(16, FIONREAD, [83]) = 0 15:42:19.940917 recvfrom(16, "\263\225\205\200\0\1\0\1\0\1\0\0\0013\0016\00216\003172\7in-addr\4a"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("172.16.6.3")}, [16]) = 83 15:42:19.940973 close(16) = 0 15:42:19.941020 uname({sys="Linux", node="hermod.mgr", ...}) = 0 15:42:19.941095 --- SIGSEGV (Segmentation fault) @ 0 (0) --- So the last thing done successfully is a DNS query for the LDAP server. The ltrace is a little more informative. However, it looks like the SEGFAULT is somewhere in SASL, probably the thread should continue there ... 3922 15:46:41.771218 read(0, "dXNlcm5hbWU9ImN5cnVzIixyZWFsbT0i"..., 4096) = 354 3922 15:46:41.771278 strlen("") = 0 3922 15:46:41.771321 strncasecmp(0x7fffc67534d0, 0x46193b, 0, 5, 22) = 0 3922 15:46:41.771380 strlen("") = 0 3922 15:46:41.771423 strlen("dXNlcm5hbWU9ImN5cnVzIixyZWFsbT0i"...) = 352 3922 15:46:41.771470 sasl_decode64(0x7fffc67534d0, 352, 0x7fffc67534d0, 21848, 0x7fffc6758a3c) = 0 3922 15:46:41.771520 sasl_server_step(0x1f48140, 0x7fffc67534d0, 262, 0x7fffc6758a30, 0x7fffc6758a38 3922 15:46:41.771554 malloc(250) = 0x1f4a260 3922 15:46:41.771631 malloc(263) = 0x1f4a260 3922 15:46:41.771680 malloc(6) = 0x1f4a370 3922 15:46:41.771722 malloc(11) = 0x1f4a390 3922 15:46:41.771763 malloc(45) = 0x1f4a3b0 3922 15:46:41.771805 malloc(45) = 0x1f4a3f0 3922 15:46:41.771847 malloc(10) = 0x1f4a430 3922 15:46:41.771891 malloc(4) = 0x1f4a450 3922 15:46:41.771933 malloc(16) = 0x1f4a470 3922 15:46:41.771975 malloc(33) = 0x1f49b20 3922 15:46:41.772018 malloc(1219) = 0x1f4a490 3922 15:46:41.772061 memcpy(0x1f48ee1, "cyrus", 5) = 0x1f48ee1 3922 15:46:41.772105 strlen("cyrus") = 5 3922 15:46:41.772148 strlen("cyrus") = 5 3922 15:46:41.772193 strcmp("unix", "unix") = 0 3922 15:46:41.772241 strlen("cyrus") = 5 3922 15:46:41.772283 memmove(0x6fc420, 0x1f48ee1, 5, 2, 1) = 0x6fc420 3922 15:46:41.772333 __ctype_b_loc() = 0x7f87be74d6a0 3922 15:46:41.772376 __ctype_b_loc() = 0x7f87be74d6a0 3922 15:46:41.772429 __ctype_b_loc() = 0x7f87be74d6a0 3922 15:46:41.772472 __ctype_b_loc() = 0x7f87be74d6a0 3922 15:46:41.772514 __ctype_b_loc() = 0x7f87be74d6a0 3922 15:46:41.772556 strlen("cyrus") = 5 3922 15:46:41.772599 strcpy(0x1f48ee1, "cyrus") = 0x1f48ee1 3922 15:46:41.772643 strlen("sasl_") = 5 3922 15:46:41.772685 snprintf("\001\200\255\373\377\177", 5102666, "\377\377\377\377\377\377\377\377"...) = 27 3922 15:46:41.772736 malloc(7) = 0x1f49670 3922 15:46:41.772780 strlen("sasl_") = 5 3922 15:46:41.772822 snprintf("\001\200\255\373", 5102666, "\377\377\377\377\377\377\377\377"...) = 24 3922 15:46:41.772871 strcmp("imap_sasl_auxprop_plugin", "partition-default") = -1 3922 15:46:41.772920 strcmp("sasl_auxprop_plugin", "partition-default") = 1 3922 15:46:41.772967 strcmp("sasl_auxprop_plugin", "sasl_auxprop_plugin") = 0 3922 15:46:41.773015 malloc(7) = 0x1f49670 3922 15:46:41.773056 malloc(48) = 0x1f49690 3922 15:46:41.773482 malloc(8) = 0x1f496d0 3922 15:46:41.795036 malloc(4568) = 0x1f8fc10 3922 15:46:41.795087 malloc(192) = 0x1f8c020 3922 15:46:41.795130 malloc(5) = 0x1f574b0 3922 15:46:41.795172 malloc(150) = 0x1f8c0f0 3922 15:46:41.795212 malloc(150) = 0x1f8c190 3922 15:46:41.795253 malloc(8) = 0x1f55c30 3922 15:46:41.795294 malloc(392) = 0x1f8a220 3922 15:46:41.795335 malloc(12) = 0x1f5fc10 3922 15:46:41.795387 malloc(11) = 0x1f5fc90 3922 15:46:41.795435 malloc(336) = 0x1f8a3b0 3922 15:46:41.795479 --- SIGSEGV (Segmentation fault) --- 3922 15:46:41.796184 +++ killed by SIGSEGV +++ From tschlabach at gmx.net Sat Jan 3 11:34:26 2009 From: tschlabach at gmx.net (Torsten Schlabach) Date: Sat, 03 Jan 2009 17:34:26 +0100 Subject: ldapdb auxprop configuration In-Reply-To: <495F87BF.8000700@lhanke.de> References: <495E65D3.7050805@lhanke.de> <495E71FA.1070708@olp.net> <495E8904.6020506@lhanke.de> <495E9604.7030201@olp.net> <495F87BF.8000700@lhanke.de> Message-ID: <495F9392.80902@gmx.net> Hi all! If I may briefly jump in: One comment suggested that the problem might be one of the Debian specific patches! Did you try to build a package without them? I never found the time ... Regards, Torsten Lars Hanke schrieb: > _S_eems we're coming closer ... > >> 'signaled to death by 11' is a big red flag... your imapd process is >> seg faulting. It's possibly caused by an old SASL/OpenLDAP reentrant >> bug (are you running an old version of libldap?). > Looking at SASL Bug 3032 > and Debian Bug > #409495 it looks quite like that. Both bugs are not resolved according > to the bugtrackers. The install is Lenny current as per today. These are > the relevant packages: > > hermod:/# dpkg -l '*cyrus*' | grep '^ii' > ii cyrus-admin-2.2 2.2.13-14 Cyrus mail > system (administration tools) > ii cyrus-clients-2.2 2.2.13-14+b3 Cyrus mail > system (test clients) > ii cyrus-common-2.2 2.2.13-14+b3 Cyrus mail > system (common files) > ii cyrus-imapd-2.2 2.2.13-14+b3 Cyrus mail > system (IMAP support) > ii libcyrus-imap-perl22 2.2.13-14+b3 Interface > to Cyrus imap client imclient libr > hermod:/# dpkg -l '*ldap*' | grep '^ii' > ii ldap-utils 2.4.11-1 OpenLDAP > utilities > ii libldap-2.4-2 2.4.11-1 OpenLDAP > libraries > hermod:/# dpkg -l '*sasl*' | grep '^ii' > ii libsasl2-2 2.1.22.dfsg1-23 Cyrus SASL > - authentication abstraction libr > ii libsasl2-modules 2.1.22.dfsg1-23 Cyrus SASL > - pluggable authentication module > ii libsasl2-modules-gssapi-mit 2.1.22.dfsg1-23 Cyrus SASL > - pluggable authentication module > ii libsasl2-modules-ldap 2.1.22.dfsg1-23 Cyrus SASL > - pluggable authentication module > ii sasl2-bin 2.1.22.dfsg1-23 Cyrus SASL > - administration programs for SASL > > The Debian Bugtracker has it that avoiding DIGEST-MD5 would work around > the bug. So I removed that MECH and eventually replaced it with > CRAM-MD5, but the SEGFAULT persists. >> You can specify a debug_command in your imapd.conf to generate a back >> trace. See: >> https://langhorst.com/cgi-bin/dwww//usr/share/doc/cyrus21-common/README.Debian.debug.gz >> > Okay, if the error is known and open, it's worth the while to create a > debugging version of probably all the packages. Does anyone know more > about the current state of the bug? I could supply an OpenVZ Container, > which readily produces it. ;) > > The strace ends in: > > 15:42:19.940818 poll([{fd=16, events=POLLIN}], 1, 5000) = 1 ([{fd=16, > revents=POLLIN}]) > 15:42:19.940883 ioctl(16, FIONREAD, [83]) = 0 > 15:42:19.940917 recvfrom(16, > "\263\225\205\200\0\1\0\1\0\1\0\0\0013\0016\00216\003172\7in-addr\4a"..., > 1024, 0, {sa_family=AF_INET, sin_port=htons(53), > sin_addr=inet_addr("172.16.6.3")}, [16]) = 83 > 15:42:19.940973 close(16) = 0 > 15:42:19.941020 uname({sys="Linux", node="hermod.mgr", ...}) = 0 > 15:42:19.941095 --- SIGSEGV (Segmentation fault) @ 0 (0) --- > > So the last thing done successfully is a DNS query for the LDAP server. > The ltrace is a little more informative. However, it looks like the > SEGFAULT is somewhere in SASL, probably the thread should continue there ... > > 3922 15:46:41.771218 read(0, "dXNlcm5hbWU9ImN5cnVzIixyZWFsbT0i"..., > 4096) = 354 > 3922 15:46:41.771278 strlen("") = 0 > 3922 15:46:41.771321 strncasecmp(0x7fffc67534d0, 0x46193b, 0, 5, 22) = 0 > 3922 15:46:41.771380 strlen("") = 0 > 3922 15:46:41.771423 strlen("dXNlcm5hbWU9ImN5cnVzIixyZWFsbT0i"...) = 352 > 3922 15:46:41.771470 sasl_decode64(0x7fffc67534d0, 352, 0x7fffc67534d0, > 21848, 0x7fffc6758a3c) = 0 > 3922 15:46:41.771520 sasl_server_step(0x1f48140, 0x7fffc67534d0, 262, > 0x7fffc6758a30, 0x7fffc6758a38 inished ...> > 3922 15:46:41.771554 malloc(250) = 0x1f4a260 > 3922 15:46:41.771631 malloc(263) = 0x1f4a260 > 3922 15:46:41.771680 malloc(6) = 0x1f4a370 > 3922 15:46:41.771722 malloc(11) = 0x1f4a390 > 3922 15:46:41.771763 malloc(45) = 0x1f4a3b0 > 3922 15:46:41.771805 malloc(45) = 0x1f4a3f0 > 3922 15:46:41.771847 malloc(10) = 0x1f4a430 > 3922 15:46:41.771891 malloc(4) = 0x1f4a450 > 3922 15:46:41.771933 malloc(16) = 0x1f4a470 > 3922 15:46:41.771975 malloc(33) = 0x1f49b20 > 3922 15:46:41.772018 malloc(1219) = 0x1f4a490 > 3922 15:46:41.772061 memcpy(0x1f48ee1, "cyrus", 5) = 0x1f48ee1 > 3922 15:46:41.772105 strlen("cyrus") = 5 > 3922 15:46:41.772148 strlen("cyrus") = 5 > 3922 15:46:41.772193 strcmp("unix", "unix") = 0 > 3922 15:46:41.772241 strlen("cyrus") = 5 > 3922 15:46:41.772283 memmove(0x6fc420, 0x1f48ee1, 5, 2, 1) = 0x6fc420 > 3922 15:46:41.772333 __ctype_b_loc() = > 0x7f87be74d6a0 > 3922 15:46:41.772376 __ctype_b_loc() = > 0x7f87be74d6a0 > 3922 15:46:41.772429 __ctype_b_loc() = > 0x7f87be74d6a0 > 3922 15:46:41.772472 __ctype_b_loc() = > 0x7f87be74d6a0 > 3922 15:46:41.772514 __ctype_b_loc() = > 0x7f87be74d6a0 > 3922 15:46:41.772556 strlen("cyrus") = 5 > 3922 15:46:41.772599 strcpy(0x1f48ee1, "cyrus") = 0x1f48ee1 > 3922 15:46:41.772643 strlen("sasl_") = 5 > 3922 15:46:41.772685 snprintf("\001\200\255\373\377\177", 5102666, > "\377\377\377\377\377\377\377\377"...) = 27 > 3922 15:46:41.772736 malloc(7) = 0x1f49670 > 3922 15:46:41.772780 strlen("sasl_") = 5 > 3922 15:46:41.772822 snprintf("\001\200\255\373", 5102666, > "\377\377\377\377\377\377\377\377"...) = 24 > 3922 15:46:41.772871 strcmp("imap_sasl_auxprop_plugin", > "partition-default") = -1 > 3922 15:46:41.772920 strcmp("sasl_auxprop_plugin", > "partition-default") = 1 > 3922 15:46:41.772967 strcmp("sasl_auxprop_plugin", > "sasl_auxprop_plugin") = 0 > 3922 15:46:41.773015 malloc(7) = 0x1f49670 > 3922 15:46:41.773056 malloc(48) = 0x1f49690 > 3922 15:46:41.773482 malloc(8) = 0x1f496d0 > 3922 15:46:41.795036 malloc(4568) = 0x1f8fc10 > 3922 15:46:41.795087 malloc(192) = 0x1f8c020 > 3922 15:46:41.795130 malloc(5) = 0x1f574b0 > 3922 15:46:41.795172 malloc(150) = 0x1f8c0f0 > 3922 15:46:41.795212 malloc(150) = 0x1f8c190 > 3922 15:46:41.795253 malloc(8) = 0x1f55c30 > 3922 15:46:41.795294 malloc(392) = 0x1f8a220 > 3922 15:46:41.795335 malloc(12) = 0x1f5fc10 > 3922 15:46:41.795387 malloc(11) = 0x1f5fc90 > 3922 15:46:41.795435 malloc(336) = 0x1f8a3b0 > 3922 15:46:41.795479 --- SIGSEGV (Segmentation fault) --- > 3922 15:46:41.796184 +++ killed by SIGSEGV +++ > > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html From robm at fastmail.fm Sat Jan 3 22:56:59 2009 From: robm at fastmail.fm (Rob Mueller) Date: Sun, 4 Jan 2009 14:56:59 +1100 Subject: choosing a file system References: <1617f8010812300849k1c7c878bl2f17e8d4287c10be@mail.gmail.com> <20081231142023.GB32353@helsinki.fi> <8765283395DA4A1AB912FC5D3DED2F89@robmhp> <200901020850.22355.jmadden@ivytech.edu> <1230963401.20171.95.camel@darkstar.netcore.co.in> Message-ID: > Running multiple cyrus instances with different dbs ? How do we do that. > I have seen the ultimate io-contention point is the mailboxes.db file. > And that has to be single. > Do you mean dividing the users to different cyrus instances. That is a > maintenance issue IMHO. As Bron said, yes it is, but if you have more than 1 machines worth of users anyway, you have maintenance issues anyway. So rather than just one instance per machine, we run multiple instances per machine. The only issue it really introduces is that folder sharing between arbitrary users isn't possible (unless you used murder to join all the instances together again, but we don't), only users within an instance can share. > I had the feeling whatever optimizations done at the FS level would give > us a max of 5-10% benefit. > We migrated from ext3 to reiserfs on our cyrus servers with 30k > mailboxes. I am not sure I saw a great benefit in terms of the iowait. > At peak times I always see a iowait of 40-60% To be honest, that's not what we saw in our ext3 <-> reiserfs tests. What mount options are you using? Are you using the mount options I mentioned? noatime,nodiratime,notail,data=ordered > But the new Solid-State-Disks seem very promising. They are claimed to > give 30x the throughput of a 15k rpm disk. If IO improves by 30 times > that should make all these optimizations unnecessary. > As my boss used to tell me ... Good hardware always compensates for > not-so-good software. What we've found is that the meta-data (eg mailbox.db, seen db's, quota files, cyrus.* files) use WAY more IO than the email data, but only use 1/20th the space. By separating the meta data onto RAID1 10k/15k RPM drives, and the email data onto RAID5/6 7.2k RPM drives, you can get a good balance of space/speed. Rob From robm at fastmail.fm Sat Jan 3 23:07:48 2009 From: robm at fastmail.fm (Rob Mueller) Date: Sun, 4 Jan 2009 15:07:48 +1100 Subject: choosing a file system References: <1617f8010812300849k1c7c878bl2f17e8d4287c10be@mail.gmail.com><495A95F2.1010003@email.arizona.edu><20081231005129.GA15299@brong.net><1617f8010812310247l4c94e47fr296efceb30951d8a@mail.gmail.com><1230727101.5300.6.camel@linux-nnci.site><20081231134616.GA32353@helsinki.fi><1230743048.6328.11.camel@linux-nnci.site> <20090103144519.GB7152@khazad-dum.debian.net> Message-ID: <0A2E12D65D13402F87DBB682386AD4C1@robmhp> > Ext4, I never tried. Nor reiser3. I may have to, we will build a brand > new > Cyrus spool (small, just 5K users) next month, and the XFS unlink > [lack of] performance worries me. >From what I can tell, all filesystems seem to have relatively poor unlink performance and unlinks often cause excessive contention and IO for what you think they should be doing. And it's not just filesystems. SQL deletes in MySQL InnoDB are way slower than you'd expect as well. Maybe deletes in general are just not as optimised a path, or there's something tricky about making atomic deletes work, I admit I've never really looked into it. Anyway, that's part of the reason we sponsored Ken to create the "delayed expunge" mode code for cyrus, which allows us to delay unlinks to the weekends when IO load due to other things is the lowest. --- Added support for "delayed" expunge, in which messages are removed from the mailbox index at the time of the EXPUNGE (hiding them from the client), but the message files and cache entries are left behind, to be purged at a later time by cyr_expire. This reduces the amount of I/O that takes place at the time of EXPUNGE and should result in greater responsiveness for the client, especially when expunging a large number of messages. The new expunge_mode option in imapd.conf controls whether expunges are "immediate" or "delayed". Development sponsored by FastMail. --- Rob From Pascal.Gienger at uni-konstanz.de Sun Jan 4 02:19:31 2009 From: Pascal.Gienger at uni-konstanz.de (Pascal Gienger) Date: Sun, 04 Jan 2009 08:19:31 +0100 Subject: choosing a file system In-Reply-To: <20090103144519.GB7152@khazad-dum.debian.net> References: <1617f8010812300849k1c7c878bl2f17e8d4287c10be@mail.gmail.com> <495A95F2.1010003@email.arizona.edu> <20081231005129.GA15299@brong.net> <1617f8010812310247l4c94e47fr296efceb30951d8a@mail.gmail.com> <1230727101.5300.6.camel@linux-nnci.site> <20081231134616.GA32353@helsinki.fi> <1230743048.6328.11.camel@linux-nnci.site> <20090103144519.GB7152@khazad-dum.debian.net> Message-ID: Henrique de Moraes Holschuh wrote: > Ext4, I never tried. Nor reiser3. I may have to, we will build a brand > new Cyrus spool (small, just 5K users) next month, and the XFS unlink > [lack of] performance worries me. Nobody likes deletes. Even databases used to mark deleted space only as "deleted" until a vacuum (Postgres) or other periodical maintenance command was run. Cyrus offers a similiar construct named "delayed expunge". Before we migrated our mail system to Solaris 10 it ran on Linux 2.4 with XFS on a FC SAN device. Deletes were extremely slow so we had to delay the expunges until the weekend, even on night they were too slow and too IO congesting. On the other hand, XFS was the only Linux filesystems capable to handle our 5 million files (at that time, we're now at 33 million) we had in these days with an acceptable performance. Ext3 was way too slow with directories with > 1000 files (but many things have changed from kernel 2.4.x to nowadays kernels), IBM jfs was not stable (it crashed during a high load test, which was an immediate k.o.). We were reluctant to use Reiser then as it was "too new" in 2001. From lars at lhanke.de Sun Jan 4 13:06:39 2009 From: lars at lhanke.de (Lars Hanke) Date: Sun, 04 Jan 2009 19:06:39 +0100 Subject: ldapdb auxprop configuration In-Reply-To: <495F9392.80902@gmx.net> References: <495E65D3.7050805@lhanke.de> <495E71FA.1070708@olp.net> <495E8904.6020506@lhanke.de> <495E9604.7030201@olp.net> <495F87BF.8000700@lhanke.de> <495F9392.80902@gmx.net> Message-ID: <4960FAAF.9050106@lhanke.de> Hi all! Sorry for cross-posting, but since this appears to be SASL related, I switch to the SASL list and leave this message in the cyrus-imap list for others to follow. So when answering to this, please check that you're not crossposting the answer. Summary for the SASL list subscribers, who have missed the start of this thread: I'm running cyrus-imap to authenticate users using the ldapdb auxprop against a remote ldaps: host. During the DIGEST-MD5 or CRAM-MD5 authentication of the user using imtest imapd SEGFAULTs. The ltrace suggests that it happens somewhere in the SASL layer. The setup is Debian Lenny kept current daily on an Intel Core2-Quad, i.e. amd64 build. @ Torsten Schlabach: > One comment suggested that the problem might be one of the Debian > specific patches! Did you try to build a package without them? Not yet, but I'm determined to get that issue resolved. One of the larger problems could be that Debian uses GnuTLS instead of OpenSSL. I had some severe issues with that kind of porting some years back with OpenLDAP. @ Dan White: I produced debugging versions of cyrus-imap , cyrus-sasl, and openldap and created a backtrace of the crash. See the end of this message. @ cyrus-imap list For some reason the method using the "debug_command" in /etc/imapd.conf and the "-D" option for imapd in "/etc/cyrus.conf" as described in https://langhorst.com/cgi-bin/dwww//usr/share/doc/cyrus21-common/README.Debian.debug.gz does not work, i.e. it does not produce any logs in /tmp. Am I missing something? So what I did was to use CYRUS_VERBOSE=100 in /etc/default/cyrus2.2 and used the 15 second delay to attach a gdb. The following happened and produced the backtrace of the SEGFAULT: hermod:/# imtest -u cyrus -a cyrus -v -p imap -m DIGEST-MD5 hermod.mgr S: * OK hermod.mgr Cyrus IMAP4 v2.2.13-Debian-2.2.13-14 server ready C: C01 CAPABILITY S: * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS ID NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE IDLE STARTTLS AUTH=NTLM AUTH=DIGEST-MD5 AUTH=CRAM-MD5 SASL-IR S: C01 OK Completed C: A01 AUTHENTICATE DIGEST-MD5 S: + bm9uY2U9IjNFZzIrY2xsci84dmREdXprTkd3a1VmL25XYTRBVnRXQmMxSGpndFBiVEk9IixyZWFsbT0iaGVybW9kLm1nciIscW9wPSJhdXRoLGF1dGgtaW50LGF1dGgtY29uZiIsY2lwaGVyPSJyYzQtNDAscmM0LTU2LHJjNCxkZXMsM2RlcyIsbWF4YnVmPTQwOTYsY2hhcnNldD11dGYtOCxhbGdvcml0aG09bWQ1LXNlc3M= Please enter your password: C: dXNlcm5hbWU9ImN5cnVzIixyZWFsbT0iaGVybW9kLm1nciIsbm9uY2U9IjNFZzIrY2xsci84dmREdXprTkd3a1VmL25XYTRBVnRXQmMxSGpndFBiVEk9Iixjbm9uY2U9IjluczF0dmwwMUhWU095dzlNZXRXK0ltRnVyWHRINDd4TFhyUjEvcXpNZHM9IixuYz0wMDAwMDAwMSxxb3A9YXV0aC1jb25mLGNpcGhlcj1yYzQsbWF4YnVmPTEwMjQsZGlnZXN0LXVyaT0iaW1hcC9oZXJtb2QubWdyIixyZXNwb25zZT1lZmYxZjk2MjUyNzlmY2UyMDY3MmIxOTg1NjIzZmIwYw== failure: prot layer failure Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fa6ca1e3700 (LWP 5409)] 0x00007fa6c72ed4aa in pthread_mutex_lock () from /lib/libpthread.so.0 (gdb) bt #0 0x00007fa6c72ed4aa in pthread_mutex_lock () from /lib/libpthread.so.0 #1 0x00007fa6c32b75a9 in ldap_pvt_thread_mutex_lock (mutex=0x1) at /home/admin/packages/openldap/openldap-2.4.11/libraries/libldap_r/thr_posix.c:296 #2 0x00007fa6c32c112b in ldap_pvt_sasl_mutex_lock (mutex=0x1) at cyrus.c:1294 #3 0x00007fa6c4b69828 in digestmd5_client_mech_step (conn_context=0x2094440, params=0x20960b0, serverin=0x0, serverinlen=0, prompt_need=0x7fffd21e8760, clientout=0x7fffd21e8748, clientoutlen=0x7fffd21e875c, oparams=0x209a510) at digestmd5.c:3955 #4 0x00007fa6c9dc25e6 in sasl_client_step (conn=0x2099ca0, serverin=0x0, serverinlen=0, prompt_need=0x7fffd21e8760, clientout=0x7fffd21e8748, clientoutlen=0x7fffd21e875c) at client.c:658 #5 0x00007fa6c9dc2445 in sasl_client_start (conn=0x2099ca0, mechlist=0x2041d40 "DIGEST-MD5", prompt_need=0x7fffd21e8760, clientout=0x7fffd21e8748, clientoutlen=0x7fffd21e875c, mech=0x7fffd21e8778) at client.c:606 #6 0x00007fa6c32bfc79 in ldap_int_sasl_bind (ld=0x2053880, dn=0x0, mechs=0x2041d40 "DIGEST-MD5", sctrls=0x0, cctrls=0x0, flags=2, interact=0x7fa6c34fd704 , defaults=0x204dce0) at cyrus.c:689 #7 0x00007fa6c32c3b7f in ldap_sasl_interactive_bind_s (ld=0x2053880, dn=0x0, mechs=0x2041d40 "DIGEST-MD5", serverControls=0x0, clientControls=0x0, flags=2, interact=0x7fa6c34fd704 , defaults=0x204dce0) at sasl.c:464 #8 0x00007fa6c34fd96c in ldapdb_connect (ctx=0x204dce0, sparams=0x20516c0, user=0x2052f71 "cyrus", ulen=5, cp=0x7fffd21e8910) at ldapdb.c:106 #9 0x00007fa6c34fdd45 in ldapdb_auxprop_lookup (glob_context=0x204dce0, sparams=0x20516c0, flags=0, user=0x2052f71 "cyrus", ulen=5) at ldapdb.c:178 #10 0x00007fa6c9dbe881 in _sasl_auxprop_lookup (sparams=0x20516c0, flags=0, user=0x2052f71 "cyrus", ulen=5) at auxprop.c:898 #11 0x00007fa6c9dbf309 in _sasl_canon_user (conn=0x20521d0, user=0x2052f71 "cyrus", ulen=5, flags=1, oparams=0x2052a40) at canonusr.c:190 #12 0x00007fa6c4b6556b in digestmd5_server_mech_step2 (stext=0x2054080, sparams=0x20516c0, clientin=0x7fffd21e8e10 "username=\"cyrus\",realm=\"hermod.mgr\",nonce=\"3Eg2+cllr/8vdDuzkNGwkUf/nWa4AVtWBc1HjgtPbTI=\",cnonce=\"9ns1tvl01HVSOyw9MetW+ImFurXtH47xLXrR1/qzMds=\",nc=00000001,qop=auth-conf,cipher=rc4,maxbuf=1024,digest-u"..., clientinlen=262, serverout=0x7fffd21e8e00, serveroutlen=0x7fffd21e8dfc, oparams=0x2052a40) at digestmd5.c:2301 #13 0x00007fa6c4b666cc in digestmd5_server_mech_step (conn_context=0x2054080, sparams=0x20516c0, clientin=0x7fffd21e8e10 "username=\"cyrus\",realm=\"hermod.mgr\",nonce=\"3Eg2+cllr/8vdDuzkNGwkUf/nWa4AVtWBc1HjgtPbTI=\",cnonce=\"9ns1tvl01HVSOyw9MetW+ImFurXtH47xLXrR1/qzMds=\",nc=00000001,qop=auth-conf,cipher=rc4,maxbuf=1024,digest-u"..., clientinlen=262, serverout=0x7fffd21e8e00, serveroutlen=0x7fffd21e8dfc, oparams=0x2052a40) at digestmd5.c:2689 #14 0x00007fa6c9dcd696 in sasl_server_step (conn=0x20521d0, clientin=0x7fffd21e8e10 "username=\"cyrus\",realm=\"hermod.mgr\",nonce=\"3Eg2+cllr/8vdDuzkNGwkUf/nWa4AVtWBc1HjgtPbTI=\",cnonce=\"9ns1tvl01HVSOyw9MetW+ImFurXtH47xLXrR1/qzMds=\",nc=00000001,qop=auth-conf,cipher=rc4,maxbuf=1024,digest-u"..., clientinlen=262, serverout=0x7fffd21e8e00, serveroutlen=0x7fffd21e8dfc) at server.c:1433 #15 0x000000000044ae85 in saslserver (conn=0x20521d0, mech=0x2054010 "DIGEST-MD5", init_resp=0x0, resp_prefix=0x473e03 "", continuation=0x473e27 "+ ", empty_chal=0x473e03 "", pin=0x2045a20, pout=0x2045ad0, sasl_result=0x7fffd21ee614, success_data=0x0) at saslserver.c:134 #16 0x000000000040e617 in cmd_authenticate (tag=0x2053eb0 "A01", authtype=0x2054010 "DIGEST-MD5", resp=0x0) at imapd.c:1888 #17 0x000000000040ae83 in cmdloop () at imapd.c:921 #18 0x000000000040a59e in service_main (argc=1, argv=0x2041010, envp=0x7fffd21f0f48) at imapd.c:691 #19 0x00000000004083a1 in main (argc=3, argv=0x7fffd21f0f28, envp=0x7fffd21f0f48) at service.c:533 Versions: hermod:~/imap# dpkg -l '*cyrus*' | grep '^ii' ii cyrus-admin-2.2 2.2.13-14 Cyrus mail system (administration tools) ii cyrus-clients-2.2 2.2.13-14+b3 Cyrus mail system (test clients) ii cyrus-common-2.2 2.2.13-14 Cyrus mail system (common files) ii cyrus-imapd-2.2 2.2.13-14 Cyrus mail system (IMAP support) ii libcyrus-imap-perl22 2.2.13-14+b3 Interface to Cyrus imap client imclient libr hermod:~/imap# dpkg -l '*sasl*' | grep '^ii' ii libsasl2-2 2.1.22.dfsg1-23 Cyrus SASL - authentication abstraction libr ii libsasl2-modules 2.1.22.dfsg1-23 Cyrus SASL - pluggable authentication module ii libsasl2-modules-gssapi-mit 2.1.22.dfsg1-23 Cyrus SASL - pluggable authentication module ii libsasl2-modules-ldap 2.1.22.dfsg1-23 Cyrus SASL - pluggable authentication module ii sasl2-bin 2.1.22.dfsg1-23 Cyrus SASL - administration programs for SAS hermod:~# dpkg -l '*ldap*' | grep '^ii' ii ldap-utils 2.4.11-1 OpenLDAP utilities ii libldap-2.4-2 2.4.11-1 OpenLDAP libraries From tschlabach at gmx.net Sun Jan 4 13:26:03 2009 From: tschlabach at gmx.net (Torsten Schlabach) Date: Sun, 04 Jan 2009 19:26:03 +0100 Subject: ldapdb auxprop configuration In-Reply-To: <4960FAAF.9050106@lhanke.de> References: <495E65D3.7050805@lhanke.de> <495E71FA.1070708@olp.net> <495E8904.6020506@lhanke.de> <495E9604.7030201@olp.net> <495F87BF.8000700@lhanke.de> <495F9392.80902@gmx.net> <4960FAAF.9050106@lhanke.de> Message-ID: <4960FF3B.3060408@gmx.net> Hi Lars! > @ Torsten Schlabach: >> One comment suggested that the problem might be one of the Debian >> specific patches! Did you try to build a package without them? > Not yet, but I'm determined to get that issue resolved. One of the > larger problems could be that Debian uses GnuTLS instead of OpenSSL. I > had some severe issues with that kind of porting some years back with > OpenLDAP. It would be nice if you could give it a try. Actually, I would need 1-2 days to get myself a fresh server to replicate the setup and join the effort. What I do remember is: I had build the respective parts from source tarballs once and it did *not* segfault. But it's too long ago to tell if I had been using GnuTLS or OpenSSL. > @ Dan White: > I produced debugging versions of cyrus-imap , cyrus-sasl, and openldap If you have all the infrastructure set up to create your own version of the packages, it should be a five minute exercise to empty the debian/patches directory, re-build, re-install and see if the issue goes away. In case it does, we're on the wrong mailist list and should continue the discussion in a Debian developer forum IMO. Regards, Torsten Lars Hanke schrieb: > Hi all! > > Sorry for cross-posting, but since this appears to be SASL related, I > switch to the SASL list and leave this message in the cyrus-imap list > for others to follow. So when answering to this, please check that > you're not crossposting the answer. > > Summary for the SASL list subscribers, who have missed the start of this > thread: > > I'm running cyrus-imap to authenticate users using the ldapdb auxprop > against a remote ldaps: host. During the DIGEST-MD5 or CRAM-MD5 > authentication of the user using imtest imapd SEGFAULTs. The ltrace > suggests that it happens somewhere in the SASL layer. The setup is > Debian Lenny kept current daily on an Intel Core2-Quad, i.e. amd64 build. > > @ Torsten Schlabach: >> One comment suggested that the problem might be one of the Debian >> specific patches! Did you try to build a package without them? > Not yet, but I'm determined to get that issue resolved. One of the > larger problems could be that Debian uses GnuTLS instead of OpenSSL. I > had some severe issues with that kind of porting some years back with > OpenLDAP. > > @ Dan White: > I produced debugging versions of cyrus-imap , cyrus-sasl, and openldap > and created a backtrace of the crash. See the end of this message. > > @ cyrus-imap list > For some reason the method using the "debug_command" in /etc/imapd.conf > and the "-D" option for imapd in "/etc/cyrus.conf" as described in > https://langhorst.com/cgi-bin/dwww//usr/share/doc/cyrus21-common/README.Debian.debug.gz > does not work, i.e. it does not produce any logs in /tmp. Am I missing > something? > > So what I did was to use CYRUS_VERBOSE=100 in /etc/default/cyrus2.2 and > used the 15 second delay to attach a gdb. The following happened and > produced the backtrace of the SEGFAULT: > > hermod:/# imtest -u cyrus -a cyrus -v -p imap -m DIGEST-MD5 hermod.mgr > S: * OK hermod.mgr Cyrus IMAP4 v2.2.13-Debian-2.2.13-14 server ready > C: C01 CAPABILITY > S: * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS ID > NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT > THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE IDLE STARTTLS > AUTH=NTLM AUTH=DIGEST-MD5 AUTH=CRAM-MD5 SASL-IR > S: C01 OK Completed > C: A01 AUTHENTICATE DIGEST-MD5 > S: + > bm9uY2U9IjNFZzIrY2xsci84dmREdXprTkd3a1VmL25XYTRBVnRXQmMxSGpndFBiVEk9IixyZWFsbT0iaGVybW9kLm1nciIscW9wPSJhdXRoLGF1dGgtaW50LGF1dGgtY29uZiIsY2lwaGVyPSJyYzQtNDAscmM0LTU2LHJjNCxkZXMsM2RlcyIsbWF4YnVmPTQwOTYsY2hhcnNldD11dGYtOCxhbGdvcml0aG09bWQ1LXNlc3M= > > Please enter your password: > C: > dXNlcm5hbWU9ImN5cnVzIixyZWFsbT0iaGVybW9kLm1nciIsbm9uY2U9IjNFZzIrY2xsci84dmREdXprTkd3a1VmL25XYTRBVnRXQmMxSGpndFBiVEk9Iixjbm9uY2U9IjluczF0dmwwMUhWU095dzlNZXRXK0ltRnVyWHRINDd4TFhyUjEvcXpNZHM9IixuYz0wMDAwMDAwMSxxb3A9YXV0aC1jb25mLGNpcGhlcj1yYzQsbWF4YnVmPTEwMjQsZGlnZXN0LXVyaT0iaW1hcC9oZXJtb2QubWdyIixyZXNwb25zZT1lZmYxZjk2MjUyNzlmY2UyMDY3MmIxOTg1NjIzZmIwYw== > > failure: prot layer failure > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x7fa6ca1e3700 (LWP 5409)] > 0x00007fa6c72ed4aa in pthread_mutex_lock () from /lib/libpthread.so.0 > (gdb) bt > #0 0x00007fa6c72ed4aa in pthread_mutex_lock () from /lib/libpthread.so.0 > #1 0x00007fa6c32b75a9 in ldap_pvt_thread_mutex_lock (mutex=0x1) > at > /home/admin/packages/openldap/openldap-2.4.11/libraries/libldap_r/thr_posix.c:296 > > #2 0x00007fa6c32c112b in ldap_pvt_sasl_mutex_lock (mutex=0x1) at > cyrus.c:1294 > #3 0x00007fa6c4b69828 in digestmd5_client_mech_step > (conn_context=0x2094440, params=0x20960b0, > serverin=0x0, serverinlen=0, prompt_need=0x7fffd21e8760, > clientout=0x7fffd21e8748, > clientoutlen=0x7fffd21e875c, oparams=0x209a510) at digestmd5.c:3955 > #4 0x00007fa6c9dc25e6 in sasl_client_step (conn=0x2099ca0, serverin=0x0, > serverinlen=0, > prompt_need=0x7fffd21e8760, clientout=0x7fffd21e8748, > clientoutlen=0x7fffd21e875c) at client.c:658 > #5 0x00007fa6c9dc2445 in sasl_client_start (conn=0x2099ca0, > mechlist=0x2041d40 "DIGEST-MD5", > prompt_need=0x7fffd21e8760, clientout=0x7fffd21e8748, > clientoutlen=0x7fffd21e875c, > mech=0x7fffd21e8778) at client.c:606 > #6 0x00007fa6c32bfc79 in ldap_int_sasl_bind (ld=0x2053880, dn=0x0, > mechs=0x2041d40 "DIGEST-MD5", > sctrls=0x0, cctrls=0x0, flags=2, interact=0x7fa6c34fd704 > , defaults=0x204dce0) > at cyrus.c:689 > #7 0x00007fa6c32c3b7f in ldap_sasl_interactive_bind_s (ld=0x2053880, > dn=0x0, > mechs=0x2041d40 "DIGEST-MD5", serverControls=0x0, clientControls=0x0, > flags=2, > interact=0x7fa6c34fd704 , defaults=0x204dce0) at > sasl.c:464 > #8 0x00007fa6c34fd96c in ldapdb_connect (ctx=0x204dce0, > sparams=0x20516c0, user=0x2052f71 "cyrus", > ulen=5, cp=0x7fffd21e8910) at ldapdb.c:106 > #9 0x00007fa6c34fdd45 in ldapdb_auxprop_lookup (glob_context=0x204dce0, > sparams=0x20516c0, flags=0, > user=0x2052f71 "cyrus", ulen=5) at ldapdb.c:178 > #10 0x00007fa6c9dbe881 in _sasl_auxprop_lookup (sparams=0x20516c0, > flags=0, user=0x2052f71 "cyrus", > ulen=5) at auxprop.c:898 > #11 0x00007fa6c9dbf309 in _sasl_canon_user (conn=0x20521d0, > user=0x2052f71 "cyrus", ulen=5, flags=1, > oparams=0x2052a40) at canonusr.c:190 > #12 0x00007fa6c4b6556b in digestmd5_server_mech_step2 (stext=0x2054080, > sparams=0x20516c0, > clientin=0x7fffd21e8e10 > "username=\"cyrus\",realm=\"hermod.mgr\",nonce=\"3Eg2+cllr/8vdDuzkNGwkUf/nWa4AVtWBc1HjgtPbTI=\",cnonce=\"9ns1tvl01HVSOyw9MetW+ImFurXtH47xLXrR1/qzMds=\",nc=00000001,qop=auth-conf,cipher=rc4,maxbuf=1024,digest-u"..., > clientinlen=262, serverout=0x7fffd21e8e00, > serveroutlen=0x7fffd21e8dfc, oparams=0x2052a40) at digestmd5.c:2301 > #13 0x00007fa6c4b666cc in digestmd5_server_mech_step > (conn_context=0x2054080, sparams=0x20516c0, > clientin=0x7fffd21e8e10 > "username=\"cyrus\",realm=\"hermod.mgr\",nonce=\"3Eg2+cllr/8vdDuzkNGwkUf/nWa4AVtWBc1HjgtPbTI=\",cnonce=\"9ns1tvl01HVSOyw9MetW+ImFurXtH47xLXrR1/qzMds=\",nc=00000001,qop=auth-conf,cipher=rc4,maxbuf=1024,digest-u"..., > clientinlen=262, serverout=0x7fffd21e8e00, > serveroutlen=0x7fffd21e8dfc, oparams=0x2052a40) at digestmd5.c:2689 > #14 0x00007fa6c9dcd696 in sasl_server_step (conn=0x20521d0, > clientin=0x7fffd21e8e10 > "username=\"cyrus\",realm=\"hermod.mgr\",nonce=\"3Eg2+cllr/8vdDuzkNGwkUf/nWa4AVtWBc1HjgtPbTI=\",cnonce=\"9ns1tvl01HVSOyw9MetW+ImFurXtH47xLXrR1/qzMds=\",nc=00000001,qop=auth-conf,cipher=rc4,maxbuf=1024,digest-u"..., > clientinlen=262, serverout=0x7fffd21e8e00, serveroutlen=0x7fffd21e8dfc) > at server.c:1433 > #15 0x000000000044ae85 in saslserver (conn=0x20521d0, mech=0x2054010 > "DIGEST-MD5", init_resp=0x0, > resp_prefix=0x473e03 "", continuation=0x473e27 "+ ", empty_chal=0x473e03 > "", pin=0x2045a20, > pout=0x2045ad0, sasl_result=0x7fffd21ee614, success_data=0x0) at > saslserver.c:134 > #16 0x000000000040e617 in cmd_authenticate (tag=0x2053eb0 "A01", > authtype=0x2054010 "DIGEST-MD5", > resp=0x0) at imapd.c:1888 > #17 0x000000000040ae83 in cmdloop () at imapd.c:921 > #18 0x000000000040a59e in service_main (argc=1, argv=0x2041010, > envp=0x7fffd21f0f48) at imapd.c:691 > #19 0x00000000004083a1 in main (argc=3, argv=0x7fffd21f0f28, > envp=0x7fffd21f0f48) at service.c:533 > > Versions: > hermod:~/imap# dpkg -l '*cyrus*' | grep '^ii' > ii cyrus-admin-2.2 2.2.13-14 Cyrus mail system (administration tools) > ii cyrus-clients-2.2 2.2.13-14+b3 Cyrus mail system (test clients) > ii cyrus-common-2.2 2.2.13-14 Cyrus mail system (common files) > ii cyrus-imapd-2.2 2.2.13-14 Cyrus mail system (IMAP support) > ii libcyrus-imap-perl22 2.2.13-14+b3 Interface to Cyrus imap client > imclient libr > hermod:~/imap# dpkg -l '*sasl*' | grep '^ii' > ii libsasl2-2 2.1.22.dfsg1-23 Cyrus SASL - authentication abstraction libr > ii libsasl2-modules 2.1.22.dfsg1-23 Cyrus SASL - pluggable > authentication module > ii libsasl2-modules-gssapi-mit 2.1.22.dfsg1-23 Cyrus SASL - pluggable > authentication module > ii libsasl2-modules-ldap 2.1.22.dfsg1-23 Cyrus SASL - pluggable > authentication module > ii sasl2-bin 2.1.22.dfsg1-23 Cyrus SASL - administration programs for SAS > hermod:~# dpkg -l '*ldap*' | grep '^ii' > ii ldap-utils 2.4.11-1 OpenLDAP utilities > ii libldap-2.4-2 2.4.11-1 OpenLDAP libraries From lars at lhanke.de Sun Jan 4 13:33:49 2009 From: lars at lhanke.de (Lars Hanke) Date: Sun, 04 Jan 2009 19:33:49 +0100 Subject: ldapdb auxprop configuration In-Reply-To: <4960FAAF.9050106@lhanke.de> References: <495E65D3.7050805@lhanke.de> <495E71FA.1070708@olp.net> <495E8904.6020506@lhanke.de> <495E9604.7030201@olp.net> <495F87BF.8000700@lhanke.de> <495F9392.80902@gmx.net> <4960FAAF.9050106@lhanke.de> Message-ID: <4961010D.3070900@lhanke.de> Hi there, > For some reason the method using the "debug_command" in /etc/imapd.conf > and the "-D" option for imapd in "/etc/cyrus.conf" as described in > https://langhorst.com/cgi-bin/dwww//usr/share/doc/cyrus21-common/README.Debian.debug.gz > does not work, i.e. it does not produce any logs in /tmp. Am I missing > something? > Sorry, after dumping /etc/cyrus.conf another time I found that -D was in the wrong place on the command line. Now it works. Regards, - lars. From awilliam at whitemice.org Sun Jan 4 15:58:32 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Sun, 04 Jan 2009 15:58:32 -0500 Subject: choosing a file system In-Reply-To: References: <1617f8010812300849k1c7c878bl2f17e8d4287c10be@mail.gmail.com> <20081231142023.GB32353@helsinki.fi> <8765283395DA4A1AB912FC5D3DED2F89@robmhp> <200901020850.22355.jmadden@ivytech.edu> <1230963401.20171.95.camel@darkstar.netcore.co.in> Message-ID: <1231102712.4921.11.camel@linux-nnci.site> > > I had the feeling whatever optimizations done at the FS level would give > > us a max of 5-10% benefit. > > We migrated from ext3 to reiserfs on our cyrus servers with 30k > > mailboxes. I am not sure I saw a great benefit in terms of the iowait. > > At peak times I always see a iowait of 40-60% > To be honest, that's not what we saw in our ext3 <-> reiserfs tests. > What mount options are you using? Are you using the mount options I > mentioned? > noatime,nodiratime,notail,data=ordered FYI, noatime implies nodiratime. You can set nodiratime without atime, but not atime without nodiratime. > > But the new Solid-State-Disks seem very promising. They are claimed to > > give 30x the throughput of a 15k rpm disk. If IO improves by 30 times > > that should make all these optimizations unnecessary. > > As my boss used to tell me ... Good hardware always compensates for > > not-so-good software. > What we've found is that the meta-data (eg mailbox.db, seen db's, quota > files, cyrus.* files) use WAY more IO than the email data, but only use > 1/20th the space. Ditto. The meta-data is very much the hot-spot for I/O. > By separating the meta data onto RAID1 10k/15k RPM drives, and the email > data onto RAID5/6 7.2k RPM drives, you can get a good balance of > space/speed. Agree. From awilliam at whitemice.org Sun Jan 4 16:02:14 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Sun, 04 Jan 2009 16:02:14 -0500 Subject: choosing a file system In-Reply-To: References: <1617f8010812300849k1c7c878bl2f17e8d4287c10be@mail.gmail.com> <495A95F2.1010003@email.arizona.edu> <20081231005129.GA15299@brong.net> <1617f8010812310247l4c94e47fr296efceb30951d8a@mail.gmail.com> <1230727101.5300.6.camel@linux-nnci.site> <20081231134616.GA32353@helsinki.fi> <1230743048.6328.11.camel@linux-nnci.site> <20090103144519.GB7152@khazad-dum.debian.net> Message-ID: <1231102934.4921.15.camel@linux-nnci.site> > On the other hand, XFS was the only Linux filesystems capable to handle our > 5 million files (at that time, we're now at 33 million) we had in these > days with an acceptable performance. Ext3 was way too slow with directories > with > 1000 files ?(but many things have changed from kernel 2.4.x to > nowadays kernels) It has; not Cyrus but another application we had, and we had to make a 'hashed' directory structure to avoid the many-files-in-a-directory situation. But this isn't true anymore, ext3 performs well with very large directories. From mangoo at wpkg.org Mon Jan 5 06:51:15 2009 From: mangoo at wpkg.org (Tomasz Chmielewski) Date: Mon, 05 Jan 2009 12:51:15 +0100 Subject: Cyrus "hanging" when connecting to POP3 port several times Message-ID: <4961F433.9060900@wpkg.org> My Cyrus is hanging when I try to connect to the POP3 or POP3S port several times, i.e. when connecting like this three-four-five times in a row: $ telnet 127.0.0.1 110 Trying 127.0.0.1... Connected to localhost (127.0.0.1). Escape character is '^]'. +OK <2041089544.1231155840 at imap> imap1 Cyrus POP3 v2.3.13 server ready quit +OK Connection closed by foreign host. $ telnet 127.0.0.1 110 Trying 127.0.0.1... Connected to localhost (127.0.0.1). Escape character is '^]'. +OK <2300500967.1231155842 at imap> imap1 Cyrus POP3 v2.3.13 server ready quit +OK Connection closed by foreign host. $ telnet 127.0.0.1 110 Trying 127.0.0.1... Connected to localhost (127.0.0.1). Escape character is '^]'. +OK <2951207308.1231155844 at imap> imap1 Cyrus POP3 v2.3.13 server ready quit +OK Connection closed by foreign host. $ telnet 127.0.0.1 110 Trying 127.0.0.1... Connected to localhost (127.0.0.1). Escape character is '^]'. And it waits like that for 10, 20, 30 seconds, until I get a reply, or just indefinitely, and I don't get any reply and connection times out. In the meantime, I can open yet another connection from another terminal. Or not, because it will hang similarly. The phenomenon does not happen for IMAP connections. I tried Cyrus versions 2.3.11, 2.3.12 and 2.3.13 with different settings on different machines and I could reproduce this issue everywhere. There is no firewall limiting the number of connections. Is it a known problem? If yes, are there any workarounds to that? If no, what more information should I provide to help fix it? -- Tomasz Chmielewski http://wpkg.org From mangoo at wpkg.org Mon Jan 5 08:07:33 2009 From: mangoo at wpkg.org (Tomasz Chmielewski) Date: Mon, 05 Jan 2009 14:07:33 +0100 Subject: Cyrus "hanging" when connecting to POP3 port several times In-Reply-To: <4961F433.9060900@wpkg.org> References: <4961F433.9060900@wpkg.org> Message-ID: <49620615.301@wpkg.org> Tomasz Chmielewski schrieb: > My Cyrus is hanging when I try to connect to the POP3 or POP3S port > several times, i.e. when connecting like this three-four-five times in a > row: > $ telnet 127.0.0.1 110 > Trying 127.0.0.1... > Connected to localhost (127.0.0.1). > Escape character is '^]'. Stracing it leads me to an empty /dev/random. After looking at man pages, I see there is no option to specify an alternative random file location (i.e., /dev/urandom). Is recompilation the only way to get around this problem? -- Tomasz Chmielewski http://wpkg.org From Rudy.Gevaert at UGent.be Mon Jan 5 08:16:36 2009 From: Rudy.Gevaert at UGent.be (Rudy Gevaert) Date: Mon, 05 Jan 2009 14:16:36 +0100 Subject: Cyrus "hanging" when connecting to POP3 port several times In-Reply-To: <49620615.301@wpkg.org> References: <4961F433.9060900@wpkg.org> <49620615.301@wpkg.org> Message-ID: <49620834.1040509@UGent.be> Tomasz Chmielewski wrote: > Tomasz Chmielewski schrieb: >> My Cyrus is hanging when I try to connect to the POP3 or POP3S port >> several times, i.e. when connecting like this three-four-five times in a >> row: > >> $ telnet 127.0.0.1 110 >> Trying 127.0.0.1... >> Connected to localhost (127.0.0.1). >> Escape character is '^]'. > > Stracing it leads me to an empty /dev/random. > > After looking at man pages, I see there is no option to specify an > alternative random file location (i.e., /dev/urandom). > > Is recompilation the only way to get around this problem? You should recompile sasl to use /dev/urandom. Rudy -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Rudy Gevaert Rudy.Gevaert at UGent.be tel:+32 9 264 4734 Directie ICT, afd. Infrastructuur ICT Department, Infrastructure office Groep Systemen Systems group Universiteit Gent Ghent University Krijgslaan 281, gebouw S9, 9000 Gent, Belgie www.UGent.be -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- From mangoo at wpkg.org Mon Jan 5 08:36:44 2009 From: mangoo at wpkg.org (Tomasz Chmielewski) Date: Mon, 05 Jan 2009 14:36:44 +0100 Subject: Cyrus "hanging" when connecting to POP3 port several times In-Reply-To: <20090105131621.GA14015@mail.me.com> References: <4961F433.9060900@wpkg.org> <49620615.301@wpkg.org> <20090105131621.GA14015@mail.me.com> Message-ID: <49620CEC.3070703@wpkg.org> Mike Eggleston schrieb: > On Mon, 05 Jan 2009, Tomasz Chmielewski might have said: >> Stracing it leads me to an empty /dev/random. >> >> After looking at man pages, I see there is no option to specify an >> alternative random file location (i.e., /dev/urandom). >> >> Is recompilation the only way to get around this problem? > > What about making a symlink from /dev/urandom to /dev/random? > > Mike > > # ln -s /dev/urandom /dev/random # ln -s /dev/urandom /dev/random ln: creating symbolic link `/dev/random': File exists I know, I could (re)move /dev/random first, but then other programs won't have real random data where they usually expect it (/dev/random). Question is, do other programs really need a true /dev/random on a machine running IMAP/S and POP/S... Another "workaround" is disabling APOP in imapd.conf, but it potentially lowers the security... The best way would be connecting a mouse, keyboard, enabling X and placing some gamer to play a shooter - this should generate enough entropy... ;) -- Tomasz Chmielewski http://wpkg.org From jvoorhees1 at gmail.com Mon Jan 5 09:13:09 2009 From: jvoorhees1 at gmail.com (Jason Voorhees) Date: Mon, 05 Jan 2009 09:13:09 -0500 Subject: Basic question In-Reply-To: <495B8E74.5020903@olp.net> References: <495A60B2.7070701@gmail.com> <495B8E74.5020903@olp.net> Message-ID: <49621575.10604@gmail.com> That was perfect, Thank you very much Dan! Now I know what configuration is appropiate for me. Bytes! Dan White escribi?: > Jason Voorhees wrote: >> Hi there: >> >> I'm planning to use Cyrus IMAP and OpenLDAP to authenticate users. >> Long time ago I used to configure Cyrus IMAP + Cyrus SASL using >> saslauthd with pam module. It was something simple. >> >> Then I used to configure Cyrus IMAP + Cyrus SASL using saslauthd with >> ldap module and /etc/saslauthd.conf without problems. That's fine. >> >> >> Now I would like to use Cyrus IMAP with OpenLDAP too, but I found that >> there are at least 2 ways: >> >> 1. Use Cyrus SASL with auxprop to authenticate users trough LDAP using >> auxprop_plugin: ldapdb, sasl_ldap_servers among other sasl_* directives. >> Right? >> >> 2. The other way is to use ldap_* directives like ldap_uri, ldap_filter >> among others. But I believe that I would need to use 'pts' module in >> auth_mech directive, right? >> >> The question is: What are pts, unix, krb and krb5 modules used for? >> What's the difference between them? Should I use pts module to make >> Cyrus talk directly to OpenLDAP...? Or should I use Cyrus SASL with >> auxprop plugin to make the authentication to OpenLDAP? >> >> Is there a place where I can get some clear information about these >> items? Man pages are not too clear :S >> >> Thanks people :) >> > > Jason, > > Available documentation that I'm aware of includes: > > /doc/options.html (within the cyrus-sasl source) which documents how to > configure the ldapdb auxprop plugin > > /saslauthd/LDAP_SASLAUTHD (within the cyrus-sasl source) which discusses > how to configure the ldap saslauthd backend > > /doc/overview.html (within the cyrus-imap source), in the 'Kerberos vs. > Unix Authorization' section, which discusses authorization. > > As I understand it, the ldapdb auxprop plugin is entirely within the > realm of cyrus sasl (authentication), and the auth_mech directive in > imapd.conf is cyrus imapd specific, and only handles authorization. > > The auth_mech options (pts, unix, krb and krb5) direct how cyrus imapd > authorizes users to access mailboxes/resources *after* they have been > authenticated. The kerberos options direct imapd to perform some > canonicalization of the authenticating user before opening their mailbox > - so if a user connects as jsmith at EXAMPLE.COM, the kerberos options > could canonicalize that to 'jsmith', so that the server can open the > 'jsmith' mailbox instead of searching for a 'jsmith at EXAMPLE.COM' mailbox. > > The unix and pts options should only come in to play if you have > specified a 'group:staff' style ACL for your mailboxes. It tells the > imapd server how to resolve group membership to grant access to the > mailbox. The 'unix' option will perform a unix getgrent call, or > something like that, to determine if a user belongs to a group - using > nss for instance, which in turn can use the nss-ldap or nss-mysql > modules to lookup groups. However, that's pretty slow in my experience > and you'd need to make sure you're properly optimizing your LDAP database. > > The pts route can be used to reference and LDAP server directly to > resolve group membership within an LDAP database. > > - Dan From jmadden at ivytech.edu Mon Jan 5 10:49:59 2009 From: jmadden at ivytech.edu (John Madden) Date: Mon, 5 Jan 2009 10:49:59 -0500 Subject: choosing a file system In-Reply-To: References: <1617f8010812300849k1c7c878bl2f17e8d4287c10be@mail.gmail.com> <200901020850.22355.jmadden@ivytech.edu> Message-ID: <200901051049.59526.jmadden@ivytech.edu> > $ mount | wc -l > 92 Wow. > We've found that splitting the data up into more volumes + more cyrus > instances seems to help as well because it seems to reduce overall > contention points in the kernel + software (eg filesystem locks spread > across multiple mounts, db locks are spread across multiple dbs, etc) Makes sense. Single cyrus env here, might consider that in the future. At that point though, I'd probably consider Murder or similar. > Also one thing I did fail to mention, was that for the data volumes, you > should definitely be using the "notail" mount option. Unfortunately that's > not the default, and I think it probably should be. Tails packing is neat > for saving space, but it reduces the average meta-data density, which makes > "stating" lots of files in a directory a lot slower. I think that's what > you might have been seeing. Of course you also mounted "noatime,nodiratime" > on both? Yes, we were using notail,noatime,nodiratime. John -- John Madden Sr. UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden at ivytech.edu From david.lang at digitalinsight.com Mon Jan 5 12:12:33 2009 From: david.lang at digitalinsight.com (David Lang) Date: Mon, 5 Jan 2009 09:12:33 -0800 (PST) Subject: choosing a file system In-Reply-To: References: <1617f8010812300849k1c7c878bl2f17e8d4287c10be@mail.gmail.com><20081231142023.GB32353@helsinki.fi><8765283395DA4A1AB912FC5D3DED2F89@robmhp><200901020850.22355.jmadden@ivytech.edu><1230963401.20171.95.camel@darkstar.netcore.co.in> Message-ID: On Sat, 3 Jan 2009, Rob Mueller wrote: >> But the new Solid-State-Disks seem very promising. They are claimed to >> give 30x the throughput of a 15k rpm disk. If IO improves by 30 times >> that should make all these optimizations unnecessary. >> As my boss used to tell me ... Good hardware always compensates for >> not-so-good software. > > What we've found is that the meta-data (eg mailbox.db, seen db's, quota > files, cyrus.* files) use WAY more IO than the email data, but only use > 1/20th the space. > > By separating the meta data onto RAID1 10k/15k RPM drives, and the email > data onto RAID5/6 7.2k RPM drives, you can get a good balance of > space/speed. how do you move the cyrus* files onto other drives? David Lang From boutilpj at ednet.ns.ca Mon Jan 5 13:57:46 2009 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Mon, 05 Jan 2009 14:57:46 -0400 Subject: choosing a file system In-Reply-To: References: <1617f8010812300849k1c7c878bl2f17e8d4287c10be@mail.gmail.com><20081231142023.GB32353@helsinki.fi><8765283395DA4A1AB912FC5D3DED2F89@robmhp><200901020850.22355.jmadden@ivytech.edu><1230963401.20171.95.camel@darkstar.netcore.co.in> Message-ID: <4962582A.9040804@ednet.ns.ca> David Lang wrote: > On Sat, 3 Jan 2009, Rob Mueller wrote: > >>> But the new Solid-State-Disks seem very promising. They are claimed to >>> give 30x the throughput of a 15k rpm disk. If IO improves by 30 times >>> that should make all these optimizations unnecessary. >>> As my boss used to tell me ... Good hardware always compensates for >>> not-so-good software. >> What we've found is that the meta-data (eg mailbox.db, seen db's, quota >> files, cyrus.* files) use WAY more IO than the email data, but only use >> 1/20th the space. >> >> By separating the meta data onto RAID1 10k/15k RPM drives, and the email >> data onto RAID5/6 7.2k RPM drives, you can get a good balance of >> space/speed. > > how do you move the cyrus* files onto other drives? metapartition_files and metapartition-default imapd.conf options in cyrus-imapd 2.3.x > > David Lang > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -------------- next part -------------- A non-text attachment was scrubbed... Name: boutilpj.vcf Type: text/x-vcard Size: 286 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090105/240a6894/attachment.vcf From dom.lalot at gmail.com Mon Jan 5 14:26:17 2009 From: dom.lalot at gmail.com (LALOT Dominique) Date: Mon, 5 Jan 2009 20:26:17 +0100 Subject: choosing a file system In-Reply-To: <4962582A.9040804@ednet.ns.ca> References: <1617f8010812300849k1c7c878bl2f17e8d4287c10be@mail.gmail.com> <20081231142023.GB32353@helsinki.fi> <8765283395DA4A1AB912FC5D3DED2F89@robmhp> <200901020850.22355.jmadden@ivytech.edu> <1230963401.20171.95.camel@darkstar.netcore.co.in> <4962582A.9040804@ednet.ns.ca> Message-ID: <1617f8010901051126p5787d098m28f83249745de4fa@mail.gmail.com> 2009/1/5 Patrick Boutilier > David Lang wrote: > >> On Sat, 3 Jan 2009, Rob Mueller wrote: >> >> But the new Solid-State-Disks seem very promising. They are claimed to >>>> give 30x the throughput of a 15k rpm disk. If IO improves by 30 times >>>> that should make all these optimizations unnecessary. >>>> As my boss used to tell me ... Good hardware always compensates for >>>> not-so-good software. >>>> >>> What we've found is that the meta-data (eg mailbox.db, seen db's, quota >>> files, cyrus.* files) use WAY more IO than the email data, but only use >>> 1/20th the space. >>> >>> By separating the meta data onto RAID1 10k/15k RPM drives, and the email >>> data onto RAID5/6 7.2k RPM drives, you can get a good balance of >>> space/speed. >>> >> >> how do you move the cyrus* files onto other drives? >> > > metapartition_files and metapartition-default imapd.conf options in > cyrus-imapd 2.3.x So, then, may be we can easily store pure email data on an NFS appliance, keeping metadata on traditionnal filesystem, which can be synced using low level tools Dom > > > > >> David Lang >> ---- >> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ >> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki >> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html >> > > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > -- Dominique LALOT Ing?nieur Syst?mes et R?seaux http://annuaire.univmed.fr/showuser?uid=lalot -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090105/a1883b7d/attachment.html From robm at fastmail.fm Mon Jan 5 17:14:25 2009 From: robm at fastmail.fm (Rob Mueller) Date: Tue, 6 Jan 2009 09:14:25 +1100 Subject: choosing a file system References: <1617f8010812300849k1c7c878bl2f17e8d4287c10be@mail.gmail.com> <200901020850.22355.jmadden@ivytech.edu> <200901051049.59526.jmadden@ivytech.edu> Message-ID: <4C30AD5B38874B96A458631F9A71E19C@Atticus> >> We've found that splitting the data up into more volumes + more cyrus >> instances seems to help as well because it seems to reduce overall >> contention points in the kernel + software (eg filesystem locks spread >> across multiple mounts, db locks are spread across multiple dbs, etc) > > Makes sense. Single cyrus env here, might consider that in the future. > At > that point though, I'd probably consider Murder or similar. That should work fine as well. I believe murder just does two main things. 1. It's merges the mailboxes.db from each instance into each other instance, so you end up with just one giant single namespace 2. It proxies everything (imap/pop/lmtp) as needed to the appropriate instance if it's not the local one We don't use murder as we don't really need (1), and we do (2) ourselves with a combination of nginx and custom lmtpproxy tool. Rob From joar at linpro.no Tue Jan 6 07:11:41 2009 From: joar at linpro.no (Joar Jegleim) Date: Tue, 06 Jan 2009 13:11:41 +0100 Subject: howto list inactive users in cyrus: In-Reply-To: <494BD388.9000300@onlight.com> References: <494B8067.7060606@linpro.no> <494BD388.9000300@onlight.com> Message-ID: <49634A7D.1000100@linpro.no> Sorry for late reply, had a nice and long xmas vacation here :) Happy new year to everybody :) > I am looking into this right now. I think I have a Perl script which > could be easily modified to do what you want. Very interesting But as morgan states: " Are you sure the is the information you need? Here is what I see on a test server: localhost> info user.morgan {user.morgan}: condstore: false duplicatedeliver: false lastpop: lastupdate: 31-Oct-2008 14:27:57 -0700 partition: default sharedseen: false size: 5277441 The lastupdate field corresponds with the last time the account received an email, not my last login (which was yesterday). " if this is correct then info doesn't give me the information I need :( Joseph Brennan wrote: "How about checking the timestamp on the user.seen files? It seems to get touched at login, even if you don't open a message. Joseph Brennan Columbia University Information Technology " And Wesley Craig wrote: "There's a "last read" timestamp in the seen DB, which is probably more accurate. :wes" I'm looking into this right now Appreciate your replies . I'll post on the list how things turn out. regards Joar Jegleim Nic Bernstein wrote: > Joar Jegleim wrote: >> is there any way to easily list 'last login' for users with cyradm ? >> I got over 1000 accounts and want to find out if there are any inactive >> accounts as in "haven't logged on" for like 6 months or so' . >> >> ">cyradm info some.mailbox" gives me the info I'm looking for >> (basically), but I'm having trouble with howto script this command and >> sort it, is there any written perl script for this out there ? >> Or mabye there are other ways to get what I want ? >> > I am looking into this right now. I think I have a Perl script which > could be easily modified to do what you want. > > Cheers, > -nic > From blake at ispn.net Wed Jan 7 12:01:11 2009 From: blake at ispn.net (Blake Hudson) Date: Wed, 07 Jan 2009 11:01:11 -0600 Subject: howto list inactive users in cyrus: In-Reply-To: <05ECE370-7D29-4F7D-906D-E31B54BA62EE@umich.edu> References: <494B8067.7060606@linpro.no> <9B733BFFD7A4289F99CF942A@[192.168.2.14]> <05ECE370-7D29-4F7D-906D-E31B54BA62EE@umich.edu> Message-ID: <4964DFD7.8060203@ispn.net> I brought this subject up on the list a few times over the years and never received any interest in updating cyradm/fud to report on the last pop(login) date correctly. Rather than try to fix cyrus myself, I decided to write my own script which tails the mail log and looks for successful pop/imap/smtp logins and updates our user sql database with the current time and IP the user used to access his or her account. If someone were interested in fixing this feature of cyrus it would be great. -Blake -------- Original Message -------- Subject: Re: howto list inactive users in cyrus: From: Wesley Craig To: Joseph Brennan Cc: info-cyrus at lists.andrew.cmu.edu Date: Friday, December 19, 2008 12:41:18 PM > On 19 Dec 2008, at 12:38, Joseph Brennan wrote: > >> How about checking the timestamp on the user.seen files? It seems to >> get touched at login, even if you don't open a message. >> > > There's a "last read" timestamp in the seen DB, which is probably > more accurate. > > :wes > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090107/75d6bd4d/attachment.html From rjaffey at artic.edu Wed Jan 7 18:12:00 2009 From: rjaffey at artic.edu (Raphael Jaffey) Date: Wed, 7 Jan 2009 17:12:00 -0600 Subject: quota_db recommendations Message-ID: <20090107171200.52wcg6ua8cc0gwk8@webmail-test.artic.edu> I've encountered a bug in the 2.3.13 IMAP server running on RHEL AS 4.6, which appears to be related to the use of the default 'quota_db: quotalegacy' and certain patterns within the user namespace. When there are a large number of entries in a branch of the quota hierarchy which share a common root in the username and a new entry is created which matches the common root exactly, the user matching the common root cannot replicate and the user's inbox cannot be deleted. Example: There are 50 users ranging from aa0 to aa49 all having a quota on their inboxes (mailboxes user.aa0, user.aa1, etc, and quota/a/user.aa0, quota/a/user.aa1,...). Replication is active. Creating a user aa (mailbox user.aa) causes replication to fail repeatedly until the "USER aa" record is removed from the replication log. Deleting the mailbox ("sam user.aa cyrus all", "dm user.aa") causes either a segfault in glibc's _int_malloc(), a "*** glibc detected *** double free or corruption (!prev):...", or "*** glibc detected *** corrupted double-linked list:" error reported by the imap process. There may be other errors as well. While I'm in the process of testing a number of system configurations (RHEL AS 4 and 5, vanilla cyrus, invoca cyrus package) and will submit details to bugzilla, I need to address this problem immediately as this affects a production environment with close to 20K inboxes. So far, I've eliminated the aspect which affects replication by switching the quota_db to something other than 'quotalegacy' on the production replicant. I've also confirmed in a test environment that switching quota_db to something other than 'quotalegacy' eliminates the deletion problem and am considering doing this on the production IMAP server. I've been successful with 'skiplist', but unsuccessful with both 'berkeley' and 'berkeley-hash'. In the latter two cases, while processing close to 40K 'sq user... storage ...' commands the associated IMAP process freezes at random points in the list of 'sq' commands and no further manipulation of quotas are possible via IMAP or the 'quota' binary. There doesn't seem to be a way to recover when using 'berkeley', however, it appears that when using 'berkeley-hash', restarting cyrus (which runs 'ctl_cyrusdb -r') temporarily enables manipulation of quotas by the aforementioned methods. Needless to say, I'm not sure I trust either 'berkeley' or 'berkeley-hash' at this point. Does anyone have experience using "flat" or "skiplist" for quotas? Are either of these suitable for a moderately high volume production environment of the scale I mentioned? From baconm at email.unc.edu Wed Jan 7 18:34:47 2009 From: baconm at email.unc.edu (Michael Bacon) Date: Wed, 07 Jan 2009 18:34:47 -0500 Subject: GSSAPI authentication ceased working In-Reply-To: <495E3E9D.10907@lhanke.de> References: <495E3E9D.10907@lhanke.de> Message-ID: <56F1441DB3402973A4D16D9D@ITSBACONM.depts.unc.edu> Shot in the dark here, but are you using AFS? If so, you can run into some nasty things if it tries to grab libraries out of AFS that you have access to when you have AFS tokens, but which become unavailable when they expire. You start up the process with the tokens, but when you log back in, you obtain tokens for yourself, but not the PAG that the process started in. If you want to know what you're linked against, use ldd on your binaries and on your SASL plugins. If you see paths in AFS, that's likely your problem. -Michael --On Friday, January 02, 2009 5:19 PM +0100 Lars Hanke wrote: > I'm currently setting up a new imap server to replace my old one. > Yesterday I had GSSAPI authentication running, today it ceased working. > I did quite some configuration in the meantime mostly on the LDAP > server, but nothing I'd readily associate with cyrus-imap authentication. > > I appreciate any ideas for more systematic troubleshooting. > > Regards, > - lars. > > The setup: > KDC and LDAP is a sever called hel. The KDC uses LDAP as backend. > Cyrus-Imap (v2.2.13-Debian-2.2.13-14+b3) runs on hermod. > > What worked yesterday: > > kinit cyrus > imtest -v -u cyrus -a cyrus -p imap -r MGR hermod.mgr > cyradm --user cyrus --auth GSSAPI --server hermod.mgr > > What still works today: > kinit cyrus > > Diagnostics: ># kinit cyrus > hermod:~# klist > Ticket cache: FILE:/tmp/krb5cc_0 > Default principal: cyrus at MGR > > Valid starting Expires Service principal > 01/02/09 16:41:41 01/03/09 02:41:41 krbtgt/MGR at MGR > renew until 01/03/09 16:41:41 > > > Kerberos 4 ticket cache: /tmp/tkt0 > klist: You have no tickets cached > hermod:~# imtest -v -u cyrus -a cyrus -p imap -r MGR hermod.mgr > S: * OK hermod.mgr Cyrus IMAP4 v2.2.13-Debian-2.2.13-14+b3 server ready > C: C01 CAPABILITY > S: * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS ID > NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT > THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE IDLE STARTTLS > AUTH=GSSAPI AUTH=NTLM AUTH=DIGEST-MD5 AUTH=CRAM-MD5 SASL-IR S: C01 OK > Completed > Authentication failed. generic failure > Security strength factor: 0 > C: Q01 LOGOUT > * BYE LOGOUT received > Q01 OK Completed > Connection closed. > > hermod: /var/log/auth.log > Jan 2 17:07:54 hermod imtest: GSSAPI Error: Unspecified GSS failure. > Minor code may provide more information (Decrypt integrity check failed) > > hel: /var/log/syslog > Jan 2 16:07:54 hel krb5kdc[1652]: TGS_REQ (7 etypes {18 17 16 23 1 3 2}) > 172.16.6.5: PROCESS_TGS: authtime 0, for > imap/hermod.mgr at MGR, Decrypt integrity check failed Jan 2 16:07:54 hel > last message repeated 3 times > > > What I tried: > > Since "Decrypt integrity check failed" means "wrong password" I recreated > the principal "imap/hermod.mgr" and replaced the keytab file with the new > key. I also removed the ldapdb auxprop, which I had installed in the > meantime, but nothing helped. If I remove the ticket for cyrus, I receive: > Jan 2 17:13:36 hermod imtest: GSSAPI Error: Unspecified GSS failure. > Minor code may provide more information (No credentials cache found) as I > would expect. > > > > > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html From lars at lhanke.de Wed Jan 7 19:43:28 2009 From: lars at lhanke.de (Lars Hanke) Date: Thu, 08 Jan 2009 01:43:28 +0100 Subject: GSSAPI authentication ceased working In-Reply-To: <56F1441DB3402973A4D16D9D@ITSBACONM.depts.unc.edu> References: <495E3E9D.10907@lhanke.de> <56F1441DB3402973A4D16D9D@ITSBACONM.depts.unc.edu> Message-ID: <49654C30.2000208@lhanke.de> Hi Michael, > Shot in the dark here, but are you using AFS? If so, you can run into > some nasty things if it tries to grab libraries out of AFS that you > have access to when you have AFS tokens, but which become unavailable > when they expire. You start up the process with the tokens, but when > you log back in, you obtain tokens for yourself, but not the PAG that > the process started in. There are strange things out there. Thanks for the idea, but I definitely have never used AFS and nothing is installed, which I would associate with AFS. BTW: It's still not working. I put it to PRI2, since the important ldapdb stuff is running. Kerberized imap is rarely used here, so people can do without. But still I'd like to understand, what is happening. Regards, - lars. From dave64 at andrew.cmu.edu Thu Jan 8 07:42:13 2009 From: dave64 at andrew.cmu.edu (Dave McMurtrie) Date: Thu, 08 Jan 2009 07:42:13 -0500 Subject: GSSAPI authentication ceased working In-Reply-To: <49654C30.2000208@lhanke.de> References: <495E3E9D.10907@lhanke.de> <56F1441DB3402973A4D16D9D@ITSBACONM.depts.unc.edu> <49654C30.2000208@lhanke.de> Message-ID: <4965F4A5.9090002@andrew.cmu.edu> Lars Hanke wrote: > BTW: It's still not working. I put it to PRI2, since the important > ldapdb stuff is running. Kerberized imap is rarely used here, so people > can do without. But still I'd like to understand, what is happening. Is the keytab readable by the cyrus user (the Unix uid)? Thanks, Dave -- Dave McMurtrie, SPE Email Systems Team Leader Carnegie Mellon University, Computing Services From blake at ispn.net Thu Jan 8 10:08:40 2009 From: blake at ispn.net (Blake Hudson) Date: Thu, 08 Jan 2009 09:08:40 -0600 Subject: Storage Sizing: IOPS per mailbox In-Reply-To: <1230957617.20171.17.camel@darkstar.netcore.co.in> References: <1230957617.20171.17.camel@darkstar.netcore.co.in> Message-ID: <496616F8.6090606@ispn.net> -------- Original Message -------- Subject: Storage Sizing: IOPS per mailbox From: ram To: info-cyrus Date: Friday, January 02, 2009 10:40:17 PM > When sizing a storage device for a large cyrus server, the typical > question asked by storage vendors is what is the IOPS required per > mailbox > M$$ Exchange has this concept of IOPS. and they suggest 1.5 IOPS per > mailbox ( heavy users ) > > If I use postfix and cyrus , on my imap server ( pure IMAP server .. All > spam filtering , outgoing mails , authentication etc happens on > different servers ) > > > If the storage is used only for imap storage , what is the typical > "IOPS" requirement per user > We will probably assume 30-50 mails a day of average 100k , and an email > client checking for new mail every 5minutes > > > In my experience I would estimate 0.1 IOPS per user for heavy users (thousands of emails per day, checked every few minutes) and 0.01 IOPS per user for typical ISP accounts (under a dozen emails, checked a few times daily). Our systems use MySQL for authentication and account verification and primarily skiplist databases within Cyrus. These figures may be on the safe side as none of our systems do just postfix/imap. --Blake From janne.peltonen at helsinki.fi Thu Jan 8 10:20:00 2009 From: janne.peltonen at helsinki.fi (Janne Peltonen) Date: Thu, 8 Jan 2009 17:20:00 +0200 Subject: choosing a file system In-Reply-To: References: <200901020850.22355.jmadden@ivytech.edu> Message-ID: <20090108152000.GH13471@helsinki.fi> Hm. ReiserFS: If I'm still following after reading through all this discussion, everyone who is actually using ReiserFS (v3) appears to be very content with it, even with very large installations. Apparently the fact that ReiserFS uses the BKL in places doesn't hurt performance too badly, even with multi core systems? Another thing I don't recall being mentioned was fragmentation - ext3 appears to have a problem with it, in typical Cyrus usage, but how does ReiserFS compare to it? Also, the write barrier problem mentioned in response to my earlier post on ext3 would apparently be there with ReiserFS, too, wouldn't it? GFS: Nobody mentioned using GFS, which /is/ a clustered file system and as such, probably overkill if it's only mounted on one node at a time, but I'm curious... the overhead of a clustered FS is the fact that all metadata operations take a long time, because there is a lot of cluster-wide locking. But how much metadata operations there are, after all, in Cyrus? Also, GFS is one of the two file systems available when using RH clustering... Ext3: I'm using this happily, with 50k users, 24 distinct mailspools of 240G each. Full backups take quite a while to complete (~2 days), but normal usage is quite fast. There is the barrier problem, of course... I'm using noatime (implying nodiratime) and data=ordered, since data=writeback resulted in corrupted skiplist files on crash, while data=ordered mostly didn't. Also, ext3 is the other FS available when using RH clustering. (Of course, it isn't a clustered FS, so it is only available when using the cluster in active-passive mode.) XFS: There was someone using this, too, and happy with it. JFS: Mm, apparently no comments on this, not positive, at least. Future: Ext4 just got stable, so there is no real world Cyrus user experience on it. Among other things, it contains an online defragmenter. Journal checksumming might also help around the write barrier problem on LVM logical volumes, if I've understood correctly. Reiser4 might have a future, at least Andrew Morton's -mm patch contains it and there are people developing it. But I don't know if it ever will be included in the "standard" kernel tree. Btrfs is in so early development that I don't know yet what to say about it, but the fact of ZFS's being incompatible with GPL might be mitigated by this. Conclusion: I'm going to continue using ext3 for now, and probably ext4 when it's available from certain commercial enterprise linux vendor (personally, I'd be using Debian, but the department has an official policy of using RH / Centos). I'm eagerly waiting for btrfs to appear... I probably /would/ switch to ReiserFS for now, if RH cluster would support ReiserFS FS resources. Hmm, maybe I should just start hacking... On the other hand, the upgrade path from ext3 to ext4 is quite easy, and I don't know yet which would be better, ReiserFS or ext4. -- Janne Peltonen PGP Key ID: 0x9CFAC88B Please consider membership of the Hospitality Club (http://www.hospitalityclub.org) From ian.batten at uk.fujitsu.com Thu Jan 8 10:53:13 2009 From: ian.batten at uk.fujitsu.com (Ian Batten) Date: Thu, 8 Jan 2009 15:53:13 +0000 Subject: Storage Sizing: IOPS per mailbox In-Reply-To: <496616F8.6090606@ispn.net> References: <1230957617.20171.17.camel@darkstar.netcore.co.in> <496616F8.6090606@ispn.net> Message-ID: <82271048-7EA9-40B5-8D10-AB845F3272CF@uk.fujitsu.com> On 08 Jan 09, at 1508, Blake Hudson wrote: > -------- Original Message -------- > Subject: Storage Sizing: IOPS per mailbox > From: ram > To: info-cyrus > Date: Friday, January 02, 2009 10:40:17 PM >> When sizing a storage device for a large cyrus server, the typical >> question asked by storage vendors is what is the IOPS required per >> mailbox >> M$$ Exchange has this concept of IOPS. and they suggest 1.5 IOPS per >> mailbox ( heavy users ) >> >> If I use postfix and cyrus , on my imap server ( pure IMAP >> server .. All >> spam filtering , outgoing mails , authentication etc happens on >> different servers ) >> >> >> If the storage is used only for imap storage , what is the typical >> "IOPS" requirement per user >> We will probably assume 30-50 mails a day of average 100k , and an >> email >> client checking for new mail every 5minutes >> >> >> > In my experience I would estimate 0.1 IOPS per user for heavy users > (thousands of emails per day, checked every few minutes) and 0.01 IOPS > per user for typical ISP accounts (under a dozen emails, checked a few > times daily). Our IMAP server has as I type 1020 imap connections up, representing most of our staff. The metadata (both /var/imap and per-mailbox) is in a ZFS pool configured as a two-way mirror of two-way stripes of SAS drives. The load on that is low. The zfs statistics on the metadata are as an example: mailhost-new# zpool iostat 1 capacity operations bandwidth pool used avail read write read write ---------- ----- ----- ----- ----- ----- ----- pool1 54.4G 23.6G 12 77 489K 499K pool1 54.4G 23.6G 0 785 0 3.74M pool1 54.4G 23.6G 0 14 0 135K pool1 54.4G 23.6G 0 24 0 877K pool1 54.4G 23.6G 0 33 0 242K pool1 54.4G 23.6G 0 10 0 43.6K pool1 54.4G 23.6G 0 417 1.48K 2.06M pool1 54.4G 23.6G 0 22 0 139K pool1 54.4G 23.6G 0 1 0 7.92K pool1 54.4G 23.6G 0 7 0 31.7K pool1 54.4G 23.6G 0 20 0 83.2K pool1 54.4G 23.6G 0 504 1013 1.93M pool1 54.4G 23.6G 0 23 0 574K pool1 54.4G 23.6G 2 17 96.5K 123K pool1 54.4G 23.6G 0 40 0 285K pool1 54.4G 23.6G 0 26 0 123K pool1 54.4G 23.6G 0 698 1.98K 3.41M pool1 54.4G 23.6G 0 3 0 15.8K pool1 54.4G 23.6G 0 24 0 744K pool1 54.4G 23.6G 0 16 0 713K pool1 54.4G 23.6G 3 15 209K 147K pool1 54.4G 23.6G 5 569 760K 2.71M pool1 54.4G 23.6G 0 16 0 222K ^Cmailhost-new# You can see the five-second sync. The first line, average, figures aren't representative because they of course include backup activity. The actual messages are stored in the lowest (`archive') QoS band of a Pillar Axiom 500, in NAS mode. The load is very small on each of two 2TB-ish partitions (it's approaching four pm, so the business is going at close to full load): mailpool1 I/O Operations Read I/Os per second: 58.677 Write I/Os per second: 7.129 Average Request Time: 4.475 ms Current MB per second: 1.003 General Statistics Read/Write Cache Hit Percentage: 69% Read/Write I/O Ratio: 89:11 mailpool2 I/O Operations Read I/Os per second: 46.733 Write I/Os per second: 9.467 Average Request Time: 1.923 ms Current MB per second: 0.544 General Statistics Read/Write Cache Hit Percentage: 56% Read/Write I/O Ratio: 83:17 From vbfox at ucdavis.edu Thu Jan 8 11:01:04 2009 From: vbfox at ucdavis.edu (Vincent Fox) Date: Thu, 08 Jan 2009 08:01:04 -0800 Subject: choosing a file system In-Reply-To: <20090108152000.GH13471@helsinki.fi> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> Message-ID: <49662340.7070103@ucdavis.edu> (Summary of filesystem discussion) You left out ZFS. Sometimes Linux admins remind me of Windows admins. I have adminned a half-dozen UNIX variants professionally but keep running into admins who only do ONE and for whom every problem is solved with "how can I do this with one OS only?" I admin numerous Linux systems in our data center (Perdition proxy in front of Cyrus for one) but frankly you want me to go back into filesystem Dark Ages now for terabytes of mail volume I'd throw a professional fit. Even the idea that I need to tune my filesystem for inodes and to avoid it wanting to fsck on reboot #20 or whatever seems like caveman discussion. Any of them offer cheap and nearly-instant snapshots & online scrubbing? No? Then why use it for large number of files of important nature? I love Linux, I surely do. Virtually everything of an appliance nature here will probably shift over to it in the long run I think and for good reasons. But filesystem is one area where the bazaar model has fallen into a very deep rut and can't muster energy to climb out. So far ZFS ticking along with no problems and low iostat numbers with everything in one big pool. I have separate fs for data, imap, mail but haven't seen any need to carve mail spool into chunks at all. There were initial problems noted here in the mailing lists way back in Solaris 10u3 but that was solved with the fsync patch and since then it's been like butter. Mail-store systems nobody ever needs to look at them because it "just works". From wes at umich.edu Thu Jan 8 14:21:30 2009 From: wes at umich.edu (Wesley Craig) Date: Thu, 8 Jan 2009 14:21:30 -0500 Subject: GSSAPI authentication ceased working In-Reply-To: <495E3E9D.10907@lhanke.de> References: <495E3E9D.10907@lhanke.de> Message-ID: <40675BDB-8862-4727-B7B2-CBAC110D3389@umich.edu> On 02 Jan 2009, at 11:19, Lars Hanke wrote: > hermod: /var/log/auth.log > Jan 2 17:07:54 hermod imtest: GSSAPI Error: Unspecified GSS > failure. Minor code may provide more information (Decrypt > integrity check failed) > > hel: /var/log/syslog > Jan 2 16:07:54 hel krb5kdc[1652]: TGS_REQ (7 etypes {18 17 16 23 1 > 3 2}) 172.16.6.5: PROCESS_TGS: authtime 0, for > imap/hermod.mgr at MGR, Decrypt integrity check failed As I read this, hel is saying that the TGT is bad. You're trying to obtain a service ticket for imap/hermod, but the TGT you're attempting to use is not accepted by the KDC. If you klist after running imtest, you have no imap/hermod ticket. I've never seen an error like that. It suggests that you KDC is really broken :) Something like the key used to encrypt your TGT isn't valid for obtaining service tickets. :wes From brong at fastmail.fm Thu Jan 8 19:23:23 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 9 Jan 2009 11:23:23 +1100 Subject: choosing a file system In-Reply-To: <20090108152000.GH13471@helsinki.fi> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> Message-ID: <20090109002323.GA17001@brong.net> On Thu, Jan 08, 2009 at 05:20:00PM +0200, Janne Peltonen wrote: > If I'm still following after reading through all this discussion, > everyone who is actually using ReiserFS (v3) appears to be very content > with it, even with very large installations. Apparently the fact that > ReiserFS uses the BKL in places doesn't hurt performance too badly, even > with multi core systems? Another thing I don't recall being mentioned > was fragmentation - ext3 appears to have a problem with it, in typical > Cyrus usage, but how does ReiserFS compare to it? Yeah, I'm surprised the BKL hasn't hurt us more. Fragmentation, yeah it does hurt performance a bit. We run a patch which causes a skiplist checkpoint every time it runs a "recovery", which includes every restart. We also tune skiplists to checkpoint more frequently in everyday use. This helps reduce meta fragmentation. For data fragmentation - we don't care. Honestly. Data IO is so rare. The main time it matters is if someone does a body search. Which leaves... index files. The worst case are files that are only ever appended to, never any records deleted. Each time you expunge a mailbox (even with delayed expunge) it causes a complete rewrite of the cyrus.index file. I also wrote a filthy little script (attached) which can repack cyrus meta directories. I'm not 100% certain that it's problem free though, so I only run it on replicas. Besides, it's not "protected" like most of our auto-system functions, which check the database to see if the machine is reporting high load problems and choke themselves until the load drops back down again. > I'm using this happily, with 50k users, 24 distinct mailspools of 240G > each. Full backups take quite a while to complete (~2 days), but normal > usage is quite fast. There is the barrier problem, of course... I'm > using noatime (implying nodiratime) and data=ordered, since > data=writeback resulted in corrupted skiplist files on crash, while > data=ordered mostly didn't. Yeah, full backups. Ouch. I think the last time we had to do that it took somewhat over a week. Mainly CPU limited on the backup server, which is doing a LOT of gzipping! Our incremental backups take about 4 hours. We could probably speed this up a little more, but given that it's now down from about 12 hours two weeks ago, I'm happy. We were actually rate limited by Perl 'unpack' and hash creation, believe it or not! I wound up rewriting Cyrus::IndexFile to provide a raw interface, and unpacking just the fields that I needed. I also asserted index file version == 10 in the backup library so I can guarantee the offsets are correct. I've described our backup system here before - it's _VERY_ custom, based on a deep understanding of the Cyrus file structures. In this case it's definitely worth it - it allows us to reconstruct partial mailbox recoveries with flags intact. Unfortunately, "seen" information is much trickier. I've been tempted for a while to patch cyrus's seen support to store seen information for the user themselves in the cyrus.index file, and only seen information for unowned folders in the user.seen files. The way it works now seems optimised for the uncommon case at the expense of the common. That always annoys me! > Ext4 just got stable, so there is no real world Cyrus user experience on > it. Among other things, it contains an online defragmenter. Journal > checksumming might also help around the write barrier problem on LVM > logical volumes, if I've understood correctly. Yeah, it's interesting. Local fiddling suggests it's worse for my Maildir performance than even btrfs, and btrfs feels more jerky than reiser3, so I stick with reiser3. > Reiser4 might have a future, at least Andrew Morton's -mm patch contains > it and there are people developing it. But I don't know if it ever will > be included in the "standard" kernel tree. Yeah, the mailing list isn't massively active at the moment either... I do keep an eye on it. > Btrfs is in so early development that I don't know yet what to say about > it, but the fact of ZFS's being incompatible with GPL might be mitigated > by this. Yeah, btrfs looks interesting. Especially with their work on improving locking - even on my little dual processor laptop (yay core processors) I would expect to see an improvement when they merge the new locking code. > I'm going to continue using ext3 for now, and probably ext4 when it's > available from certain commercial enterprise linux vendor (personally, > I'd be using Debian, but the department has an official policy of using > RH / Centos). I'm eagerly waiting for btrfs to appear... I probably /would/ > switch to ReiserFS for now, if RH cluster would support ReiserFS FS > resources. Hmm, maybe I should just start hacking... On the other hand, > the upgrade path from ext3 to ext4 is quite easy, and I don't know yet > which would be better, ReiserFS or ext4. Sounds sane. If vendor support matters, then ext4 is probably the immediate future good choice. It's had a fair bit of work. I'm tempted to keep an eye on tux3 too. Exciting times in the linux filesystem world at the moment. Bron. From brong at fastmail.fm Thu Jan 8 19:46:20 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 9 Jan 2009 11:46:20 +1100 Subject: choosing a file system In-Reply-To: <49662340.7070103@ucdavis.edu> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> Message-ID: <20090109004620.GB17001@brong.net> On Thu, Jan 08, 2009 at 08:01:04AM -0800, Vincent Fox wrote: > (Summary of filesystem discussion) > > You left out ZFS. > > Sometimes Linux admins remind me of Windows admins. > > I have adminned a half-dozen UNIX variants professionally but > keep running into admins who only do ONE and for whom every > problem is solved with "how can I do this with one OS only?" We run one zfs machine. I've seen it report issues on a scrub only to not have them on the second scrub. While it looks shiny and great, it's also relatively new. Besides, we had a disk _fail_ early on in our x4500 - Sun shipped a replacement drive, but the kernel was unable to recognise it: --- "Nothing odd about how it snaps in. We can see the connectors in the slot - they seem fine as far as we can tell. The drive's 'ok' light is on and the blue led lit." Which suggests the server thinks the drive is fine, but the dmesg data definitely suggests it isn't. I've also included the output of hdadm display below as well, which shows that currently it thinks the drive is not present, even though the last thing reported in the dmesg log is that the device was connected. Aug 14 21:59:13 backup1 SATA device attached at port 0 Aug 14 21:59:13 backup1 sata: [ID 663010 kern.info] +/pci at 2,0/pci1022,7458 at 8/pci11ab,11ab at 1 : The output of hdadm display shows that the machine definitely thinks the drive is NOT connected. --- Sun's response was to wait for the next kernel upgrade - there was a bug that made that channel unusable even after a reboot. > So far ZFS ticking along with no problems and low iostat numbers > with everything in one big pool. I have separate fs for data, imap, mail > but haven't seen any need to carve mail spool into chunks at all. > There were initial problems noted here in the mailing lists way back > in Solaris 10u3 but that was solved with the fsync patch and since then > it's been like butter. Mail-store systems nobody ever needs to look > at them because it "just works". I'd sure hate to lose the entire basket, say due to an unknown bug in zfs. Besides, I _know_ Debian quite well. We don't have any Solaris experience in our team. The documentation looks quite good, but it's still a lot of things that work differently. I tell you what, maintaining Solaris and using the Solaris userland feels like going back 20 years - and the whole "need a sunsolve password and only get some patches - permission denied on others" crap. I don't need that. So while I apprciate that ZFS has some advantages, I'd have to say that they need to be weighed up against the rest of the system, and the "all the eggs in a relatively new basket" argument. Also, the response we've had from Linus when we find kernel issues has been absolutely fantastic. Bron ( Debian on the Solaris kernel would be interesting... ) From daleg at elemental.org Thu Jan 8 20:03:46 2009 From: daleg at elemental.org (Dale Ghent) Date: Thu, 8 Jan 2009 20:03:46 -0500 Subject: choosing a file system In-Reply-To: <20090109004620.GB17001@brong.net> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> <20090109004620.GB17001@brong.net> Message-ID: <7E97FA07-7CB8-4333-B731-F5837EBC88D8@elemental.org> On Jan 8, 2009, at 7:46 PM, Bron Gondwana wrote: > We run one zfs machine. I've seen it report issues on a scrub > only to not have them on the second scrub. While it looks shiny > and great, it's also relatively new. Wait, weren't you just crowing about ext4? The filesystem that was marked GA in the linux kernel release that happened just a few weeks ago? You also sound pretty enthusiastic, rather than cautious, when talking about brtfs and tux3. ZFS, and anyone who even remotely seriously follows Solaris would know this, has been GA for 3 years now. For someone who doesn't have their nose buried in Solaris much or with any serious attention span, I guess it could still seem new. As for your x4500, I can't tell if those syslog lines you pasted were from Aug. 2008 or 2007, but certainly since 2007 the marvel SATA driver has seen some huge improvements to work around some pretty nasty bugs in the marvell chipset. If you still have that x4500, and have not applied the current patch for the marvell88sx driver, I highly suggest doing so. Problems with that chip are some of the reasons Sun switched to the LSI 1068E as the controller in the x4540. /dale From brong at fastmail.fm Thu Jan 8 22:32:04 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 09 Jan 2009 14:32:04 +1100 Subject: choosing a file system In-Reply-To: <7E97FA07-7CB8-4333-B731-F5837EBC88D8@elemental.org> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> <20090109004620.GB17001@brong.net> <7E97FA07-7CB8-4333-B731-F5837EBC88D8@elemental.org> Message-ID: <1231471924.9875.1293804941@webmail.messagingengine.com> On Thu, 08 Jan 2009 20:03 -0500, "Dale Ghent" wrote: > On Jan 8, 2009, at 7:46 PM, Bron Gondwana wrote: > > > We run one zfs machine. I've seen it report issues on a scrub > > only to not have them on the second scrub. While it looks shiny > > and great, it's also relatively new. > > Wait, weren't you just crowing about ext4? The filesystem that was > marked GA in the linux kernel release that happened just a few weeks > ago? You also sound pretty enthusiastic, rather than cautious, when > talking about brtfs and tux3. I was saying I find it interesting. I wouldn't seriously consider using it for production mail stores just yet. But I have been testing it on my laptop, where I'm running an offlineimap replicated copy of my mail. I wouldn't consider btrfs for production yet either, and tux3 isn't even on the radar. They're interesting to watch though, as is ZFS. I also said (or at least meant) that if you have commercial support, ext4 is probably going to be the next evolutionary step from ext3. > ZFS, and anyone who even remotely seriously follows Solaris would know > this, has been GA for 3 years now. For someone who doesn't have their > nose buried in Solaris much or with any serious attention span, I > guess it could still seem new. Yeah, it's true - but I've heard anecdotes of people losing entire zpools due to bugs. Google turns up things like: http://www.techcrunch.com/2008/01/15/joyent-suffers-major-downtime-due-to-zfs-bug/ which points to this thread: http://www.opensolaris.org/jive/thread.jspa?threadID=49020&tstart=0 and finally this comment: http://www.joyeur.com/2008/01/16/strongspace-and-bingodisk-update#c008480 Not something I would want happening to my entire universe, which is why having ~280 separate filesystems (at the moment) with our email spread across them means that a rare filesystem bug is only likely to affect a single store if it bites - and we can restore one store's worth of users a lot quicker than the whole system. It's the same reason we prefer Cyrus replication (and put a LOT of work into making it stable - check this mailing list from a couple of years ago. I wrote most of the patches the stabilised replication between 2.3.3 and 2.3.8) If all your files are on a single filesystem then a rare bug only has to hit once. A frequent bug on the other hand, well - you'll know about them pretty fast... :) None of the filesystems mentioned have frequent bugs (except btrfs and probably tux3 - but they ship with big fat warnings all over) > As for your x4500, I can't tell if those syslog lines you pasted were > from Aug. 2008 or 2007, but certainly since 2007 the marvel SATA > driver has seen some huge improvements to work around some pretty > nasty bugs in the marvell chipset. If you still have that x4500, and > have not applied the current patch for the marvell88sx driver, I > highly suggest doing so. Problems with that chip are some of the > reasons Sun switched to the LSI 1068E as the controller in the x4540. I think it was 2007 actually. We haven't had any trouble with it for a while, but then it does pretty little. The big zpool is just used for backups, which are pretty much one .tar.gz and one .sqlite3 file per user - and the .sqlite3 file is just indexing the .tar.gz file, we can rebuild it by reading the tar file if needed. As a counterpoint to some of the above, we had an issue with Linux where there was a bug in 64 bit writev handling of mmaped space. If you were doing a writev with a mmaped space that crossed a page boundary and the following page wasn't mapped in, it would inject spurious zero bytes in the output where the start of the next page belonged. It took me a few days to prove it was the kernel and create a repeatable test case, and then backwards and forwards with Linus and a couple of other developers we fixed it and tested it _that_day_. I don't know anyone with even unobtanium level support with a commercial vendor who has actually had that sort of turnaround. This caused pretty massive file corruption of especially our skiplist files, but bits of every other meta file too. Luckily, as per above, we had only upgraded one machine. We generally do that with new kernels or software versions - upgrade one production machine and watch it for a bit. We also test things on testbed machines first, but you always find something different on production. The mmap over boundaries case was pretty rare - only a few per day would actually cause a crash, the others were silent corruption that wasn't detected at the time. If something like this hit an only machine, we would have been seriously screwed. Since it only hit one machine, we could apply the fix and re-replicate all the damaged data from the other machine. No actual dataloss. Bron. -- Bron Gondwana brong at fastmail.fm From rob at nofocus.org Thu Jan 8 23:57:18 2009 From: rob at nofocus.org (Robert Banz) Date: Thu, 8 Jan 2009 20:57:18 -0800 Subject: choosing a file system In-Reply-To: <20090109004620.GB17001@brong.net> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> <20090109004620.GB17001@brong.net> Message-ID: On Jan 8, 2009, at 4:46 PM, Bron Gondwana wrote: > On Thu, Jan 08, 2009 at 08:01:04AM -0800, Vincent Fox wrote: >> (Summary of filesystem discussion) >> >> You left out ZFS. >> >> Sometimes Linux admins remind me of Windows admins. >> >> I have adminned a half-dozen UNIX variants professionally but >> keep running into admins who only do ONE and for whom every >> problem is solved with "how can I do this with one OS only?" > > We run one zfs machine. I've seen it report issues on a scrub > only to not have them on the second scrub. While it looks shiny > and great, it's also relatively new. You'd be surprised how unreliable disks and the transport between the disk and host can be. This isn't a ZFS problem, but a statistical certainty as we're pushing a large amount of bits down the wire. You can, with a large enough corpus, have on-disk data corruption, or data corruption that appeared en-flight to the disk, or in the controller, that your standard disk CRCs can't correct for. As we keep pushing the limits, data integrity checking at the filesystem layer -- before the information is presented for your application to consume -- has basically become a requirement. BTW, the reason that the first scrub saw the error, and the second scrub didn't, is that the first scrub fixed it -- that's the job of a ZFS scrub. -rob From brong at fastmail.fm Fri Jan 9 00:08:00 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 9 Jan 2009 16:08:00 +1100 Subject: choosing a file system In-Reply-To: References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> <20090109004620.GB17001@brong.net> Message-ID: <20090109050800.GA13154@brong.net> On Thu, Jan 08, 2009 at 08:57:18PM -0800, Robert Banz wrote: > > On Jan 8, 2009, at 4:46 PM, Bron Gondwana wrote: > >> On Thu, Jan 08, 2009 at 08:01:04AM -0800, Vincent Fox wrote: >>> (Summary of filesystem discussion) >>> >>> You left out ZFS. >>> >>> Sometimes Linux admins remind me of Windows admins. >>> >>> I have adminned a half-dozen UNIX variants professionally but >>> keep running into admins who only do ONE and for whom every >>> problem is solved with "how can I do this with one OS only?" There's a significant upfront cost to learning a whole new system for one killer feature, especially if it comes along with signifiant regressions in lots of other features (like a non-sucky userland out of the box). Applying patches on Solaris seems to be a choice between incredibly low-level command line tools or boot up a whole graphical environment on a machine in a datacentre on the other side of the world. >> We run one zfs machine. I've seen it report issues on a scrub >> only to not have them on the second scrub. While it looks shiny >> and great, it's also relatively new. > > You'd be surprised how unreliable disks and the transport between the > disk and host can be. This isn't a ZFS problem, but a statistical > certainty as we're pushing a large amount of bits down the wire. > > You can, with a large enough corpus, have on-disk data corruption, or > data corruption that appeared en-flight to the disk, or in the > controller, that your standard disk CRCs can't correct for. As we keep > pushing the limits, data integrity checking at the filesystem layer -- > before the information is presented for your application to consume -- > has basically become a requirement. > > BTW, the reason that the first scrub saw the error, and the second scrub > didn't, is that the first scrub fixed it -- that's the job of a ZFS # zpool status -v rpool pool: rpool state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: scrub in progress for 0h0m, 0.69% done, 1h40m to go config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 mirror ONLINE 0 0 0 c5t0d0s0 ONLINE 0 0 0 c5t4d0s0 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: //dev/dsk ------- if that's an "error that the scrub fixed" then it's a really badly written error message. Same error didn't exist next scrub, which was what confused me. Bron. From brong at fastmail.fm Fri Jan 9 00:19:53 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 9 Jan 2009 16:19:53 +1100 Subject: choosing a file system In-Reply-To: <49662340.7070103@ucdavis.edu> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> Message-ID: <20090109051953.GA13886@brong.net> On Thu, Jan 08, 2009 at 08:01:04AM -0800, Vincent Fox wrote: > (Summary of filesystem discussion) > > You left out ZFS. Just to come back to this - I should say that I'm a big fan of ZFS and what Sun have done with filesystem design. Despite the issues we've had with that machine, I know it's great for people who are using it... BUT - if someone is asking "what's the best filesystem to use on Linux" and gets told ZFS, and by the way you should switch operating systems and ditch all the rest of your custom setup/ experience then you're as bad as a Linux weenie saying "just use Cyrus on Linux" in a "how should I tune NTFS on my Exchange server" discussion. >From the original post: Message-ID: <1617f8010812300849k1c7c878bl2f17e8d4287c10be at mail.gmail.com> "zfs (but we should switch to solaris or freebsd and throw away our costly SAN)" I'd love to do some load testing on a ZFS box with our setup at some point. There would be some advantages, though I suspect having one big mailboxes.db vs the lots of little ones we have would be a point of contention - and fine-grained skiplist locking is still very much a wishlist item. I'd want to take some time testing it before unleashing it on the world! Bron. From vbfox at ucdavis.edu Fri Jan 9 01:00:04 2009 From: vbfox at ucdavis.edu (Vincent Fox) Date: Thu, 08 Jan 2009 22:00:04 -0800 Subject: choosing a file system In-Reply-To: <20090109051953.GA13886@brong.net> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> <20090109051953.GA13886@brong.net> Message-ID: <4966E7E4.6050607@ucdavis.edu> Bron Gondwana wrote: > BUT - if someone is asking "what's the best filesystem to use > on Linux" and gets told ZFS, and by the way you should switch > operating systems and ditch all the rest of your custom setup/ > experience then you're as bad as a Linux weenie saying "just > use Cyrus on Linux" in a "how should I tune NTFS on my > Exchange server" discussion. > > Point taken. We can go around that circle all day long but I *am* saying there are other UNIX OS out there than just Linux and quite frankly it blows my mind sometimes how people fall into ruts. Numerous times in my career I have had to switch some application from AIX to HP-UX, or IRIX to Linux. The differing flavors of UNIX are not so different to me as others perhaps. Particularly when it's a single app on a dedicated server I usually find it odd how people get stuck on something and won't change. Or they take the safe institutional path and never fight it. Collect your paycheck and go home at 4. I sleep very well at night knowing the Cyrus mail-stores are on ZFS. Once in a while I run a scrub just for fun. No futzing around. This was no cakewalk. I was pushing a boulder up a hill particularly when we ran head-first into the ZFS fsync bottleneck start of Fall quarter. Managers said we needed a crash program to convert everything to Linux or Exchange or whatever. I dug into the bugs instead and Sun got us an interim patch to fix it and we moved on. Now as I said it's like butter and one of those setups nobody thinks about. There are always excuses why you will stick with "established" practice even if it's antiquated and full of aches and pains, and I fought that and won. It seems to me there is no bigger deal than having a RELIABLE filesystem for mail-store and this is where all other filesystem I have worked with since 1989 have been a frigging nightmare. Everything from bad controllers to double-disk failures in RAID-5 sets keeps me wondering am I paranoid ENOUGH. I'll be all over btrfs when it hits beta. I'm not married to ZFS. But I'm quite unashamedly looking down my nose at any filesystem now that leaves me possibly looking at fsck prompt. I've done enough of that in my career already it's time to move beyond 30+ years worth of cruft atop antique designs that seemed tolerable when a huge disk was 20 gigs. From rob at nofocus.org Fri Jan 9 01:13:25 2009 From: rob at nofocus.org (Robert Banz) Date: Thu, 8 Jan 2009 22:13:25 -0800 Subject: choosing a file system In-Reply-To: <20090109050800.GA13154@brong.net> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> <20090109004620.GB17001@brong.net> <20090109050800.GA13154@brong.net> Message-ID: <2EE6AC0B-BA73-4AA2-BBB9-BE13113E38F0@nofocus.org> > > There's a significant upfront cost to learning a whole new system > for one killer feature, especially if it comes along with signifiant > regressions in lots of other features (like a non-sucky userland > out of the box). ... The "non-sucky" userland comment is simply a matter of preference, and bait for a religious war, which I'm not going to bite. What I will say is that switching between Solaris, Linux, IRIX, Ultrix, FreeBSD, HP-UX, OSF/1 -- any *nix variant, should not be considered a stumbling block. Your comment shows the narrow-mindedness of the current Linux culture, many of us were brought up supporting and using a collection of these platforms at any one time. (notice, didn't mention AIX. I've got my standards ;) Patching is always an issue on any OS, and you do have the choice of running X applications remotely (booting an entire graphic environment!?), and many other tools available such as pca to help you patch on Solaris, which provide many of the features that you're used to. -rob From ram at netcore.co.in Fri Jan 9 01:47:33 2009 From: ram at netcore.co.in (ram) Date: Fri, 09 Jan 2009 12:17:33 +0530 Subject: 32-bit to 64-bit migration seen flags Message-ID: <1231483653.9135.34.camel@darkstar.netcore.co.in> I am migrating mailboxes from a 32 bit cyrus (cyrus-2.3.7) to a 64 bit cyrus (2.3.13) server When I copy the mailbox seen flags(skiplist) from the 32 bit server to the 64 bit servers it does not work. All the mails are flagged as unseen on the new server Is there a way I can migrate the seen flags Thanks Ram From janne.peltonen at helsinki.fi Fri Jan 9 03:54:10 2009 From: janne.peltonen at helsinki.fi (Janne Peltonen) Date: Fri, 9 Jan 2009 10:54:10 +0200 Subject: choosing a file system In-Reply-To: <49662340.7070103@ucdavis.edu> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> Message-ID: <20090109085409.GF5876@helsinki.fi> On Thu, Jan 08, 2009 at 08:01:04AM -0800, Vincent Fox wrote: > (Summary of filesystem discussion) > > You left out ZFS. > > Sometimes Linux admins remind me of Windows admins. I didn't. --clip-- Btrfs is in so early development that I don't know yet what to say about it, but the fact of ZFS's being incompatible with GPL might be mitigated by this. --clip-- > I have adminned a half-dozen UNIX variants professionally but > keep running into admins who only do ONE and for whom every > problem is solved with "how can I do this with one OS only?" So have I. But in the current Cyrus installation, I'm stuck with Linux, so I concentrated on what's available on Linux. Moreover, I don't want to use non-free operating systems - if anything, I've become more ideological with age... I'd happily use /any/ free unix variant that ran ZFS, but. > Dark Ages now for terabytes of mail volume I'd throw a professional fit. > Even the idea that I need to tune my filesystem for inodes and to avoid it > wanting to fsck on reboot #20 or whatever seems like caveman discussion. > Any of them offer cheap and nearly-instant snapshots & online scrubbing? > No? Then why use it for large number of files of important nature? Because there isn't a free FS that does those things (yet). And there are free systems that do enough... > I love Linux, I surely do. Virtually everything of an appliance nature here > will probably shift over to it in the long run I think and for good reasons. > But filesystem is one area where the bazaar model has fallen into a very > deep rut and can't muster energy to climb out. Really? Btrfs /does/ appear promising to me. I might be wrong, though. > So far ZFS ticking along with no problems and low iostat numbers > with everything in one big pool. I have separate fs for data, imap, mail > but haven't seen any need to carve mail spool into chunks at all. > There were initial problems noted here in the mailing lists way back > in Solaris 10u3 but that was solved with the fsync patch and since then > it's been like butter. Mail-store systems nobody ever needs to look > at them because it "just works". Well, that's nice. It's a shame they made it GPL-incompatible. BR -- Janne Peltonen PGP Key ID: 0x9CFAC88B Please consider membership of the Hospitality Club (http://www.hospitalityclub.org) From brong at fastmail.fm Fri Jan 9 01:59:59 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 9 Jan 2009 17:59:59 +1100 Subject: choosing a file system In-Reply-To: <2EE6AC0B-BA73-4AA2-BBB9-BE13113E38F0@nofocus.org> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> <20090109004620.GB17001@brong.net> <20090109050800.GA13154@brong.net> <2EE6AC0B-BA73-4AA2-BBB9-BE13113E38F0@nofocus.org> Message-ID: <20090109065959.GA6798@brong.net> On Thu, Jan 08, 2009 at 10:13:25PM -0800, Robert Banz wrote: >> >> There's a significant upfront cost to learning a whole new system >> for one killer feature, especially if it comes along with signifiant >> regressions in lots of other features (like a non-sucky userland >> out of the box). > > The "non-sucky" userland comment is simply a matter of preference, and > bait for a religious war, which I'm not going to bite. Well, yeah. Point. Though most Solaris admins I know tend to pull in gnu or bsd utilities pretty quickly. I'll take that one back, it was baity. > What I will say is that switching between Solaris, Linux, IRIX, Ultrix, > FreeBSD, HP-UX, OSF/1 -- any *nix variant, should not be considered a > stumbling block. Your comment shows the narrow-mindedness of the current > Linux culture, many of us were brought up supporting and using a > collection of these platforms at any one time. There's a switching cost, particularly if you don't have any experience with a new system. You have to consider that cost when making an upgrade choice. I agree that ZFS is better than anything currently available on Linux - but the question is "does that outweight the disadvantages of learning and supporting a new platform?". There are basically two worthwhile things on Solaris: ZFS and DTrace. Other things - fork behaviour caused us pain recently, it's just not as cheap as on Linux, and forking from a big process caused lots of swapping because even though it was execing pretty quickly, it had to commit the memory first. Oops. There are downsides to Linux's overcommit, but having to add complexity to our backup manager because forking for every backup was too expensive was annoying. (I do take offence to being considered narrow minded for not blindly following the latest fashion and wanting to switch everything over to Solaris because it has the latest bling - I've considered it, but the numbers just don't add up. We have something that works, is reliable and is fast. Our redundancy is just at a different level) Hey - back on topic for cyrus. We store sha1s of message files in the index file now. We don't have checksums on index files (yet, I have crc32 patches half finished somewhere), but we're at a point where userland scrubs are possible. Along with replication, you can restore any damaged file from the replica. Actually, with out backup system, you can even pull the original file from the backup, knowing its sha1 because it gets recalculated again and checked during the backup phase. > (notice, didn't mention AIX. I've got my standards ;) Hey - I have a friend who _likes_ AIX. There are odd people in the world. > Patching is always an issue on any OS, and you do have the choice of > running X applications remotely (booting an entire graphic > environment!?), and many other tools available such as pca to help you > patch on Solaris, which provide many of the features that you're used > to. I take it you haven't run X applications remotely from the other side of the world before? I'd hardly call it "running". Crawling maybe. My current approach is to run up an vncserver on a box in the same colo and run X applications remotely to there. It's significantly less painful, and also gives me a place to run an iceweasel to talk to the web interfaces of things things that won't talk to me any other way. Uploading firmware via the web from locally is similarly less sucky than pushing it out from Australia. And I'm seeing there are quite a few third party tools that people have written to ease the pain of patch management on Solaris (I believe it's actually one of the nicer unixes to manage patches on, but when you're used to apt-get, there's a whole world of WTFery in manually downloading and applying patch sets - especially when you get permission denied on a bunch of things that the tool has just suggested as being missing) In short - I'm not sold on the value to FastMail of at least two of us (bus factor) learning to maintain Solaris to the level that we'd want for running something so core to our operations as the IMAP servers. Bron ( happy to either stop the flamewar or take it off list at this point. I don't think we're contributing anything meaningful any more ) From archi.laurent at gmail.com Fri Jan 9 07:21:26 2009 From: archi.laurent at gmail.com (Laurent Archambault) Date: Fri, 9 Jan 2009 13:21:26 +0100 Subject: Delete an user + virtual domains Message-ID: <692f4d2f0901090421k354f2c20o8664e510ec2b9708@mail.gmail.com> Hello, With this values in imap.conf : "virtdomains: yes" + "unixhierarchysep: yes" (do after for it). - In this directory : "/var/spool/cyrus/mail/domain/d/domaine2.com" - I have for this an user this structural directory ... (for me it's strange but i don't known all) *./Sent/l/user/laurent^amt ./Sent/l/user/laurent^amt/cyrus.cache ./Sent/l/user/laurent^amt/cyrus.index ./Sent/l/user/laurent^amt/cyrus.squat ./Sent/l/user/laurent^amt/cyrus.header ./l/user/laurent^amt ./l/user/laurent^amt/cyrus.cache ./l/user/laurent^amt/cyrus.index ./l/user/laurent^amt/cyrus.squat ./l/user/laurent^amt/cyrus.header ./Drafts/l/user/laurent^amt ./Drafts/l/user/laurent^amt/cyrus.cache ./Drafts/l/user/laurent^amt/cyrus.index ./Drafts/l/user/laurent^amt/cyrus.squat ./Drafts/l/user/laurent^amt/cyrus.header ./Trash/l/user/laurent^amt ./Trash/l/user/laurent^amt/cyrus.cache ./Trash/l/user/laurent^amt/cyrus.index ./Trash/l/user/laurent^amt/cyrus.squat ./Trash/l/user/laurent^amt/cyrus.header ./Junk/l/user/laurent^amt ./Junk/l/user/laurent^amt/cyrus.cache ./Junk/l/user/laurent^amt/cyrus.index ./Junk/l/user/laurent^amt/cyrus.squat ./Junk/l/user/laurent^amt/cyrus.header * With this structure, Cyrus run perfectly... but when i want to delete an mailbox (or account), its not clean for me ... *./cyrus-admin.pl -u cyrus -p secret -m laurent.amt at domaine2.com --delete Mailbox:define acl lrswipkxtecda for user/laurent.amt at domaine2.com by cyrus Deleted Mailbox: user/laurent.amt at domaine2.com * And now i read my structural Cyrus and i can see this : *just lhis line "./l/user/laurent^amt*" is not present or deleted ... i can see almost all ... - And it's similar with "cyradm ...." ./Sent/l/user/laurent^amt ./Sent/l/user/laurent^amt/cyrus.cache ./Sent/l/user/laurent^amt/cyrus.index ./Sent/l/user/laurent^amt/cyrus.squat ./Sent/l/user/laurent^amt/cyrus.header ./Drafts/l/user/laurent^amt ./Drafts/l/user/laurent^amt/cyrus.cache ./Drafts/l/user/laurent^amt/cyrus.index ./Drafts/l/user/laurent^amt/cyrus.squat ./Drafts/l/user/laurent^amt/cyrus.header ./Trash/l/user/laurent^amt ./Trash/l/user/laurent^amt/cyrus.cache ./Trash/l/user/laurent^amt/cyrus.index ./Trash/l/user/laurent^amt/cyrus.squat ./Trash/l/user/laurent^amt/cyrus.header ./Junk/l/user/laurent^amt ./Junk/l/user/laurent^amt/cyrus.cache ./Junk/l/user/laurent^amt/cyrus.index ./Junk/l/user/laurent^amt/cyrus.squat ./Junk/l/user/laurent^amt/cyrus.header Thanks a lot ... where is my error .. -- ----~o00o-----//{ ??`(_)??` }\\-----o00o~------ Laurent Archambault Under Linux -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090109/834aa096/attachment.html From nic at onlight.com Fri Jan 9 07:35:17 2009 From: nic at onlight.com (Nic Bernstein) Date: Fri, 09 Jan 2009 06:35:17 -0600 Subject: choosing a file system In-Reply-To: <20090109065959.GA6798@brong.net> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> <20090109004620.GB17001@brong.net> <20090109050800.GA13154@brong.net> <2EE6AC0B-BA73-4AA2-BBB9-BE13113E38F0@nofocus.org> <20090109065959.GA6798@brong.net> Message-ID: <49674485.2050204@onlight.com> On 01/09/2009 12:59 AM, Bron Gondwana wrote: > On Thu, Jan 08, 2009 at 10:13:25PM -0800, Robert Banz wrote: > >>> There's a significant upfront cost to learning a whole new system >>> for one killer feature, especially if it comes along with signifiant >>> regressions in lots of other features (like a non-sucky userland >>> out of the box). >>> >> The "non-sucky" userland comment is simply a matter of preference, and >> bait for a religious war, which I'm not going to bite. >> > > Well, yeah. Point. Though most Solaris admins I know tend to pull in > gnu or bsd utilities pretty quickly. I'll take that one back, it was > baity. > So at the risk of entering into a flame war, I must say I am surprised that no one has mentioned Nexenta/OS. http://www.nexenta.org/os They have bolted the Ubuntu/Debian userland onto OpenSolaris to give the Linux lovers out there a linuxy experience with access to all of that shiny new Solaris bling, such as zfs and dtrace. You may want to give it a look-see. >> Patching is always an issue on any OS, and you do have the choice of >> running X applications remotely (booting an entire graphic >> environment!?), and many other tools available such as pca to help you >> patch on Solaris, which provide many of the features that you're used >> to. >> > > > And I'm seeing there are quite a few third party tools that people have > written to ease the pain of patch management on Solaris (I believe it's > actually one of the nicer unixes to manage patches on, but when you're > used to apt-get, there's a whole world of WTFery in manually downloading > and applying patch sets - especially when you get permission denied on > a bunch of things that the tool has just suggested as being missing) > Oh yeah, apt-get included. Cheers, -nic PS - This has been a very interesting thread to read. Some of us just don't have the exposure to large systems like the participants in this thread have, and this can be very educational. -- Nic Bernstein nic at onlight.com Onlight llc. www.onlight.com 2266 North Prospect Avenue #610 v. 414.272.4477 Milwaukee, Wisconsin 53202-6306 f. 414.290.0335 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090109/94191d77/attachment.html From dave64 at andrew.cmu.edu Fri Jan 9 08:21:10 2009 From: dave64 at andrew.cmu.edu (Dave McMurtrie) Date: Fri, 09 Jan 2009 08:21:10 -0500 Subject: choosing a file system In-Reply-To: <49674485.2050204@onlight.com> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> <20090109004620.GB17001@brong.net> <20090109050800.GA13154@brong.net> <2EE6AC0B-BA73-4AA2-BBB9-BE13113E38F0@nofocus.org> <20090109065959.GA6798@brong.net> <49674485.2050204@onlight.com> Message-ID: <49674F46.8090603@andrew.cmu.edu> Nic Bernstein wrote: > PS - This has been a very interesting thread to read. Some of us just > don't have the exposure to large systems like the participants in this > thread have, and this can be very educational. It's actually been helpful to us, as well. All of our mail backends are currently Solaris with SAN storage using vxfs. We're considering a move to Linux, but which filesystem to choose is still a major unanswered question for us. After reading this entire thread, it makes me realize that I've been taking vxfs for granted. It's been rock solid and the performance is fine. Thanks, Dave -- Dave McMurtrie, SPE Email Systems Team Leader Carnegie Mellon University, Computing Services From Martin.Wendel at its.uu.se Fri Jan 9 09:21:01 2009 From: Martin.Wendel at its.uu.se (Martin Wendel) Date: Fri, 09 Jan 2009 15:21:01 +0100 Subject: choosing a file system In-Reply-To: <20090109065959.GA6798@brong.net> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> <20090109004620.GB17001@brong.net> <20090109050800.GA13154@brong.net> <2EE6AC0B-BA73-4AA2-BBB9-BE13113E38F0@nofocus.org> <20090109065959.GA6798@brong.net> Message-ID: <49675D4D.6040001@its.uu.se> Bron Gondwana wrote: > On Thu, Jan 08, 2009 at 10:13:25PM -0800, Robert Banz wrote: >> (notice, didn't mention AIX. I've got my standards ;) > > Hey - I have a friend who _likes_ AIX. There are odd people in the > world. We at Uppsala university have been running cyrus on AIX for a little more than 10 years. Back then, there was no acceptable alternative to the AIX LVM. AIX still is, as I see it, very competent when it comes to handling disk I/O. About three years ago I thought time was ready for running cyrus on a shared file system. We had a large installation of IBM SANFS on another system that at the time was performing well. We purchased SANFS and four RS/6000 servers for cyrus and needed to take it in production rather abruptly after a filesystem crash on the old cyrus server. However time was not quite ready for running cyrus on SANFS. After about two years of SANFS problems (although at no time did SANFS get corrupt, it was very stable but just could not handle the load. Sometimes we needed to restart the filesystem several times a week) we decided to follow the examples published on this list by the FastMail guys. I did not dare to go with GPFS when SANFS was discontinued. We've since splitted our 20 million cyrus files onto eight IBM blade servers, running AIX virtualization server handling SAN connections for the 6 virtual RedHat servers running on each of them. We run one cyrus instance on each RedHat server, 3 primary servers and 3 replicas on each blade. We thus have 24 primary servers, and 24 replicas, with about 1 million cyrus files each. We did some tests on which file system to choose but there were not that much difference so we decided on ext3. We also have 4 additional blades running Debian, 2 for LVS and 2 for Nginx, and about 10TB of SAN disk area dedicated to cyrus. The system has been running very nicely for six months now. So I guess this is a success story inspired by FastMail. But I still would not choose anything other than AIX for our TSM servers. From lambert at lambertfam.org Fri Jan 9 09:41:38 2009 From: lambert at lambertfam.org (Scott Lambert) Date: Fri, 9 Jan 2009 08:41:38 -0600 Subject: choosing a file system In-Reply-To: <20090109085409.GF5876@helsinki.fi> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> <20090109085409.GF5876@helsinki.fi> Message-ID: <20090109144138.GA37579@sysmon.tcworks.net> On Fri, Jan 09, 2009 at 10:54:10AM +0200, Janne Peltonen wrote: > So have I. But in the current Cyrus installation, I'm stuck with Linux, > so I concentrated on what's available on Linux. Moreover, I don't want > to use non-free operating systems - if anything, I've become more > ideological with age... I'd happily use /any/ free unix variant that ran > ZFS, but. Well, fire up your test environment and start playing with FreeBSD. ZFS and DTrace and "free." The better ZFS is in 8-CURRENT. Apparently, you have to tweak things a bit on 7-STABLE still. But, by the time you (third person non-specific) get comfortable with "not Linux" 8 may be -STABLE. -- Scott Lambert KC5MLE Unix SysAdmin lambert at lambertfam.org From janne.peltonen at helsinki.fi Fri Jan 9 10:20:02 2009 From: janne.peltonen at helsinki.fi (Janne Peltonen) Date: Fri, 9 Jan 2009 17:20:02 +0200 Subject: choosing a file system In-Reply-To: <20090109144138.GA37579@sysmon.tcworks.net> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> <20090109085409.GF5876@helsinki.fi> <20090109144138.GA37579@sysmon.tcworks.net> Message-ID: <20090109152002.GH5876@helsinki.fi> On Fri, Jan 09, 2009 at 08:41:38AM -0600, Scott Lambert wrote: > On Fri, Jan 09, 2009 at 10:54:10AM +0200, Janne Peltonen wrote: > > So have I. But in the current Cyrus installation, I'm stuck with Linux, > > so I concentrated on what's available on Linux. Moreover, I don't want > > to use non-free operating systems - if anything, I've become more > > ideological with age... I'd happily use /any/ free unix variant that ran > > ZFS, but. > > Well, fire up your test environment and start playing with FreeBSD. ZFS > and DTrace and "free." The better ZFS is in 8-CURRENT. Apparently, > you have to tweak things a bit on 7-STABLE still. But, by the time you > (third person non-specific) get comfortable with "not Linux" 8 may be > -STABLE. OK, I was oversimplifying things. ZFS isn't actually non-free as such, it's just GPL-incompatible. And the "but" above did include quite a lot of things, like for instance "us" being committed to red hat / centos. If I haven't been able to even alter the Linux distribution here, just how hard would you think it'd be to try altering the unix variant... For example, one thing is, our SAN vendor says their product supports Red Hat, and I'm already treading over thin ice by using Centos. I've even been playing a little with userland ZFS, but it's far from usable in production (was a nice little toy. though, and a /lot/ faster than could be believed). I think other points concerning why not to change to another OS completely for the benefits available in ZFS were already covered by Bron, so I'm not going to waste bandwidth any more with this matter. :) --Janne -- Janne Peltonen PGP Key ID: 0x9CFAC88B Please consider membership of the Hospitality Club (http://www.hospitalityclub.org) From wes at umich.edu Fri Jan 9 11:53:17 2009 From: wes at umich.edu (Wesley Craig) Date: Fri, 9 Jan 2009 11:53:17 -0500 Subject: 32-bit to 64-bit migration seen flags In-Reply-To: <1231483653.9135.34.camel@darkstar.netcore.co.in> References: <1231483653.9135.34.camel@darkstar.netcore.co.in> Message-ID: <72D6090D-A220-43FD-B55D-8A49191287CC@umich.edu> How are you copying? :wes On 09 Jan 2009, at 01:47, ram wrote: > I am migrating mailboxes from a 32 bit cyrus (cyrus-2.3.7) to a 64 bit > cyrus (2.3.13) server > > When I copy the mailbox seen flags(skiplist) from the 32 bit server to > the 64 bit servers it does not work. All the mails are flagged as > unseen > on the new server > > Is there a way I can migrate the seen flags From blake at ispn.net Fri Jan 9 12:37:30 2009 From: blake at ispn.net (Blake Hudson) Date: Fri, 09 Jan 2009 11:37:30 -0600 Subject: howto list inactive users in cyrus: In-Reply-To: <4964DFD7.8060203@ispn.net> References: <494B8067.7060606@linpro.no> <9B733BFFD7A4289F99CF942A@[192.168.2.14]> <05ECE370-7D29-4F7D-906D-E31B54BA62EE@umich.edu> <4964DFD7.8060203@ispn.net> Message-ID: <49678B5A.4060507@ispn.net> -------- Original Message -------- Subject: Re: howto list inactive users in cyrus: From: Chris Conn To: Blake Hudson Date: Friday, January 09, 2009 10:38:38 AM > Blake Hudson wrote: >> I brought this subject up on the list a few times over the years and >> never received any interest in updating cyradm/fud to report on the >> last pop(login) date correctly. >> >> Rather than try to fix cyrus myself, I decided to write my own script >> which tails the mail log and looks for successful pop/imap/smtp >> logins and updates our user sql database with the current time and IP >> the user used to access his or her account. >> >> If someone were interested in fixing this feature of cyrus it would >> be great. >> >> -Blake > > Hello, > > You wouldn't care to post this script to the list or to share the loop > you use to extract that information from the log? I am currently > looking to work on one and if you don't mind I'd appreciate the > timesaver. > > Thanks in advance, > > Chris This is accomplished via two scripts. First, a shell script that starts, monitors, and restarts the process that pipes input to the actual logging script. And second, the script that gets data from stdin and puts that data into sql. Once the data is in sql, we can do a number of operations to classify users as active or inactive based on the last login or other criteria we already track in sql. login-runner.sh --------------------------------------------------- #!/bin/sh # # login-runner.sh # # Checks for running login-runner, if none starts a tail of the maillog # that outputs pop/imap logins to login2sql.php which puts the data into a db. # l2sLOG="/var/log/someLogFile.log" l2sRCP="monitoring-address at someDomain.com" l2sPIDS=`ps ax | grep -v grep | grep '/usr/bin/php ./login2sql.php' | awk '{print $1}'` if [ "$l2sPIDS" != "" ]; then COUNT=`echo $l2sPIDS | wc -w | tr -d " "` else COUNT=0 fi if [ $COUNT -lt 1 ]; then STAMP=`date "+%m/%d/%y %H:%M:%S"` ALERT="$COUNT processes. Restarting login2sql." echo "$STAMP `basename $0`: $ALERT" >> $l2sLOG if [ "$l2sRCP" != "" ]; then echo "`basename $0`: $ALERT" | mail -s "login2sql Restarted" $l2sRCP fi #kill $PIDS 2>/dev/null; tail -F /var/log/maillog | grep ': login:' | awk -F ']' '{print $2 $3}' | awk -F '[' '{print $2 $3}' | awk '{print $1,$2}'| ./login2sql.php else echo "$COUNT processes already running. Exiting..." fi --------------------------------------------------- login2sql.php --------------------------------------------------- #!/usr/bin/php --------------------------------------------------- From cyrus09 at mnoha.de Sat Jan 10 03:58:29 2009 From: cyrus09 at mnoha.de (mno) Date: Sat, 10 Jan 2009 09:58:29 +0100 Subject: different Cert for POP/IMAP Message-ID: Hi List, I'm using cyrus with IMAPs and POP3s and would like to use different certificates Looking at the default imap.conf file I have different parameters for a global certificate As well as individual params for Cert/keys for IMAP/ POPs etc. When configuring the individual parameter pop3_tls_cert_file and pop3_tls_key_file it seems these params are ignord. Looking at the manpage (man imapd.conf) are not mentioned. Though I' not a programmer, I had a look at the source itself and did not find any hint for the Use of pop3_tls_cert_file and pop3_tls_key_file. These params are useless - can anybody confirm this? How can I use different Certificates for pop and imap ? BR MNO From brong at fastmail.fm Sat Jan 10 04:56:00 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sat, 10 Jan 2009 20:56:00 +1100 Subject: choosing a file system In-Reply-To: <20090109152002.GH5876@helsinki.fi> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> <20090109085409.GF5876@helsinki.fi> <20090109144138.GA37579@sysmon.tcworks.net> <20090109152002.GH5876@helsinki.fi> Message-ID: <20090110095600.GA26886@brong.net> On Fri, Jan 09, 2009 at 05:20:02PM +0200, Janne Peltonen wrote: > I've even been playing a little with userland ZFS, but it's far from > usable in production (was a nice little toy. though, and a /lot/ faster > than could be believed). Yeah - zfs-on-fuse is not something I'd want to trust production data to. > I think other points concerning why not to change to another OS > completely for the benefits available in ZFS were already covered by > Bron, so I'm not going to waste bandwidth any more with this matter. :) I did get a bit worked up about it ;) Thankfully, I don't get confronted with fsck prompts very often, because my response to fsck required is pretty simple these days :) a) it's a system partition - reinstall. Takes 10 minutes from start to finish (ok, 15 on some of the bigger servers, POST being the extra) and doesn't blat data partitions. Our machines are installed using FAI to bring the base operating system up and install the "fastmail-server" Debian package, which pulls in all the packages we use as dependencies. It then checks out the latest subversion repository and does "make -C conf install" which sets up everything else. This is all per-role and per machine configured in a config file which contains lots of little micro languages optimised for being easy to read in a 'diff -u', since that's what our subversion commit hook emails us. b) if it's a cyrus partition, nuke the data and meta partitions and re-sync all users from the replicated pair. c) if it's a VFS partition, nuke it and let the automated balancing script fill it back up in its own time (this is the nicest one, all key-value based with sha1. I know I'll probably have to migrate the whole thing to sha3 at some stage, but happy to wait until it's finalised) d) oh yeah, mysql. That's replicated between two machines as well, and dumped with ibbackup every night. If we lose one of these we restore from the previous night's backup and let replication catch up. It's never happened (yet) on the primary pair - I've had to rebuild a few slaves though, so the process is well tested. So - no filesystem is sacred. Except for bloody out1 with its 1000+ queued postfix emails and no replication. It's been annoying me for over a year now, because EVERYTHING ELSE is replicated. We've got some new hardware in place, so I'm investigating drbd as an option here. Not convined. It still puts us at the mercy of a filesystem crash. I'd prefer a higher level replication solution, but I don't know any product that replicates outbound mail queues nicely between multiple machines in a way that guarantees that every mail will be delivered at least once, and if there's a machine failure the only possible failure mode is that the second machine isn't aware that the message hasn't been delivered yet, so delivers it again. That's what I want. I'd also like a replication mode for our IMAP server that guaranteed the message was actually committed to disk on both machines before returning OK to the lmtpd or imapd. That's a whole lot of work though. (we actually lost an entire external drive unit the other day, and had to move replicas to new machines. ZFS wouldn't have helped here, the failure was hardware. We would still have had perfectly good filesystems that were offline. Can't serve up emails while offline) Bron. From ml at awinkelmann.de Sat Jan 10 05:07:06 2009 From: ml at awinkelmann.de (Andreas Winkelmann) Date: Sat, 10 Jan 2009 11:07:06 +0100 (CET) Subject: different Cert for POP/IMAP In-Reply-To: <200901100901.n0A8xKli016299@lists2.andrew.cmu.edu> References: <200901100901.n0A8xKli016299@lists2.andrew.cmu.edu> Message-ID: <2b323316aa6f2388297141bddddc030e.squirrel@a-angels.ath.cx> > I'm using cyrus with IMAPs and POP3s and would like to use different > certificates > Looking at the default imap.conf file I have different parameters for a > global certificate > As well as individual params for Cert/keys for IMAP/ POPs etc. > > When configuring the individual parameter pop3_tls_cert_file and > pop3_tls_key_file it seems these params are ignord. > Looking at the manpage (man imapd.conf) are not mentioned. > Though I' not a programmer, I had a look at the source itself and did not > find any hint for the > Use of pop3_tls_cert_file and pop3_tls_key_file. > These params are useless - can anybody confirm this? > How can I use different Certificates for pop and imap ? This overrides do not work with all Options. The two you mentioned will not work this way. Your chance is to use diffrent imapd.conf-Files for each Daemon and specify it in cyrus.conf. Create one common imapd.conf for all common options and for each Daemon one with the special Options. Then you can use the include-Statement to include the common one to the specials. imapd -C imapd_imapd.conf pop3d -C pop3d_imapd.conf man imapd.conf man pop3d man imapd -- Andreas From Hagedorn at uni-koeln.de Sat Jan 10 07:38:37 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Sat, 10 Jan 2009 13:38:37 +0100 Subject: different Cert for POP/IMAP In-Reply-To: <200901100901.n0A8xKmX016299@lists2.andrew.cmu.edu> References: <200901100901.n0A8xKmX016299@lists2.andrew.cmu.edu> Message-ID: -- mno is rumored to have mumbled on 10. Januar 2009 09:58:29 +0100 regarding different Cert for POP/IMAP: > When configuring the individual parameter pop3_tls_cert_file and > pop3_tls_key_file it seems these params are ignord. > Looking at the manpage (man imapd.conf) are not mentioned. > Though I' not a programmer, I had a look at the source itself and did not > find any hint for the > Use of pop3_tls_cert_file and pop3_tls_key_file. > These params are useless - can anybody confirm this? No. > How can I use different Certificates for pop and imap ? Yes. We've got this in imapd.conf and it works just fine: sieve_tls_cert_file: /var/lib/imap/imapd.cert sieve_tls_key_file: /var/lib/imap/imapd.key imaps_tls_cert_file: /var/lib/imap/imapd.cert imap_tls_cert_file: /var/lib/imap/imapd.cert imaps_tls_key_file: /var/lib/imap/imapd.key imap_tls_key_file: /var/lib/imap/imapd.key pop3s_tls_cert_file: /var/lib/imap/popd.cert pop3_tls_cert_file: /var/lib/imap/popd.cert pop3s_tls_key_file: /var/lib/imap/popd.key pop3_tls_key_file: /var/lib/imap/popd.key tls_ca_file: /var/lib/imap/chain.pem Perhaps you need pop3s? -- Sebastian Hagedorn - RZKR-R1 (Flachbau), Zi. 18, Robert-Koch-Str. 10 Zentrum f?r angewandte Informatik - Universit?tsweiter Service RRZK Universit?t zu K?ln / Cologne University - Tel. +49-221-478-5587 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090110/13a07f2e/attachment.bin From list at joreybump.com Sat Jan 10 09:24:06 2009 From: list at joreybump.com (Jorey Bump) Date: Sat, 10 Jan 2009 09:24:06 -0500 Subject: different Cert for POP/IMAP In-Reply-To: <200901100901.n0A8xKjb016299@lists2.andrew.cmu.edu> References: <200901100901.n0A8xKjb016299@lists2.andrew.cmu.edu> Message-ID: <4968AF86.8080107@joreybump.com> mno wrote, at 01/10/2009 03:58 AM: > I'm using cyrus with IMAPs and POP3s and would like to use different > certificates > Looking at the default imap.conf file I have different parameters for a > global certificate > As well as individual params for Cert/keys for IMAP/ POPs etc. > > When configuring the individual parameter pop3_tls_cert_file and > pop3_tls_key_file it seems these params are ignord. > Looking at the manpage (man imapd.conf) are not mentioned. > Though I' not a programmer, I had a look at the source itself and did not > find any hint for the > Use of pop3_tls_cert_file and pop3_tls_key_file. > These params are useless - can anybody confirm this? > How can I use different Certificates for pop and imap ? What's in your cyrus.conf? Many parameters are global and can be overridden by prepending a *name* defined in the SERVICES section of cyrus.conf. The names are merely labels to describe the corresponding arguments, which is where the actual service is defined. While it is conventional to use names that match the service (pop3 for port 110, pop3s for port 995, imap for port 143, etc.), it's not a requirement, and is why you cannot find pop3_tls_key_file in the source. If you want to support both STARTTLS on port 143 and POP3 over SSL on port 995, you need to configure certificates for both names in the SERVICES section of cyrus.conf. From list at joreybump.com Sat Jan 10 14:35:53 2009 From: list at joreybump.com (Jorey Bump) Date: Sat, 10 Jan 2009 14:35:53 -0500 Subject: choosing a file system In-Reply-To: <20090110095600.GA26886@brong.net> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> <20090109085409.GF5876@helsinki.fi> <20090109144138.GA37579@sysmon.tcworks.net> <20090109152002.GH5876@helsinki.fi> <20090110095600.GA26886@brong.net> Message-ID: <4968F899.5000305@joreybump.com> Bron Gondwana wrote, at 01/10/2009 04:56 AM: > So - no filesystem is sacred. Except for bloody out1 with its 1000+ > queued postfix emails and no replication. It's been annoying me for > over a year now, because EVERYTHING ELSE is replicated. We've got > some new hardware in place, so I'm investigating drbd as an option > here. Not convined. It still puts us at the mercy of a filesystem > crash. > > I'd prefer a higher level replication solution, but I don't know > any product that replicates outbound mail queues nicely between > multiple machines in a way that guarantees that every mail will be > delivered at least once, and if there's a machine failure the only > possible failure mode is that the second machine isn't aware that > the message hasn't been delivered yet, so delivers it again. That's > what I want. You could regularly rsync or rdiff-backup your Postfix queue directory to another machine where Postfix lies dormant, but with a similar configuration. In the event of a machine failure, you can start up Postfix on the backup, which may even be able to function as a complete replacement (submission, MX, delivery over LMTP). There is still opportunity for minor race conditions and automating failover needs to be worked out, but it's better than nothing. Jorey ( big fan of Bron's occasional parenthetical sig comments! ) From brong at fastmail.fm Sat Jan 10 19:07:28 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sun, 11 Jan 2009 11:07:28 +1100 Subject: choosing a file system In-Reply-To: <4968F899.5000305@joreybump.com> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> <20090109085409.GF5876@helsinki.fi> <20090109144138.GA37579@sysmon.tcworks.net> <20090109152002.GH5876@helsinki.fi> <20090110095600.GA26886@brong.net> <4968F899.5000305@joreybump.com> Message-ID: <20090111000728.GA10334@brong.net> On Sat, Jan 10, 2009 at 02:35:53PM -0500, Jorey Bump wrote: > Bron Gondwana wrote, at 01/10/2009 04:56 AM: > > > So - no filesystem is sacred. Except for bloody out1 with its 1000+ > > queued postfix emails and no replication. It's been annoying me for > > over a year now, because EVERYTHING ELSE is replicated. We've got > > some new hardware in place, so I'm investigating drbd as an option > > here. Not convined. It still puts us at the mercy of a filesystem > > crash. > > > > I'd prefer a higher level replication solution, but I don't know > > any product that replicates outbound mail queues nicely between > > multiple machines in a way that guarantees that every mail will be > > delivered at least once, and if there's a machine failure the only > > possible failure mode is that the second machine isn't aware that > > the message hasn't been delivered yet, so delivers it again. That's > > what I want. > > You could regularly rsync or rdiff-backup your Postfix queue directory > to another machine where Postfix lies dormant, but with a similar > configuration. In the event of a machine failure, you can start up > Postfix on the backup, which may even be able to function as a complete > replacement (submission, MX, delivery over LMTP). There is still > opportunity for minor race conditions and automating failover needs to > be worked out, but it's better than nothing. Yeah, except Postfix encodes the inode of the queue files in its queue IDs, so it gets very confused if you do this. Same with restoring queues from backups. My searches on the postfix mailing list archives have shown similar questions being asked a couple of times, but nobody has come up with a really good solution so far. We do keep inhouse patches against postfix - I think we apply 6 at the moment. So I'm happy to make small changes to support this :) > Jorey ( big fan of Bron's occasional parenthetical sig comments! ) Bron ( I try ;) ) From chitnis.ashay at gmail.com Mon Jan 12 02:20:05 2009 From: chitnis.ashay at gmail.com (Ashay Chitnis) Date: Mon, 12 Jan 2009 12:50:05 +0530 Subject: Issue with POP accounts getting locked. Message-ID: Dear All, I have been facing this issue for quite some time now. We have a shared hosting server and cyrus-imapd-2.3.7-4. The issue is as below. 1. Most of the time some of our roaming POP service users connect to the server via a Data card on their laptops. Due to the travel they receive varying bandwidth through their ISP and at times their email client disconnects the POP server without properly terminating the POP session. Thus leaving the stale lock on the pop server. We have to manually remove the lock every time such thing happens. (We also use a script to do this) 2. Issue occurs when there is access to the email POP account from multiple Email Client at the same time. The users face issue of POP lock. I assume this is protocol level issue faced by cyrus, Can anyone tell me if there is any updated cyrus version /patch where in we can eliminate the issue of 1. Stale lock getting ever created on the server. 2. Multiple email clients can login at the same time. For issue 1. is concerned as far as my experience goes cyrus faces the issue of stale locks more often than other POP servers. Does anyone has any remedy on this? regards, Ashay. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090112/a26669a9/attachment.html From ram at netcore.co.in Mon Jan 12 04:26:46 2009 From: ram at netcore.co.in (ram) Date: Mon, 12 Jan 2009 14:56:46 +0530 Subject: 32-bit to 64-bit migration seen flags In-Reply-To: <72D6090D-A220-43FD-B55D-8A49191287CC@umich.edu> References: <1231483653.9135.34.camel@darkstar.netcore.co.in> <72D6090D-A220-43FD-B55D-8A49191287CC@umich.edu> Message-ID: <1231752406.5608.87.camel@darkstar.netcore.co.in> On Fri, 2009-01-09 at 11:53 -0500, Wesley Craig wrote: > How are you copying? > > :wes scp-ing the files > > On 09 Jan 2009, at 01:47, ram wrote: > > I am migrating mailboxes from a 32 bit cyrus (cyrus-2.3.7) to a 64 bit > > cyrus (2.3.13) server > > > > When I copy the mailbox seen flags(skiplist) from the 32 bit server to > > the 64 bit servers it does not work. All the mails are flagged as > > unseen > > on the new server > > > > Is there a way I can migrate the seen flags From ml at awinkelmann.de Mon Jan 12 05:44:09 2009 From: ml at awinkelmann.de (Andreas Winkelmann) Date: Mon, 12 Jan 2009 11:44:09 +0100 (CET) Subject: Issue with POP accounts getting locked. In-Reply-To: References: Message-ID: <2dbdd8e34ece4628747116e2f74fde1f.squirrel@a-angels.ath.cx> > I have been facing this issue for quite some time now. > > We have a shared hosting server and cyrus-imapd-2.3.7-4. The issue is as > below. > > 1. Most of the time some of our roaming POP service users connect to the > server via a Data card on their laptops. Due to the travel they receive > varying bandwidth through their ISP and at times their email client > disconnects the POP server without properly terminating the POP session. > Thus leaving the stale lock on the pop server. We have to manually remove > the lock every time such thing happens. (We also use a script to do this) > > 2. Issue occurs when there is access to the email POP account from > multiple > Email Client at the same time. The users face issue of POP lock. > > I assume this is protocol level issue faced by cyrus, Can anyone tell me > if > there is any updated cyrus version /patch where in we can eliminate the > issue of > > 1. Stale lock getting ever created on the server. There is an option poptimeout in imapd.conf you can modify it, but not setting lower than 10 minutes without modifying the Sourcecode. This limit is given from rfc. rfc1939: A POP3 server MAY have an inactivity autologout timer. Such a timer MUST be of at least 10 minutes' duration. The receipt of any command from the client during that interval should suffice to reset the autologout timer. When the timer expires, the session does NOT enter the UPDATE state--the server should close the TCP connection without removing any messages or sending any response to the client. Check if the ISP really closes the TCP/IP Connection or maybe it keeps it open for a faster reconnect of the roaming Users. > 2. Multiple email clients can login at the same time. This is designed by Protocol. If a Client connects via POP3, the Mailbox must be locked. Maybe IMAP is an option? > For issue 1. is concerned as far as my experience goes cyrus faces the > issue of stale locks more often than other POP servers. > > Does anyone has any remedy on this? -- Andreas From aspineux at gmail.com Mon Jan 12 08:14:08 2009 From: aspineux at gmail.com (Alain Spineux) Date: Mon, 12 Jan 2009 14:14:08 +0100 Subject: Delete an user + virtual domains In-Reply-To: <692f4d2f0901090421k354f2c20o8664e510ec2b9708@mail.gmail.com> References: <692f4d2f0901090421k354f2c20o8664e510ec2b9708@mail.gmail.com> Message-ID: <71fe4e760901120514k4cd1618ao2043c1b5327dcde3@mail.gmail.com> 2009/1/9 Laurent Archambault : > Hello, > > With this values in imap.conf : "virtdomains: yes" + "unixhierarchysep: yes" > (do after for it). > - In this directory : "/var/spool/cyrus/mail/domain/d/domaine2.com" > - I have for this an user this structural directory ... (for me it's strange > but i don't known all) > > ./Sent/l/user/laurent^amt > ./Sent/l/user/laurent^amt/cyrus.cache > ./Sent/l/user/laurent^amt/cyrus.index > ./Sent/l/user/laurent^amt/cyrus.squat > ./Sent/l/user/laurent^amt/cyrus.header > ./l/user/laurent^amt > ./l/user/laurent^amt/cyrus.cache > ./l/user/laurent^amt/cyrus.index > ./l/user/laurent^amt/cyrus.squat > ./l/user/laurent^amt/cyrus.header > ./Drafts/l/user/laurent^amt > ./Drafts/l/user/laurent^amt/cyrus.cache > ./Drafts/l/user/laurent^amt/cyrus.index > ./Drafts/l/user/laurent^amt/cyrus.squat > ./Drafts/l/user/laurent^amt/cyrus.header > ./Trash/l/user/laurent^amt > ./Trash/l/user/laurent^amt/cyrus.cache > ./Trash/l/user/laurent^amt/cyrus.index > ./Trash/l/user/laurent^amt/cyrus.squat > ./Trash/l/user/laurent^amt/cyrus.header > ./Junk/l/user/laurent^amt > ./Junk/l/user/laurent^amt/cyrus.cache > ./Junk/l/user/laurent^amt/cyrus.index > ./Junk/l/user/laurent^amt/cyrus.squat > ./Junk/l/user/laurent^amt/cyrus.header > > With this structure, Cyrus run perfectly... but when i want to delete an > mailbox (or account), its not clean for me ... > ./cyrus-admin.pl -u cyrus -p secret -m laurent.amt at domaine2.com --delete > Mailbox:define acl lrswipkxtecda for user/laurent.amt at domaine2.com by cyrus > Deleted Mailbox: user/laurent.amt at domaine2.com > > And now i read my structural Cyrus and i can see this : > just lhis line "./l/user/laurent^amt" is not present or deleted ... i can > see almost all ... > - And it's similar with "cyradm ...." cyradm only delete one mailbox, not the existing subfolders, you have to delete them yourself > > ./Sent/l/user/laurent^amt > ./Sent/l/user/laurent^amt/cyrus.cache > ./Sent/l/user/laurent^amt/cyrus.index > ./Sent/l/user/laurent^amt/cyrus.squat > ./Sent/l/user/laurent^amt/cyrus.header > ./Drafts/l/user/laurent^amt > ./Drafts/l/user/laurent^amt/cyrus.cache > ./Drafts/l/user/laurent^amt/cyrus.index > ./Drafts/l/user/laurent^amt/cyrus.squat > ./Drafts/l/user/laurent^amt/cyrus.header > ./Trash/l/user/laurent^amt > ./Trash/l/user/laurent^amt/cyrus.cache > ./Trash/l/user/laurent^amt/cyrus.index > ./Trash/l/user/laurent^amt/cyrus.squat > ./Trash/l/user/laurent^amt/cyrus.header > ./Junk/l/user/laurent^amt > ./Junk/l/user/laurent^amt/cyrus.cache > ./Junk/l/user/laurent^amt/cyrus.index > ./Junk/l/user/laurent^amt/cyrus.squat > ./Junk/l/user/laurent^amt/cyrus.header > > Thanks a lot ... where is my error .. > -- > ----~o00o-----//{ ??`(_)??` }\\-----o00o~------ > > Laurent Archambault > Under Linux > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > -- Alain Spineux aspineux gmail com May the sources be with you From wes at umich.edu Mon Jan 12 11:20:07 2009 From: wes at umich.edu (Wesley Craig) Date: Mon, 12 Jan 2009 11:20:07 -0500 Subject: 32-bit to 64-bit migration seen flags In-Reply-To: <1231752406.5608.87.camel@darkstar.netcore.co.in> References: <1231483653.9135.34.camel@darkstar.netcore.co.in> <72D6090D-A220-43FD-B55D-8A49191287CC@umich.edu> <1231752406.5608.87.camel@darkstar.netcore.co.in> Message-ID: <5EBE2ECB-4B35-4467-8E46-92DAB5559766@umich.edu> On 12 Jan 2009, at 04:26, ram wrote: > On Fri, 2009-01-09 at 11:53 -0500, Wesley Craig wrote: >> How are you copying? > > scp-ing the files Well, the data in the seen files is stored by unique ID of the mailbox -- which is stored as a string. The data stored is the "seen version" (which has been 1 for a long time), the last read time (time_t cast to int), last uid (int), last change time (time_t cast to int), and seen uids (a string created earlier from unsigned long). I suppose it's possible that the casts of time_t to int are corrupting your results. What value is time_t on your system? It absolutely true that casting time_t to int is *wrong*, and should be handled in some other way, e.g., TIME_T_FMT. Of course, that has (different) implications for migrating between platforms as well. You might try dumping the data with cyr_dbtool on yor 32-bit system and reloading it with cyr_dbtool on your 64-bit system. I doubt that will help, but you never know. I think a much more likely scenario is that however you're copying the mailboxes is causing the unique ID of the mailboxes to be re-assigned. Since all of the data in the seen DB is keyed on the unique ID of the mailbox, if the unique ID of the mailbox changes, all of the seen data is effectively lost. Probably a uid validity change would have a similar effect. :wes >> On 09 Jan 2009, at 01:47, ram wrote: >>> I am migrating mailboxes from a 32 bit cyrus (cyrus-2.3.7) to a >>> 64 bit >>> cyrus (2.3.13) server >>> >>> When I copy the mailbox seen flags(skiplist) from the 32 bit >>> server to >>> the 64 bit servers it does not work. All the mails are flagged as >>> unseen >>> on the new server >>> >>> Is there a way I can migrate the seen flags From janne.peltonen at helsinki.fi Mon Jan 12 11:47:10 2009 From: janne.peltonen at helsinki.fi (Janne Peltonen) Date: Mon, 12 Jan 2009 18:47:10 +0200 Subject: Elusive replication bug in 2.3.13 Message-ID: <20090112164709.GA1417@helsinki.fi> Hi! Since I upgraded to 2.3.13 (Invoca RPM rev 4) I've been running into a mysterious replication bug. In some circumstances, creating a user with a three letter long username causes the sync master process to choke, on either signal 11 or 6. Like this: --clip-- i08.mappi.helsinki.fi> cm user.jmm --clip-- Jan 12 18:31:01 scn3 i08/syncserver[25821]: Failed to access inbox for jmm Jan 12 18:31:01 scn3 i08/master[31569]: process 25821 exited, signaled to death by 6 Jan 12 18:31:07 scn3 i08/syncserver[30193]: login: pcn3.mappi.helsinki.fi [128.214.20.131] cyr_sync DIGEST-MD5 User logged in Jan 12 18:31:07 scn3 i08/syncserver[30193]: Failed to access inbox for jmm Jan 12 18:31:07 scn3 i08/master[31569]: process 30193 exited, signaled to death by 11 [etc, the sync_client keeps retrying] --clip-- [jmmpelto at pcn3 ~]$ ps -ef|grep sync.*i08 cyrus 8255 1 0 18:16 ? 00:00:00 /usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master cyrus 22092 8255 0 18:33 ? 00:00:00 /usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master jmmpelto 22095 20355 0 18:33 pts/4 00:00:00 grep sync.*i08 [jmmpelto at pcn3 ~]$ sudo kill 8255 Password: [jmmpelto at pcn3 ~]$ ps -ef|grep sync.*i08 jmmpelto 25898 20355 0 18:34 pts/4 00:00:00 grep sync.*i08 [jmmpelto at pcn3 ~]$ [the child died, again, because the server died] --clip-- [jmmpelto at pcn3 ~]$ sudo su - cyrus -c "/usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master" [jmmpelto at pcn3 ~]$ ps -ef|grep sync.*i08 cyrus 11483 1 0 18:35 ? 00:00:00 /usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master cyrus 11484 11483 1 18:35 ? 00:00:00 /usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master jmmpelto 13382 20355 0 18:35 pts/4 00:00:00 grep sync.*i08 [after restart, rolling replication runs OK] --clip-- [jmmpelto at scn3 ~]$ sudo /usr/lib/cyrus-imapd/mbpath -C /etc/imapd.conf.i08.replica user.jmm Invalid mailbox name: user.jmm [but the mailbox never got replicated] --clip-- -bash-3.2$ cat log-8255 USER jmm -bash-3.2$ /usr/lib/cyrus-imapd/sync_client -v -o -r -C /etc/imapd.conf.i08.master -f log-8255 USER jmm USER jmm USER jmm --clip-- Jan 12 18:39:19 scn3 i08/syncserver[30016]: login: pcn3.mappi.helsinki.fi [128.214.20.131] cyr_sync DIGEST-MD5 User logged in Jan 12 18:39:19 scn3 i08/syncserver[30016]: Failed to access inbox for jmm Jan 12 18:39:19 scn3 i08/master[31569]: process 30016 exited, signaled to death by 11 [jmmpelto at scn3 ~]$ sudo /usr/lib/cyrus-imapd/mbpath -C /etc/imapd.conf.i08.replica user.jmm Invalid mailbox name: user.jmm [trying to say "USER jmm" results in sig11 death, no replication] --clip-- -bash-3.2$ /usr/lib/cyrus-imapd/sync_client -v -o -r -C /etc/imapd.conf.i08.master -f - ACL user.jmm SETACL user.jmm Promoting: ACL user.jmm -> MAILBOX user.jmm MAILBOXES user.jmm META jmm [..however, doing something that gets escalated to creating the user...] --clip-- [jmmpelto at scn3 ~]$ sudo /usr/lib/cyrus-imapd/mbpath -C /etc/imapd.conf.i08.replica user.jmm /var/spool/imap/jjjr/j/user/jmm [...doesn't result in process death, instead, the user & mailbox do get replicated] --clip-- The irritating thing about this is, I can't reliably recreate the bug. It appears that - the user name has to be a prefix of already existing usernames - there has to be at least some (ten? twenty?) usernames the new username is a prefix of - however, when trying to create users jmbtest1..9 and jmbtes10..20, I wasn't able to recreate the bug with user.jmb (there were 5 'real' usernames with the prefix jmb), even though I was able to hit it with user.jme that only has 18 real usernames... I didn't have this problem with 2.3.11, so it has to be relatively new. Might anyone else have noticed anything similar? --Janne -- Janne Peltonen PGP Key ID: 0x9CFAC88B Please consider membership of the Hospitality Club (http://www.hospitalityclub.org) From anebi at iguanait.com Mon Jan 12 12:00:52 2009 From: anebi at iguanait.com (anebi at iguanait.com) Date: Mon, 12 Jan 2009 19:00:52 +0200 Subject: Issue with pop3s abnormally terminating Message-ID: <1231779652.11639.10.camel@hugo.iguanait.com> Hi, we have installed cyrus-imapd-2.3.7-2 on Centos 5. Everything work very well, but last days i noticed that we get an error related with pop3s. The error is this: Jan 4 05:45:20 herm pop3s[21598]: pop3s failed: Dynamic-IP-cr20011xxxx.xxxx.xxx [200.xxx.xxx.xxx] Jan 4 05:45:20 herm pop3s[21598]: Fatal error: tls_start_servertls() failed Jan 4 05:45:20 herm master[3688]: process 21598 exited, status 75 Jan 4 05:45:20 herm master[3688]: service pop3s pid 21598 in BUSY state: terminated abnormally I tested pop3s with pop3test tool and also with a real user that was configured to use pop3s. It is login in successful and i tried to reproduce this error, but i still cannot do it. I checked my configuration files related with cyrus-imap. Everything is ok. Imap/Pop3 with TLS and Imaps/Pop3s are working good. Why this happen and how to solve this problem? Do i need to increase some number of processes for pop3 or something different? I read in the net a lot of comments about this problem, but i still can't find a solution for this. I will be happy if someone give me solve direction to troubleshoot this problem and try to solve it. Thanks in advanced! Regards, Ali Nebi! From cyrus09 at mnoha.de Mon Jan 12 12:34:49 2009 From: cyrus09 at mnoha.de (mno) Date: Mon, 12 Jan 2009 18:34:49 +0100 Subject: AW: different Cert for POP/IMAP In-Reply-To: <200901100859.n0A8xKd6016299@lists2.andrew.cmu.edu> References: <200901100859.n0A8xKd6016299@lists2.andrew.cmu.edu> Message-ID: FYI - SOLVED, solved, both solutions work 1) change in cyrus.conf and use different imapd.conf - files for pops and imaps 2) the right name for the option is pop3s_tls_cert_file and pop3s_tls_key_file, Only one imapd.conf - file is needed. Thanks Mno -----Urspr?ngliche Nachricht----- Von: info-cyrus-bounces+cyrus09=mnoha.de at lists.andrew.cmu.edu [mailto:info-cyrus-bounces+cyrus09=mnoha.de at lists.andrew.cmu.edu] Im Auftrag von mno Gesendet: Samstag, 10. Januar 2009 09:58 An: info-cyrus at lists.andrew.cmu.edu Betreff: different Cert for POP/IMAP Hi List, I'm using cyrus with IMAPs and POP3s and would like to use different certificates Looking at the default imap.conf file I have different parameters for a global certificate As well as individual params for Cert/keys for IMAP/ POPs etc. When configuring the individual parameter pop3_tls_cert_file and pop3_tls_key_file it seems these params are ignord. Looking at the manpage (man imapd.conf) are not mentioned. Though I' not a programmer, I had a look at the source itself and did not find any hint for the Use of pop3_tls_cert_file and pop3_tls_key_file. These params are useless - can anybody confirm this? How can I use different Certificates for pop and imap ? BR MNO ---- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html From rjaffey at artic.edu Mon Jan 12 12:38:56 2009 From: rjaffey at artic.edu (Raphael Jaffey) Date: Mon, 12 Jan 2009 11:38:56 -0600 Subject: Elusive replication bug in 2.3.13 Message-ID: <20090112113856.8xc60s5v0gwgwo8k@webmail-test.artic.edu> Hi Janne, I've reproduced this with both the vanilla Cyrus tarball and the Invoca RPM. Check out bug report 3130 at: https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3130 and my post at: http://lists.andrew.cmu.edu/pipermail/info-cyrus/2009-January/030279.html for a discussion of what looks like the same problem you're experiencing and a workaround that seems to solve the problem. In my case it's related to how many of those inboxes have quotas associated with them and the use of the default 'quota_db' of 'quotalegacy'. I was able to work around the replication problem by switching 'quota_db' to 'skiplist' on the replica. Unfortunately I couldn't get berkeley or berkeley-hash to work reliably and as I'm unsure as to which database other than 'quotalegacy' would be most appropriate. So far, no one has suggested anything. What happens when you try to delete user.jmm on the master? Does the IMAP process handling the delete segfault? Regards, Rafe Quoting Janne Peltonen : > Hi! > > Since I upgraded to 2.3.13 (Invoca RPM rev 4) I've been running into a > mysterious replication bug. In some circumstances, creating a user > with a three > letter long username causes the sync master process to choke, on > either signal > 11 or 6. Like this: > > --clip-- > i08.mappi.helsinki.fi> cm user.jmm > --clip-- > Jan 12 18:31:01 scn3 i08/syncserver[25821]: Failed to access inbox for jmm > Jan 12 18:31:01 scn3 i08/master[31569]: process 25821 exited, > signaled to death by 6 > Jan 12 18:31:07 scn3 i08/syncserver[30193]: login: > pcn3.mappi.helsinki.fi [128.214.20.131] cyr_sync DIGEST-MD5 User > logged in > Jan 12 18:31:07 scn3 i08/syncserver[30193]: Failed to access inbox for jmm > Jan 12 18:31:07 scn3 i08/master[31569]: process 30193 exited, > signaled to death by 11 > [etc, the sync_client keeps retrying] > --clip-- > [jmmpelto at pcn3 ~]$ ps -ef|grep sync.*i08 > cyrus 8255 1 0 18:16 ? 00:00:00 > /usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master > cyrus 22092 8255 0 18:33 ? 00:00:00 > /usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master > jmmpelto 22095 20355 0 18:33 pts/4 00:00:00 grep sync.*i08 > [jmmpelto at pcn3 ~]$ sudo kill 8255 > Password: > [jmmpelto at pcn3 ~]$ ps -ef|grep sync.*i08 > jmmpelto 25898 20355 0 18:34 pts/4 00:00:00 grep sync.*i08 > [jmmpelto at pcn3 ~]$ > [the child died, again, because the server died] > --clip-- > [jmmpelto at pcn3 ~]$ sudo su - cyrus -c > "/usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master" > [jmmpelto at pcn3 ~]$ ps -ef|grep sync.*i08 > cyrus 11483 1 0 18:35 ? 00:00:00 > /usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master > cyrus 11484 11483 1 18:35 ? 00:00:00 > /usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master > jmmpelto 13382 20355 0 18:35 pts/4 00:00:00 grep sync.*i08 > [after restart, rolling replication runs OK] > --clip-- > [jmmpelto at scn3 ~]$ sudo /usr/lib/cyrus-imapd/mbpath -C > /etc/imapd.conf.i08.replica user.jmm > Invalid mailbox name: user.jmm > [but the mailbox never got replicated] > --clip-- > -bash-3.2$ cat log-8255 > USER jmm > -bash-3.2$ /usr/lib/cyrus-imapd/sync_client -v -o -r -C > /etc/imapd.conf.i08.master -f log-8255 > USER jmm > USER jmm > USER jmm > --clip-- > Jan 12 18:39:19 scn3 i08/syncserver[30016]: login: > pcn3.mappi.helsinki.fi [128.214.20.131] cyr_sync DIGEST-MD5 User > logged in > Jan 12 18:39:19 scn3 i08/syncserver[30016]: Failed to access inbox for jmm > Jan 12 18:39:19 scn3 i08/master[31569]: process 30016 exited, > signaled to death by 11 > [jmmpelto at scn3 ~]$ sudo /usr/lib/cyrus-imapd/mbpath -C > /etc/imapd.conf.i08.replica user.jmm > Invalid mailbox name: user.jmm > [trying to say "USER jmm" results in sig11 death, no replication] > --clip-- > -bash-3.2$ /usr/lib/cyrus-imapd/sync_client -v -o -r -C > /etc/imapd.conf.i08.master -f - > ACL user.jmm > SETACL user.jmm > Promoting: ACL user.jmm -> MAILBOX user.jmm > MAILBOXES user.jmm > META jmm > [..however, doing something that gets escalated to creating the user...] > --clip-- > [jmmpelto at scn3 ~]$ sudo /usr/lib/cyrus-imapd/mbpath -C > /etc/imapd.conf.i08.replica user.jmm > /var/spool/imap/jjjr/j/user/jmm > [...doesn't result in process death, instead, the user & mailbox do > get replicated] > --clip-- > > The irritating thing about this is, I can't reliably recreate the > bug. It appears that > > - the user name has to be a prefix of already existing usernames > > - there has to be at least some (ten? twenty?) usernames the new > username is a > prefix of - however, when trying to create users jmbtest1..9 and > jmbtes10..20, > I wasn't able to recreate the bug with user.jmb (there were 5 'real' > usernames > with the prefix jmb), even though I was able to hit it with user.jme > that only > has 18 real usernames... > > I didn't have this problem with 2.3.11, so it has to be relatively new. Might > anyone else have noticed anything similar? > > > --Janne > -- > Janne Peltonen PGP Key ID: 0x9CFAC88B > Please consider membership of the Hospitality Club > (http://www.hospitalityclub.org) > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > From list at joreybump.com Mon Jan 12 12:57:06 2009 From: list at joreybump.com (Jorey Bump) Date: Mon, 12 Jan 2009 12:57:06 -0500 Subject: AW: different Cert for POP/IMAP In-Reply-To: <200901121737.n0CHZTGK016169@lists2.andrew.cmu.edu> References: <200901100859.n0A8xKd6016299@lists2.andrew.cmu.edu> <200901121737.n0CHZTGK016169@lists2.andrew.cmu.edu> Message-ID: <496B8472.3050800@joreybump.com> mno wrote, at 01/12/2009 12:34 PM: > 2) the right name for the option is pop3s_tls_cert_file and > pop3s_tls_key_file, [snip] > Though I' not a programmer, I had a look at the source itself and did not > find any hint for the > Use of pop3_tls_cert_file and pop3_tls_key_file. > These params are useless - can anybody confirm this? Note that the pop3_tls_(cert|key)_file params are not useless. If you support STARTTLS on the standard pop3 port 143 (you should), you will want to set these as well, if you want the same certificate to be presented as on pop3s port 995. From davidk at cs.umass.edu Mon Jan 12 15:32:47 2009 From: davidk at cs.umass.edu (David Korpiewski) Date: Mon, 12 Jan 2009 15:32:47 -0500 Subject: Error dragging and dropping a folder from one server to another for only one user Message-ID: <496BA8EF.6030502@cs.umass.edu> I have two cyrus servers and I'm dragging and dropping a folder within mail.app from one server to another and it is giving me the error for only one user: "Some items could not be copied to "mailserver"" "The IMAP command "CREATE" failed with server error: INvalid mailbox name." Does anyone know why this could be happening? The user can create a new mailbox and copy email into it, but cannot drag and drop an existing folder. Strange. Thanks! David -- =========================================== David Korpiewski Software Specialist I CSCF - Computer Science Computing Facility Department of Computer Science Phone: 413-545-4319 Fax: 413-577-2285 =========================================== From wes at umich.edu Mon Jan 12 16:04:16 2009 From: wes at umich.edu (Wesley Craig) Date: Mon, 12 Jan 2009 16:04:16 -0500 Subject: Issue with pop3s abnormally terminating In-Reply-To: <1231779652.11639.10.camel@hugo.iguanait.com> References: <1231779652.11639.10.camel@hugo.iguanait.com> Message-ID: <36F9F73B-06AE-4EF4-B53A-227C6FE4E61F@umich.edu> Kinda looks like a port probe. telnet to the pop3s port and then use the telnet escape key to quit the connection. If you'd like to submit a patch to reduce the level of logging for these errors to something a little more sane (like DEBUG), I'm sure it would be accepted through Bugzilla. :wes On 12 Jan 2009, at 12:00, anebi at iguanait.com wrote: > we have installed cyrus-imapd-2.3.7-2 on Centos 5. Everything work > very > well, but last days i noticed that we get an error related with pop3s. > The error is this: > > Jan 4 05:45:20 herm pop3s[21598]: pop3s failed: > Dynamic-IP-cr20011xxxx.xxxx.xxx [200.xxx.xxx.xxx] > Jan 4 05:45:20 herm pop3s[21598]: Fatal error: tls_start_servertls() > failed > Jan 4 05:45:20 herm master[3688]: process 21598 exited, status 75 > Jan 4 05:45:20 herm master[3688]: service pop3s pid 21598 in BUSY > state: terminated abnormally > > I tested pop3s with pop3test tool and also with a real user that was > configured to use pop3s. It is login in successful and i tried to > reproduce this error, but i still cannot do it. > > I checked my configuration files related with cyrus-imap. > Everything is > ok. Imap/Pop3 with TLS and Imaps/Pop3s are working good. > > Why this happen and how to solve this problem? > > Do i need to increase some number of processes for pop3 or something > different? > > > I read in the net a lot of comments about this problem, but i still > can't find a solution for this. > > I will be happy if someone give me solve direction to troubleshoot > this > problem and try to solve it. From wes at umich.edu Mon Jan 12 16:05:50 2009 From: wes at umich.edu (Wesley Craig) Date: Mon, 12 Jan 2009 16:05:50 -0500 Subject: Error dragging and dropping a folder from one server to another for only one user In-Reply-To: <496BA8EF.6030502@cs.umass.edu> References: <496BA8EF.6030502@cs.umass.edu> Message-ID: <047B7C85-72E1-489E-8A3B-72F3E2D469C9@umich.edu> On 12 Jan 2009, at 15:32, David Korpiewski wrote: > I have two cyrus servers and I'm dragging and dropping a folder within > mail.app from one server to another and it is giving me the error for > only one user: > > "Some items could not be copied to "mailserver"" > "The IMAP command "CREATE" failed with server error: INvalid > mailbox name." > > Does anyone know why this could be happening? The user can create a > new mailbox and copy email into it, but cannot drag and drop an > existing > folder. Enable telemetry and see what Mail.app is doing. :wes From davidk at cs.umass.edu Mon Jan 12 16:59:28 2009 From: davidk at cs.umass.edu (David Korpiewski) Date: Mon, 12 Jan 2009 16:59:28 -0500 Subject: Error dragging and dropping a folder from one server to another for only one user In-Reply-To: <047B7C85-72E1-489E-8A3B-72F3E2D469C9@umich.edu> References: <496BA8EF.6030502@cs.umass.edu> <047B7C85-72E1-489E-8A3B-72F3E2D469C9@umich.edu> Message-ID: <496BBD40.2030006@cs.umass.edu> Forgive me, but what is "telemetry". I talked to our mac guru and he was unaware what that was either. This problem continues to plague us even if we use cyradm and delete the account outright. We verify that the account was deleted and the directory in the mail store was gone. Then when it comes back the problem still exists. A new finding (and work-around) is that if we drag local folders to the mail server having issues, the folders copy. The problem exists from server to server (same versions and all). So our work-around is to copy the one server's folders to the local disk and then over to the new server. Thanks David Wesley Craig wrote: > On 12 Jan 2009, at 15:32, David Korpiewski wrote: >> I have two cyrus servers and I'm dragging and dropping a folder within >> mail.app from one server to another and it is giving me the error for >> only one user: >> >> "Some items could not be copied to "mailserver"" >> "The IMAP command "CREATE" failed with server error: INvalid >> mailbox name." >> >> Does anyone know why this could be happening? The user can create a >> new mailbox and copy email into it, but cannot drag and drop an >> existing >> folder. > > Enable telemetry and see what Mail.app is doing. > > :wes > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- =========================================== David Korpiewski Software Specialist I CSCF - Computer Science Computing Facility Department of Computer Science Phone: 413-545-4319 Fax: 413-577-2285 =========================================== From brong at fastmail.fm Mon Jan 12 19:14:18 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 13 Jan 2009 11:14:18 +1100 Subject: 32-bit to 64-bit migration seen flags In-Reply-To: <5EBE2ECB-4B35-4467-8E46-92DAB5559766@umich.edu> References: <1231483653.9135.34.camel@darkstar.netcore.co.in> <72D6090D-A220-43FD-B55D-8A49191287CC@umich.edu> <1231752406.5608.87.camel@darkstar.netcore.co.in> <5EBE2ECB-4B35-4467-8E46-92DAB5559766@umich.edu> Message-ID: <20090113001418.GA8018@brong.net> On Mon, Jan 12, 2009 at 11:20:07AM -0500, Wesley Craig wrote: > On 12 Jan 2009, at 04:26, ram wrote: > > On Fri, 2009-01-09 at 11:53 -0500, Wesley Craig wrote: > >> How are you copying? > > > > scp-ing the files > > Well, the data in the seen files is stored by unique ID of the > mailbox -- which is stored as a string. The data stored is the "seen > version" (which has been 1 for a long time), the last read time > (time_t cast to int), last uid (int), last change time (time_t cast > to int), and seen uids (a string created earlier from unsigned long). > > I suppose it's possible that the casts of time_t to int are > corrupting your results. What value is time_t on your system? It > absolutely true that casting time_t to int is *wrong*, and should be > handled in some other way, e.g., TIME_T_FMT. Of course, that has > (different) implications for migrating between platforms as well. Yeah - when I was playing with skiplist2 a while back, I set aside 64 bits for each time value (there are only a couple of them in the header, and we have spare slots in the header anyway. > You might try dumping the data with cyr_dbtool on yor 32-bit system > and reloading it with cyr_dbtool on your 64-bit system. I doubt that > will help, but you never know. I think a much more likely scenario > is that however you're copying the mailboxes is causing the unique ID > of the mailboxes to be re-assigned. Since all of the data in the > seen DB is keyed on the unique ID of the mailbox, if the unique ID of > the mailbox changes, all of the seen data is effectively lost. > Probably a uid validity change would have a similar effect. the uniqueid is stored in the cyrus.header file. If you're copying the cyrus.header files then you shouldn't lose it. Bron. From brong at fastmail.fm Mon Jan 12 19:50:09 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 13 Jan 2009 11:50:09 +1100 Subject: Error dragging and dropping a folder from one server to another for only one user In-Reply-To: <496BBD40.2030006@cs.umass.edu> References: <496BA8EF.6030502@cs.umass.edu> <047B7C85-72E1-489E-8A3B-72F3E2D469C9@umich.edu> <496BBD40.2030006@cs.umass.edu> Message-ID: <20090113005009.GB8608@brong.net> On Mon, Jan 12, 2009 at 04:59:28PM -0500, David Korpiewski wrote: > Forgive me, but what is "telemetry". I talked to our mac guru and he > was unaware what that was either. The first hit on google for "cyrus telemetry" give this: http://www.irbs.net/internet/info-cyrus/0604/0313.html Which is a pretty good description. Basically, you create a directory in /log/ with the username of the user you want to watch, and it logs all traffic on connections for that user. (something like /var/imap/log/brong for me) > This problem continues to plague us > even if we use cyradm and delete the account outright. We verify that > the account was deleted and the directory in the mail store was gone. > Then when it comes back the problem still exists. It sounds like there's an issue with the encoding of a folder name to me - when Mac mail sends it. Telemetry will show it. > A new finding (and work-around) is that if we drag local folders to the > mail server having issues, the folders copy. The problem exists from > server to server (same versions and all). So our work-around is to > copy the one server's folders to the local disk and then over to the new > server. Sounds like there's an ImapUTF-7 encoding/decoding issue - probably not re-encoding the value for one end. Bron. From Leo.Cheng at bdci.com.hk Mon Jan 12 20:54:30 2009 From: Leo.Cheng at bdci.com.hk (Leo P CHENG) Date: Tue, 13 Jan 2009 09:54:30 +0800 Subject: help: i want to unsubscribe mail list Message-ID: Hi : Excuse me , I want to unsubscribe mail list from info-cyrus at lists.andrew.cmu.edu . But , how to do it ? Help! Thanks leocheng -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090113/4ab68cf8/attachment.html From awilliam at whitemice.org Mon Jan 12 21:01:55 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Mon, 12 Jan 2009 21:01:55 -0500 Subject: help: i want to unsubscribe mail list In-Reply-To: References: Message-ID: <1231812115.5923.1.camel@linux-nnci.site> On Tue, 2009-01-13 at 09:54 +0800, Leo P CHENG wrote: > Hi : > > > > Excuse me , I want to unsubscribe mail list from > info-cyrus at lists.andrew.cmu.edu . > > But , how to do it ? As included in every message: From awilliam at whitemice.org Mon Jan 12 21:02:25 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Mon, 12 Jan 2009 21:02:25 -0500 Subject: help: i want to unsubscribe mail list In-Reply-To: References: Message-ID: <1231812145.5923.3.camel@linux-nnci.site> On Tue, 2009-01-13 at 09:54 +0800, Leo P CHENG wrote: > Hi : > Excuse me , I want to unsubscribe mail list from > info-cyrus at lists.andrew.cmu.edu . > But , how to do it ? As included in every message - ?List-Unsubscribe: , From janne.peltonen at helsinki.fi Tue Jan 13 03:50:18 2009 From: janne.peltonen at helsinki.fi (Janne Peltonen) Date: Tue, 13 Jan 2009 10:50:18 +0200 Subject: Elusive replication bug in 2.3.13 In-Reply-To: <20090112113856.8xc60s5v0gwgwo8k@webmail-test.artic.edu> References: <20090112113856.8xc60s5v0gwgwo8k@webmail-test.artic.edu> Message-ID: <20090113085017.GE30870@helsinki.fi> Hi, On Mon, Jan 12, 2009 at 11:38:56AM -0600, Raphael Jaffey wrote: > for a discussion of what looks like the same problem you're > experiencing and a workaround that seems to solve the problem. Thanks for the info. > In my case it's related to how many of those inboxes have quotas > associated with them and the use of the default 'quota_db' of > 'quotalegacy'. I was able to work around the replication problem by > switching 'quota_db' to 'skiplist' on the replica. Unfortunately I > couldn't get berkeley or berkeley-hash to work reliably and as I'm > unsure as to which database other than 'quotalegacy' would be most > appropriate. So far, no one has suggested anything. I could only find this thread that remotely touches the subject: http://osdir.com/ml/security.cyrus.sasl/2005-02/msg00024.html and it's four years old... How long did it take you to convert your quota db from the legacy format to skiplist? Was it doable? > What happens when you try to delete user.jmm on the master? Does the > IMAP process handling the delete segfault? Yes, it does. --Janne -- Janne Peltonen PGP Key ID: 0x9CFAC88B Please consider membership of the Hospitality Club (http://www.hospitalityclub.org) From s.dewal at computel.nl Tue Jan 13 04:34:51 2009 From: s.dewal at computel.nl (s.dewal at computel.nl) Date: Tue, 13 Jan 2009 10:34:51 +0100 Subject: migrate from 64 to 32 bits Message-ID: <002f01c97562$2f0ff6f0$8d2fe4d0$@dewal@computel.nl> Hi, A customer of outs wishes to migrate cyrus and postfix from a 64 bits to a 32 bits server. Normally I just copy yhe configuration files and several directories ( /var/lib/imap and /var/spool/imap) but the question is, are the file compatible in a 32 bits environment when copied from a 64 bits environment? Regards, Stefan de Wal RHCE From rjaffey at artic.edu Tue Jan 13 08:36:12 2009 From: rjaffey at artic.edu (Raphael Jaffey) Date: Tue, 13 Jan 2009 07:36:12 -0600 Subject: Elusive replication bug in 2.3.13 In-Reply-To: <20090113085017.GE30870@helsinki.fi> References: <20090112113856.8xc60s5v0gwgwo8k@webmail-test.artic.edu> <20090113085017.GE30870@helsinki.fi> Message-ID: <20090113073612.hwfqmbtkowk844og@webmail-test.artic.edu> Quoting Janne Peltonen : > I could only find this thread that remotely touches the subject: > > http://osdir.com/ml/security.cyrus.sasl/2005-02/msg00024.html > > and it's four years old... How long did it take you to convert your quota > db from the legacy format to skiplist? Was it doable? > Our system contains ~120K mailboxes and about 40K have quotas on them. I don't recall exactly, but I think the entire conversion took less than an hour. Cyrus was only down for about 15 seconds in order to switch to the new 'quota_db' setting. Rafe From andre at digirati.com.br Tue Jan 13 18:47:47 2009 From: andre at digirati.com.br (Andre Nathan) Date: Tue, 13 Jan 2009 21:47:47 -0200 Subject: Ruby/ManageSieve 0.4.0 released Message-ID: <1231890467.5998.11.camel@homesick> Hello. This a new release of Ruby/ManageSieve, adding TLS support. Ruby/ManageSieve is a pure-ruby library for the MANAGESIEVE protocol. It also includes the command-line ``sievectl'' utility for managing Sieve scripts. Please check the Ruby/ManageSieve homepage for documentation and downloads: http://managesieve.rubyforge.org/ A rubygem is also available: gem install ruby-managesieve == Changes from 0.3.0: - TLS support (implemented by Alban Peignier. Thank you!). - Avoiding breakage due to deprecated rubygems method calls. Best regards, Andre From robm at fastmail.fm Tue Jan 13 20:25:02 2009 From: robm at fastmail.fm (Rob Mueller) Date: Wed, 14 Jan 2009 12:25:02 +1100 Subject: Elusive replication bug in 2.3.13 In-Reply-To: <20090112164709.GA1417@helsinki.fi> References: <20090112164709.GA1417@helsinki.fi> Message-ID: <8C506188988F4A159973A4661E0955C9@Atticus> > Since I upgraded to 2.3.13 (Invoca RPM rev 4) I've been running into a > mysterious replication bug. In some circumstances, creating a user with a > three > letter long username causes the sync master process to choke, on either > signal > 11 or 6. Like this: Interestingly, we just encountered this as well after a new rebuild from CVS. I got a stack back trace, which showed realloc() was choking. I looked at the code around there, and couldn't see anything obviously wrong. Bron looked at it and saw the problem in 10 seconds. I think I've been out of practice with C programming for too long :) The bug occured in this commit where Ken changed a glob() call to a custom readdir() loop. http://github.com/brong/cyrus-imapd/commit/56d0823d3192ea13fa3afe397d625252b16252b9 There's a missing sizeof() in there meaning that if you have > 25 quotas in a dir (or > 12 on a 64-bit build), you'll have memory corruption occuring. Here's the patch to fix: http://github.com/brong/cyrus-imapd/commit/1de9d758aeb360714236388c4e1689db0522c21e All the fastmail patches are now on github, you can see them here: http://github.com/brong/cyrus-imapd/commits/fastmail/ http://github.com/brong/cyrus-sieve/commits/fastmail/ Rob FYI: We haven't changed to skiplist for quota db's because they get a lot of writes, and a skiplist db requires locking the entire db for the period of the write, so I'm worried about contention issues. To be honest, I haven't tested, so it might not be a problem at all, it's just something we'd need to test first. From ram at netcore.co.in Wed Jan 14 01:45:55 2009 From: ram at netcore.co.in (ram) Date: Wed, 14 Jan 2009 12:15:55 +0530 Subject: migrate from 64 to 32 bits In-Reply-To: <002f01c97562$2f0ff6f0$8d2fe4d0$@dewal@computel.nl> References: <002f01c97562$2f0ff6f0$8d2fe4d0$@dewal@computel.nl> Message-ID: <1231915555.30300.57.camel@darkstar.netcore.co.in> On Tue, 2009-01-13 at 10:34 +0100, s.dewal at computel.nl wrote: > Hi, > > A customer of outs wishes to migrate cyrus and postfix from a 64 bits to a > 32 bits server. Normally I just copy yhe configuration files and several > directories ( /var/lib/imap and /var/spool/imap) but the question is, are > the file compatible in a 32 bits environment when copied from a 64 bits > environment? > > Regards, > How strange , I had a problem doing right the opposite. ( from 32 to 64 ) We migrated the mailboxes , from 32 bit to 64 bit , and all the seen flags got messed up. I had to run imapsync to get all the seen flags set up The best way to do migration is , setup a new server migrate one or two mailboxes and see for yourself what breaks > Stefan de Wal > RHCE > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html From simon.matter at invoca.ch Wed Jan 14 01:49:40 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Wed, 14 Jan 2009 07:49:40 +0100 (CET) Subject: Elusive replication bug in 2.3.13 In-Reply-To: <8C506188988F4A159973A4661E0955C9@Atticus> References: <20090112164709.GA1417@helsinki.fi> <8C506188988F4A159973A4661E0955C9@Atticus> Message-ID: <47cadc79b48df3ce5780e2598a378e55.squirrel@webmail.bi.corp.invoca.ch> >> Since I upgraded to 2.3.13 (Invoca RPM rev 4) I've been running into a >> mysterious replication bug. In some circumstances, creating a user with >> a >> three >> letter long username causes the sync master process to choke, on either >> signal >> 11 or 6. Like this: > > Interestingly, we just encountered this as well after a new rebuild from > CVS. > > I got a stack back trace, which showed realloc() was choking. I looked at > the code around there, and couldn't see anything obviously wrong. Bron > looked at it and saw the problem in 10 seconds. I think I've been out of > practice with C programming for too long :) > > The bug occured in this commit where Ken changed a glob() call to a custom > readdir() loop. > > http://github.com/brong/cyrus-imapd/commit/56d0823d3192ea13fa3afe397d625252b16252b9 > > There's a missing sizeof() in there meaning that if you have > 25 quotas > in > a dir (or > 12 on a 64-bit build), you'll have memory corruption occuring. > > Here's the patch to fix: > > http://github.com/brong/cyrus-imapd/commit/1de9d758aeb360714236388c4e1689db0522c21e Maybe it's just the lack of caffeine in the early morning, but, isn't there a way to simply _download_ a single diff from github.com? Thanks for any insights. Simon From simon.matter at invoca.ch Wed Jan 14 02:22:29 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Wed, 14 Jan 2009 08:22:29 +0100 (CET) Subject: migrate from 64 to 32 bits In-Reply-To: <1231915555.30300.57.camel@darkstar.netcore.co.in> References: <002f01c97562$2f0ff6f0$8d2fe4d0$@dewal@computel.nl> <1231915555.30300.57.camel@darkstar.netcore.co.in> Message-ID: <93d163f22dba982881e23fd7b13f30f8.squirrel@webmail.bi.corp.invoca.ch> > > On Tue, 2009-01-13 at 10:34 +0100, s.dewal at computel.nl wrote: >> Hi, >> >> A customer of outs wishes to migrate cyrus and postfix from a 64 bits to >> a >> 32 bits server. Normally I just copy yhe configuration files and several >> directories ( /var/lib/imap and /var/spool/imap) but the question is, >> are >> the file compatible in a 32 bits environment when copied from a 64 bits >> environment? >> >> Regards, >> > How strange , I had a problem doing right the opposite. ( from 32 to > 64 ) > > We migrated the mailboxes , from 32 bit to 64 bit , and all the seen > flags got messed up. I had to run imapsync to get all the seen flags set > up Interesting. Did you have the same version of Cyrus-IMAPd on both servers? I recently did a migration of 2.3.13 from RHEL3/i386 to RHEL5/x86_64 without any such issues. So, I'm really wondering why it sometimes works and for others it does not. Simon From boutilpj at ednet.ns.ca Wed Jan 14 05:45:31 2009 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Wed, 14 Jan 2009 06:45:31 -0400 Subject: migrate from 64 to 32 bits In-Reply-To: <93d163f22dba982881e23fd7b13f30f8.squirrel@webmail.bi.corp.invoca.ch> References: <002f01c97562$2f0ff6f0$8d2fe4d0$@dewal@computel.nl> <1231915555.30300.57.camel@darkstar.netcore.co.in> <93d163f22dba982881e23fd7b13f30f8.squirrel@webmail.bi.corp.invoca.ch> Message-ID: Simon Matter wrote: >> On Tue, 2009-01-13 at 10:34 +0100, s.dewal at computel.nl wrote: >>> Hi, >>> >>> A customer of outs wishes to migrate cyrus and postfix from a 64 bits to >>> a >>> 32 bits server. Normally I just copy yhe configuration files and several >>> directories ( /var/lib/imap and /var/spool/imap) but the question is, >>> are >>> the file compatible in a 32 bits environment when copied from a 64 bits >>> environment? >>> >>> Regards, >>> >> How strange , I had a problem doing right the opposite. ( from 32 to >> 64 ) >> >> We migrated the mailboxes , from 32 bit to 64 bit , and all the seen >> flags got messed up. I had to run imapsync to get all the seen flags set >> up > > Interesting. Did you have the same version of Cyrus-IMAPd on both servers? > I recently did a migration of 2.3.13 from RHEL3/i386 to RHEL5/x86_64 > without any such issues. So, I'm really wondering why it sometimes works > and for others it does not. We went from Fedora Core 2 (32 bit) to CentOS 5.2 (64 bit) without any issues. Both were running Cyrus 2.3.x. > > Simon > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html From brong at fastmail.fm Wed Jan 14 07:29:59 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 14 Jan 2009 23:29:59 +1100 Subject: Elusive replication bug in 2.3.13 In-Reply-To: <47cadc79b48df3ce5780e2598a378e55.squirrel@webmail.bi.corp.invoca.ch> References: <20090112164709.GA1417@helsinki.fi> <8C506188988F4A159973A4661E0955C9@Atticus> <47cadc79b48df3ce5780e2598a378e55.squirrel@webmail.bi.corp.invoca.ch> Message-ID: <20090114122959.GA18511@brong.net> On Wed, Jan 14, 2009 at 07:49:40AM +0100, Simon Matter wrote: > > http://github.com/brong/cyrus-imapd/commit/1de9d758aeb360714236388c4e1689db0522c21e > > Maybe it's just the lack of caffeine in the early morning, but, isn't > there a way to simply _download_ a single diff from github.com? > Thanks for any insights. Man, I see your point. They don't seem to have a "get this as a plain diff file". How rude. I've attached it. Pretty easy within git. Bron ( still enjoying all the other stuff github gives you! ) -------------- next part -------------- A non-text attachment was scrubbed... Name: quotafix.diff Type: text/x-diff Size: 1132 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090114/291718bc/attachment.bin From murch at andrew.cmu.edu Wed Jan 14 10:55:04 2009 From: murch at andrew.cmu.edu (Ken Murchison) Date: Wed, 14 Jan 2009 10:55:04 -0500 Subject: Elusive replication bug in 2.3.13 In-Reply-To: <20090114122959.GA18511@brong.net> References: <20090112164709.GA1417@helsinki.fi> <8C506188988F4A159973A4661E0955C9@Atticus> <47cadc79b48df3ce5780e2598a378e55.squirrel@webmail.bi.corp.invoca.ch> <20090114122959.GA18511@brong.net> Message-ID: <496E0AD8.1000200@andrew.cmu.edu> Committed to CVS. Thnaks! Bron Gondwana wrote: > On Wed, Jan 14, 2009 at 07:49:40AM +0100, Simon Matter wrote: >>> http://github.com/brong/cyrus-imapd/commit/1de9d758aeb360714236388c4e1689db0522c21e >> Maybe it's just the lack of caffeine in the early morning, but, isn't >> there a way to simply _download_ a single diff from github.com? >> Thanks for any insights. > > Man, I see your point. They don't seem to have a "get this as a plain > diff file". How rude. > > I've attached it. Pretty easy within git. > > Bron ( still enjoying all the other stuff github gives you! ) > -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University From kbah at linuxmail.org Wed Jan 14 13:43:14 2009 From: kbah at linuxmail.org (k bah) Date: Wed, 14 Jan 2009 15:43:14 -0300 Subject: Auto set quota to specific folder thru imapd.conf/cyrus.conf ?? Message-ID: <20090114184314.131FB4446B@ws5-1.us4.outblaze.com> cyrus-imapd-2.3.8-51 cyrus-sasl-2.1.22-82 -- user/someuser~domain.com (\HasChildren) user/someuser~domain.com/Drafts (\HasNoChildren) user/someuser~domain.com/INBOX (\HasNoChildren) user/someuser~domain.com/2008 Messages (\HasNoChildren) user/someuser~domain.com/Sent (\HasNoChildren) user/someuser~domain.com/Spam (\HasNoChildren) user/someuser~domain.com/Trash (\HasNoChildren) -- Hi, Is it possible to set a specific quota for a folder by editing cyrus.conf or imapd.conf? I already set a quota there for INBOX, but I need to auto set a quota for another folder, Trash, the ideal would be to set Trash with an unlimited quota, since the only supported IMAP client my users have access to is Webmail, and it automatically deletes all messages from the Trash folder monthly. I could run some script to set a higher quota on user/myuser~domain.com/Trash (calling cyradm), but that would be my last option. That way, new users would have Trash with a higher quota, not affecting their INBOX % usage. If possible I want to do that (unlimit the quota for Trash, keeping the default quota for INBOX and subfolders) by editing imapd.conf or some other config file. thanks in advance = -- Powered by Outblaze From kbah at linuxmail.org Wed Jan 14 13:51:22 2009 From: kbah at linuxmail.org (k bah) Date: Wed, 14 Jan 2009 15:51:22 -0300 Subject: Quota on subfolders question. Message-ID: <20090114185122.591E04446B@ws5-1.us4.outblaze.com> cyrus-imapd-2.3.8 cyrus-sasl-2.1.22 -- user/someuser~domain.com (\HasChildren) user/someuser~domain.com/Drafts (\HasNoChildren) user/someuser~domain.com/INBOX (\HasNoChildren) user/someuser~domain.com/2008 Messages (\HasNoChildren) user/someuser~domain.com/Sent (\HasNoChildren) user/someuser~domain.com/Spam (\HasNoChildren) user/someuser~domain.com/Trash (\HasNoChildren) -- Hi, The above folder/subfolder structure is how my users mailboxes are now. Q1) If INBOX has a 50Mb quota (autocreatequota on imapd.conf) and the other subfolders do not have a quota configured via cyradm, all of them respect the 50Mb limit, right? The sum of all of them cannot be more than 50Mb. Q2) If I set a quota higher than 50Mb to any of those subfolders, say 60Mb for Trash, then can the user move messages to the Trash folder until they sum 55Mb (all messages on Trash sum 55Mb), *without* affecting the INBOX limit? I mean, the user will still be able to receive messages if Trash has 55Mb? thanks for you time = Slatwall - Guaranteed Low Prices Save Money. Buy Slatwall Direct From The Manufacturer. Ships Same Day. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=c33f3e7389ca71081776d3da9be71f6a -- Powered by Outblaze From baconm at email.unc.edu Wed Jan 14 14:44:28 2009 From: baconm at email.unc.edu (Michael Bacon) Date: Wed, 14 Jan 2009 14:44:28 -0500 Subject: 32 to 64 bit -- murder and replication Message-ID: <1393523B1D6CEC26210BD41E@ITSBACONM.depts.unc.edu> Hi all, To throw another 32/64 question on the fire here, I'm hoping to upgrade the ancient 1.6 (yes, I know!) install here to 2.3, and then we're going to use the murder to help us break out of the single giant 6800 we have and move to the distributed model. However, for a time, the 32 bit code on the old server is going to need to play nicely with the 64 bit code on the other servers, both front-end and back-end. We'd also like to get replication into the mix here at some point, and may have the same split. Everything will be running the same code, but it will be a different build, naturally (the old housed on Solaris 9, the new on Solaris 10). Can anyone think of any nastiness we might run into along the way? Ram's warning about the seen flags means I'll probably dump the seen databases before migration just to be safe, but I think we can handle migration issues. It's the ongoing interop issues I'm worried about. Thanks, Michael From morgan at orst.edu Wed Jan 14 19:42:04 2009 From: morgan at orst.edu (Andrew Morgan) Date: Wed, 14 Jan 2009 16:42:04 -0800 (PST) Subject: Doing Cyrus *right* for a large installation Message-ID: I have an existing Cyrus installation consisting of: (2) backend servers, each with 800GB of SAN storage, 1 cyrus partition (3) frontend servers (1) mupdate master Obviously, we are running Cyrus as a murder cluster and we really like that. I'm interested in upgrading both the servers and the storage for Cyrus, but I need some advice. I would like to add a *lot* more storage so that we can increase our email quotas (currently 200MB per user). It seems like the proper way to scale up is to split the Cyrus metadata off and use some large SATA drives for the message files. I was considering adding a shelf of 1TB SATA drives to our SAN. I could store the metadata on existing FC drives on the SAN, or just use internal disks on the servers. But then I started thinking about how I was going to backup all this new data... Our backup administrator isn't too excited about trying to backup 12TB of email data. What if we used Cyrus replication in combination with delayed expunge as a form of "backup"? We currently only keep 1 month of daily backups anyways... For those of you running large installations, how do you back them up? Thanks, Andy From simon.matter at invoca.ch Thu Jan 15 01:49:33 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Thu, 15 Jan 2009 07:49:33 +0100 (CET) Subject: Doing Cyrus *right* for a large installation In-Reply-To: References: Message-ID: > But then I started thinking about how I was going to backup all this new > data... Our backup administrator isn't too excited about trying to backup > 12TB of email data. > > What if we used Cyrus replication in combination with delayed expunge as a > form of "backup"? We currently only keep 1 month of daily backups > anyways... > > For those of you running large installations, how do you back them up? I don't have a really large installation anymore but what I still do is: 1) use 'expunge_mode: delayed' and 'delete_mode: delayed' (a nice side effect is increase of performance on certain operations). (expunge/delete period is 7 days in our case) 2) use rsnapshot to replicate all cyrus data to a separate storage and maintain some history while still saving space. (daily in our case) 3) create offsite backup of the rsnapshot storage. (weekly in our case) That's our poor mans solution we use here, and it is very KISS. Other ideas may include using advanced filesystems like ZFS or build something around DRBD, maybe all in combination with deduplication (something like using a Datadomain box instead of rsnapshot). I'm also interested to hear what others do. Regards, Simon From Pascal.Gienger at uni-konstanz.de Thu Jan 15 02:01:56 2009 From: Pascal.Gienger at uni-konstanz.de (Pascal Gienger) Date: Thu, 15 Jan 2009 08:01:56 +0100 Subject: Doing Cyrus *right* for a large installation In-Reply-To: References: Message-ID: <496EDF64.10407@uni-konstanz.de> Andrew Morgan wrote: > But then I started thinking about how I was going to backup all this new > data... Our backup administrator isn't too excited about trying to backup > 12TB of email data. Just a side note: For most backup systems it is not the size which matters much (because only a small portion of data gets changed each day on a typical IMAP storage) it is the *NUMBER* of files, because - for an incremental backup - each file stat data has to be read and compared. With ZFS (gzip compression activated), we get 31 million files in 3-4 hours (example with tivoli storage manager): 01/03/09 23:38:48 --- SCHEDULEREC STATUS BEGIN 01/03/09 23:38:48 Total number of objects inspected: 31,305,555 01/03/09 23:38:48 Total number of objects backed up: 25,347 01/03/09 23:38:48 Total number of objects updated: 0 01/03/09 23:38:48 Total number of objects rebound: 0 01/03/09 23:38:48 Total number of objects deleted: 0 01/03/09 23:38:48 Total number of objects expired: 308,991 01/03/09 23:38:48 Total number of objects failed: 0 01/03/09 23:38:48 Total number of bytes transferred: 2.56 GB 01/03/09 23:38:48 Data transfer time: 280.87 sec 01/03/09 23:38:48 Network data transfer rate: 9,594.01 KB/sec 01/03/09 23:38:48 Aggregate data transfer rate: 209.05 KB/sec 01/03/09 23:38:48 Objects compressed by: 15% 01/03/09 23:38:48 Elapsed processing time: 03:34:49 01/03/09 23:38:48 --- SCHEDULEREC STATUS END 01/03/09 23:38:48 --- SCHEDULEREC OBJECT END SA_SO_MAIL 01/03/09 20:00:00 01/03/09 23:38:48 Executing Operating System command or script: /mail/bin/pr_backupsnapshot_off Just to give you a "real life" number... Pascal From Rudy.Gevaert at UGent.be Thu Jan 15 02:57:49 2009 From: Rudy.Gevaert at UGent.be (Rudy Gevaert) Date: Thu, 15 Jan 2009 08:57:49 +0100 Subject: Doing Cyrus *right* for a large installation In-Reply-To: References: Message-ID: <496EEC7D.9030804@UGent.be> Andrew Morgan wrote: > For those of you running large installations, how do you back them up? We have 7 backends: 400G 320G 81G 80% /mail/mail1 400G 273G 128G 69% /mail/mail2 450G 390G 61G 87% /mail/mail3 400G 322G 79G 81% /mail/mail4 400G 270G 131G 68% /mail/mail5 450G 397G 54G 89% /mail/mail6 50G 4.8G 46G 10% /mail/mail7 For mail1-6 incremental backup time takes between 2h and 7h, we suspect a storage (misconfiguration) issue. Fulls take a lot longer, but I can't give you any time estimates now. For the moment we don't backup the replica's. Nor use them. But we could do it. We are 'planning' a storage migration too the coming months. Rudy -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Rudy Gevaert Rudy.Gevaert at UGent.be tel:+32 9 264 4734 Directie ICT, afd. Infrastructuur ICT Department, Infrastructure office Groep Systemen Systems group Universiteit Gent Ghent University Krijgslaan 281, gebouw S9, 9000 Gent, Belgie www.UGent.be -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- From morgan at orst.edu Thu Jan 15 15:04:29 2009 From: morgan at orst.edu (Andrew Morgan) Date: Thu, 15 Jan 2009 12:04:29 -0800 (PST) Subject: Doing Cyrus *right* for a large installation In-Reply-To: References: Message-ID: On Thu, 15 Jan 2009, Simon Matter wrote: >> But then I started thinking about how I was going to backup all this new >> data... Our backup administrator isn't too excited about trying to backup >> 12TB of email data. >> >> What if we used Cyrus replication in combination with delayed expunge as a >> form of "backup"? We currently only keep 1 month of daily backups >> anyways... >> >> For those of you running large installations, how do you back them up? > > I don't have a really large installation anymore but what I still do is: > 1) use 'expunge_mode: delayed' and 'delete_mode: delayed' (a nice side > effect is increase of performance on certain operations). (expunge/delete > period is 7 days in our case) > 2) use rsnapshot to replicate all cyrus data to a separate storage and > maintain some history while still saving space. (daily in our case) > 3) create offsite backup of the rsnapshot storage. (weekly in our case) > > That's our poor mans solution we use here, and it is very KISS. Other > ideas may include using advanced filesystems like ZFS or build something > around DRBD, maybe all in combination with deduplication (something like > using a Datadomain box instead of rsnapshot). > > I'm also interested to hear what others do. rsnapshot, neat! I haven't seen that until now. If I'm reading this right, rsnapshot works like ZFS snapshots, where the only "extra" data in each snapshot is the data that has changed? This kinda goes back to a discussion last week about filesystems. I think I would like ZFS, but I'm not sure I want to switch our Cyrus stuff over to Solaris. We run a fair amount of Solaris on SPARC here, but no Solaris x86. Unfortunately the SPARC hardware is not as cheap as x86. I was pondering using delayed expunge and delayed delete with a 30 day period, and just replicating (Cyrus replication) to another data center. Hmm, I should try to figure out how much overhead it would be to keep 30 days of delayed expunge/delete... Andy From morgan at orst.edu Thu Jan 15 15:08:24 2009 From: morgan at orst.edu (Andrew Morgan) Date: Thu, 15 Jan 2009 12:08:24 -0800 (PST) Subject: Doing Cyrus *right* for a large installation In-Reply-To: <496EEC7D.9030804@UGent.be> References: <496EEC7D.9030804@UGent.be> Message-ID: On Thu, 15 Jan 2009, Rudy Gevaert wrote: > Andrew Morgan wrote: > >> For those of you running large installations, how do you back them up? > > We have 7 backends: > 400G 320G 81G 80% /mail/mail1 > 400G 273G 128G 69% /mail/mail2 > 450G 390G 61G 87% /mail/mail3 > 400G 322G 79G 81% /mail/mail4 > 400G 270G 131G 68% /mail/mail5 > 450G 397G 54G 89% /mail/mail6 > 50G 4.8G 46G 10% /mail/mail7 > > > For mail1-6 incremental backup time takes between 2h and 7h, we suspect > a storage (misconfiguration) issue. > > Fulls take a lot longer, but I can't give you any time estimates now. Our incrementals take about 2 hours (both backends are done at the same time). Our fulls take an entire day, but we only do fulls once a month. I'm hoping that using smaller partitions will let us do backups in shorter times, assuming we stay with tape-backups. > For the moment we don't backup the replica's. Nor use them. But we > could do it. So you are using Cyrus Murder with replication? How does that work? > We are 'planning' a storage migration too the coming months. I'm curious what storage hardware you are exploring. What are you using now for your 7 backends? Thanks, Andy From simon.matter at invoca.ch Thu Jan 15 17:40:06 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Thu, 15 Jan 2009 23:40:06 +0100 (CET) Subject: Doing Cyrus *right* for a large installation In-Reply-To: References: Message-ID: > On Thu, 15 Jan 2009, Simon Matter wrote: > >>> But then I started thinking about how I was going to backup all this >>> new >>> data... Our backup administrator isn't too excited about trying to >>> backup >>> 12TB of email data. >>> >>> What if we used Cyrus replication in combination with delayed expunge >>> as a >>> form of "backup"? We currently only keep 1 month of daily backups >>> anyways... >>> >>> For those of you running large installations, how do you back them up? >> >> I don't have a really large installation anymore but what I still do is: >> 1) use 'expunge_mode: delayed' and 'delete_mode: delayed' (a nice side >> effect is increase of performance on certain operations). >> (expunge/delete >> period is 7 days in our case) >> 2) use rsnapshot to replicate all cyrus data to a separate storage and >> maintain some history while still saving space. (daily in our case) >> 3) create offsite backup of the rsnapshot storage. (weekly in our case) >> >> That's our poor mans solution we use here, and it is very KISS. Other >> ideas may include using advanced filesystems like ZFS or build something >> around DRBD, maybe all in combination with deduplication (something like >> using a Datadomain box instead of rsnapshot). >> >> I'm also interested to hear what others do. > > rsnapshot, neat! I haven't seen that until now. If I'm reading this > right, rsnapshot works like ZFS snapshots, where the only "extra" data in > each snapshot is the data that has changed? I don't know the details of ZFS but I guess it works on the block level but not like most LVM on raw device blocks but on the filesystem level. Rsnapshot works completely on the file level. It's nice for mail spools because the payload data doesn't change and that means those file only exist once and all other instances are hardlinks. It does not work well if mails are moved around, of course, but usually that happens only once and then they stay there forever. We are using rsnapshot for all our servers and it really works very well. We have hit different kernel issues over the years (the first box ran RedHat 9) but finally things look very stable with RHEL5.2. > > This kinda goes back to a discussion last week about filesystems. I think > I would like ZFS, but I'm not sure I want to switch our Cyrus stuff over > to Solaris. We run a fair amount of Solaris on SPARC here, but no Solaris > x86. Unfortunately the SPARC hardware is not as cheap as x86. > > I was pondering using delayed expunge and delayed delete with a 30 day > period, and just replicating (Cyrus replication) to another data center. > > Hmm, I should try to figure out how much overhead it would be to keep 30 > days of delayed expunge/delete... We are using 7 days for delayed expunge/delete and I don't remember any proplems with it. I have never tried 30 days. Simon From baconm at email.unc.edu Thu Jan 15 18:04:35 2009 From: baconm at email.unc.edu (Michael Bacon) Date: Thu, 15 Jan 2009 18:04:35 -0500 Subject: Possible bug in ctl_mboxlist -u (or perhaps I'm just crazy) Message-ID: Hi, all, In the "bugs that increasingly small number of people are likely to encounter" department, I've run into this bug on our upgrade test box on the path out of the dark ages, a.k.a. between 1.6 and 2.3. The 1.6 install uses the flatfile format, and did not include the mbtype field. If viewed with a less -U, it nicely shows all the fields delimited with a tab (^I) character: user.baconm^I112^Ibaconm^Ilrswipcda^I user.baconm.bedeworks-users^I112^Ibaconm^Ilrswipcda^I user.baconm.info-cyrus^I112^Ibaconm^Ilrswipcda^I The upgrade instructions (which are cumulative and ancient, I realize) indicate that a ctl_mboxlist -u on a flatfile mailboxes database should be enough. However, this doesn't incorporate the adding of the mbtype field. This would be Bug #1 as I see it. (or something that just requires an additional step in the upgrade document, which admittedly is not a step many people will ever go through again.) Getting around this is a little tricky, because simply adding an additional tab-delimited field with 0 in it right after the mailbox name does not work. Dumping the mailboxes database from a legitimate (I think) skiplist file gets me something that looks like this (with less -U, which again is turning tabs into ^I): user.baconm^I0 112 baconm^Ilrswipcda^I user.baconm.bedeworks-users^I0 112 baconm^Ilrswipcda^I user.baconm.info-cyrus^I0 112 baconm^Ilrswipcda^I The partition here is "112". Now, at first glance, this seems pretty wacky, to have the fields delimited with an odd mix of spaces and tabs, but hey, whatever works. I'm pretty sure this isn't just my system being buggy, because the code is pretty explicit about what's doing (imap/mboxlist.c): char *mboxlist_makeentry(int mbtype, const char *part, const char *acl) { char *mboxent = (char *) xmalloc(sizeof(char) * (30 + strlen(acl) + strlen(part))); sprintf(mboxent, "%d %s %s", mbtype, part, acl); return mboxent; } The workaround I think I've developed is to just run the old mailboxes file through the following perl snippet before undumping it: perl -ane 's/\t(\w+)\t/\t0 $1 /; print' That replaces what was ^I^I^I^I... with ^I ^I^I... (and always uses "0" for the mbtype, which as I understand it is proper for an old mailboxes database) It appears to work on my test system, but this is just strange enough that I want to double-check with the kind folks here to make sure this is proper and sane before I go and run this on a single 1.6 install with 80k users and 800k mailboxes. So, if this is actually the right thing to be doing, please let me know, and if I'm doing something horribly broken, I REALLY want to know. Thanks, y'all, Michael Bacon ITS Messaging UNC Chapel Hill From brong at fastmail.fm Thu Jan 15 18:47:34 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 16 Jan 2009 10:47:34 +1100 Subject: Doing Cyrus *right* for a large installation In-Reply-To: References: Message-ID: <20090115234734.GA13760@brong.net> On Wed, Jan 14, 2009 at 04:42:04PM -0800, Andrew Morgan wrote: > But then I started thinking about how I was going to backup all this new > data... Our backup administrator isn't too excited about trying to backup > 12TB of email data. > > What if we used Cyrus replication in combination with delayed expunge as a > form of "backup"? We currently only keep 1 month of daily backups > anyways... We don't trust that enough as an only solution - in particular, user deletes will replicate immediately. Delayed delete as well as delayed expunge gets you a lot closer. But we still want real backups that are offline. > For those of you running large installations, how do you back them up? We use a custom set of libraries and scripts that compress everything into a tar.gz file. The file contains actual emails named by their sha1, index, expunge and header files named by the mailbox uniqueid, sieve scripts, seen and subs files, and symlinks from the actual folder names to to the uniqueid directories. This allows us to recreate the original mailboxes precisely, but also copes with message copies between folders (just store the new cyrus.index file, we already have the sha1 file in backup, so no need to re-fetch), folder renames (just replace the symlink). Because we just append to the tar file, a folder rename is actually two files, the new symlink and a symlink with the old name pointing to an empty string. Every so often (starting at 80% live and going up 1% per week, so guaranteed to run every 20 weeks) we re-compress the backup, by zcatting the tar file through a decider function which either pipes on the file to the output or eats it - meaning only "alive" items get kept. Even a user who never deletes anything will have stale old copies of their cyrus.index files, because we take one snapshot per day. ---- This is supported by a server on each imap machine which provides direct access to all the files. It uses fcntl locking so that it will block while Cyrus is using the files, and so that Cyrus will block while it is working. The meta directories are handled in two passes. The first pass is just a stat pass, where we ask for a stat(2) on each file. If ANY of them have changed, we come back and lock all the files in the directory, before streaming them all to the backup tool. That way we're guaranteed consistency. Data files we just fetch and check the sha1 on the result. They either fetch correctly or we abort with an error for the sysadmin (that would be me) to check and fix. It should never happen. The one inconsistency we do have is that seen files aren't locked for the entire time, so they might fall out of sync with mailboxes. I don't consider this a giant risk in practice. ---- I am in the process of tidying up and documenting all the modules that we use for processing backups. I'd like to publish the whole lot, perhaps even putting them inside the "perl" directory of the Cyrus distribution so they can be kept in sync with the rest of Cyrus. They do include pack/unpack re-implementations of the cyrus.index and cyrus.header formats. I've also got a perl implementation of the skiplist format that I should polish up and include. Bron. Bron. From doconnor at gsoft.com.au Thu Jan 15 18:52:22 2009 From: doconnor at gsoft.com.au (Daniel O'Connor) Date: Fri, 16 Jan 2009 10:22:22 +1030 Subject: Doing Cyrus *right* for a large installation In-Reply-To: References: Message-ID: On Friday 16 January 2009 06:34:29 Andrew Morgan wrote: > This kinda goes back to a discussion last week about filesystems. I think > I would like ZFS, but I'm not sure I want to switch our Cyrus stuff over > to Solaris. We run a fair amount of Solaris on SPARC here, but no Solaris > x86. Unfortunately the SPARC hardware is not as cheap as x86. Try OpenSolaris or FreeBSD :) -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090116/f34028c9/attachment.bin From wes at umich.edu Thu Jan 15 19:53:06 2009 From: wes at umich.edu (Wesley Craig) Date: Thu, 15 Jan 2009 19:53:06 -0500 Subject: Possible bug in ctl_mboxlist -u (or perhaps I'm just crazy) In-Reply-To: References: Message-ID: <6F56E678-E3B2-4A4C-8413-AC90FA05EFA6@umich.edu> Regarding the odd mix of tab vs spaces: https://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/imap/ ctl_mboxlist.c.diff?r1=1.56;r2=1.57 According to the change log, this was to be compatible with cyr_dbtool. However, on undump, it appears to accept space or tab. Regarding the additional flag: https://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/imap/ ctl_mboxlist.c.diff?r1=1.43.2.6;r2=1.43.2.7 The flag is optional on undump, defaulting to 0 (a local, active mailbox). If either of these backwards compatible features is tripping you up, then please file a bugzilla report: it should work. I do notice that since your partition names start with a digit, probably the backward compatibility code is insufficient... Hope this helps on your adventures navigating from 1.6 to 2.3! :wes On 15 Jan 2009, at 18:04, Michael Bacon wrote: > In the "bugs that increasingly small number of people are likely to > encounter" department, I've run into this bug on our upgrade test > box on > the path out of the dark ages, a.k.a. between 1.6 and 2.3. > > The 1.6 install uses the flatfile format, and did not include the > mbtype > field. If viewed with a less -U, it nicely shows all the fields > delimited > with a tab (^I) character: > > user.baconm^I112^Ibaconm^Ilrswipcda^I > user.baconm.bedeworks-users^I112^Ibaconm^Ilrswipcda^I > user.baconm.info-cyrus^I112^Ibaconm^Ilrswipcda^I > > The upgrade instructions (which are cumulative and ancient, I realize) > indicate that a ctl_mboxlist -u on a flatfile mailboxes database > should be > enough. However, this doesn't incorporate the adding of the mbtype > field. > This would be Bug #1 as I see it. (or something that just requires an > additional step in the upgrade document, which admittedly is not a > step > many people will ever go through again.) > > Getting around this is a little tricky, because simply adding an > additional > tab-delimited field with 0 in it right after the mailbox name does not > work. Dumping the mailboxes database from a legitimate (I think) > skiplist > file gets me something that looks like this (with less -U, which > again is > turning tabs into ^I): > > user.baconm^I0 112 baconm^Ilrswipcda^I > user.baconm.bedeworks-users^I0 112 baconm^Ilrswipcda^I > user.baconm.info-cyrus^I0 112 baconm^Ilrswipcda^I > > The partition here is "112". Now, at first glance, this seems pretty > wacky, to have the fields delimited with an odd mix of spaces and > tabs, but > hey, whatever works. I'm pretty sure this isn't just my system being > buggy, because the code is pretty explicit about what's doing > (imap/mboxlist.c): > > char *mboxlist_makeentry(int mbtype, const char *part, const char > *acl) > { > char *mboxent = (char *) xmalloc(sizeof(char) * > (30 + strlen(acl) + strlen > (part))); > sprintf(mboxent, "%d %s %s", mbtype, part, acl); > return mboxent; > } > > The workaround I think I've developed is to just run the old > mailboxes file > through the following perl snippet before undumping it: > > perl -ane 's/\t(\w+)\t/\t0 $1 /; print' > > That replaces what was > > ^I^I^I^I... > > with > > ^I ^I^I... > > (and always uses "0" for the mbtype, which as I understand it is > proper for > an old mailboxes database) > > It appears to work on my test system, but this is just strange > enough that > I want to double-check with the kind folks here to make sure this > is proper > and sane before I go and run this on a single 1.6 install with 80k > users > and 800k mailboxes. So, if this is actually the right thing to be > doing, > please let me know, and if I'm doing something horribly broken, I > REALLY > want to know. > > Thanks, y'all, > > Michael Bacon > ITS Messaging > UNC Chapel Hill > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > > !DSPAM:496fc1d3290211336712104! > > > From robm at fastmail.fm Thu Jan 15 18:49:19 2009 From: robm at fastmail.fm (Rob Mueller) Date: Fri, 16 Jan 2009 10:49:19 +1100 Subject: Doing Cyrus *right* for a large installation In-Reply-To: References: Message-ID: <0900EDD0E1B54BCDB36306BA0FBAC396@Atticus> > I would like to add a *lot* more storage so that we can increase our email > quotas (currently 200MB per user). It seems like the proper way to scale > up is to split the Cyrus metadata off and use some large SATA drives for > the message files. I was considering adding a shelf of 1TB SATA drives to > our SAN. I could store the metadata on existing FC drives on the SAN, or > just use internal disks on the servers. We split our meta/data onto 10k/15k RPM RAID1 for meta vs 7.2k RPM RAID5 for data. The meta is 1/20th the size of the data. The meta drives get more data written to them, the data drives get more data read (we have lots of memory now, so probably lots of meta is cached). On average, utilisation for meta is still higher than data, but they're relatively well balanced with that split it seems. > But then I started thinking about how I was going to backup all this new > data... Our backup administrator isn't too excited about trying to backup > 12TB of email data. We backup to a X4500 server. Bron built our custom backup system for cyrus. Each cyrus machine has a backup daemon that has a simple network protocol. The daemon knows where meta and data files are, and can read and understand cyrus.* files. A backup process on the X4500 runs each day and connects to the daemon on each cyrus machine and uses it to find out changes for each user for each folder and updates the backup on the X4500. All backups are stored in .tgz streams with a copy of every email, every cyrus.* file. Meta data is stored in an sqlite file. In general the backup process just appends to the .tgz stream. When it calculates that the ratio of "old" data in the .tgz is too high, it re-packs the whole thing removing the old data. The whole thing relies a lot on internal knowledge of our setup, so it's not something we can easily release unfortunately. > What if we used Cyrus replication in combination with delayed expunge as a > form of "backup"? We currently only keep 1 month of daily backups > anyways... It's an option, but it's a bit scary still. What if there's a replication protocol error that blows away your replica? Unlikely, but possible. I think we might be a bit paranoid. We don't like relying on any one thing. Filesystems, software, hardware, etc. Net result is we've ended up with quite a few levels of redundancy. 1. All data on RAID so any HD failure is just a replacement HD no downtime at all 2. Delayed delete, so any user deletion error can be fixed by re-inserting the deleted messages 3. All data replicated, so any server/storage unit failure is just switching master/replica 4. Nightly backups to a completely separate server, with different OS and filesystem, and with no shared credentials or trust. Basically a last resort in case of major hardware/OS/security screw up that you absolutely hope you never have to use. Rob From jblaine at kickflop.net Fri Jan 16 06:01:48 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Fri, 16 Jan 2009 06:01:48 -0500 Subject: 2.2.12 to 2.3.13 upgrade : Message-ID: <4970691C.70900@kickflop.net> Our happy and functioning 2.2.12 server setup shows the following when switched over to 2.3.13. The odd thing is that we're using skiplist, not berkeley, for our dbs. If anyone has comments, I'd love to hear them. bash-2.05# cat /etc/imapd.conf configdirectory: /var/imap defaultpartition: default partition-default: /var/spool/imap imap_admins: root cyrus sasl_pwcheck_method: saslauthd sieveusehomedir: true autocreatequota: 200000 duplicate_db: skiplist tlscache_db: skiplist annotation_db: skiplist tls_cert_file: /var/imap/ca.crt tls_key_file: /var/imap/server.pem tls_ca_file: /var/imap/ca.crt bash-2.05# # /etc/init.d/imap stop Cyrus: Shutting dowm imapd instances Cyrus: Shutting down master process # /etc/init.d/imap start ... Jan 16 05:25:15 imapsrv ctl_cyrusdb[13271]: [ID 866726 local6.warning] DBERROR db4: Program version 4.3 doesn't match environment version Jan 16 05:25:15 imapsrv ctl_cyrusdb[13271]: [ID 866726 local6.warning] DBERROR db4: Skipping log file /var/imap/db/log.0000000224: historic log version 7 Jan 16 05:25:16 imapsrv ctl_cyrusdb[13271]: [ID 866726 local6.warning] DBERROR db4: /var/imap/db/log.0000000225: log file open failed: No such file or directory Jan 16 05:25:16 imapsrv ctl_cyrusdb[13271]: [ID 866726 local6.warning] DBERROR db4: PANIC: No such file or directory Jan 16 05:25:16 imapsrv ctl_cyrusdb[13272]: [ID 866726 local6.warning] DBERROR db4: PANIC: fatal region error detected; run recovery From Hagedorn at uni-koeln.de Fri Jan 16 06:15:48 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Fri, 16 Jan 2009 12:15:48 +0100 Subject: 2.2.12 to 2.3.13 upgrade : In-Reply-To: <4970691C.70900@kickflop.net> References: <4970691C.70900@kickflop.net> Message-ID: <7CF41FD1BD9D923A95C99081@tyrion.rrz.uni-koeln.de> Hello Jeff, --On 16. Januar 2009 06:01:48 -0500 Jeff Blaine wrote: > Our happy and functioning 2.2.12 server setup shows the > following when switched over to 2.3.13. The odd thing is > that we're using skiplist, not berkeley, for our dbs. > > If anyone has comments, I'd love to hear them. > > bash-2.05# cat /etc/imapd.conf > configdirectory: /var/imap > defaultpartition: default > partition-default: /var/spool/imap > imap_admins: root cyrus > sasl_pwcheck_method: saslauthd > sieveusehomedir: true > autocreatequota: 200000 > duplicate_db: skiplist > tlscache_db: skiplist > annotation_db: skiplist > tls_cert_file: /var/imap/ca.crt > tls_key_file: /var/imap/server.pem > tls_ca_file: /var/imap/ca.crt > bash-2.05# what about mboxlist_db? And even if all your dbs *are* skiplist, you might still have remants from previous Berkeley dbs in your /var/lib/imap/db directory. If you are sure you don't need those, just delete them. Obviously the new Cyrus has found a Berkeley environment somewhere ... -- .:.Sebastian Hagedorn - RZKR-R1 (Geb?ude 52), Zimmer 18.:. Zentrum f?r angewandte Informatik - Universit?tsweiter Service RRZK .:.Universit?t zu K?ln / Cologne University - ? +49-221-478-5587.:. .:.:.:.Skype: shagedorn.:.:.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090116/3c8e9020/attachment.bin From jblaine at kickflop.net Fri Jan 16 06:26:48 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Fri, 16 Jan 2009 06:26:48 -0500 Subject: 2.2.12 to 2.3.13 upgrade : In-Reply-To: <7CF41FD1BD9D923A95C99081@tyrion.rrz.uni-koeln.de> References: <4970691C.70900@kickflop.net> <7CF41FD1BD9D923A95C99081@tyrion.rrz.uni-koeln.de> Message-ID: <49706EF8.4080905@kickflop.net> Sebastian Hagedorn wrote: > Hello Jeff, > > --On 16. Januar 2009 06:01:48 -0500 Jeff Blaine > wrote: > >> Our happy and functioning 2.2.12 server setup shows the >> following when switched over to 2.3.13. The odd thing is >> that we're using skiplist, not berkeley, for our dbs. >> >> If anyone has comments, I'd love to hear them. >> >> bash-2.05# cat /etc/imapd.conf >> configdirectory: /var/imap >> defaultpartition: default >> partition-default: /var/spool/imap >> imap_admins: root cyrus >> sasl_pwcheck_method: saslauthd >> sieveusehomedir: true >> autocreatequota: 200000 >> duplicate_db: skiplist >> tlscache_db: skiplist >> annotation_db: skiplist >> tls_cert_file: /var/imap/ca.crt >> tls_key_file: /var/imap/server.pem >> tls_ca_file: /var/imap/ca.crt >> bash-2.05# > > what about mboxlist_db? And even if all your dbs *are* skiplist, you > might still have remants from previous Berkeley dbs in your > /var/lib/imap/db directory. If you are sure you don't need those, just > delete them. Obviously the new Cyrus has found a Berkeley environment > somewhere ... Thanks for the reply, Sebastian. I found a log.00000001 from 2005 in the db directory. When I deleted it, the new 2.3.13 instance started cleanly (so far). Thanks for the nudge in the right direction. From jblaine at kickflop.net Fri Jan 16 06:38:27 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Fri, 16 Jan 2009 06:38:27 -0500 Subject: Expire (manually) TLS sessions? Message-ID: <497071B3.1010006@kickflop.net> Maybe we're doing something wrong in the process, but it seems that every time we perform offline maintenance (upgrade, whatever) on Cyrus IMAPd ... our users complain that TLS breaks afterward, but then fixes itself in time. I've demonstrated this to myself just now with the upgrade to 2.3.13 from 2.2.12. My TLS session is cached but broken with the new setup (or for whatever other reason). That is, even after restarting Thunderbird, I get the following: Jan 16 06:31:50 imapsrv imap[19690]: [ID 239158 local6.notice] STARTTLS negotiation failed: bva-172.our.com Is there a way to zero/flush all TLS cached sessions? I have to imagine there is, but I don't know how. From Hagedorn at uni-koeln.de Fri Jan 16 06:46:32 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Fri, 16 Jan 2009 12:46:32 +0100 Subject: Expire (manually) TLS sessions? In-Reply-To: <497071B3.1010006@kickflop.net> References: <497071B3.1010006@kickflop.net> Message-ID: <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> Hello Jeff, --On 16. Januar 2009 06:38:27 -0500 Jeff Blaine wrote: > Maybe we're doing something wrong in the process, but it > seems that every time we perform offline maintenance > (upgrade, whatever) on Cyrus IMAPd ... our users complain > that TLS breaks afterward, but then fixes itself in time. > > I've demonstrated this to myself just now with the upgrade > to 2.3.13 from 2.2.12. My TLS session is cached but broken > with the new setup (or for whatever other reason). That is, > even after restarting Thunderbird, I get the following: > > Jan 16 06:31:50 imapsrv imap[19690]: [ID 239158 local6.notice] STARTTLS > negotiation failed: bva-172.our.com > > Is there a way to zero/flush all TLS cached sessions? I > have to imagine there is, but I don't know how. as before: just delete the tls_sessions files before you start cyrus-imapd. They will be recreated automatically. You could even make that part of the initscript, because those session don't survive a restart anyway. -- .:.Sebastian Hagedorn - RZKR-R1 (Geb?ude 52), Zimmer 18.:. Zentrum f?r angewandte Informatik - Universit?tsweiter Service RRZK .:.Universit?t zu K?ln / Cologne University - ? +49-221-478-5587.:. .:.:.:.Skype: shagedorn.:.:.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090116/45b340e4/attachment.bin From jblaine at kickflop.net Fri Jan 16 06:58:06 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Fri, 16 Jan 2009 06:58:06 -0500 Subject: Expire (manually) TLS sessions? In-Reply-To: <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> Message-ID: <4970764E.5080209@kickflop.net> Hmm. That's not working for me. bash-2.05# pwd /var/imap # rm tls_sessions.db # /etc/init.d/imap start # Jan 16 06:54:36 imapsrv imap[20300]: [ID 286863 local6.notice] imapd:Loading hard-coded DH parameters Jan 16 06:54:36 imapsrv imap[20300]: [ID 239158 local6.notice] STARTTLS negotiation failed: bva-172.our.com bash-2.05# pwd /var/imap bash-2.05# ls -l tls* tls*: No such file or directory bash-2.05# Sebastian Hagedorn wrote: > Hello Jeff, > > --On 16. Januar 2009 06:38:27 -0500 Jeff Blaine > wrote: > >> Maybe we're doing something wrong in the process, but it >> seems that every time we perform offline maintenance >> (upgrade, whatever) on Cyrus IMAPd ... our users complain >> that TLS breaks afterward, but then fixes itself in time. >> >> I've demonstrated this to myself just now with the upgrade >> to 2.3.13 from 2.2.12. My TLS session is cached but broken >> with the new setup (or for whatever other reason). That is, >> even after restarting Thunderbird, I get the following: >> >> Jan 16 06:31:50 imapsrv imap[19690]: [ID 239158 local6.notice] STARTTLS >> negotiation failed: bva-172.our.com >> >> Is there a way to zero/flush all TLS cached sessions? I >> have to imagine there is, but I don't know how. > > as before: just delete the tls_sessions files before you start > cyrus-imapd. They will be recreated automatically. You could even make > that part of the initscript, because those session don't survive a > restart anyway. From Hagedorn at uni-koeln.de Fri Jan 16 07:08:09 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Fri, 16 Jan 2009 13:08:09 +0100 Subject: Expire (manually) TLS sessions? In-Reply-To: <4970764E.5080209@kickflop.net> References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> Message-ID: --On 16. Januar 2009 06:58:06 -0500 Jeff Blaine wrote: > Hmm. That's not working for me. > > bash-2.05# pwd > /var/imap ># rm tls_sessions.db ># /etc/init.d/imap start ># > > Jan 16 06:54:36 imapsrv imap[20300]: [ID 286863 local6.notice] > imapd:Loading hard-coded DH parameters > Jan 16 06:54:36 imapsrv imap[20300]: [ID 239158 local6.notice] STARTTLS > negotiation failed: bva-172.our.com > > bash-2.05# pwd > /var/imap > bash-2.05# ls -l tls* > tls*: No such file or directory > bash-2.05# Strange. I know for a fact that it has worked for us in the past. I can't verify that easily right now, because I don't currently have a test set-up. What happens if you touch the file prior to starting imap? -- .:.Sebastian Hagedorn - RZKR-R1 (Geb?ude 52), Zimmer 18.:. Zentrum f?r angewandte Informatik - Universit?tsweiter Service RRZK .:.Universit?t zu K?ln / Cologne University - ? +49-221-478-5587.:. .:.:.:.Skype: shagedorn.:.:.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090116/91063797/attachment-0001.bin From jblaine at kickflop.net Fri Jan 16 07:18:22 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Fri, 16 Jan 2009 07:18:22 -0500 Subject: Expire (manually) TLS sessions? In-Reply-To: References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> Message-ID: <49707B0E.3000007@kickflop.net> Sebastian Hagedorn wrote: > --On 16. Januar 2009 06:58:06 -0500 Jeff Blaine > wrote: >> Hmm. That's not working for me. >> >> bash-2.05# pwd >> /var/imap >> # rm tls_sessions.db >> # /etc/init.d/imap start >> # >> >> Jan 16 06:54:36 imapsrv imap[20300]: [ID 286863 local6.notice] >> imapd:Loading hard-coded DH parameters >> Jan 16 06:54:36 imapsrv imap[20300]: [ID 239158 local6.notice] STARTTLS >> negotiation failed: bva-172.our.com >> >> bash-2.05# pwd >> /var/imap >> bash-2.05# ls -l tls* >> tls*: No such file or directory >> bash-2.05# > > Strange. I know for a fact that it has worked for us in the past. I > can't verify that easily right now, because I don't currently have a > test set-up. What happens if you touch the file prior to starting imap? I tried that first: cp /dev/null tls_sessions.db Same situation :( From jblaine at kickflop.net Fri Jan 16 07:48:18 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Fri, 16 Jan 2009 07:48:18 -0500 Subject: Expire (manually) TLS sessions? In-Reply-To: <49707B0E.3000007@kickflop.net> References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> <49707B0E.3000007@kickflop.net> Message-ID: <49708212.4010409@kickflop.net> More info after increasing local6.info to local6.debug for syslog: accepted connection imapd:Loading hard-coded DH parameters SSL_accept() incomplete -> wait decryption failed or bad record mac in SSL_accept() -> fail STARTTLS negotiation failed: bva-172.our.com Our TLS all worked fine before the upgrade :( Jeff Blaine wrote: > Sebastian Hagedorn wrote: >> --On 16. Januar 2009 06:58:06 -0500 Jeff Blaine >> wrote: >>> Hmm. That's not working for me. >>> >>> bash-2.05# pwd >>> /var/imap >>> # rm tls_sessions.db >>> # /etc/init.d/imap start >>> # >>> >>> Jan 16 06:54:36 imapsrv imap[20300]: [ID 286863 local6.notice] >>> imapd:Loading hard-coded DH parameters >>> Jan 16 06:54:36 imapsrv imap[20300]: [ID 239158 local6.notice] STARTTLS >>> negotiation failed: bva-172.our.com >>> >>> bash-2.05# pwd >>> /var/imap >>> bash-2.05# ls -l tls* >>> tls*: No such file or directory >>> bash-2.05# >> Strange. I know for a fact that it has worked for us in the past. I >> can't verify that easily right now, because I don't currently have a >> test set-up. What happens if you touch the file prior to starting imap? > > I tried that first: cp /dev/null tls_sessions.db > > Same situation :( > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > From Hagedorn at uni-koeln.de Fri Jan 16 07:54:19 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Fri, 16 Jan 2009 13:54:19 +0100 Subject: Expire (manually) TLS sessions? In-Reply-To: <49708212.4010409@kickflop.net> References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> <49707B0E.3000007@kickflop.net> <49708212.4010409@kickflop.net> Message-ID: --On 16. Januar 2009 07:48:18 -0500 Jeff Blaine wrote: > More info after increasing local6.info to local6.debug for > syslog: > > accepted connection > imapd:Loading hard-coded DH parameters > SSL_accept() incomplete -> wait > decryption failed or bad record mac in SSL_accept() -> fail > STARTTLS negotiation failed: bva-172.our.com > > Our TLS all worked fine before the upgrade :( I'm pretty sure the tls_cache is a red herring. The SSL/TLS code changed a lot between 2.2 and 2.3. My guess would be that there lies the actual problem. I wonder where the line "Loading hard-coded DH parameters" comes from. I haven't seen that before. Anyway, I guess you need an SSL expert to make sense of that. How old is your certificate? Maybe the new code doesn't like it? Did you build the binary yourself or where did you get it? -- .:.Sebastian Hagedorn - RZKR-R1 (Geb?ude 52), Zimmer 18.:. Zentrum f?r angewandte Informatik - Universit?tsweiter Service RRZK .:.Universit?t zu K?ln / Cologne University - ? +49-221-478-5587.:. .:.:.:.Skype: shagedorn.:.:.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090116/ca2532cb/attachment.bin From jblaine at kickflop.net Fri Jan 16 08:16:36 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Fri, 16 Jan 2009 08:16:36 -0500 Subject: Expire (manually) TLS sessions? In-Reply-To: References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> <49707B0E.3000007@kickflop.net> <49708212.4010409@kickflop.net> Message-ID: <497088B4.5080707@kickflop.net> Sebastian Hagedorn wrote: > --On 16. Januar 2009 07:48:18 -0500 Jeff Blaine > wrote: > >> More info after increasing local6.info to local6.debug for >> syslog: >> >> accepted connection >> imapd:Loading hard-coded DH parameters >> SSL_accept() incomplete -> wait >> decryption failed or bad record mac in SSL_accept() -> fail >> STARTTLS negotiation failed: bva-172.our.com >> >> Our TLS all worked fine before the upgrade :( > > I'm pretty sure the tls_cache is a red herring. The SSL/TLS code changed > a lot between 2.2 and 2.3. My guess would be that there lies the actual > problem. > > I wonder where the line "Loading hard-coded DH parameters" comes from. I > haven't seen that before. Anyway, I guess you need an SSL expert to make > sense of that. How old is your certificate? Maybe the new code doesn't > like it? Did you build the binary yourself or where did you get it? The certificate is 1 year 10 months old. Everything was built by hand (as it was with our 2.2.12 install as well). I'll try redoing the cert. From jblaine at kickflop.net Fri Jan 16 09:43:02 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Fri, 16 Jan 2009 09:43:02 -0500 Subject: Expire (manually) TLS sessions? In-Reply-To: <497088B4.5080707@kickflop.net> References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> <49707B0E.3000007@kickflop.net> <49708212.4010409@kickflop.net> <497088B4.5080707@kickflop.net> Message-ID: <49709CF6.9040609@kickflop.net> A new cert did not solve the problem: Jan 16 09:41:30 imapsrv imap[12264]: [ID 921384 local6.debug] accepted connection Jan 16 09:41:30 imapsrv imap[12264]: [ID 192010 local6.debug] wrong version number in SSL_accept() -> fail Jan 16 09:41:30 imapsrv imap[12264]: [ID 239158 local6.notice] STARTTLS negotiation failed: bva-172.our.com Jeff Blaine wrote: > Sebastian Hagedorn wrote: >> --On 16. Januar 2009 07:48:18 -0500 Jeff Blaine >> wrote: >> >>> More info after increasing local6.info to local6.debug for >>> syslog: >>> >>> accepted connection >>> imapd:Loading hard-coded DH parameters >>> SSL_accept() incomplete -> wait >>> decryption failed or bad record mac in SSL_accept() -> fail >>> STARTTLS negotiation failed: bva-172.our.com >>> >>> Our TLS all worked fine before the upgrade :( >> >> I'm pretty sure the tls_cache is a red herring. The SSL/TLS code >> changed a lot between 2.2 and 2.3. My guess would be that there lies >> the actual problem. >> >> I wonder where the line "Loading hard-coded DH parameters" comes from. >> I haven't seen that before. Anyway, I guess you need an SSL expert to >> make sense of that. How old is your certificate? Maybe the new code >> doesn't like it? Did you build the binary yourself or where did you >> get it? > > The certificate is 1 year 10 months old. > > Everything was built by hand (as it was with our 2.2.12 > install as well). > > I'll try redoing the cert. > From Hagedorn at uni-koeln.de Fri Jan 16 09:50:43 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Fri, 16 Jan 2009 15:50:43 +0100 Subject: Expire (manually) TLS sessions? In-Reply-To: <49709CF6.9040609@kickflop.net> References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> <49707B0E.3000007@kickflop.net> <49708212.4010409@kickflop.net> <497088B4.5080707@kickflop.net> <49709CF6.9040609@kickflop.net> Message-ID: <0EAD03513BDEB07A363D29A5@tyrion.rrz.uni-koeln.de> --On 16. Januar 2009 09:43:02 -0500 Jeff Blaine wrote: > A new cert did not solve the problem: > > Jan 16 09:41:30 imapsrv imap[12264]: [ID 921384 local6.debug] accepted > connection > Jan 16 09:41:30 imapsrv imap[12264]: [ID 192010 local6.debug] wrong > version number in SSL_accept() -> fail But it results in a different error message. > Jan 16 09:41:30 imapsrv imap[12264]: [ID 239158 local6.notice] STARTTLS > negotiation failed: bva-172.our.com That reminds me of something. Try removing this line from your config: tls_ca_file: /var/imap/ca.crt Also, try using different clients. IIRC, there is an issue specifically with Thunderbird and that setting. I don't remember the details, but you should be able to find them in the archives. -- .:.Sebastian Hagedorn - RZKR-R1 (Geb?ude 52), Zimmer 18.:. Zentrum f?r angewandte Informatik - Universit?tsweiter Service RRZK .:.Universit?t zu K?ln / Cologne University - ? +49-221-478-5587.:. .:.:.:.Skype: shagedorn.:.:.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090116/6f495448/attachment.bin From jblaine at kickflop.net Fri Jan 16 10:12:00 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Fri, 16 Jan 2009 10:12:00 -0500 Subject: Expire (manually) TLS sessions? In-Reply-To: <0EAD03513BDEB07A363D29A5@tyrion.rrz.uni-koeln.de> References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> <49707B0E.3000007@kickflop.net> <49708212.4010409@kickflop.net> <497088B4.5080707@kickflop.net> <49709CF6.9040609@kickflop.net> <0EAD03513BDEB07A363D29A5@tyrion.rrz.uni-koeln.de> Message-ID: <4970A3C0.1050407@kickflop.net> With the tls_ca_file line removed, Thunderbird asked me to specify a client certificate, I chose my cert and entered my password to access it. Jan 16 10:08:33 imapsrv imap[15668]: [ID 921384 local6.debug] accepted connection Jan 16 10:08:33 imapsrv imap[15668]: [ID 636471 local6.notice] TLS server engine: cannot load CA data Jan 16 10:08:33 imapsrv imap[15668]: [ID 286863 local6.notice] imapd:Loading hard-coded DH parameters Jan 16 10:08:33 imapsrv imap[15668]: [ID 277171 local6.error] TLS server engine: No CA file specified. Client side certs may not work Jan 16 10:08:33 imapsrv imap[15668]: [ID 574029 local6.debug] SSL_accept() incomplete -> wait Jan 16 10:08:43 imapsrv imap[15668]: [ID 160154 local6.debug] Doing a peer verify Jan 16 10:08:43 imapsrv imap[15668]: [ID 227675 local6.error] verify error:num=20:unable to get local issuer certificate Jan 16 10:08:43 imapsrv imap[15668]: [ID 192010 local6.debug] no certificate returned in SSL_accept() -> fail Jan 16 10:08:43 imapsrv imap[15668]: [ID 239158 local6.notice] STARTTLS negotiation failed: bva-172.our.com Sebastian Hagedorn wrote: > --On 16. Januar 2009 09:43:02 -0500 Jeff Blaine > wrote: > >> A new cert did not solve the problem: >> >> Jan 16 09:41:30 imapsrv imap[12264]: [ID 921384 local6.debug] accepted >> connection >> Jan 16 09:41:30 imapsrv imap[12264]: [ID 192010 local6.debug] wrong >> version number in SSL_accept() -> fail > > But it results in a different error message. > >> Jan 16 09:41:30 imapsrv imap[12264]: [ID 239158 local6.notice] STARTTLS >> negotiation failed: bva-172.our.com > > That reminds me of something. Try removing this line from your config: > > tls_ca_file: /var/imap/ca.crt > > Also, try using different clients. IIRC, there is an issue specifically > with Thunderbird and that setting. I don't remember the details, but you > should be able to find them in the archives. From Rudy.Gevaert at ugent.be Fri Jan 16 10:17:23 2009 From: Rudy.Gevaert at ugent.be (Rudy Gevaert) Date: Fri, 16 Jan 2009 16:17:23 +0100 Subject: Doing Cyrus *right* for a large installation In-Reply-To: References: <496EEC7D.9030804@UGent.be> Message-ID: <20090116151723.GA10261@ugent.be> On Thu, Jan 15, 2009 at 12:08:24PM -0800, Andrew Morgan wrote: > > Fulls take a lot longer, but I can't give you any time estimates now. > > Our incrementals take about 2 hours (both backends are done at the same > time). Our fulls take an entire day, but we only do fulls once a month. > I'm hoping that using smaller partitions will let us do backups in shorter > times, assuming we stay with tape-backups. In the past we did the backups in two shifts on the mailstores: shift 1 a-k, shift2 l-z (e.g.) And did them in parallel. > > For the moment we don't backup the replica's. Nor use them. But we > > could do it. > > So you are using Cyrus Murder with replication? How does that work? No we don't have murder running (yet) I am not confident enough in the setup. I have seen to many bad messages on the list. But I haven't set it up yet, so I may be 'scared' for nothing. > > We are 'planning' a storage migration too the coming months. > > I'm curious what storage hardware you are exploring. What are you using > now for your 7 backends? we have 1 cx400 and 1 cx500 now. I think we are looking at something from IBM, but I'm not sure, that is not 'my departement'. We also have a lot of NetApp but we are not satisfied with it. Are primary plan was to migrate to the NetApp but now we aren't going to. Btw for backup we use Veritas Netbackup Best whishes, -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Rudy Gevaert Rudy.Gevaert at UGent.be tel:+32 9 264 4734 Directie ICT, afd. Infrastructuur Direction ICT, Infrastructure dept. Groep Systemen Systems group Universiteit Gent Ghent University Krijgslaan 281, gebouw S9, 9000 Gent, Belgie www.UGent.be -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- From jblaine at kickflop.net Fri Jan 16 10:19:51 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Fri, 16 Jan 2009 10:19:51 -0500 Subject: Expire (manually) TLS sessions? In-Reply-To: <4970A3C0.1050407@kickflop.net> References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> <49707B0E.3000007@kickflop.net> <49708212.4010409@kickflop.net> <497088B4.5080707@kickflop.net> <49709CF6.9040609@kickflop.net> <0EAD03513BDEB07A363D29A5@tyrion.rrz.uni-koeln.de> <4970A3C0.1050407@kickflop.net> Message-ID: <4970A597.4050808@kickflop.net> Outlook 2007 works. Unfortunately, this is not an option for us as our users use Thunderbird. Jan 16 10:18:14 imapsrv imap[16000]: [ID 921384 local6.debug] accepted connection Jan 16 10:18:14 imapsrv imap[16000]: [ID 636471 local6.notice] TLS server engine: cannot load CA data Jan 16 10:18:14 imapsrv imap[16000]: [ID 286863 local6.notice] imapd:Loading hard-coded DH parameters Jan 16 10:18:14 imapsrv imap[16000]: [ID 277171 local6.error] TLS server engine: No CA file specified. Client side certs may not work Jan 16 10:18:15 imapsrv imap[16000]: [ID 574029 local6.debug] SSL_accept() incomplete -> wait Jan 16 10:18:15 imapsrv imap[16000]: [ID 867439 local6.debug] SSL_accept() succeeded -> done Jan 16 10:18:15 imapsrv imap[16000]: [ID 379946 local6.notice] starttls: TLSv1 with cipher RC4-MD5 (128/128 bits new) no authentication Jan 16 10:18:15 imapsrv imap[16000]: [ID 277583 local6.notice] login: bva-172.our.com jblaine plaintext+TLS User logged in Jeff Blaine wrote: > With the tls_ca_file line removed, Thunderbird asked me > to specify a client certificate, I chose my cert and > entered my password to access it. > > Jan 16 10:08:33 imapsrv imap[15668]: [ID 921384 local6.debug] accepted > connection > Jan 16 10:08:33 imapsrv imap[15668]: [ID 636471 local6.notice] TLS > server engine: cannot load CA data > Jan 16 10:08:33 imapsrv imap[15668]: [ID 286863 local6.notice] > imapd:Loading hard-coded DH parameters > Jan 16 10:08:33 imapsrv imap[15668]: [ID 277171 local6.error] TLS server > engine: No CA file specified. Client side certs may not work > Jan 16 10:08:33 imapsrv imap[15668]: [ID 574029 local6.debug] > SSL_accept() incomplete -> wait > Jan 16 10:08:43 imapsrv imap[15668]: [ID 160154 local6.debug] Doing a > peer verify > Jan 16 10:08:43 imapsrv imap[15668]: [ID 227675 local6.error] verify > error:num=20:unable to get local issuer certificate > Jan 16 10:08:43 imapsrv imap[15668]: [ID 192010 local6.debug] no > certificate returned in SSL_accept() -> fail > Jan 16 10:08:43 imapsrv imap[15668]: [ID 239158 local6.notice] STARTTLS > negotiation failed: bva-172.our.com > > Sebastian Hagedorn wrote: >> --On 16. Januar 2009 09:43:02 -0500 Jeff Blaine >> wrote: >> >>> A new cert did not solve the problem: >>> >>> Jan 16 09:41:30 imapsrv imap[12264]: [ID 921384 local6.debug] accepted >>> connection >>> Jan 16 09:41:30 imapsrv imap[12264]: [ID 192010 local6.debug] wrong >>> version number in SSL_accept() -> fail >> But it results in a different error message. >> >>> Jan 16 09:41:30 imapsrv imap[12264]: [ID 239158 local6.notice] STARTTLS >>> negotiation failed: bva-172.our.com >> That reminds me of something. Try removing this line from your config: >> >> tls_ca_file: /var/imap/ca.crt >> >> Also, try using different clients. IIRC, there is an issue specifically >> with Thunderbird and that setting. I don't remember the details, but you >> should be able to find them in the archives. > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > From list at joreybump.com Fri Jan 16 10:47:26 2009 From: list at joreybump.com (Jorey Bump) Date: Fri, 16 Jan 2009 10:47:26 -0500 Subject: Expire (manually) TLS sessions? In-Reply-To: <4970A3C0.1050407@kickflop.net> References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> <49707B0E.3000007@kickflop.net> <49708212.4010409@kickflop.net> <497088B4.5080707@kickflop.net> <49709CF6.9040609@kickflop.net> <0EAD03513BDEB07A363D29A5@tyrion.rrz.uni-koeln.de> <4970A3C0.1050407@kickflop.net> Message-ID: <4970AC0E.9020507@joreybump.com> Jeff Blaine wrote, at 01/16/2009 10:12 AM: > With the tls_ca_file line removed, Thunderbird asked me > to specify a client certificate, I chose my cert and > entered my password to access it. That sounds backwards. My understanding is that setting tls_ca_file is what will cause some clients to prompt for a client certificate, and that commenting out the setting avoids this problem if you don't use client certs. > Jan 16 10:08:33 imapsrv imap[15668]: [ID 921384 local6.debug] accepted > connection > Jan 16 10:08:33 imapsrv imap[15668]: [ID 636471 local6.notice] TLS > server engine: cannot load CA data That's fine. It's a spurious log message as a result of removing tls_ca_file. > Jan 16 10:08:33 imapsrv imap[15668]: [ID 286863 local6.notice] > imapd:Loading hard-coded DH parameters This is also normal, nothing to worry about. > Jan 16 10:08:33 imapsrv imap[15668]: [ID 277171 local6.error] TLS server > engine: No CA file specified. Client side certs may not work More harmless noise from the removal of tls_ca_file. > Jan 16 10:08:33 imapsrv imap[15668]: [ID 574029 local6.debug] > SSL_accept() incomplete -> wait > Jan 16 10:08:43 imapsrv imap[15668]: [ID 160154 local6.debug] Doing a > peer verify > Jan 16 10:08:43 imapsrv imap[15668]: [ID 227675 local6.error] verify > error:num=20:unable to get local issuer certificate > Jan 16 10:08:43 imapsrv imap[15668]: [ID 192010 local6.debug] no > certificate returned in SSL_accept() -> fail > Jan 16 10:08:43 imapsrv imap[15668]: [ID 239158 local6.notice] STARTTLS > negotiation failed: bva-172.our.com This is probably related to your client certificate, now that you don't have a CA store for verification. I don't know why Thunderbird prompted you for a certificate, though. You might want to test from another Thunderbird with no client certs installed. In any case, this might be easier to troubleshoot if you post your imapd.conf (and maybe even cyrus.conf). I found it was a little tricky going upgrading within 2.3.x due to some TLS changes, but I still managed to maintain a very simple configuration. Yours just might need a couple of tweaks. From jblaine at kickflop.net Fri Jan 16 11:18:50 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Fri, 16 Jan 2009 11:18:50 -0500 Subject: Expire (manually) TLS sessions? In-Reply-To: <4970AC0E.9020507@joreybump.com> References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> <49707B0E.3000007@kickflop.net> <49708212.4010409@kickflop.net> <497088B4.5080707@kickflop.net> <49709CF6.9040609@kickflop.net> <0EAD03513BDEB07A363D29A5@tyrion.rrz.uni-koeln.de> <4970A3C0.1050407@kickflop.net> <4970AC0E.9020507@joreybump.com> Message-ID: <4970B36A.8030902@kickflop.net> Jorey Bump wrote: > Jeff Blaine wrote, at 01/16/2009 10:12 AM: >> With the tls_ca_file line removed, Thunderbird asked me >> to specify a client certificate, I chose my cert and >> entered my password to access it. > > That sounds backwards. My understanding is that setting tls_ca_file is > what will cause some clients to prompt for a client certificate, and > that commenting out the setting avoids this problem if you don't use > client certs. > >> Jan 16 10:08:33 imapsrv imap[15668]: [ID 921384 local6.debug] accepted >> connection >> Jan 16 10:08:33 imapsrv imap[15668]: [ID 636471 local6.notice] TLS >> server engine: cannot load CA data > > That's fine. It's a spurious log message as a result of removing > tls_ca_file. > >> Jan 16 10:08:33 imapsrv imap[15668]: [ID 286863 local6.notice] >> imapd:Loading hard-coded DH parameters > > This is also normal, nothing to worry about. > >> Jan 16 10:08:33 imapsrv imap[15668]: [ID 277171 local6.error] TLS server >> engine: No CA file specified. Client side certs may not work > > More harmless noise from the removal of tls_ca_file. > >> Jan 16 10:08:33 imapsrv imap[15668]: [ID 574029 local6.debug] >> SSL_accept() incomplete -> wait >> Jan 16 10:08:43 imapsrv imap[15668]: [ID 160154 local6.debug] Doing a >> peer verify >> Jan 16 10:08:43 imapsrv imap[15668]: [ID 227675 local6.error] verify >> error:num=20:unable to get local issuer certificate >> Jan 16 10:08:43 imapsrv imap[15668]: [ID 192010 local6.debug] no >> certificate returned in SSL_accept() -> fail >> Jan 16 10:08:43 imapsrv imap[15668]: [ID 239158 local6.notice] STARTTLS >> negotiation failed: bva-172.our.com > > This is probably related to your client certificate, now that you don't > have a CA store for verification. I don't know why Thunderbird prompted > you for a certificate, though. You might want to test from another > Thunderbird with no client certs installed. > > In any case, this might be easier to troubleshoot if you post your > imapd.conf (and maybe even cyrus.conf). I found it was a little tricky > going upgrading within 2.3.x due to some TLS changes, but I still > managed to maintain a very simple configuration. Yours just might need a > couple of tweaks. Here ya go: allowplaintext: true configdirectory: /var/imap defaultpartition: default partition-default: /var/spool/imap imap_admins: root cyrus sasl_pwcheck_method: saslauthd unix_group_enable: 0 sieveusehomedir: true autocreatequota: 200000 duplicate_db: skiplist annotation_db: skiplist mboxkey_db: skiplist mboxlist_db: skiplist tlscache_db: skiplist tls_session_timeout: 0 tls_cert_file: /var/imap/ca.crt tls_key_file: /var/imap/server.pk START { recover cmd="ctl_cyrusdb -r" } SERVICES { imap cmd="imapd" listen="imap" prefork=5 proto=tcp4 imaps cmd="imapd -s" listen="imaps" prefork=1 proto=tcp4 lmtpunix cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=1 } EVENTS { checkpoint cmd="ctl_cyrusdb -c" period=10 delprune cmd="cyr_expire -E 3" at=0400 tlsprune cmd="tls_prune" at=0400 } From sebelk at gmail.com Fri Jan 16 11:31:38 2009 From: sebelk at gmail.com (Sergio Belkin) Date: Fri, 16 Jan 2009 14:31:38 -0200 Subject: Last access Message-ID: <8c6f7f450901160831u48c96035u638866e714e6519c@mail.gmail.com> HI, I am using cyrus-imapd-2.3.7-1.1.el5 on Centos 5. How I can get the last access to his mailbox of a given user? Thanks in advance -- -- Open Kairos http://www.openkairos.com Watch More TV http://sebelk.blogspot.com Sergio Belkin - From sebelk at gmail.com Fri Jan 16 12:44:53 2009 From: sebelk at gmail.com (Sergio Belkin) Date: Fri, 16 Jan 2009 15:44:53 -0200 Subject: howto list inactive users in cyrus: In-Reply-To: <05ECE370-7D29-4F7D-906D-E31B54BA62EE@umich.edu> References: <494B8067.7060606@linpro.no> <9B733BFFD7A4289F99CF942A@192.168.2.14> <05ECE370-7D29-4F7D-906D-E31B54BA62EE@umich.edu> Message-ID: <8c6f7f450901160944s5af9a616wbc88ad2b05eb2938@mail.gmail.com> 2008/12/19 Wesley Craig : > On 19 Dec 2008, at 12:38, Joseph Brennan wrote: >> How about checking the timestamp on the user.seen files? It seems to >> get touched at login, even if you don't open a message. > > There's a "last read" timestamp in the seen DB, which is probably > more accurate. > > :wes > > ---- Sorry, but I skipped this thread, and I've found that has to do with I'm looking for. I've read the script but I am not using MySQL. is there a cyrus tool to get the last read read from DB? Would be it enough with timestamp of cyrus.seen files? Please fix me I am wrong... -- -- Open Kairos http://www.openkairos.com Watch More TV http://sebelk.blogspot.com Sergio Belkin - From wes at umich.edu Fri Jan 16 13:55:30 2009 From: wes at umich.edu (Wesley Craig) Date: Fri, 16 Jan 2009 13:55:30 -0500 Subject: Doing Cyrus *right* for a large installation In-Reply-To: <20090116151723.GA10261@ugent.be> References: <496EEC7D.9030804@UGent.be> <20090116151723.GA10261@ugent.be> Message-ID: On 16 Jan 2009, at 10:17, Rudy Gevaert wrote: > No we don't have murder running (yet) > > I am not confident enough in the setup. I have seen to many bad > messages on the list. > > But I haven't set it up yet, so I may be 'scared' for nothing. There are always bugs, of course, but in recent versions, replication is working very well. IMHO, the major piece of basic functionality missing for replication to be 100% solid is an out of band method to verify that it's working properly. I know of several (many?) solutions that people have in place. Perhaps something will be contributed soon. :wes From brong at fastmail.fm Fri Jan 16 21:23:09 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sat, 17 Jan 2009 13:23:09 +1100 Subject: Doing Cyrus *right* for a large installation In-Reply-To: References: <496EEC7D.9030804@UGent.be> <20090116151723.GA10261@ugent.be> Message-ID: <20090117022309.GA13178@brong.net> On Fri, Jan 16, 2009 at 01:55:30PM -0500, Wesley Craig wrote: > On 16 Jan 2009, at 10:17, Rudy Gevaert wrote: > > No we don't have murder running (yet) > > > > I am not confident enough in the setup. I have seen to many bad > > messages on the list. > > > > But I haven't set it up yet, so I may be 'scared' for nothing. > > There are always bugs, of course, but in recent versions, replication > is working very well. IMHO, the major piece of basic functionality > missing for replication to be 100% solid is an out of band method to > verify that it's working properly. I know of several (many?) > solutions that people have in place. Perhaps something will be > contributed soon. Yeah - ours is way complex, because it checks at lots of different points! Unfortunately, it needs to log in separately as each user to check seen database information. I'd much prefer to be able to do the whole thing as the admin user. Bron. From brong at fastmail.fm Fri Jan 16 21:24:41 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sat, 17 Jan 2009 13:24:41 +1100 Subject: Expire (manually) TLS sessions? In-Reply-To: <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> Message-ID: <20090117022441.GB13178@brong.net> On Fri, Jan 16, 2009 at 12:46:32PM +0100, Sebastian Hagedorn wrote: > Hello Jeff, > > --On 16. Januar 2009 06:38:27 -0500 Jeff Blaine > wrote: > >> Maybe we're doing something wrong in the process, but it >> seems that every time we perform offline maintenance >> (upgrade, whatever) on Cyrus IMAPd ... our users complain >> that TLS breaks afterward, but then fixes itself in time. >> >> I've demonstrated this to myself just now with the upgrade >> to 2.3.13 from 2.2.12. My TLS session is cached but broken >> with the new setup (or for whatever other reason). That is, >> even after restarting Thunderbird, I get the following: >> >> Jan 16 06:31:50 imapsrv imap[19690]: [ID 239158 local6.notice] STARTTLS >> negotiation failed: bva-172.our.com >> >> Is there a way to zero/flush all TLS cached sessions? I >> have to imagine there is, but I don't know how. > > as before: just delete the tls_sessions files before you start > cyrus-imapd. They will be recreated automatically. You could even make > that part of the initscript, because those session don't survive a > restart anyway. In that case maybe Cyrus should do this itself? Sounds like a candidate for a fix. (we don't do TLS on our backends, because all connections come to nginx, and it always connects plaintext to the backends) Bron. From Rudy.Gevaert at ugent.be Sat Jan 17 03:22:02 2009 From: Rudy.Gevaert at ugent.be (Rudy Gevaert) Date: Sat, 17 Jan 2009 09:22:02 +0100 Subject: Doing Cyrus *right* for a large installation In-Reply-To: <20090117022309.GA13178@brong.net> References: <496EEC7D.9030804@UGent.be> <20090116151723.GA10261@ugent.be> <20090117022309.GA13178@brong.net> Message-ID: <20090117082202.GA8166@ugent.be> On Sat, Jan 17, 2009 at 01:23:09PM +1100, Bron Gondwana wrote: > On Fri, Jan 16, 2009 at 01:55:30PM -0500, Wesley Craig wrote: > > On 16 Jan 2009, at 10:17, Rudy Gevaert wrote: > > > No we don't have murder running (yet) > > > > > > I am not confident enough in the setup. I have seen to many bad > > > messages on the list. > > > > > > But I haven't set it up yet, so I may be 'scared' for nothing. > > > > There are always bugs, of course, but in recent versions, replication > > is working very well. IMHO, the major piece of basic functionality > > missing for replication to be 100% solid is an out of band method to > > verify that it's working properly. I know of several (many?) > > solutions that people have in place. Perhaps something will be > > contributed soon. > > Yeah - ours is way complex, because it checks at lots of different > points! > > Unfortunately, it needs to log in separately as each user to check > seen database information. I'd much prefer to be able to do the whole > thing as the admin user. I'm using the fastmail patches to check replication. The only thing that I'm not doing is on a regular basis. But when I need to restore from backup and reconstruct a mailbox I check the replica with my script. -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Rudy Gevaert Rudy.Gevaert at UGent.be tel:+32 9 264 4734 Directie ICT, afd. Infrastructuur Direction ICT, Infrastructure dept. Groep Systemen Systems group Universiteit Gent Ghent University Krijgslaan 281, gebouw S9, 9000 Gent, Belgie www.UGent.be -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- From Rudy.Gevaert at ugent.be Sat Jan 17 03:22:31 2009 From: Rudy.Gevaert at ugent.be (Rudy Gevaert) Date: Sat, 17 Jan 2009 09:22:31 +0100 Subject: Doing Cyrus *right* for a large installation In-Reply-To: References: <496EEC7D.9030804@UGent.be> <20090116151723.GA10261@ugent.be> Message-ID: <20090117082231.GB8166@ugent.be> On Fri, Jan 16, 2009 at 01:55:30PM -0500, Wesley Craig wrote: > On 16 Jan 2009, at 10:17, Rudy Gevaert wrote: >> No we don't have murder running (yet) >> >> I am not confident enough in the setup. I have seen to many bad >> messages on the list. >> >> But I haven't set it up yet, so I may be 'scared' for nothing. > > There are always bugs, of course, but in recent versions, replication is > working very well. IMHO, the major piece of basic functionality missing > for replication to be 100% solid is an out of band method to verify that > it's working properly. I know of several (many?) solutions that people > have in place. Perhaps something will be contributed soon. I meant that I'm 'scared' to use murder. Rudy -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Rudy Gevaert Rudy.Gevaert at UGent.be tel:+32 9 264 4734 Directie ICT, afd. Infrastructuur Direction ICT, Infrastructure dept. Groep Systemen Systems group Universiteit Gent Ghent University Krijgslaan 281, gebouw S9, 9000 Gent, Belgie www.UGent.be -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- From toomas.aas at raad.tartu.ee Sat Jan 17 09:39:56 2009 From: toomas.aas at raad.tartu.ee (Toomas Aas) Date: Sat, 17 Jan 2009 16:39:56 +0200 Subject: Last access In-Reply-To: <8c6f7f450901160831u48c96035u638866e714e6519c@mail.gmail.com> References: <8c6f7f450901160831u48c96035u638866e714e6519c@mail.gmail.com> Message-ID: <200901171639.56717.toomas.aas@raad.tartu.ee> Friday 16 January 2009 18:31:38 kirjutas Sergio Belkin: > > How I can get the last access to his mailbox of a given user? > Maybe I'm naive, but I have always just looked at the timestamp of that user's seen file. -- Toomas From Hagedorn at uni-koeln.de Sat Jan 17 09:44:36 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Sat, 17 Jan 2009 15:44:36 +0100 Subject: Expire (manually) TLS sessions? In-Reply-To: <4970A3C0.1050407@kickflop.net> References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> <49707B0E.3000007@kickflop.net> <49708212.4010409@kickflop.net> <497088B4.5080707@kickflop.net> <49709CF6.9040609@kickflop.net> <0EAD03513BDEB07A363D29A5@tyrion.rrz.uni-koeln.de> <4970A3C0.1050407@kickflop.net> Message-ID: -- Jeff Blaine is rumored to have mumbled on 16. Januar 2009 10:12:00 -0500 regarding Re: Expire (manually) TLS sessions?: > With the tls_ca_file line removed, Thunderbird asked me > to specify a client certificate, I chose my cert and > entered my password to access it. It's quite possible that authentication with a certificate is broken. I helped fix a few SSL-related bugs that affected us, but we don't use certificates. That means that the changes that were made weren't tested using such a setup. -- Sebastian Hagedorn - RZKR-R1 (Flachbau), Zi. 18, Robert-Koch-Str. 10 Zentrum f?r angewandte Informatik - Universit?tsweiter Service RRZK Universit?t zu K?ln / Cologne University - Tel. +49-221-478-5587 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090117/16b0ee7f/attachment-0001.bin From Hagedorn at uni-koeln.de Sat Jan 17 09:46:54 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Sat, 17 Jan 2009 15:46:54 +0100 Subject: Expire (manually) TLS sessions? In-Reply-To: <4970A597.4050808@kickflop.net> References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> <49707B0E.3000007@kickflop.net> <49708212.4010409@kickflop.net> <497088B4.5080707@kickflop.net> <49709CF6.9040609@kickflop.net> <0EAD03513BDEB07A363D29A5@tyrion.rrz.uni-koeln.de> <4970A3C0.1050407@kickflop.net> <4970A597.4050808@kickflop.net> Message-ID: <1D1C8ECCD65FCB3DC2CAF6E7@G5.voip.local> -- Jeff Blaine is rumored to have mumbled on 16. Januar 2009 10:19:51 -0500 regarding Re: Expire (manually) TLS sessions?: > Outlook 2007 works. Using a password, that is. I guess you didn't try it with a certificate? (I assume Outlook supports that, but I'm not sure) > Unfortunately, this is not an option > for us as our users use Thunderbird. How about Thunderbird using a password for authentication? Is that an option at all? -- Sebastian Hagedorn - RZKR-R1 (Flachbau), Zi. 18, Robert-Koch-Str. 10 Zentrum f?r angewandte Informatik - Universit?tsweiter Service RRZK Universit?t zu K?ln / Cologne University - Tel. +49-221-478-5587 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090117/bef2f3f7/attachment.bin From Hagedorn at uni-koeln.de Sat Jan 17 09:49:30 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Sat, 17 Jan 2009 15:49:30 +0100 Subject: Expire (manually) TLS sessions? In-Reply-To: <20090117022441.GB13178@brong.net> References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <20090117022441.GB13178@brong.net> Message-ID: -- Bron Gondwana is rumored to have mumbled on 17. Januar 2009 13:24:41 +1100 regarding Re: Expire (manually) TLS sessions?: > just delete the tls_sessions files before you start >> cyrus-imapd. They will be recreated automatically. You could even make >> that part of the initscript, because those session don't survive a >> restart anyway. > > In that case maybe Cyrus should do this itself? Sounds like a candidate > for a fix. I'm not sure. Right now it seems to me that Jeff's problem didn't have anything to do with that file. The problems we were having with it were years ago, back when we used Berkeley DB. We have since switched to skiplist and haven't had any trouble anymore. -- Sebastian Hagedorn - RZKR-R1 (Flachbau), Zi. 18, Robert-Koch-Str. 10 Zentrum f?r angewandte Informatik - Universit?tsweiter Service RRZK Universit?t zu K?ln / Cologne University - Tel. +49-221-478-5587 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090117/3dfe5724/attachment.bin From morgan at orst.edu Sat Jan 17 12:10:54 2009 From: morgan at orst.edu (Andrew Morgan) Date: Sat, 17 Jan 2009 09:10:54 -0800 (PST) Subject: Doing Cyrus *right* for a large installation In-Reply-To: <20090117082231.GB8166@ugent.be> References: <496EEC7D.9030804@UGent.be> <20090116151723.GA10261@ugent.be> <20090117082231.GB8166@ugent.be> Message-ID: On Sat, 17 Jan 2009, Rudy Gevaert wrote: > On Fri, Jan 16, 2009 at 01:55:30PM -0500, Wesley Craig wrote: >> On 16 Jan 2009, at 10:17, Rudy Gevaert wrote: >>> No we don't have murder running (yet) >>> >>> I am not confident enough in the setup. I have seen to many bad >>> messages on the list. >>> >>> But I haven't set it up yet, so I may be 'scared' for nothing. >> >> There are always bugs, of course, but in recent versions, replication is >> working very well. IMHO, the major piece of basic functionality missing >> for replication to be 100% solid is an out of band method to verify that >> it's working properly. I know of several (many?) solutions that people >> have in place. Perhaps something will be contributed soon. > > I meant that I'm 'scared' to use murder. Scared of what happening? We've been using murder here for the past 4 years. Andy From wes at umich.edu Sat Jan 17 13:14:53 2009 From: wes at umich.edu (Wesley Craig) Date: Sat, 17 Jan 2009 13:14:53 -0500 Subject: Last access In-Reply-To: <200901171639.56717.toomas.aas@raad.tartu.ee> References: <8c6f7f450901160831u48c96035u638866e714e6519c@mail.gmail.com> <200901171639.56717.toomas.aas@raad.tartu.ee> Message-ID: On 17 Jan 2009, at 09:39, Toomas Aas wrote: > Friday 16 January 2009 18:31:38 kirjutas Sergio Belkin: >> How I can get the last access to his mailbox of a given user? > > Maybe I'm naive, but I have always just looked at the timestamp of > that > user's seen file. That's definitely a good estimate. The only obvious caveats I can think of are related to reconstruct and sync_server: I don't think either take pains to maintain the timestamp of the file. :wes From wes at umich.edu Sat Jan 17 13:20:07 2009 From: wes at umich.edu (Wesley Craig) Date: Sat, 17 Jan 2009 13:20:07 -0500 Subject: Doing Cyrus *right* for a large installation In-Reply-To: <20090117082231.GB8166@ugent.be> References: <496EEC7D.9030804@UGent.be> <20090116151723.GA10261@ugent.be> <20090117082231.GB8166@ugent.be> Message-ID: On 17 Jan 2009, at 03:22, Rudy Gevaert wrote: > I meant that I'm 'scared' to use murder. "Standard" murder is pretty solid. "Unified" is not yet solid, but I think it's fair to say that that's the logical direction for Cyrus to go -- it solves many inconsistencies, and it's a step along the path of full redundancy with a minimal number of machines. I don't have any operational experience with "replicated" murder, but I also haven't heard many horror stories. :wes From Rudy.Gevaert at ugent.be Sun Jan 18 14:43:50 2009 From: Rudy.Gevaert at ugent.be (Rudy Gevaert) Date: Sun, 18 Jan 2009 20:43:50 +0100 Subject: Doing Cyrus *right* for a large installation In-Reply-To: References: <496EEC7D.9030804@UGent.be> <20090116151723.GA10261@ugent.be> <20090117082231.GB8166@ugent.be> Message-ID: <20090118194350.GC8528@ugent.be> On Sat, Jan 17, 2009 at 09:10:54AM -0800, Andrew Morgan wrote: > Scared of what happening? We've been using murder here for the past 4 > years. First, I haven't even tried to set up a murder. However from time to time I see posts on the list (from you e.g. :)) about things going not as they should be. Maybe I interpreted them wrongly, but they gave me a bad feeling... Rudy -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Rudy Gevaert Rudy.Gevaert at UGent.be tel:+32 9 264 4734 Directie ICT, afd. Infrastructuur Direction ICT, Infrastructure dept. Groep Systemen Systems group Universiteit Gent Ghent University Krijgslaan 281, gebouw S9, 9000 Gent, Belgie www.UGent.be -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- From Rudy.Gevaert at ugent.be Sun Jan 18 14:58:48 2009 From: Rudy.Gevaert at ugent.be (Rudy Gevaert) Date: Sun, 18 Jan 2009 20:58:48 +0100 Subject: Doing Cyrus *right* for a large installation In-Reply-To: References: <496EEC7D.9030804@UGent.be> <20090116151723.GA10261@ugent.be> <20090117082231.GB8166@ugent.be> Message-ID: <20090118195848.GD8528@ugent.be> On Sat, Jan 17, 2009 at 01:20:07PM -0500, Wesley Craig wrote: > On 17 Jan 2009, at 03:22, Rudy Gevaert wrote: > > I meant that I'm 'scared' to use murder. > > "Standard" murder is pretty solid. "Unified" is not yet solid, but I > think it's fair to say that that's the logical direction for Cyrus to > go -- it solves many inconsistencies, and it's a step along the path > of full redundancy with a minimal number of machines. I don't have > any operational experience with "replicated" murder, but I also > haven't heard many horror stories. Hi Wesley I can't find any info about the unified on the wiki/site. I once did read about it. In the docs I can't find a definition of the unified setup. Btw has anybody migrated a normal setup to a murder setup? Can it easily be done? E.g. stop, edit configs, start. More or less :) Thanks!! -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Rudy Gevaert Rudy.Gevaert at UGent.be tel:+32 9 264 4734 Directie ICT, afd. Infrastructuur Direction ICT, Infrastructure dept. Groep Systemen Systems group Universiteit Gent Ghent University Krijgslaan 281, gebouw S9, 9000 Gent, Belgie www.UGent.be -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- From andrewm at object-craft.com.au Mon Jan 19 01:29:29 2009 From: andrewm at object-craft.com.au (Andrew McNamara) Date: Mon, 19 Jan 2009 17:29:29 +1100 Subject: choosing a file system In-Reply-To: <20090111000728.GA10334@brong.net> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> <20090109085409.GF5876@helsinki.fi> <20090109144138.GA37579@sysmon.tcworks.net> <20090109152002.GH5876@helsinki.fi> <20090110095600.GA26886@brong.net> <4968F899.5000305@joreybump.com> <20090111000728.GA10334@brong.net> Message-ID: <20090119062929.4C85E5AC015@longblack.object-craft.com.au> >Yeah, except Postfix encodes the inode of the queue files in its queue >IDs, so it gets very confused if you do this. Same with restoring >queues from backups. You should be able to get away with this if, when moving the queue to another machine, you move the queued mail from hold, incoming, active and deferred directories into the maildrop directory on the target instance. This (somewhat old, but still correct, I think) message from Wietse might shed more light on it: Date: Thu, 12 Sep 2002 20:33:08 -0400 (EDT) From: wietse at porcupine.org (Wietse Venema) Subject: Re: postfix migration > I want to migrate postfix to another machine. What are also the steps so > that I won't lose mails on the process? This is the safe procedure. 1) On the old machine, stop Postfix. 2) On the old machine, run as super-user: postsuper -r ALL This moves all queue files to the maildrop queue. 3) On the old machine, back up /var/spool/postfix/maildrop 4) On the new machine, make sure Postfix works. 5) On the new machine, stop Postfix. 6) On the new machine, restore /var/spool/postfix/maildrop 7) On the new machine, start Postfix. There are ways to skip the "postsuper -r ALL" step, and copy the incoming + active + deferred + bounce + defer + flush + hold directories to the new machine, but that would be safe only with an empty queue on the new machine. -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/ From archi.laurent at gmail.com Mon Jan 19 06:24:27 2009 From: archi.laurent at gmail.com (Laurent Archambault) Date: Mon, 19 Jan 2009 12:24:27 +0100 Subject: hashimapspool + fulldirhash Message-ID: <692f4d2f0901190324l2af5e03coc1753db833539d49@mail.gmail.com> Hello all, I has activate this values for Cyrus "hashimapspool: true (+) fulldirhash: true", and i has do this after advice by a an "old"(cyrus) administator. For him, it's more better. Ok for this. Now i search now the real explain (and same technical) for this 2 options. - And i think it's important : cyrus is activate with "virtdomains = yes !!!! Have a good day and thanks Best regards -- ----~o00o-----//{ ??`(_)??` }\\-----o00o~------ Laurent Archambault Under Linux -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090119/43953f44/attachment.html From peo at intersonic.se Mon Jan 19 11:08:04 2009 From: peo at intersonic.se (Per olof Ljungmark) Date: Mon, 19 Jan 2009 17:08:04 +0100 Subject: Any plans for a 2.3.14? Message-ID: <4974A564.80209@intersonic.se> Hi, Any plans to get a new release out soonish? We are planning for other work on the mail system and it would be nice to have the latest bugfixes and improvements done. Thanks for a great piece of software! -- per From iane at sussex.ac.uk Mon Jan 19 12:36:17 2009 From: iane at sussex.ac.uk (Ian Eiloart) Date: Mon, 19 Jan 2009 17:36:17 +0000 Subject: murder configuration issue final stretch In-Reply-To: References: <417DD550405AF84F95040DA44520C8E505191C@pxtbenexd02.pxt.primeexalia.com> <3B5E8BFE-5E46-452C-936A-569822312718@umich.edu> <417DD550405AF84F95040DA44520C8E505191E@pxtbenexd02.pxt.primeexalia.com> <417DD550405AF84F95040DA44520C8E5051922@pxtbenexd02.pxt.primeexalia.com> <417DD550405AF84F95040DA44520C8E5051925@pxtbenexd02.pxt.primeexalia.com> <417DD550405AF84F95040DA44520C8E505192D@pxtbenexd02.pxt.primeexalia.com> <417DD550405AF84F95040DA44520C8E5051930@pxtbenexd02.pxt.primeexalia.com> Message-ID: <999C350A3BE0D8E612B62A4F@lewes.staff.uscs.susx.ac.uk> Hi, I've got a similar problem. Can I clarify what this solution means, please? It looks like we need lmtp listening on a TCP/IP socket at the back end, of course. I've got that. At the front end I'm delivering into a unix socket, so I have that set up. But does the solution below suggest that I also need to set up a tcp listener on the front end? Perhaps that's the process that's doing the final delivery? Or is the solution below one line for the front end and one for the back end? --On 12 December 2008 09:46:05 -0800 Andrew Morgan wrote: > On Fri, 12 Dec 2008, Gary W. Smith wrote: > >> Andrew, >> >> Any idea how this works with the lmtpproxy? Also, I've tried to setup >> the lmtp using the local port but for some reason when I restart >> cyrus-imapd it doesn't who up in netstat. >> >> Tried >> lmtp cmd="lmtpd" listen="lmtp" prefork=1 >> Tried >> lmtp cmd="lmtpd" listen="/var/lib/imap/socket/lmtp" prefork=1 >> maxchild=20 Current: >> lmtp cmd="lmtpproxyd" listen="/var/lib/imap/socket/lmtpunix" >> prefork=1 >> >> Do you have any examples of setting it up on a different port and >> forwarding it onwards to the backend (or I assume that we would forward >> it to the proxy) using postfix?. > > I believe you want: > > lmtp cmd="lmtpproxyd" listen="2004" prefork=1 maxchild=50 > lmtpunix cmd="lmtpproxyd" listen="/var/lib/imap/socket/lmtpunix" > prefork=1 > > This should listen on port 2004 (instead of the usual 2003) on all > interfaces, plus listen on the unix socket you had defined earlier. > > Andy > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- Ian Eiloart IT Services, University of Sussex x3148 From list at joreybump.com Mon Jan 19 13:06:41 2009 From: list at joreybump.com (Jorey Bump) Date: Mon, 19 Jan 2009 13:06:41 -0500 Subject: choosing a file system In-Reply-To: <20090119062929.4C85E5AC015@longblack.object-craft.com.au> References: <200901020850.22355.jmadden@ivytech.edu> <20090108152000.GH13471@helsinki.fi> <49662340.7070103@ucdavis.edu> <20090109085409.GF5876@helsinki.fi> <20090109144138.GA37579@sysmon.tcworks.net> <20090109152002.GH5876@helsinki.fi> <20090110095600.GA26886@brong.net> <4968F899.5000305@joreybump.com> <20090111000728.GA10334@brong.net> <20090119062929.4C85E5AC015@longblack.object-craft.com.au> Message-ID: <4974C131.40007@joreybump.com> Andrew McNamara wrote, at 01/19/2009 01:29 AM: >> Yeah, except Postfix encodes the inode of the queue files in its queue >> IDs, so it gets very confused if you do this. Same with restoring >> queues from backups. > > You should be able to get away with this if, when moving the queue to > another machine, you move the queued mail from hold, incoming, active and > deferred directories into the maildrop directory on the target instance. > > This (somewhat old, but still correct, I think) message from Wietse > might shed more light on it: > > Date: Thu, 12 Sep 2002 20:33:08 -0400 (EDT) > From: wietse at porcupine.org (Wietse Venema) > Subject: Re: postfix migration > > > I want to migrate postfix to another machine. What are also the steps so > > that I won't lose mails on the process? > > This is the safe procedure. > > 1) On the old machine, stop Postfix. > > 2) On the old machine, run as super-user: > > postsuper -r ALL > > This moves all queue files to the maildrop queue. > > 3) On the old machine, back up /var/spool/postfix/maildrop > > 4) On the new machine, make sure Postfix works. > > 5) On the new machine, stop Postfix. > > 6) On the new machine, restore /var/spool/postfix/maildrop > > 7) On the new machine, start Postfix. > > There are ways to skip the "postsuper -r ALL" step, and copy the > incoming + active + deferred + bounce + defer + flush + hold > directories to the new machine, but that would be safe only with > an empty queue on the new machine. > This has become somewhat off-topic for this list, but you might be able to simply sync the entire Postfix queue to the backup machine, and run postsuper -s before starting Postfix on the backup. From the postsuper man page: -s Structure check and structure repair. This should be done once before Postfix startup. Rename files whose name does not match the message file inode number. This operation is necessary after restoring a mail queue from a different machine, or from backup media. The important thing to keep in mind is that Postfix embeds the inode number in the filename simply to keep the name unique while the message resides on the filesystem. Obviously, this approach breaks when the files are copied to another filesystem. Renaming them appropriately on the new destination ensures no files will be overwritten as the queue is processed or new messages enter the queue. Of course, the scheme I proposed earlier requires that once the backup Postfix is brought up, it must be impossible for the primary to begin resyncing files to the same location on the backup if it becomes active again (or refuses to die a graceful death). Certainly tricky, but it sounds like the use case is to preserve the queue in case of a total failure, just to make sure the mail goes out (even it means it goes out twice). From gary at primeexalia.com Mon Jan 19 13:11:19 2009 From: gary at primeexalia.com (Gary W. Smith) Date: Mon, 19 Jan 2009 10:11:19 -0800 Subject: murder configuration issue final stretch In-Reply-To: <999C350A3BE0D8E612B62A4F@lewes.staff.uscs.susx.ac.uk> References: <417DD550405AF84F95040DA44520C8E505191C@pxtbenexd02.pxt.primeexalia.com> <3B5E8BFE-5E46-452C-936A-569822312718@umich.edu> <417DD550405AF84F95040DA44520C8E505191E@pxtbenexd02.pxt.primeexalia.com> <417DD550405AF84F95040DA44520C8E5051922@pxtbenexd02.pxt.primeexalia.com> <417DD550405AF84F95040DA44520C8E5051925@pxtbenexd02.pxt.primeexalia.com> <417DD550405AF84F95040DA44520C8E505192D@pxtbenexd02.pxt.primeexalia.com> <417DD550405AF84F95040DA44520C8E5051930@pxtbenexd02.pxt.primeexalia.com> <999C350A3BE0D8E612B62A4F@lewes.staff.uscs.susx.ac.uk> Message-ID: <417DD550405AF84F95040DA44520C8E5A8CF@pxtbenexd02.pxt.primeexalia.com> Unfortunately I was unable to reproduce a working environment. It's still on my to do list but I've been over tasked over the last couple weeks and this was a lower priority. We will be setting up a new environment in the next 30 days and are hoping to tackle our problem then. We are getting new hardware which will free up more hardware as a test environment. Gary -----Original Message----- From: iane at sussex.ac.uk [mailto:iane at sussex.ac.uk] Sent: Monday, January 19, 2009 9:36 AM To: Gary W. Smith Cc: info-cyrus at lists.andrew.cmu.edu Subject: RE: murder configuration issue final stretch Hi, I've got a similar problem. Can I clarify what this solution means, please? It looks like we need lmtp listening on a TCP/IP socket at the back end, of course. I've got that. At the front end I'm delivering into a unix socket, so I have that set up. But does the solution below suggest that I also need to set up a tcp listener on the front end? Perhaps that's the process that's doing the final delivery? Or is the solution below one line for the front end and one for the back end? --On 12 December 2008 09:46:05 -0800 Andrew Morgan wrote: > On Fri, 12 Dec 2008, Gary W. Smith wrote: > >> Andrew, >> >> Any idea how this works with the lmtpproxy? Also, I've tried to setup >> the lmtp using the local port but for some reason when I restart >> cyrus-imapd it doesn't who up in netstat. >> >> Tried >> lmtp cmd="lmtpd" listen="lmtp" prefork=1 >> Tried >> lmtp cmd="lmtpd" listen="/var/lib/imap/socket/lmtp" prefork=1 >> maxchild=20 Current: >> lmtp cmd="lmtpproxyd" listen="/var/lib/imap/socket/lmtpunix" >> prefork=1 >> >> Do you have any examples of setting it up on a different port and >> forwarding it onwards to the backend (or I assume that we would forward >> it to the proxy) using postfix?. > > I believe you want: > > lmtp cmd="lmtpproxyd" listen="2004" prefork=1 maxchild=50 > lmtpunix cmd="lmtpproxyd" listen="/var/lib/imap/socket/lmtpunix" > prefork=1 > > This should listen on port 2004 (instead of the usual 2003) on all > interfaces, plus listen on the unix socket you had defined earlier. > > Andy > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- Ian Eiloart IT Services, University of Sussex x3148 From wes at umich.edu Mon Jan 19 13:26:51 2009 From: wes at umich.edu (Wesley Craig) Date: Mon, 19 Jan 2009 13:26:51 -0500 Subject: Doing Cyrus *right* for a large installation In-Reply-To: <20090118195848.GD8528@ugent.be> References: <496EEC7D.9030804@UGent.be> <20090116151723.GA10261@ugent.be> <20090117082231.GB8166@ugent.be> <20090118195848.GD8528@ugent.be> Message-ID: <1718860C-A694-4C51-92C1-AC67EEFD86C3@umich.edu> On 18 Jan 2009, at 14:58, Rudy Gevaert wrote: > I can't find any info about the unified on the wiki/site. I once did > read about it. In the docs I can't find a definition of the unified > setup. From imapd.conf man page: mupdate_config: standard The configuration of the mupdate servers in the Cyrus Murder. The "standard" config is one in which there are discreet frontend (proxy) and backend servers. The "unified" config is one in which a server can be both a frontend and backend. The "repli- cated" config is one in which multiple backend servers all share the same mailspool, but each have their own "replicated" copy of mailboxes.db. Allowed values: standard, unified, replicated I'm not sure what I'd add to this description, other than the assertions I've made about the relative stability of the various components. > Btw has anybody migrated a normal setup to a murder setup? > Can it easily be done? E.g. stop, edit configs, start. More or > less :) What's a "normal setup"? :wes From brong at fastmail.fm Mon Jan 19 17:43:08 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 20 Jan 2009 09:43:08 +1100 Subject: Any plans for a 2.3.14? In-Reply-To: <4974A564.80209@intersonic.se> References: <4974A564.80209@intersonic.se> Message-ID: <20090119224308.GA8107@brong.net> On Mon, Jan 19, 2009 at 05:08:04PM +0100, Per olof Ljungmark wrote: > Hi, > > Any plans to get a new release out soonish? We are planning for other > work on the mail system and it would be nice to have the latest bugfixes > and improvements done. > > Thanks for a great piece of software! Sounds good to me :) I have a small queue of bugfixes to push up to CVS before the next release. I've been waiting on getting my CVS password to commit them myself, but I'm happy to just post them for someone else to commit if we're going to do a release soonish. Bron. From hijinks at gmail.com Mon Jan 19 15:47:47 2009 From: hijinks at gmail.com (Mike Zupan) Date: Mon, 19 Jan 2009 15:47:47 -0500 Subject: murder help Message-ID: <7227c6c70901191247s1e835cbakc5e7523e273d6bf6@mail.gmail.com> We currently have a cyrus server setup. The issue is we need to migrate users off one type of storage to another with little to no downtime. I heard about murder for cyrus and wondering if it could work for a already running imap setup. Basically what we need is something to tell cyrus to send mail here now since we moved the mailbox over there and if it can handle moving mailboxes in real time. The plan is to setup a new cyrus server to place behind it with our old server and hopefully move over mailboxes on the fly. Will murder work for this? or do we need to look in another direction? or is this even possible? Thanks! Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090119/a0c503dc/attachment.html From dom.lalot at gmail.com Tue Jan 20 02:25:33 2009 From: dom.lalot at gmail.com (LALOT Dominique) Date: Tue, 20 Jan 2009 08:25:33 +0100 Subject: choosing a file system In-Reply-To: <4974C131.40007@joreybump.com> References: <200901020850.22355.jmadden@ivytech.edu> <49662340.7070103@ucdavis.edu> <20090109085409.GF5876@helsinki.fi> <20090109144138.GA37579@sysmon.tcworks.net> <20090109152002.GH5876@helsinki.fi> <20090110095600.GA26886@brong.net> <4968F899.5000305@joreybump.com> <20090111000728.GA10334@brong.net> <20090119062929.4C85E5AC015@longblack.object-craft.com.au> <4974C131.40007@joreybump.com> Message-ID: <1617f8010901192325t2383dc3y8d5484030f943158@mail.gmail.com> Please, we are following a long thread I introduced a while ago which was speaking about file system. If you want to ask question about something you red in that thread, but not properly speaking about file system, please, be kind enough to start another thread and stop replying in that thread. Dom 2009/1/19 Jorey Bump : > Andrew McNamara wrote, at 01/19/2009 01:29 AM: >>> Yeah, except Postfix encodes the inode of the queue files in its queue >>> IDs, so it gets very confused if you do this. Same with restoring >>> queues from backups. >> >> You should be able to get away with this if, when moving the queue to >> another machine, you move the queued mail from hold, incoming, active and >> deferred directories into the maildrop directory on the target instance. >> >> This (somewhat old, but still correct, I think) message from Wietse >> might shed more light on it: >> >> Date: Thu, 12 Sep 2002 20:33:08 -0400 (EDT) >> From: wietse at porcupine.org (Wietse Venema) >> Subject: Re: postfix migration >> >> > I want to migrate postfix to another machine. What are also the steps so >> > that I won't lose mails on the process? >> >> This is the safe procedure. >> >> 1) On the old machine, stop Postfix. >> >> 2) On the old machine, run as super-user: >> >> postsuper -r ALL >> >> This moves all queue files to the maildrop queue. >> >> 3) On the old machine, back up /var/spool/postfix/maildrop >> >> 4) On the new machine, make sure Postfix works. >> >> 5) On the new machine, stop Postfix. >> >> 6) On the new machine, restore /var/spool/postfix/maildrop >> >> 7) On the new machine, start Postfix. >> >> There are ways to skip the "postsuper -r ALL" step, and copy the >> incoming + active + deferred + bounce + defer + flush + hold >> directories to the new machine, but that would be safe only with >> an empty queue on the new machine. >> > > This has become somewhat off-topic for this list, but you might be able > to simply sync the entire Postfix queue to the backup machine, and run > postsuper -s before starting Postfix on the backup. From the postsuper > man page: > > -s Structure check and structure repair. This should be done > once before Postfix startup. > > Rename files whose name does not match the message file inode > number. This operation is necessary after restoring a mail > queue from a different machine, or from backup media. > > The important thing to keep in mind is that Postfix embeds the inode > number in the filename simply to keep the name unique while the message > resides on the filesystem. Obviously, this approach breaks when the > files are copied to another filesystem. Renaming them appropriately on > the new destination ensures no files will be overwritten as the queue is > processed or new messages enter the queue. Of course, the scheme I > proposed earlier requires that once the backup Postfix is brought up, it > must be impossible for the primary to begin resyncing files to the same > location on the backup if it becomes active again (or refuses to die a > graceful death). Certainly tricky, but it sounds like the use case is to > preserve the queue in case of a total failure, just to make sure the > mail goes out (even it means it goes out twice). > > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > -- Dominique LALOT Ing?nieur Syst?mes et R?seaux http://annuaire.univmed.fr/showuser?uid=lalot From aspineux at gmail.com Tue Jan 20 02:43:45 2009 From: aspineux at gmail.com (Alain Spineux) Date: Tue, 20 Jan 2009 08:43:45 +0100 Subject: Quota on subfolders question. In-Reply-To: <20090114185122.591E04446B@ws5-1.us4.outblaze.com> References: <20090114185122.591E04446B@ws5-1.us4.outblaze.com> Message-ID: <71fe4e760901192343p488a9294vfdd464d722243f86@mail.gmail.com> On Wed, Jan 14, 2009 at 7:51 PM, k bah wrote: > cyrus-imapd-2.3.8 > cyrus-sasl-2.1.22 > -- > user/someuser~domain.com (\HasChildren) > user/someuser~domain.com/Drafts (\HasNoChildren) > user/someuser~domain.com/INBOX (\HasNoChildren) > user/someuser~domain.com/2008 Messages (\HasNoChildren) > user/someuser~domain.com/Sent (\HasNoChildren) > user/someuser~domain.com/Spam (\HasNoChildren) > user/someuser~domain.com/Trash (\HasNoChildren) > -- > > Hi, > > The above folder/subfolder structure is how my users mailboxes are now. > > Q1) If INBOX has a 50Mb quota (autocreatequota on imapd.conf) and the other subfolders do not have a quota configured via cyradm, all of them respect the 50Mb limit, right? The sum of all of them cannot be more than 50Mb. > Yes, that's right, all sub-folder depend of the same quotaroot > Q2) If I set a quota higher than 50Mb to any of those subfolders, say 60Mb for Trash, then can the user move messages to the Trash folder until they sum 55Mb (all messages on Trash sum 55Mb), *without* affecting the INBOX limit? I mean, the user will still be able to receive messages if Trash has 55Mb? > Yes It should be. > thanks for you time > > > = > Slatwall - Guaranteed Low Prices > Save Money. Buy Slatwall Direct From The Manufacturer. Ships Same Day. > http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=c33f3e7389ca71081776d3da9be71f6a > > > -- > Powered by Outblaze > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > -- Alain Spineux aspineux gmail com May the sources be with you From aspineux at gmail.com Tue Jan 20 02:47:45 2009 From: aspineux at gmail.com (Alain Spineux) Date: Tue, 20 Jan 2009 08:47:45 +0100 Subject: Auto set quota to specific folder thru imapd.conf/cyrus.conf ?? In-Reply-To: <20090114184314.131FB4446B@ws5-1.us4.outblaze.com> References: <20090114184314.131FB4446B@ws5-1.us4.outblaze.com> Message-ID: <71fe4e760901192347l7aeaf952j93653bc6ae464de0@mail.gmail.com> On Wed, Jan 14, 2009 at 7:43 PM, k bah wrote: > cyrus-imapd-2.3.8-51 > cyrus-sasl-2.1.22-82 > -- > user/someuser~domain.com (\HasChildren) > user/someuser~domain.com/Drafts (\HasNoChildren) > user/someuser~domain.com/INBOX (\HasNoChildren) > user/someuser~domain.com/2008 Messages (\HasNoChildren) > user/someuser~domain.com/Sent (\HasNoChildren) > user/someuser~domain.com/Spam (\HasNoChildren) > user/someuser~domain.com/Trash (\HasNoChildren) > -- > > Hi, > > Is it possible to set a specific quota for a folder by editing cyrus.conf or imapd.conf? I don't think so. > I already set a quota there for INBOX, but I need to auto set a quota for another folder, Trash, the ideal would be to set Trash with an unlimited quota, since the only supported IMAP client my users have access to is Webmail, and it automatically deletes all messages from the Trash folder monthly. > I could run some script to set a higher quota on user/myuser~domain.com/Trash (calling cyradm), but that would be my last option. That way, new users would have Trash with a higher quota, not affecting their INBOX % usage. If possible I want to do that (unlimit the quota for Trash, keeping the default quota for INBOX and subfolders) by editing imapd.conf or some other config file. > > thanks in advance > > = > > > -- > Powered by Outblaze > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > -- Alain Spineux aspineux gmail com May the sources be with you From kbah at linuxmail.org Tue Jan 20 13:12:33 2009 From: kbah at linuxmail.org (k bah) Date: Tue, 20 Jan 2009 15:12:33 -0300 Subject: Lmtp w/ Postfix and LDAP: change mailbox name when saving mail. Message-ID: <20090120181233.D89E57BD56@ws5-10.us4.outblaze.com> cyrus-imapd-2.3.8 cyrus-sasl-2.1.22 postfix-2.4.5 -------- On my LDAP server, the uid field of each user (person) is: "username~domain.org". I correctly set up cyrus and saslauthd to authenticate with the username as the LDAP field above. A simple Horde hook does the job to make "user at whatever domain he/she typed" into "user~whatever domain...". When authenticating, if the mailbox does not exist it gets created (autocreatequota is nonzero on imapd.conf). The mailbox is created following that naming convention (username~domain..). createonpost is set to 1 on imapd.conf My problem is when saving incoming mail. The mailbox "username~domain.org" is not created, instead it creates a mailbox named "username" (actually user/username, and "username~domain.org" is actually "user/username~domain.org"). Other relevant (I think) options from imapd.conf: unixhierarchysep: 1 username_tolower: 1 normalizeuid: 1 Only /etc/saslauthd.conf contains stuff about LDAP. I tried setting ldap_* options on imapd.conf (as the imapd.conf man page tells) but I guess they are not read when mail is arriving to the system (Postfix/Cyrus lmtp). Is it possible to (automatically) create mailboxes with that naming convention? I want cyrus to save mail to "username~domain.org" instead of "username". thanks in advance = Retail-Man Point of Sale (POS) Software Low cost POS systems use Retail-Man software. Free 30 day evaluation. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=e08e956d3eddef06b5754d8411899ca9 -- Powered by Outblaze From morgan at orst.edu Tue Jan 20 14:04:41 2009 From: morgan at orst.edu (Andrew Morgan) Date: Tue, 20 Jan 2009 11:04:41 -0800 (PST) Subject: Doing Cyrus *right* for a large installation In-Reply-To: <20090118195848.GD8528@ugent.be> References: <496EEC7D.9030804@UGent.be> <20090116151723.GA10261@ugent.be> <20090117082231.GB8166@ugent.be> <20090118195848.GD8528@ugent.be> Message-ID: On Sun, 18 Jan 2009, Rudy Gevaert wrote: > Btw has anybody migrated a normal setup to a murder setup? > > Can it easily be done? E.g. stop, edit configs, start. More or less :) Yeah, it's easy. We did it here. Have your frontend(s) and mupdate master machine ready to go. Then just switch the config to murder on the backend and restart cyrus. It should populate the mupdate master with the mailbox list, and the frontend should get the mailbox list from the mupdate master (probably need to restart the frontend). In our case, the next step was to "rename" mailboxes onto the second backend server to distribute them around. Andy From morgan at orst.edu Tue Jan 20 15:02:52 2009 From: morgan at orst.edu (Andrew Morgan) Date: Tue, 20 Jan 2009 12:02:52 -0800 (PST) Subject: murder configuration issue final stretch In-Reply-To: <999C350A3BE0D8E612B62A4F@lewes.staff.uscs.susx.ac.uk> References: <417DD550405AF84F95040DA44520C8E505191C@pxtbenexd02.pxt.primeexalia.com> <3B5E8BFE-5E46-452C-936A-569822312718@umich.edu> <417DD550405AF84F95040DA44520C8E505191E@pxtbenexd02.pxt.primeexalia.com> <417DD550405AF84F95040DA44520C8E5051922@pxtbenexd02.pxt.primeexalia.com> <417DD550405AF84F95040DA44520C8E5051925@pxtbenexd02.pxt.primeexalia.com> <417DD550405AF84F95040DA44520C8E505192D@pxtbenexd02.pxt.primeexalia.com> <417DD550405AF84F95040DA44520C8E5051930@pxtbenexd02.pxt.primeexalia.com> <999C350A3BE0D8E612B62A4F@lewes.staff.uscs.susx.ac.uk> Message-ID: On Mon, 19 Jan 2009, Ian Eiloart wrote: > Hi, > > I've got a similar problem. Can I clarify what this solution means, please? > > It looks like we need lmtp listening on a TCP/IP socket at the back end, of > course. I've got that. > > At the front end I'm delivering into a unix socket, so I have that set up. > > But does the solution below suggest that I also need to set up a tcp > listener on the front end? Perhaps that's the process that's doing the > final delivery? > > Or is the solution below one line for the front end and one for the back > end? I had to use an authenticated LMTP connection to the frontends in order for the frontend to proxy the auth to the backend LMTP connection. The only way I found to do that was to deliver to the LMTP tcp socket. Andy From morgan at orst.edu Tue Jan 20 15:29:31 2009 From: morgan at orst.edu (Andrew Morgan) Date: Tue, 20 Jan 2009 12:29:31 -0800 (PST) Subject: murder help In-Reply-To: <7227c6c70901191247s1e835cbakc5e7523e273d6bf6@mail.gmail.com> References: <7227c6c70901191247s1e835cbakc5e7523e273d6bf6@mail.gmail.com> Message-ID: On Mon, 19 Jan 2009, Mike Zupan wrote: > We currently have a cyrus server setup. The issue is we need to migrate > users off one type of storage to another with little to no downtime. I heard > about murder for cyrus and wondering if it could work for a already running > imap setup. > > Basically what we need is something to tell cyrus to send mail here now > since we moved the mailbox over there and if it can handle moving mailboxes > in real time. > > The plan is to setup a new cyrus server to place behind it with our old > server and hopefully move over mailboxes on the fly. > > Will murder work for this? or do we need to look in another direction? or is > this even possible? If you have a working murder setup, it will do what you listed above. Andy From dwhite at olp.net Wed Jan 21 10:07:27 2009 From: dwhite at olp.net (Dan White) Date: Wed, 21 Jan 2009 09:07:27 -0600 Subject: Lmtp w/ Postfix and LDAP: change mailbox name when saving mail. In-Reply-To: <20090120181233.D89E57BD56@ws5-10.us4.outblaze.com> References: <20090120181233.D89E57BD56@ws5-10.us4.outblaze.com> Message-ID: <49773A2F.5010305@olp.net> k bah wrote: > cyrus-imapd-2.3.8 > cyrus-sasl-2.1.22 > postfix-2.4.5 > -------- > > On my LDAP server, the uid field of each user (person) is: "username~domain.org". > I correctly set up cyrus and saslauthd to authenticate with the username as the LDAP field above. A simple Horde hook does the job to make "user at whatever domain he/she typed" into "user~whatever domain...". > When authenticating, if the mailbox does not exist it gets created (autocreatequota is nonzero on imapd.conf). The mailbox is created following that naming convention (username~domain..). > > ... > Is it possible to (automatically) create mailboxes with that naming convention? I want cyrus to save mail to "username~domain.org" instead of "username". > You can use the ldapdb user canonicalization patch for SASL (See CVS) for situations like this. It won't automatically created mailboxes, but it will direct cyrus IMAP to use an alternative name before referencing the mailbox. For example: sasl_ldapdb_uri: ldap://ldap.example.org sasl_ldapdb_mech: GSSAPI sasl_ldapdb_canon_attr: uid #sasl_canon_user_plugin: ladpdb #imap_sasl_canon_user_plugin: ldapdb #pop3_sasl_canon_user_plugin: ldapdb #nntp_sasl_canon_user_plugin: ldapdb #sieve_sasl_canon_user_plugin: ldapdb lmtp_sasl_canon_user_plugin: ldapdb Where 'lmtp' matches the name you use within /etc/cyrus.conf. This would direct cyrus imap, after authenticating, to look up the 'uid' attribute within LDAP, and use it as the effective username (and mailbox name). You would not need to change your authentication setup from saslauthd to ldapdb auxprop, since auxprop plugins and user canonicalization plugings can function independently. See the '/doc/options.html' documentation for cyrus SASL for more options (in CVS). - Dan From dwhite at olp.net Wed Jan 21 10:16:08 2009 From: dwhite at olp.net (Dan White) Date: Wed, 21 Jan 2009 09:16:08 -0600 Subject: Lmtp w/ Postfix and LDAP: change mailbox name when saving mail. In-Reply-To: <49773A2F.5010305@olp.net> References: <20090120181233.D89E57BD56@ws5-10.us4.outblaze.com> <49773A2F.5010305@olp.net> Message-ID: <49773C38.1030407@olp.net> Dan White wrote: > > sasl_ldapdb_uri: ldap://ldap.example.org > sasl_ldapdb_mech: GSSAPI > sasl_ldapdb_canon_attr: uid > #sasl_canon_user_plugin: ladpdb > #imap_sasl_canon_user_plugin: ldapdb > #pop3_sasl_canon_user_plugin: ldapdb > #nntp_sasl_canon_user_plugin: ldapdb > #sieve_sasl_canon_user_plugin: ldapdb > lmtp_sasl_canon_user_plugin: ldapdb > > On second thought, this probably won't do anything useful, since the user is not authenticating to LMTP. A Postfix regex map (or LDAP map) may be able to convert the recipient before it gets handed off to LMTP. - Dan From jblaine at kickflop.net Wed Jan 21 11:19:31 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Wed, 21 Jan 2009 11:19:31 -0500 Subject: Expire (manually) TLS sessions? In-Reply-To: <1D1C8ECCD65FCB3DC2CAF6E7@G5.voip.local> References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> <49707B0E.3000007@kickflop.net> <49708212.4010409@kickflop.net> <497088B4.5080707@kickflop.net> <49709CF6.9040609@kickflop.net> <0EAD03513BDEB07A363D29A5@tyrion.rrz.uni-koeln.de> <4970A3C0.1050407@kickflop.net> <4970A597.4050808@kickflop.net> <1D1C8ECCD65FCB3DC2CAF6E7@G5.voip.local> Message-ID: <49774B13.5070404@kickflop.net> Sorry for the delay -- I had my wedding and a brief mini-honeymoon to attend to ;) Sebastian Hagedorn wrote: > -- Jeff Blaine is rumored to have mumbled on 16. > Januar 2009 10:19:51 -0500 regarding Re: Expire (manually) TLS sessions?: > >> Outlook 2007 works. > > Using a password, that is. I guess you didn't try it with a certificate? > (I assume Outlook supports that, but I'm not sure) > >> Unfortunately, this is not an option >> for us as our users use Thunderbird. > > How about Thunderbird using a password for authentication? Is that an > option at all? I realize this is a little "all over the road" here, but bear with me as I am just trying to get something working at this point for our users who are now without secure IMAP :( With "TLS" selected in Thunderbird, I am given no choice but to select a client certificate. See attached images. Another user reports that GNU Emacs with the Gnus client works with SSL and port 993. I've confirmed this in the log: Jan 21 11:11:03 imapsrv imaps[14170]: [ID 277583 local6.notice] login: jimbo-host.our.com [xx.xx.50.67] jimbo plaintext+TLS User logged in If I configure Thunderbird to do that (SSL via 993), I get the following: Jan 21 11:10:19 imapsrv imaps[14104]: [ID 636471 local6.notice] TLS server engine: cannot load CA data Jan 21 11:10:19 imapsrv imaps[14104]: [ID 286863 local6.notice] imapd:Loading hard-coded DH parameters Jan 21 11:10:19 imapsrv imaps[14104]: [ID 798856 local6.notice] imaps TLS negotiation failed: myclient.our.com Jan 21 11:10:19 imapsrv imaps[14104]: [ID 637875 local6.error] Fatal error: tls_start_servertls() failed -------------- next part -------------- A non-text attachment was scrubbed... Name: tbird-cyrus2.jpg Type: image/jpeg Size: 9988 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090121/bf7a871c/attachment-0002.jpg -------------- next part -------------- A non-text attachment was scrubbed... Name: tbird-cyrus.jpg Type: image/jpeg Size: 44860 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090121/bf7a871c/attachment-0003.jpg From Hagedorn at uni-koeln.de Wed Jan 21 12:24:10 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Wed, 21 Jan 2009 18:24:10 +0100 Subject: Expire (manually) TLS sessions? In-Reply-To: <49774B13.5070404@kickflop.net> References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> <49707B0E.3000007@kickflop.net> <49708212.4010409@kickflop.net> <497088B4.5080707@kickflop.net> <49709CF6.9040609@kickflop.net> <0EAD03513BDEB07A363D29A5@tyrion.rrz.uni-koeln.de> <4970A3C0.1050407@kickflop.net> <4970A597.4050808@kickflop.net> <1D1C8ECCD65FCB3DC2CAF6E7@G5.voip.local> <49774B13.5070404@kickflop.net> Message-ID: Hi Jeff, --On 21. Januar 2009 11:19:31 -0500 Jeff Blaine wrote: > Sorry for the delay -- I had my wedding and a brief > mini-honeymoon to attend to ;) congrats! >> How about Thunderbird using a password for authentication? Is that an >> option at all? > > I realize this is a little "all over the road" here, > but bear with me as I am just trying to get something > working at this point for our users who are now > without secure IMAP :( > > With "TLS" selected in Thunderbird, I am given no > choice but to select a client certificate. See > attached images. I wonder why that is. The only reason that comes to mind is that you *have* a certificate. I don't and so I'm never asked to use it. So why don't you try removing your certificate? Honestly, I would expect the same to happen that happens when you use SSL, but you never know. > Another user reports that GNU Emacs with the Gnus > client works with SSL and port 993. I've confirmed > this in the log: > > Jan 21 11:11:03 imapsrv imaps[14170]: [ID 277583 local6.notice] login: > jimbo-host.our.com [xx.xx.50.67] jimbo plaintext+TLS User logged in > > If I configure Thunderbird to do that (SSL via 993), > I get the following: > > Jan 21 11:10:19 imapsrv imaps[14104]: [ID 636471 local6.notice] TLS > server engine: cannot load CA data > Jan 21 11:10:19 imapsrv imaps[14104]: [ID 286863 local6.notice] > imapd:Loading hard-coded DH parameters > Jan 21 11:10:19 imapsrv imaps[14104]: [ID 798856 local6.notice] imaps TLS > negotiation failed: myclient.our.com > Jan 21 11:10:19 imapsrv imaps[14104]: [ID 637875 local6.error] Fatal > error: tls_start_servertls() failed I have no idea why that happens. I just tried it myself and got the following in our log: Jan 21 18:17:48 lvr13 imaps[9855]: accepted connection Jan 21 18:17:48 lvr13 imaps[9855]: SSL_accept() incomplete -> wait Jan 21 18:17:48 lvr13 imaps[9855]: SSL_accept() succeeded -> done Jan 21 18:17:48 lvr13 imaps[9855]: starttls: TLSv1 with cipher AES256-SHA (256/256 bits new) no authentication Jan 21 18:17:53 lvr13 imaps[9855]: login: [redacted] User logged in Could it be that your OpenSSL version or your certificate somehow don't support features that Thunderbird requires? I'm really no expert, but I know that client and server *negotiate* about these things. And the error reads "negotiation failed" ... If your server is accessible over the Internet, perhaps I could try connecting to it with "openssl s_client". That might tell us something. You can try that as well, of course. -- .:.Sebastian Hagedorn - RZKR-R1 (Geb?ude 52), Zimmer 18.:. Zentrum f?r angewandte Informatik - Universit?tsweiter Service RRZK .:.Universit?t zu K?ln / Cologne University - ? +49-221-478-5587.:. .:.:.:.Skype: shagedorn.:.:.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090121/df448772/attachment.bin From jblaine at kickflop.net Wed Jan 21 12:41:31 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Wed, 21 Jan 2009 12:41:31 -0500 Subject: Expire (manually) TLS sessions? In-Reply-To: References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> <49707B0E.3000007@kickflop.net> <49708212.4010409@kickflop.net> <497088B4.5080707@kickflop.net> <49709CF6.9040609@kickflop.net> <0EAD03513BDEB07A363D29A5@tyrion.rrz.uni-koeln.de> <4970A3C0.1050407@kickflop.net> <4970A597.4050808@kickflop.net> <1D1C8ECCD65FCB3DC2CAF6E7@G5.voip.local> <49774B13.5070404@kickflop.net> Message-ID: <49775E4B.8080805@kickflop.net> Sebastian Hagedorn wrote: > Hi Jeff, > > --On 21. Januar 2009 11:19:31 -0500 Jeff Blaine > wrote: > >> Sorry for the delay -- I had my wedding and a brief >> mini-honeymoon to attend to ;) > > congrats! Thanks :) >>> How about Thunderbird using a password for authentication? Is that an >>> option at all? >> >> I realize this is a little "all over the road" here, >> but bear with me as I am just trying to get something >> working at this point for our users who are now >> without secure IMAP :( >> >> With "TLS" selected in Thunderbird, I am given no >> choice but to select a client certificate. See >> attached images. > > I wonder why that is. The only reason that comes to mind is that you > *have* a certificate. I don't and so I'm never asked to use it. So why > don't you try removing your certificate? Honestly, I would expect the > same to happen that happens when you use SSL, but you never know. I'll try to do this and get back to you. >> Another user reports that GNU Emacs with the Gnus >> client works with SSL and port 993. I've confirmed >> this in the log: >> >> Jan 21 11:11:03 imapsrv imaps[14170]: [ID 277583 local6.notice] login: >> jimbo-host.our.com [xx.xx.50.67] jimbo plaintext+TLS User logged in >> >> If I configure Thunderbird to do that (SSL via 993), >> I get the following: >> >> Jan 21 11:10:19 imapsrv imaps[14104]: [ID 636471 local6.notice] TLS >> server engine: cannot load CA data >> Jan 21 11:10:19 imapsrv imaps[14104]: [ID 286863 local6.notice] >> imapd:Loading hard-coded DH parameters >> Jan 21 11:10:19 imapsrv imaps[14104]: [ID 798856 local6.notice] imaps TLS >> negotiation failed: myclient.our.com >> Jan 21 11:10:19 imapsrv imaps[14104]: [ID 637875 local6.error] Fatal >> error: tls_start_servertls() failed > > I have no idea why that happens. I just tried it myself and got the > following in our log: > > Jan 21 18:17:48 lvr13 imaps[9855]: accepted connection > Jan 21 18:17:48 lvr13 imaps[9855]: SSL_accept() incomplete -> wait > Jan 21 18:17:48 lvr13 imaps[9855]: SSL_accept() succeeded -> done > Jan 21 18:17:48 lvr13 imaps[9855]: starttls: TLSv1 with cipher > AES256-SHA (256/256 bits new) no authentication > Jan 21 18:17:53 lvr13 imaps[9855]: login: [redacted] User logged in > > Could it be that your OpenSSL version or your certificate somehow don't > support features that Thunderbird requires? I'm really no expert, but I > know that client and server *negotiate* about these things. And the > error reads "negotiation failed" ... FWIW, Thunderbird with SSL on port 993 pops up a box saying incorrect Message authentication code. I forgot to mention that. > If your server is accessible over the Internet, perhaps I could try > connecting to it with "openssl s_client". That might tell us something. > You can try that as well, of course. Obvious sanitizing below: bash-2.05# /imapsrv/bin/openssl s_client -connect imapsrv:993 CONNECTED(00000004) depth=0 /O=our.com/OU=Servers/CN=imapsrv.our.com verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 /O=our.com/OU=Servers/CN=imapsrv.our.com verify error:num=27:certificate not trusted verify return:1 depth=0 /O=our.com/OU=Servers/CN=imapsrv.our.com verify error:num=21:unable to verify the first certificate verify return:1 --- Certificate chain 0 s:/O=our.com/OU=Servers/CN=imapsrv.our.com i:/O=our.com/OU=Certificate Authority/CN=Our Corporation Primary CA-1 --- Server certificate -----BEGIN CERTIFICATE----- MIIDWzCCAkOgAwIBAgICKCQw--blah-blah... blah...6nfEfM9VDXKFAQw1EpXU= -----END CERTIFICATE----- subject=/O=our.com/OU=Servers/CN=imapsrv.our.com issuer=/O=our.com/OU=Certificate Authority/CN=Our Corporation Primary CA-1 --- No client certificate CA names sent --- SSL handshake has read 1427 bytes and written 322 bytes --- New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA Server public key is 1024 bit Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1 Cipher : DHE-RSA-AES256-SHA Session-ID: 7CFF7259D4E28002.....................8BC4F829E0C0FC90700 Session-ID-ctx: Master-Key: FDA05F594004CE18421274................490D4B93678C4.............8DBD9610C89D Key-Arg : None Start Time: 1232559254 Timeout : 300 (sec) Verify return code: 21 (unable to verify the first certificate) --- * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID AUTH=PLAIN SASL-IR] imapsrv.our.com Cyrus IMAP v2.3.13 server ready From jblaine at kickflop.net Wed Jan 21 13:02:32 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Wed, 21 Jan 2009 13:02:32 -0500 Subject: Expire (manually) TLS sessions? In-Reply-To: References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> <49707B0E.3000007@kickflop.net> <49708212.4010409@kickflop.net> <497088B4.5080707@kickflop.net> <49709CF6.9040609@kickflop.net> <0EAD03513BDEB07A363D29A5@tyrion.rrz.uni-koeln.de> <4970A3C0.1050407@kickflop.net> <4970A597.4050808@kickflop.net> <1D1C8ECCD65FCB3DC2CAF6E7@G5.voip.local> <49774B13.5070404@kickflop.net> Message-ID: <49776338.5060803@kickflop.net> I raised syslog info to local6.debug and the TLS session with Thunderbird and NO certs shows this: Jan 21 12:59:10 imapsrv imap[1518]: [ID 636471 local6.notice] TLS server engine: cannot load CA data Jan 21 12:59:10 imapsrv imap[1518]: [ID 286863 local6.notice] imapd:Loading hard-coded DH parameters Jan 21 12:59:10 imapsrv imap[1518]: [ID 277171 local6.error] TLS server engine: No CA file specified. Client side certs may not work Jan 21 12:59:10 imapsrv imap[1518]: [ID 574029 local6.debug] SSL_accept() incomplete -> wait Jan 21 12:59:10 imapsrv imap[1518]: [ID 192010 local6.debug] decryption failed or bad record mac in SSL_accept() -> fail Jan 21 12:59:10 imapsrv imap[1518]: [ID 239158 local6.notice] STARTTLS negotiation failed: myclient.our.com [xx.xx.6.52] Sebastian Hagedorn wrote: > Hi Jeff, > > --On 21. Januar 2009 11:19:31 -0500 Jeff Blaine > wrote: > >> Sorry for the delay -- I had my wedding and a brief >> mini-honeymoon to attend to ;) > > congrats! > >>> How about Thunderbird using a password for authentication? Is that an >>> option at all? >> >> I realize this is a little "all over the road" here, >> but bear with me as I am just trying to get something >> working at this point for our users who are now >> without secure IMAP :( >> >> With "TLS" selected in Thunderbird, I am given no >> choice but to select a client certificate. See >> attached images. > > I wonder why that is. The only reason that comes to mind is that you > *have* a certificate. I don't and so I'm never asked to use it. So why > don't you try removing your certificate? Honestly, I would expect the > same to happen that happens when you use SSL, but you never know. > >> Another user reports that GNU Emacs with the Gnus >> client works with SSL and port 993. I've confirmed >> this in the log: >> >> Jan 21 11:11:03 imapsrv imaps[14170]: [ID 277583 local6.notice] login: >> jimbo-host.our.com [xx.xx.50.67] jimbo plaintext+TLS User logged in >> >> If I configure Thunderbird to do that (SSL via 993), >> I get the following: >> >> Jan 21 11:10:19 imapsrv imaps[14104]: [ID 636471 local6.notice] TLS >> server engine: cannot load CA data >> Jan 21 11:10:19 imapsrv imaps[14104]: [ID 286863 local6.notice] >> imapd:Loading hard-coded DH parameters >> Jan 21 11:10:19 imapsrv imaps[14104]: [ID 798856 local6.notice] imaps TLS >> negotiation failed: myclient.our.com >> Jan 21 11:10:19 imapsrv imaps[14104]: [ID 637875 local6.error] Fatal >> error: tls_start_servertls() failed > > I have no idea why that happens. I just tried it myself and got the > following in our log: > > Jan 21 18:17:48 lvr13 imaps[9855]: accepted connection > Jan 21 18:17:48 lvr13 imaps[9855]: SSL_accept() incomplete -> wait > Jan 21 18:17:48 lvr13 imaps[9855]: SSL_accept() succeeded -> done > Jan 21 18:17:48 lvr13 imaps[9855]: starttls: TLSv1 with cipher > AES256-SHA (256/256 bits new) no authentication > Jan 21 18:17:53 lvr13 imaps[9855]: login: [redacted] User logged in > > Could it be that your OpenSSL version or your certificate somehow don't > support features that Thunderbird requires? I'm really no expert, but I > know that client and server *negotiate* about these things. And the > error reads "negotiation failed" ... > > If your server is accessible over the Internet, perhaps I could try > connecting to it with "openssl s_client". That might tell us something. > You can try that as well, of course. From kbah at linuxmail.org Wed Jan 21 12:01:14 2009 From: kbah at linuxmail.org (k bah) Date: Wed, 21 Jan 2009 14:01:14 -0300 Subject: Lmtp w/ Postfix and LDAP: change mailbox name when saving mail. Message-ID: <20090121170114.3A037CBBD1@ws5-11.us4.outblaze.com> I need to know then how things are processed when mail arrives, from the Cyrus point of view, of course. First Postfix searches the user on LDAP, like this: query_filter=(&(|(&(objectclass=postfixuser)(objectclass=inetlocalmailrecipient)) (objectclass=groupofuniquenames))(mail=%s)) result_attribute=mailRoutingAddress on my LDAP server the fields are: mail = username at domain.org uid = username~domain.org mailRoutingAddress = username~domain.org at domain.org (uid is the "user part") My question is, how Cyrus decides what is the mailbox name, from what Postfix gives or it just looks the user part of the email? I mean, Cyrus knows the destination email is username at domain.org, then it assumes the mailbox name is "username"? Or it uses the "user part" of whatever Postfix gave(?), because if this is true, Cyrus should be using the user part of the mailroutingaddress field..? Right now Postfix is using Dovecot LDA, and the mailbox names are as expected. I'm trying to understand how Cyrus decides the mailbox name, how can I debug it? I think Postfix is correctly configured to what I want it to do (mailbox name = uid field from LDAP or "user part of mailroutingaddress"). thanks again > ----- Original Message ----- > From: "Dan White" > To: "k bah" > Cc: Info-cyrus at lists.andrew.cmu.edu > Subject: Re: Lmtp w/ Postfix and LDAP: change mailbox name when saving mail. > Date: Wed, 21 Jan 2009 09:16:08 -0600 > > > Dan White wrote: > > > > sasl_ldapdb_uri: ldap://ldap.example.org > > sasl_ldapdb_mech: GSSAPI > > sasl_ldapdb_canon_attr: uid > > #sasl_canon_user_plugin: ladpdb > > #imap_sasl_canon_user_plugin: ldapdb > > #pop3_sasl_canon_user_plugin: ldapdb > > #nntp_sasl_canon_user_plugin: ldapdb > > #sieve_sasl_canon_user_plugin: ldapdb > > lmtp_sasl_canon_user_plugin: ldapdb > > > > > > On second thought, this probably won't do anything useful, since > the user is not authenticating to LMTP. > > A Postfix regex map (or LDAP map) may be able to convert the > recipient before it gets handed off to LMTP. > > - Dan = Antennas.us Store Antennas 100 kHz to 26 GHz (COTS) Commercial- Industrial-Government. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=687e4b3ea622e686069e429cb0ff5733 -- Powered by Outblaze From jblaine at kickflop.net Wed Jan 21 13:36:02 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Wed, 21 Jan 2009 13:36:02 -0500 Subject: Expire (manually) TLS sessions? In-Reply-To: <49776338.5060803@kickflop.net> References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> <49707B0E.3000007@kickflop.net> <49708212.4010409@kickflop.net> <497088B4.5080707@kickflop.net> <49709CF6.9040609@kickflop.net> <0EAD03513BDEB07A363D29A5@tyrion.rrz.uni-koeln.de> <4970A3C0.1050407@kickflop.net> <4970A597.4050808@kickflop.net> <1D1C8ECCD65FCB3DC2CAF6E7@G5.voip.local> <49774B13.5070404@kickflop.net> <49776338.5060803@kickflop.net> Message-ID: <49776B12.2040206@kickflop.net> Also: bash-2.05# su cyrus -c "/imapsrv/mail/cyrus/bin/imtest -t /var/imap/server.pem imapsrv" S: * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID STARTTLS AUTH=PLAIN SASL-IR] imapsrv.our.com Cyrus IMAP v2.3.13 server ready C: S01 STARTTLS S: S01 OK Begin TLS negotiation now verify error:num=20:unable to get local issuer certificate verify error:num=27:certificate not trusted verify error:num=21:unable to verify the first certificate SSL_connect error 0 SSL session removed failure: TLS negotiation failed! bash-2.05# Jeff Blaine wrote: > I raised syslog info to local6.debug and the TLS session with > Thunderbird and NO certs shows this: > > Jan 21 12:59:10 imapsrv imap[1518]: [ID 636471 local6.notice] TLS server > engine: cannot load CA data > Jan 21 12:59:10 imapsrv imap[1518]: [ID 286863 local6.notice] > imapd:Loading hard-coded DH parameters > Jan 21 12:59:10 imapsrv imap[1518]: [ID 277171 local6.error] TLS server > engine: No CA file specified. Client side certs may not work > Jan 21 12:59:10 imapsrv imap[1518]: [ID 574029 local6.debug] > SSL_accept() incomplete -> wait > Jan 21 12:59:10 imapsrv imap[1518]: [ID 192010 local6.debug] decryption > failed or bad record mac in SSL_accept() -> fail > Jan 21 12:59:10 imapsrv imap[1518]: [ID 239158 local6.notice] STARTTLS > negotiation failed: myclient.our.com [xx.xx.6.52] > > Sebastian Hagedorn wrote: >> Hi Jeff, >> >> --On 21. Januar 2009 11:19:31 -0500 Jeff Blaine >> wrote: >> >>> Sorry for the delay -- I had my wedding and a brief >>> mini-honeymoon to attend to ;) >> congrats! >> >>>> How about Thunderbird using a password for authentication? Is that an >>>> option at all? >>> I realize this is a little "all over the road" here, >>> but bear with me as I am just trying to get something >>> working at this point for our users who are now >>> without secure IMAP :( >>> >>> With "TLS" selected in Thunderbird, I am given no >>> choice but to select a client certificate. See >>> attached images. >> I wonder why that is. The only reason that comes to mind is that you >> *have* a certificate. I don't and so I'm never asked to use it. So why >> don't you try removing your certificate? Honestly, I would expect the >> same to happen that happens when you use SSL, but you never know. >> >>> Another user reports that GNU Emacs with the Gnus >>> client works with SSL and port 993. I've confirmed >>> this in the log: >>> >>> Jan 21 11:11:03 imapsrv imaps[14170]: [ID 277583 local6.notice] login: >>> jimbo-host.our.com [xx.xx.50.67] jimbo plaintext+TLS User logged in >>> >>> If I configure Thunderbird to do that (SSL via 993), >>> I get the following: >>> >>> Jan 21 11:10:19 imapsrv imaps[14104]: [ID 636471 local6.notice] TLS >>> server engine: cannot load CA data >>> Jan 21 11:10:19 imapsrv imaps[14104]: [ID 286863 local6.notice] >>> imapd:Loading hard-coded DH parameters >>> Jan 21 11:10:19 imapsrv imaps[14104]: [ID 798856 local6.notice] imaps TLS >>> negotiation failed: myclient.our.com >>> Jan 21 11:10:19 imapsrv imaps[14104]: [ID 637875 local6.error] Fatal >>> error: tls_start_servertls() failed >> I have no idea why that happens. I just tried it myself and got the >> following in our log: >> >> Jan 21 18:17:48 lvr13 imaps[9855]: accepted connection >> Jan 21 18:17:48 lvr13 imaps[9855]: SSL_accept() incomplete -> wait >> Jan 21 18:17:48 lvr13 imaps[9855]: SSL_accept() succeeded -> done >> Jan 21 18:17:48 lvr13 imaps[9855]: starttls: TLSv1 with cipher >> AES256-SHA (256/256 bits new) no authentication >> Jan 21 18:17:53 lvr13 imaps[9855]: login: [redacted] User logged in >> >> Could it be that your OpenSSL version or your certificate somehow don't >> support features that Thunderbird requires? I'm really no expert, but I >> know that client and server *negotiate* about these things. And the >> error reads "negotiation failed" ... >> >> If your server is accessible over the Internet, perhaps I could try >> connecting to it with "openssl s_client". That might tell us something. >> You can try that as well, of course. > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > From awilliam at whitemice.org Wed Jan 21 13:37:28 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Wed, 21 Jan 2009 13:37:28 -0500 Subject: Lmtp w/ Postfix and LDAP: change mailbox name when saving mail. In-Reply-To: <20090121170114.3A037CBBD1@ws5-11.us4.outblaze.com> References: <20090121170114.3A037CBBD1@ws5-11.us4.outblaze.com> Message-ID: <1232563048.4363.8.camel@linux-nnci.site> On Wed, 2009-01-21 at 14:01 -0300, k bah wrote: > I need to know then how things are processed when mail arrives, from the Cyrus point of view, of course. > First Postfix searches the user on LDAP, like this: > query_filter=(&(|(&(objectclass=postfixuser)(objectclass=inetlocalmailrecipient)) (objectclass=groupofuniquenames))(mail=%s)) > result_attribute=mailRoutingAddres) > on my LDAP server the fields are: > mail = username at domain.org > uid = username~domain.org > mailRoutingAddress = username~domain.org at domain.org (uid is the "user part") > My question is, how Cyrus decides what is the mailbox name, from what Postfix gives or it just looks the user part of the email? I mean, Cyrus knows the destination email is username at domain.org, then it assumes the mailbox name is "username"? Or it uses the "user part" of whatever Postfix gave(?), because if this is true, Cyrus should be using the user part of the mailroutingaddress field..? If your MTA (Postfix) is delivering via LMTP then Cyrus does what it is told. The mailbox is determined by Postfix. We use the "mailRoutingAddress" schema you describe above; the user portion of the e-mail address is completely irrelevant. If you have ... mailLoadAddress: fred.t.smith at example.com mailRoutingAddress: fred ... Then mail to "fred.t.smith at example.com" should get stuffed into the INBOX "user.fred". -- Adam Tauno Williams, Network & Systems Administrator Consultant - http://www.whitemiceconsulting.com Developer - http://www.opengroupware.org From wes at umich.edu Wed Jan 21 13:44:08 2009 From: wes at umich.edu (Wesley Craig) Date: Wed, 21 Jan 2009 13:44:08 -0500 Subject: Expire (manually) TLS sessions? In-Reply-To: <49775E4B.8080805@kickflop.net> References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> <49707B0E.3000007@kickflop.net> <49708212.4010409@kickflop.net> <497088B4.5080707@kickflop.net> <49709CF6.9040609@kickflop.net> <0EAD03513BDEB07A363D29A5@tyrion.rrz.uni-koeln.de> <4970A3C0.1050407@kickflop.net> <4970A597.4050808@kickflop.net> <1D1C8ECCD65FCB3DC2CAF6E7@G5.voip.local> <49774B13.5070404@kickflop.net> <49775E4B.8080805@kickflop.net> Message-ID: On 21 Jan 2009, at 12:41, Jeff Blaine wrote: > FWIW, Thunderbird with SSL on port 993 pops up a box saying > incorrect Message authentication code. I forgot to mention > that. That's interesting. What platform are you running this on? What compiler did you use to build openssl? What optimization flags did you use? (You might try building openssl with reduced optimization and see what results you get.) :wes From kbah at linuxmail.org Wed Jan 21 14:28:38 2009 From: kbah at linuxmail.org (k bah) Date: Wed, 21 Jan 2009 16:28:38 -0300 Subject: Lmtp w/ Postfix and LDAP: change mailbox name when saving mail. Message-ID: <20090121192838.A1AF2233F3@ws5-3.us4.outblaze.com> > ----- Original Message ----- > From: "Adam Tauno Williams" > To: "Cyrus Mailing List" > Subject: Re: Lmtp w/ Postfix and LDAP: change mailbox name when saving mail. > Date: Wed, 21 Jan 2009 13:37:28 -0500 > > > On Wed, 2009-01-21 at 14:01 -0300, k bah wrote: > > I need to know then how things are processed when mail arrives, > > from the Cyrus point of view, of course. > > First Postfix searches the user on LDAP, like this: > > > > query_filter=(&(|(&(objectclass=postfixuser)(objectclass=inetlocalmailrecipient)) > > (objectclass=groupofuniquenames))(mail=%s)) > > result_attribute=mailRoutingAddres) > > on my LDAP server the fields are: > > mail = username at domain.org > > uid = username~domain.org > > mailRoutingAddress = username~domain.org at domain.org (uid is the > > "user part") > > My question is, how Cyrus decides what is the mailbox name, from > > what Postfix gives or it just looks the user part of the email? I > > mean, Cyrus knows the destination email is username at domain.org, > > then it assumes the mailbox name is "username"? Or it uses the > > "user part" of whatever Postfix gave(?), because if this is true, > > Cyrus should be using the user part of the mailroutingaddress > > field..? > > If your MTA (Postfix) is delivering via LMTP then Cyrus does what it is > told. The mailbox is determined by Postfix. We use the > "mailRoutingAddress" schema you describe above; the user portion of the > e-mail address is completely irrelevant. If you have > > ... > mailLoadAddress: fred.t.smith at example.com > mailRoutingAddress: fred > ... > > Then mail to "fred.t.smith at example.com" should get stuffed into the > INBOX "user.fred". > -- Thanks. I started some of Postfix daemons with "-v" and noticed mailroutingaddress shows up on the logs when using Dovecot LDA, but not when Postfix is configured to use Cyrus lmtp socket. From this I started to think the issue is with Postfix config, this along with what you said points to exactly it: Postfix config. I'm gonna (re)check/study the Postfix queues order and address the problem to the Postfix mailing list. thanks! = Slatwall - Guaranteed Low Prices Save Money. Buy Slatwall Direct From The Manufacturer. Ships Same Day. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=1eaeab72fe25701fdf05fbc6900fc46f -- Powered by Outblaze From jblaine at kickflop.net Wed Jan 21 15:15:46 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Wed, 21 Jan 2009 15:15:46 -0500 Subject: Expire (manually) TLS sessions? In-Reply-To: References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> <49707B0E.3000007@kickflop.net> <49708212.4010409@kickflop.net> <497088B4.5080707@kickflop.net> <49709CF6.9040609@kickflop.net> <0EAD03513BDEB07A363D29A5@tyrion.rrz.uni-koeln.de> <4970A3C0.1050407@kickflop.net> <4970A597.4050808@kickflop.net> <1D1C8ECCD65FCB3DC2CAF6E7@G5.voip.local> <49774B13.5070404@kickflop.net> <49775E4B.8080805@kickflop.net> Message-ID: <49778272.3000809@kickflop.net> This is fixed! I built OpenSSL 0.9.8j on a whim and noticed it fixed a bizarre problem with OpenSSH 5.1p1 that I was working on in a pre-production environment. We were using an older 0.9.8 rev prior to this. I then did a 'make clean; make; sudo make install' in my cyrus-sasl source tree (which links against the static openssl libs), and did the same for the cyrus-imapd tree. Beats me! Jan 21 15:10:20 imapsrv imap[9928]: [ID 574029 local6.debug] SSL_accept() incomplete -> wait Jan 21 15:10:20 imapsrv imap[9928]: [ID 867439 local6.debug] SSL_accept() succeeded -> done Jan 21 15:10:20 imapsrv imap[9928]: [ID 379946 local6.notice] starttls: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits new) no authentication Jan 21 15:10:20 imapsrv imap[9928]: [ID 529592 local6.notice] login: myclient.our.com [xx.xx.6.52] jblaine plain+TLS User logged in Wesley Craig wrote: > On 21 Jan 2009, at 12:41, Jeff Blaine wrote: >> FWIW, Thunderbird with SSL on port 993 pops up a box saying >> incorrect Message authentication code. I forgot to mention >> that. > > That's interesting. What platform are you running this on? What > compiler did you use to build openssl? What optimization flags did you > use? (You might try building openssl with reduced optimization and see > what results you get.) > > :wes > From list at joreybump.com Wed Jan 21 15:27:54 2009 From: list at joreybump.com (Jorey Bump) Date: Wed, 21 Jan 2009 15:27:54 -0500 Subject: Expire (manually) TLS sessions? In-Reply-To: <49776B12.2040206@kickflop.net> References: <497071B3.1010006@kickflop.net> <4F9FCBE28CB223CB9D5B0950@tyrion.rrz.uni-koeln.de> <4970764E.5080209@kickflop.net> <49707B0E.3000007@kickflop.net> <49708212.4010409@kickflop.net> <497088B4.5080707@kickflop.net> <49709CF6.9040609@kickflop.net> <0EAD03513BDEB07A363D29A5@tyrion.rrz.uni-koeln.de> <4970A3C0.1050407@kickflop.net> <4970A597.4050808@kickflop.net> <1D1C8ECCD65FCB3DC2CAF6E7@G5.voip.local> <49774B13.5070404@kickflop.net> <49776338.5060803@kickflop.net> <49776B12.2040206@kickflop.net> Message-ID: <4977854A.9040502@joreybump.com> Jeff Blaine wrote, at 01/21/2009 01:36 PM: > bash-2.05# su cyrus -c "/imapsrv/mail/cyrus/bin/imtest -t > /var/imap/server.pem imapsrv" My understanding is that you only specify a keyfile if you're testing client certificate authentication. For a normal test of TLS encryption, it should be empty (but quoted): imtest -u bob -a bob -t "" mail.example.com You'll still see this: > S: * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID STARTTLS AUTH=PLAIN > SASL-IR] imapsrv.our.com Cyrus IMAP v2.3.13 server ready > C: S01 STARTTLS > S: S01 OK Begin TLS negotiation now > verify error:num=20:unable to get local issuer certificate > verify error:num=27:certificate not trusted > verify error:num=21:unable to verify the first certificate But you shouldn't see this: > SSL_connect error 0 > SSL session removed > failure: TLS negotiation failed! If it works, you'll see this instead: TLS connection established: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits) C: C01 CAPABILITY ... BTW, you probably shouldn't be advertising AUTH=PLAIN pre-STARTTLS. Try something like this in imapd.conf, adjusted for the mechanisms you support: # authentication sasl_pwcheck_method: auxprop sasl_mech_list: PLAIN LOGIN CRAM-MD5 DIGEST-MD5 allowplaintext: no # use this to enforce TLS with plaintext mechanisms sasl_minimum_layer: 128 From ian.batten at uk.fujitsu.com Thu Jan 22 12:31:44 2009 From: ian.batten at uk.fujitsu.com (Ian Batten) Date: Thu, 22 Jan 2009 17:31:44 +0000 Subject: enforcing TLS certificates for replication Message-ID: <712F52FA-900A-4B6E-8A59-5A7CAFA917B3@uk.fujitsu.com> With my private network hat on, I have a requirement to secure replication. I have one machine in a data centre which runs 2.3.13 on Solaris 10. I've recently brought up an Open Solaris machine at home, similarly running 2.3.13, with a static IP number and an appropriate hole in the firewall to run replication. Which is all good, but I'm not at all sure how good my ISP is at preventing Bad People from mis- using IP numbers, so I'd like to require the sync_server to offer a certificate to prove its good will to the sync_client. I assume I can do it, but what are the options? ian From wes at umich.edu Thu Jan 22 12:57:30 2009 From: wes at umich.edu (Wesley Craig) Date: Thu, 22 Jan 2009 12:57:30 -0500 Subject: enforcing TLS certificates for replication In-Reply-To: <712F52FA-900A-4B6E-8A59-5A7CAFA917B3@uk.fujitsu.com> References: <712F52FA-900A-4B6E-8A59-5A7CAFA917B3@uk.fujitsu.com> Message-ID: <9C5EB68F-717F-4C35-BEBD-C3E4D7F3F13E@umich.edu> On 22 Jan 2009, at 12:31, Ian Batten wrote: > With my private network hat on, I have a requirement to secure > replication. I have one machine in a data centre which runs 2.3.13 on > Solaris 10. I've recently brought up an Open Solaris machine at home, > similarly running 2.3.13, with a static IP number and an appropriate > hole in the firewall to run replication. Which is all good, but I'm > not at all sure how good my ISP is at preventing Bad People from mis- > using IP numbers, so I'd like to require the sync_server to offer a > certificate to prove its good will to the sync_client. I assume I can > do it, but what are the options? If the sync_server isn't allowed to accept clear text passwords and is configured to provide certificates, you should be all set. sync_server supports STARTTLS with the same routines as everything else, sync_client is using the same backend_connect() routine that everything else uses. It should "Just Work". :wes From sebelk at gmail.com Fri Jan 23 08:28:16 2009 From: sebelk at gmail.com (Sergio Belkin) Date: Fri, 23 Jan 2009 11:28:16 -0200 Subject: Folders created by hand are not seen by cyradm Message-ID: <8c6f7f450901230528w72a4259dl2df108bbed6218ee@mail.gmail.com> Hi, I created folders Trash, Sent, Queue, Draft, for a mailbox user, assigned permissions and owner for cyrus, recontructed, but still cyrus can't see them. Please, what can I do?? Thanks in advance! -- -- Open Kairos http://www.openkairos.com Watch More TV http://sebelk.blogspot.com Sergio Belkin - From awilliam at whitemice.org Fri Jan 23 08:31:08 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Fri, 23 Jan 2009 08:31:08 -0500 Subject: Folders created by hand are not seen by cyradm In-Reply-To: <8c6f7f450901230528w72a4259dl2df108bbed6218ee@mail.gmail.com> References: <8c6f7f450901230528w72a4259dl2df108bbed6218ee@mail.gmail.com> Message-ID: <1232717468.5449.1.camel@linux-nnci.site> > I created folders Trash, Sent, Queue, Draft, for a mailbox user, > assigned permissions and owner for cyrus, recontructed, > but still cyrus can't see them. Please, what can I do?? What does "created by hand" mean? You modified the filesystem directly in the Cyrus spool? Do not do that. If you did that: delete the folders and create the folders via cyradm or an IMAP client. From cristian at seres.fi Mon Jan 26 11:25:06 2009 From: cristian at seres.fi (Cristian Seres) Date: Mon, 26 Jan 2009 18:25:06 +0200 (EET) Subject: Sync_client retrying, but synctest works Message-ID: <26298.84.20.148.100.1232987106.squirrel@s3.bitpoint.fi> Hi! I have tried to get Cyrus replication working without success. It is RHEL 5.3 with standard cyrus-imap 2.3.7-2. Synctest with -t "" succeeds, but sync_client -v -r gives "retrying in 15 seconds". Tcpdump shows that there is traffic in both cases. Any help is appreciated. sync_ lines in imapd.conf on imapmaster host: *** sync_host: imapreplica.company.com sync_authname: cyrus sync_passwd: protected *** The following does not work: *** cyrus at imapmaster$ /usr/lib/cyrus-imapd/sync_client -v -r Can not connect to server 'imapreplica.company.com', retrying in 15 seconds *** The following works: *** cyrus at imapmaster$ synctest imapreplica.company.com -t "" S: * STARTTLS S: * OK imapreplica.company.com Cyrus sync server v2.3.7-Invoca-RPM-2.3.7-2.el5 C: STARTTLS S: OK Begin TLS negotiation now verify error:num=18:self signed certificate TLS connection established: TLSv1 with cipher AES256-SHA (256/256 bits) S: * SASL LOGIN PLAIN S: * OK imapreplica.company.com Cyrus sync server v2.3.7-Invoca-RPM-2.3.7-2.el5 C: AUTHENTICATE LOGIN S: + VXNlcm5hbWU6 Please enter your password: C: S: + C: S: OK Success (tls protection) Authenticated. Security strength factor: 256 *** With best regards, -- Cristian Seres From Duncan.Gibb at SiriusIT.co.uk Tue Jan 27 06:29:00 2009 From: Duncan.Gibb at SiriusIT.co.uk (Duncan Gibb) Date: Tue, 27 Jan 2009 11:29:00 +0000 Subject: enforcing TLS certificates for replication In-Reply-To: <9C5EB68F-717F-4C35-BEBD-C3E4D7F3F13E@umich.edu> References: <712F52FA-900A-4B6E-8A59-5A7CAFA917B3@uk.fujitsu.com> <9C5EB68F-717F-4C35-BEBD-C3E4D7F3F13E@umich.edu> Message-ID: <497EEFFC.1090901@SiriusIT.co.uk> Last week, Wesley Craig wrote: IB> I have one machine in a data centre which runs 2.3.13 IB> [and a] machine at home, similarly running 2.3.13, IB> with a static IP number and an appropriate hole in the IB> firewall to run replication. Which is all good, but IB> I'm not at all sure how good my ISP is at preventing IB> Bad People from mis-using IP numbers, so I'd like to IB> require the sync_server to offer a certificate to prove IB> its good will to the sync_client. WC> If the sync_server isn't allowed to accept clear text WC> passwords and is configured to provide certificates, WC> you should be all set. WC> [..] It should "Just Work". If you want to try also using certificates to authenticate the client to the server, you might like to look at my patch - thus far only tested for traditional murder FE->BE and FE/BE->MUPDATE authentication: https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3133 Cheers Duncan -- Duncan Gibb, Technical Director Sirius Corporation plc - The Open Source Experts http://www.siriusit.co.uk/ Tel: +44 870 608 0063 From mathieu.kretchner at sophia.inria.fr Tue Jan 27 13:04:02 2009 From: mathieu.kretchner at sophia.inria.fr (Mathieu Kretchner) Date: Tue, 27 Jan 2009 19:04:02 +0100 Subject: PLAIN authentication timsieved Message-ID: <497F4C92.3050507@sophia.inria.fr> Hello, I would like to allow connection to sieved server with PLAIN mechanism. But my configuration seems to already have this. What do I miss ? Cyrus is 2.2.12 here is my imapd.conf : configdirectory: /data/imap partition-default: /data/imap/spool servername: imap-sop.inria.fr admins: cyrus hashimapspool: yes duplicatesuppression: no sasl_pwcheck_method: saslauthd allowanonymouslogin: no tls_session_timeout: 0 allowapop: 0 sasl_mech_list: PLAIN sieveuserhomedir: no sievedir: /data/imap/sieve sieve_maxscripts: 8 sieve_maxscriptsize: 640 sendmail: /usr/sbin/sendmail tls_ca_file: /data/imap/ssl/ca.crt tls_cert_file: /data/imap/ssl/server.crt tls_key_file: /data/imap/ssl/server.key tls_ca_path: /data/imap/ssl Thank you -------------- next part -------------- A non-text attachment was scrubbed... Name: mathieu_kretchner.vcf Type: text/x-vcard Size: 258 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090127/ad6a043f/attachment.vcf From rjaffey at artic.edu Tue Jan 27 14:09:25 2009 From: rjaffey at artic.edu (Raphael Jaffey) Date: Tue, 27 Jan 2009 13:09:25 -0600 Subject: PLAIN authentication timsieved In-Reply-To: <497F4C92.3050507@sophia.inria.fr> References: <497F4C92.3050507@sophia.inria.fr> Message-ID: <20090127130925.mq60te1xmoo8ggsw@webmail-test.artic.edu> sieve_allowplaintext: yes Quoting Mathieu Kretchner : > Hello, > > I would like to allow connection to sieved server with PLAIN mechanism. > But my configuration seems to already have this. What do I miss ? > > Cyrus is 2.2.12 > here is my imapd.conf : > > configdirectory: /data/imap > partition-default: /data/imap/spool > servername: imap-sop.inria.fr > admins: cyrus > hashimapspool: yes > duplicatesuppression: no > sasl_pwcheck_method: saslauthd > allowanonymouslogin: no > tls_session_timeout: 0 > allowapop: 0 > sasl_mech_list: PLAIN > sieveuserhomedir: no > sievedir: /data/imap/sieve > sieve_maxscripts: 8 > sieve_maxscriptsize: 640 > sendmail: /usr/sbin/sendmail > tls_ca_file: /data/imap/ssl/ca.crt > tls_cert_file: /data/imap/ssl/server.crt > tls_key_file: /data/imap/ssl/server.key > tls_ca_path: /data/imap/ssl > > Thank you > > From rjaffey at artic.edu Tue Jan 27 14:42:53 2009 From: rjaffey at artic.edu (Raphael Jaffey) Date: Tue, 27 Jan 2009 13:42:53 -0600 Subject: PLAIN authentication timsieved In-Reply-To: <20090127130925.mq60te1xmoo8ggsw@webmail-test.artic.edu> References: <497F4C92.3050507@sophia.inria.fr> <20090127130925.mq60te1xmoo8ggsw@webmail-test.artic.edu> Message-ID: <20090127134253.ipvjpv9og4k40k8k@webmail-test.artic.edu> Sorry, we use this setting in our environment as we're using stunnel for sieved connections rather than its built in TLS support. The relevant parts of our current config read: sasl_pwcheck_method: saslauthd sasl_mech_list: PLAIN allowplaintext: no sasl_minimum_layer: 128 sieve_allowplaintext: yes sieve_sasl_minimum_layer: 0 tls_cert_file: tls_key_file: tls_ca_file: tls_cipher_list: !ADH:MEDIUM:HIGH sieve_tls_cert_file: disabled In your case, assuming you don't want PLAIN in the clear, I should think the following would suffice: sasl_pwcheck_method: saslauthd sasl_mech_list: PLAIN allowplaintext: no sasl_minimum_layer: 128 tls_cert_file: tls_key_file: tls_ca_file: tls_cipher_list: !ADH:MEDIUM:HIGH This assumes that your sieve client supports TLS. Quoting Raphael Jaffey : > sieve_allowplaintext: yes > > Quoting Mathieu Kretchner : > >> Hello, >> >> I would like to allow connection to sieved server with PLAIN mechanism. >> But my configuration seems to already have this. What do I miss ? >> >> Cyrus is 2.2.12 >> here is my imapd.conf : >> >> configdirectory: /data/imap >> partition-default: /data/imap/spool >> servername: imap-sop.inria.fr >> admins: cyrus >> hashimapspool: yes >> duplicatesuppression: no >> sasl_pwcheck_method: saslauthd >> allowanonymouslogin: no >> tls_session_timeout: 0 >> allowapop: 0 >> sasl_mech_list: PLAIN >> sieveuserhomedir: no >> sievedir: /data/imap/sieve >> sieve_maxscripts: 8 >> sieve_maxscriptsize: 640 >> sendmail: /usr/sbin/sendmail >> tls_ca_file: /data/imap/ssl/ca.crt >> tls_cert_file: /data/imap/ssl/server.crt >> tls_key_file: /data/imap/ssl/server.key >> tls_ca_path: /data/imap/ssl >> >> Thank you >> >> > > > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > From mathieu.kretchner at sophia.inria.fr Wed Jan 28 04:53:53 2009 From: mathieu.kretchner at sophia.inria.fr (Mathieu Kretchner) Date: Wed, 28 Jan 2009 10:53:53 +0100 Subject: PLAIN authentication timsieved In-Reply-To: <20090127134253.ipvjpv9og4k40k8k@webmail-test.artic.edu> References: <497F4C92.3050507@sophia.inria.fr> <20090127130925.mq60te1xmoo8ggsw@webmail-test.artic.edu> <20090127134253.ipvjpv9og4k40k8k@webmail-test.artic.edu> Message-ID: <49802B31.50506@sophia.inria.fr> Ok thank you for your help, I've tried the allowplaintext: yes but the proxy sieve server I use is still complaining ! I don't know why? I've done a tcp/ip trace of data transmission between proxy and sieve cyrus server and the only thing I see is that : Data (41 bytes) 0000 41 55 54 48 45 4e 54 49 43 41 54 45 20 22 50 4c AUTHENTICATE "PL 0010 41 49 4e 22 20 22 41 47 6c 74 59 58 41 79 41 47 AIN" "AGltYXAyAG 0020 6c 74 59 58 41 79 22 0d 0a ltYXAy".. Data (22 bytes) 0000 4f 4b 20 22 4c 6f 67 6f 75 74 20 43 6f 6d 70 6c OK "Logout Compl 0010 65 74 65 22 0d 0a ete".. How could I debug this ? Raphael Jaffey wrote: > Sorry, we use this setting in our environment as we're using stunnel > for sieved connections rather than its built in TLS support. > > The relevant parts of our current config read: > > sasl_pwcheck_method: saslauthd > sasl_mech_list: PLAIN > > > allowplaintext: no > sasl_minimum_layer: 128 > sieve_allowplaintext: yes > sieve_sasl_minimum_layer: 0 > > tls_cert_file: > tls_key_file: > tls_ca_file: > tls_cipher_list: !ADH:MEDIUM:HIGH > > sieve_tls_cert_file: disabled > > > > In your case, assuming you don't want PLAIN in the clear, I should > think the following would suffice: > > sasl_pwcheck_method: saslauthd > sasl_mech_list: PLAIN > > > allowplaintext: no > sasl_minimum_layer: 128 > > tls_cert_file: > tls_key_file: > tls_ca_file: > tls_cipher_list: !ADH:MEDIUM:HIGH > > This assumes that your sieve client supports TLS. > > Quoting Raphael Jaffey : > >> sieve_allowplaintext: yes >> >> Quoting Mathieu Kretchner : >> >>> Hello, >>> >>> I would like to allow connection to sieved server with PLAIN mechanism. >>> But my configuration seems to already have this. What do I miss ? >>> >>> Cyrus is 2.2.12 >>> here is my imapd.conf : >>> >>> configdirectory: /data/imap >>> partition-default: /data/imap/spool >>> servername: imap-sop.inria.fr >>> admins: cyrus >>> hashimapspool: yes >>> duplicatesuppression: no >>> sasl_pwcheck_method: saslauthd >>> allowanonymouslogin: no >>> tls_session_timeout: 0 >>> allowapop: 0 >>> sasl_mech_list: PLAIN >>> sieveuserhomedir: no >>> sievedir: /data/imap/sieve >>> sieve_maxscripts: 8 >>> sieve_maxscriptsize: 640 >>> sendmail: /usr/sbin/sendmail >>> tls_ca_file: /data/imap/ssl/ca.crt >>> tls_cert_file: /data/imap/ssl/server.crt >>> tls_key_file: /data/imap/ssl/server.key >>> tls_ca_path: /data/imap/ssl >>> >>> Thank you >>> >>> >> >> >> ---- >> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ >> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki >> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html >> > > > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -------------- next part -------------- A non-text attachment was scrubbed... Name: mathieu_kretchner.vcf Type: text/x-vcard Size: 258 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090128/ad8d8417/attachment.vcf From michael.menge at zdv.uni-tuebingen.de Wed Jan 28 05:53:40 2009 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Wed, 28 Jan 2009 11:53:40 +0100 Subject: PLAIN authentication timsieved In-Reply-To: <49802B31.50506@sophia.inria.fr> References: <497F4C92.3050507@sophia.inria.fr> <20090127130925.mq60te1xmoo8ggsw@webmail-test.artic.edu> <20090127134253.ipvjpv9og4k40k8k@webmail-test.artic.edu> <49802B31.50506@sophia.inria.fr> Message-ID: <20090128115340.16054wpjgziq28fo@webmail.uni-tuebingen.de> Hi, Quoting Mathieu Kretchner : > Ok thank you for your help, > > I've tried the allowplaintext: yes but the proxy sieve server I use is > still complaining ! > > I don't know why? I've done a tcp/ip trace of data transmission between > proxy and sieve cyrus server and the only thing I see is that : > > Data (41 bytes) > 0000 41 55 54 48 45 4e 54 49 43 41 54 45 20 22 50 4c AUTHENTICATE "PL > 0010 41 49 4e 22 20 22 41 47 6c 74 59 58 41 79 41 47 AIN" "AGltYXAyAG > 0020 6c 74 59 58 41 79 22 0d 0a ltYXAy".. > > Data (22 bytes) > 0000 4f 4b 20 22 4c 6f 67 6f 75 74 20 43 6f 6d 70 6c OK "Logout Compl > 0010 65 74 65 22 0d 0a ete".. > > > How could I debug this ? > > It seems your client tries PLAIN authentication. You can try sivtest, it will show all the data send between sieved and sivtest. Which database does saslauthd use for authentication, if you use pam you need an entry for sieve in /etc/pam.d . Are there any messages form saslauthd in the logs? > Raphael Jaffey wrote: >> Sorry, we use this setting in our environment as we're using stunnel >> for sieved connections rather than its built in TLS support. >> >> The relevant parts of our current config read: >> >> sasl_pwcheck_method: saslauthd >> sasl_mech_list: PLAIN >> >> >> allowplaintext: no >> sasl_minimum_layer: 128 >> sieve_allowplaintext: yes >> sieve_sasl_minimum_layer: 0 >> >> tls_cert_file: >> tls_key_file: >> tls_ca_file: >> tls_cipher_list: !ADH:MEDIUM:HIGH >> >> sieve_tls_cert_file: disabled >> >> >> >> In your case, assuming you don't want PLAIN in the clear, I should >> think the following would suffice: >> >> sasl_pwcheck_method: saslauthd >> sasl_mech_list: PLAIN >> >> >> allowplaintext: no >> sasl_minimum_layer: 128 >> >> tls_cert_file: >> tls_key_file: >> tls_ca_file: >> tls_cipher_list: !ADH:MEDIUM:HIGH >> >> This assumes that your sieve client supports TLS. >> >> Quoting Raphael Jaffey : >> >>> sieve_allowplaintext: yes >>> >>> Quoting Mathieu Kretchner : >>> >>>> Hello, >>>> >>>> I would like to allow connection to sieved server with PLAIN mechanism. >>>> But my configuration seems to already have this. What do I miss ? >>>> >>>> Cyrus is 2.2.12 >>>> here is my imapd.conf : >>>> >>>> configdirectory: /data/imap >>>> partition-default: /data/imap/spool >>>> servername: imap-sop.inria.fr >>>> admins: cyrus >>>> hashimapspool: yes >>>> duplicatesuppression: no >>>> sasl_pwcheck_method: saslauthd >>>> allowanonymouslogin: no >>>> tls_session_timeout: 0 >>>> allowapop: 0 >>>> sasl_mech_list: PLAIN >>>> sieveuserhomedir: no >>>> sievedir: /data/imap/sieve >>>> sieve_maxscripts: 8 >>>> sieve_maxscriptsize: 640 >>>> sendmail: /usr/sbin/sendmail >>>> tls_ca_file: /data/imap/ssl/ca.crt >>>> tls_cert_file: /data/imap/ssl/server.crt >>>> tls_key_file: /data/imap/ssl/server.key >>>> tls_ca_path: /data/imap/ssl >>>> >>>> Thank you >>>> >>>> >>> >>> >>> ---- >>> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ >>> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki >>> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html >>> >> >> >> >> ---- >> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ >> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki >> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From mathieu.kretchner at sophia.inria.fr Wed Jan 28 07:47:47 2009 From: mathieu.kretchner at sophia.inria.fr (Mathieu Kretchner) Date: Wed, 28 Jan 2009 13:47:47 +0100 Subject: PLAIN authentication timsieved In-Reply-To: <20090128115340.16054wpjgziq28fo@webmail.uni-tuebingen.de> References: <497F4C92.3050507@sophia.inria.fr> <20090127130925.mq60te1xmoo8ggsw@webmail-test.artic.edu> <20090127134253.ipvjpv9og4k40k8k@webmail-test.artic.edu> <49802B31.50506@sophia.inria.fr> <20090128115340.16054wpjgziq28fo@webmail.uni-tuebingen.de> Message-ID: <498053F3.4020107@sophia.inria.fr> It seems like I've no plain text auth capability !! [root at client ~]# telnet cyrus_server imap Trying cyrus_server ... Connected to imap-sop.inria.fr (cyrus_server). Escape character is '^]'. * OK cyrus_server Cyrus IMAP4 v2.2.12 server ready . capability * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ MAILBOX-REFERRALS NAMESPACE UIDPLUS ID NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE IDLE STARTTLS How could I configure this, like my other server do ? [root at client ~]# telnet test1 imap Trying 1.2.3.4... Connected to test1 (1.2.3.4). Escape character is '^]'. * OK INRIA mail server - test1 ready. . capability * CAPABILITY IMAP4rev1 SASL-IR SORT THREAD=REFERENCES MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS UIDPLUS LIST-EXTENDED I18NLEVEL=1 QUOTA STARTTLS *AUTH=PLAIN* Michael Menge wrote: > Hi, > > Quoting Mathieu Kretchner : > >> Ok thank you for your help, >> >> I've tried the allowplaintext: yes but the proxy sieve server I use is >> still complaining ! >> >> I don't know why? I've done a tcp/ip trace of data transmission between >> proxy and sieve cyrus server and the only thing I see is that : >> >> Data (41 bytes) >> 0000 41 55 54 48 45 4e 54 49 43 41 54 45 20 22 50 4c AUTHENTICATE "PL >> 0010 41 49 4e 22 20 22 41 47 6c 74 59 58 41 79 41 47 AIN" "AGltYXAyAG >> 0020 6c 74 59 58 41 79 22 0d 0a ltYXAy".. >> >> Data (22 bytes) >> 0000 4f 4b 20 22 4c 6f 67 6f 75 74 20 43 6f 6d 70 6c OK "Logout Compl >> 0010 65 74 65 22 0d 0a ete".. >> >> >> How could I debug this ? >> >> > > It seems your client tries PLAIN authentication. You can try sivtest, > it will show all the data send between sieved and sivtest. > Which database does saslauthd use for authentication, if you use pam > you need an entry for sieve in /etc/pam.d . > Are there any messages form saslauthd in the logs? > > > >> Raphael Jaffey wrote: >>> Sorry, we use this setting in our environment as we're using stunnel >>> for sieved connections rather than its built in TLS support. >>> >>> The relevant parts of our current config read: >>> >>> sasl_pwcheck_method: saslauthd >>> sasl_mech_list: PLAIN >>> >>> >>> allowplaintext: no >>> sasl_minimum_layer: 128 >>> sieve_allowplaintext: yes >>> sieve_sasl_minimum_layer: 0 >>> >>> tls_cert_file: >>> tls_key_file: >>> tls_ca_file: >>> tls_cipher_list: !ADH:MEDIUM:HIGH >>> >>> sieve_tls_cert_file: disabled >>> >>> >>> >>> In your case, assuming you don't want PLAIN in the clear, I should >>> think the following would suffice: >>> >>> sasl_pwcheck_method: saslauthd >>> sasl_mech_list: PLAIN >>> >>> >>> allowplaintext: no >>> sasl_minimum_layer: 128 >>> >>> tls_cert_file: >>> tls_key_file: >>> tls_ca_file: >>> tls_cipher_list: !ADH:MEDIUM:HIGH >>> >>> This assumes that your sieve client supports TLS. >>> >>> Quoting Raphael Jaffey : >>> >>>> sieve_allowplaintext: yes >>>> >>>> Quoting Mathieu Kretchner : >>>> >>>>> Hello, >>>>> >>>>> I would like to allow connection to sieved server with PLAIN mechanism. >>>>> But my configuration seems to already have this. What do I miss ? >>>>> >>>>> Cyrus is 2.2.12 >>>>> here is my imapd.conf : >>>>> >>>>> configdirectory: /data/imap >>>>> partition-default: /data/imap/spool >>>>> servername: imap-sop.inria.fr >>>>> admins: cyrus >>>>> hashimapspool: yes >>>>> duplicatesuppression: no >>>>> sasl_pwcheck_method: saslauthd >>>>> allowanonymouslogin: no >>>>> tls_session_timeout: 0 >>>>> allowapop: 0 >>>>> sasl_mech_list: PLAIN >>>>> sieveuserhomedir: no >>>>> sievedir: /data/imap/sieve >>>>> sieve_maxscripts: 8 >>>>> sieve_maxscriptsize: 640 >>>>> sendmail: /usr/sbin/sendmail >>>>> tls_ca_file: /data/imap/ssl/ca.crt >>>>> tls_cert_file: /data/imap/ssl/server.crt >>>>> tls_key_file: /data/imap/ssl/server.key >>>>> tls_ca_path: /data/imap/ssl >>>>> >>>>> Thank you >>>>> >>>>> >>>> >>>> ---- >>>> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ >>>> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki >>>> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html >>>> >>> >>> >>> ---- >>> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ >>> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki >>> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > > > > -------------------------------------------------------------------------------- > M.Menge Tel.: (49) 7071/29-70316 > Universit?t T?bingen Fax.: (49) 7071/29-5912 > Zentrum f?r Datenverarbeitung mail: > michael.menge at zdv.uni-tuebingen.de > W?chterstra?e 76 > 72074 T?bingen > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -------------- next part -------------- A non-text attachment was scrubbed... Name: mathieu_kretchner.vcf Type: text/x-vcard Size: 268 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090128/52c8b24b/attachment-0001.vcf From michael.menge at zdv.uni-tuebingen.de Wed Jan 28 08:37:37 2009 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Wed, 28 Jan 2009 14:37:37 +0100 Subject: PLAIN authentication timsieved In-Reply-To: <498053F3.4020107@sophia.inria.fr> References: <497F4C92.3050507@sophia.inria.fr> <20090127130925.mq60te1xmoo8ggsw@webmail-test.artic.edu> <20090127134253.ipvjpv9og4k40k8k@webmail-test.artic.edu> <49802B31.50506@sophia.inria.fr> <20090128115340.16054wpjgziq28fo@webmail.uni-tuebingen.de> <498053F3.4020107@sophia.inria.fr> Message-ID: <20090128143737.17773jc14bme43a9@webmail.uni-tuebingen.de> Quoting Mathieu Kretchner : > It seems like I've no plain text auth capability !! > Depending on you distribution plain authentication may be in a seperat package you have to install. -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5339 bytes Desc: S/MIME krytographische Unterschrift Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090128/dbaacd1a/attachment.bin From mathieu.kretchner at sophia.inria.fr Wed Jan 28 09:11:37 2009 From: mathieu.kretchner at sophia.inria.fr (Mathieu Kretchner) Date: Wed, 28 Jan 2009 15:11:37 +0100 Subject: PLAIN authentication timsieved In-Reply-To: <20090128143737.17773jc14bme43a9@webmail.uni-tuebingen.de> References: <497F4C92.3050507@sophia.inria.fr> <20090127130925.mq60te1xmoo8ggsw@webmail-test.artic.edu> <20090127134253.ipvjpv9og4k40k8k@webmail-test.artic.edu> <49802B31.50506@sophia.inria.fr> <20090128115340.16054wpjgziq28fo@webmail.uni-tuebingen.de> <498053F3.4020107@sophia.inria.fr> <20090128143737.17773jc14bme43a9@webmail.uni-tuebingen.de> Message-ID: <49806799.4010609@sophia.inria.fr> Ho sorry, I've done some others tests with sivtest and it's possible to connect with PLAIN. Michael Menge wrote: > > Quoting Mathieu Kretchner : > >> It seems like I've no plain text auth capability !! >> > > Depending on you distribution plain authentication may be in a seperat > package you have to install. > > > > -------------------------------------------------------------------------------- > > M.Menge Tel.: (49) 7071/29-70316 > Universit?t T?bingen Fax.: (49) 7071/29-5912 > Zentrum f?r Datenverarbeitung mail: > michael.menge at zdv.uni-tuebingen.de > W?chterstra?e 76 > 72074 T?bingen > > > ------------------------------------------------------------------------ > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -------------- next part -------------- A non-text attachment was scrubbed... Name: mathieu_kretchner.vcf Type: text/x-vcard Size: 258 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090128/f911a84f/attachment.vcf From jvoorhees1 at gmail.com Wed Jan 28 09:19:53 2009 From: jvoorhees1 at gmail.com (Jason Voorhees) Date: Wed, 28 Jan 2009 09:19:53 -0500 Subject: Considerations to migrate cyrus mailboxes Message-ID: Hi people: I'm running a CentOS 5 box with Cyrus IMAP 2.3.7. All mailboxes belong to a unique domain (no virtdomains setting) and also are using the unix hierarchy convention. Now I'm planning to migrate the mailserver to a virtual machine under OpenVZ and I want to migrate all mailbox data from the old server to the new one. But the new server -also running Cyrus and the same version- will be configured with virtdomains support, and the 'defaultdomain' will be different to the domain where I pretend to move old mailboxes. What considerations or steps must I follow? How can I convert old mailboxes (with no virtdomains support) to the new ones (with virtdomains support)? Will I need to copy all files from the Cyrus config directory and partition directory? I know that this task it could be done using imapsync between both imap servers... but.. is there any better option than this? I hope someone can help me a bit, thanks, bye :) From mhlavink at redhat.com Fri Jan 30 04:52:06 2009 From: mhlavink at redhat.com (Michal Hlavinka) Date: Fri, 30 Jan 2009 10:52:06 +0100 Subject: What is use-case for ctl_cyrusdb? man page has only too brief description Message-ID: <4982CDC6.70205@redhat.com> Hi, does anybody know, how ctl_cyrusdb works (how it should be used)? I've seen ctl_cyrusdb's man pages, but there is just few notes and nothing more. I don't need to use ctl_cyrusdb now, but I'd like to know more about this. I assume ctl_cyrusdb is used (in /etc/cyrus.conf) to generate "checkpoints" of the Cyrus databases, presumably those in /var/lib/imap. The information that man does not provide is 1) Listing of database files "checkpointed" 2) Description of checkpoint format a) "ls /var/lib/imap" suggest two sets of checkpoint database are retained. b) Description of checkpoint configuration options, if any, to control checkpoint sets 3) Checkpointed image backup requirements 4) Appropriate cross-references to utilities used to determine database corrution requiring recovery procedure 5) Appropriate recovery procedure when invocation with "-r" vs. "-r -x' is required. It'll be really appreciated if someone could explain at least some of these questions. Thanks question for developers: Could you please extend this man page? Regards, Michal Hlavinka From edwin at secureoffice.net Sat Jan 31 02:56:52 2009 From: edwin at secureoffice.net (Edwin Boersma) Date: Sat, 31 Jan 2009 08:56:52 +0100 Subject: Email bounces: mailbox does not exist Message-ID: <49840444.2020003@secureoffice.net> Hi, I have installed Cyrus 2.3.11 and it works fine together with Postfix. But there is something strange. I found these two things: 1. Cyrus doesn't seem to accept mailboxes with a dot in it, like "edwin.boersma". It then bounces the mail with a "Message contains invalid header" message: : data format error. Command output: edwin.boersma: Message contains invalid header It also says: Diagnostic-Code: x-unix; edwin.boersma: Message contains invalid header I changed to user name to "edwin_boersma" and then it works. Is this "works as designed" or just a plain bug? 2. If I send an email to 2 or more users on the same server, the message bounces with a "Mailbox does not exist" message. (expanded from ): data format error. Command output: : Mailbox does not exist (expanded from ): data format error. Command output: : Mailbox does not exist Both mailboxes are there, for sure. Sending to each user seperately works! -- Kind regards, Edwin Boersma Lead Developer Web Applications SecureOffice Europe AB Ideon Science Park B2 floor 2 Scheelev?gen 17 22363 Lund Sweden W: http://www.secureoffice.net T: +46 462868773 M: +46 709726431 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090131/e0dd6cd7/attachment.html From ml at awinkelmann.de Sat Jan 31 04:40:35 2009 From: ml at awinkelmann.de (Andreas Winkelmann) Date: Sat, 31 Jan 2009 10:40:35 +0100 (CET) Subject: Email bounces: mailbox does not exist In-Reply-To: <49840444.2020003@secureoffice.net> References: <49840444.2020003@secureoffice.net> Message-ID: > I have installed Cyrus 2.3.11 and it works fine together with Postfix. > But there is something strange. I found these two things: > > 1. Cyrus doesn't seem to accept mailboxes with a dot in it, like > "edwin.boersma". It then bounces the mail with a "Message contains > invalid header" message: > > : data format error. Command output: > edwin.boersma: Message contains invalid header > > It also says: > > Diagnostic-Code: x-unix; edwin.boersma: Message contains invalid header > > I changed to user name to "edwin_boersma" and then it works. > > Is this "works as designed" or just a plain bug? Did you enable unixhierarchysep in impad.conf? Without this the "." is a special Character in Cyrus. > 2. If I send an email to 2 or more users on the same server, the message > bounces with a "Mailbox does not exist" message. > > (expanded from > ): > data format error. Command output: : Mailbox does not exist > > (expanded from > ): > data format error. Command output: : Mailbox does not exist > > Both mailboxes are there, for sure. Sending to each user seperately works! You should not use deliver from Postfix. Deliver is just a wrapper between stdin and lmtp. Postfix can speak lmtp itself. -- Andreas