From dom.lalot at gmail.com Tue Sep 1 05:41:15 2009 From: dom.lalot at gmail.com (LALOT Dominique) Date: Tue, 1 Sep 2009 11:41:15 +0200 Subject: strange quota behaviour Message-ID: <1617f8010909010241i731bdedfm392389b963c4f4dd@mail.gmail.com> Hello, Sometimes, I have some bad quotas. Here is a trace: [cyrus at imap-perso ~]$ cyradm -user cyrus localhost Password: localhost.localdomain> lq user/xx STORAGE 197/1000000 (0.0197%) [root at imap-perso ~]# du -hs /var/spool/imap2/m/user/xx 890M /var/spool/imap2/m/user/xx reconstruct -rf user/xx [cyrus at imap-perso ~]$ quota -f user/xx user.xx: usage was 201755, now 820057155 Quota % Used Used Root 1000000 80 800837 user/xx (note that: its better but 800M and 890 makes a difference. That user has lots of folders and subfloders) my version: cyrus-imapd-2.3.14 It happens quite a lot. Too much for me. I used some years ago to force a quota -f on each users. It worked on a 2.2.12 version, but when we migrate to a 2.3.x version, suddenly, the quotas were broken with over 1000%. We went back to 2.2.12 and got rid of quota -f (too dangerous) But now, that problem is coming back again and I still don't understand why. Fixing a quota should be easy to do. quota -f is very fast. Too fast I think.. Can I generalize a reconstruct -rf followed by a quota -f without fears? Do you experience such problems? Thanks Dom -- Dominique LALOT Ing?nieur Syst?mes et R?seaux http://annuaire.univmed.fr/showuser.php?uid=lalot -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090901/c627b24b/attachment.html From michael.menge at zdv.uni-tuebingen.de Tue Sep 1 05:58:33 2009 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Tue, 01 Sep 2009 11:58:33 +0200 Subject: strange quota behaviour In-Reply-To: <1617f8010909010241i731bdedfm392389b963c4f4dd@mail.gmail.com> References: <1617f8010909010241i731bdedfm392389b963c4f4dd@mail.gmail.com> Message-ID: <20090901115833.12901vsblddnxqo9@webmail.uni-tuebingen.de> Quoting LALOT Dominique : > Hello, > > Sometimes, I have some bad quotas. Here is a trace: > > [cyrus at imap-perso ~]$ cyradm -user cyrus localhost > Password: > localhost.localdomain> lq user/xx > STORAGE 197/1000000 (0.0197%) > > [root at imap-perso ~]# du -hs /var/spool/imap2/m/user/xx > 890M /var/spool/imap2/m/user/xx > > reconstruct -rf user/xx > > [cyrus at imap-perso ~]$ quota -f user/xx > user.xx: usage was 201755, now 820057155 > Quota % Used Used Root > 1000000 80 800837 user/xx > Do you use delayed expunge, squatter? The size of cyrus.* files and eMails in delayed expunge mode are not included in the quota but in du. Reconstructing the mailbox without -k will delete the eMails in delayed expunge mode. > (note that: its better but 800M and 890 makes a difference. That user has > lots of folders and subfloders) > my version: > cyrus-imapd-2.3.14 > > It happens quite a lot. Too much for me. > I used some years ago to force a quota -f on each users. It worked on a > 2.2.12 version, but when we migrate to a 2.3.x version, suddenly, the quotas > were broken with over 1000%. > > We went back to 2.2.12 and got rid of quota -f (too dangerous) > > But now, that problem is coming back again and I still don't understand why. > Fixing a quota should be easy to do. > quota -f is very fast. Too fast I think.. > Can I generalize a reconstruct -rf followed by a quota -f without fears? > > Do you experience such problems? > > Thanks > > Dom > > -- > Dominique LALOT > Ing?nieur Syst?mes et R?seaux > http://annuaire.univmed.fr/showuser.php?uid=lalot > -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5339 bytes Desc: S/MIME krytographische Unterschrift Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090901/50ebad97/attachment.bin From dom.lalot at gmail.com Tue Sep 1 10:36:55 2009 From: dom.lalot at gmail.com (LALOT Dominique) Date: Tue, 1 Sep 2009 16:36:55 +0200 Subject: strange quota behaviour In-Reply-To: <20090901115833.12901vsblddnxqo9@webmail.uni-tuebingen.de> References: <1617f8010909010241i731bdedfm392389b963c4f4dd@mail.gmail.com> <20090901115833.12901vsblddnxqo9@webmail.uni-tuebingen.de> Message-ID: <1617f8010909010736n1ef57d8fp4be95de406467e48@mail.gmail.com> 2009/9/1 Michael Menge > Quoting LALOT Dominique : > > Hello, >> >> Sometimes, I have some bad quotas. Here is a trace: >> >> [cyrus at imap-perso ~]$ cyradm -user cyrus localhost >> Password: >> localhost.localdomain> lq user/xx >> STORAGE 197/1000000 (0.0197%) >> >> [root at imap-perso ~]# du -hs /var/spool/imap2/m/user/xx >> 890M /var/spool/imap2/m/user/xx >> >> reconstruct -rf user/xx >> >> [cyrus at imap-perso ~]$ quota -f user/xx >> user.xx: usage was 201755, now 820057155 >> Quota % Used Used Root >> 1000000 80 800837 user/xx >> >> > Do you use delayed expunge, squatter? The size of cyrus.* files and > eMails in delayed expunge mode are not included in the quota but > in du. Reconstructing the mailbox without -k will delete the eMails > in delayed expunge mode. No I have no expunge mode, my install is fairly basic.. Dom > > > > (note that: its better but 800M and 890 makes a difference. That user has >> lots of folders and subfloders) >> my version: >> cyrus-imapd-2.3.14 >> >> It happens quite a lot. Too much for me. >> I used some years ago to force a quota -f on each users. It worked on a >> 2.2.12 version, but when we migrate to a 2.3.x version, suddenly, the >> quotas >> were broken with over 1000%. >> >> We went back to 2.2.12 and got rid of quota -f (too dangerous) >> >> But now, that problem is coming back again and I still don't understand >> why. >> Fixing a quota should be easy to do. >> quota -f is very fast. Too fast I think.. >> Can I generalize a reconstruct -rf followed by a quota -f without fears? >> >> Do you experience such problems? >> >> Thanks >> >> Dom >> >> -- >> Dominique LALOT >> Ing?nieur Syst?mes et R?seaux >> http://annuaire.univmed.fr/showuser.php?uid=lalot >> >> > > > > -------------------------------------------------------------------------------- > M.Menge Tel.: (49) 7071/29-70316 > Universit?t T?bingen Fax.: (49) 7071/29-5912 > Zentrum f?r Datenverarbeitung mail: > michael.menge at zdv.uni-tuebingen.de > W?chterstra?e 76 > 72074 T?bingen > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > -- Dominique LALOT Ing?nieur Syst?mes et R?seaux http://annuaire.univmed.fr/showuser.php?uid=lalot -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090901/1ea90530/attachment-0001.html From linkages at ufl.edu Tue Sep 1 12:00:24 2009 From: linkages at ufl.edu (Eli Ben-Shoshan) Date: Tue, 01 Sep 2009 12:00:24 -0400 Subject: Cyrus v2.2 and v2.3 interoperability Message-ID: <4A9D4518.8020803@ufl.edu> We here at the University of Florida are running a good sized cyrus murder setup. We have 10 cyrus v.2.2.12 backends with 9 cyrus v2.2.12 frontends and a dedicated mupdate master running v2.2.12. We are now in the planning phase of moving to the cyrus v2.3 branch. We will be installing all v.2.3.7 nodes since that is what is the latest available in RHEL 5.3. We would like to upgrade just the frontends and mupdate first since we are redesigning our backend's storage infrastructure. Later on we plan on upgrading the backends to v2.3 with replication. Can this be done? Is it safe to run v2.3 frontends and mupdate while keeping v.2.2 backends? Should we leave mupdate at v2.2 and upgrade just the frontends to v2.3? Are there specific versions of cyrus to avoid in this situation? Has anyone actually done this before? Any insights are welcomed. Thanks in advance. -- Eli Ben-Shoshan From morgan at orst.edu Tue Sep 1 12:53:51 2009 From: morgan at orst.edu (Andrew Morgan) Date: Tue, 1 Sep 2009 09:53:51 -0700 (PDT) Subject: Cyrus v2.2 and v2.3 interoperability In-Reply-To: <4A9D4518.8020803@ufl.edu> References: <4A9D4518.8020803@ufl.edu> Message-ID: On Tue, 1 Sep 2009, Eli Ben-Shoshan wrote: > We here at the University of Florida are running a good sized cyrus murder > setup. We have 10 cyrus v.2.2.12 backends with 9 cyrus v2.2.12 frontends and a > dedicated mupdate master running v2.2.12. > > We are now in the planning phase of moving to the cyrus v2.3 branch. We will be > installing all v.2.3.7 nodes since that is what is the latest available in RHEL 5.3. > > We would like to upgrade just the frontends and mupdate first since we are > redesigning our backend's storage infrastructure. Later on we plan on upgrading > the backends to v2.3 with replication. > > Can this be done? > Is it safe to run v2.3 frontends and mupdate while keeping v.2.2 backends? > Should we leave mupdate at v2.2 and upgrade just the frontends to v2.3? > Are there specific versions of cyrus to avoid in this situation? > Has anyone actually done this before? > > Any insights are welcomed. Thanks in advance. Sorry, you must upgrade all of your backends before your frontends. 2.3 frontends try to issue an IMAP command to the backends that 2.2 doesn't understand. Quoting myself back in 2007: I proceeded assuming I could have a v2.3.10 frontend with older v2.2.13 backends. However, I was unable to get the APPEND command to work. With telemetry logging enabled, I discovered that a 2.3 frontend issues the IMAP command "Localappend" to a backend. However, my v2.2.13 backend does not recognize "Localappend" as a valid command (and it is not present in the source code). I'm not aware of any version requirements or problems for the mupdate master. I would probably upgrade in this order: mupdate > backends > frontends. Andy From michael.menge at zdv.uni-tuebingen.de Tue Sep 1 14:02:24 2009 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Tue, 01 Sep 2009 20:02:24 +0200 Subject: Cyrus v2.2 and v2.3 interoperability In-Reply-To: <4A9D4518.8020803@ufl.edu> References: <4A9D4518.8020803@ufl.edu> Message-ID: <20090901200224.87786twzjh0uux80@webmail.uni-tuebingen.de> Quoting Eli Ben-Shoshan : > We are now in the planning phase of moving to the cyrus v2.3 branch. > We will be > installing all v.2.3.7 nodes since that is what is the latest > available in RHEL 5.3. I don't know which patches where backported to v.2.3.7 in RHEL 5.3 but there where many bugs in 2.3.7 that are now fixed in 2.3.14 which are IMHO worth installing cyrus 2.3.14 from the source > > We would like to upgrade just the frontends and mupdate first since we are > redesigning our backend's storage infrastructure. Later on we plan > on upgrading > the backends to v2.3 with replication. > > Can this be done? > Is it safe to run v2.3 frontends and mupdate while keeping v.2.2 backends? You must upgrade the backends first. > Should we leave mupdate at v2.2 and upgrade just the frontends to v2.3? > Are there specific versions of cyrus to avoid in this situation? > Has anyone actually done this before? > > Any insights are welcomed. Thanks in advance. > -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen From awilliam at whitemice.org Tue Sep 1 14:17:00 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Tue, 01 Sep 2009 14:17:00 -0400 Subject: Cyrus v2.2 and v2.3 interoperability In-Reply-To: <20090901200224.87786twzjh0uux80@webmail.uni-tuebingen.de> References: <4A9D4518.8020803@ufl.edu> <20090901200224.87786twzjh0uux80@webmail.uni-tuebingen.de> Message-ID: <1251829020.7274.1.camel@linux-m3mt> On Tue, 2009-09-01 at 20:02 +0200, Michael Menge wrote: > Quoting Eli Ben-Shoshan : > > We are now in the planning phase of moving to the cyrus v2.3 branch. > > We will be > > installing all v.2.3.7 nodes since that is what is the latest > > available in RHEL 5.3. > I don't know which patches where backported to v.2.3.7 in RHEL 5.3 > but there where many bugs in 2.3.7 that are now fixed in 2.3.14 which > are IMHO worth installing cyrus 2.3.14 from the source Use Simon's excellent RPMs for RHEL5/CentOS5. -- OpenGroupware developer: awilliam at whitemice.org OpenGroupare & Cyrus IMAPd documenation @ From hans.moser at ofd-sth.niedersachsen.de Wed Sep 2 07:58:12 2009 From: hans.moser at ofd-sth.niedersachsen.de (Marc Patermann) Date: Wed, 02 Sep 2009 13:58:12 +0200 Subject: Logging region out of memory Message-ID: <4A9E5DD4.6000905@ofd-sth.niedersachsen.de> Hi, I have IMAPd 2.2.12 and BDB 4.2.52: When I got: Sep 2 11:28:39 rzhs199 local6:warn|warning lmtpunix[1171642]: DBERROR db4: Logging region out of memory; you may need to increase its size Sep 2 11:28:39 rzhs199 local6:err|error lmtpunix[1171642]: DBERROR: opening /mail/imap/ptclient/ptscache.db: Not enough space Sep 2 11:28:39 rzhs199 local6:err|error lmtpunix[1171642]: DBERROR: opening /mail/imap/ptclient/ptscache.db: cyrusdb error Sep 2 11:28:39 rzhs199 local6:warn|warning lmtpunix[4227234]: DBERROR db4: Logging region out of memory; you may need to increase its size Sep 2 11:28:39 rzhs199 local6:err|error lmtpunix[4227234]: DBERROR: opening /mail/imap/deliver.db: Not enough space Sep 2 11:28:39 rzhs199 local6:err|error lmtpunix[4227234]: DBERROR: opening /mail/imap/deliver.db: cyrusdb error Sep 2 11:28:39 rzhs199 local6:err|error lmtpunix[4227234]: FATAL: lmtpd: unable to init duplicate delivery database Mass mail could not be delivered in time because LMTP had errors. I found DB_CONFIG in /mail/imap/ :/mail/imap # cat DB_CONFIG set_cachesize 0 8388608 8 set_lg_regionmax 524288 set_lg_bsize 2097152 and these files :/mail/imap # l 45888 insgesamt drwxr-xr-x 12 cyrus cyrus 4096 02 Sep 12:00 . drwxr-xr-x 6 cyrus cyrus 256 28 Jun 2007 .. -rw------- 1 cyrus cyrus 144 02 Sep 12:03 annotations.db drwxrws--- 2 cyrus cyrus 4096 02 Sep 04:14 db drwx------ 2 cyrus cyrus 256 02 Sep 11:44 db.backup1 drwx------ 2 cyrus cyrus 256 02 Sep 11:14 db.backup2 -rw-r--r-- 1 cyrus cyrus 72 08 Jan 2009 DB_CONFIG -rw------- 1 cyrus cyrus 21336064 02 Sep 12:03 deliver.db drwxr-xr-x 2 root system 256 29 Nov 2007 lost+found -rw------- 1 cyrus cyrus 1106000 02 Sep 12:03 mailboxes.db -rw-rw---- 1 cyrus cyrus 689810 02 Sep 11:37 mailboxes.tsm -rw-r----- 1 root system 8 20 Apr 10:45 master.pid drwxrws--- 2 cyrus cyrus 256 29 Nov 2007 msg drwxrws--- 2 cyrus cyrus 102400 02 Sep 12:04 proc drwxrws--- 2 cyrus cyrus 256 20 Apr 10:45 ptclient drwxrws--- 26 cyrus cyrus 4096 29 Nov 2007 quota drwxrws--- 2 cyrus cyrus 256 20 Apr 10:45 socket -rw------- 1 cyrus cyrus 139264 02 Sep 12:03 tls_sessions.db drwx------ 26 cyrus cyrus 4096 30 Nov 2007 user But the DB_CONFIG setting seemed not to be active. :/mail/imap # db_stat -m -h db 641KB 604B Total cache size. 1 Number of caches. 648KB Pool individual cache size. 0 Requested pages mapped into the process' address space. 53M Requested pages found in the cache (96%). 2318503 Requested pages not found in the cache. 23 Pages created in the cache. 2318494 Pages read into the cache. 1311629 Pages written from the cache to the backing file. 1479063 Clean pages forced from the cache. 839315 Dirty pages forced from the cache. 0 Dirty pages written by trickle-sync thread. 158 Current total page count. 89 Current clean page count. 69 Current dirty page count. 67 Number of hash buckets used for page location. 57M Total number of times hash chains searched for a page. 15 The longest hash chain searched for a page. 177M Total number of hash buckets examined for page location. 128M The number of hash bucket locks granted without waiting. 1030 The number of hash bucket locks granted after waiting. 297 The maximum number of times any hash bucket lock was waited for. 12M The number of region locks granted without waiting. 1747 The number of region locks granted after waiting. 2318893 The number of page allocations. 4713896 The number of hash buckets examined during allocations 5 The max number of hash buckets examined for an allocation 2318377 The number of pages examined during allocations 2 The max number of pages examined for an allocation =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Pool File: /mail/imap/ptclient/ptscache.db 4096 Page size. 0 Requested pages mapped into the process' address space. 9465518 Requested pages found in the cache (92%). 809878 Requested pages not found in the cache. 23 Pages created in the cache. 809878 Pages read into the cache. 279900 Pages written from the cache to the backing file. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Pool File: /mail/imap/deliver.db 4096 Page size. 0 Requested pages mapped into the process' address space. 41M Requested pages found in the cache (97%). 1443266 Requested pages not found in the cache. 0 Pages created in the cache. 1443257 Pages read into the cache. 957015 Pages written from the cache to the backing file. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Pool File: /mail/imap/tls_sessions.db 4096 Page size. 0 Requested pages mapped into the process' address space. 1780221 Requested pages found in the cache (96%). 65359 Requested pages not found in the cache. 0 Pages created in the cache. 65359 Pages read into the cache. 74714 Pages written from the cache to the backing file. # db_stat -l -h . 40988 Log magic number. 8 Log version number. 32KB Log record cache size. 0644 Log file mode. 10Mb Current log file size. Log bytes written. Log bytes written since last checkpoint. 0 Total log file writes. 0 Total log file write due to overflow. 0 Total log file flushes. 1311 Current log file number. 2653252 Current log file offset. 1311 On-disk log file number. 2653252 On-disk log file offset. 0 Max commits in a log flush. 0 Min commits in a log flush. 96KB Log region size. 0 The number of region locks granted after waiting. 0 The number of region locks granted without waiting. I moved DB_CONFIG to db/ made "db_recover -h" and it looked much better. After that change ptloader (against openLDAP) complained: Sep 2 13:42:28 rzhs199 local6:debug ptloader[2212060]: accepted connection Sep 2 13:42:28 rzhs199 local6:err|error ptloader[2212060]: Unable to set LDAP_OPT_X_SASL_SECPROPS. Sep 2 13:42:28 rzhs199 local6:err|error imap[3440844]: ptload(): bad response from ptloader server: ptsmodule_connect() failed Whereas some user were still be able to log in others don't. I removed DB_CONFIG from db, did a db_recover again. I have the much smaller standard values again active, but the ptloader errors are gone. Where is the configuration fault? Marc From pcravero at as2594.net Wed Sep 2 08:30:18 2009 From: pcravero at as2594.net (Paolo Cravero) Date: Wed, 02 Sep 2009 14:30:18 +0200 Subject: incremental squatter Message-ID: <4A9E655A.2080101@as2594.net> Hi. Perhaps it is a bug in the documentation. 'man squatter' in the DESCRIPTION says there's no incremental update: "Squatter creates an index of ALL messages in the mailbox, not just those since the last time that it was run (i.e., it does NOT do incremental updates)." but in the OPTIONS a "-i" switch mentions incremental. Is it just a documentation error, so incremental indexing does work in 2.3.14? TIA, Paolo From Hagedorn at uni-koeln.de Wed Sep 2 08:40:42 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Wed, 02 Sep 2009 14:40:42 +0200 Subject: incremental squatter In-Reply-To: <4A9E655A.2080101@as2594.net> References: <4A9E655A.2080101@as2594.net> Message-ID: <43C11F14D59B8A28E1AFD2B6@tyrion.local> --On 2. September 2009 14:30:18 +0200 Paolo Cravero wrote: > Is it just a documentation error, so incremental indexing does work in > 2.3.14? Yes, it does. -- .:.Sebastian Hagedorn - RZKR-R1 (Geb?ude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universit?t zu K?ln / Cologne University - ? +49-221-478-5587.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pkcs7-signature Size: 5292 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090902/0f00a087/attachment-0001.bin From awilliam at whitemice.org Wed Sep 2 08:44:59 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Wed, 02 Sep 2009 08:44:59 -0400 Subject: incremental squatter In-Reply-To: <4A9E655A.2080101@as2594.net> References: <4A9E655A.2080101@as2594.net> Message-ID: <1251895499.5267.11.camel@linux-m3mt> On Wed, 2009-09-02 at 14:30 +0200, Paolo Cravero wrote: > Perhaps it is a bug in the documentation. > 'man squatter' in the DESCRIPTION says there's no incremental update: > "Squatter creates an index of ALL messages in the mailbox, not just those > since the last time that it was run (i.e., it does NOT do incremental > updates)." > but in the OPTIONS a "-i" switch mentions incremental. > Is it just a documentation error, so incremental indexing does work in 2.3.14? It may be a matter of interpretation. Squatter creates an index of ALL messages in the mailbox, not just those since time that it was run (i.e., it does NOT do incremental updates). Any messages appended to the mailbox after squatter is run, will NOT be included in the index. To include new messages in the index, squatter must be run again. I took "does NOT do incremental updates" to mean the index is not updated on-the-fly when new messages enter the mailbox, only when squatter is run. -- OpenGroupware developer: awilliam at whitemice.org OpenGroupare & Cyrus IMAPd documenation @ From billcamer at gmail.com Wed Sep 2 11:19:12 2009 From: billcamer at gmail.com (Bill Cameron) Date: Wed, 2 Sep 2009 09:19:12 -0600 Subject: Migrating 32bit to 64bit Debian Lenny Message-ID: <7f1434c60909020819s5d90fad7s1ccee10e74328a49@mail.gmail.com> Hi, I've seen some questions about migrating from 32 bit to 64 bit. We migrated last weekend and these are the steps we took. Old server: - rsync the cyrus data while cyrus is running rsync -vaH --delete -e ssh /var/lib/cyrus/ new-server:/var/lib/cyrus rsync -vaH --delete -e ssh /var/spool/cyrus/ new-server:/var/spool/cyrus rsync -vaH --delete -e ssh /var/spool/sieve/ new-server:/var/spool/sieve - shut down cyrus - repeat rsyncing of the three directories to provide stable environment and databases. This will be a lot faster than the original rsync. - dump /var/lib/cyrus/mailboxes.db to a text file /usr/sbin/ctl_mboxlist -d > mboxlist.txt - copy this text file to the new server New server: - make sure cyrus is shutdown - switch to user 'cyrus' su - cyrus - remove some of the databases rm /var/lib/cyrus/db/* rm /var/lib/cyrus/db.backup1/* rm /var/lib/cyrus/db.backup2/* rm /var/lib/cyrus/deliver.db rm /var/lib/cyrus/tls_sessions.db rm /var/lib/cyrus/mailboxes.db - build new mailboxes.db from mboxlist.txt file /usr/sbin/ctl_mboxlist -u < mboxlist.txt - check /var/log/mail.err and /var/log/mail.info for any errors from the above command and the following commands. There should only be one error about missing timestamp file but it is automatically created. - run the following commands and check logs for errors /usr/sbin/ctl_cyrusdb -r - the above command will verify mailboxes.db and annotations.db /usr/sbin/tls_prune - the above command will create a new tls_prune database /usr/sbin/ctl_cyrusdb -c /usr/sbin/cyr_expire -E 3 - you can also run the squatter command but it doesn't really need to run until it's scheduled time. It takes a awhile to run. - start cyrus and check that it is working correctly. - you will need to reset any annotations (e.g.: expire) on mailboxes/folders. We didn't have any annotations set on mailboxes so I didn't try migrating that database. The database types are defined in /etc/imapd.conf. They don't appear in the default Lenny conf file since they use the predefined default types. The man page for imapd.conf lists those defaults: annotation_db: skiplist, duplicate_db: berkeley-nosync... The current database types are listed in /usr/lib/cyrus/cyrus-db-types.active. We had to take the server off-line to migrate some other applications on the server so I didn't use imapsync. We are using imapsync to migrate from Lotus Notes to cyrus. Bill C. From simon.matter at invoca.ch Wed Sep 2 11:56:26 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Wed, 2 Sep 2009 17:56:26 +0200 Subject: Migrating 32bit to 64bit Debian Lenny In-Reply-To: <7f1434c60909020819s5d90fad7s1ccee10e74328a49@mail.gmail.com> References: <7f1434c60909020819s5d90fad7s1ccee10e74328a49@mail.gmail.com> Message-ID: <8f86614351325fa8d8e3138253ae3e2c.squirrel@webmail.bi.corp.invoca.ch> > Hi, > > I've seen some questions about migrating from 32 bit to 64 bit. We Hi, I understand that you also migrated to a newer OS and cyrus-imapd version, right? > migrated last weekend and these are the steps we took. > > Old server: > - rsync the cyrus data while cyrus is running > rsync -vaH --delete -e ssh /var/lib/cyrus/ > new-server:/var/lib/cyrus > rsync -vaH --delete -e ssh /var/spool/cyrus/ > new-server:/var/spool/cyrus > rsync -vaH --delete -e ssh /var/spool/sieve/ > new-server:/var/spool/sieve > - shut down cyrus > - repeat rsyncing of the three directories to provide stable > environment and databases. This will be a lot faster than the original > rsync. > - dump /var/lib/cyrus/mailboxes.db to a text file > /usr/sbin/ctl_mboxlist -d > mboxlist.txt > - copy this text file to the new server > > New server: > - make sure cyrus is shutdown > - switch to user 'cyrus' > su - cyrus > - remove some of the databases > rm /var/lib/cyrus/db/* > rm /var/lib/cyrus/db.backup1/* > rm /var/lib/cyrus/db.backup2/* > rm /var/lib/cyrus/deliver.db > rm /var/lib/cyrus/tls_sessions.db > rm /var/lib/cyrus/mailboxes.db > - build new mailboxes.db from mboxlist.txt file > /usr/sbin/ctl_mboxlist -u < mboxlist.txt > - check /var/log/mail.err and /var/log/mail.info for any errors > from the above command and the following commands. There should only > be one error about missing timestamp file but it is automatically > created. > - run the following commands and check logs for errors > /usr/sbin/ctl_cyrusdb -r > - the above command will verify mailboxes.db and annotations.db > /usr/sbin/tls_prune > - the above command will create a new tls_prune database > /usr/sbin/ctl_cyrusdb -c > /usr/sbin/cyr_expire -E 3 > - you can also run the squatter command but it doesn't really need > to run until it's scheduled time. It takes a awhile to run. > - start cyrus and check that it is working correctly. > - you will need to reset any annotations (e.g.: expire) on > mailboxes/folders. We didn't have any annotations set on mailboxes so > I didn't try migrating that database. > > The database types are defined in /etc/imapd.conf. They don't appear > in the default Lenny conf file since they use the predefined default > types. The man page for imapd.conf lists those defaults: > annotation_db: skiplist, duplicate_db: berkeley-nosync... The current > database types are listed in /usr/lib/cyrus/cyrus-db-types.active. I'm wondering how much of all this was really needed for the migration from 32bit to 64bit? Are the BerkeleyDB ondisk files different on 32/64bit? Because, the last migration I did was from RHEL3/32bit to RHEL5/64bit using our own cyrus-imapd RPMs and the migration was as easy as stopping cyrus-imapd, rsyncing /var/lib/imap + /var/spool/imap to the new box and starting cyrus-imapd on the new box. Now, the RPMs do quite some magic on the database files to make sure they are all skiplist after shutdown, and switched back on startup. So my question remains, what parts of cyrus-imapd are possibly arch dependant? Regards, Simon From Pascal.Gienger at uni-konstanz.de Wed Sep 2 12:18:28 2009 From: Pascal.Gienger at uni-konstanz.de (Pascal Gienger) Date: Wed, 02 Sep 2009 18:18:28 +0200 Subject: Migrating 32bit to 64bit Debian Lenny In-Reply-To: <8f86614351325fa8d8e3138253ae3e2c.squirrel@webmail.bi.corp.invoca.ch> References: <7f1434c60909020819s5d90fad7s1ccee10e74328a49@mail.gmail.com> <8f86614351325fa8d8e3138253ae3e2c.squirrel@webmail.bi.corp.invoca.ch> Message-ID: <4A9E9AD4.2050009@uni-konstanz.de> Simon Matter wrote: > I'm wondering how much of all this was really needed for the migration > from 32bit to 64bit? Are the BerkeleyDB ondisk files different on > 32/64bit? Yes they are. It's not the OS that matters but the architecture of the libdb4.so file. It is still a good idea not to use Berkeley DB for real important data. Here at our university's cyrus we are using Berkeley for the duplicate delivery and the tls databases - both of them are easily set to zero in case of problems without deep impact on the functionality (in case the delivery db crashes users can get some mails two times (doubling), in the latter case (tls db crash) a returning client has to re-initiate a TLS handshake including key exchange). Pascal Gienger -- Pascal Gienger University of Konstanz, IT Services Department ("Rechenzentrum") Electronic Communications and Web Services Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739 From billcamer at gmail.com Wed Sep 2 12:46:35 2009 From: billcamer at gmail.com (Bill Cameron) Date: Wed, 2 Sep 2009 10:46:35 -0600 Subject: Migrating 32bit to 64bit Debian Lenny In-Reply-To: <8f86614351325fa8d8e3138253ae3e2c.squirrel@webmail.bi.corp.invoca.ch> References: <7f1434c60909020819s5d90fad7s1ccee10e74328a49@mail.gmail.com> <8f86614351325fa8d8e3138253ae3e2c.squirrel@webmail.bi.corp.invoca.ch> Message-ID: <7f1434c60909020946hfe3b913yaec0892d97207157@mail.gmail.com> > I'm wondering how much of all this was really needed for the migration > from 32bit to 64bit? Are the BerkeleyDB ondisk files different on > 32/64bit? > I initially tried just using rsync with cyrus shutdown on both servers but cyrus failed to start on the new server due to db errors. As mentioned this is due to the architecture of the libdb4.so file. Further testing showed which files worked and which ones didn't. Bill C. From brong at fastmail.fm Wed Sep 2 18:06:25 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 3 Sep 2009 08:06:25 +1000 Subject: Migrating 32bit to 64bit Debian Lenny In-Reply-To: <7f1434c60909020946hfe3b913yaec0892d97207157@mail.gmail.com> References: <7f1434c60909020819s5d90fad7s1ccee10e74328a49@mail.gmail.com> <8f86614351325fa8d8e3138253ae3e2c.squirrel@webmail.bi.corp.invoca.ch> <7f1434c60909020946hfe3b913yaec0892d97207157@mail.gmail.com> Message-ID: <20090902220625.GA15835@brong.net> On Wed, Sep 02, 2009 at 10:46:35AM -0600, Bill Cameron wrote: > > I'm wondering how much of all this was really needed for the migration > > from 32bit to 64bit? Are the BerkeleyDB ondisk files different on > > 32/64bit? > > > > I initially tried just using rsync with cyrus shutdown on both servers > but cyrus failed to start on the new server due to db errors. As > mentioned this is due to the architecture of the libdb4.so file. > > Further testing showed which files worked and which ones didn't. Thankfully skiplist is immune to such nonsense and just keeps working across any architecture or underlying version change (apart from a couple of bugs that are only fixed in CVS at the moment - *sigh*) We've recently switched to using skiplist even for deliver and TLS databases. It works fine. Possibly not quite as efficient, but we haven't noticed a difference in the load on the machines. Though - I do wonder how many problems we've had with BDB were caused by idled forking with an environment reference and then closing it in the parent, causing the reference count to be incorrect. There's a patch for that in CVS too. Bron. From support at harmsconsulting.com Wed Sep 2 19:39:05 2009 From: support at harmsconsulting.com (Harms Consulting IT support desk) Date: Thu, 03 Sep 2009 09:39:05 +1000 Subject: Logging region out of memory In-Reply-To: <4A9E5DD4.6000905@ofd-sth.niedersachsen.de> References: <4A9E5DD4.6000905@ofd-sth.niedersachsen.de> Message-ID: <4A9F0219.6020802@harmsconsulting.com> G'day Marc, Marc Patermann wrote: > Hi, > > I have IMAPd 2.2.12 and BDB 4.2.52: > > When I got: > > Sep 2 11:28:39 rzhs199 local6:warn|warning lmtpunix[1171642]: DBERROR > db4: Logging region out of memory; you may need to increase its size > I'm not good for a lot of help in this mailing list, but whenever DBERROR appears in the logs, the recommended solution is to change database formats away from Berkeley DB. Search this mail list for DBERROR and see this page in the wiki for more info: http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/WhatDatabaseBackend HTH, Josh > Sep 2 11:28:39 rzhs199 local6:err|error lmtpunix[1171642]: DBERROR: > opening /mail/imap/ptclient/ptscache.db: Not enough space > Sep 2 11:28:39 rzhs199 local6:err|error lmtpunix[1171642]: DBERROR: > opening /mail/imap/ptclient/ptscache.db: cyrusdb error > > Sep 2 11:28:39 rzhs199 local6:warn|warning lmtpunix[4227234]: DBERROR > db4: Logging region out of memory; you may need to increase its size > Sep 2 11:28:39 rzhs199 local6:err|error lmtpunix[4227234]: DBERROR: > opening /mail/imap/deliver.db: Not enough space > Sep 2 11:28:39 rzhs199 local6:err|error lmtpunix[4227234]: DBERROR: > opening /mail/imap/deliver.db: cyrusdb error > Sep 2 11:28:39 rzhs199 local6:err|error lmtpunix[4227234]: FATAL: > lmtpd: unable to init duplicate delivery database > > Mass mail could not be delivered in time because LMTP had errors. > > I found DB_CONFIG in /mail/imap/ > > :/mail/imap # cat DB_CONFIG > set_cachesize 0 8388608 8 > set_lg_regionmax 524288 > set_lg_bsize 2097152 > > and these files > > :/mail/imap # l > 45888 insgesamt > drwxr-xr-x 12 cyrus cyrus 4096 02 Sep 12:00 . > drwxr-xr-x 6 cyrus cyrus 256 28 Jun 2007 .. > -rw------- 1 cyrus cyrus 144 02 Sep 12:03 annotations.db > drwxrws--- 2 cyrus cyrus 4096 02 Sep 04:14 db > drwx------ 2 cyrus cyrus 256 02 Sep 11:44 db.backup1 > drwx------ 2 cyrus cyrus 256 02 Sep 11:14 db.backup2 > -rw-r--r-- 1 cyrus cyrus 72 08 Jan 2009 DB_CONFIG > -rw------- 1 cyrus cyrus 21336064 02 Sep 12:03 deliver.db > drwxr-xr-x 2 root system 256 29 Nov 2007 lost+found > -rw------- 1 cyrus cyrus 1106000 02 Sep 12:03 mailboxes.db > -rw-rw---- 1 cyrus cyrus 689810 02 Sep 11:37 mailboxes.tsm > -rw-r----- 1 root system 8 20 Apr 10:45 master.pid > drwxrws--- 2 cyrus cyrus 256 29 Nov 2007 msg > drwxrws--- 2 cyrus cyrus 102400 02 Sep 12:04 proc > drwxrws--- 2 cyrus cyrus 256 20 Apr 10:45 ptclient > drwxrws--- 26 cyrus cyrus 4096 29 Nov 2007 quota > drwxrws--- 2 cyrus cyrus 256 20 Apr 10:45 socket > -rw------- 1 cyrus cyrus 139264 02 Sep 12:03 tls_sessions.db > drwx------ 26 cyrus cyrus 4096 30 Nov 2007 user > > But the DB_CONFIG setting seemed not to be active. > > :/mail/imap # db_stat -m -h db > 641KB 604B Total cache size. > 1 Number of caches. > 648KB Pool individual cache size. > 0 Requested pages mapped into the process' address space. > 53M Requested pages found in the cache (96%). > 2318503 Requested pages not found in the cache. > 23 Pages created in the cache. > 2318494 Pages read into the cache. > 1311629 Pages written from the cache to the backing file. > 1479063 Clean pages forced from the cache. > 839315 Dirty pages forced from the cache. > 0 Dirty pages written by trickle-sync thread. > 158 Current total page count. > 89 Current clean page count. > 69 Current dirty page count. > 67 Number of hash buckets used for page location. > 57M Total number of times hash chains searched for a page. > 15 The longest hash chain searched for a page. > 177M Total number of hash buckets examined for page location. > 128M The number of hash bucket locks granted without waiting. > 1030 The number of hash bucket locks granted after waiting. > 297 The maximum number of times any hash bucket lock was waited for. > 12M The number of region locks granted without waiting. > 1747 The number of region locks granted after waiting. > 2318893 The number of page allocations. > 4713896 The number of hash buckets examined during allocations > 5 The max number of hash buckets examined for an allocation > 2318377 The number of pages examined during allocations > 2 The max number of pages examined for an allocation > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Pool File: /mail/imap/ptclient/ptscache.db > 4096 Page size. > 0 Requested pages mapped into the process' address space. > 9465518 Requested pages found in the cache (92%). > 809878 Requested pages not found in the cache. > 23 Pages created in the cache. > 809878 Pages read into the cache. > 279900 Pages written from the cache to the backing file. > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Pool File: /mail/imap/deliver.db > 4096 Page size. > 0 Requested pages mapped into the process' address space. > 41M Requested pages found in the cache (97%). > 1443266 Requested pages not found in the cache. > 0 Pages created in the cache. > 1443257 Pages read into the cache. > 957015 Pages written from the cache to the backing file. > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Pool File: /mail/imap/tls_sessions.db > 4096 Page size. > 0 Requested pages mapped into the process' address space. > 1780221 Requested pages found in the cache (96%). > 65359 Requested pages not found in the cache. > 0 Pages created in the cache. > 65359 Pages read into the cache. > 74714 Pages written from the cache to the backing file. > > # db_stat -l -h . > 40988 Log magic number. > 8 Log version number. > 32KB Log record cache size. > 0644 Log file mode. > 10Mb Current log file size. > Log bytes written. > Log bytes written since last checkpoint. > 0 Total log file writes. > 0 Total log file write due to overflow. > 0 Total log file flushes. > 1311 Current log file number. > 2653252 Current log file offset. > 1311 On-disk log file number. > 2653252 On-disk log file offset. > 0 Max commits in a log flush. > 0 Min commits in a log flush. > 96KB Log region size. > 0 The number of region locks granted after waiting. > 0 The number of region locks granted without waiting. > > I moved DB_CONFIG to db/ made "db_recover -h" and it looked much better. > > After that change ptloader (against openLDAP) complained: > > Sep 2 13:42:28 rzhs199 local6:debug ptloader[2212060]: accepted connection > Sep 2 13:42:28 rzhs199 local6:err|error ptloader[2212060]: Unable to > set LDAP_OPT_X_SASL_SECPROPS. > Sep 2 13:42:28 rzhs199 local6:err|error imap[3440844]: ptload(): bad > response from ptloader server: ptsmodule_connect() failed > > Whereas some user were still be able to log in others don't. > > I removed DB_CONFIG from db, did a db_recover again. > I have the much smaller standard values again active, but the ptloader > errors are gone. > > Where is the configuration fault? > > > Marc > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > From wcooley at nakedape.cc Wed Sep 2 20:46:47 2009 From: wcooley at nakedape.cc (Wil Cooley) Date: Wed, 02 Sep 2009 17:46:47 -0700 Subject: Resources for MUA support of IMAP features? Message-ID: <4A9F11F7.3070207@nakedape.cc> Does anyone know of a mailing list or a web site with information about MUA support of various IMAP features? For example, for IMAP IDLE the Wikipedia entry is good: http://en.wikipedia.org/wiki/IMAP_IDLE (Although it is lacking in some details about what to expect from a server supporting it, especially server-specific information such as using idled vs not.) Or something like this (probably dated) reference about server features: http://www.melnikov.ca/mel/devel/ServerReference.html The UW lists at http://www.washington.edu/imap/ seem (imap-use@ especially) like they would be an appropriate place, but they seem kinda... dead. Wil -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 260 bytes Desc: OpenPGP digital signature Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090902/0a0ba584/attachment.bin From Pascal.Gienger at uni-konstanz.de Thu Sep 3 01:14:14 2009 From: Pascal.Gienger at uni-konstanz.de (Pascal Gienger) Date: Thu, 03 Sep 2009 07:14:14 +0200 Subject: Logging region out of memory In-Reply-To: <4A9E5DD4.6000905@ofd-sth.niedersachsen.de> References: <4A9E5DD4.6000905@ofd-sth.niedersachsen.de> Message-ID: <4A9F50A6.7050407@uni-konstanz.de> Marc Patermann wrote: > Hi, > > I have IMAPd 2.2.12 and BDB 4.2.52: > > When I got: > > Sep 2 11:28:39 rzhs199 local6:warn|warning lmtpunix[1171642]: DBERROR > db4: Logging region out of memory; you may need to increase its size Increase logging region size. > I found DB_CONFIG in /mail/imap/ > > :/mail/imap # cat DB_CONFIG > set_cachesize 0 8388608 8 > set_lg_regionmax 524288 > set_lg_bsize 2097152 > > and these files The file DB_CONFIG has to be in the "db" subdirectory (/mail/imap/db in your case). Be warned: Some parameters of DB_CONFIG also change the on-disk-format, so backup your db files before (after shutting down cyrus) and restart after your changes. Don't delete "skipstamp" in this db directory as it is used by your skiplist databases. Just a personal biased hint: You should not use Berkeley DB for important data of your cyrus system. Berkeley has a very rapid random read performance which is important (in our case) with the duplicate delivery database (now 1,3 GB in size). But even that should be feasible with skiplist. Pascal -- Pascal Gienger University of Konstanz, IT Services Department ("Rechenzentrum") Electronic Communications and Web Services Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739 From pcravero at as2594.net Thu Sep 3 02:51:19 2009 From: pcravero at as2594.net (Paolo Cravero) Date: Thu, 03 Sep 2009 08:51:19 +0200 Subject: Resources for MUA support of IMAP features? In-Reply-To: <4A9F11F7.3070207@nakedape.cc> References: <4A9F11F7.3070207@nakedape.cc> Message-ID: <4A9F6767.5080808@as2594.net> Wil Cooley wrote: > Does anyone know of a mailing list or a web site with information about > MUA support of various IMAP features? For example, for IMAP IDLE the I once came across this wiki: http://www.imapwiki.org/ that links to this http://uplib.parc.com/misc/imapclients.html for your specific question. HTH, Paolo PS: nice domain name Wil :) From hans.moser at ofd-sth.niedersachsen.de Thu Sep 3 03:47:52 2009 From: hans.moser at ofd-sth.niedersachsen.de (Marc Patermann) Date: Thu, 03 Sep 2009 09:47:52 +0200 Subject: Logging region out of memory In-Reply-To: <4A9F50A6.7050407@uni-konstanz.de> References: <4A9E5DD4.6000905@ofd-sth.niedersachsen.de> <4A9F50A6.7050407@uni-konstanz.de> Message-ID: <4A9F74A8.2020105@ofd-sth.niedersachsen.de> Pascal Gienger schrieb: >> Sep 2 11:28:39 rzhs199 local6:warn|warning lmtpunix[1171642]: DBERROR >> db4: Logging region out of memory; you may need to increase its size > Increase logging region size. That's what I did (tried). But then I got this strange ptloader errors. Sep 2 13:42:28 rzhs199 local6:debug ptloader[2212060]: accepted connection Sep 2 13:42:28 rzhs199 local6:err|error ptloader[2212060]: Unable to set LDAP_OPT_X_SASL_SECPROPS. Sep 2 13:42:28 rzhs199 local6:err|error imap[3440844]: ptload(): bad response from ptloader server: ptsmodule_connect() failed Does the DB_CONFIG interfere with the /mail/imap/ptclient/ptscache.db here? Marc From ross at biostat.ucsf.edu Thu Sep 3 12:57:46 2009 From: ross at biostat.ucsf.edu (Ross Boylan) Date: Thu, 03 Sep 2009 09:57:46 -0700 Subject: incremental squatter In-Reply-To: <1251895499.5267.11.camel@linux-m3mt> References: <4A9E655A.2080101@as2594.net> <1251895499.5267.11.camel@linux-m3mt> Message-ID: <1251997066.29746.28.camel@corn.betterworld.us> On Wed, 2009-09-02 at 08:44 -0400, Adam Tauno Williams wrote: > On Wed, 2009-09-02 at 14:30 +0200, Paolo Cravero wrote: > > Perhaps it is a bug in the documentation. > > 'man squatter' in the DESCRIPTION says there's no incremental update: > > "Squatter creates an index of ALL messages in the mailbox, not just those > > since the last time that it was run (i.e., it does NOT do incremental > > updates)." > > but in the OPTIONS a "-i" switch mentions incremental. > > Is it just a documentation error, so incremental indexing does work in 2.3.14? > > It may be a matter of interpretation. > > > Squatter creates an index of ALL messages in the mailbox, not just those > since time that it was run (i.e., it does NOT do incremental updates). > Any messages appended to the mailbox after squatter is run, will NOT > be included in the index. To include new messages in the index, squatter > must be run again. > > > I took "does NOT do incremental updates" to mean the index is not > updated on-the-fly when new messages enter the mailbox, only when > squatter is run. I think that is a leftover from 2.2, and the sense of incremental there was that when squatter runs reindexes the entire content, rather than reusing what it can from the existing index. So incremental referred to a batch, rather than real-time, update. Apparently 2.3 does have a batch incremental mode. That sounds like a huge win. Also, does anyone know what this means for searches on material that has changed since the last squatter run? I have assumed, and hope, that the search procedure is something like this: search in the squatter index remove results referring to deleted items do an unindexed search on items added since last index. Is that right? Or, for example, are new messages just ignored? Ross Boylan From robm at fastmail.fm Thu Sep 3 17:54:03 2009 From: robm at fastmail.fm (Rob Mueller) Date: Fri, 4 Sep 2009 07:54:03 +1000 Subject: incremental squatter References: <4A9E655A.2080101@as2594.net> <1251895499.5267.11.camel@linux-m3mt> <1251997066.29746.28.camel@corn.betterworld.us> Message-ID: <0C0F8D6686684DAD8B86DE53053B658E@jem> > Also, does anyone know what this means for searches on material that has > changed since the last squatter run? I have assumed, and hope, that the > search procedure is something like this: > search in the squatter index > remove results referring to deleted items > do an unindexed search on items added since last index. > > Is that right? Or, for example, are new messages just ignored? The squatter index isn't a perfect index. What it does is given a search term, it returns a list of messages that might contain the term, and excludes messages that definitely do not contain the search term. For each message that squatter says might contain the search term, cyrus then opens the message and does a complete search on it to see if it definitely contains the search term. Because of that, if squatter sees a message id it hasn't indexed, it will always return that id, because that id might contain the term, it doesn't know. The net result is that things work as expected. New messages that haven't been squatter indexed are always searched, you never miss messages. Rob From boutilpj at ednet.ns.ca Fri Sep 4 08:06:43 2009 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Fri, 04 Sep 2009 09:06:43 -0300 Subject: Question regarding ctl_mboxlist -u Message-ID: <4AA102D3.6000801@ednet.ns.ca> Cyrus 2.2.14 I have a user that I can not delete: [root at student2 cyrus]# /home/cyrus/deluser.pl cyrust Login: No Errors User ACL: OK Completed IMAP::Admin [ delete ]: couldn't delete user.cyrust : * BYE Fatal error: Internal error: assertion failed: cyrusdb_skiplist.c: 758: db->is_open So I plan on fixing this by dumping and then importing mailboxes.db using something like the following. My question is will this be sufficient? I wouldn't have to run any reconstructs afterwards would I? service cyrus stop /usr/local/cyrus/bin/ctl_mboxlist -d > /home/cyrus/mailboxes.txt grep -v 'user.cyrust' /home/cyrus/mailboxes.txt > /home/cyrus/mailboxes-new.txt cat /home/cyrus/mailboxes-new.txt|/usr/local/cyrus/bin/ctl_mboxlist -u service cyrus start -------------- next part -------------- A non-text attachment was scrubbed... Name: boutilpj.vcf Type: text/x-vcard Size: 286 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090904/3de9b93f/attachment.vcf From pcravero at as2594.net Fri Sep 4 09:53:11 2009 From: pcravero at as2594.net (Paolo Cravero) Date: Fri, 04 Sep 2009 15:53:11 +0200 Subject: cyrus-imapd in (veritas) cluster: anyone? Message-ID: <4AA11BC7.2060500@as2594.net> Hi. I've read Sims message of 2009.08.04 and following replies with interest. Unfortunately I couldn't find an answer (in archives) to the following question. Those using cyrus-imapd in a clustered environment, which cluster environment do they use? Veritas, RedHat, ... ? We're currently running an active-passive cluster with Sun Messaging under Veritas. There is a specific Veritas agent for that product that handles checks and switchover in case of node failure, but I guess nothing for cyrus. We will (try to) establish an active-passive cyrus cluster, possibly with Veritas. RedHat cluster as second choice. I would appreciate some pointers if someone has already been there. Have a nice weekend, Paolo Cravero From jmasterson at modwest.com Fri Sep 4 13:49:25 2009 From: jmasterson at modwest.com (John Masterson) Date: Fri, 04 Sep 2009 11:49:25 -0600 Subject: -authz no longer working? Message-ID: <4AA15325.7050308@modwest.com> I have searched high and low but am coming up with nothing on this: For years we used a simple script that invoked Cyrus::IMAP::Admin to connect as the cyrus admin user with -authz for a user we were interested in seeing what shared mailboxes they had access to. It stopped working last year some time, possibly with a cyrus upgrade (to debian's 2.2.13-10), but since we use the script only upon an employee exiting, I'm not sure. The behavior is that list("users/*) returns 0 mailboxes, even when the user certainly does have access to some shared mailboxes. I doublechecked this with the cyradm tool and the -authz argument, and got the same behavior. If I use cyradm to authenticate directly as the user in question, then I correctly see the shared mailboxes. But when I authenticate as the cyrus admin user and -authz as the user I am interested in, no such luck. Did some behavior change, or am I otherwise missing something? Thanks, -- John Masterson Modwest, Inc. http://www.modwest.com 1-888-549-0917 1-406-541-4678 From dwhite at olp.net Fri Sep 4 15:31:51 2009 From: dwhite at olp.net (Dan White) Date: Fri, 4 Sep 2009 14:31:51 -0500 Subject: -authz no longer working? In-Reply-To: <4AA15325.7050308@modwest.com> References: <4AA15325.7050308@modwest.com> Message-ID: <20090904193151.GH16139@dan.olp.net> On 04/09/09?11:49?-0600, John Masterson wrote: >If I use cyradm to authenticate directly as the user in question, then I >correctly see the shared mailboxes. > >But when I authenticate as the cyrus admin user and -authz as the user I >am interested in, no such luck. The ability to authz is controlled by the proxyservers option within /etc/imapd.conf. In your syslog (auth) log, you should not see the administrator connecting. When you successfully authz, you should see the proxied user connecting. -- Dan White From jmasterson at modwest.com Fri Sep 4 15:50:56 2009 From: jmasterson at modwest.com (John Masterson) Date: Fri, 04 Sep 2009 13:50:56 -0600 Subject: -authz no longer working? In-Reply-To: <20090904193151.GH16139@dan.olp.net> References: <4AA15325.7050308@modwest.com> <20090904193151.GH16139@dan.olp.net> Message-ID: <4AA16FA0.70405@modwest.com> Dan White wrote: > The ability to authz is controlled by the proxyservers option within > /etc/imapd.conf. > > In your syslog (auth) log, you should not see the administrator > connecting. When you successfully authz, you should see the proxied user > connecting. Thanks. The server I'm connecting to has the 'cyrus' user listed in proxyservers in /etc/imapd.conf: proxyservers: cyrus mupdate When i connect via cyradm and an authz argument, the log on the destination server indicates I am 'cyrus', not the user I provided in my authz arg: $ cyradm --user cyrus --authz johnm mbe1 IMAP Password: mbe1.msomt.modwest.com> Log: Sep 4 13:47:43 mbe1 cyrus/imap[17219]: login: mgmt.modwest.com [204.11.245.21] cyrus plaintext User logged in I am sure I am just missing something and appreciate the help. -- John Masterson Modwest, Inc. http://www.modwest.com 1-888-549-0917 1-406-541-4678 From dwhite at olp.net Fri Sep 4 16:13:16 2009 From: dwhite at olp.net (Dan White) Date: Fri, 4 Sep 2009 15:13:16 -0500 Subject: -authz no longer working? In-Reply-To: <4AA16FA0.70405@modwest.com> References: <4AA15325.7050308@modwest.com> <20090904193151.GH16139@dan.olp.net> <4AA16FA0.70405@modwest.com> Message-ID: <20090904201315.GM16139@dan.olp.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 04/09/09?13:50?-0600, John Masterson wrote: > Thanks. The server I'm connecting to has the 'cyrus' user listed in > proxyservers in /etc/imapd.conf: > > proxyservers: cyrus mupdate > > When i connect via cyradm and an authz argument, the log on the > destination server indicates I am 'cyrus', not the user I provided in my > authz arg: > > $ cyradm --user cyrus --authz johnm mbe1 > IMAP Password: > > mbe1.msomt.modwest.com> > > Log: > > Sep 4 13:47:43 mbe1 cyrus/imap[17219]: login: mgmt.modwest.com > [204.11.245.21] cyrus plaintext User logged in Looks like this problem is related to the mechanism that you are connecting with. If I explicitly request the 'PLAIN' mechanism, then I see: Sep 4 15:06:25 neo cyrus/imap[28930]: login: vpn.olp.net [67.217.151.100] dwhite PLAIN User logged in but if I do the LOGIN mechanism (which I think is really the RFC 3501 6.2.3 login command, which doesn't support authz), then I get something similar to what you're getting: Sep 4 15:06:00 neo cyrus/imap[28930]: login: vpn.olp.net [67.217.151.100] cyrus plaintext User logged in - -- Dan White -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkqhdNsACgkQjEHNWladFEXGCwCgp4/ZRh3/HN/hlHjTVhqz8n0U ILsAn2OH2p460aY+UXlbjktUtQqaMcmX =XjSP -----END PGP SIGNATURE----- From jmasterson at modwest.com Fri Sep 4 17:01:42 2009 From: jmasterson at modwest.com (John Masterson) Date: Fri, 04 Sep 2009 15:01:42 -0600 Subject: -authz no longer working? In-Reply-To: <20090904201315.GM16139@dan.olp.net> References: <4AA15325.7050308@modwest.com> <20090904193151.GH16139@dan.olp.net> <4AA16FA0.70405@modwest.com> <20090904201315.GM16139@dan.olp.net> Message-ID: <4AA18036.2060203@modwest.com> Dan White wrote: > Looks like this problem is related to the mechanism that you are connecting > with. If I explicitly request the 'PLAIN' mechanism, then I see: > > Sep 4 15:06:25 neo cyrus/imap[28930]: login: vpn.olp.net [67.217.151.100] > dwhite PLAIN User logged in > > but if I do the LOGIN mechanism (which I think is really the RFC 3501 6.2.3 > login command, which doesn't support authz), then I get something similar > to what you're getting: > > Sep 4 15:06:00 neo cyrus/imap[28930]: login: vpn.olp.net [67.217.151.100] > cyrus plaintext User logged in Looks like you're on to something! jmasterson at mgmt:~$ cyradm -user cyrus -authz magnafix -auth PLAIN mbe1 Password: IMAP Password: mbe1.msomt.modwest.com> mbe1.msomt.modwest.com> had to enter password twice, resulting in logs: Sep 4 14:59:09 mbe1 cyrus/imap[18587]: badlogin: mgmt.modwest.com [204.11.245.21] PLAIN [SASL(-16): encryption needed to use mechanism: security flags do not match required] Sep 4 14:59:39 mbe1 cyrus/imap[18587]: login: mgmt.modwest.com [204.11.245.21] cyrus plaintext User logged in That's something new for me to search for anyway. If you have a silver bullet for this, I do appreciate the help. -- John Masterson http://modwest.com From dwhite at olp.net Fri Sep 4 17:09:53 2009 From: dwhite at olp.net (Dan White) Date: Fri, 4 Sep 2009 16:09:53 -0500 Subject: -authz no longer working? In-Reply-To: <4AA18036.2060203@modwest.com> References: <4AA15325.7050308@modwest.com> <20090904193151.GH16139@dan.olp.net> <4AA16FA0.70405@modwest.com> <20090904201315.GM16139@dan.olp.net> <4AA18036.2060203@modwest.com> Message-ID: <20090904210952.GN16139@dan.olp.net> On 04/09/09?15:01?-0600, John Masterson wrote: > Sep 4 14:59:09 mbe1 cyrus/imap[18587]: badlogin: mgmt.modwest.com > [204.11.245.21] PLAIN [SASL(-16): encryption needed to use mechanism: > security flags do not match required] > Sep 4 14:59:39 mbe1 cyrus/imap[18587]: login: mgmt.modwest.com > [204.11.245.21] cyrus plaintext User logged in You probably do not have this turned on in /etc/imapd.conf: # Allow plaintext logins by default (SASL PLAIN) allowplaintext: yes You can either: * connect using TLS (which will provide the required security bits), then connect with PLAIN * enable the allowplaintext option * or connect with another mechanism (like DIGEST-MD5) which would also provide the appropriate level of network security. you might also need to adjust your sasl_minimum_layer setting. -- Dan White From jmasterson at modwest.com Fri Sep 4 19:13:02 2009 From: jmasterson at modwest.com (John Masterson) Date: Fri, 04 Sep 2009 17:13:02 -0600 Subject: [0.9] Re: -authz no longer working? In-Reply-To: <20090904210952.GN16139@dan.olp.net> References: <4AA15325.7050308@modwest.com> <20090904193151.GH16139@dan.olp.net> <4AA16FA0.70405@modwest.com> <20090904201315.GM16139@dan.olp.net> <4AA18036.2060203@modwest.com> <20090904210952.GN16139@dan.olp.net> Message-ID: <4AA19EFE.8010000@modwest.com> Dan White wrote: > On 04/09/09 15:01 -0600, John Masterson wrote: >> Sep 4 14:59:09 mbe1 cyrus/imap[18587]: badlogin: mgmt.modwest.com >> [204.11.245.21] PLAIN [SASL(-16): encryption needed to use mechanism: >> security flags do not match required] >> Sep 4 14:59:39 mbe1 cyrus/imap[18587]: login: mgmt.modwest.com >> [204.11.245.21] cyrus plaintext User logged in > > You probably do not have this turned on in /etc/imapd.conf: > > # Allow plaintext logins by default (SASL PLAIN) > allowplaintext: yes > > You can either: > > * connect using TLS (which will provide the required security bits), > then connect with PLAIN > > * enable the allowplaintext option > > * or connect with another mechanism (like DIGEST-MD5) which would also > provide the appropriate level of network security. > > you might also need to adjust your sasl_minimum_layer setting. > Hmm: $ cyradm -user cyrus -authz magnafix -auth DIGEST-MD5 mbe1 cyradm: cannot authenticate to server with DIGEST-MD5 as cyrus root at mbe1:~# grep allowplaintext: /etc/imapd.conf allowplaintext: yes lmtpproxy_allowplaintext: yes lmtp_allowplaintext: yes imap_allowplaintext: yes root at mbe1:~# grep sasl_min /etc/imapd.conf # sasl_minimum_layer and allowapop below, too. sasl_minimum_layer: 0 I will keep researching. Thanks for your help so far! -- John Masterson Modwest, Inc. http://www.modwest.com 1-888-549-0917 1-406-541-4678 From boutilpj at ednet.ns.ca Sat Sep 5 08:02:36 2009 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Sat, 05 Sep 2009 09:02:36 -0300 Subject: Question regarding ctl_mboxlist -u In-Reply-To: <4AA102D3.6000801@ednet.ns.ca> References: <4AA102D3.6000801@ednet.ns.ca> Message-ID: <14538_1252152159_n85C2cfM003606_4AA2535C.8020807@ednet.ns.ca> On 09/04/2009 09:06 AM, Patrick Boutilier wrote: > Cyrus 2.2.14 > > I have a user that I can not delete: > > [root at student2 cyrus]# /home/cyrus/deluser.pl cyrust > Login: No Errors > User ACL: OK Completed > IMAP::Admin [ delete ]: couldn't delete user.cyrust : * BYE Fatal error: > Internal error: assertion failed: cyrusdb_skiplist.c: 758: db->is_open > > > So I plan on fixing this by dumping and then importing mailboxes.db > using something like the following. My question is will this be > sufficient? I wouldn't have to run any reconstructs afterwards would I? > > > service cyrus stop > /usr/local/cyrus/bin/ctl_mboxlist -d > /home/cyrus/mailboxes.txt > grep -v 'user.cyrust' /home/cyrus/mailboxes.txt > > /home/cyrus/mailboxes-new.txt #Important to remove mailboxes.db at this point otherwise you are just #adding to mailboxes.db, not replacing it. mv /var/imap/mailboxes.db /var/imap/mailboxes.db.OLD > cat /home/cyrus/mailboxes-new.txt|/usr/local/cyrus/bin/ctl_mboxlist -u > service cyrus start > > > ------------------------------------------------------------------------ > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html From forrie at forrie.com Sat Sep 5 10:28:59 2009 From: forrie at forrie.com (Forrest Aldrich) Date: Sat, 05 Sep 2009 10:28:59 -0400 Subject: SYSLOG facility in *.conf ? Message-ID: <4AA275AB.4090000@forrie.com> Would it be feasible to enable configuring the syslog facility used by cyrus in the imapd.conf file? From awilliam at whitemice.org Sat Sep 5 12:47:30 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Sat, 05 Sep 2009 12:47:30 -0400 Subject: Resources for MUA support of IMAP features? In-Reply-To: <4A9F6767.5080808@as2594.net> References: <4A9F11F7.3070207@nakedape.cc> <4A9F6767.5080808@as2594.net> Message-ID: <1252169250.5109.7.camel@linux-m3mt> On Thu, 2009-09-03 at 08:51 +0200, Paolo Cravero wrote: > Wil Cooley wrote: > > Does anyone know of a mailing list or a web site with information about > > MUA support of various IMAP features? For example, for IMAP IDLE the > I once came across this wiki: > http://www.imapwiki.org/ > that links to this http://uplib.parc.com/misc/imapclients.html for your > specific question. That page looks familiar; do you have any idea if it is up-to-date? From brong at fastmail.fm Sat Sep 5 20:07:58 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sun, 6 Sep 2009 10:07:58 +1000 Subject: SYSLOG facility in *.conf ? In-Reply-To: <4AA275AB.4090000@forrie.com> References: <4AA275AB.4090000@forrie.com> Message-ID: <20090906000758.GA30228@brong.net> On Sat, Sep 05, 2009 at 10:28:59AM -0400, Forrest Aldrich wrote: > Would it be feasible to enable configuring the syslog facility used by > cyrus in the imapd.conf file? You can configure it at compile time. That said, I suspect we'd accept a patch if you wanted to write it. Just nobody can be particularly bothered writing it right now! We've got other fish to fry :) Bron. From forrie at forrie.com Sat Sep 5 22:24:19 2009 From: forrie at forrie.com (Forrest Aldrich) Date: Sat, 05 Sep 2009 22:24:19 -0400 Subject: SYSLOG facility in *.conf ? In-Reply-To: <20090906000758.GA30228@brong.net> References: <4AA275AB.4090000@forrie.com> <20090906000758.GA30228@brong.net> Message-ID: <4AA31D53.1070302@forrie.com> On 9/5/09 8:07 PM, Bron Gondwana wrote: > On Sat, Sep 05, 2009 at 10:28:59AM -0400, Forrest Aldrich wrote: > >> Would it be feasible to enable configuring the syslog facility used by >> cyrus in the imapd.conf file? >> > You can configure it at compile time. > > That said, I suspect we'd accept a patch if you wanted to write it. Just > nobody can be particularly bothered writing it right now! We've got other > fish to fry :) > > Bron. > Yes, I know about the compile time switch :-) I just think it would offer more flexibility if we could modify the syslog facility in the configuration file - it's a lot of work to recompile the lot, just for something simple like that. That said, I know there are alternate syslog servers (rsyslog) that can filter based on regex for which a syslog_prefix is provided for in the present imapd.conf. I'm not much of a programmer, but I suspect a patch wouldn't be too difficult -- but I may be wrong! :-) I'll look. Thanks, and have fun frying those fish ;-) Forrest From ask at andreas.kerber.name Sun Sep 6 04:35:36 2009 From: ask at andreas.kerber.name (Andreas S. Kerber) Date: Sun, 6 Sep 2009 10:35:36 +0200 Subject: Migrated Murder to 2.3 Message-ID: <20090906083536.GA21814@eb6.speedkom.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Just a heads up for anybody trying to update a murder to version 2.3. I've just upgraded several servers from 2.1.16 to 2.3.14 without major problems. As many pointed out before, the backends should be upgraded first. The mupdate master followed and the frontends where upgraded in the last step. There we're only minor problems, mostly related to my specific environment: * Had compilation problems on SuSE 9 due to lacking com_et (needed to install e2fsprogs-1.38) * Selection of bdb library used to newest library. Configure switches didn't help so I edited configure by hand * Had to add entry for mupdate service in /etc/services due to: "failed: Servname not supported for ai_socktype" error * LMTP delivery problems during upgrade from old frontends to new backends (uses another username for authentication) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) iD8DBQFKo3RYP0gnkXA27R8RAqSEAKCcvfbe6yRt7rwwgc/pwF/d1vyM2wCcD/I1 NhVt8ZEzUkOYh+wIG3mw7uM= =YWLJ -----END PGP SIGNATURE----- From ross at biostat.ucsf.edu Mon Sep 7 17:34:03 2009 From: ross at biostat.ucsf.edu (Ross Boylan) Date: Mon, 07 Sep 2009 14:34:03 -0700 Subject: Migrating 32bit to 64bit Debian Lenny In-Reply-To: <7f1434c60909020819s5d90fad7s1ccee10e74328a49@mail.gmail.com> References: <7f1434c60909020819s5d90fad7s1ccee10e74328a49@mail.gmail.com> Message-ID: <1252359243.17043.57.camel@corn.betterworld.us> Did you install cyrus on the new server before or after the steps described below? It sounds as if it was before, just by the references to shutting it down on the new server and to /usr/lib/cyrus/cyrus-db-types.active. Also, do you have any sense of the speed and accuracy of this method as opposed to imapsync? Ross Boylan On Wed, 2009-09-02 at 09:19 -0600, Bill Cameron wrote: > Hi, > > I've seen some questions about migrating from 32 bit to 64 bit. We > migrated last weekend and these are the steps we took. > > Old server: > - rsync the cyrus data while cyrus is running > rsync -vaH --delete -e ssh /var/lib/cyrus/ new-server:/var/lib/cyrus > rsync -vaH --delete -e ssh /var/spool/cyrus/ new-server:/var/spool/cyrus > rsync -vaH --delete -e ssh /var/spool/sieve/ new-server:/var/spool/sieve > - shut down cyrus > - repeat rsyncing of the three directories to provide stable > environment and databases. This will be a lot faster than the original > rsync. > - dump /var/lib/cyrus/mailboxes.db to a text file > /usr/sbin/ctl_mboxlist -d > mboxlist.txt > - copy this text file to the new server > > New server: > - make sure cyrus is shutdown > - switch to user 'cyrus' > su - cyrus > - remove some of the databases > rm /var/lib/cyrus/db/* > rm /var/lib/cyrus/db.backup1/* > rm /var/lib/cyrus/db.backup2/* > rm /var/lib/cyrus/deliver.db > rm /var/lib/cyrus/tls_sessions.db > rm /var/lib/cyrus/mailboxes.db > - build new mailboxes.db from mboxlist.txt file > /usr/sbin/ctl_mboxlist -u < mboxlist.txt > - check /var/log/mail.err and /var/log/mail.info for any errors > from the above command and the following commands. There should only > be one error about missing timestamp file but it is automatically > created. > - run the following commands and check logs for errors > /usr/sbin/ctl_cyrusdb -r > - the above command will verify mailboxes.db and annotations.db > /usr/sbin/tls_prune > - the above command will create a new tls_prune database > /usr/sbin/ctl_cyrusdb -c > /usr/sbin/cyr_expire -E 3 > - you can also run the squatter command but it doesn't really need > to run until it's scheduled time. It takes a awhile to run. > - start cyrus and check that it is working correctly. > - you will need to reset any annotations (e.g.: expire) on > mailboxes/folders. We didn't have any annotations set on mailboxes so > I didn't try migrating that database. > > The database types are defined in /etc/imapd.conf. They don't appear > in the default Lenny conf file since they use the predefined default > types. The man page for imapd.conf lists those defaults: > annotation_db: skiplist, duplicate_db: berkeley-nosync... The current > database types are listed in /usr/lib/cyrus/cyrus-db-types.active. > > We had to take the server off-line to migrate some other applications > on the server so I didn't use imapsync. We are using imapsync to > migrate from Lotus Notes to cyrus. > > Bill C. > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html From billcamer at gmail.com Mon Sep 7 19:16:20 2009 From: billcamer at gmail.com (Bill Cameron) Date: Mon, 7 Sep 2009 17:16:20 -0600 Subject: Migrating 32bit to 64bit Debian Lenny In-Reply-To: <1252359243.17043.57.camel@corn.betterworld.us> References: <7f1434c60909020819s5d90fad7s1ccee10e74328a49@mail.gmail.com> <1252359243.17043.57.camel@corn.betterworld.us> Message-ID: <7f1434c60909071616t3178f833jc65a0f7d9a53af3c@mail.gmail.com> On Mon, Sep 7, 2009 at 3:34 PM, Ross Boylan wrote: > Did you install cyrus on the new server before or after the steps > described below? ?It sounds as if it was before, just by the references > to shutting it down on the new server and > to /usr/lib/cyrus/cyrus-db-types.active. I installed the cyrus packages before transferring any data between servers. I then shut down cyrus on the new server and started rsyncing the data. > > Also, do you have any sense of the speed and accuracy of this method as > opposed to imapsync? > I've only used imapsync for moving mailboxes from Lotus Domino to cyrus. Domino is slow serving imap and it took longer to imapsync the same mailbox from Domino to cyrus then rsync it from 32 bit to 64 bit cyrus machines. I would think that imapsync would be slower because of all the imap communication overhead. The advantage of imapsync is that you don't have to shutdown either system. Bill C. > Ross Boylan From mail_list at computer-gott.de Tue Sep 8 03:43:15 2009 From: mail_list at computer-gott.de (mail_list at computer-gott.de) Date: Tue, 08 Sep 2009 09:43:15 +0200 Subject: Cyrus-imapd 2.3.14 - timsieved - Fedora 11-problem Message-ID: <1411545.1199401252395795437.JavaMail.servlet@kundenserver> Hi all ! on my installation , there's a problem with timsieved and horde-app "INGO". i don?t know if cyrus or horde causses this problem, so there ist also an similiar thread on lists.horde.org I am using Cyrus-imapd with webcyradm, mysql, amavisd, spamassasin, horde, clamd, postfix, postgrey,dcc, razo, pyzor. I installed all packages on Fedora 11 and it works fine, only the INGO app won?t work. INGO is an ap to activate and manipulate scripts for vacation or filtering with sieve ( in this case: timsieved). I get an "Not ins TRANSACTION state"if i try to activate any script. So the app can?t communicate with timsieved. i did some tests.: # sivtest -v localhost -u cyrus -a cyrus S: "IMPLEMENTATION" "Cyrus timsieved v2.3.14-Fedora-RPM-2.3.14-1.fc11" S: "SASL" "LOGIN DIGEST-MD5" S: "SIEVE" "comparator-i;ascii-numeric fileinto reject vacation imapflags notify envelope relational regex subaddress copy" S: "STARTTLS" S: OK C: AUTHENTICATE "DIGEST-MD5" S: {252} S: bm9uY2U9IkNkU0w2TllXbnkvMURUNEhmYXV........EZXQwLHJjNC01NixyYzQsZGVzLDNkZXMiLG1heGJ1Zj00MDk2LGNoYXJzZXQ9dXRmLTgsYWxnb3JpdGhtPW1kNS1zZXNz Please enter your password: C: {356+} C: dXNlcm5h...c0VGEwPSIsbmM9MDAwMDAwMDEscW9wPWF1dGgtY29uZixjaXBoZXI9cmM0LG1heGJ1Zj0xMDI0LGRpZ2VzdC11cmk9In.........zZT04MjFiYzM5NzA4OTU2MzgzYTVkODg2MGY4NGEwZjI2NA== S: OK (SASL "cnNwYXV0a.......MTY5ND...hNg==") Authenticated. Security strength factor: 128 Asking for capabilities again since they might have changed C: CAPABILITY S: "IMPLEMENTATION" "Cyrus timsieved v2.3.14-Fedora-RPM-2.3.14-1.fc11" S: "SASL" "LOGIN DIGEST-MD5" S: "SIEVE" "comparator-i;ascii-numeric fileinto reject vacation imapflags notify envelope relational regex subaddress copy" S: OK # ps fax | grep sieve 18744 pts/0 S+ 0:00 \_ grep sieve 18651 ? S 0:00 \_ timsieved 18659 ? S 0:00 \_ timsieved # telnet localhost 2000 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. "IMPLEMENTATION" "Cyrus timsieved v2.3.14-Fedora-RPM-2.3.14-1.fc11" "SASL" "LOGIN DIGEST-MD5" "SIEVE" "comparator-i;ascii-numeric fileinto reject vacation imapflags notify envelope relational regex subaddress copy" "STARTTLS" OK #cat /etc/cyrus conf | grep sieve sieve cmd="timsieved" listen="sieve" prefork=1 So, the daemon ist running. Maybe there is a also or just only problem with the pear net_sieve. Cause of requirements, i am not allowed to use a newer version of cyrus-imapd. Where is the problem ? Thanks Georg From michael.menge at zdv.uni-tuebingen.de Tue Sep 8 04:45:39 2009 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Tue, 08 Sep 2009 10:45:39 +0200 Subject: Cyrus-imapd 2.3.14 - timsieved - Fedora 11-problem In-Reply-To: <1411545.1199401252395795437.JavaMail.servlet@kundenserver> References: <1411545.1199401252395795437.JavaMail.servlet@kundenserver> Message-ID: <20090908104539.3851240j4ksk92nn@webmail.uni-tuebingen.de> Hi, Quoting mail_list at computer-gott.de: > Hi all ! > > on my installation , there's a problem with timsieved and horde-app "INGO". > i don?t know if cyrus or horde causses this problem, so there ist > also an similiar thread on lists.horde.org > I am using Cyrus-imapd with webcyradm, mysql, amavisd, spamassasin, > horde, clamd, postfix, postgrey,dcc, razo, pyzor. > > I installed all packages on Fedora 11 and it works fine, only the > INGO app won?t work. > INGO is an ap to activate and manipulate scripts for vacation or > filtering with sieve ( in this case: timsieved). > > I get an "Not ins TRANSACTION state"if i try to activate any script. > So the app can?t communicate with timsieved. > This error comes from pear Net_Sieve, and indicates that the Connection is not in an Authenticated state. In this state you can upload scripts for a syntax check, but you can't do much more. As one of the mails in the ingo Mailinglist indicates, this problem does not exist in 2.3.11. The CAPABILITY handling after TLS changed in 2.3.12 and caused some clients and libraries not working with cyrus. You may whant to have a look at the following bugs http://bugs.horde.org/ticket/6338 http://pear.php.net/bugs/bug.php?id=14205 > i did some tests.: > > # sivtest -v localhost -u cyrus -a cyrus > S: "IMPLEMENTATION" "Cyrus timsieved v2.3.14-Fedora-RPM-2.3.14-1.fc11" > S: "SASL" "LOGIN DIGEST-MD5" > S: "SIEVE" "comparator-i;ascii-numeric fileinto reject vacation > imapflags notify envelope relational regex subaddress copy" > S: "STARTTLS" > S: OK > C: AUTHENTICATE "DIGEST-MD5" > S: {252} > S: > bm9uY2U9IkNkU0w2TllXbnkvMURUNEhmYXV........EZXQwLHJjNC01NixyYzQsZGVzLDNkZXMiLG1heGJ1Zj00MDk2LGNoYXJzZXQ9dXRmLTgsYWxnb3JpdGhtPW1kNS1zZXNz > Please enter your password: > C: {356+} > C: > dXNlcm5h...c0VGEwPSIsbmM9MDAwMDAwMDEscW9wPWF1dGgtY29uZixjaXBoZXI9cmM0LG1heGJ1Zj0xMDI0LGRpZ2VzdC11cmk9In.........zZT04MjFiYzM5NzA4OTU2MzgzYTVkODg2MGY4NGEwZjI2NA== > S: OK (SASL "cnNwYXV0a.......MTY5ND...hNg==") > Authenticated. > Security strength factor: 128 > Asking for capabilities again since they might have changed > C: CAPABILITY > S: "IMPLEMENTATION" "Cyrus timsieved v2.3.14-Fedora-RPM-2.3.14-1.fc11" > S: "SASL" "LOGIN DIGEST-MD5" > S: "SIEVE" "comparator-i;ascii-numeric fileinto reject vacation > imapflags notify envelope relational regex subaddress copy" > S: OK > Does the logfiles indecate that the ingo session was authenticated? -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5339 bytes Desc: S/MIME krytographische Unterschrift Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090908/b7211d22/attachment.bin From bernd at firmix.at Tue Sep 8 09:57:54 2009 From: bernd at firmix.at (Bernd Petrovitsch) Date: Tue, 08 Sep 2009 15:57:54 +0200 Subject: Cyrus-Imapd on Linux-iSCSI Message-ID: <4417_1252418282_n88Dw193005410_1252418274.16073.116.camel@spike.firmix.at> An embedded and charset-unspecified text was scrubbed... Name: not available Url: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090908/2a34f238/attachment.ksh From simon.matter at invoca.ch Tue Sep 8 10:26:31 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Tue, 8 Sep 2009 16:26:31 +0200 Subject: Cyrus-Imapd on Linux-iSCSI In-Reply-To: <4417_1252418282_n88Dw193005410_1252418274.16073.116.camel@spike.firmix.at > References: <4417_1252418282_n88Dw193005410_1252418274.16073.116.camel@spike.firmix.at> Message-ID: > Hi all! > > We are running cyrus-imapd with all files (mailboxes, Berkeley-DB, > mmap()ed files) on an ext3 filesystem (with data=ordered) on a iSCSI > block device (and the actual storage is on a NetApp box if that matters) > on a Xeon 3.2GHz CPU (with 4 cores/2 HT thread - ). > The kernel is 2.6.18-128.el5 (from the RHEL5/CentOS-5 updates). > > On a low-(to-now-)volume test cluster, everything runs fine. On the > high-volume production cluster[0], we experienced a corrupted BDB. > Googling didn't yield any useful result on the cause (or hint on > improvement of the configuration). > > Does anyone have experience with such a setup? > Is it possible to (simply) move the __* files to a local drive (or a > tmpfs/ramfs/ramdisk)? > > Any hints or questions or feedback or ... is greatly appreciated, Hi, As a first step I suggest to get rid of the Berkeley-DB files and convert them to skiplist. Regards, Simon From bsd at todoo.biz Tue Sep 8 10:30:47 2009 From: bsd at todoo.biz (bsd) Date: Tue, 8 Sep 2009 16:30:47 +0200 Subject: unsubscribe Message-ID: <0B0E7399-5255-48E6-9311-6C1D1F0CD3E5@todoo.biz> unsubscribe From bernd at firmix.at Tue Sep 8 10:48:01 2009 From: bernd at firmix.at (Bernd Petrovitsch) Date: Tue, 08 Sep 2009 16:48:01 +0200 Subject: Cyrus-Imapd on Linux-iSCSI In-Reply-To: References: <4417_1252418282_n88Dw193005410_1252418274.16073.116.camel@spike.firmix.at> Message-ID: <4429_1252421298_n88EmGZF011904_1252421281.16073.127.camel@spike.firmix.at> An embedded and charset-unspecified text was scrubbed... Name: not available Url: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090908/f68bfd0a/attachment.ksh From simon.matter at invoca.ch Tue Sep 8 10:51:58 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Tue, 8 Sep 2009 16:51:58 +0200 Subject: Cyrus-Imapd on Linux-iSCSI In-Reply-To: <1252421281.16073.127.camel@spike.firmix.at> References: <4417_1252418282_n88Dw193005410_1252418274.16073.116.camel@spike.firmix.at> <1252421281.16073.127.camel@spike.firmix.at> Message-ID: <4438ef30d4d2e37160531350d6261f7a.squirrel@webmail.bi.corp.invoca.ch> > On Tue, 2009-09-08 at 16:26 +0200, Simon Matter wrote: > [ problems with delivery.db] >> > Any hints or questions or feedback or ... is greatly appreciated, > [...] >> As a first step I suggest to get rid of the Berkeley-DB files and >> convert >> them to skiplist. > > I knew I forgot something in the last mail;-) - the delivery.db is > pretty much the last on "berkeley. > Is there any performance impact on that? In theory yes, there may be an impact. But it works. And I never heard someone to go back from skiplist... > Any URLs? Check the list archives, this has been discussed many times. Regards, Simon From michael.menge at zdv.uni-tuebingen.de Tue Sep 8 10:56:16 2009 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Tue, 08 Sep 2009 16:56:16 +0200 Subject: Cyrus-Imapd on Linux-iSCSI In-Reply-To: <4417_1252418282_n88Dw193005410_1252418274.16073.116.camel@spike.firmix.at> References: <4417_1252418282_n88Dw193005410_1252418274.16073.116.camel@spike.firmix.at> Message-ID: <20090908165616.375635ezxkktj5ow@webmail.uni-tuebingen.de> Hi, Quoting Bernd Petrovitsch : > Hi all! > > We are running cyrus-imapd with all files (mailboxes, Berkeley-DB, > mmap()ed files) on an ext3 filesystem (with data=ordered) on a iSCSI > block device (and the actual storage is on a NetApp box if that matters) > on a Xeon 3.2GHz CPU (with 4 cores/2 HT thread - ). > The kernel is 2.6.18-128.el5 (from the RHEL5/CentOS-5 updates). > Don't know if i can help as we have an different setting cyrus-2.3.14, SLES 10, xfs on iSCSI and Skiplist instead of BDB. > On a low-(to-now-)volume test cluster, everything runs fine. On the > high-volume production cluster[0], we experienced a corrupted BDB. > Googling didn't yield any useful result on the cause (or hint on > improvement of the configuration). > Which Version of Cyrus did you use? Which DB was Corrupted? Did this happen only once or on a regulare base? > Does anyone have experience with such a setup? With the differences mentioned above, this setups works fine. -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5339 bytes Desc: S/MIME krytographische Unterschrift Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090908/d0c49bd8/attachment.bin From bernd at firmix.at Tue Sep 8 11:26:42 2009 From: bernd at firmix.at (Bernd Petrovitsch) Date: Tue, 08 Sep 2009 17:26:42 +0200 Subject: Cyrus-Imapd on Linux-iSCSI In-Reply-To: <20090908165616.375635ezxkktj5ow@webmail.uni-tuebingen.de> References: <4417_1252418282_n88Dw193005410_1252418274.16073.116.camel@spike.firmix.at> <20090908165616.375635ezxkktj5ow@webmail.uni-tuebingen.de> Message-ID: <4404_1252423610_n88FQmlo011780_1252423602.16073.141.camel@spike.firmix.at> An embedded and charset-unspecified text was scrubbed... Name: not available Url: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090908/ba2d1523/attachment.ksh From robm at fastmail.fm Wed Sep 9 00:51:20 2009 From: robm at fastmail.fm (Rob Mueller) Date: Wed, 9 Sep 2009 14:51:20 +1000 Subject: Cyrus-Imapd on Linux-iSCSI References: <4417_1252418282_n88Dw193005410_1252418274.16073.116.camel@spike.firmix.at><20090908165616.375635ezxkktj5ow@webmail.uni-tuebingen.de> <4404_1252423610_n88FQmlo011780_1252423602.16073.141.camel@spike.firmix.at> Message-ID: <75D8393E1A0C4205A0E19D44E76ECDD3@jem> > - cyrus-imapd-2.3.7 (from RHEL5/CentOS-5) with some minor patches in the > popd (UUID format and an enhancement to the authentication - both > shouldn't have any impact on the storage part) As I'm sure others will mention, this is a quite old cyrus now with many known bugs. You chould definitely consider upgrading to Simon Matters RPMs > It happened now and then and is somewhat inconvenient. > And it's not fun if the /var/log partition is full just because > cyr_expire stumbles over a corrupted BDB. We used to see this every now and then as well, but it's stopped since we fixed this bug: http://www.mail-archive.com/cyrus-devel at lists.andrew.cmu.edu/msg01123.html I'm not sure if Bron has commited this back to cyrus CVS for the next release yet, but if not, it should definitely get included. > We switched to iSCSI just recently - so that's the only really "new" > thing in the setup. I think it's either a coincidence (eg the bug is not iSCSI related), or if it is iSCSI related, it's not directly related, and probably just caused by timing race changes created by a change in the storage layer. Rob From brong at fastmail.fm Wed Sep 9 02:23:37 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 09 Sep 2009 16:23:37 +1000 Subject: Cyrus-Imapd on Linux-iSCSI In-Reply-To: <75D8393E1A0C4205A0E19D44E76ECDD3@jem> References: <4417_1252418282_n88Dw193005410_1252418274.16073.116.camel@spike.firmix.at><20090908165616.375635ezxkktj5ow@webmail.uni-tuebingen.de> <4404_1252423610_n88FQmlo011780_1252423602.16073.141.camel@spike.firmix.at> <75D8393E1A0C4205A0E19D44E76ECDD3@jem> Message-ID: <1252477417.5122.1333907623@webmail.messagingengine.com> On Wed, 09 Sep 2009 14:51 +1000, "Rob Mueller" wrote: > > It happened now and then and is somewhat inconvenient. > > And it's not fun if the /var/log partition is full just because > > cyr_expire stumbles over a corrupted BDB. > > We used to see this every now and then as well, but it's stopped since we > fixed this bug: > > http://www.mail-archive.com/cyrus-devel at lists.andrew.cmu.edu/msg01123.html > > I'm not sure if Bron has commited this back to cyrus CVS for the next > release yet, but if not, it should definitely get included. Yes - it's in CVS. It should come out tomorrow hopefully in the next release. Bron. -- Bron Gondwana brong at fastmail.fm From mail_list at computer-gott.de Wed Sep 9 04:04:46 2009 From: mail_list at computer-gott.de (Georg Stich) Date: Wed, 09 Sep 2009 10:04:46 +0200 Subject: Cyrus-imapd 2.3.14 - timsieved - Fedora 11-problem Message-ID: <31055638.1401361252483486668.JavaMail.servlet@kundenserver> >This error comes from pear Net_Sieve, and indicates that the Connection is >not in an Authenticated state. In this state you can upload scripts >for a syntax check, but you can't do much more. > >As one of the mails in the ingo Mailinglist indicates, this problem does not >exist in 2.3.11. The CAPABILITY handling after TLS changed in 2.3.12 and >caused some clients and libraries not working with cyrus. > >You may whant to have a look at the following bugs > >http://bugs.horde.org/ticket/6338 >http://pear.php.net/bugs/bug.php?id=14205 > Hi! Thank you. I added the lines to my Sieve.php (/usr/share/pear/Net/Sieve.php) Now ingo freezes. No information in the logs. > >> i did some tests.: >> >> # sivtest -v localhost -u cyrus -a cyrus >> S: "IMPLEMENTATION" "Cyrus timsieved v2.3.14-Fedora-RPM-2.3.14-1.fc11" >> S: "SASL" "LOGIN DIGEST-MD5" >> S: "SIEVE" "comparator-i;ascii-numeric fileinto reject vacation >> imapflags notify envelope relational regex subaddress copy" >> S: "STARTTLS" >> S: OK >> C: AUTHENTICATE "DIGEST-MD5" >> S: {252} > >> C: CAPABILITY >> S: "IMPLEMENTATION" "Cyrus timsieved v2.3.14-Fedora-RPM-2.3.14-1.fc11" >> S: "SASL" "LOGIN DIGEST-MD5" >> S: "SIEVE" "comparator-i;ascii-numeric fileinto reject vacation >> imapflags notify envelope relational regex subaddress copy" >> S: OK >> > >Does the logfiles indecate that the ingo session was authenticated? Hmm, i found no entries about it, but if i use the wrong password: # sieveshell -u user at dom.de -a user at dom.de localhost in the log appear a: do_auth : auth failure: [user=user at dom.de] [service=sieve] [realm=dom.de] [mech=pam] [reason=PAM auth error] so i think the authentication works. Thanks Georg PS: Greetings from Esslingen / Echterdingen From michael.menge at zdv.uni-tuebingen.de Wed Sep 9 04:17:17 2009 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Wed, 09 Sep 2009 10:17:17 +0200 Subject: Cyrus-imapd 2.3.14 - timsieved - Fedora 11-problem In-Reply-To: <31055638.1401361252483486668.JavaMail.servlet@kundenserver> References: <31055638.1401361252483486668.JavaMail.servlet@kundenserver> Message-ID: <20090909101717.12664nd0svaq24ct@webmail.uni-tuebingen.de> Quoting Georg Stich : > > Hi! > > Thank you. > I added the lines to my Sieve.php (/usr/share/pear/Net/Sieve.php) > > Now ingo freezes. > No information in the logs. > We use Net_Sieve 1.1.6 with the Patch form the Horde Bug-Tracker and Ingo 1.2.1 >> Does the logfiles indecate that the ingo session was authenticated? > > Hmm, i found no entries about it, but if i use the wrong password: > > # sieveshell -u user at dom.de -a user at dom.de localhost > > in the log appear a: > do_auth : auth failure: [user=user at dom.de] [service=sieve] > [realm=dom.de] [mech=pam] [reason=PAM auth error] > > so i think the authentication works. > After sucsessful authentication you can use Cyrus telemetry-logging to debug this problem. Cyrus will log all send and recieved data for an observed account. > > Thanks > > > Georg > > > > PS: Greetings from Esslingen / Echterdingen > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5339 bytes Desc: S/MIME krytographische Unterschrift Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090909/3169f16b/attachment.bin From michael.menge at zdv.uni-tuebingen.de Wed Sep 9 04:36:59 2009 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Wed, 09 Sep 2009 10:36:59 +0200 Subject: Cyrus-Imapd on Linux-iSCSI In-Reply-To: <4404_1252423610_n88FQmlo011780_1252423602.16073.141.camel@spike.firmix.at> References: <4417_1252418282_n88Dw193005410_1252418274.16073.116.camel@spike.firmix.at> <20090908165616.375635ezxkktj5ow@webmail.uni-tuebingen.de> <4404_1252423610_n88FQmlo011780_1252423602.16073.141.camel@spike.firmix.at> Message-ID: <20090909103659.173803s1puqxbup7@webmail.uni-tuebingen.de> Quoting Bernd Petrovitsch : > On Tue, 2009-09-08 at 16:56 +0200, Michael Menge wrote: > [...] >> Quoting Bernd Petrovitsch : > [...] >> > We are running cyrus-imapd with all files (mailboxes, Berkeley-DB, >> > mmap()ed files) on an ext3 filesystem (with data=ordered) on a iSCSI >> > block device (and the actual storage is on a NetApp box if that matters) >> > on a Xeon 3.2GHz CPU (with 4 cores/2 HT thread - ). >> > The kernel is 2.6.18-128.el5 (from the RHEL5/CentOS-5 updates). >> >> Don't know if i can help as we have an different setting >> cyrus-2.3.14, SLES 10, xfs on iSCSI and Skiplist instead of BDB. > How large is the setup? We have 6 server, each with ~3000-9000 users, about 300 imapds and few popds > We have ~25K mailboxes with ~1E6 mails per day (on the average) and run > up to 300 popds and some imapds in parallel. -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5339 bytes Desc: S/MIME krytographische Unterschrift Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090909/c87c811f/attachment-0001.bin From kai at ich-geh-kaputt.de Wed Sep 9 05:08:30 2009 From: kai at ich-geh-kaputt.de (Kai Moritz) Date: Wed, 09 Sep 2009 11:08:30 +0200 Subject: Caching imap proxy Message-ID: <1252487310.4113.15.camel@macbook> Hi all! We need something like a caching imap proxy. If an imap client accesses the proxy, instead of delegating the client to the real server, the proxy should first try to serve the request out of its cache. So that, if several clients access the proxy and ask for the contents of the same message in the same folder, it is only requested once from the original server. As far as I know, imap proxies normally does not do stuff like this. They only delegate requests to an original server. But the murder documentation sounds a little bit, like cyrus murder might be capable to do it. If cyrus murder cannot cache imap read requests, perhaps anyone has an idea, how this caching problem can be solved. Or is it only possible to cache imap request in the imap client itself? Greetings Kai Moritz From bernd at firmix.at Wed Sep 9 06:16:40 2009 From: bernd at firmix.at (Bernd Petrovitsch) Date: Wed, 09 Sep 2009 12:16:40 +0200 Subject: Cyrus-Imapd on Linux-iSCSI In-Reply-To: <75D8393E1A0C4205A0E19D44E76ECDD3@jem> References: <4417_1252418282_n88Dw193005410_1252418274.16073.116.camel@spike.firmix.at> <20090908165616.375635ezxkktj5ow@webmail.uni-tuebingen.de> <4404_1252423610_n88FQmlo011780_1252423602.16073.141.camel@spike.firmix.at> <75D8393E1A0C4205A0E19D44E76ECDD3@jem> Message-ID: <1252491400.24298.24.camel@spike.firmix.at> An embedded and charset-unspecified text was scrubbed... Name: not available Url: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090909/b36abb84/attachment.ksh From murch at andrew.cmu.edu Wed Sep 9 09:47:14 2009 From: murch at andrew.cmu.edu (Ken Murchison) Date: Wed, 09 Sep 2009 09:47:14 -0400 Subject: Cyrus IMAPd 2.2.13p1 & 2.3.15 Released Message-ID: <4AA7B1E2.9010507@andrew.cmu.edu> I'd like to announce the releases of Cyrus IMAPd 2.2.13p1 and 2.3.15. These releases should both be considered production quality. These releases are being made at this time to fix the potential buffer overflow vulnerability described in CERT VU#336053: http://www.kb.cert.org/vuls/id/336053 The 2.2.13p1 release is no different from 2.2.13 other than the buffer overflow fix. The 2.3.15 release contains several other non-critical bugfixes and feature enhancements. For full details, please see doc/changes.html and doc/install-upgrade.html which are included in the distribution. I'd personally like to thank Bron Gondwana of Fastmail.fm for finding and fixing the buffer overflow, as well as his numerous other contributions to the 2.3.15 release. URLs for these releases: ftp://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.2.13p1.tar.gz ftp://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.3.15.tar.gz or http://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.2.13p1.tar.gz http://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.3.15.tar.gz Questions and comments can be directed to info-cyrus at lists.andrew.cmu.edu (public list), or cyrus-bugs at andrew.cmu.edu. -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University From dbosso+lists.cyrus at lsit.ucsb.edu Wed Sep 9 13:07:31 2009 From: dbosso+lists.cyrus at lsit.ucsb.edu (David R Bosso) Date: Wed, 09 Sep 2009 10:07:31 -0700 Subject: Cyrus IMAPd 2.2.13p1 & 2.3.15 Released In-Reply-To: <4AA7B1E2.9010507@andrew.cmu.edu> References: <4AA7B1E2.9010507@andrew.cmu.edu> Message-ID: <05C20D0D3976979436F43D2B@host127-29.lsdepts.ucsb.edu> Is there a specific reason the patch for bug #3159 wasn't included? I've been adding it locally and just wondering if should continue to do so. Thanks. -David --On September 9, 2009 9:47:14 AM -0400 Ken Murchison wrote: > I'd like to announce the releases of Cyrus IMAPd 2.2.13p1 and 2.3.15. > These releases should both be considered production quality. These > releases are being made at this time to fix the potential buffer > overflow vulnerability described in CERT VU#336053: > http://www.kb.cert.org/vuls/id/336053 > > The 2.2.13p1 release is no different from 2.2.13 other than the buffer > overflow fix. The 2.3.15 release contains several other non-critical > bugfixes and feature enhancements. For full details, please see > doc/changes.html and doc/install-upgrade.html which are included in the > distribution. > > I'd personally like to thank Bron Gondwana of Fastmail.fm for finding > and fixing the buffer overflow, as well as his numerous other > contributions to the 2.3.15 release. > > URLs for these releases: > ftp://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.2.13p1.tar.gz > ftp://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.3.15.tar.gz > or > http://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.2.13p1.tar.gz > http://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.3.15.tar.gz > > > Questions and comments can be directed to > info-cyrus at lists.andrew.cmu.edu (public list), or > cyrus-bugs at andrew.cmu.edu. > > -- > Kenneth Murchison > Systems Programmer > Project Cyrus Developer/Maintainer > Carnegie Mellon University > > > > > > > > > > > > > > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html From Hagedorn at uni-koeln.de Wed Sep 9 13:31:51 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Wed, 09 Sep 2009 19:31:51 +0200 Subject: Cyrus IMAPd 2.2.13p1 & 2.3.15 Released In-Reply-To: <05C20D0D3976979436F43D2B@host127-29.lsdepts.ucsb.edu> References: <4AA7B1E2.9010507@andrew.cmu.edu> <05C20D0D3976979436F43D2B@host127-29.lsdepts.ucsb.edu> Message-ID: <2E590928ED0A30D6B2D65D5E@G5.local> -- David R Bosso is rumored to have mumbled on 9. September 2009 10:07:31 -0700 regarding Re: Cyrus IMAPd 2.2.13p1 & 2.3.15 Released: > Is there a specific reason the patch for bug #3159 wasn't included? I've > been adding it locally and just wondering if should continue to do so. While we're at it, what about #2642? We were recently bitten by that particular problem. -- Sebastian Hagedorn - RZKR-R1 (Flachbau), Zi. 18, Robert-Koch-Str. 10 Regionales Rechenzentrum (RRZK) Universit?t zu K?ln / Cologne University - Tel. +49-221-478-5587 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pkcs7-signature Size: 5292 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090909/c5716eb7/attachment.bin From awilliam at whitemice.org Wed Sep 9 13:40:17 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Wed, 09 Sep 2009 13:40:17 -0400 Subject: Cyrus IMAPd 2.2.13p1 & 2.3.15 Released In-Reply-To: <4AA7B1E2.9010507@andrew.cmu.edu> References: <4AA7B1E2.9010507@andrew.cmu.edu> Message-ID: <1252518017.5045.19.camel@linux-m3mt> On Wed, 2009-09-09 at 09:47 -0400, Ken Murchison wrote: > I'd like to announce the releases of Cyrus IMAPd 2.2.13p1 and 2.3.15. > These releases should both be considered production quality. These > releases are being made at this time to fix the potential buffer > overflow vulnerability described in CERT VU#336053: > http://www.kb.cert.org/vuls/id/336053 > The 2.2.13p1 release is no different from 2.2.13 other than the buffer > overflow fix. The 2.3.15 release contains several other non-critical > bugfixes and feature enhancements. For full details, please see > doc/changes.html and doc/install-upgrade.html which are included in the > distribution. > I'd personally like to thank Bron Gondwana of Fastmail.fm for finding > and fixing the buffer overflow, as well as his numerous other > contributions to the 2.3.15 release. I've update the Freshmeat entry for 2.3.15 From awilliam at whitemice.org Wed Sep 9 13:47:40 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Wed, 09 Sep 2009 13:47:40 -0400 Subject: Patch: add regex support to ipurge to improve folder selection In-Reply-To: <902EF0C4-D4B0-4244-8ABA-E3FE203A7C11@columbia.edu> References: <20090721144346.GB7271@open-xchange.com> <902EF0C4-D4B0-4244-8ABA-E3FE203A7C11@columbia.edu> Message-ID: <1252518460.5045.23.camel@linux-m3mt> On Tue, 2009-07-21 at 11:57 -0400, Matt Selsky wrote: > On Jul 21, 2009, at 10:43 AM, Carsten Hoeger wrote: > > > attached the patch to add regex pattern matching to folders for > > ipurge. > > Can you add this to bugzilla for tracking? Was this ever uploaded to the bugzilla? From simon.matter at invoca.ch Wed Sep 9 16:20:33 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Wed, 9 Sep 2009 22:20:33 +0200 Subject: Cyrus IMAPd 2.2.13p1 & 2.3.15 Released In-Reply-To: <4AA7B1E2.9010507@andrew.cmu.edu> References: <4AA7B1E2.9010507@andrew.cmu.edu> Message-ID: <47bfc51579f0edde512eb5b76e3c4dd8.squirrel@webmail.bi.corp.invoca.ch> > I'd like to announce the releases of Cyrus IMAPd 2.2.13p1 and 2.3.15. > These releases should both be considered production quality. These > releases are being made at this time to fix the potential buffer > overflow vulnerability described in CERT VU#336053: > http://www.kb.cert.org/vuls/id/336053 > > The 2.2.13p1 release is no different from 2.2.13 other than the buffer > overflow fix. The 2.3.15 release contains several other non-critical > bugfixes and feature enhancements. For full details, please see > doc/changes.html and doc/install-upgrade.html which are included in the > distribution. > > I'd personally like to thank Bron Gondwana of Fastmail.fm for finding > and fixing the buffer overflow, as well as his numerous other > contributions to the 2.3.15 release. Hello Cyrus IMAP team, Thanks for the new release. While upgrading our RPMs I found two small issues: 1) Old (ancient) GCC doesn't like some of the new code. A patch to fix the issue is attached. 2) Old (ancient) zlib doesn't have the deflateBound() function. Looks like at least zlib >= 1.2.x is needed. Maybe the zlib detection could also check the version of the deflateBound() function? For those interested, the package is available in the usual place http://www.invoca.ch/pub/packages/cyrus-imapd/ Regards, Simon -------------- next part -------------- A non-text attachment was scrubbed... Name: cyrus-imapd-2.3.15-oldgcc.patch Type: text/x-patch Size: 1744 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090909/faa76f9a/attachment.bin From brong at fastmail.fm Thu Sep 10 00:49:14 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 10 Sep 2009 14:49:14 +1000 Subject: Cyrus IMAPd 2.2.13p1 & 2.3.15 Released In-Reply-To: <2E590928ED0A30D6B2D65D5E@G5.local> References: <4AA7B1E2.9010507@andrew.cmu.edu> <05C20D0D3976979436F43D2B@host127-29.lsdepts.ucsb.edu> <2E590928ED0A30D6B2D65D5E@G5.local> Message-ID: <20090910044914.GB11606@brong.net> On Wed, Sep 09, 2009 at 07:31:51PM +0200, Sebastian Hagedorn wrote: > -- David R Bosso is rumored to > have mumbled on 9. September 2009 10:07:31 -0700 regarding Re: Cyrus > IMAPd 2.2.13p1 & 2.3.15 Released: > > >Is there a specific reason the patch for bug #3159 wasn't included? I've > >been adding it locally and just wondering if should continue to do so. Yeah, that's trivial and obviously. I've added it to my tree, and I'll push it back to CVS so it makes it into the next release for sure. > While we're at it, what about #2642? > > > > We were recently bitten by that particular problem. Now, that's a bit more complicated! Also, we use nginx in front of cyrus, so we don't use the built-in tls engine at all. I wouldn't feel comfortable testing this one. Is anyone running it on top of 2.3.14, or only on the 2.2 series? Bron. From brong at fastmail.fm Thu Sep 10 00:51:15 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 10 Sep 2009 14:51:15 +1000 Subject: Cyrus IMAPd 2.2.13p1 & 2.3.15 Released In-Reply-To: <47bfc51579f0edde512eb5b76e3c4dd8.squirrel@webmail.bi.corp.invoca.ch> References: <4AA7B1E2.9010507@andrew.cmu.edu> <47bfc51579f0edde512eb5b76e3c4dd8.squirrel@webmail.bi.corp.invoca.ch> Message-ID: <20090910045115.GC11606@brong.net> (un-CCed CERT, they don't care!) On Wed, Sep 09, 2009 at 10:20:33PM +0200, Simon Matter wrote: > > I'd like to announce the releases of Cyrus IMAPd 2.2.13p1 and 2.3.15. > > These releases should both be considered production quality. These > > releases are being made at this time to fix the potential buffer > > overflow vulnerability described in CERT VU#336053: > > http://www.kb.cert.org/vuls/id/336053 > > > > The 2.2.13p1 release is no different from 2.2.13 other than the buffer > > overflow fix. The 2.3.15 release contains several other non-critical > > bugfixes and feature enhancements. For full details, please see > > doc/changes.html and doc/install-upgrade.html which are included in the > > distribution. > > > > I'd personally like to thank Bron Gondwana of Fastmail.fm for finding > > and fixing the buffer overflow, as well as his numerous other > > contributions to the 2.3.15 release. > > Hello Cyrus IMAP team, > > Thanks for the new release. While upgrading our RPMs I found two small > issues: > > 1) Old (ancient) GCC doesn't like some of the new code. A patch to fix the > issue is attached. Applied to my git tree - I'll push it back to CVS. Thanks. > 2) Old (ancient) zlib doesn't have the deflateBound() function. Looks like > at least zlib >= 1.2.x is needed. Maybe the zlib detection could also > check the version of the deflateBound() function? It shouldn't be too hard to rewrite it to not use deflateBound() at all. I'll have a look at that. Who still has ancient zlib? RH 7.3? > For those interested, the package is available in the usual place > http://www.invoca.ch/pub/packages/cyrus-imapd/ Cool :) I don't actually have a redhat machine to test things on, but it's good to have these packages out there. Bron ( not everyone wants to be hand-building Cyrus all the time! ) From simon.matter at invoca.ch Thu Sep 10 02:41:13 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Thu, 10 Sep 2009 08:41:13 +0200 Subject: Cyrus IMAPd 2.2.13p1 & 2.3.15 Released In-Reply-To: <20090910045115.GC11606@brong.net> References: <4AA7B1E2.9010507@andrew.cmu.edu> <47bfc51579f0edde512eb5b76e3c4dd8.squirrel@webmail.bi.corp.invoca.ch> <20090910045115.GC11606@brong.net> Message-ID: <1f7dd6cc6951864401dc9129505dc48a.squirrel@webmail.bi.corp.invoca.ch> > (un-CCed CERT, they don't care!) > > On Wed, Sep 09, 2009 at 10:20:33PM +0200, Simon Matter wrote: >> > I'd like to announce the releases of Cyrus IMAPd 2.2.13p1 and 2.3.15. >> > These releases should both be considered production quality. These >> > releases are being made at this time to fix the potential buffer >> > overflow vulnerability described in CERT VU#336053: >> > http://www.kb.cert.org/vuls/id/336053 >> > >> > The 2.2.13p1 release is no different from 2.2.13 other than the buffer >> > overflow fix. The 2.3.15 release contains several other non-critical >> > bugfixes and feature enhancements. For full details, please see >> > doc/changes.html and doc/install-upgrade.html which are included in >> the >> > distribution. >> > >> > I'd personally like to thank Bron Gondwana of Fastmail.fm for finding >> > and fixing the buffer overflow, as well as his numerous other >> > contributions to the 2.3.15 release. >> >> Hello Cyrus IMAP team, >> >> Thanks for the new release. While upgrading our RPMs I found two small >> issues: >> >> 1) Old (ancient) GCC doesn't like some of the new code. A patch to fix >> the >> issue is attached. > > Applied to my git tree - I'll push it back to CVS. Thanks. Great, thanks. > >> 2) Old (ancient) zlib doesn't have the deflateBound() function. Looks >> like >> at least zlib >= 1.2.x is needed. Maybe the zlib detection could also >> check the version of the deflateBound() function? > > It shouldn't be too hard to rewrite it to not use deflateBound() at all. > I'll have a look at that. Sounds good, I was sure it could be done but not by me. If I try it nobody will be happy with the result :( > > Who still has ancient zlib? RH 7.3? Yes, or RHEL2.1, which is now EOL. Really, it's low priority but still nice to have as much compatibility as possible which makes life of packagers easier. Just let me know if you have patches to test... > >> For those interested, the package is available in the usual place >> http://www.invoca.ch/pub/packages/cyrus-imapd/ > > Cool :) I don't actually have a redhat machine to test things on, but > it's good to have these packages out there. > > Bron ( not everyone wants to be hand-building Cyrus all the time! ) Some years back I did quite a number of Cyrus installs for companies around the world. That's gone now because of my own packages which have even be included in RedHat EL. Regards, Simon From Duncan.Gibb at SiriusIT.co.uk Thu Sep 10 03:00:07 2009 From: Duncan.Gibb at SiriusIT.co.uk (Duncan Gibb) Date: Thu, 10 Sep 2009 08:00:07 +0100 Subject: Cyrus IMAPd 2.2.13p1 & 2.3.15 Released In-Reply-To: <20090910044914.GB11606@brong.net> References: <4AA7B1E2.9010507@andrew.cmu.edu> <05C20D0D3976979436F43D2B@host127-29.lsdepts.ucsb.edu> <2E590928ED0A30D6B2D65D5E@G5.local> <20090910044914.GB11606@brong.net> Message-ID: <4AA8A3F7.1020306@SiriusIT.co.uk> Bron Gondwana wrote: DBR> BG> we use nginx in front of cyrus, so we don't use the built-in BG> tls engine at all. I wouldn't feel comfortable testing this BG> one. Is anyone running it on top of 2.3.14, or only on the BG> 2.2 series? Not yet. I'm going test it on the Sirius tree (and hence sort out interactions with https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3119 and https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3133 ) once I've merged 2.3.15 there and into Debian mainline. "tls_require_cert optional" would open the possibility of doing both cert-based auth FE-to-BE and still allowing referrals with non-cert clients... Cheers Duncan -- Duncan Gibb - Technical Director Sirius Corporation plc - control through freedom http://www.siriusit.co.uk/ || t: +44 870 608 0063 Debian Cyrus Team - https://alioth.debian.org/projects/pkg-cyrus-imapd/ From choeger at open-xchange.com Thu Sep 10 03:03:51 2009 From: choeger at open-xchange.com (Carsten Hoeger) Date: Thu, 10 Sep 2009 09:03:51 +0200 Subject: Patch: add regex support to ipurge to improve folder selection In-Reply-To: <1252518460.5045.23.camel@linux-m3mt> References: <20090721144346.GB7271@open-xchange.com> <902EF0C4-D4B0-4244-8ABA-E3FE203A7C11@columbia.edu> <1252518460.5045.23.camel@linux-m3mt> Message-ID: <20090910070351.GA5374@open-xchange.com> On Wed, Sep 09, Adam Tauno Williams wrote: > On Tue, 2009-07-21 at 11:57 -0400, Matt Selsky wrote: > > On Jul 21, 2009, at 10:43 AM, Carsten Hoeger wrote: > > > > > attached the patch to add regex pattern matching to folders for > > > ipurge. > > > > Can you add this to bugzilla for tracking? > > Was this ever uploaded to the bugzilla? Yes: https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3167 -- With best regards, Carsten Hoeger -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090910/b4aa4a6c/attachment.bin From brong at fastmail.fm Thu Sep 10 03:04:50 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 10 Sep 2009 17:04:50 +1000 Subject: Cyrus IMAPd 2.2.13p1 & 2.3.15 Released In-Reply-To: <47bfc51579f0edde512eb5b76e3c4dd8.squirrel@webmail.bi.corp.invoca.ch> References: <4AA7B1E2.9010507@andrew.cmu.edu> <47bfc51579f0edde512eb5b76e3c4dd8.squirrel@webmail.bi.corp.invoca.ch> Message-ID: <20090910070450.GA507@brong.net> On Wed, Sep 09, 2009 at 10:20:33PM +0200, Simon Matter wrote: > 2) Old (ancient) zlib doesn't have the deflateBound() function. Looks like > at least zlib >= 1.2.x is needed. Maybe the zlib detection could also > check the version of the deflateBound() function? Try this patch (attached) and see if that is happy with your old zlib) Bron. -------------- next part -------------- A non-text attachment was scrubbed... Name: zlib.diff Type: text/x-diff Size: 6777 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090910/e9d03ae2/attachment.bin From kmauz at htwg-konstanz.de Thu Sep 10 03:10:57 2009 From: kmauz at htwg-konstanz.de (Konrad Mauz) Date: Thu, 10 Sep 2009 09:10:57 +0200 Subject: Cyrus IMAPd 2.3.15 - Fatal error: cyrusdb backend berkeley-nosync not supported Message-ID: <20090910071057.GA4852@rz-ap-mauz.htwg-konstanz.de> Hi all, I've tried to upgrade Cyrus from 2.3.8 to 2.3.15. I compiled 2.3.15 on the same machines with the same configure call: ./configure --prefix=/usr/cyrus --enable-nntp \ --enable-murder --enable-replication \ --enable-listext --enable-netscapehack --without-bdb But wenn I tried to restart imapd and pop3d process generate a lot of error messages like: Sep 10 06:44:11 mailsrv master[10325]: service pop3 pid 11258 in READY state: terminated abnormally Sep 10 06:44:11 mailsrv imap[11259]: Fatal error: cyrusdb backend berkeley-nosync not supported Sep 10 06:44:11 mailsrv pop3[11260]: Fatal error: cyrusdb backend berkeley-nosync not supported Sep 10 06:44:11 mailsrv lmtpunix[11261]: FATAL: cyrusdb backend berkeley-nosync not supported Sep 10 06:44:11 mailsrv imap[11262]: Fatal error: cyrusdb backend berkeley-nosync not supported Sep 10 06:44:11 mailsrv pop3[11263]: Fatal error: cyrusdb backend berkeley-nosync not supported Sep 10 06:44:11 mailsrv master[10325]: service imap pid 11259 in READY state: terminated abnormally Sep 10 06:44:11 mailsrv master[10325]: service pop3 pid 11260 in READY state: terminated abnormally Sep 10 06:44:11 mailsrv master[10325]: service lmtpunix pid 11261 in READY state: terminated abnormally Sep 10 06:44:11 mailsrv master[10325]: service imap pid 11262 in READY state: terminated abnormally Sep 10 06:44:11 mailsrv master[10325]: service pop3 pid 11263 in READY state: terminated abnormally Sep 10 06:44:11 mailsrv imap[11264]: Fatal error: cyrusdb backend berkeley-nosync not supported Sep 10 06:44:11 mailsrv pop3[11265]: Fatal error: cyrusdb backend berkeley-nosync not supported The relevant part in /etc/imapd.conf: # DB Settings mboxlist_db: skiplist seenstate_db: skiplist quota_db: skiplist duplicate_db: skiplist annotation_db: skiplist ptscache_db: skiplist subscription_db: flat tlscache_db: skiplist I can not find any reference to a berkley db, so I don't know why the daemons thow errors about the missing berkeley support. The only to dbs which are berkeley-nosync by default are: duplicate_db and tlscache_db, but I have them set to skiplist in my config. Can anyone tell me, which db defaultsetting is berkeley-nosync and not overwritten in my imapd.conf? Regards, Konrad PS: the old imapd binary are not linked with berkeley db - checked with ldd -- Konrad Mauz Rechenzentrum Hochschule Technik, Wirtschaft und Gestaltung Braunegger-Strasse 55, D 78462 Konstanz e-mail: kmauz at htwg-konstanz.de Tel.: +49 7531 206-472 Fax.: +49 7531 206-153 From simon.matter at invoca.ch Thu Sep 10 03:22:00 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Thu, 10 Sep 2009 09:22:00 +0200 Subject: Cyrus IMAPd 2.3.15 - Fatal error: cyrusdb backend berkeley-nosync not supported In-Reply-To: <20090910071057.GA4852@rz-ap-mauz.htwg-konstanz.de> References: <20090910071057.GA4852@rz-ap-mauz.htwg-konstanz.de> Message-ID: <11f5ec38d49c919e6a629ab83244cb15.squirrel@webmail.bi.corp.invoca.ch> > Hi all, > > I've tried to upgrade Cyrus from 2.3.8 to 2.3.15. > I compiled 2.3.15 on the same machines with the same configure call: > > ./configure --prefix=/usr/cyrus --enable-nntp \ > --enable-murder --enable-replication \ > --enable-listext --enable-netscapehack --without-bdb > > But wenn I tried to restart imapd and pop3d process generate a lot > of error messages like: > > Sep 10 06:44:11 mailsrv master[10325]: service pop3 pid 11258 in READY > state: terminated abnormally > Sep 10 06:44:11 mailsrv imap[11259]: Fatal error: cyrusdb backend > berkeley-nosync not supported > Sep 10 06:44:11 mailsrv pop3[11260]: Fatal error: cyrusdb backend > berkeley-nosync not supported > Sep 10 06:44:11 mailsrv lmtpunix[11261]: FATAL: cyrusdb backend > berkeley-nosync not supported > Sep 10 06:44:11 mailsrv imap[11262]: Fatal error: cyrusdb backend > berkeley-nosync not supported > Sep 10 06:44:11 mailsrv pop3[11263]: Fatal error: cyrusdb backend > berkeley-nosync not supported > Sep 10 06:44:11 mailsrv master[10325]: service imap pid 11259 in READY > state: terminated abnormally > Sep 10 06:44:11 mailsrv master[10325]: service pop3 pid 11260 in READY > state: terminated abnormally > Sep 10 06:44:11 mailsrv master[10325]: service lmtpunix pid 11261 in READY > state: terminated abnormally > Sep 10 06:44:11 mailsrv master[10325]: service imap pid 11262 in READY > state: terminated abnormally > Sep 10 06:44:11 mailsrv master[10325]: service pop3 pid 11263 in READY > state: terminated abnormally > Sep 10 06:44:11 mailsrv imap[11264]: Fatal error: cyrusdb backend > berkeley-nosync not supported > Sep 10 06:44:11 mailsrv pop3[11265]: Fatal error: cyrusdb backend > berkeley-nosync not supported > > The relevant part in /etc/imapd.conf: > > # DB Settings > mboxlist_db: skiplist > seenstate_db: skiplist > quota_db: skiplist > duplicate_db: skiplist > annotation_db: skiplist > ptscache_db: skiplist > subscription_db: flat > tlscache_db: skiplist > > I can not find any reference to a berkley db, so I don't know why the > daemons thow errors about > the missing berkeley support. The only to dbs which are berkeley-nosync by > default are: > duplicate_db and tlscache_db, but I have them set to skiplist in my > config. > > Can anyone tell me, which db defaultsetting is berkeley-nosync and not > overwritten in my imapd.conf? That may be 'statuscache_db'. Check man imapd.conf to be sure. Regards, Simon From simon.matter at invoca.ch Thu Sep 10 03:59:32 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Thu, 10 Sep 2009 09:59:32 +0200 Subject: Cyrus IMAPd 2.2.13p1 & 2.3.15 Released In-Reply-To: <20090910070450.GA507@brong.net> References: <4AA7B1E2.9010507@andrew.cmu.edu> <47bfc51579f0edde512eb5b76e3c4dd8.squirrel@webmail.bi.corp.invoca.ch> <20090910070450.GA507@brong.net> Message-ID: > On Wed, Sep 09, 2009 at 10:20:33PM +0200, Simon Matter wrote: >> 2) Old (ancient) zlib doesn't have the deflateBound() function. Looks >> like >> at least zlib >= 1.2.x is needed. Maybe the zlib detection could also >> check the version of the deflateBound() function? > > Try this patch (attached) and see if that is happy with your old zlib) Hi Bron, Now it builds fine even with old zlib. While we are at it, and because this patch affects the Clean-Shutdown patch - are there any plans to include the Clean-Shutdown patch into upstream? It has worked very well for us in the past and I'd love to see this one go in. Thanks, Simon From kmauz at htwg-konstanz.de Thu Sep 10 04:21:10 2009 From: kmauz at htwg-konstanz.de (Konrad Mauz) Date: Thu, 10 Sep 2009 10:21:10 +0200 Subject: Cyrus IMAPd 2.3.15 - Fatal error: cyrusdb backend berkeley-nosync not supported In-Reply-To: <20090910071057.GA4852@rz-ap-mauz.htwg-konstanz.de> References: <20090910071057.GA4852@rz-ap-mauz.htwg-konstanz.de> Message-ID: <20090910082110.GC4852@rz-ap-mauz.htwg-konstanz.de> On Thu, Sep 10, 2009 at 09:10:57AM +0200, Konrad Mauz wrote: > Hi all, > > I've tried to upgrade Cyrus from 2.3.8 to 2.3.15. > I compiled 2.3.15 on the same machines with the same configure call: > > ./configure --prefix=/usr/cyrus --enable-nntp \ > --enable-murder --enable-replication \ > --enable-listext --enable-netscapehack --without-bdb > > But wenn I tried to restart imapd and pop3d process generate a lot > of error messages like: > > Sep 10 06:44:11 mailsrv master[10325]: service pop3 pid 11258 in READY state: terminated abnormally > Sep 10 06:44:11 mailsrv imap[11259]: Fatal error: cyrusdb backend berkeley-nosync not supported > Sep 10 06:44:11 mailsrv pop3[11260]: Fatal error: cyrusdb backend berkeley-nosync not supported > Sep 10 06:44:11 mailsrv lmtpunix[11261]: FATAL: cyrusdb backend berkeley-nosync not supported > Sep 10 06:44:11 mailsrv imap[11262]: Fatal error: cyrusdb backend berkeley-nosync not supported > Sep 10 06:44:11 mailsrv pop3[11263]: Fatal error: cyrusdb backend berkeley-nosync not supported > Sep 10 06:44:11 mailsrv master[10325]: service imap pid 11259 in READY state: terminated abnormally > Sep 10 06:44:11 mailsrv master[10325]: service pop3 pid 11260 in READY state: terminated abnormally > Sep 10 06:44:11 mailsrv master[10325]: service lmtpunix pid 11261 in READY state: terminated abnormally > Sep 10 06:44:11 mailsrv master[10325]: service imap pid 11262 in READY state: terminated abnormally > Sep 10 06:44:11 mailsrv master[10325]: service pop3 pid 11263 in READY state: terminated abnormally > Sep 10 06:44:11 mailsrv imap[11264]: Fatal error: cyrusdb backend berkeley-nosync not supported > Sep 10 06:44:11 mailsrv pop3[11265]: Fatal error: cyrusdb backend berkeley-nosync not supported > > The relevant part in /etc/imapd.conf: > > # DB Settings > mboxlist_db: skiplist > seenstate_db: skiplist > quota_db: skiplist > duplicate_db: skiplist > annotation_db: skiplist > ptscache_db: skiplist > subscription_db: flat > tlscache_db: skiplist > > I can not find any reference to a berkley db, so I don't know why the daemons thow errors about > the missing berkeley support. The only to dbs which are berkeley-nosync by default are: > duplicate_db and tlscache_db, but I have them set to skiplist in my config. > > Can anyone tell me, which db defaultsetting is berkeley-nosync and not overwritten in my imapd.conf? > I missed the the new feature "statuscache" and statuscache_db ( since version 2.3.11 ). The db statuscache_db is by default berkeley-nosync. Thanks to Simon Matter for this usefull hint. Regards, Konrad -- Konrad Mauz Rechenzentrum Hochschule Technik, Wirtschaft und Gestaltung Braunegger-Strasse 55, D 78462 Konstanz e-mail: kmauz at htwg-konstanz.de Tel.: +49 7531 206-472 Fax.: +49 7531 206-153 From brong at fastmail.fm Thu Sep 10 05:02:28 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 10 Sep 2009 19:02:28 +1000 Subject: Cyrus IMAPd 2.2.13p1 & 2.3.15 Released In-Reply-To: References: <4AA7B1E2.9010507@andrew.cmu.edu> <47bfc51579f0edde512eb5b76e3c4dd8.squirrel@webmail.bi.corp.invoca.ch> <20090910070450.GA507@brong.net> Message-ID: <20090910090228.GA6426@brong.net> On Thu, Sep 10, 2009 at 09:59:32AM +0200, Simon Matter wrote: > > On Wed, Sep 09, 2009 at 10:20:33PM +0200, Simon Matter wrote: > >> 2) Old (ancient) zlib doesn't have the deflateBound() function. Looks > >> like > >> at least zlib >= 1.2.x is needed. Maybe the zlib detection could also > >> check the version of the deflateBound() function? > > > > Try this patch (attached) and see if that is happy with your old zlib) > > Hi Bron, > > Now it builds fine even with old zlib. Great. I've tested it with my development thunderbird copy that has COMPRESS=DEFLATE support built in. Haven't rolled it out to production yet, but probably will in a few hours once I'm happy that everything has rebased correctly. > While we are at it, and because this patch affects the Clean-Shutdown > patch - are there any plans to include the Clean-Shutdown patch into > upstream? Yeah, like this. I have a fixed clean-shutdown that applies over the zlib changes in my github repository. I went through and re-ordered everything so I could pop this to the top of the patch queue. > It has worked very well for us in the past and I'd love to see this one go > in. Yeah - I need to make sure it's squeeky-clean safe, but it's been pretty good for us too. We do often have hundreds of processes left after 10 seconds though, waiting on locks - which kind of sucks. Would be nice to fix that too! Bron. From Hagedorn at uni-koeln.de Thu Sep 10 06:56:06 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Thu, 10 Sep 2009 12:56:06 +0200 Subject: Cyrus IMAPd 2.2.13p1 & 2.3.15 Released In-Reply-To: <20090910044914.GB11606@brong.net> References: <4AA7B1E2.9010507@andrew.cmu.edu> <05C20D0D3976979436F43D2B@host127-29.lsdepts.ucsb.edu> <2E590928ED0A30D6B2D65D5E@G5.local> <20090910044914.GB11606@brong.net> Message-ID: <8FAC403E9DE870BCC70B8A22@G5.local> -- Bron Gondwana is rumored to have mumbled on 10. September 2009 14:49:14 +1000 regarding Re: Cyrus IMAPd 2.2.13p1 & 2.3.15 Released: >> While we're at it, what about #2642? >> >> >> >> We were recently bitten by that particular problem. > > Now, that's a bit more complicated! Also, we use nginx in front > of cyrus, so we don't use the built-in tls engine at all. I > wouldn't feel comfortable testing this one. Is anyone running it > on top of 2.3.14, or only on the 2.2 series? I have to admit that we currently don't. I applied the patch, compiled successfully, copied over just the imapd binary, created a special entry in cyrus.conf, that used that binary, and found that it segfaulted. I didn't investigate further, because I was under pressure. I reversed the patch and added just a single line in tls.c: askcert = 0; That worked, but of course that disables all client certificates. Because I didn't want to maintain a local patch, I came up with this workaround: Still, that's not exactly a clean solution. So ideally the patch in #2642 should be made to work in 2.3.15 ... I'd be happy to beta-test it if someone can clean it up. -- Sebastian Hagedorn - RZKR-R1 (Flachbau), Zi. 18, Robert-Koch-Str. 10 Regionales Rechenzentrum (RRZK) Universit?t zu K?ln / Cologne University - Tel. +49-221-478-5587 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pkcs7-signature Size: 5292 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090910/d467a286/attachment.bin From rosenbaumlm at ornl.gov Thu Sep 10 16:11:37 2009 From: rosenbaumlm at ornl.gov (Rosenbaum, Larry M.) Date: Thu, 10 Sep 2009 16:11:37 -0400 Subject: v2.3.15 IMAP connection hangs Message-ID: <43C68785C2728049AF86B0ECB240A151117DB6E4E7@EXCHMB.ornl.gov> I have just installed v2.3.15 on Solaris 9 Sparc with Cyrus-SASL 2.1.24rc1. When I connect to the IMAP port, I get a banner message but there is no response to any command I give it (including "0 logout"). I am able to give commands to the POP3 port and LMTP port just fine. Previous versions of Cyrus IMAP work OK too. Where should I look for the problem? Thanks, Larry -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090910/37d40bd5/attachment.html From brong at fastmail.fm Thu Sep 10 18:10:36 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 11 Sep 2009 08:10:36 +1000 Subject: v2.3.15 IMAP connection hangs In-Reply-To: <43C68785C2728049AF86B0ECB240A151117DB6E4E7@EXCHMB.ornl.gov> References: <43C68785C2728049AF86B0ECB240A151117DB6E4E7@EXCHMB.ornl.gov> Message-ID: <20090910221036.GA1177@brong.net> On Thu, Sep 10, 2009 at 04:11:37PM -0400, Rosenbaum, Larry M. wrote: > I have just installed v2.3.15 on Solaris 9 Sparc with Cyrus-SASL 2.1.24rc1. When I connect to the IMAP port, I get a banner message but there is no response to any command I give it (including "0 logout"). I am able to give commands to the POP3 port and LMTP port just fine. Previous versions of Cyrus IMAP work OK too. Where should I look for the problem? Can you get a trace on the process (truss or similar) that shows what file it's sitting waiting on a lock for? It's probably a locking issue. Bron. From simon.matter at invoca.ch Fri Sep 11 01:42:06 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Fri, 11 Sep 2009 07:42:06 +0200 Subject: Cyrus IMAPd 2.2.13p1 & 2.3.15 Released In-Reply-To: <20090910090228.GA6426@brong.net> References: <4AA7B1E2.9010507@andrew.cmu.edu> <47bfc51579f0edde512eb5b76e3c4dd8.squirrel@webmail.bi.corp.invoca.ch> <20090910070450.GA507@brong.net> <20090910090228.GA6426@brong.net> Message-ID: <3927454ba5e0bdf7ee60497593c2d33a.squirrel@webmail.bi.corp.invoca.ch> > > I have a fixed clean-shutdown that applies over the zlib changes in my > github repository. I went through and re-ordered everything so I could > pop this to the top of the patch queue. > >> It has worked very well for us in the past and I'd love to see this one >> go >> in. > > Yeah - I need to make sure it's squeeky-clean safe, but it's been > pretty good for us too. We do often have hundreds of processes > left after 10 seconds though, waiting on locks - which kind of > sucks. Would be nice to fix that too! What I saw on some hosts while upgrading from 2.3.14 to 2.3.15 is that I had idled processes hanging around. Looks like they have not been terminated. I tried to reproduce it on several hosts but I don't get stuck idled's now. Maybe it just happens on busy servers or after they ran for some time. Regards, Simon From rosenbaumlm at ornl.gov Fri Sep 11 09:22:47 2009 From: rosenbaumlm at ornl.gov (Rosenbaum, Larry M.) Date: Fri, 11 Sep 2009 09:22:47 -0400 Subject: v2.3.15 IMAP connection hangs In-Reply-To: <20090910221036.GA1177@brong.net> References: <43C68785C2728049AF86B0ECB240A151117DB6E4E7@EXCHMB.ornl.gov> <20090910221036.GA1177@brong.net> Message-ID: <43C68785C2728049AF86B0ECB240A151117DB6E653@EXCHMB.ornl.gov> > From: Bron Gondwana [mailto:brong at fastmail.fm] > > On Thu, Sep 10, 2009 at 04:11:37PM -0400, Rosenbaum, Larry M. wrote: > > I have just installed v2.3.15 on Solaris 9 Sparc with Cyrus-SASL > 2.1.24rc1. When I connect to the IMAP port, I get a banner message but > there is no response to any command I give it (including "0 logout"). > I am able to give commands to the POP3 port and LMTP port just fine. > Previous versions of Cyrus IMAP work OK too. Where should I look for > the problem? > > Can you get a trace on the process (truss or similar) that shows what > file > it's sitting waiting on a lock for? It's probably a locking issue. Here is a truss showing an attempt at running cyradm: ornl71# truss -f -p 18499 18499: accept(4, 0x00000000, 0x00000000, 1) (sleeping...) 18499: accept(4, 0x00000000, 0x00000000, 1) = 10 18499: fcntl(9, F_SETLKW, 0xFFBFF290) = 0 18499: alarm(0) = 0 18499: getpid() = 18499 [17305] 18499: write(3, "\0\0\002\0\0 H C", 8) = 8 18499: fstat(5, 0xFFBFEFC8) = 0 18499: time() = 1252675062 18499: getpid() = 18499 [17305] 18499: putmsg(5, 0xFFBFE680, 0xFFBFE674, 0) = 0 18499: open("/var/run/syslog_door", O_RDONLY) = 11 18499: door_info(11, 0xFFBFE5B8) = 0 18499: getpid() = 18499 [17305] 18499: door_call(11, 0xFFBFE5A0) = 0 18499: close(11) = 0 18499: fcntl(10, F_DUP2FD, 0x00000000) = 0 18499: fcntl(10, F_DUP2FD, 0x00000001) = 1 18499: fcntl(10, F_DUP2FD, 0x00000002) = 2 18499: close(10) = 0 18499: getpid() = 18499 [17305] 18499: write(3, "\0\0\003\0\0 H C", 8) = 8 18499: getpeername(0, 0xFFBFF010, 0xFFBFEB84, 1) = 0 18499: open("/etc/netconfig", O_RDONLY|O_LARGEFILE) = 10 18499: fcntl(10, F_DUPFD, 0x00000100) Err#22 EINVAL 18499: read(10, " # p r a g m a i d e n".., 1024) = 1024 18499: read(10, " t s t p i _ c".., 1024) = 215 18499: read(10, 0x00136728, 1024) = 0 18499: lseek(10, 0, SEEK_SET) = 0 18499: read(10, " # p r a g m a i d e n".., 1024) = 1024 18499: read(10, " t s t p i _ c".., 1024) = 215 18499: read(10, 0x00136728, 1024) = 0 18499: close(10) = 0 18499: open("/dev/udp", O_RDONLY) = 10 18499: ioctl(10, 0xC00C6982, 0xFFBFE8FC) = 0 18499: close(10) = 0 18499: open64("/var/run/name_service_door", O_RDONLY) = 10 18499: fcntl(10, F_SETFD, 0x00000001) = 0 18499: door_info(10, 0xFEEC26C0) = 0 18499: door_call(10, 0xFFBFC868) = 0 18499: door_info(10, 0xFFBFC800) = 0 18499: door_call(10, 0xFFBFC7E8) = 0 18499: getsockname(0, 0xFFBFF110, 0xFFBFEB84, 1) = 0 18499: brk(0x0013D190) = 0 18499: brk(0x0013F190) = 0 18499: getpid() = 18499 [17305] 18499: open("/usr/local/cyrus/data/proc/18499", O_RDWR|O_CREAT|O_TRUNC, 0666) = 11 18499: llseek(11, 0, SEEK_CUR) = 0 18499: llseek(11, 0, SEEK_SET) = 0 18499: fstat64(11, 0xFFBFE950) = 0 18499: fstat64(11, 0xFFBFE7F8) = 0 18499: ioctl(11, TCGETA, 0xFFBFE8DC) Err#25 ENOTTY 18499: write(11, " e m a i l . o r n l . g".., 29) = 29 18499: llseek(11, 0, SEEK_CUR) = 29 18499: fcntl(11, F_FREESP, 0xFFBFEA84) = 0 18499: time() = 1252675063 18499: open("/usr/local/cyrus/data/msg/motd", O_RDONLY) Err#2 ENOENT 18499: write(1, " * O K [ C A P A B I".., 126) = 126 18499: time() = 1252675063 18499: poll(0xFFBFB860, 1, 1800000) = 1 18499: time() = 1252675063 18499: time() = 1252675063 18499: poll(0xFFBFC858, 1, 1800000) = 1 18499: time() = 1252675063 18499: time() = 1252675063 18499: read(0, " 1 C A P A B I L I T Y".., 4096) = 14 18499: time() = 1252675063 18499: poll(0xFFBFC858, 1, 1800000) (sleeping...) From brong at fastmail.fm Fri Sep 11 18:53:09 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sat, 12 Sep 2009 08:53:09 +1000 Subject: v2.3.15 IMAP connection hangs In-Reply-To: <43C68785C2728049AF86B0ECB240A151117DB6E653@EXCHMB.ornl.gov> References: <43C68785C2728049AF86B0ECB240A151117DB6E4E7@EXCHMB.ornl.gov> <20090910221036.GA1177@brong.net> <43C68785C2728049AF86B0ECB240A151117DB6E653@EXCHMB.ornl.gov> Message-ID: <20090911225309.GA4536@brong.net> On Fri, Sep 11, 2009 at 09:22:47AM -0400, Rosenbaum, Larry M. wrote: > > From: Bron Gondwana [mailto:brong at fastmail.fm] > > On Thu, Sep 10, 2009 at 04:11:37PM -0400, Rosenbaum, Larry M. wrote: > > > I have just installed v2.3.15 on Solaris 9 Sparc with Cyrus-SASL > > 2.1.24rc1. When I connect to the IMAP port, I get a banner message but > > there is no response to any command I give it (including "0 logout"). > > I am able to give commands to the POP3 port and LMTP port just fine. > > Previous versions of Cyrus IMAP work OK too. Where should I look for > > the problem? > > > > Can you get a trace on the process (truss or similar) that shows what > > file > > it's sitting waiting on a lock for? It's probably a locking issue. > > Here is a truss showing an attempt at running cyradm: That's kind of pointless, sorry. I really need a trace of the imapd process, since that's what's locking up. The only interesting bit is: > 18499: read(0, " 1 C A P A B I L I T Y".., 4096) = 14 All we ever got was the "1 CAPABILITY" response (14 characters) without the actual contents of the capabilities. A trace of the connected imapd would be a lot more useful. You should be able to figure it out by following the port numbers (I would use netstat on Linux for that, or lsof - not sure what you do on Solaris) Hmm - I don't see any code that can generate "1 CAPABILITY" - it's all either "* CAPABILITY ..." or " OK [CAPABILITY ...]" Sorry - I need more debugging info (and probably your imapd.conf) to have a chance of understanding what's going on! Regards, Bron. From rosenbaumlm at ornl.gov Sat Sep 12 14:38:19 2009 From: rosenbaumlm at ornl.gov (Rosenbaum, Larry M.) Date: Sat, 12 Sep 2009 14:38:19 -0400 Subject: v2.3.15 IMAP connection hangs In-Reply-To: <20090911225309.GA4536@brong.net> References: <43C68785C2728049AF86B0ECB240A151117DB6E4E7@EXCHMB.ornl.gov> <20090910221036.GA1177@brong.net> <43C68785C2728049AF86B0ECB240A151117DB6E653@EXCHMB.ornl.gov> <20090911225309.GA4536@brong.net> Message-ID: <43C68785C2728049AF86B0ECB240A151117DB6E9F1@EXCHMB.ornl.gov> > From: Bron Gondwana [mailto:brong at fastmail.fm] > > On Fri, Sep 11, 2009 at 09:22:47AM -0400, Rosenbaum, Larry M. wrote: > > > From: Bron Gondwana [mailto:brong at fastmail.fm] > > > On Thu, Sep 10, 2009 at 04:11:37PM -0400, Rosenbaum, Larry M. > wrote: > > > > I have just installed v2.3.15 on Solaris 9 Sparc with Cyrus-SASL > > > 2.1.24rc1. When I connect to the IMAP port, I get a banner message > but > > > there is no response to any command I give it (including "0 > logout"). > > > I am able to give commands to the POP3 port and LMTP port just > fine. > > > Previous versions of Cyrus IMAP work OK too. Where should I look > for > > > the problem? > > > > > > Can you get a trace on the process (truss or similar) that shows > what > > > file > > > it's sitting waiting on a lock for? It's probably a locking issue. > > > > Here is a truss showing an attempt at running cyradm: > > That's kind of pointless, sorry. I really need a trace of the imapd > process, since that's what's locking up. I think I may have confused you by giving a poor description of what I was doing. This isn't a trace of cyradm. I put a truss trace on the imapd process. Then I ran cyradm to connect to the imapd process and watched the imapd trace to see what happened. It got to the following and stopped: 18499: read(0, " 1 C A P A B I L I T Y".., 4096) = 14 18499: time() = 1252675063 18499: poll(0xFFBFC858, 1, 1800000) (sleeping...) I then copied the output and sent it to you. > The only interesting bit is: > > > 18499: read(0, " 1 C A P A B I L I T Y".., 4096) = 14 > > All we ever got was the "1 CAPABILITY" response (14 characters) without > the > actual contents of the capabilities. > ... > Hmm - I don't see any code that can generate "1 CAPABILITY" - it's all This is cyradm sending a CAPABILITY command. It apparently uses "1" as the tag. > Sorry - I need more debugging info (and probably your imapd.conf) to > have a chance of understanding what's going on! I have attached the imapd.conf file. I hope that my explanation has given you what you need. If not, please tell me what else you need. Thanks, Larry -------------- next part -------------- A non-text attachment was scrubbed... Name: imapd.conf Type: application/octet-stream Size: 1077 bytes Desc: imapd.conf Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090912/006c7291/attachment.obj From PerSteinar.Iversen at hio.no Sun Sep 13 07:43:45 2009 From: PerSteinar.Iversen at hio.no (Per Steinar Iversen) Date: Sun, 13 Sep 2009 13:43:45 +0200 (CEST) Subject: Cyrus IMAPd 2.3.15 - cvt_cyrusdb changes owner of seen-file Message-ID: I just upgraded our imap-server to 2.3.15 using the cyrus-imapd-2.3.15-2 rpm made available by Simon Matter (thanks!). The OS is Redhat 5.4 on x86_64. All works well so far, except for one new strangeness: We have a script, run by root, that counts the number of unread mails in inboxes. To be able to parse this information easily the seen-files are converted from skiplist to the flat format first. This worked well for years, but now there is a change, sometimes the conversion triggers a checkpoint, the log is then something like this: Sep 13 13:06:18 imap cvt_cyrusdb[21330]: skiplist: checkpointed /var/lib/imap/user/d/dxxxx.seen (8 records, 716 bytes) in 0 seconds What is new now is that the seen-file is owned by root and not cyrus after the checkpoint. Is this a bug or just me doing something stupid? -psi From brong at fastmail.fm Sun Sep 13 07:54:01 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sun, 13 Sep 2009 21:54:01 +1000 Subject: Cyrus IMAPd 2.3.15 - cvt_cyrusdb changes owner of seen-file In-Reply-To: References: Message-ID: <20090913115401.GA2802@brong.net> On Sun, Sep 13, 2009 at 01:43:45PM +0200, Per Steinar Iversen wrote: > All works well so far, except for one new strangeness: We have a script, > run by root, that counts the number of unread mails in inboxes. To be able > to parse this information easily the seen-files are converted from > skiplist to the flat format first. This worked well for years, but now > there is a change, sometimes the conversion triggers a checkpoint, the log > is then something like this: You read the flat files with a tool? > Sep 13 13:06:18 imap cvt_cyrusdb[21330]: skiplist: checkpointed /var/lib/imap/user/d/dxxxx.seen (8 records, 716 bytes) in 0 seconds > > What is new now is that the seen-file is owned by root and not cyrus after > the checkpoint. > > Is this a bug or just me doing something stupid? The change that's triggering it is is that recovery can cause a checkpoint now. The cvt_cyrusdb should be running as Cyrus (you could argue it should drop privs automatically) Alternatively, there's cyr_dbtool. You can use that to dump the .seen db and pipe the output to somewhere. Run that as cyrus user as well. There's always a risk that the file will choose to run "recovery", and hence trigger the checkpoint. Bron. From cyruspy at gmail.com Sun Sep 13 14:42:07 2009 From: cyruspy at gmail.com (Ciro Iriarte) Date: Sun, 13 Sep 2009 14:42:07 -0400 Subject: Active/Active cyrus on OCFS2 Message-ID: Hi, I've been reading the archives about clustering cyrus and I have a question. Now that OCFS2 v1.4 supports Shared Writable MMAP, would it be possible to run a two-node cluster with a cyrus instance in each node sharing the same mail spool on top of OCFS2?, would that have a good performance and be reliable?. We're toying with the idea to build a two-node cluster on SLES10SP2 and LinuxHA, running Postfix+Cyrus+OpenLDAP to provide service for about 30k mailboxes/accounts. Anyone has done something similar with OCFS2? Regards, -- Ciro Iriarte http://cyruspy.wordpress.com -- From jmasterson at modwest.com Sun Sep 13 17:22:18 2009 From: jmasterson at modwest.com (John Masterson) Date: Sun, 13 Sep 2009 15:22:18 -0600 Subject: debian etch upgrade, now 64bit machines cannot connect to backends Message-ID: <4AAD628A.3060400@modwest.com> We just upgraded to 2.2.13-10+etch2 and now our 64bit front end machines cannot connect to our back end servers, e.g., Sep 13 11:55:49 mfe5 cyrus/mc2pop3[6764]: connect(mbe2.msomt.modwest.com) failed: Invalid argument Sep 13 11:55:49 mfe5 cyrus/mc2pop3[6764]: couldn't authenticate to backend server also: Sep 13 15:15:39 mfe5 cyrus/mupdate[1327]: Thread timed out waiting for listener_lock A 32-bit front end machine is working fine. We don't think there are any other differences between these installations. cyrus.conf, mupdate.conf, and imap.conf are identical except for the obvious, listen addresses, hostnames. Any ideas or experience with this? -- John Masterson http://modwest.com From michael.menge at zdv.uni-tuebingen.de Mon Sep 14 05:18:16 2009 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Mon, 14 Sep 2009 11:18:16 +0200 Subject: syncserver segfaults Message-ID: <20090914111816.163928dn79adp1so@webmail.uni-tuebingen.de> Hi, I need some help debuging a problem with the syncserver. After upgrading to from cyrus 2.3.14 to 2.3.15 the syncserver segfaults. This happens on i586 and x86_64 systems after a syncclient tries to connect. Telnet to the syncserver shows /home/mailserv08# telnet mailserv01 2005 Trying 192.168.3.17... Connected to mailserv01. Escape character is '^]'. * SASL PLAIN LOGIN * STARTTLS * OK mailserv01 Cyrus sync server v2.3.15 Connection closed by foreign host. Attacht are the strace from the i586 and x86_64 server. What other informations are needed to debug this. Regards Michael Menge -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen -------------- next part -------------- accept(4, 0, NULL) = 9 fcntl64(8, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0 alarm(0) = 0 write(3, "\2\0\0\0\272#\0\0", 8) = 8 time(NULL) = 1252918704 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=842, ...}) = 0 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=842, ...}) = 0 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=842, ...}) = 0 send(5, "<183>Sep 14 10:58:24 syncserver["..., 58, MSG_NOSIGNAL) = 58 dup2(9, 0) = 0 dup2(9, 1) = 1 dup2(9, 2) = 2 close(9) = 0 write(3, "\3\0\0\0\272#\0\0", 8) = 8 getpeername(0, {sa_family=AF_INET, sin_port=htons(53531), sin_addr=inet_addr("192.168.3.31")}, [16]) = 0 socket(PF_FILE, SOCK_STREAM, 0) = 9 fcntl64(9, F_GETFL) = 0x2 (flags O_RDWR) fcntl64(9, F_SETFL, O_RDWR|O_NONBLOCK) = 0 connect(9, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = 0 poll([{fd=9, events=POLLOUT|POLLERR|POLLHUP, revents=POLLOUT}], 1, 5000) = 1 send(9, "\2\0\0\0\r\0\0\0\6\0\0\0hosts\0\0\0", 20, MSG_NOSIGNAL) = 20 poll([{fd=9, events=POLLIN|POLLERR|POLLHUP, revents=POLLIN|POLLERR|POLLHUP}], 1, 5000) = 1 recvmsg(9, {msg_name(0)=NULL, msg_iov(1)=[{"hosts\0", 6}], msg_controllen=16, {cmsg_len=16, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, {10}}, msg_flags=0}, 0) = 6 fstat64(10, {st_mode=S_IFREG|0600, st_size=217016, ...}) = 0 pread64(10, "\1\0\0\0h\0\0\0\260k\10\0\1\0\0\0\177P\365I\0\0\0\0\323"..., 104, 0) = 104 mmap2(NULL, 217016, PROT_READ, MAP_SHARED, 10, 0) = 0xb6181000 close(10) = 0 close(9) = 0 getsockname(0, {sa_family=AF_INET, sin_port=htons(2005), sin_addr=inet_addr("192.168.3.19")}, [16]) = 0 socket(PF_NETLINK, SOCK_RAW, 0) = 9 bind(9, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 getsockname(9, {sa_family=AF_NETLINK, pid=9146, groups=00000000}, [12]) = 0 time(NULL) = 1252918704 sendto(9, "\24\0\0\0\26\0\1\3\260\5\256J\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20 recvmsg(9, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"D\0\0\0\24\0\2\0\260\5\256J\272#\0\0\2\10\200\376\1\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 324 recvmsg(9, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"@\0\0\0\24\0\2\0\260\5\256J\272#\0\0\n\200\200\376\1\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 320 recvmsg(9, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0\260\5\256J\272#\0\0\0\0\0\0\1\0\0\0\24"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 20 close(9) = 0 socket(PF_NETLINK, SOCK_RAW, 0) = 9 bind(9, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 getsockname(9, {sa_family=AF_NETLINK, pid=9146, groups=00000000}, [12]) = 0 time(NULL) = 1252918704 sendto(9, "\24\0\0\0\26\0\1\3\260\5\256J\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20 recvmsg(9, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"D\0\0\0\24\0\2\0\260\5\256J\272#\0\0\2\10\200\376\1\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 324 recvmsg(9, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"@\0\0\0\24\0\2\0\260\5\256J\272#\0\0\n\200\200\376\1\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 320 recvmsg(9, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0\260\5\256J\272#\0\0\0\0\0\0\1\0\0\0\24"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 20 close(9) = 0 open("/etc/nsswitch.conf", O_RDONLY) = 9 fstat64(9, {st_mode=S_IFREG|0644, st_size=1192, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6180000 read(9, "#\n# /etc/nsswitch.conf\n#\n# An ex"..., 4096) = 1192 read(9, "", 4096) = 0 close(9) = 0 munmap(0xb6180000, 4096) = 0 open("/etc/ld.so.cache", O_RDONLY) = 9 fstat64(9, {st_mode=S_IFREG|0644, st_size=59359, ...}) = 0 mmap2(NULL, 59359, PROT_READ, MAP_PRIVATE, 9, 0) = 0xb6172000 close(9) = 0 open("/lib/libnss_files.so.2", O_RDONLY) = 9 read(9, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\360\33"..., 512) = 512 fstat64(9, {st_mode=S_IFREG|0755, st_size=42485, ...}) = 0 mmap2(NULL, 37528, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 9, 0) = 0xb6168000 madvise(0xb6168000, 37528, MADV_SEQUENTIAL|0x1) = 0 mmap2(0xb6170000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 9, 0x7) = 0xb6170000 close(9) = 0 munmap(0xb6172000, 59359) = 0 open("/etc/protocols", O_RDONLY) = 9 fcntl64(9, F_GETFD) = 0 fcntl64(9, F_SETFD, FD_CLOEXEC) = 0 fstat64(9, {st_mode=S_IFREG|0644, st_size=6561, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6180000 read(9, "#\n# protocols\tThis file describe"..., 4096) = 4096 close(9) = 0 munmap(0xb6180000, 4096) = 0 setsockopt(1, SOL_TCP, TCP_NODELAY, [1], 4) = 0 open("/home/mailserv09_replic/proc/9146", O_RDWR|O_CREAT|O_TRUNC, 0666) = 9 fstat64(9, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6180000 _llseek(9, 0, [0], SEEK_SET) = 0 write(9, "mailserv09.uni-tuebingen.de [192"..., 43) = 43 _llseek(9, 0, [43], SEEK_CUR) = 0 ftruncate(9, 43) = 0 write(1, "* SASL PLAIN LOGIN\r\n* STARTTLS\r\n"..., 75) = 75 time(NULL) = 1252918704 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=842, ...}) = 0 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=842, ...}) = 0 stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=842, ...}) = 0 send(5, "<183>Sep 14 10:58:24 syncserver["..., 57, MSG_NOSIGNAL) = 57 mmap2(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6127000 --- SIGSEGV (Segmentation fault) @ 0 (0) --- -------------- next part -------------- accept(4, 0, NULL) = 9 fcntl(8, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0 alarm(0) = 0 write(3, "\2\0\0\0pm\0\0", 8) = 8 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=842, ...}) = 0 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=842, ...}) = 0 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=842, ...}) = 0 sendto(5, "<183>Sep 14 11:00:01 syncserver["..., 59, MSG_NOSIGNAL, NULL, 0) = 59 dup2(9, 0) = 0 dup2(9, 1) = 1 dup2(9, 2) = 2 close(9) = 0 write(3, "\3\0\0\0pm\0\0", 8) = 8 getpeername(0, {sa_family=AF_INET, sin_port=htons(38226), sin_addr=inet_addr("192.168.3.30")}, [12884901904]) = 0 socket(PF_FILE, SOCK_STREAM, 0) = 9 fcntl(9, F_GETFL) = 0x2 (flags O_RDWR) fcntl(9, F_SETFL, O_RDWR|O_NONBLOCK) = 0 connect(9, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = 0 poll([{fd=9, events=POLLOUT|POLLERR|POLLHUP, revents=POLLOUT}], 1, 5000) = 1 sendto(9, "\2\0\0\0\r\0\0\0\6\0\0\0hosts\0\252\264", 20, MSG_NOSIGNAL, NULL, 0) = 20 poll([{fd=9, events=POLLIN|POLLERR|POLLHUP, revents=POLLIN|POLLERR|POLLHUP}], 1, 5000) = 1 recvmsg(9, {msg_name(0)=NULL, msg_iov(1)=[{"hosts\0", 6}], msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, {10}}, msg_flags=0}, 0) = 6 fstat(10, {st_mode=S_IFREG|0600, st_size=217016, ...}) = 0 pread(10, "\1\0\0\0h\0\0\0H\25\t\0\1\0\0\0\35P\365I\0\0\0\0\323\0"..., 104, 0) = 104 mmap(NULL, 217016, PROT_READ, MAP_SHARED, 10, 0) = 0x2b1bb761e000 close(10) = 0 close(9) = 0 getsockname(0, {sa_family=AF_INET, sin_port=htons(2005), sin_addr=inet_addr("192.168.3.17")}, [12884901904]) = 0 socket(PF_NETLINK, SOCK_RAW, 0) = 9 bind(9, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 getsockname(9, {sa_family=AF_NETLINK, pid=28016, groups=00000000}, [17780093728359186444]) = 0 sendto(9, "\24\0\0\0\26\0\1\3\21\6\256J\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20 recvmsg(9, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"D\0\0\0\24\0\2\0\21\6\256Jpm\0\0\2\10\200\376\1\0\0\0\10"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 324 recvmsg(9, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"@\0\0\0\24\0\2\0\21\6\256Jpm\0\0\n\200\200\376\1\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 320 recvmsg(9, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0\21\6\256Jpm\0\0\0\0\0\0\1\0\0\0\24\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 20 close(9) = 0 socket(PF_NETLINK, SOCK_RAW, 0) = 9 bind(9, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 getsockname(9, {sa_family=AF_NETLINK, pid=28016, groups=00000000}, [17780093453481279500]) = 0 sendto(9, "\24\0\0\0\26\0\1\3\21\6\256J\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20 recvmsg(9, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"D\0\0\0\24\0\2\0\21\6\256Jpm\0\0\2\10\200\376\1\0\0\0\10"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 324 recvmsg(9, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"@\0\0\0\24\0\2\0\21\6\256Jpm\0\0\n\200\200\376\1\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 320 recvmsg(9, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0\21\6\256Jpm\0\0\0\0\0\0\1\0\0\0\24\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 20 close(9) = 0 open("/etc/nsswitch.conf", O_RDONLY) = 9 fstat(9, {st_mode=S_IFREG|0644, st_size=1192, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b1bb7653000 read(9, "#\n# /etc/nsswitch.conf\n#\n# An ex"..., 4096) = 1192 read(9, "", 4096) = 0 close(9) = 0 munmap(0x2b1bb7653000, 4096) = 0 open("/etc/ld.so.cache", O_RDONLY) = 9 fstat(9, {st_mode=S_IFREG|0644, st_size=124944, ...}) = 0 mmap(NULL, 124944, PROT_READ, MAP_PRIVATE, 9, 0) = 0x2b1bb7653000 close(9) = 0 open("/lib64/libnss_files.so.2", O_RDONLY) = 9 read(9, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260#\0\0"..., 832) = 832 fstat(9, {st_mode=S_IFREG|0755, st_size=53489, ...}) = 0 mmap(NULL, 1090888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 9, 0) = 0x2b1bb7672000 madvise(0x2b1bb7672000, 1090888, MADV_SEQUENTIAL|0x1) = 0 mprotect(0x2b1bb767c000, 1044480, PROT_NONE) = 0 mmap(0x2b1bb777b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 9, 0x9000) = 0x2b1bb777b000 close(9) = 0 munmap(0x2b1bb7653000, 124944) = 0 open("/etc/protocols", O_RDONLY) = 9 fcntl(9, F_GETFD) = 0 fcntl(9, F_SETFD, FD_CLOEXEC) = 0 fstat(9, {st_mode=S_IFREG|0644, st_size=6561, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b1bb7653000 read(9, "#\n# protocols\tThis file describe"..., 4096) = 4096 close(9) = 0 munmap(0x2b1bb7653000, 4096) = 0 setsockopt(1, SOL_TCP, TCP_NODELAY, [1], 4) = 0 open("/home/mailserv08_replic/proc/28016", O_RDWR|O_CREAT|O_TRUNC, 0666) = 9 fstat(9, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b1bb7653000 lseek(9, 0, SEEK_SET) = 0 write(9, "mailserv08.uni-tuebingen.de [192"..., 43) = 43 lseek(9, 0, SEEK_CUR) = 43 ftruncate(9, 43) = 0 write(1, "* SASL PLAIN LOGIN\r\n* STARTTLS\r\n"..., 75) = 75 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=842, ...}) = 0 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=842, ...}) = 0 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=842, ...}) = 0 sendto(5, "<183>Sep 14 11:00:01 syncserver["..., 58, MSG_NOSIGNAL, NULL, 0) = 58 mmap(NULL, 528384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b1bb777d000 --- SIGSEGV (Segmentation fault) @ 0 (0) --- -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5339 bytes Desc: S/MIME krytographische Unterschrift Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090914/71c43aeb/attachment-0001.bin From PerSteinar.Iversen at hio.no Mon Sep 14 05:19:06 2009 From: PerSteinar.Iversen at hio.no (Per Steinar Iversen) Date: Mon, 14 Sep 2009 11:19:06 +0200 (CEST) Subject: Cyrus IMAPd 2.3.15 - cvt_cyrusdb changes owner of seen-file In-Reply-To: <20090913115401.GA2802@brong.net> References: <20090913115401.GA2802@brong.net> Message-ID: On Sun, 13 Sep 2009, Bron Gondwana wrote: > On Sun, Sep 13, 2009 at 01:43:45PM +0200, Per Steinar Iversen wrote: >> All works well so far, except for one new strangeness: We have a script, >> run by root, that counts the number of unread mails in inboxes. To be able >> to parse this information easily the seen-files are converted from >> skiplist to the flat format first. This worked well for years, but now >> there is a change, sometimes the conversion triggers a checkpoint, the log >> is then something like this: > > You read the flat files with a tool? Yes. I added "su cyrus" to this step in the script, that does seem to work well. > >> Sep 13 13:06:18 imap cvt_cyrusdb[21330]: skiplist: checkpointed /var/lib/imap/user/d/dxxxx.seen (8 records, 716 bytes) in 0 seconds >> >> What is new now is that the seen-file is owned by root and not cyrus after >> the checkpoint. >> >> Is this a bug or just me doing something stupid? > > The change that's triggering it is is that recovery can cause a checkpoint > now. The cvt_cyrusdb should be running as Cyrus (you could argue it should > drop privs automatically) > > Alternatively, there's cyr_dbtool. You can use that to dump the .seen > db and pipe the output to somewhere. Run that as cyrus user as well. > There's always a risk that the file will choose to run "recovery", and > hence trigger the checkpoint. > > Bron. -psi From michael.menge at zdv.uni-tuebingen.de Mon Sep 14 10:31:47 2009 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Mon, 14 Sep 2009 16:31:47 +0200 Subject: syncserver segfaults In-Reply-To: <4AAE1279.9010804@yandex-team.ru> References: <4AAE1279.9010804@yandex-team.ru> Message-ID: <20090914163147.26015kcdohq8fnbn@webmail.uni-tuebingen.de> Quoting Oleg Gorokhov : > Michael, > > it could be useful some observations and details in my situation > after update cyrus from 2.3.14 -> 2.3.15 murder+sync_server > combination (actually with our patches applied). Our problem with > segfaults syncserver that in imapd.conf the default-partition should > be explictly set. Default configuration has it just commented and in > 2.3.15 brokes sync_server (might be something also). It is amasing > but after stated > > default-partition: default > "defaultpartition: default" Thanks that was the problem -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5339 bytes Desc: S/MIME krytographische Unterschrift Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090914/83e3d3f0/attachment.bin From rosenbaumlm at ornl.gov Tue Sep 15 14:35:48 2009 From: rosenbaumlm at ornl.gov (Rosenbaum, Larry M.) Date: Tue, 15 Sep 2009 14:35:48 -0400 Subject: v2.3.15 IMAP connection hangs In-Reply-To: <20090911225309.GA4536@brong.net> References: <43C68785C2728049AF86B0ECB240A151117DB6E4E7@EXCHMB.ornl.gov> <20090910221036.GA1177@brong.net> <43C68785C2728049AF86B0ECB240A151117DB6E653@EXCHMB.ornl.gov> <20090911225309.GA4536@brong.net> Message-ID: <43C68785C2728049AF86B0ECB240A151117DC7C86E@EXCHMB.ornl.gov> > From: Bron Gondwana [mailto:brong at fastmail.fm] > >... > That's kind of pointless, sorry. I really need a trace of the imapd > process, since that's what's locking up. Here's another trace of an imapd process: ornl71# truss -f -p 23237 23237: accept(4, 0x00000000, 0x00000000, 1) (sleeping...) (at this point I used Telnet to connect to port 143) 23237: accept(4, 0x00000000, 0x00000000, 1) = 10 23237: fcntl(9, F_SETLKW, 0xFFBFF290) = 0 23237: alarm(0) = 0 23237: getpid() = 23237 [17305] 23237: write(3, "\0\0\002\0\0 ZC5", 8) = 8 23237: fstat(5, 0xFFBFEFC8) = 0 23237: time() = 1253039206 23237: getpid() = 23237 [17305] 23237: putmsg(5, 0xFFBFE680, 0xFFBFE674, 0) = 0 23237: open("/var/run/syslog_door", O_RDONLY) = 11 23237: door_info(11, 0xFFBFE5B8) = 0 23237: getpid() = 23237 [17305] 23237: door_call(11, 0xFFBFE5A0) = 0 23237: close(11) = 0 23237: fcntl(10, F_DUP2FD, 0x00000000) = 0 23237: fcntl(10, F_DUP2FD, 0x00000001) = 1 23237: fcntl(10, F_DUP2FD, 0x00000002) = 2 23237: close(10) = 0 23237: getpid() = 23237 [17305] 23237: write(3, "\0\0\003\0\0 ZC5", 8) = 8 23237: getpeername(0, 0xFFBFF010, 0xFFBFEB84, 1) = 0 23237: open("/etc/netconfig", O_RDONLY|O_LARGEFILE) = 10 23237: fcntl(10, F_DUPFD, 0x00000100) Err#22 EINVAL 23237: read(10, " # p r a g m a i d e n".., 1024) = 1024 23237: read(10, " t s t p i _ c".., 1024) = 215 23237: read(10, 0x00136728, 1024) = 0 23237: lseek(10, 0, SEEK_SET) = 0 23237: read(10, " # p r a g m a i d e n".., 1024) = 1024 23237: read(10, " t s t p i _ c".., 1024) = 215 23237: read(10, 0x00136728, 1024) = 0 23237: close(10) = 0 23237: open("/dev/udp", O_RDONLY) = 10 23237: ioctl(10, 0xC00C6982, 0xFFBFE8FC) = 0 23237: close(10) = 0 23237: open64("/var/run/name_service_door", O_RDONLY) = 10 23237: fcntl(10, F_SETFD, 0x00000001) = 0 23237: door_info(10, 0xFEEC26C0) = 0 23237: door_call(10, 0xFFBFC868) = 0 23237: door_info(10, 0xFFBFC800) = 0 23237: door_call(10, 0xFFBFC7E8) = 0 23237: getsockname(0, 0xFFBFF110, 0xFFBFEB84, 1) = 0 23237: brk(0x0013D190) = 0 23237: brk(0x0013F190) = 0 23237: getpid() = 23237 [17305] 23237: open("/usr/local/cyrus/data/proc/23237", O_RDWR|O_CREAT|O_TRUNC, 0666) = 11 23237: llseek(11, 0, SEEK_CUR) = 0 23237: llseek(11, 0, SEEK_SET) = 0 23237: fstat64(11, 0xFFBFE950) = 0 23237: fstat64(11, 0xFFBFE7F8) = 0 23237: ioctl(11, TCGETA, 0xFFBFE8DC) Err#25 ENOTTY 23237: write(11, " e m a i l . o r n l . g".., 29) = 29 23237: llseek(11, 0, SEEK_CUR) = 29 23237: fcntl(11, F_FREESP, 0xFFBFEA84) = 0 23237: time() = 1253039206 23237: open("/usr/local/cyrus/data/msg/motd", O_RDONLY) Err#2 ENOENT 23237: write(1, " * O K [ C A P A B I".., 126) = 126 23237: time() = 1253039206 23237: poll(0xFFBFB860, 1, 1800000) (sleeping...) (at this time I typed "0 logout" at the Telnet session) 23237: poll(0xFFBFB860, 1, 1800000) = 1 23237: time() = 1253039218 23237: time() = 1253039218 23237: poll(0xFFBFC858, 1, 1788000) = 1 23237: time() = 1253039218 23237: time() = 1253039218 23237: read(0, " 0 l o g o u t\r\n", 4096) = 10 23237: time() = 1253039218 23237: poll(0xFFBFC858, 1, 1800000) (sleeping...) (At this point the trace paused, and the Telnet session got no response. I aborted the Telnet session and the trace continued as follows:) 23237: poll(0xFFBFC858, 1, 1800000) = 1 23237: time() = 1253039249 23237: time() = 1253039249 23237: read(0, 0x00135688, 4096) = 0 23237: llseek(11, 0, SEEK_CUR) = 29 23237: close(11) = 0 23237: unlink("/usr/local/cyrus/data/proc/23237") = 0 23237: open("/dev/null", O_RDWR) = 11 23237: shutdown(0, 0, 1) = 0 23237: fcntl(11, F_DUP2FD, 0x00000000) = 0 23237: shutdown(1, 0, 1) = 0 23237: fcntl(11, F_DUP2FD, 0x00000001) = 1 23237: shutdown(2, 0, 1) = 0 23237: fcntl(11, F_DUP2FD, 0x00000002) = 2 23237: close(11) = 0 23237: getpid() = 23237 [17305] 23237: write(3, "\0\0\001\0\0 ZC5", 8) = 8 23237: sigaction(SIGALRM, 0xFFBFF170, 0x00000000) = 0 23237: sigaction(SIGHUP, 0xFFBFF170, 0x00000000) = 0 23237: sigaction(SIGINT, 0xFFBFF170, 0x00000000) = 0 23237: sigaction(SIGQUIT, 0xFFBFF170, 0x00000000) = 0 23237: alarm(114) = 0 23237: fcntl(9, F_SETLKW, 0xFFBFF3B8) (sleeping...) So what was it waiting for? Thanks, Larry -------------- next part -------------- A non-text attachment was scrubbed... Name: imapd.conf Type: application/octet-stream Size: 1077 bytes Desc: imapd.conf Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090915/759e1ecb/attachment.obj From qralston+ml.info-cyrus at andrew.cmu.edu Tue Sep 15 23:13:46 2009 From: qralston+ml.info-cyrus at andrew.cmu.edu (James Ralston) Date: Tue, 15 Sep 2009 23:13:46 -0400 Subject: autocreate shared folders during delivery? Message-ID: <7B9157258942D7F2A5DE9CEF@shieldbreaker.sei.cmu.edu> We have a high-volume mailing list that we want to have delivered to a series of shared folders based on the date that the message is received: sharedfolder.example-list.2009.01 sharedfolder.example-list.2009.02 sharedfolder.example-list.2009.03 ... sharedfolder.example-list.2009.04 We may need to get it down to the granularity of a single day; e.g.: sharedfolder.example-list.2009.09.01 sharedfolder.example-list.2009.09.02 sharedfolder.example-list.2009.09.03 ... sharedfolder.example-list.2009.09.15 Now, on the mail hub, we have a process that expands the "example-list" alias appropriately based on the current date/time. E.g., on 2009-09-15, this: example-list at example.org becomes this: +sharedfolder.example-list.2009.09 at imap.example.org ...when using a "monthly" expansion rule. The problem we've run into is that there doesn't appear to be any way to get Cyrus to autocreate the shared folders on the fly during delivery. (The autocreate patches from the University of Athens handle autocreation only for regular user accounts.) Having someone manually create the folders in advance would have overhead, as well as be subject to human error. Trying to script it would also be problematic, because storing authentication credentials would be required. Has anyone encountered a need to do something like this before? If so, what did you do? From pcravero at as2594.net Wed Sep 16 04:23:56 2009 From: pcravero at as2594.net (Paolo Cravero) Date: Wed, 16 Sep 2009 10:23:56 +0200 Subject: fyi, problem with compile_sieve on RHEL 5.4 x86_64 (2.3.7-7) Message-ID: <4AB0A09C.2090705@as2594.net> For the records, in case someone gets stuck in the same problem. I installed a test environment x86_64 with RedHat Enterprise Linux 5.4 using their official RPMs cyrus-imapd-2.3.7-7.el5.x86_64.rpm and related. I couldn't get complie_sieve to work, it coredumped right after opening the output file for writing. Solution: compiled invoca's 2.3.14 SRPM. Since I was still setting up the system I don't know if other /usr/lib/cyrus-imapd programs fail. At the time being there is no more recent RPM on RH official repository. One more note for x86_64 and PAM authentication with LDAP backend. In /etc/pam.d/imap (smtp, pop, sieve, lmtp) files I had to use explicitly 64bit libraries: auth sufficient /lib64/security/pam_ldap.so account sufficient /lib64/security/pam_ldap.so while /lib/security/pam_ldap.so did not log any error but did not authenticate at all, not even opened the TCP connection to LDAP server. I'm not that much into 64bit yet, so others might find this info useful too. Paolo From m.roshany at gmail.com Wed Sep 16 05:27:15 2009 From: m.roshany at gmail.com (mozafar roshany) Date: Wed, 16 Sep 2009 13:57:15 +0430 Subject: Cyrus LMTP and Maildir[++] Format Message-ID: <849aa58a0909160227nad0bf9epae12894e86c8667@mail.gmail.com> Hi everyone. I want to use the Cyrus LMTP server to get mail from Postfix remotely. But, the problem is that I can't use Cyrus-IMAP/POP itself for some reasons and I have courier as my IMAP/POP server which uses Maildir++ mailbox format. Is there any way (any configuration parameter, or any patch or ...) to make Cyrus LMTP understand Maildir++ format? If not, why? I haven't find a strict and clear answer to this question at my searches. Thank you so much. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090916/7197e35c/attachment.html From michael.menge at zdv.uni-tuebingen.de Wed Sep 16 05:57:45 2009 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Wed, 16 Sep 2009 11:57:45 +0200 Subject: Cyrus LMTP and Maildir[++] Format In-Reply-To: <849aa58a0909160227nad0bf9epae12894e86c8667@mail.gmail.com> References: <849aa58a0909160227nad0bf9epae12894e86c8667@mail.gmail.com> Message-ID: <20090916115745.44905bw7qg8miftl@webmail.uni-tuebingen.de> Quoting mozafar roshany : > Hi everyone. > > I want to use the Cyrus LMTP server to get mail from Postfix remotely. But, > the problem is that I can't use Cyrus-IMAP/POP itself for some reasons and I > have courier as my IMAP/POP server which uses Maildir++ mailbox format. Is > there any way (any configuration parameter, or any patch or ...) to make > Cyrus LMTP understand Maildir++ format? If not, why? I haven't find a strict > and clear answer to this question at my searches. > The LMTP Protocol is integrated in Cyrus. It will store the mails in cyrus own format. Cyrus does not know how to handle maildir or mbox as there is no need for this. You may be able to write an patch for this, but i doubt that it will be added to cyrus. If you want to use Cyrus, we may be able to help you solve the problems/reasons. If you want to use Courier, you should ask on the courier mailinglist how to deliver mails from a remote postfix. -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5339 bytes Desc: S/MIME krytographische Unterschrift Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090916/5eeb5663/attachment-0001.bin From awilliam at whitemice.org Wed Sep 16 06:05:17 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Wed, 16 Sep 2009 06:05:17 -0400 Subject: Cyrus LMTP and Maildir[++] Format In-Reply-To: <849aa58a0909160227nad0bf9epae12894e86c8667@mail.gmail.com> References: <849aa58a0909160227nad0bf9epae12894e86c8667@mail.gmail.com> Message-ID: <1253095517.5043.9.camel@linux-m3mt> On Wed, 2009-09-16 at 13:57 +0430, mozafar roshany wrote: > Hi everyone. > I want to use the Cyrus LMTP server to get mail from Postfix remotely. > But, the problem is that I can't use Cyrus-IMAP/POP itself for some > reasons and I have courier as my IMAP/POP server which uses Maildir++ > mailbox format. That won't work (and doesn't make much sense). Switch to Cyrus, it is a superior IMAP/POP server. > Is there any way (any configuration parameter, or any patch or ...) to > make Cyrus LMTP understand Maildir++ format? No. > If not, why? Why do so? The purpose of Cyrus' lmtpd is to delivery mail into the Cyrus mail store. > I haven't find a strict and clear answer to this question at my > searches. IMO, because maildir is a bad idea. There is no good reason for someone to be mucking around in the filesystem underneath my mail server. You can access the mail via IMAP [which is why you have an IMAP server!] - via which your actions can be audited, sanitized, and made safe for concurrent actions [ none of which is possible if some rogue user is dorking the filesystem ]. From brong at fastmail.fm Wed Sep 16 11:12:13 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 17 Sep 2009 01:12:13 +1000 Subject: v2.3.15 IMAP connection hangs In-Reply-To: <43C68785C2728049AF86B0ECB240A151117DC7C86E@EXCHMB.ornl.gov> References: <43C68785C2728049AF86B0ECB240A151117DB6E4E7@EXCHMB.ornl.gov> <20090910221036.GA1177@brong.net> <43C68785C2728049AF86B0ECB240A151117DB6E653@EXCHMB.ornl.gov> <20090911225309.GA4536@brong.net> <43C68785C2728049AF86B0ECB240A151117DC7C86E@EXCHMB.ornl.gov> Message-ID: <20090916151213.GA25255@brong.net> On Tue, Sep 15, 2009 at 02:35:48PM -0400, Rosenbaum, Larry M. wrote: > > From: Bron Gondwana [mailto:brong at fastmail.fm] > > > >... > > That's kind of pointless, sorry. I really need a trace of the imapd > > process, since that's what's locking up. > > Here's another trace of an imapd process: Sorry I haven't replied before - I'm on holiday over the other side of Australia for a week - so we were flying most of yesterday to get here! > 23237: poll(0xFFBFB860, 1, 1800000) = 1 > 23237: time() = 1253039218 > 23237: time() = 1253039218 > 23237: poll(0xFFBFC858, 1, 1788000) = 1 > 23237: time() = 1253039218 > 23237: time() = 1253039218 > 23237: read(0, " 0 l o g o u t\r\n", 4096) = 10 > 23237: time() = 1253039218 > 23237: poll(0xFFBFC858, 1, 1800000) (sleeping...) > > (At this point the trace paused, and the Telnet session got no response. > I aborted the Telnet session and the trace continued as follows:) This is the same path at the same point as last time. Clearly something going on! The pain is, I can't see it - unless it's happening deep inside lib/prot.c somewhere. And the strace doesn't tell us anything - it will read up to 4096 characters straight off, and it's getting 10 of them, which should be enough! I'm wondering if something in the COMPRESS changes has broken an assuption. I don't suppose you're able to gdb the process and get a backtrace from that poll are you? Sorry to keep asking for more stuff, I just can't figure out any more than that from the truss output! > 23237: poll(0xFFBFC858, 1, 1800000) = 1 Yeah, so disconnect caused the poll to finish! And a signal handler had fired, so it knows to close down... > 23237: close(11) = 0 > 23237: unlink("/usr/local/cyrus/data/proc/23237") = 0 > 23237: open("/dev/null", O_RDWR) = 11 > 23237: shutdown(0, 0, 1) = 0 > > So what was it waiting for? That's a really good question! It feels to me that it had gone back into the loop to get some more data. Hmm - question: do you have zlib support compiled in? [brong at utility1 ~]$ ldd /usr/cyrus/bin/imapd | grep libz libz.so.1 => /usr/lib/libz.so.1 (0x55990000) Maybe there's a bug in the codepaths for COMPRESS when zlib doesn't exist (because they're all #ifdef HAVE_ZLIB - maybe things don't work nicely without it!) Long shot - but you might want to try it with the attached patch applied. It rewrites the zlib handling logic quite a lot, though it wasn't originally designed to deal with your problem! I'm going to push it to CVS soon. Bron. -------------- next part -------------- A non-text attachment was scrubbed... Name: rewrite_zlib.diff Type: text/x-diff Size: 6777 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090917/a006e57c/attachment.bin From rosenbaumlm at ornl.gov Wed Sep 16 11:23:09 2009 From: rosenbaumlm at ornl.gov (Rosenbaum, Larry M.) Date: Wed, 16 Sep 2009 11:23:09 -0400 Subject: v2.3.15 IMAP connection hangs In-Reply-To: <20090916151213.GA25255@brong.net> References: <43C68785C2728049AF86B0ECB240A151117DB6E4E7@EXCHMB.ornl.gov> <20090910221036.GA1177@brong.net> <43C68785C2728049AF86B0ECB240A151117DB6E653@EXCHMB.ornl.gov> <20090911225309.GA4536@brong.net> <43C68785C2728049AF86B0ECB240A151117DC7C86E@EXCHMB.ornl.gov> <20090916151213.GA25255@brong.net> Message-ID: <43C68785C2728049AF86B0ECB240A151117DC7CCB9@EXCHMB.ornl.gov> > From: Bron Gondwana [mailto:brong at fastmail.fm] >... > Hmm - question: do you have zlib support compiled in? > > [brong at utility1 ~]$ ldd /usr/cyrus/bin/imapd | grep libz > libz.so.1 => /usr/lib/libz.so.1 (0x55990000) > > Maybe there's a bug in the codepaths for COMPRESS when zlib doesn't > exist (because they're all #ifdef HAVE_ZLIB - maybe things don't work > nicely without it!) > > Long shot - but you might want to try it with the attached patch > applied. > It rewrites the zlib handling logic quite a lot, though it wasn't > originally > designed to deal with your problem! I'm going to push it to CVS soon. I already have the patch installed because we have an old version of Zlib. Could the patch be causing the problem? Larry From jvoorhees1 at gmail.com Wed Sep 16 11:51:02 2009 From: jvoorhees1 at gmail.com (Jose Perez) Date: Wed, 16 Sep 2009 10:51:02 -0500 Subject: Recovering of mailboxes Message-ID: HI all: I'm running Cyrus IMAP 2.3.7 in a CentOS 5.3 box. We have about 400 users, most of them POP3 users but some are IMAP users with their mail stored at the mailserver. I recently had a out of space problem, so I removed so many unnecesary files (logs, backups, among others) and then restarted mail services (MailScanner, MySQL, ClamAV, Cyrus IMAP, etc)... but I had this problem with Cyrus IMAP: # /etc/init.d/cyrus restart Apagando cyrus-imapd: [ OK ] Exportando la base de datos cyrus-imapd : [ OK ] ... ... ... ... this takes so long that I decided to investigate what was the problem and I found that /var/lib/imap/db uses 21 GB. Is is normal that uses so many space? I found around 1700 files named log.000* and I just decided to not use them: # cd /var/lib/imap # mv db db2 # cp -a db.backup db # /etc/init.d/cyrus-imapd start And then Cyrus IMAP started correctly (because /var/lib/imap/db.backup ocuped just a few KB) but now IMAP users are telling me that their e-mails are gone! However I can see that all their mails are under /var/spool/imap/domain/X/XDOMAIN/u/user using a few hundreds of MB so I can believe that their emails aren't gone. What happened to users emails? How can I recover them? I would appreciate some help, thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090916/7ae1b7e3/attachment.html From simon.matter at invoca.ch Wed Sep 16 12:26:43 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Wed, 16 Sep 2009 18:26:43 +0200 Subject: Recovering of mailboxes In-Reply-To: References: Message-ID: <0a70efa86d3c6a041ef1268b8dfead67.squirrel@webmail.bi.corp.invoca.ch> > HI all: > > I'm running Cyrus IMAP 2.3.7 in a CentOS 5.3 box. We have about 400 users, > most of them POP3 users but some are IMAP users with their mail stored at > the mailserver. > I recently had a out of space problem, so I removed so many unnecesary > files > (logs, backups, among others) and then restarted mail services > (MailScanner, > MySQL, ClamAV, Cyrus IMAP, etc)... but I had this problem with Cyrus IMAP: > > # /etc/init.d/cyrus restart > Apagando cyrus-imapd: [ OK ] > Exportando la base de datos cyrus-imapd : [ OK ] > > ... ... > ... ... > > this takes so long that I decided to investigate what was the problem and > I > found that /var/lib/imap/db uses 21 GB. Is is normal that uses so many > space? I found around 1700 files named log.000* and I just decided to not > use them: > > # cd /var/lib/imap > # mv db db2 > # cp -a db.backup db > # /etc/init.d/cyrus-imapd start > > And then Cyrus IMAP started correctly (because /var/lib/imap/db.backup > ocuped just a few KB) but now IMAP users are telling me that their e-mails > are gone! However I can see that all their mails are under > /var/spool/imap/domain/X/XDOMAIN/u/user using a few hundreds of MB so I > can > believe that their emails aren't gone. > > What happened to users emails? How can I recover them? Did you try to run reconstruct? Something like su - cyrus -c "/usr/lib/cyrus-imapd/reconstruct -r -f user.xxxxxxxxx" may help. Regards, Simon From simon.matter at invoca.ch Wed Sep 16 12:46:17 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Wed, 16 Sep 2009 18:46:17 +0200 Subject: v2.3.15 IMAP connection hangs In-Reply-To: <43C68785C2728049AF86B0ECB240A151117DC7CCB9@EXCHMB.ornl.gov> References: <43C68785C2728049AF86B0ECB240A151117DB6E4E7@EXCHMB.ornl.gov> <20090910221036.GA1177@brong.net> <43C68785C2728049AF86B0ECB240A151117DB6E653@EXCHMB.ornl.gov> <20090911225309.GA4536@brong.net> <43C68785C2728049AF86B0ECB240A151117DC7C86E@EXCHMB.ornl.gov> <20090916151213.GA25255@brong.net> <43C68785C2728049AF86B0ECB240A151117DC7CCB9@EXCHMB.ornl.gov> Message-ID: <8962e5139ced2fbdc48dff39999ae00a.squirrel@webmail.bi.corp.invoca.ch> >> From: Bron Gondwana [mailto:brong at fastmail.fm] >>... >> Hmm - question: do you have zlib support compiled in? >> >> [brong at utility1 ~]$ ldd /usr/cyrus/bin/imapd | grep libz >> libz.so.1 => /usr/lib/libz.so.1 (0x55990000) >> >> Maybe there's a bug in the codepaths for COMPRESS when zlib doesn't >> exist (because they're all #ifdef HAVE_ZLIB - maybe things don't work >> nicely without it!) >> >> Long shot - but you might want to try it with the attached patch >> applied. >> It rewrites the zlib handling logic quite a lot, though it wasn't >> originally >> designed to deal with your problem! I'm going to push it to CVS soon. > > I already have the patch installed because we have an old version of Zlib. > Could the patch be causing the problem? Hi, I have the same patch in my RPMs and tested it on a ancient OS/zlib and didn't see any problems. Of course it doesn't mean much because I'm running Linux. Did you already try to build and run without zlib support? Simon From rosenbaumlm at ornl.gov Wed Sep 16 12:53:41 2009 From: rosenbaumlm at ornl.gov (Rosenbaum, Larry M.) Date: Wed, 16 Sep 2009 12:53:41 -0400 Subject: v2.3.15 IMAP connection hangs In-Reply-To: <8962e5139ced2fbdc48dff39999ae00a.squirrel@webmail.bi.corp.invoca.ch> References: <43C68785C2728049AF86B0ECB240A151117DB6E4E7@EXCHMB.ornl.gov> <20090910221036.GA1177@brong.net> <43C68785C2728049AF86B0ECB240A151117DB6E653@EXCHMB.ornl.gov> <20090911225309.GA4536@brong.net> <43C68785C2728049AF86B0ECB240A151117DC7C86E@EXCHMB.ornl.gov> <20090916151213.GA25255@brong.net> <43C68785C2728049AF86B0ECB240A151117DC7CCB9@EXCHMB.ornl.gov> <8962e5139ced2fbdc48dff39999ae00a.squirrel@webmail.bi.corp.invoca.ch> Message-ID: <43C68785C2728049AF86B0ECB240A151117DC7CDB1@EXCHMB.ornl.gov> > From: Simon Matter [mailto:simon.matter at invoca.ch] >... > >> Long shot - but you might want to try it with the attached patch > >> applied. > >> It rewrites the zlib handling logic quite a lot, though it wasn't > >> originally > >> designed to deal with your problem! I'm going to push it to CVS > soon. > > > > I already have the patch installed because we have an old version of > Zlib. > > Could the patch be causing the problem? > > Hi, > > I have the same patch in my RPMs and tested it on a ancient OS/zlib and > didn't see any problems. Of course it doesn't mean much because I'm > running Linux. > Did you already try to build and run without zlib support? I hadn't thought of building it without zlib support. I have just installed a newer zlib, removed the patch, and rebuilt Cyrus IMAP with the new zlib. Now it works! Larry From simon.matter at invoca.ch Wed Sep 16 13:04:05 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Wed, 16 Sep 2009 19:04:05 +0200 Subject: v2.3.15 IMAP connection hangs In-Reply-To: <43C68785C2728049AF86B0ECB240A151117DC7CDB1@EXCHMB.ornl.gov> References: <43C68785C2728049AF86B0ECB240A151117DB6E4E7@EXCHMB.ornl.gov> <20090910221036.GA1177@brong.net> <43C68785C2728049AF86B0ECB240A151117DB6E653@EXCHMB.ornl.gov> <20090911225309.GA4536@brong.net> <43C68785C2728049AF86B0ECB240A151117DC7C86E@EXCHMB.ornl.gov> <20090916151213.GA25255@brong.net> <43C68785C2728049AF86B0ECB240A151117DC7CCB9@EXCHMB.ornl.gov> <8962e5139ced2fbdc48dff39999ae00a.squirrel@webmail.bi.corp.invoca.ch> <43C68785C2728049AF86B0ECB240A151117DC7CDB1@EXCHMB.ornl.gov> Message-ID: <71ea10c069b05fb055857a78141776a0.squirrel@webmail.bi.corp.invoca.ch> >> From: Simon Matter [mailto:simon.matter at invoca.ch] >>... >> >> Long shot - but you might want to try it with the attached patch >> >> applied. >> >> It rewrites the zlib handling logic quite a lot, though it wasn't >> >> originally >> >> designed to deal with your problem! I'm going to push it to CVS >> soon. >> > >> > I already have the patch installed because we have an old version of >> Zlib. >> > Could the patch be causing the problem? >> >> Hi, >> >> I have the same patch in my RPMs and tested it on a ancient OS/zlib and >> didn't see any problems. Of course it doesn't mean much because I'm >> running Linux. >> Did you already try to build and run without zlib support? > > I hadn't thought of building it without zlib support. > > I have just installed a newer zlib, removed the patch, and rebuilt Cyrus > IMAP with the new zlib. Now it works! Good, but I'm sure Bron would still like to know why it didn't work before... Simon From cyruspy at gmail.com Wed Sep 16 18:49:25 2009 From: cyruspy at gmail.com (Ciro Iriarte) Date: Wed, 16 Sep 2009 18:49:25 -0400 Subject: Two cyrus processes, one mail spool Message-ID: Hi, is it safe to run two cyrus processes on the same filesystem/mail spool to provide scalability and HA?, in example, two servers accessing one clustered filesystem or NFS directory?. Regards, -- Ciro Iriarte http://cyruspy.wordpress.com -- From brong at fastmail.fm Wed Sep 16 20:21:59 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 17 Sep 2009 10:21:59 +1000 Subject: v2.3.15 IMAP connection hangs In-Reply-To: <71ea10c069b05fb055857a78141776a0.squirrel@webmail.bi.corp.invoca.ch> References: <43C68785C2728049AF86B0ECB240A151117DB6E4E7@EXCHMB.ornl.gov> <20090910221036.GA1177@brong.net> <43C68785C2728049AF86B0ECB240A151117DB6E653@EXCHMB.ornl.gov> <20090911225309.GA4536@brong.net> <43C68785C2728049AF86B0ECB240A151117DC7C86E@EXCHMB.ornl.gov> <20090916151213.GA25255@brong.net> <43C68785C2728049AF86B0ECB240A151117DC7CCB9@EXCHMB.ornl.gov> <8962e5139ced2fbdc48dff39999ae00a.squirrel@webmail.bi.corp.invoca.ch> <43C68785C2728049AF86B0ECB240A151117DC7CDB1@EXCHMB.ornl.gov> <71ea10c069b05fb055857a78141776a0.squirrel@webmail.bi.corp.invoca.ch> Message-ID: <20090917002159.GA5172@brong.net> On Wed, Sep 16, 2009 at 07:04:05PM +0200, Simon Matter wrote: > >> From: Simon Matter [mailto:simon.matter at invoca.ch] > >>... > >> >> Long shot - but you might want to try it with the attached patch > >> >> applied. > >> >> It rewrites the zlib handling logic quite a lot, though it wasn't > >> >> originally > >> >> designed to deal with your problem! I'm going to push it to CVS > >> soon. > >> > > >> > I already have the patch installed because we have an old version of > >> Zlib. > >> > Could the patch be causing the problem? > >> > >> Hi, > >> > >> I have the same patch in my RPMs and tested it on a ancient OS/zlib and > >> didn't see any problems. Of course it doesn't mean much because I'm > >> running Linux. > >> Did you already try to build and run without zlib support? > > > > I hadn't thought of building it without zlib support. > > > > I have just installed a newer zlib, removed the patch, and rebuilt Cyrus > > IMAP with the new zlib. Now it works! > > Good, but I'm sure Bron would still like to know why it didn't work before... I sure would! Given that we've been running that patch on top of CVS for a few months not (Linux with a newer zlib) without problems, I'd love to know what's wrong with the patch - particularly since I'm planning to add it to CVS if there's nothing else wrong with it. So it goes into an infinite loop on signals_poll, obviously. Can you please try building with the patch, but with the new zlib? Thanks, Bron. From pcravero at as2594.net Thu Sep 17 02:47:49 2009 From: pcravero at as2594.net (Paolo Cravero) Date: Thu, 17 Sep 2009 08:47:49 +0200 Subject: Two cyrus processes, one mail spool In-Reply-To: References: Message-ID: <4AB1DB95.3010109@as2594.net> Ciro Iriarte wrote: > Hi, is it safe to run two cyrus processes on the same filesystem/mail > spool to provide scalability and HA?, in example, two servers > accessing one clustered filesystem or NFS directory?. See http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/CyrusCluster See also the archives and my message of 2009-09-04 "cyrus-imapd in (veritas) cluster: anyone?" that received no answers (except one private saying they're using Sun Cluster). Then try with more specific questions. Paolo From simon.matter at invoca.ch Thu Sep 17 02:49:41 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Thu, 17 Sep 2009 08:49:41 +0200 Subject: Two cyrus processes, one mail spool In-Reply-To: References: Message-ID: <497a288455c1f206f04c0752956081e9.squirrel@webmail.bi.corp.invoca.ch> > Hi, is it safe to run two cyrus processes on the same filesystem/mail > spool to provide scalability and HA?, in example, two servers > accessing one clustered filesystem or NFS directory?. Hi, NFS no, that doesn't seem to work even on NFSv4. At least no one has confirmed it to work IIRC. With a clustered filesystem it should work, but make sure you don't run Berkeley DB backends, run it with skiplist instead. Regards, Simon From support at spectrum.ru Thu Sep 17 08:42:10 2009 From: support at spectrum.ru (Alexander Demin) Date: Thu, 17 Sep 2009 16:42:10 +0400 Subject: Synchronisation two cyrus-imapd servers Message-ID: <1253191330.1801.14.camel@support.spectrum.ru> Hello. I have problem with synchronisation two cyrus-imapd servers. ******* Start "Replica" host configuration ******* OS: FreeBSD 7.2-STABLE i386 cyrus-imapd-2.3.15 WITH_BDB=true WITH_REPLICATION=true cyrus-sasl-2.1.23 WITH_AUTHDAEMOND=true WITH_LOGIN=true WITH_PLAIN=true WITH_CRAM=true WITH_DIGEST=true cyrus-sasl-saslauthd-2.1.23 All soft installed from ports. Cyrus configuration: /usr/local/etc/cyrus.conf START { recover cmd="ctl_cyrusdb -r" } SERVICES { imap cmd="imapd" listen="imap" prefork=0 imaps cmd="imapd -s" listen="imaps" prefork=0 pop3 cmd="pop3d" listen="pop3" prefork=0 pop3s cmd="pop3d -s" listen="pop3s" prefork=0 sieve cmd="timsieved" listen="sieve" prefork=0 lmtpunix cmd="lmtpd" listen="/data/imap/socket/lmtp" prefork=0 smmap cmd="smmapd" listen="/data/imap/socket/smmap" prefork=1 syncserver cmd="sync_server" listen="csync" prefork=1 } EVENTS { checkpoint cmd="ctl_cyrusdb -c" period=30 delprune cmd="cyr_expire -E 3" at=0400 tlsprune cmd="tls_prune" at=0400 } /usr/local/etc/imapd.conf configdirectory: /backup/imap partition-default: /backup/spool/imap unixhierarchysep: no altnamespace: yes allowanonymouslogin: no allowplaintext: yes imapidresponse: yes admins: cyrus munge8bit: 0 rfc2046_strict: 0 sievedir: /backup/imap/sieve sendmail: /usr/sbin/sendmail postmaster: postmaster annotation_db: skiplist duplicate_db: berkeley-nosync mboxlist_db: skiplist ptscache_db: berkeley seenstate_db: skiplist subscription_db: flat sasl_pwcheck_method: auxprop sasl_auxprop_plugin: sasldb sasl_log_level: 7 sasl_mech_list: plain cram-md5 digest-md5 login lmtpsocket: /backup/imap/socket/lmtp virtdomains: userid lmtp_downcase_rcpt: 1 # # EOF /etc/services csync 2005/tcp /etc/rc.conf (show only cyrus/sasl params) cyrus_imapd_enable="YES" saslauthd_enable="YES" saslauthd_flags="-a sasldb" ******* End "Replica" host configuration ******* ******* Start "Master" host configuration ******* OS: FreeBSD 7.2-STABLE amd64 cyrus-imapd-2.3.15 WITH_BDB=true WITH_REPLICATION=true cyrus-sasl-2.1.23 WITH_AUTHDAEMOND=true WITH_LOGIN=true WITH_PLAIN=true WITH_CRAM=true WITH_DIGEST=true cyrus-sasl-saslauthd-2.1.23 All soft installed from ports. Cyrus configuration: /usr/local/etc/cyrus.conf START { recover cmd="ctl_cyrusdb -r" } SERVICES { imap cmd="imapd" listen="imap" prefork=0 imaps cmd="imapd -s" listen="imaps" prefork=0 pop3 cmd="pop3d" listen="pop3" prefork=0 pop3s cmd="pop3d -s" listen="pop3s" prefork=0 sieve cmd="timsieved" listen="sieve" prefork=0 lmtpunix cmd="lmtpd" listen="/data/imap/socket/lmtp" prefork=0 smmap cmd="smmapd" listen="/data/imap/socket/smmap" prefork=1 syncclient cmd="sync_client -r" listen="csync" prefork=1 } EVENTS { checkpoint cmd="ctl_cyrusdb -c" period=30 delprune cmd="cyr_expire -E 3" at=0400 tlsprune cmd="tls_prune" at=0400 } /usr/local/etc/imapd.conf configdirectory: /data/imap partition-default: /data/spool/imap unixhierarchysep: no altnamespace: yes allowanonymouslogin: no allowplaintext: yes imapidresponse: yes admins: cyrus cyrus at spectrum.ru munge8bit: 0 rfc2046_strict: 0 sievedir: /data/imap/sieve sendmail: /usr/sbin/sendmail postmaster: postmaster annotation_db: skiplist duplicate_db: berkeley-nosync mboxlist_db: skiplist ptscache_db: berkeley seenstate_db: skiplist subscription_db: flat sasl_pwcheck_method: auxprop sasl_auxprop_plugin: sasldb sasl_log_level: 7 sasl_mech_list: plain cram-md5 digest-md5 login tls_cert_file: /etc/ssl/imapserver.pem tls_key_file: /etc/ssl/imapserver.pem tls_ca_file: /etc/ssl/imapserver.pem tls_session_timeout: 0 lmtpsocket: /data/imap/socket/lmtp virtdomains: userid lmtp_downcase_rcpt: 1 sync_repeat_interval: 10 sync_host: support.spectrum.ru sync_authname: cyrus sync_password: *********** sync_log: 1 # # EOF /etc/services csync 2005/tcp /etc/rc.conf (show only cyrus/sasl params) cyrus_imapd_enable="YES" saslauthd_enable="YES" saslauthd_flags="-a sasldb" "Master" - it's production mail server of my company. All services worked is fine. ******* End "Master" host configuration ******* Step-by-step: "Replica" host 1. /usr/local/etc/rc.d/imapd start 2. imtest -a cyrus localhost - has passed successfully 3. synctest -u cyrus localhost - failed S: * SASL LOGIN PLAIN DIGEST-MD5 CRAM-MD5 S: * OK support.spectrum.ru Cyrus sync server v2.3.15 C: AUTHENTICATE DIGEST-MD5 failure: prot layer failure 4. ps -ax | grep sync_server 65257 ?? I 0:00,00 sync_server 65617 ?? I 0:00,00 sync_server 5. grep sync /var/log/all.log Sep 17 15:59:24 support syncserver[65589]: accepted connection Sep 17 15:59:24 support master[65616]: about to exec /usr/local/cyrus/bin/sync_server Sep 17 15:59:24 support kernel: pid 65589 (sync_server), uid 60: exited on signal 11 Sep 17 15:59:24 support syncserver[65589]: cmdloop(): startup Sep 17 15:59:24 support syncserver[65616]: executed Sep 17 15:59:24 support master[65253]: service syncserver pid 65589 in BUSY state: terminated abnormally Sep 17 15:59:24 support syncserver[65616]: accepted connection Sep 17 15:59:24 support master[65617]: about to exec /usr/local/cyrus/bin/sync_server Sep 17 15:59:24 support kernel: pid 65616 (sync_server), uid 60: exited on signal 11 Sep 17 15:59:24 support syncserver[65616]: cmdloop(): startup Sep 17 15:59:24 support syncserver[65617]: executed Sep 17 15:59:24 support master[65253]: service syncserver pid 65616 in BUSY state: terminated abnormally "Master" host 1. /usr/local/etc/rc.d/imapd restart 2. imtest -a cyrus localhost - has passed successfully 3. ps -ax | grep sync_client 63196 ?? S 0:00,01 sync_client -r 63197 ?? S 0:00,01 sync_client -r 4. grep sync /var/log/all.log Sep 17 16:24:18 mail sync_client[63196]: couldn't authenticate to backend server: generic failure Sep 17 16:24:18 mail sync_client[63197]: couldn't authenticate to backend server: generic failure Sep 17 16:25:18 mail sync_client[63196]: couldn't authenticate to backend server: generic failure Sep 17 16:25:18 mail sync_client[63197]: couldn't authenticate to backend server: generic failure Did i make something not correctly? Help me, please, to find the mistake and understand this problem. Thanks. -- Demin Alexander / Network Administrator Group of companies Spectrum / tel. (+7 495) 995-8999 Russia, Moscow, 103009, Strastnoy blvr. 8 Web: http://www.spectrum.ru/ From rosenbaumlm at ornl.gov Thu Sep 17 10:13:10 2009 From: rosenbaumlm at ornl.gov (Rosenbaum, Larry M.) Date: Thu, 17 Sep 2009 10:13:10 -0400 Subject: v2.3.15 IMAP connection hangs In-Reply-To: <20090917002159.GA5172@brong.net> References: <43C68785C2728049AF86B0ECB240A151117DB6E4E7@EXCHMB.ornl.gov> <20090910221036.GA1177@brong.net> <43C68785C2728049AF86B0ECB240A151117DB6E653@EXCHMB.ornl.gov> <20090911225309.GA4536@brong.net> <43C68785C2728049AF86B0ECB240A151117DC7C86E@EXCHMB.ornl.gov> <20090916151213.GA25255@brong.net> <43C68785C2728049AF86B0ECB240A151117DC7CCB9@EXCHMB.ornl.gov> <8962e5139ced2fbdc48dff39999ae00a.squirrel@webmail.bi.corp.invoca.ch> <43C68785C2728049AF86B0ECB240A151117DC7CDB1@EXCHMB.ornl.gov> <71ea10c069b05fb055857a78141776a0.squirrel@webmail.bi.corp.invoca.ch> <20090917002159.GA5172@brong.net> Message-ID: <43C68785C2728049AF86B0ECB240A151117DC7D1D5@EXCHMB.ornl.gov> > From: Bron Gondwana [mailto:brong at fastmail.fm] > Sent: Wednesday, September 16, 2009 8:22 PM > To: Simon Matter > Cc: Rosenbaum, Larry M.; Bron Gondwana; info-cyrus at lists.andrew.cmu.edu > Subject: Re: v2.3.15 IMAP connection hangs > > On Wed, Sep 16, 2009 at 07:04:05PM +0200, Simon Matter wrote: > > >> From: Simon Matter [mailto:simon.matter at invoca.ch] > > >>... > > >> >> Long shot - but you might want to try it with the attached > patch > > >> >> applied. > > >> >> It rewrites the zlib handling logic quite a lot, though it > wasn't > > >> >> originally > > >> >> designed to deal with your problem! I'm going to push it to > CVS > > >> soon. > > >> > > > >> > I already have the patch installed because we have an old > version of > > >> Zlib. > > >> > Could the patch be causing the problem? > > >> > > >> Hi, > > >> > > >> I have the same patch in my RPMs and tested it on a ancient > OS/zlib and > > >> didn't see any problems. Of course it doesn't mean much because > I'm > > >> running Linux. > > >> Did you already try to build and run without zlib support? > > > > > > I hadn't thought of building it without zlib support. > > > > > > I have just installed a newer zlib, removed the patch, and rebuilt > Cyrus > > > IMAP with the new zlib. Now it works! > > > > Good, but I'm sure Bron would still like to know why it didn't work > before... > > I sure would! Given that we've been running that patch on top of CVS > for > a few months not (Linux with a newer zlib) without problems, I'd love > to > know what's wrong with the patch - particularly since I'm planning to > add > it to CVS if there's nothing else wrong with it. > > So it goes into an infinite loop on signals_poll, obviously. > > Can you please try building with the patch, but with the new zlib? With the new zlib and the patch, it hangs. Perhaps the problem with the patch is Solaris-specific. From cyruspy at gmail.com Thu Sep 17 10:34:05 2009 From: cyruspy at gmail.com (Ciro Iriarte) Date: Thu, 17 Sep 2009 10:34:05 -0400 Subject: Two cyrus processes, one mail spool In-Reply-To: <4AB1DB95.3010109@as2594.net> References: <4AB1DB95.3010109@as2594.net> Message-ID: 2009/9/17 Paolo Cravero : > Ciro Iriarte wrote: > >> Hi, is it safe to run two cyrus processes on the same filesystem/mail >> spool to provide scalability and HA?, in example, two servers >> accessing one clustered filesystem or NFS directory?. > > See > > http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/CyrusCluster Thanks, that helps > > See also the archives and my message of ?2009-09-04 "cyrus-imapd in (veritas) > cluster: anyone?" that received no answers (except one private saying they're > using Sun Cluster). > > Then try with more specific questions. I'm more concerned about file locking (most important) and FS overall speed... I'm considering to built the solution on top of Linux+OCFS2 > > Paolo Regards, -- Ciro Iriarte http://cyruspy.wordpress.com -- From cyruspy at gmail.com Thu Sep 17 10:36:03 2009 From: cyruspy at gmail.com (Ciro Iriarte) Date: Thu, 17 Sep 2009 10:36:03 -0400 Subject: Two cyrus processes, one mail spool In-Reply-To: <497a288455c1f206f04c0752956081e9.squirrel@webmail.bi.corp.invoca.ch> References: <497a288455c1f206f04c0752956081e9.squirrel@webmail.bi.corp.invoca.ch> Message-ID: 2009/9/17 Simon Matter : >> Hi, is it safe to run two cyrus processes on the same filesystem/mail >> spool to provide scalability and HA?, in example, two servers >> accessing one clustered filesystem or NFS directory?. > > Hi, NFS no, that doesn't seem to work even on NFSv4. At least no one has > confirmed it to work IIRC. > With a clustered filesystem it should work, but make sure you don't run > Berkeley DB backends, run it with skiplist instead. > > Regards, > Simon > > BDB is the default, isn't it?, how would skiplist affect functionality/speed?, is that just a configuration parameter o cyrus hast to be built/compiled again to support it?. Regards, -- Ciro Iriarte http://cyruspy.wordpress.com -- From simon.matter at invoca.ch Thu Sep 17 10:42:46 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Thu, 17 Sep 2009 16:42:46 +0200 Subject: Two cyrus processes, one mail spool In-Reply-To: References: <497a288455c1f206f04c0752956081e9.squirrel@webmail.bi.corp.invoca.ch> Message-ID: <9192199210c633d23586975631968caa.squirrel@webmail.bi.corp.invoca.ch> > 2009/9/17 Simon Matter : >>> Hi, is it safe to run two cyrus processes on the same filesystem/mail >>> spool to provide scalability and HA?, in example, two servers >>> accessing one clustered filesystem or NFS directory?. >> >> Hi, NFS no, that doesn't seem to work even on NFSv4. At least no one has >> confirmed it to work IIRC. >> With a clustered filesystem it should work, but make sure you don't run >> Berkeley DB backends, run it with skiplist instead. >> >> Regards, >> Simon >> >> > > BDB is the default, isn't it?, how would skiplist affect > functionality/speed?, is that just a configuration parameter o cyrus > hast to be built/compiled again to support it?. It all depends on which version you are using. In recent versions most databases are skiplist by default and it's a config option, not compile time anymore. Many people including me run skiplist only and it works well. This has been discussed many times on the list so you may find more infos in the archive. Simon From jmasterson at modwest.com Thu Sep 17 11:20:31 2009 From: jmasterson at modwest.com (John Masterson) Date: Thu, 17 Sep 2009 09:20:31 -0600 Subject: debian etch upgrade, now 64bit machines cannot connect to backends (SSL) In-Reply-To: <4AAD628A.3060400@modwest.com> References: <4AAD628A.3060400@modwest.com> Message-ID: <4AB253BF.7060306@modwest.com> For posterity and the list archives: We believe we identified our problem. A previous administrator made source changes to our packages to remove the requirement of the frontends' SSL communication with the backends, since these machines are utterly private inside our network and he wanted to 1) make debugging easier, and 2) reduce performance and administration overhead. John Masterson wrote: > We just upgraded to 2.2.13-10+etch2 and now our 64bit front end machines > cannot connect to our back end servers, e.g., > > > Sep 13 11:55:49 mfe5 cyrus/mc2pop3[6764]: > connect(mbe2.msomt.modwest.com) failed: Invalid argument > Sep 13 11:55:49 mfe5 cyrus/mc2pop3[6764]: couldn't authenticate to > backend server > > also: > > Sep 13 15:15:39 mfe5 cyrus/mupdate[1327]: Thread timed out waiting for > listener_lock > > > > A 32-bit front end machine is working fine. We don't think there are any > other differences between these installations. cyrus.conf, mupdate.conf, > and imap.conf are identical except for the obvious, listen addresses, > hostnames. > > > Any ideas or experience with this? > > -- > John Masterson > http://modwest.com > > > > > > > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > > > From vfort at provident-solutions.com Thu Sep 17 20:26:17 2009 From: vfort at provident-solutions.com (Vernon A. Fort) Date: Thu, 17 Sep 2009 19:26:17 -0500 Subject: deliver to mailbox folder not working Message-ID: <4AB2D3A9.6080801@provident-solutions.com> I have a script that re-delivers email messages from one mailbox to another. The following worked with 2.3.13 but does NOT work with 2.3.14 /usr/lib/cyrus/deliver -a testuser -m user.testuser.subfolder < email.txt All i get is mailbox does not exist and a "lmtpunix: verify_user(user.textuser.subfolder) failed: Mailbox does not exist. All the acl are correct - set to 'all' on both the inbox and subfolder. The message IS a valid email message and again, the deliver command worked with the previous version. I'm using gentoo and they removed the 2.3.13 version from portage. Has the command syntax changes.... Vernon From simon.matter at invoca.ch Fri Sep 18 02:23:33 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Fri, 18 Sep 2009 08:23:33 +0200 Subject: deliver to mailbox folder not working In-Reply-To: <4AB2D3A9.6080801@provident-solutions.com> References: <4AB2D3A9.6080801@provident-solutions.com> Message-ID: > I have a script that re-delivers email messages from one mailbox to > another. The following worked with 2.3.13 but does NOT work with 2.3.14 > > /usr/lib/cyrus/deliver -a testuser -m user.testuser.subfolder < email.txt > > All i get is mailbox does not exist and a "lmtpunix: > verify_user(user.textuser.subfolder) failed: Mailbox does not exist. > All the acl are correct - set to 'all' on both the inbox and subfolder. > The message IS a valid email message and again, the deliver command > worked with the previous version. I'm using gentoo and they removed the > 2.3.13 version from portage. Has the command syntax changes.... IIRC that kind of error may be caused by a bug which is in 2.3.14. I strongly suggest to update to 2.3.15 anyway because it also fixes a security issue. Please let us know whether the issue has gone with 2.3.15. Regards, Simon From arbatovevgeniy at gmail.com Fri Sep 18 08:43:20 2009 From: arbatovevgeniy at gmail.com (Evgeniy Arbatov) Date: Fri, 18 Sep 2009 15:43:20 +0300 Subject: Same mailbox with different logins Message-ID: <56c989d50909180543v5e2adf66w7adcd834f869b1c7@mail.gmail.com> Hello, A question: I want users authenticating with two different usernames to Cyrus IMAP to be able to access the same mailbox. For example, I wish this to be the case ("login name" -> "mailbox"): username -> user/firstname.lastname at domain firstname.lastname at domain -> user/firstname.lastname at domain What would be the way to proceed? At the moment I have: username -> user/username firstname.lastname at domain -> user/firstname.lastname at domain With the two mailboxes being completely separate. Thank you in advance! Best regards, Evgeniy From tschlabach at gmx.net Fri Sep 18 09:23:49 2009 From: tschlabach at gmx.net (Torsten Schlabach) Date: Fri, 18 Sep 2009 15:23:49 +0200 Subject: Same mailbox with different logins In-Reply-To: <56c989d50909180543v5e2adf66w7adcd834f869b1c7@mail.gmail.com> References: <56c989d50909180543v5e2adf66w7adcd834f869b1c7@mail.gmail.com> Message-ID: <4AB389E5.1040102@gmx.net> Hi Evgeniy! > For example, I wish this to be the case ("login name" -> "mailbox"): > > username -> user/firstname.lastname at domain > firstname.lastname at domain -> user/firstname.lastname at domain First of all, I personally don't recommend this naming scheme for a number of reasons: 1. Names of people may change over time or just need a correction. But renaming a mailbox in Cyrus is non trivial! 2. Same for domain names. Think additional domain names, for example. We have set up our system in a way that the mailbox name is an entirely arbitrary technical id, such as XXXyyyyy with XXX beding a kind of group / customer prefix and yyyyy being just a unique number. Independent of that and despite a number of wrong answers to this question on the archives of this list: If the authentication id of the user (commonly called "username") and the mailbox name are going to be any different from each other, you need to use a canon_user sasl plugin to make this happen. You can *not* solve this using any kind of mapping which is hidden in your authentication mechanism, such as PAM or LDAP, simple because after a successful authentication imapd / popd will always search for a mailbox with the name of the authentication id and the only hook you have there is the canon_user API, which is what the canon_user API got made for. HTH Regards, Torsten Evgeniy Arbatov schrieb: > Hello, > > A question: I want users authenticating with two different usernames > to Cyrus IMAP to be able to access the same mailbox. > > For example, I wish this to be the case ("login name" -> "mailbox"): > > username -> user/firstname.lastname at domain > firstname.lastname at domain -> user/firstname.lastname at domain > > What would be the way to proceed? > > At the moment I have: > > username -> user/username > firstname.lastname at domain -> user/firstname.lastname at domain > > With the two mailboxes being completely separate. > > Thank you in advance! > > Best regards, > Evgeniy > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html From brong at fastmail.fm Fri Sep 18 09:24:08 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 18 Sep 2009 23:24:08 +1000 Subject: Same mailbox with different logins In-Reply-To: <56c989d50909180543v5e2adf66w7adcd834f869b1c7@mail.gmail.com> References: <56c989d50909180543v5e2adf66w7adcd834f869b1c7@mail.gmail.com> Message-ID: <20090918132408.GA7710@brong.net> On Fri, Sep 18, 2009 at 03:43:20PM +0300, Evgeniy Arbatov wrote: > Hello, > > A question: I want users authenticating with two different usernames > to Cyrus IMAP to be able to access the same mailbox. > > For example, I wish this to be the case ("login name" -> "mailbox"): > > username -> user/firstname.lastname at domain > firstname.lastname at domain -> user/firstname.lastname at domain > > What would be the way to proceed? Well - we just put an nginx proxy in front, and have the authentication daemon return the "correct" username to pass to the backend, regardless of the username used. That's one option. What's the use case anyway? You can use aliases for delivery, why does the one user need two different usernames? Bron. From brong at fastmail.fm Fri Sep 18 10:26:12 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sat, 19 Sep 2009 00:26:12 +1000 Subject: Same mailbox with different logins In-Reply-To: <4AB389E5.1040102@gmx.net> References: <56c989d50909180543v5e2adf66w7adcd834f869b1c7@mail.gmail.com> <4AB389E5.1040102@gmx.net> Message-ID: <20090918142612.GA3534@brong.net> On Fri, Sep 18, 2009 at 03:23:49PM +0200, Torsten Schlabach wrote: > 1. Names of people may change over time or just need a correction. But > renaming a mailbox in Cyrus is non trivial! Really? telnet $hostname 143 . login admin . rename user.olduser user.newuser Doesn't tickle my "non trivial" buttons much. (we have perl APIs that wrap all that and make it even easier, but still) Bron. From lambert at lambertfam.org Fri Sep 18 19:06:52 2009 From: lambert at lambertfam.org (Scott Lambert) Date: Fri, 18 Sep 2009 18:06:52 -0500 Subject: Same mailbox with different logins In-Reply-To: <20090918132408.GA7710@brong.net> References: <56c989d50909180543v5e2adf66w7adcd834f869b1c7@mail.gmail.com> <20090918132408.GA7710@brong.net> Message-ID: <20090918230652.GF66091@sysmon.tcworks.net> On Fri, Sep 18, 2009 at 11:24:08PM +1000, Bron Gondwana wrote: > On Fri, Sep 18, 2009 at 03:43:20PM +0300, Evgeniy Arbatov wrote: > > Hello, > > > > A question: I want users authenticating with two different usernames > > to Cyrus IMAP to be able to access the same mailbox. > > > > For example, I wish this to be the case ("login name" -> "mailbox"): > > > > username -> user/firstname.lastname at domain > > firstname.lastname at domain -> user/firstname.lastname at domain > > > > What would be the way to proceed? > > Well - we just put an nginx proxy in front, and have the authentication > daemon return the "correct" username to pass to the backend, regardless > of the username used. That's one option. > > What's the use case anyway? You can use aliases for delivery, why does > the one user need two different usernames? I don't know about the original poster but the ISP I work for has been doing e-mail with sendmail and IMAP-UW. All usernames are of the form username at maindomain, only they don't use @maindomain. Customer domain users use mailboxes of the form domainSpecificPrefixOf2or3characters_localpart for localpart at customerdomain. virtusertable: localpart at customerdomain.com cd_localpart We would like to get those users out of the maindomain and have them switch to using mailboxes of localpart at customerdomain.com and be able to login as localpart at customerdomain.com but have compatibility with the old way of loging in as cd_localpart as well, at least during the transition period. But we haven't gotten to giving that a lot of thought yet. -- Scott Lambert KC5MLE Unix SysAdmin lambert at lambertfam.org From brong at fastmail.fm Fri Sep 18 20:19:01 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sat, 19 Sep 2009 10:19:01 +1000 Subject: Same mailbox with different logins In-Reply-To: <20090918230652.GF66091@sysmon.tcworks.net> References: <56c989d50909180543v5e2adf66w7adcd834f869b1c7@mail.gmail.com> <20090918132408.GA7710@brong.net> <20090918230652.GF66091@sysmon.tcworks.net> Message-ID: <20090919001901.GA3599@brong.net> On Fri, Sep 18, 2009 at 06:06:52PM -0500, Scott Lambert wrote: > On Fri, Sep 18, 2009 at 11:24:08PM +1000, Bron Gondwana wrote: > > On Fri, Sep 18, 2009 at 03:43:20PM +0300, Evgeniy Arbatov wrote: > > > Hello, > > > > > > A question: I want users authenticating with two different usernames > > > to Cyrus IMAP to be able to access the same mailbox. > > > > > > For example, I wish this to be the case ("login name" -> "mailbox"): > > > > > > username -> user/firstname.lastname at domain > > > firstname.lastname at domain -> user/firstname.lastname at domain > > > > > > What would be the way to proceed? > > > > Well - we just put an nginx proxy in front, and have the authentication > > daemon return the "correct" username to pass to the backend, regardless > > of the username used. That's one option. > > > > What's the use case anyway? You can use aliases for delivery, why does > > the one user need two different usernames? > > I don't know about the original poster but the ISP I work for has been > doing e-mail with sendmail and IMAP-UW. > > All usernames are of the form username at maindomain, only they don't use > @maindomain. > > Customer domain users use mailboxes of the form > domainSpecificPrefixOf2or3characters_localpart for > localpart at customerdomain. > > virtusertable: > localpart at customerdomain.com cd_localpart > > We would like to get those users out of the maindomain and have them > switch to using mailboxes of localpart at customerdomain.com and be able > to login as localpart at customerdomain.com but have compatibility with > the old way of loging in as cd_localpart as well, at least during the > transition period. That's almost exactly what we did at FastMail - though the other way around. We still have old mailboxes with the old names, but everyone has to log in as username at domain now. Nginx supports sending an arbitrary username to the backend. The authentication protocol is basically HTTP (username and password sent as headers in the request and result as headers in the response) so it's easy to extend and support new fields, and you can write your authentication daemon in any language pretty easily, especially if it has a HTTP library. Bron. From arbatovevgeniy at gmail.com Mon Sep 21 05:11:53 2009 From: arbatovevgeniy at gmail.com (Evgeniy Arbatov) Date: Mon, 21 Sep 2009 12:11:53 +0300 Subject: Same mailbox with different logins In-Reply-To: <20090919001901.GA3599@brong.net> References: <56c989d50909180543v5e2adf66w7adcd834f869b1c7@mail.gmail.com> <20090918132408.GA7710@brong.net> <20090918230652.GF66091@sysmon.tcworks.net> <20090919001901.GA3599@brong.net> Message-ID: <56c989d50909210211o69fe93e5n37802c34ee7d8b8d@mail.gmail.com> Thank you for your replies! I've decided to go with canon_user plugin. My next question is how to use this plugin. I am trying to use LDAP as authentication backend. What I could find are following imapd.conf settings: sasl_pwcheck_method: saslauthd sasl_mech_list: login plain sasl_auxprop_plugin: ldapdb sasl_ldapdb_uri: ldap://ldap.example.net/ sasl_ldapdb_canon_attr: mail sasl_canon_user_plugin: ldapd imap_sasl_canon_user_plugin: ldapdb pop3_sasl_canon_user_plugin: ldapdb Will this give me canonified username -> firstname.lastname at domain? Do I need to make changes to LDAP for those settings to work? After I configure this ldapdb plugin I see in logs: mail imaps[10161]: canonified earbatov -> earbatov mail imaps[10161]: badlogin: host [10.10.10.10] plain [SASL(-4): no mechanism available: desired canon_user plugin ldapdb not found] mail imaps[10161]: badlogin: host [10.10.10.10] plaintext earbatov SASL(-4): no mechanism available: desired canon_user plugin ldapdb not found I put my complete imapd.conf here http://pastebin.com/m2dbf3951 Evgeniy From rudy.gevaert at ugent.be Mon Sep 21 05:35:35 2009 From: rudy.gevaert at ugent.be (Rudy Gevaert) Date: Mon, 21 Sep 2009 11:35:35 +0200 Subject: Same mailbox with different logins In-Reply-To: <56c989d50909180543v5e2adf66w7adcd834f869b1c7@mail.gmail.com> References: <56c989d50909180543v5e2adf66w7adcd834f869b1c7@mail.gmail.com> Message-ID: <20090921113535.12873ixx5wip7i7b@langoest.ugent.be> Hi, I haven't taken the time to read the other replies but we here allow people to log in with username OR firstname.lastname at domain.com. (At the bottom of the reply is a question regarding canon plugin and shared folders) The users are connecting to a perdition proxy that does login name translation to firstname.lastname at domain.com and redirect the connection to the correct backend (we have 7 backends). Also, we have one of the 7 back ends with the patched saslauthd. On this backend we have 'shared mailboxes'. These are mailboxes that are shared amongst several people. The all log in with a the login name of the shared mailbox. BUT the password they have to use is their **personal** password. Our saslauthd goes to LDAP to fetch all information. The credentials of the shared mailbox are also in the ldap. It holds references to the people who are allowed to log in. If I would have to do it again. I would not the C code in saslauthd! But I would write my own saslauthd daemon. This is trivial, but I did not know that back then :). Some other opinions: - renaming a mailbox in cyrus is not difficult, as Bron has shown - it still is a pain in the ass to have the mailbox name of the user tied to their real name Renames are not so frequent, but still you have to do some book keeping on whose name belonged to which mailbox in the past. E.g. people who change their sex change their name too, but want to still receive mail sent to their old name. (Fix this with an alias.) If I would have to redo our setup I would go with - some_id at domain.com - rewriting whatever.they.want at domain.com to some_id at domain.com with a proxy in front of it (IMAP/POP proxy is no problem, but what about a SIEVE proxy? Can nginx do this? Bron? Perdition can't. :)) Or rewriting with the canon plugin. The only thing I'm still wondering is, what about shared folders? We can't ask a user to share their folders with some_id at domain.com. They should still be able to share with first.last_otheruser at domain.com Bye! Rudy -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Rudy Gevaert Rudy.Gevaert at UGent.be tel:+32 9 264 4734 Directie ICT, afd. Infrastructuur Direction ICT, Infrastructure dept. Groep Systemen Systems group Universiteit Gent Ghent University Krijgslaan 281, gebouw S9, 9000 Gent, Belgie www.UGent.be -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- From dwhite at olp.net Mon Sep 21 10:28:49 2009 From: dwhite at olp.net (Dan White) Date: Mon, 21 Sep 2009 09:28:49 -0500 Subject: Same mailbox with different logins In-Reply-To: <56c989d50909210211o69fe93e5n37802c34ee7d8b8d@mail.gmail.com> References: <56c989d50909180543v5e2adf66w7adcd834f869b1c7@mail.gmail.com> <20090918132408.GA7710@brong.net> <20090918230652.GF66091@sysmon.tcworks.net> <20090919001901.GA3599@brong.net> <56c989d50909210211o69fe93e5n37802c34ee7d8b8d@mail.gmail.com> Message-ID: <20090921142849.GE5045@dan.olp.net> On 21/09/09?12:11?+0300, Evgeniy Arbatov wrote: >Thank you for your replies! I've decided to go with canon_user plugin. >My next question is how to use this plugin. I am trying to use LDAP >as authentication backend. What I could find are following imapd.conf >settings: > >sasl_pwcheck_method: saslauthd >sasl_mech_list: login plain >sasl_auxprop_plugin: ldapdb >sasl_ldapdb_uri: ldap://ldap.example.net/ >sasl_ldapdb_canon_attr: mail >sasl_canon_user_plugin: ldapd >imap_sasl_canon_user_plugin: ldapdb >pop3_sasl_canon_user_plugin: ldapdb > >Will this give me canonified username -> firstname.lastname at domain? Do >I need to make changes to LDAP for those settings to work? > >After I configure this ldapdb plugin I see in logs: > >mail imaps[10161]: canonified earbatov -> earbatov >mail imaps[10161]: badlogin: host [10.10.10.10] plain [SASL(-4): no >mechanism available: desired canon_user plugin ldapdb not found] >mail imaps[10161]: badlogin: host [10.10.10.10] plaintext earbatov >SASL(-4): no mechanism available: desired canon_user plugin ldapdb not >found > >I put my complete imapd.conf here http://pastebin.com/m2dbf3951 Evgeniy, ldapdb, as a canon_user plugin, is not currently found in the 2.1.23 cyrus sasl release. You will need to obtain cyrus sasl source from CVS. There is an upcoming 2.1.24 sasl release that hopefully includes this functionality. Documentation is found within 'docs/options.html' in the sasl source. You will need to configure your openldap server to support proxy authorization, as discussed here: "http://www.openldap.org/doc/admin24/sasl.html#SASL Proxy Authorization" 'sasl_auxprop_plugin: ldapdb' is probably not necessary, since you are using saslauthd for login/plain (only) authentication. Assuming you have openldap proxy authorization set up properly for your environment, the mail attribute (per your config) should return the username you wish to ultimately use. cyrus imap will pretty much remain ignorant of which username you originally authenticated as, and use the identity returned from sasl when searching for mailboxes and applying ACLs. -- Dan White From dwhite at olp.net Mon Sep 21 10:42:05 2009 From: dwhite at olp.net (Dan White) Date: Mon, 21 Sep 2009 09:42:05 -0500 Subject: Same mailbox with different logins In-Reply-To: <20090921113535.12873ixx5wip7i7b@langoest.ugent.be> References: <56c989d50909180543v5e2adf66w7adcd834f869b1c7@mail.gmail.com> <20090921113535.12873ixx5wip7i7b@langoest.ugent.be> Message-ID: <20090921144205.GF5045@dan.olp.net> On 21/09/09?11:35?+0200, Rudy Gevaert wrote: >Hi, > >I haven't taken the time to read the other replies but we here allow >people to log in with username OR firstname.lastname at domain.com. > >(At the bottom of the reply is a question regarding canon plugin and >shared folders) > >If I would have to redo our setup I would go with >- some_id at domain.com >- rewriting whatever.they.want at domain.com to some_id at domain.com with a >proxy in front of it (IMAP/POP proxy is no problem, but what about a >SIEVE proxy? Can nginx do this? Bron? Perdition can't. :)) Or >rewriting with the canon plugin. With a canon_user plugin, all protocols should be supported, including sieve. This is one of the main reasons we moved away from perdition in our environment. -- Dan White From Hagedorn at uni-koeln.de Wed Sep 23 07:04:47 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Wed, 23 Sep 2009 13:04:47 +0200 Subject: delete_mode: delayed not always working? Message-ID: <6396F9955E89A3D753A521BD@tyrion.rrz.uni-koeln.de> Hi, we're running 2.3.14 with "delete_mode: delayed" in /etc/imapd.conf (actually it's the default in Simon Matter's RPM). It's generally working fine, and I have restored a few accidentally deleted mailboxes this way. But today there was a case where a mailbox seems to have been deleted immediately. Here's what I've found: Sep 22 11:41:39 lvr13 imap[3032]: Deleted mailbox user.xxx.Fakultaet.Raumfragen.Dekanat It may be important to know that the mailbox in question had an ACL that allowed other users access. One of the other users accidentally deleted it, not the owner! So I looked for the mailbox, but it couldn't be found: > lm DELETED/user/xxx/* > The output of "ctl_mboxlist -d | grep DELETED" didn't include it, either. Fortunately I had a flashcopy from which I could restore the mailbox, but still I wonder what happened. Before I look at the code myself, I wanted to ask if someone knows offhand if there are known circumstances that cause mailboxes to be deleted immediately. NB: it couldn't have been expunged in the meantime. We call cyr_expire like this: cyr_expire -E 3 -X 3 -D 3. There isn't a cyr_expire line for that mailbox, either. -- .:.Sebastian Hagedorn - RZKR-R1 (Geb?ude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universit?t zu K?ln / Cologne University - ? +49-221-478-5587.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pkcs7-signature Size: 5292 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090923/b349b984/attachment.bin From brong at fastmail.fm Wed Sep 23 07:32:55 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 23 Sep 2009 21:32:55 +1000 Subject: delete_mode: delayed not always working? In-Reply-To: <6396F9955E89A3D753A521BD@tyrion.rrz.uni-koeln.de> References: <6396F9955E89A3D753A521BD@tyrion.rrz.uni-koeln.de> Message-ID: <20090923113255.GA6199@brong.net> On Wed, Sep 23, 2009 at 01:04:47PM +0200, Sebastian Hagedorn wrote: > Hi, > > we're running 2.3.14 with "delete_mode: delayed" in /etc/imapd.conf > (actually it's the default in Simon Matter's RPM). It's generally > working fine, and I have restored a few accidentally deleted > mailboxes this way. But today there was a case where a mailbox seems > to have been deleted immediately. Here's what I've found: > > [...] > > The output of "ctl_mboxlist -d | grep DELETED" didn't include it, > either. Fortunately I had a flashcopy from which I could restore the > mailbox, but still I wonder what happened. Before I look at the code > myself, I wanted to ask if someone knows offhand if there are known > circumstances that cause mailboxes to be deleted immediately. Take a look at the code yourself, and search for the magic string "Localdelete". It bypasses delayed delete. Bron. From Hagedorn at uni-koeln.de Wed Sep 23 07:52:36 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Wed, 23 Sep 2009 13:52:36 +0200 Subject: delete_mode: delayed not always working? In-Reply-To: <20090923113255.GA6199@brong.net> References: <6396F9955E89A3D753A521BD@tyrion.rrz.uni-koeln.de> <20090923113255.GA6199@brong.net> Message-ID: <9E9F6C922C1E7E8A4DA5DA14@tyrion.rrz.uni-koeln.de> Thanks for your reply! --On 23. September 2009 21:32:55 +1000 Bron Gondwana wrote: >> The output of "ctl_mboxlist -d | grep DELETED" didn't include it, >> either. Fortunately I had a flashcopy from which I could restore the >> mailbox, but still I wonder what happened. Before I look at the code >> myself, I wanted to ask if someone knows offhand if there are known >> circumstances that cause mailboxes to be deleted immediately. > > Take a look at the code yourself, and search for the magic string > "Localdelete". It bypasses delayed delete. Localdelete seems to be an actual (private) command that a client can issue, right? I think it's completely impossible that the mailbox was deleted using this command ... -- .:.Sebastian Hagedorn - RZKR-R1 (Geb?ude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universit?t zu K?ln / Cologne University - ? +49-221-478-5587.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pkcs7-signature Size: 5292 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090923/c6bda19a/attachment.bin From simon.matter at invoca.ch Wed Sep 23 08:00:49 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Wed, 23 Sep 2009 14:00:49 +0200 Subject: delete_mode: delayed not always working? In-Reply-To: <9E9F6C922C1E7E8A4DA5DA14@tyrion.rrz.uni-koeln.de> References: <6396F9955E89A3D753A521BD@tyrion.rrz.uni-koeln.de> <20090923113255.GA6199@brong.net> <9E9F6C922C1E7E8A4DA5DA14@tyrion.rrz.uni-koeln.de> Message-ID: > Thanks for your reply! > > --On 23. September 2009 21:32:55 +1000 Bron Gondwana > wrote: > >>> The output of "ctl_mboxlist -d | grep DELETED" didn't include it, >>> either. Fortunately I had a flashcopy from which I could restore the >>> mailbox, but still I wonder what happened. Before I look at the code >>> myself, I wanted to ask if someone knows offhand if there are known >>> circumstances that cause mailboxes to be deleted immediately. >> >> Take a look at the code yourself, and search for the magic string >> "Localdelete". It bypasses delayed delete. > > Localdelete seems to be an actual (private) command that a client can > issue, right? I think it's completely impossible that the mailbox was > deleted using this command ... Could the user in question be an admin or a proxyadmin? Simon From Hagedorn at uni-koeln.de Wed Sep 23 08:09:35 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Wed, 23 Sep 2009 14:09:35 +0200 Subject: delete_mode: delayed not always working? In-Reply-To: References: <6396F9955E89A3D753A521BD@tyrion.rrz.uni-koeln.de> <20090923113255.GA6199@brong.net> <9E9F6C922C1E7E8A4DA5DA14@tyrion.rrz.uni-koeln.de> Message-ID: <14C3BA863F9E91663E68F28D@tyrion.rrz.uni-koeln.de> --On 23. September 2009 14:00:49 +0200 Simon Matter wrote: >> Localdelete seems to be an actual (private) command that a client can >> issue, right? I think it's completely impossible that the mailbox was >> deleted using this command ... > > Could the user in question be an admin or a proxyadmin? No. I know for a fact that he wasn't. -- .:.Sebastian Hagedorn - RZKR-R1 (Geb?ude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universit?t zu K?ln / Cologne University - ? +49-221-478-5587.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pkcs7-signature Size: 5292 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090923/1cea5be2/attachment-0001.bin From Eric.Luyten at vub.ac.be Wed Sep 23 10:19:19 2009 From: Eric.Luyten at vub.ac.be (Eric Luyten) Date: Wed, 23 Sep 2009 16:19:19 +0200 (CEST) Subject: MIgrating from Cyrus 2.2 to 2.3 : extra numerical field in mailboxes DB Message-ID: <64507.134.184.15.103.1253715559.squirrel@nuts.vub.ac.be> Folks, I copied all data from a (Cyrus 2.2.13 / Solaris 9) production server to a new (Cyrus 2.3.15 / Solaris 10) server and am now trying to fire the new server up. I recreated the mailboxes using 'ctl_mboxlist -d' on the old server and 'ctl_mboxlist -u' on the new one, which returned no errors, but when I subsequently start Cyrus, he insists on removing about one quarter of all mailboxes (about 100,000 out of slightly over 400,000) while logging lines could not remove reserved mailbox '.....': Unknown/invalid partition or removed reserved mailbox '.....' depending on the partition the mailboxes are located in (there are 9 partitions on our server, only partitions numbered 2, 3, 6 and 7 pose problems). When dumping the resulting mailboxes DB on the Cyrus 2.3.15 system I noticed (for newly created mailboxes) an extra numerical field in position 2 (position 3 appears to be the partition number). Where is it used for ("reservations" ? Murder ?) and what is the best way of moving from the old to the new format ? Thx, Eric Luyten, Brussels Free University Computing Centre. From wes at umich.edu Wed Sep 23 11:05:57 2009 From: wes at umich.edu (Wesley Craig) Date: Wed, 23 Sep 2009 11:05:57 -0400 Subject: MIgrating from Cyrus 2.2 to 2.3 : extra numerical field in mailboxes DB In-Reply-To: <64507.134.184.15.103.1253715559.squirrel@nuts.vub.ac.be> References: <64507.134.184.15.103.1253715559.squirrel@nuts.vub.ac.be> Message-ID: <383F22FF-110D-4BDF-A223-855E00D810BD@umich.edu> On 23 Sep 2009, at 10:19, Eric Luyten wrote: > When dumping the resulting mailboxes DB on the Cyrus 2.3.15 system > I noticed (for newly created mailboxes) an extra numerical field in > position 2 (position 3 appears to be the partition number). > > Where is it used for ("reservations" ? Murder ?) and what is the best > way of moving from the old to the new format ? The new field is flags. You should massage the old data into the new format, i.e., add that flag in. I seem to recall that the backward compatibility in the new ctl_mboxlist -u code isn't prepared for the case where partitions names may be confused with flags. :wes From Eric.Luyten at vub.ac.be Wed Sep 23 12:38:40 2009 From: Eric.Luyten at vub.ac.be (Eric Luyten) Date: Wed, 23 Sep 2009 18:38:40 +0200 (CEST) Subject: MIgrating from Cyrus 2.2 to 2.3 : extra numerical field in mailboxes DB In-Reply-To: <383F22FF-110D-4BDF-A223-855E00D810BD@umich.edu> References: <64507.134.184.15.103.1253715559.squirrel@nuts.vub.ac.be> <383F22FF-110D-4BDF-A223-855E00D810BD@umich.edu> Message-ID: <65447.134.184.15.103.1253723920.squirrel@nuts.vub.ac.be> On Wed, September 23, 2009 5:05 pm, Wesley Craig wrote: > On 23 Sep 2009, at 10:19, Eric Luyten wrote: > >> When dumping the resulting mailboxes DB on the Cyrus 2.3.15 system >> I noticed (for newly created mailboxes) an extra numerical field in >> position 2 (position 3 appears to be the partition number). >> >> Where is it used for ("reservations" ? Murder ?) and what is the best >> way of moving from the old to the new format ? > > The new field is flags. You should massage the old data into the new > format, i.e., add that flag in. I seem to recall that the backward > compatibility in the new ctl_mboxlist -u code isn't prepared for the case > where partitions names may be confused with flags. Replacing the "partition-number" field with "0partition-number" produces the desired result after a 'ctl_mboxlist -u' operation. Thx, Wes. Eric. From mercier.frederic at iname.com Wed Sep 23 15:05:06 2009 From: mercier.frederic at iname.com (=?ISO-8859-1?Q?Fr=E9d=E9ric_MERCIER?=) Date: Wed, 23 Sep 2009 21:05:06 +0200 Subject: INBOX Prefix problem with sasldb authentication Message-ID: <4ABA7162.6060901@iname.com> Hi, I have actually a cyrus IMAP server which work fine from a long time with an saslauthd and pam. I want to switch from salsauthd to salsdb2 to be able to use a secure authentication (DIGEST-MD5 and CRAM-MD5). My new configuration work fine for authentication, but with the sasldb2 one, I don't have the INBOX prefix. For example : Authentication with saslauthd : myserver:~# telnet localhost 993 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID AUTH=PLAIN AUTH=CRAM-MD5 AUTH=DIGEST-MD5 SASL-IR COMPRESS=DEFLATE] myserver.mydomain.net Cyrus IMAP v2.3.15 server ready . login test mypassword . OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID LOGINDISABLED COMPRESS=DEFLATE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE CATENATE CONDSTORE SCAN IDLE URLAUTH] User logged in . list "*" "*" * LIST (\HasChildren) "." "INBOX" * LIST (\HasChildren) "." "INBOX.Administratif" etc ... Authentication with sasldb2 : myserver:~# telnet localhost 993 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID AUTH=PLAIN AUTH=CRAM-MD5 AUTH=DIGEST-MD5 SASL-IR COMPRESS=DEFLATE] myserver.mydomain.net Cyrus IMAP v2.3.15 server ready . login test at mydomain.net mypassword . OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID LOGINDISABLED COMPRESS=DEFLATE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE CATENATE CONDSTORE SCAN IDLE URLAUTH] User logged in . list "*" "*" * LIST (\HasChildren) "." "user.test" * LIST (\HasChildren) "." "user.test.Administratif" etc ... user.test is not set as INBOX Prefix. The result is that my MUA is not able to find my mails ... Anyone have an idea ? Regards, Frederic. From morgan at orst.edu Wed Sep 23 15:11:43 2009 From: morgan at orst.edu (Andrew Morgan) Date: Wed, 23 Sep 2009 12:11:43 -0700 (PDT) Subject: INBOX Prefix problem with sasldb authentication In-Reply-To: <4ABA7162.6060901@iname.com> References: <4ABA7162.6060901@iname.com> Message-ID: On Wed, 23 Sep 2009, Fr?d?ric MERCIER wrote: > Hi, > > I have actually a cyrus IMAP server which work fine from a long time > with an saslauthd and pam. > > I want to switch from salsauthd to salsdb2 to be able to use a secure > authentication (DIGEST-MD5 and CRAM-MD5). > > My new configuration work fine for authentication, but with the sasldb2 > one, I don't have the INBOX prefix. > > For example : > > Authentication with saslauthd : > > myserver:~# telnet localhost 993 > Trying 127.0.0.1... > Connected to localhost. > Escape character is '^]'. > * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID AUTH=PLAIN AUTH=CRAM-MD5 > AUTH=DIGEST-MD5 SASL-IR COMPRESS=DEFLATE] myserver.mydomain.net Cyrus > IMAP v2.3.15 server ready > . login test mypassword > . OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID LOGINDISABLED > COMPRESS=DEFLATE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE > UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT > SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE > CATENATE CONDSTORE SCAN IDLE URLAUTH] User logged in > . list "*" "*" > * LIST (\HasChildren) "." "INBOX" > * LIST (\HasChildren) "." "INBOX.Administratif" > etc ... > > > Authentication with sasldb2 : > > myserver:~# telnet localhost 993 > Trying 127.0.0.1... > Connected to localhost. > Escape character is '^]'. > * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID AUTH=PLAIN AUTH=CRAM-MD5 > AUTH=DIGEST-MD5 SASL-IR COMPRESS=DEFLATE] myserver.mydomain.net Cyrus > IMAP v2.3.15 server ready > . login test at mydomain.net mypassword > . OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID LOGINDISABLED > COMPRESS=DEFLATE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE > UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT > SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE > CATENATE CONDSTORE SCAN IDLE URLAUTH] User logged in > . list "*" "*" > * LIST (\HasChildren) "." "user.test" > * LIST (\HasChildren) "." "user.test.Administratif" > etc ... > > user.test is not set as INBOX Prefix. The result is that my MUA is not > able to find my mails ... It looks like test at mydomain.net is a cyrus admin, so it sees the internal namespace instead of the client namespace. Andy From mercier.frederic at iname.com Wed Sep 23 15:30:35 2009 From: mercier.frederic at iname.com (=?ISO-8859-15?Q?Fr=E9d=E9ric_MERCIER?=) Date: Wed, 23 Sep 2009 21:30:35 +0200 Subject: INBOX Prefix problem with sasldb authentication In-Reply-To: References: <4ABA7162.6060901@iname.com> Message-ID: <4ABA775B.2060007@iname.com> Andrew Morgan a ?crit : > On Wed, 23 Sep 2009, Fr?d?ric MERCIER wrote: > >> Hi, >> >> I have actually a cyrus IMAP server which work fine from a long time >> with an saslauthd and pam. >> >> I want to switch from salsauthd to salsdb2 to be able to use a secure >> authentication (DIGEST-MD5 and CRAM-MD5). >> >> My new configuration work fine for authentication, but with the sasldb2 >> one, I don't have the INBOX prefix. >> >> For example : >> >> Authentication with saslauthd : >> >> myserver:~# telnet localhost 993 >> Trying 127.0.0.1... >> Connected to localhost. >> Escape character is '^]'. >> * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID AUTH=PLAIN AUTH=CRAM-MD5 >> AUTH=DIGEST-MD5 SASL-IR COMPRESS=DEFLATE] myserver.mydomain.net Cyrus >> IMAP v2.3.15 server ready >> . login test mypassword >> . OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID LOGINDISABLED >> COMPRESS=DEFLATE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE >> UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT >> SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE >> CATENATE CONDSTORE SCAN IDLE URLAUTH] User logged in >> . list "*" "*" >> * LIST (\HasChildren) "." "INBOX" >> * LIST (\HasChildren) "." "INBOX.Administratif" >> etc ... >> >> >> Authentication with sasldb2 : >> >> myserver:~# telnet localhost 993 >> Trying 127.0.0.1... >> Connected to localhost. >> Escape character is '^]'. >> * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID AUTH=PLAIN AUTH=CRAM-MD5 >> AUTH=DIGEST-MD5 SASL-IR COMPRESS=DEFLATE] myserver.mydomain.net Cyrus >> IMAP v2.3.15 server ready >> . login test at mydomain.net mypassword >> . OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID LOGINDISABLED >> COMPRESS=DEFLATE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE >> UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT >> SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE >> CATENATE CONDSTORE SCAN IDLE URLAUTH] User logged in >> . list "*" "*" >> * LIST (\HasChildren) "." "user.test" >> * LIST (\HasChildren) "." "user.test.Administratif" >> etc ... >> >> user.test is not set as INBOX Prefix. The result is that my MUA is not >> able to find my mails ... > > It looks like test at mydomain.net is a cyrus admin, so it sees the > internal namespace instead of the client namespace. > > Andy In imapd.conf and impad-local.conf I have only cyrus or cyrus at mydomain.net behind the admins: option. Frederic. From dwhite at olp.net Wed Sep 23 15:48:07 2009 From: dwhite at olp.net (Dan White) Date: Wed, 23 Sep 2009 14:48:07 -0500 Subject: INBOX Prefix problem with sasldb authentication In-Reply-To: <4ABA775B.2060007@iname.com> References: <4ABA7162.6060901@iname.com> <4ABA775B.2060007@iname.com> Message-ID: <20090923194807.GC8729@dan.olp.net> On 23/09/09?21:30?+0200, Fr?d?ric MERCIER wrote: >>> Authentication with sasldb2 : >>> >>> myserver:~# telnet localhost 993 >>> Trying 127.0.0.1... >>> Connected to localhost. >>> Escape character is '^]'. >>> * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID AUTH=PLAIN AUTH=CRAM-MD5 >>> AUTH=DIGEST-MD5 SASL-IR COMPRESS=DEFLATE] myserver.mydomain.net Cyrus >>> IMAP v2.3.15 server ready >>> . login test at mydomain.net mypassword >>> . OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID LOGINDISABLED >>> COMPRESS=DEFLATE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE >>> UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT >>> SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE >>> CATENATE CONDSTORE SCAN IDLE URLAUTH] User logged in >>> . list "*" "*" >>> * LIST (\HasChildren) "." "user.test" >>> * LIST (\HasChildren) "." "user.test.Administratif" >>> etc ... >>> >>> user.test is not set as INBOX Prefix. The result is that my MUA is not >>> able to find my mails ... The 'user' prefix is displayed for mailboxes that the authenticated user has ACL access to, but is not perceived to be the user's own mailboxes. Or in other words, if you were to authenticate as jsmith, and attempt to view the mailboxes for msmith, you will get the 'user' prefix. cyrus imapd believes that test at mydomain.net and test are different users. You can either create a mailbox for test at mydomain.net, or configure your virt domain options (defaultdomain). -- Dan White From rvh at abcreek.wisequackranch.com Wed Sep 23 17:00:20 2009 From: rvh at abcreek.wisequackranch.com (rvh) Date: Wed, 23 Sep 2009 14:00:20 -0700 Subject: cyrus stopped delivery Message-ID: <20090923210020.GA1314@abcreek.wisequackranch.com> Hi, The cyrus server was shutdown over night. After restarting this morning it won't accept mail for clients. Everything I've tried has failed. I am on cyrus 2.1. The errors I'm seeing now are: DBERROR: error exiting application: DB_RUNRECOVERY: Fatal error, run database recovery and unable to create imap listener socket: Address family not supported by protocol This is a production server that has been in service for quite a while without change. Suggestions greatly appreciated. bob rvh at wisequackranch.com From dwhite at olp.net Wed Sep 23 17:34:01 2009 From: dwhite at olp.net (Dan White) Date: Wed, 23 Sep 2009 16:34:01 -0500 Subject: cyrus stopped delivery In-Reply-To: <20090923210020.GA1314@abcreek.wisequackranch.com> References: <20090923210020.GA1314@abcreek.wisequackranch.com> Message-ID: <20090923213400.GD8729@dan.olp.net> On 23/09/09?14:00?-0700, rvh wrote: >Hi, The cyrus server was shutdown over night. After restarting >this morning it won't accept mail for clients. Everything I've >tried has failed. > >I am on cyrus 2.1. > >The errors I'm seeing now are: >DBERROR: error exiting application: DB_RUNRECOVERY: Fatal error, run database recovery >and >unable to create imap listener socket: Address family not supported by protocol > >This is a production server that has been in service for quite a while >without change. > >Suggestions greatly appreciated. Can you tell from the logs which database it's giving the DBERROR for? Some databases can safely be removed. Can you include your /etc/cyrus.conf config? -- Dan White From dwhite at olp.net Wed Sep 23 18:27:16 2009 From: dwhite at olp.net (Dan White) Date: Wed, 23 Sep 2009 17:27:16 -0500 Subject: cyrus stopped delivery In-Reply-To: <20090923215639.GA17122@abcreek.wisequackranch.com> References: <20090923210020.GA1314@abcreek.wisequackranch.com> <20090923213400.GD8729@dan.olp.net> <20090923215639.GA17122@abcreek.wisequackranch.com> Message-ID: <20090923222716.GF8729@dan.olp.net> I'm CCing the list. On 23/09/09?14:56?-0700, rvh wrote: > Over the last several restarts of cyrus the database error has > not come up so that might have been resolved. Now it's just the: > unable to create imap listener socket: Address family not supported by protocol > I'll include the cyrus.conf file below. > > SERVICES { > # --- Normal cyrus spool, or Murder backends --- > # add or remove based on preferences > imap cmd="imapd -U 30" listen="imap" prefork=0 maxchild=100 > #imaps cmd="imapd -s -U 30" listen="imaps" prefork=0 maxchild=100 > pop3 cmd="pop3d -U 30" listen="pop3" prefork=0 maxchild=50 > #pop3s cmd="pop3d -s -U 30" listen="pop3s" prefork=0 maxchild=50 > > # At least one form of LMTP is required for delivery > # (you must keep the Unix socket name in sync with imap.conf) > lmtp cmd="lmtpd -a" listen="localhost:lmtp" prefork=0 maxchild=20 > # lmtpunix cmd="lmtpd" listen="/var/run/cyrus/socket/lmtp" prefork=0 maxchild=20 > # ---------------------------------------------- > > # useful if you need to give users remote access to sieve > # by default, we limit this to localhost in Debian > sieve cmd="timsieved" listen="localhost:sieve" prefork=0 maxchild=100 > > # this one is needed for the notification services > notify cmd="notifyd" listen="/var/run/cyrus/socket/notify" proto="udp" prefork=1 It appears to be complaining about your imap line in SERVICES. It's saying that the address family is not supported. I assume that it's trying to listen on tcp6 socket. Try adding "proto=tcp4", like this: imap cmd="imapd -U 30" listen="imap" proto=tcp4 prefork=0 maxchild=100 If that works for imap, you'll need to replicate for pop3, lmpt and sieve. See the man page for cyrus.conf for more information. -- Dan White From mercier.frederic at iname.com Wed Sep 23 18:39:08 2009 From: mercier.frederic at iname.com (=?ISO-8859-1?Q?Fr=E9d=E9ric_MERCIER?=) Date: Thu, 24 Sep 2009 00:39:08 +0200 Subject: INBOX Prefix problem with sasldb authentication In-Reply-To: <20090923194807.GC8729@dan.olp.net> References: <4ABA7162.6060901@iname.com> <4ABA775B.2060007@iname.com> <20090923194807.GC8729@dan.olp.net> Message-ID: <4ABAA38C.5090403@iname.com> Dan White a ?crit : > On 23/09/09 21:30 +0200, Fr?d?ric MERCIER wrote: >>>> Authentication with sasldb2 : >>>> >>>> myserver:~# telnet localhost 993 >>>> Trying 127.0.0.1... >>>> Connected to localhost. >>>> Escape character is '^]'. >>>> * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID AUTH=PLAIN AUTH=CRAM-MD5 >>>> AUTH=DIGEST-MD5 SASL-IR COMPRESS=DEFLATE] myserver.mydomain.net Cyrus >>>> IMAP v2.3.15 server ready >>>> . login test at mydomain.net mypassword >>>> . OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID LOGINDISABLED >>>> COMPRESS=DEFLATE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE >>>> UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT >>>> SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE >>>> CATENATE CONDSTORE SCAN IDLE URLAUTH] User logged in >>>> . list "*" "*" >>>> * LIST (\HasChildren) "." "user.test" >>>> * LIST (\HasChildren) "." "user.test.Administratif" >>>> etc ... >>>> >>>> user.test is not set as INBOX Prefix. The result is that my MUA is not >>>> able to find my mails ... > > The 'user' prefix is displayed for mailboxes that the authenticated > user has > ACL access to, but is not perceived to be the user's own mailboxes. > > Or in other words, if you were to authenticate as jsmith, and attempt to > view the mailboxes for msmith, you will get the 'user' prefix. > > cyrus imapd believes that test at mydomain.net and test are different > users. You can either create a mailbox for test at mydomain.net, or > configure your > virt domain options (defaultdomain). > I'm ok with the diagnostic, but I don't understand why ... I have try to create the mailbox test at mydomain.net and to use the virtual domains without success. I need to read more docs on this subject to try another time. Whith the defaultdomain option set to mydomain.net, the authentication don't work anymore because it try to authenticate with test instead of test at mydomain.net From rosenbaumlm at ornl.gov Thu Sep 24 16:37:02 2009 From: rosenbaumlm at ornl.gov (Larry Rosenbaum) Date: Thu, 24 Sep 2009 16:37:02 -0400 Subject: v2.3.15 IMAP connection hangs In-Reply-To: <43C68785C2728049AF86B0ECB240A151117DC7D1D5@EXCHMB.ornl.gov> References: <43C68785C2728049AF86B0ECB240A151117DB6E4E7@EXCHMB.ornl.gov> <20090910221036.GA1177@brong.net> <43C68785C2728049AF86B0ECB240A151117DB6E653@EXCHMB.ornl.gov> <20090911225309.GA4536@brong.net> <43C68785C2728049AF86B0ECB240A151117DC7C86E@EXCHMB.ornl.gov> <20090916151213.GA25255@brong.net> <43C68785C2728049AF86B0ECB240A151117DC7CCB9@EXCHMB.ornl.gov> <8962e5139ced2fbdc48dff39999ae00a.squirrel@webmail.bi.corp.invoca.ch> <43C68785C2728049AF86B0ECB240A151117DC7CDB1@EXCHMB.ornl.gov> <71ea10c069b05fb055857a78141776a0.squirrel@webmail.bi.corp.invoca.ch> <20090917002159.GA5172@brong.net> <43C68785C2728049AF86B0ECB240A151117DC7D1D5@EXCHMB.ornl.gov> Message-ID: <000601ca3d56$c5c3b660$514b2320$@gov> OK I've finally figured it out. The connection only hangs if I use the following build procedure: Using either old or new zlib, start without the zlib patch configure make (at this point we get the deflateBound build error if we're using the old zlib) run the patch (patches prot.h and prot.c) make make install stop and restart Cyrus master connect to port 143 and issue a command Voila! the connection hangs. Note that there is no "make depend" in this sequence. If I follow the same sequence but I do a "make depend" after the configure step, then the server works properly. I think this means there's nothing wrong with the patch. So is it better to use the patch or the new zlib or both? Thanks, Larry > -----Original Message----- > From: info-cyrus-bounces+info-cyrus=ornl.gov at lists.andrew.cmu.edu > [mailto:info-cyrus-bounces+info-cyrus=ornl.gov at lists.andrew.cmu.edu] On > Behalf Of Rosenbaum, Larry M. > Sent: Thursday, September 17, 2009 10:13 AM > To: Bron Gondwana; Simon Matter > Cc: info-cyrus at lists.andrew.cmu.edu > Subject: RE: v2.3.15 IMAP connection hangs > > > From: Bron Gondwana [mailto:brong at fastmail.fm] > > Sent: Wednesday, September 16, 2009 8:22 PM > > To: Simon Matter > > Cc: Rosenbaum, Larry M.; Bron Gondwana; info- > cyrus at lists.andrew.cmu.edu > > Subject: Re: v2.3.15 IMAP connection hangs > > > > On Wed, Sep 16, 2009 at 07:04:05PM +0200, Simon Matter wrote: > > > >> From: Simon Matter [mailto:simon.matter at invoca.ch] > > > >>... > > > >> >> Long shot - but you might want to try it with the attached > > patch > > > >> >> applied. > > > >> >> It rewrites the zlib handling logic quite a lot, though it > > wasn't > > > >> >> originally > > > >> >> designed to deal with your problem! I'm going to push it to > > CVS > > > >> soon. > > > >> > > > > >> > I already have the patch installed because we have an old > > version of > > > >> Zlib. > > > >> > Could the patch be causing the problem? > > > >> > > > >> Hi, > > > >> > > > >> I have the same patch in my RPMs and tested it on a ancient > > OS/zlib and > > > >> didn't see any problems. Of course it doesn't mean much because > > I'm > > > >> running Linux. > > > >> Did you already try to build and run without zlib support? > > > > > > > > I hadn't thought of building it without zlib support. > > > > > > > > I have just installed a newer zlib, removed the patch, and > rebuilt > > Cyrus > > > > IMAP with the new zlib. Now it works! > > > > > > Good, but I'm sure Bron would still like to know why it didn't work > > before... > > > > I sure would! Given that we've been running that patch on top of CVS > > for > > a few months not (Linux with a newer zlib) without problems, I'd love > > to > > know what's wrong with the patch - particularly since I'm planning to > > add > > it to CVS if there's nothing else wrong with it. > > > > So it goes into an infinite loop on signals_poll, obviously. > > > > Can you please try building with the patch, but with the new zlib? > > With the new zlib and the patch, it hangs. > > Perhaps the problem with the patch is Solaris-specific. > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html From steffo76 at gmx.de Fri Sep 25 04:07:40 2009 From: steffo76 at gmx.de (steffo76 at gmx.de) Date: Fri, 25 Sep 2009 10:07:40 +0200 Subject: Compiling fails imap.xs:77 Message-ID: <20090925080740.304210@gmx.net> Hi list, I got trouble compiling cyrus-imapd-2.3.15. I get the following error: IMAP.xs: In function ?fatal?: IMAP.xs:77: error: format not a string literal and no format arguments make[2]: *** [IMAP.o] Error 1 I remember vaguely that there was a thread on this list which said that a compiler flag is to be set so this doesn't happen but I just can't find the thread. Could someone please tell me how to get past this error ? Thanks, Stephan -- Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 - sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser From Eric.Luyten at vub.ac.be Fri Sep 25 05:34:23 2009 From: Eric.Luyten at vub.ac.be (Eric Luyten) Date: Fri, 25 Sep 2009 11:34:23 +0200 (CEST) Subject: Moving Cyrus TLS component files (.CA, .crt, .key) between servers ? Message-ID: <58213.134.184.15.103.1253871263.squirrel@nuts.vub.ac.be> Colleagues, I am in the process of moving Cyrus service from a Cyrus 2.2.13 / Solaris9 system to a Cyrus 2.3.15 / Solaris10 system and all is running pretty well EXCEPT for authenticated IMAP/POP sessions. CSRs and key on the first server were generated using OpenSSL 0.9.7e while there is OpenSSL 0.9.8k on the second one (if that matters). I copied the three files referenced by tls_cert_file: tls_key_file: tls_ca_file: in /etc/imapd.conf to the new server but appear to have overlooked something, because my logfile now shows : Sep 24 16:42:44 mini pop3smvub[5569]: [ID 897861 local6.error] unable to get certificate from '/usr/local/ssl/official/popimap-vub.crt' Sep 24 16:42:44 mini pop3smvub[5569]: [ID 925991 local6.error] TLS server engine: cannot load cert/key data Sep 24 16:42:44 mini pop3smvub[5569]: [ID 528583 local6.error] [pop3d] error initializing TLS Sep 24 16:42:44 mini pop3smvub[5569]: [ID 637875 local6.error] Fatal error: tls_init() failed Can I still repair this situation or am I heading towards the generation of completely new CSRs and key ? Eric Luyten. From boutilpj at ednet.ns.ca Fri Sep 25 05:40:00 2009 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Fri, 25 Sep 2009 06:40:00 -0300 Subject: Moving Cyrus TLS component files (.CA, .crt, .key) between servers ? In-Reply-To: <58213.134.184.15.103.1253871263.squirrel@nuts.vub.ac.be> References: <58213.134.184.15.103.1253871263.squirrel@nuts.vub.ac.be> Message-ID: <7079_1253871623_n8P9eMlH012543_4ABC8FF0.3070004@ednet.ns.ca> Eric Luyten wrote: > Colleagues, > > I am in the process of moving Cyrus service from a Cyrus 2.2.13 / Solaris9 > system to a Cyrus 2.3.15 / Solaris10 system and all is running pretty well > > EXCEPT > > for authenticated IMAP/POP sessions. > > CSRs and key on the first server were generated using OpenSSL 0.9.7e while > there is OpenSSL 0.9.8k on the second one (if that matters). > > I copied the three files referenced by > tls_cert_file: > tls_key_file: > tls_ca_file: > in /etc/imapd.conf to the new server but appear to have overlooked something, > because my logfile now shows : > > Sep 24 16:42:44 mini pop3smvub[5569]: [ID 897861 local6.error] unable to get > certificate from '/usr/local/ssl/official/popimap-vub.crt' > Sep 24 16:42:44 mini pop3smvub[5569]: [ID 925991 local6.error] TLS server > engine: cannot load cert/key data > Sep 24 16:42:44 mini pop3smvub[5569]: [ID 528583 local6.error] [pop3d] error > initializing TLS > Sep 24 16:42:44 mini pop3smvub[5569]: [ID 637875 local6.error] Fatal error: > tls_init() failed > > > Can I still repair this situation or am I heading towards the generation of > completely new CSRs and key ? Does the cyrus user have permissions to read the cert and key? > > > Eric Luyten. > > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html From hans.moser at ofd-sth.niedersachsen.de Fri Sep 25 05:52:34 2009 From: hans.moser at ofd-sth.niedersachsen.de (Marc Patermann) Date: Fri, 25 Sep 2009 11:52:34 +0200 Subject: INBOX Prefix problem with sasldb authentication In-Reply-To: <4ABAA38C.5090403@iname.com> References: <4ABA7162.6060901@iname.com> <4ABA775B.2060007@iname.com> <20090923194807.GC8729@dan.olp.net> <4ABAA38C.5090403@iname.com> Message-ID: <4ABC92E2.5060002@ofd-sth.niedersachsen.de> Hi, Fr?d?ric MERCIER schrieb: > Whith the defaultdomain option set to mydomain.net, the authentication > don't work anymore because it try to authenticate with test instead of > test at mydomain.net You may have to check of that before. Otherwise all users have to change their authentication credentials (from user to user at domain). What does your sasldb entries look like? You have to check the realm. Marc From Eric.Luyten at vub.ac.be Fri Sep 25 09:49:11 2009 From: Eric.Luyten at vub.ac.be (Eric Luyten) Date: Fri, 25 Sep 2009 15:49:11 +0200 (CEST) Subject: Moving Cyrus TLS component files (.CA, .crt, .key) between servers ? In-Reply-To: <7079_1253871623_n8P9eMlH012543_4ABC8FF0.3070004@ednet.ns.ca> References: <58213.134.184.15.103.1253871263.squirrel@nuts.vub.ac.be> <7079_1253871623_n8P9eMlH012543_4ABC8FF0.3070004@ednet.ns.ca> Message-ID: <60306.134.184.15.103.1253886551.squirrel@nuts.vub.ac.be> On Fri, September 25, 2009 11:40 am, Patrick Boutilier wrote: > Eric Luyten wrote: > >> Colleagues, >> >> >> I am in the process of moving Cyrus service from a Cyrus 2.2.13 / Solaris9 >> system to a Cyrus 2.3.15 / Solaris10 system and all is running pretty well >> >> EXCEPT >> >> >> for authenticated IMAP/POP sessions. >> >> CSRs and key on the first server were generated using OpenSSL 0.9.7e while >> there is OpenSSL 0.9.8k on the second one (if that matters). >> >> I copied the three files referenced by >> tls_cert_file: >> tls_key_file: >> tls_ca_file: >> in /etc/imapd.conf to the new server but appear to have overlooked >> something, because my logfile now shows : > Does the cyrus user have permissions to read the cert and key? > Patrick, Head on ! File access bits on the files themselves and directories above were correct but *group ownership* of the three files (by which I make the 'cyrus' user read them) had not been preserved. Thx a lot for your hint (should've found out by myself but long working days are starting to take their toll :-) Eric Luyten. From m.roshany at gmail.com Sat Sep 26 08:21:13 2009 From: m.roshany at gmail.com (mozafar roshany) Date: Sat, 26 Sep 2009 15:51:13 +0330 Subject: Trouble with building Cyrus on amd64 Message-ID: <849aa58a0909260521w69e98a51ocd45c66cf0dee922@mail.gmail.com> Hello everyone. I want to apply this patch: http://www.pramberger.at/peter/software/patches/#cyrus ( cyrus-imapd-2.2.12-md5hash.patch) to increase the directory levels of mailboxes. But I've had many problems with compiling Cyrus. My system is: Debian Lenny, 2.6.26-2-amd64 x86_64. The last one I can't resolve. The following is the end of make: +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ... ### Making all in /root/src/cyrus/cyrus-imapd-2.2.12/perl make[1]: Entering directory `/root/src/cyrus/cyrus-imapd-2.2.12/perl' ### Making all in /root/src/cyrus/cyrus-imapd-2.2.12/perl/imap Writing Makefile for Cyrus::IMAP make[2]: Entering directory `/root/src/cyrus/cyrus-imapd-2.2.12/perl/imap' cc -c -I../../lib -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DVERSION=\"1.00\" -DXS_VERSION=\"1.00\" -fPIC "-I/usr/lib/perl/5.10/CORE" -DPERL_POLLUTE IMAP.c Running Mkbootstrap for Cyrus::IMAP () chmod 644 IMAP.bs rm -f blib/arch/auto/Cyrus/IMAP/IMAP.so cc -shared -O2 -g -L/usr/local/lib IMAP.o -o blib/arch/auto/Cyrus/IMAP/IMAP.so ../../lib/libcyrus.a ../../lib/libcyrus_min.a \ -ldb-4 -lsasl2 -lssl -lcrypto \ /usr/bin/ld: ../../lib/libcyrus.a(imclient.o): relocation R_X86_64_32S against `a local symbol' can not be used when making a shared object; recompile with -fPIC ../../lib/libcyrus.a: could not read symbols: Bad value collect2: ld returned 1 exit status make[2]: *** [blib/arch/auto/Cyrus/IMAP/IMAP.so] Error 1 make[2]: Leaving directory `/root/src/cyrus/cyrus-imapd-2.2.12/perl/imap' make[1]: *** [all] Error 1 make[1]: Leaving directory `/root/src/cyrus/cyrus-imapd-2.2.12/perl' make: *** [all] Error 1 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ It tells about adding -fPIC flag, but I've had no success with adding it to Makefiles. I should mention the number of my test users are more than that be handled by fulldirhash parameter. With this I will have several thousands of mailboxes at one [A-Z] directory. Thanks for any suggestions. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090926/554d9c7c/attachment.html From boutilpj at ednet.ns.ca Sat Sep 26 10:44:58 2009 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Sat, 26 Sep 2009 11:44:58 -0300 Subject: Trouble with building Cyrus on amd64 In-Reply-To: <849aa58a0909260521w69e98a51ocd45c66cf0dee922@mail.gmail.com> References: <849aa58a0909260521w69e98a51ocd45c66cf0dee922@mail.gmail.com> Message-ID: <29405_1253976301_n8QEj0kh028105_4ABE28EA.5040202@ednet.ns.ca> On 09/26/2009 09:21 AM, mozafar roshany wrote: > Hello everyone. > > I want to apply this patch: > http://www.pramberger.at/peter/software/patches/#cyrus > (cyrus-imapd-2.2.12-md5hash.patch > ) > to increase the directory levels of mailboxes. But I've had many > problems with compiling Cyrus. My system is: Debian Lenny, > 2.6.26-2-amd64 x86_64. The last one I can't resolve. The following is > the end of make: > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > ... > ### Making all in /root/src/cyrus/cyrus-imapd-2.2.12/perl > make[1]: Entering directory `/root/src/cyrus/cyrus-imapd-2.2.12/perl' > ### Making all in /root/src/cyrus/cyrus-imapd-2.2.12/perl/imap > Writing Makefile for Cyrus::IMAP > make[2]: Entering directory `/root/src/cyrus/cyrus-imapd-2.2.12/perl/imap' > cc -c -I../../lib -D_REENTRANT -D_GNU_SOURCE -DDEBIAN > -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE > -D_FILE_OFFSET_BITS=64 -O2 -g -DVERSION=\"1.00\" -DXS_VERSION=\"1.00\" > -fPIC "-I/usr/lib/perl/5.10/CORE" -DPERL_POLLUTE IMAP.c > Running Mkbootstrap for Cyrus::IMAP () > chmod 644 IMAP.bs > rm -f blib/arch/auto/Cyrus/IMAP/IMAP.so > cc -shared -O2 -g -L/usr/local/lib IMAP.o -o > blib/arch/auto/Cyrus/IMAP/IMAP.so ../../lib/libcyrus.a > ../../lib/libcyrus_min.a \ > -ldb-4 -lsasl2 -lssl -lcrypto \ > > /usr/bin/ld: ../../lib/libcyrus.a(imclient.o): relocation R_X86_64_32S > against `a local symbol' can not be used when making a shared object; > recompile with -fPIC > ../../lib/libcyrus.a: could not read symbols: Bad value > collect2: ld returned 1 exit status > make[2]: *** [blib/arch/auto/Cyrus/IMAP/IMAP.so] Error 1 > make[2]: Leaving directory `/root/src/cyrus/cyrus-imapd-2.2.12/perl/imap' > make[1]: *** [all] Error 1 > make[1]: Leaving directory `/root/src/cyrus/cyrus-imapd-2.2.12/perl' > make: *** [all] Error 1 > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > It tells about adding -fPIC flag, but I've had no success with adding it > to Makefiles. make all CFLAGS=-fPIC > > I should mention the number of my test users are more than that be > handled by fulldirhash parameter. With this I will have several > thousands of mailboxes at one [A-Z] directory. > > Thanks for any suggestions. > > > ------------------------------------------------------------------------ > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html From m.roshany at gmail.com Sun Sep 27 06:33:35 2009 From: m.roshany at gmail.com (mozafar roshany) Date: Sun, 27 Sep 2009 14:03:35 +0330 Subject: Trouble with building Cyrus on amd64 In-Reply-To: <4ABE28EA.5040202@ednet.ns.ca> References: <849aa58a0909260521w69e98a51ocd45c66cf0dee922@mail.gmail.com> <4ABE28EA.5040202@ednet.ns.ca> Message-ID: <849aa58a0909270333u43fb94edq369d48b7285e468d@mail.gmail.com> Thank you Patrick, it did worked. On Sat, Sep 26, 2009 at 6:14 PM, Patrick Boutilier wrote: > On 09/26/2009 09:21 AM, mozafar roshany wrote: > >> Hello everyone. >> >> I want to apply this patch: >> http://www.pramberger.at/peter/software/patches/#cyrus >> (cyrus-imapd-2.2.12-md5hash.patch >> < >> ftp://ftp.pramberger.at/software/patches/cyrus-imapd-2.2.12-md5hash.patch >> >) >> >> to increase the directory levels of mailboxes. But I've had many >> problems with compiling Cyrus. My system is: Debian Lenny, >> 2.6.26-2-amd64 x86_64. The last one I can't resolve. The following is >> the end of make: >> >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> ... >> ### Making all in /root/src/cyrus/cyrus-imapd-2.2.12/perl >> make[1]: Entering directory `/root/src/cyrus/cyrus-imapd-2.2.12/perl' >> ### Making all in /root/src/cyrus/cyrus-imapd-2.2.12/perl/imap >> Writing Makefile for Cyrus::IMAP >> make[2]: Entering directory `/root/src/cyrus/cyrus-imapd-2.2.12/perl/imap' >> cc -c -I../../lib -D_REENTRANT -D_GNU_SOURCE -DDEBIAN >> -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE >> -D_FILE_OFFSET_BITS=64 -O2 -g -DVERSION=\"1.00\" -DXS_VERSION=\"1.00\" >> -fPIC "-I/usr/lib/perl/5.10/CORE" -DPERL_POLLUTE IMAP.c >> Running Mkbootstrap for Cyrus::IMAP () >> chmod 644 IMAP.bs >> rm -f blib/arch/auto/Cyrus/IMAP/IMAP.so >> cc -shared -O2 -g -L/usr/local/lib IMAP.o -o >> blib/arch/auto/Cyrus/IMAP/IMAP.so ../../lib/libcyrus.a >> ../../lib/libcyrus_min.a \ >> -ldb-4 -lsasl2 -lssl -lcrypto \ >> >> /usr/bin/ld: ../../lib/libcyrus.a(imclient.o): relocation R_X86_64_32S >> against `a local symbol' can not be used when making a shared object; >> recompile with -fPIC >> ../../lib/libcyrus.a: could not read symbols: Bad value >> collect2: ld returned 1 exit status >> make[2]: *** [blib/arch/auto/Cyrus/IMAP/IMAP.so] Error 1 >> make[2]: Leaving directory `/root/src/cyrus/cyrus-imapd-2.2.12/perl/imap' >> make[1]: *** [all] Error 1 >> make[1]: Leaving directory `/root/src/cyrus/cyrus-imapd-2.2.12/perl' >> make: *** [all] Error 1 >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> It tells about adding -fPIC flag, but I've had no success with adding it >> to Makefiles. >> > > make all CFLAGS=-fPIC > > >> I should mention the number of my test users are more than that be >> handled by fulldirhash parameter. With this I will have several >> thousands of mailboxes at one [A-Z] directory. >> >> Thanks for any suggestions. >> >> >> ------------------------------------------------------------------------ >> >> ---- >> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ >> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki >> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090927/41689003/attachment.html From tolomelli.luca at gmail.com Sun Sep 27 11:56:00 2009 From: tolomelli.luca at gmail.com (Luca Tolomelli) Date: Sun, 27 Sep 2009 17:56:00 +0200 Subject: reconstruct problem Message-ID: <5f8ec0530909270856r648bc8e9xefe5cce2cb2fb55f@mail.gmail.com> I have this problem with the /usr/lib/cyrus-imapd/reconstruct -r -f user.username If I try to restore a mail that was saved on the cyrus server all work as intended. If I try to restore a mail that was saved on a NTFS (copying the mail from the NTSF server to the cyrus server on the username directory) no mail is shown (the owner of the restored mail is "cyrus"). What could it be? Thanks From brong at fastmail.fm Sun Sep 27 19:01:32 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Mon, 28 Sep 2009 09:01:32 +1000 Subject: reconstruct problem In-Reply-To: <5f8ec0530909270856r648bc8e9xefe5cce2cb2fb55f@mail.gmail.com> References: <5f8ec0530909270856r648bc8e9xefe5cce2cb2fb55f@mail.gmail.com> Message-ID: <20090927230132.GA2372@brong.net> On Sun, Sep 27, 2009 at 05:56:00PM +0200, Luca Tolomelli wrote: > I have this problem with the /usr/lib/cyrus-imapd/reconstruct -r -f > user.username > > If I try to restore a mail that was saved on the cyrus server all work > as intended. > > If I try to restore a mail that was saved on a NTFS (copying the mail > from the NTSF server to the cyrus server on the username directory) no > mail is shown (the owner of the restored mail is "cyrus"). > > What could it be? Ownership of the parent directory? Just a thought. That would cause the cyrus.index and cyrus.cache creation to fail. What's showing up in syslog? Reconstruct logs any errors it finds. Also, what version of cyrus-imapd? Bron ( not that the version makes much difference, probably ) From tolomelli.luca at gmail.com Mon Sep 28 03:06:02 2009 From: tolomelli.luca at gmail.com (Luca Tolomelli) Date: Mon, 28 Sep 2009 09:06:02 +0200 Subject: reconstruct problem In-Reply-To: <20090927230132.GA2372@brong.net> References: <5f8ec0530909270856r648bc8e9xefe5cce2cb2fb55f@mail.gmail.com> <20090927230132.GA2372@brong.net> Message-ID: <5f8ec0530909280006i595acca5va9aa983f91202ce8@mail.gmail.com> I've just found the problem: NTFS removes the "." (dot) character at the end of the file names. All the mail file names have a number follower by a "." (dot)...! On Mon, Sep 28, 2009 at 01:01, Bron Gondwana wrote: > On Sun, Sep 27, 2009 at 05:56:00PM +0200, Luca Tolomelli wrote: >> I have this problem with the /usr/lib/cyrus-imapd/reconstruct -r -f >> user.username >> >> If I try to restore a mail that was saved on the cyrus server all work >> as intended. >> >> If I try to restore a mail that was saved on a NTFS (copying the mail >> from the NTSF server to the cyrus server on the username directory) no >> mail is shown (the owner of the restored mail is "cyrus"). >> >> What could it be? > > Ownership of the parent directory? ?Just a thought. ?That would cause the > cyrus.index and cyrus.cache creation to fail. > > What's showing up in syslog? ?Reconstruct logs any errors it finds. > > Also, what version of cyrus-imapd? > > Bron ( not that the version makes much difference, probably ) > From lucaszc at gmail.com Mon Sep 28 11:45:36 2009 From: lucaszc at gmail.com (Lucas Zinato Carraro) Date: Mon, 28 Sep 2009 12:45:36 -0300 Subject: Implement Cyrus IMAPD in High Load Enviromment Message-ID: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> Hi, I am deploing cyrus-imapd in my organization: My organization need to have: - 75000 maiboxes - 10000 simultaneous connections (IMAP) - 3000 mailboxes with 1Gb and 65000 with 200Mb - 12 Tb for spool ( EMC Clarion Storage ) - 15 servers with Xeon 2.6ghz - 8gb RAM - GNU/Linux 2.6 for Operating System ( RedHat, Debian etc ... ) - ~ 150000 messages / day - Postfix as MTA I made several tests to choose filesystem. Cluster FS( gfs, ocfs ), ext3, ext4 , reiserfs , xfs and etc.. my choice ext3 with dir_index, noatime, nodiratime and partition my problem with Cyrus Murder. My doubts are: - Exist a recommended number of conections to a front end server ? ( a parameter in sysctl.conf in GNU/Linux ) - Exist a recommended size to a Backend server ( Ex: 1 Tb )? Cyrus IMAP works great with large mailboxes an billions of small files. Backup is my big problem. - Exist any solution better than make a SNAPSHOT of STORAGE and backup all file system (/dev/sdaX ) ? I know that performance is terrible with billions of small files, and make tests using tar . ( ex: a.tar.gz = /var/spool/imap/a/ ), this approach increase the performance with my LTO tapes. Anyone has another suggestion ? Regards, Lucas Zinato Carraro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090928/f8b880fd/attachment.html From vbfox at ucdavis.edu Mon Sep 28 11:59:43 2009 From: vbfox at ucdavis.edu (Vincent Fox) Date: Mon, 28 Sep 2009 08:59:43 -0700 Subject: Implement Cyrus IMAPD in High Load Enviromment In-Reply-To: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> References: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> Message-ID: <4AC0DD6F.9090309@ucdavis.edu> Lucas Zinato Carraro wrote: > > - Exist a recommended size to a Backend server ( Ex: 1 Tb )? > Hardware-wise your setup is probably overkill. Nothing wrong with that. Sizing of filesystems IMO should be based on your tolerance for long fsck during a disaster. I run ZFS which has none of that and don't want to ever see it again on a mail-spool. Linux journals IME reduce the probability of it but you will still find yourself looking at fsck prompt and having to decide: Y = hours of downtime while I make sure it's actuallly OK N = get it going, cross fingers. Most Linux admins don't turn on full data journalling anyhow quoting "performance reasons" they leave the default which is journalling metadata. So you don't really know how your data is doing until it goes kablooey and you do an fsck with the filesystem unmounted. I wouldn't go over 500 megs per FS until Linux has production BTRFS or something similar. In ZFS the backups are trivial. A script does a snapshot at 23:55 takes a few seconds to complete, then the backup is made of the most recent snapshot. We keep 14 days of snapshots in the pool almost all recovery operations are satisfied from that without hitting tape. The overhead of our snapshots increases storage about 50% but we are still FAR below max usage at only about 20% filled pools with LZJB compression in the meta dirs and gzip on the messages. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3250 bytes Desc: S/MIME Cryptographic Signature Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090928/49051d79/attachment.bin From cyruspy at gmail.com Mon Sep 28 12:03:22 2009 From: cyruspy at gmail.com (Ciro Iriarte) Date: Mon, 28 Sep 2009 12:03:22 -0400 Subject: Implement Cyrus IMAPD in High Load Enviromment In-Reply-To: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> References: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> Message-ID: 2009/9/28 Lucas Zinato Carraro : > Hi, I am deploing cyrus-imapd in my organization: > > My organization need to have: > > ?- 75000? maiboxes > ?- 10000 simultaneous connections (IMAP) > ?- 3000 mailboxes with 1Gb and 65000 with 200Mb > ?- 12 Tb for spool ( EMC Clarion Storage ) > ?- 15 servers with Xeon 2.6ghz - 8gb RAM > ?- GNU/Linux 2.6 for Operating System ( RedHat, Debian etc ... ) > ?- ~ 150000 messages / day > ?- Postfix as MTA > > I made several tests? to choose filesystem. > ?Cluster FS( gfs, ocfs ),? ext3, ext4 , reiserfs , xfs and etc.. I'm really interested in this part, do you have any test numbers to share?, I would like to use OCFS2 for Active/Active configuration, was it that bad?. Was ext3 faster than reiserfs?, I would though it should be the other way around. > my choice > > ext3? with ? dir_index, noatime, nodiratime > > and partition my? problem with Cyrus Murder. > > My doubts are: > > - Exist a recommended number of conections to a front end server ? > ( a parameter in sysctl.conf in GNU/Linux ) > > - Exist a recommended size to a Backend server ( Ex: 1 Tb )? > > Cyrus IMAP works great with large mailboxes an billions of small files. > Backup is my big problem. > > - Exist any solution better than make a SNAPSHOT of STORAGE and backup all > file system (/dev/sdaX ) ? It's the best approach. Depends on your backup tool, there are comercial tools that allow you to backup at the filesystem level (filesystem image) and restore at the file level. That would be a lot faster than backing up the files per se. > > I know that performance is? terrible with billions of small files, and make > tests using tar . (? ex: a.tar.gz = /var/spool/imap/a/ ), this approach > increase the performance with my LTO tapes. > > Anyone has another suggestion? ? > > > Regards, > Lucas Zinato Carraro > Regards, -- Ciro Iriarte http://cyruspy.wordpress.com -- From brong at fastmail.fm Mon Sep 28 18:08:31 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 29 Sep 2009 08:08:31 +1000 Subject: Implement Cyrus IMAPD in High Load Enviromment In-Reply-To: <4AC0DD6F.9090309@ucdavis.edu> References: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> <4AC0DD6F.9090309@ucdavis.edu> Message-ID: <20090928220831.GA7260@brong.net> On Mon, Sep 28, 2009 at 08:59:43AM -0700, Vincent Fox wrote: > Lucas Zinato Carraro wrote: > > > >- Exist a recommended size to a Backend server ( Ex: 1 Tb )? > > > Hardware-wise your setup is probably overkill. > Nothing wrong with that. Yeah, that's a fair few machines! Nice to have space for it. > Sizing of filesystems IMO should be based on your > tolerance for long fsck during a disaster. I run ZFS which > has none of that and don't want to ever see it again on > a mail-spool. Linux journals IME reduce the probability of it > but you will still find yourself looking at fsck prompt and > having to decide: > > Y = hours of downtime while I make sure it's actuallly OK > N = get it going, cross fingers. Yeah, that's painful. Thankfully with replication we don't have it too bad. Just run up a new replica and then blow away the old one (ordering! Nice to be able to recover _something_ if you lose the other side of the replica somehow during the resync...) > Most Linux admins don't turn on full data journalling anyhow > quoting "performance reasons" they leave the default which is > journalling metadata. So you don't really know how your data > is doing until it goes kablooey and you do an fsck with the > filesystem unmounted. I wouldn't go over 500 megs per FS > until Linux has production BTRFS or something similar. I assume you mean 500 gigs! We're switching from 300 to 500 on new filesystems because we have one business customer that's over 150Gb now and we want to keep all their users on the one partition for folder sharing. We don't do any murder though. We run reiserfs (rw,noatime,nodiratime,notail,data=ordered) > In ZFS the backups are trivial. A script does a > snapshot at 23:55 takes a few seconds to complete, then > the backup is made of the most recent snapshot. We keep > 14 days of snapshots in the pool almost all recovery operations > are satisfied from that without hitting tape. The overhead of > our snapshots increases storage about 50% but we are > still FAR below max usage at only about 20% filled pools with > LZJB compression in the meta dirs and gzip on the messages. Yeah - that sounds pretty nice. Our backups have a custom file streaming and locking daemon (it can fcntl lock all the meta files and then stream them together, to guarantee consistency) on each imap server. The backup server pulls a list of users to backup from the database and then forks I think 12 daemons at the moment, which grab users in batches of 50 on a single drive unit and processes them - meaning that we don't hammer any one drive unit too hard, but spread the load around pretty randomly. The backups are stored in a .tar.gz file (I think I've posted about the internal format before, it's very UUID/UniqueID centred, so it handles renames cheaply, and does single instance stores automatically because the files have the same sha1), and there's an associated sqlite database for easy lookup, but that can be re-created just by reading through the tar file. gzip is nice because you can concatenate multiple gzip files and the result is uncompressible with a single gzip read, just possibly less efficiently packed. Tar is nice because everything is in 512 byte blocks. We have a custom Perl module that can read and write tar files, and also modules that can read and write cyrus.index files. So far I've only bothered with read access to cyrus.header files, but it shouldn't be too hard to write them either! I really should productise this thing at some point! It's a very nice backup system, but it's quite hard-coded. In particular, I should rewrite backupcyrusd.pl as a C daemon that is managed by Cyrus, instead of something standalone. Bron. From vbfox at ucdavis.edu Mon Sep 28 18:33:44 2009 From: vbfox at ucdavis.edu (Vincent Fox) Date: Mon, 28 Sep 2009 15:33:44 -0700 Subject: Implement Cyrus IMAPD in High Load Enviromment In-Reply-To: <20090928220831.GA7260@brong.net> References: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> <4AC0DD6F.9090309@ucdavis.edu> <20090928220831.GA7260@brong.net> Message-ID: <4AC139C8.3070108@ucdavis.edu> Bron Gondwana wrote: > I assume you mean 500 gigs! We're switching from 300 to 500 on new > filesystems because we have one business customer that's over 150Gb > now and we want to keep all their users on the one partition for > folder sharing. We don't do any murder though. > > Oops yes. I meant 500 gigs. The potential downside of running an fsck on terabyte+ filesystems is not worth the risks IMO. The tremendous speed & efficiency of Cyrus is in it's small files and the indexes. However you have to keep that in mind when estimating not just backups and other daily/weekly items but more serious items. Really I've looked at fsck too many times in my life and don't ever want to again. Anyone who tells me "oh yes but journalling solved all that long ago...." will get an earful from me about how they haven't run a big enough setup with enough stress on it to SEE real problems. I have seen both journalled Linux and logged Solaris filesystem turn up with data corruption and ended up staring at that fsck prompt wondering how many hours until it's done..... The antiquated filesystems that 99% of admins tolerate and work with every day should be lumped under some kind of Geneva provision against torture. It's a mystery to me why it's not resolved years ago and why there isn't a big push for it from anyone. "It doesn't matter how fast it is, if it isn't CORRECT!" should be some kind of mantra for a production data center but it still seems majority of my colleagues talk same as in 1980s' about how if we turn off this or that safety feature we can make the filesystem faster. OK stepping off my soapbox now. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3250 bytes Desc: S/MIME Cryptographic Signature Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090928/d3b67694/attachment.bin From brong at fastmail.fm Tue Sep 29 01:42:38 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 29 Sep 2009 15:42:38 +1000 Subject: Implement Cyrus IMAPD in High Load Enviromment In-Reply-To: <4AC139C8.3070108@ucdavis.edu> References: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> <4AC0DD6F.9090309@ucdavis.edu> <20090928220831.GA7260@brong.net> <4AC139C8.3070108@ucdavis.edu> Message-ID: <20090929054238.GC15988@brong.net> On Mon, Sep 28, 2009 at 03:33:44PM -0700, Vincent Fox wrote: > Bron Gondwana wrote: > >I assume you mean 500 gigs! We're switching from 300 to 500 on new > >filesystems because we have one business customer that's over > >150Gb now and we want to keep all their users on the one partition > >for > >folder sharing. We don't do any murder though. > > > Oops yes. I meant 500 gigs. The potential downside of > running an fsck on terabyte+ filesystems is not worth > the risks IMO. The tremendous speed & efficiency of > Cyrus is in it's small files and the indexes. However you > have to keep that in mind when estimating not just backups > and other daily/weekly items but more serious items. For sure. > Really I've looked at fsck too many times in my life and > don't ever want to again. Anyone who tells me "oh yes but > journalling solved all that long ago...." will get an earful > from me about how they haven't run a big enough setup > with enough stress on it to SEE real problems. I have seen > both journalled Linux and logged Solaris filesystem turn up > with data corruption and ended up staring at that fsck > prompt wondering how many hours until it's done..... Yep. Which is why we treat filesystems as disposable :) There are multiple real-time replicated copies of anything we care about, so we can blow away a filesystem and just recreate it. Even after a successful fsck I might just decide it's cheaper to recreate it than run a full sha1 checking audit_slot on the contents! > The antiquated filesystems that 99% of admins tolerate and > work with every day should be lumped under some kind of > Geneva provision against torture. It's a mystery to me why > it's not resolved years ago and why there isn't a big push > for it from anyone. Patents I suspect, at least partially. > "It doesn't matter how fast it is, if it isn't CORRECT!" should > be some kind of mantra for a production data center but it > still seems majority of my colleagues talk same as in 1980s' > about how if we turn off this or that safety feature we can > make the filesystem faster. Everything's a tradeoff, hey. With enough checksums and replication, I'm willing to treat every layer as less than 100% reliable, because that's reality. I haven't heard too many horror stories of ZFS recently, but we certainly hit a bug where we needed a software update before we could replace a failed disk, because ZFS refused to consider anything plugged into the same controller again, even after a reboot. That was odd. > OK stepping off my soapbox now. It's an interesting one. For real reliability, I want to have multiple replication target supported cleanly. It's not even that hard. Basically you would chain sync_client instances, such that there was an initial task that just reads $conf/sync/log and appends the contents to both $conf/sync/stream1/log and $conf/sync/stream2/log, then a separate sync_client instance that operates in each of $conf/sync/stream1 and $conf/sync/stream2, replicating to separate backends. This would involve minimal code changes I suspect, and allow a replica to be offline while the other two are up-to-date, and still know what needed syncing when you turned it back on! Then we'd be able to bring up a new replica BEFORE removing the old one. It's like RAID1 with three disks :) Add a new one, remove the old. Always 2 up-to-date copies. Then add management tools to make that easy to start and stop! It's an ongoing task to improve reliability. I actually wonder if it's possible to have multiple Cyrus instances running in a mesh. Each one running a sync_server and with sync_client instances running on every other one. In THEORY so long as you only wrote to one at any one time you could read from any of them, or even if you only had connections for a single user happening to one at any one time you'd be OK. You could hash users amongst them to balance the load. Then - well, I already have checksums coded into index files, just waiting code review from Ken to push that upstream. Along with sha1s, that's 99% of the data covered by checksums. Flat files (quota and the like) I don't think are viable, but it might be possible to add checksums to skiplist as well, at the expense of a format change. Not sure about BDB. I'm not a giant fan of it anyway - at least how it's being used in Cyrus. All our DBs are skiplist now, and we're pretty happy with it :) Bron. From simon.matter at invoca.ch Tue Sep 29 03:45:53 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Tue, 29 Sep 2009 09:45:53 +0200 Subject: Implement Cyrus IMAPD in High Load Enviromment In-Reply-To: <4AC139C8.3070108@ucdavis.edu> References: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> <4AC0DD6F.9090309@ucdavis.edu> <20090928220831.GA7260@brong.net> <4AC139C8.3070108@ucdavis.edu> Message-ID: > Bron Gondwana wrote: >> I assume you mean 500 gigs! We're switching from 300 to 500 on new >> filesystems because we have one business customer that's over 150Gb >> now and we want to keep all their users on the one partition for >> folder sharing. We don't do any murder though. >> >> > Oops yes. I meant 500 gigs. The potential downside of > running an fsck on terabyte+ filesystems is not worth > the risks IMO. The tremendous speed & efficiency of > Cyrus is in it's small files and the indexes. However you > have to keep that in mind when estimating not just backups > and other daily/weekly items but more serious items. > > Really I've looked at fsck too many times in my life and > don't ever want to again. Anyone who tells me "oh yes but > journalling solved all that long ago...." will get an earful > from me about how they haven't run a big enough setup > with enough stress on it to SEE real problems. I have seen > both journalled Linux and logged Solaris filesystem turn up > with data corruption and ended up staring at that fsck > prompt wondering how many hours until it's done..... > > The antiquated filesystems that 99% of admins tolerate and > work with every day should be lumped under some kind of > Geneva provision against torture. It's a mystery to me why > it's not resolved years ago and why there isn't a big push > for it from anyone. > > "It doesn't matter how fast it is, if it isn't CORRECT!" should > be some kind of mantra for a production data center but it > still seems majority of my colleagues talk same as in 1980s' > about how if we turn off this or that safety feature we can > make the filesystem faster. > > OK stepping off my soapbox now. What you said is not wrong, but it matters how you look at it. It's true, looking at an fsck prompt is something very boring and it can make one very nervous. But after many years of Unix and Linux experience it doesn't look _so_ bad considering the issues people have in the non *X world. I have seen much less fsck in the last 10 years than before. I have seen it with all kind of traditional Unix filesystems on HP/UX, Solaris, AIX and SCO Unix, I have seen it on Linux with ext2, then with ext3 and reiserfs. Linux with XFS has shown almost no issues (despite having bad support from my main distributor RedHat and some bad behavior in earlier releases). What is really bad is if you end up with a broken filesystem which can not be fixed anymore, corrupt, dead, disaster. The bad news is, it seems to be possible with every filesystem, more or less. It's because software can have bugs and so it can do something wrong. Features like checksumming data and metadata are nice but don't prevent from the worst. I really hope we could see something like ZFS on other platforms (I know there are already implementations on *BSD but I'm not sure they are as stable as on Solaris). As it is now if you choose to use ZFS your are limited to Solaris. And from my point of view it still greatly lacks features in other areas. That said, you always have to choose what is important for you and find a way to work around the disadvantages of choosen solution. What I'm really wondering, what filesystem disasters have others seen? How many times was it fsck only, how many times was it really broken. I'm not talking about laptop and desktop users but about production systems in a production environment with production class hardware and operating systems. Would be really interesting to get some of the good and bad stories even if not directly related to Cyrus-IMAP. Regards, Simon From mail_list at computer-gott.de Tue Sep 29 04:48:17 2009 From: mail_list at computer-gott.de (Georg Stich) Date: Tue, 29 Sep 2009 10:48:17 +0200 Subject: Cyrus-imapd 2.3.14 - timsieved - Fedora 11-problem Message-ID: <4AC1C9D1.6060809@computer-gott.de> Hello again ! I tried to patch all files described in the Bug Tracker Horde and elsewhere. It doesnt. work. I got "Not in Authetication State" May be i am a little confused about all, hmm . Could anybody post a (simple) workaround or howto get running ingo 1.2.1 or 1.22 with cyrus-imapd 2.3.15-1.fc11 for fedora. I read the posting http://lists.roundcube.net/mail-archive/dev/2009-09/0000141.html about the broken Sieve implemetation. But i didn't found a workaround, also not for F11 Thank you for your response! Georg Stich From selsky at columbia.edu Tue Sep 29 05:15:26 2009 From: selsky at columbia.edu (Matt Selsky) Date: Tue, 29 Sep 2009 05:15:26 -0400 Subject: Cyrus-imapd 2.3.14 - timsieved - Fedora 11-problem In-Reply-To: <4AC1C9D1.6060809@computer-gott.de> References: <4AC1C9D1.6060809@computer-gott.de> Message-ID: On Sep 29, 2009, at 4:48 AM, Georg Stich wrote: > I tried to patch all files described in the Bug Tracker Horde and > elsewhere. It doesnt. work. I got "Not in Authetication State" May > be i > am a little confused about all, hmm . > Could anybody post a (simple) workaround or howto get running ingo > 1.2.1 > or 1.22 with cyrus-imapd 2.3.15-1.fc11 for fedora. > > I read the posting > http://lists.roundcube.net/mail-archive/dev/2009-09/0000141.html > about the broken Sieve implemetation. But i didn't found a workaround, > also not for F11 What version of Net_Sieve do you have? For your version of Cyrus, you either need Net_Sieve 1.1.7 or you need to apply the patch mentioned at http://pear.php.net/bugs/bug.php?id=14205 -- Matt From mail_list at computer-gott.de Tue Sep 29 05:39:12 2009 From: mail_list at computer-gott.de (Georg Stich) Date: Tue, 29 Sep 2009 11:39:12 +0200 Subject: Cyrus-imapd 2.3.14 - timsieved - Fedora 11-problem In-Reply-To: References: <4AC1C9D1.6060809@computer-gott.de> Message-ID: <4AC1D5C0.6020200@computer-gott.de> Matt Selsky wrote: > On Sep 29, 2009, at 4:48 AM, Georg Stich wrote: > >> I tried to patch all files described in the Bug Tracker Horde and >> elsewhere. It doesnt. work. I got "Not in Authetication State" May be i >> am a little confused about all, hmm . >> Could anybody post a (simple) workaround or howto get running ingo 1.2.1 >> or 1.22 with cyrus-imapd 2.3.15-1.fc11 for fedora. >> >> I read the posting >> http://lists.roundcube.net/mail-archive/dev/2009-09/0000141.html >> about the broken Sieve implemetation. But i didn't found a workaround, >> also not for F11 > > What version of Net_Sieve do you have? For your version of Cyrus, you > either need Net_Sieve 1.1.7 or you need to apply the patch mentioned > at http://pear.php.net/bugs/bug.php?id=14205 > > Hi! Thanks for reply. I use now a fresh installation wir php-pear-Net-Sieve 1.1.7 . Do i need install TLS /SSL Support on this System ( in postfix or somewhere else ) Greeting Georg From brong at fastmail.fm Tue Sep 29 06:39:43 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 29 Sep 2009 20:39:43 +1000 Subject: Implement Cyrus IMAPD in High Load Enviromment In-Reply-To: References: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> <4AC0DD6F.9090309@ucdavis.edu> <20090928220831.GA7260@brong.net> <4AC139C8.3070108@ucdavis.edu> Message-ID: <20090929103943.GA3570@brong.net> On Tue, Sep 29, 2009 at 09:45:53AM +0200, Simon Matter wrote: > What I'm really wondering, what filesystem disasters have others seen? How > many times was it fsck only, how many times was it really broken. I'm not > talking about laptop and desktop users but about production systems in a > production environment with production class hardware and operating > systems. Well - we lost three drives in a 3TB RAID6 partition within 24 hours. That was sad. The third drive wasn't totally lost, just throwing enough errors that we remounted the whole thing readonly and kept it around to supplement the backup restores. > Would be really interesting to get some of the good and bad stories even > if not directly related to Cyrus-IMAP. Honestly, the biggest thing is - I've got a unit I've just switched drives in. It has 4 x 300GB 15kRPM drives in two RAID1 sets, and 8 x 2TB drives in two RAID5 sets. That's 12TB of data space plus a bit of room for meta. Those 2TB drives spin at 7k2RPM, that's not that fast. It takes weeks to fill one of those things, and weeks again to copy data off. Once you start talking multi-day downtimes to restore data, that's when your customers take their business elsewhere, and fair enough. Ok if you're a university or business with a captive customer base, but not so nice if you're trying to keep customers! Bron. From bernd at firmix.at Tue Sep 29 06:59:39 2009 From: bernd at firmix.at (Bernd Petrovitsch) Date: Tue, 29 Sep 2009 12:59:39 +0200 Subject: Implement Cyrus IMAPD in High Load Enviromment In-Reply-To: <4AC139C8.3070108@ucdavis.edu> References: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> <4AC0DD6F.9090309@ucdavis.edu> <20090928220831.GA7260@brong.net> <4AC139C8.3070108@ucdavis.edu> Message-ID: <1254221979.14774.17.camel@tara.firmix.at> On Mon, 2009-09-28 at 15:33 -0700, Vincent Fox wrote: [...] > Really I've looked at fsck too many times in my life and > don't ever want to again. Anyone who tells me "oh yes but Especially not in the >100GB area. [...] > The antiquated filesystems that 99% of admins tolerate and > work with every day should be lumped under some kind of > Geneva provision against torture. It's a mystery to me why > it's not resolved years ago and why there isn't a big push > for it from anyone. > > "It doesn't matter how fast it is, if it isn't CORRECT!" should > be some kind of mantra for a production data center but it It is. > still seems majority of my colleagues talk same as in 1980s' > about how if we turn off this or that safety feature we can > make the filesystem faster. How much performance do you gain and what are the risks? So - in a larger environment - buying a few disks more and/or faster disks and/or battery-backed controllers and more RAM usually outweighs the risk of losing reputation and (commercial) customers. The next question is: Why do I - as the techie/admin/.. - win by saving a few 100? (or 2.000?) on the hardware (and how much is the total hardware cost?) for *my* decision to use $BRAND_NEW_FAST_FS instead of ext3 and what can I loose (like personal reputation or some sleepless nights and killed weekends in the future)? Does anyone has scripts/tools to - at least - simulate 1000s of (semi-realistic) parallel IMAP clients on a big setup? Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services From brong at fastmail.fm Tue Sep 29 07:23:45 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 29 Sep 2009 21:23:45 +1000 Subject: Implement Cyrus IMAPD in High Load Enviromment In-Reply-To: <1254221979.14774.17.camel@tara.firmix.at> References: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> <4AC0DD6F.9090309@ucdavis.edu> <20090928220831.GA7260@brong.net> <4AC139C8.3070108@ucdavis.edu> <1254221979.14774.17.camel@tara.firmix.at> Message-ID: <20090929112345.GA26891@brong.net> On Tue, Sep 29, 2009 at 12:59:39PM +0200, Bernd Petrovitsch wrote: > Does anyone has scripts/tools to - at least - simulate 1000s of > (semi-realistic) parallel IMAP clients on a big setup? Yeah, I've got one. I need to tidy it up a bit more though, and they're a bit less realistic than I'd like. They pretty much simulate what our web interface USED to do. Bron. From simon.matter at invoca.ch Tue Sep 29 08:50:31 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Tue, 29 Sep 2009 14:50:31 +0200 Subject: Implement Cyrus IMAPD in High Load Enviromment In-Reply-To: <20090929103943.GA3570@brong.net> References: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> <4AC0DD6F.9090309@ucdavis.edu> <20090928220831.GA7260@brong.net> <4AC139C8.3070108@ucdavis.edu> <20090929103943.GA3570@brong.net> Message-ID: <1c2a7feede45971fd5408d949b1b0e7f.squirrel@webmail.bi.corp.invoca.ch> > On Tue, Sep 29, 2009 at 09:45:53AM +0200, Simon Matter wrote: >> What I'm really wondering, what filesystem disasters have others seen? >> How >> many times was it fsck only, how many times was it really broken. I'm >> not >> talking about laptop and desktop users but about production systems in a >> production environment with production class hardware and operating >> systems. > > Well - we lost three drives in a 3TB RAID6 partition within 24 hours. > That was sad. The third drive wasn't totally lost, just throwing enough > errors that we remounted the whole thing readonly and kept it around to > supplement the backup restores. > >> Would be really interesting to get some of the good and bad stories even >> if not directly related to Cyrus-IMAP. > > Honestly, the biggest thing is - I've got a unit I've just switched drives > in. It has 4 x 300GB 15kRPM drives in two RAID1 sets, and 8 x 2TB drives > in two RAID5 sets. That's 12TB of data space plus a bit of room for meta. > > Those 2TB drives spin at 7k2RPM, that's not that fast. It takes weeks to > fill one of those things, and weeks again to copy data off. > > Once you start talking multi-day downtimes to restore data, that's when > your > customers take their business elsewhere, and fair enough. Ok if you're a > university or business with a captive customer base, but not so nice if > you're trying to keep customers! The interesting point is that the discussion started as a ZFS vs. $ANY_OTHER_FS thing but it quickly turns out that the filesystem is only one part of the picture. If your storage fails on the block level I doubt the filesystem matters that much. One of the biggest issues is the cheap big drives which are put together into huge RAID arrrays. There is a good chance that if one disk fails, errors show up on another disk. What I do with Linux software raid is to split every big disk into smaller chunks like with a 500G disk I create 10x50G segments on it. The I create independant RAID devs over every segment of each disk. The whole RAID segments are then put into LVM volgroups. That prevents a disk from getting kicked out of the RAID completely if only a small part of the disk is defect. IIRC ZFS does something which in the end has similar effects and also AIX SoftRAID does someting like that. I end here before getting too OT. Simon From bawood at umich.edu Tue Sep 29 09:19:13 2009 From: bawood at umich.edu (Brian Awood) Date: Tue, 29 Sep 2009 09:19:13 -0400 Subject: Implement Cyrus IMAPD in High Load Enviromment In-Reply-To: <1254221979.14774.17.camel@tara.firmix.at> References: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> <4AC139C8.3070108@ucdavis.edu> <1254221979.14774.17.camel@tara.firmix.at> Message-ID: <200909290919.14057.bawood@umich.edu> On Tuesday 29 September 2009 @ 06:59, Bernd Petrovitsch wrote: > On Mon, 2009-09-28 at 15:33 -0700, Vincent Fox wrote: > [...] > > > Really I've looked at fsck too many times in my life and > > don't ever want to again. Anyone who tells me "oh yes but > > Especially not in the >100GB area. We haven't looked at ZFS, though as Bron suggested, I doubt it will solve all filesystem issues. We use ext3 on large partitions, ranging from 2-5TB. While it takes 14-18hrs to fsck, that doesn't really matter if you have replication, we can promote a replica to a primary in about 15minutes. > > How much performance do you gain and what are the risks? > So - in a larger environment - buying a few disks more and/or > faster disks and/or battery-backed controllers and more RAM usually > outweighs the risk of losing reputation and (commercial) customers. > > The next question is: Why do I - as the techie/admin/.. - win by > saving a few 100? (or 2.000?) on the hardware (and how much is the > total hardware cost?) for *my* decision to use $BRAND_NEW_FAST_FS > instead of ext3 and what can I loose (like personal reputation or > some sleepless nights and killed weekends in the future)? > > Does anyone has scripts/tools to - at least - simulate 1000s of > (semi-realistic) parallel IMAP clients on a big setup? > With cyrus, murder+replication+comodity hardware is the least expensive and probably most scalable way to go. We have some perl scripts for imap load testing, you can find links to them at the bottom of this page, http://blackops.mail.umich.edu/cyrus along with other info on our cyrus implementation. Brian From carles at unlimitedmail.org Tue Sep 29 10:29:30 2009 From: carles at unlimitedmail.org (Carles Xavier Munyoz =?utf-8?q?Bald=C3=B3?=) Date: Tue, 29 Sep 2009 16:29:30 +0200 Subject: deliver.db in /dev/shm Message-ID: <200909291629.31028.carles@unlimitedmail.org> Hello, May I put the database file deliver.db in the /dev/shm partition. I have disabled duplicatesuppression and I believe that I will save lot of I/O requests to my hard drives if I put this file in memory. I'am right? Which problems will I have with this file in memory? I know that I will lost this database file when system reboots, but this is not a problem for me. Will the cyrus create a new deliver.db if it doesn't exist? Thank you very much for your help. Best regards. ======================================== SOLTECSIS SOLUCIONES TECNOLOGICAS, S.L. Carles Xavier Munyoz Bald? Departamento de Tecnolog?a Tel./Fax: 966 446 046 cmunyoz at soltecsis.com www.soltecsis.com ======================================== --- La informaci?n contenida en este e-mail es confidencial, siendo para uso exclusivo del destinatario arriba mencionado. Le informamos que est? totalmente prohibida cualquier utilizaci?n, divulgaci?n, distribuci?n y/o reproducci?n de esta comunicaci?n sin autorizaci?n expresa en virtud de la legislaci?n vigente. Si ha recibido este mensaje por error, le rogamos nos lo notifique inmediatamente por la misma v?a y proceda a su eliminaci?n. --- From selsky at columbia.edu Tue Sep 29 11:20:57 2009 From: selsky at columbia.edu (Matt Selsky) Date: Tue, 29 Sep 2009 11:20:57 -0400 Subject: Cyrus-imapd 2.3.14 - timsieved - Fedora 11-problem In-Reply-To: <4AC1D5C0.6020200@computer-gott.de> References: <4AC1C9D1.6060809@computer-gott.de> <4AC1D5C0.6020200@computer-gott.de> Message-ID: <5E44D553-160A-4E8C-88DE-CE1B4B30B734@columbia.edu> On Sep 29, 2009, at 5:39 AM, Georg Stich wrote: > I use now a fresh installation wir php-pear-Net-Sieve 1.1.7 . > > Do i need install TLS /SSL Support on this System ( in postfix or > somewhere else ) If you have TLS enabled in Cyrus for sieve, then you'll need to make sure your PHP has TLS support. From vbfox at ucdavis.edu Tue Sep 29 12:24:21 2009 From: vbfox at ucdavis.edu (Vincent Fox) Date: Tue, 29 Sep 2009 09:24:21 -0700 Subject: Implement Cyrus IMAPD in High Load Enviromment In-Reply-To: <20090929054238.GC15988@brong.net> References: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> <4AC0DD6F.9090309@ucdavis.edu> <20090928220831.GA7260@brong.net> <4AC139C8.3070108@ucdavis.edu> <20090929054238.GC15988@brong.net> Message-ID: <4AC234B5.7000400@ucdavis.edu> Bron Gondwana wrote: > > It's an interesting one. For real reliability, I want to > have multiple replication target supported cleanly. So the issues for me with Cyrus replication: 1) Is it working? Is the replica actually up to date and more importantly what if I switch to it and there is some corruption in it that becomes apparent? 2) It's ANOTHER backup scheme to understand & maintain Yes I like the idea of replication. However I like even more the idea of being able to do "zpool scrub" on hot filesystem with mirrors &. Thus having to rely a bit less on replica safety net. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3250 bytes Desc: S/MIME Cryptographic Signature Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090929/4807004c/attachment.bin From vbfox at ucdavis.edu Tue Sep 29 12:45:59 2009 From: vbfox at ucdavis.edu (Vincent Fox) Date: Tue, 29 Sep 2009 09:45:59 -0700 Subject: Implement Cyrus IMAPD in High Load Enviromment In-Reply-To: References: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> <4AC0DD6F.9090309@ucdavis.edu> <20090928220831.GA7260@brong.net> <4AC139C8.3070108@ucdavis.edu> Message-ID: <4AC239C7.2090100@ucdavis.edu> Simon Matter wrote: > What I'm really wondering, what filesystem disasters have others seen? How > many times was it fsck only, how many times was it really broken. I'm not > talking about laptop and desktop users but about production systems in a > production environment with production class hardware and operating > systems. > > Would be really interesting to get some of the good and bad stories even > if not directly related to Cyrus-IMAP. > > So we ran UFS (with logging) on multiple UW-IMAP backends before moving to Cyrus. I can tell you at LEAST half a dozen times we would have some hardware or software crash that would leave someone looking at this: fsck /var/mail Y/N? The "correct" answer is Y but then you have hours and hours of downtime so sometimes you say N and cross your fingers. We had one system someone hit N and left it that way for weeks not know if it was going to develop cancer at any moment, until we could migrate users off it. It seemed working OK but we had no way to verify that while "hot" and no downtime available in the intervening perid so we crossed fingers..... Since I've started working here at UC Davis in 2005 I've seen double-disk failures in a RAID-5 set THREE TIMES when I had never seen it in previous 15 years. I've seen double-controller RAID arrays go into total lockup when one controller failed and the code that was supposed to switch smoothly to other controller didn't work. What's going on inside that black-box array controller? Who knows. The original developer is long gone and all the replacements that upgraded it over the years don't really know how it all works. It's often astonishing to me that Linux admins will use hardware controllers and even EMC sans for quite large datasets and blindly trust the black box. RAID6? I am a member of BAARF. RAID5/6 are not to be trusted. See http://www.baarf.com/ So yes I'm the paranoid soul, that if you hand me RAID6 LUNs from an EMC SAN device, I will ZFS mirror them together for additional safety on top of that since I know from experience I cannot trust the black boxes to do what they claim. Really I'm not trying to beat anyone over the head with ZFS particularly, I'm just stating that currently it's the only filesystem I can use in production for large datasets that I actually TRUST. I like a lot being able to once in a while when I replace a disk go "zpool scrub" even during peak usage hours and KNOW it's all correct. When Linux has something similar I'll use it in a second. Until then I prefer Linux for app servers and Solaris for back-end storage. YMMV. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3250 bytes Desc: S/MIME Cryptographic Signature Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090929/96c81caa/attachment.bin From rob at nofocus.org Tue Sep 29 13:33:35 2009 From: rob at nofocus.org (Robert Banz) Date: Tue, 29 Sep 2009 10:33:35 -0700 Subject: Implement Cyrus IMAPD in High Load Enviromment In-Reply-To: <4AC239C7.2090100@ucdavis.edu> References: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> <4AC0DD6F.9090309@ucdavis.edu> <20090928220831.GA7260@brong.net> <4AC139C8.3070108@ucdavis.edu> <4AC239C7.2090100@ucdavis.edu> Message-ID: <30545E21-1E75-4866-8C0C-AD8AF81F40C3@nofocus.org> With my ever-growing experience with these things, I'm tending to think that application-level HA solutions are a much more robust way of dealing with the potential failure modes of hardware or software. While this doesn't mean you shouldn't buy reasonably robust hardware (not the cheapest thing that falls off the truck), what it does is mean is that you should probably spend that extra money you would have spent on that redundant storage system and hot-plug-gable whatsit with replicating your server environment and allowing application-level replication such as Cyrus offers to do the rest. This does mean that we need to lean on software "vendors", be it people we pay or ourselves to create applications that provide for application-layer high availability. However, in the long run it IS a much more robust solution, and covers a much larger number of failure domains for the time and money spent than throwing money and effort at covering the handful of failure-domains that comprise a hardware system -- since that really is an infinite money pit. These "quality" enterprise software solutions that we're sold today that rely on "fault tolerant" hardware, instead of being "fault tolerant" themselves are not acceptable. -rob From maria at shadlen.org Tue Sep 29 18:28:59 2009 From: maria at shadlen.org (Maria McKinley) Date: Tue, 29 Sep 2009 15:28:59 -0700 Subject: moving to new cyrus Message-ID: <4AC28A2B.7040906@shadlen.org> Hello, My mail server is ancient, both in hardware and software, so I am working on updating it. I started with a fresh install on a new machine. My /var/cyrus directory is on its own hard drive. I copied everything from the /var/cyrus directory onto a separate hard drive, which I mounted on the new machine for testing purposes. My plan was to get things working, shut down the old server, move the up-to-date /var/cyrus drive to the new machine and go from there. Unfortunately, while installing/updating to newest cyrus on the new machine I noticed lines like this: Creating/updating cyrus control directories in /var/cyrus/data... Creating/updating partition spool /var/cyrus/mail... which leads me to believe there are changes that have been made to /var/cyrus that will not be on my old /var/cyrus hard drive. Any ideas on how to handle this? My old cyrus is ii cyrus-admin-2.2 2.2.13-13 Cyrus mail system (administration tools) ii cyrus-clients-2.2 2.2.13-13+b2 Cyrus mail system (test clients) ii cyrus-common-2.2 2.2.13-13 Cyrus mail system (common files) ii cyrus-imapd-2.2 2.2.13-13 Cyrus mail system (IMAP support) ii libcyrus-imap-perl22 2.2.13-14+b1 Interface to Cyrus imap client imclient library and the new one: ii cyrus-admin-2.2 2.2.13-14+lenny3 Cyrus mail system (administration tools) ii cyrus-clients-2.2 2.2.13-14+lenny3 Cyrus mail system (test clients) ii cyrus-common-2.2 2.2.13-14+lenny3 Cyrus mail system (common files) ii cyrus-imapd-2.2 2.2.13-14+lenny3 Cyrus mail system (IMAP support) ii cyrus-sasl2-dbg 2.1.22.dfsg1-23+lenny1 Cyrus SASL - debugging symbols ii libcyrus-imap-perl22 2.2.13-14+lenny3 Interface to Cyrus imap client imclient library thanks, maria From brong at fastmail.fm Tue Sep 29 18:41:29 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 30 Sep 2009 08:41:29 +1000 Subject: Implement Cyrus IMAPD in High Load Enviromment In-Reply-To: <200909290919.14057.bawood@umich.edu> References: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> <4AC139C8.3070108@ucdavis.edu> <1254221979.14774.17.camel@tara.firmix.at> <200909290919.14057.bawood@umich.edu> Message-ID: <20090929224129.GB2567@brong.net> On Tue, Sep 29, 2009 at 09:19:13AM -0400, Brian Awood wrote: > > > On Tuesday 29 September 2009 @ 06:59, Bernd Petrovitsch wrote: > > On Mon, 2009-09-28 at 15:33 -0700, Vincent Fox wrote: > > [...] > > > > > Really I've looked at fsck too many times in my life and > > > don't ever want to again. Anyone who tells me "oh yes but > > > > Especially not in the >100GB area. > > We haven't looked at ZFS, though as Bron suggested, I doubt it will > solve all filesystem issues. We use ext3 on large partitions, > ranging from 2-5TB. While it takes 14-18hrs to fsck, that doesn't > really matter if you have replication, we can promote a replica to a > primary in about 15minutes. 15 _MINUTES_? My god. Does it need a massage and having its nails done? It takes us roughly 15 _seconds_ to do a failover. And most of that is monitoring that makes sure everything has started up properly (including database checkpoints completing) Possibly the secret is that we use IPAddr2 from linux-ha to force ARP flushes, and we transfer the primary IP address between machines, so nothing else needs to know - we just shut down one end and bring up the other with the IP and it's all good. Our process is: a) check there are less than 10kb of files in $conf/sync/ - else abort b) shut down the master (host A) c) run sync_client -f $file on each file in $conf/sync (if any) c2) (if any sync fails, restart the master (host A)) d) shut down the replica (host B) e) update the database with the new master location f) start up the replica (host A) g) start up the master (host B) This means replication starts immediately, because the replica is already there when the master starts. Bron. From brong at fastmail.fm Tue Sep 29 18:53:47 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 30 Sep 2009 08:53:47 +1000 Subject: deliver.db in /dev/shm In-Reply-To: <200909291629.31028.carles@unlimitedmail.org> References: <200909291629.31028.carles@unlimitedmail.org> Message-ID: <20090929225347.GC2567@brong.net> On Tue, Sep 29, 2009 at 04:29:30PM +0200, Carles Xavier Munyoz Bald? wrote: > Hello, > May I put the database file deliver.db in the /dev/shm partition. > > I have disabled duplicatesuppression and I believe that I will save lot of I/O > requests to my hard drives if I put this file in memory. > I'am right? Not heaps really. Deliver.db IO is pretty low, and mainly affects lmtp processes - which aren't very time sensitive anyway. What you might want to do it symlink the $conf/proc directory into memory somewhere, because that really doesn't have to last over a restart! > Which problems will I have with this file in memory? > I know that I will lost this database file when system reboots, but this is > not a problem for me. > Will the cyrus create a new deliver.db if it doesn't exist? It should be fine - except: how are you going to put it in the /dev/shm partition? It sits in the $conf directory, so you really need to symlink it out. I'm not convinced cyrus will create a new one over a symlink. Also, I don't know how happy BDB would be. It uses environment stuff in $conf/db/ as well. Skiplist would break on the first checkpoint, because it would create a new file in $conf/deliver.db.NEW, and then rename that over the symlink. That said - if you patched a new deliver.db location into the code, that would work fine! Bron. From maria at shadlen.org Tue Sep 29 19:14:05 2009 From: maria at shadlen.org (Maria McKinley) Date: Tue, 29 Sep 2009 16:14:05 -0700 Subject: moving to new cyrus In-Reply-To: References: <4AC28A2B.7040906@shadlen.org> Message-ID: <4AC294BD.7000807@shadlen.org> brian wrote: > You might want to consider doing a complete, fresh install and using > imapsync to move the mailboxes from one machine to the other. That > way, you'll know that the newer Cyrus has everything it needs and is > up to date. > > I've only used it once but it worked flawlessly. > > http://www.linux-france.org/prj/imapsync/ Fantastic. There is even a debian package. I'm putting my reply back on the list, in case others have the same question. cheers, maria From Duncan.Gibb at SiriusIT.co.uk Tue Sep 29 19:24:57 2009 From: Duncan.Gibb at SiriusIT.co.uk (Duncan Gibb) Date: Wed, 30 Sep 2009 00:24:57 +0100 Subject: moving to new cyrus In-Reply-To: <4AC28A2B.7040906@shadlen.org> References: <4AC28A2B.7040906@shadlen.org> Message-ID: <4AC29749.9050405@SiriusIT.co.uk> Maria McKinley wrote: MMcK> while installing/updating to newest cyrus on the new machine MMcK> I noticed lines like this: > Creating/updating cyrus control directories in /var/cyrus/data... > Creating/updating partition spool /var/cyrus/mail... MMcK> which leads me to believe there are changes that have been MMcK> made to /var/cyrus that will not be on my old /var/cyrus hard MMcK> drive. Any ideas on how to handle this? That's just the packaging scripts making sure the relevant directories exist and have the correct permissions and ownership. The differences between 2.2.13 and 2.2.13-14+lenny3 are tiny so you almost certainly have nothing to worry about. If you're paranoid there's always diff. And rsync ;-) Cheers Duncan -- Duncan Gibb - Technical Director Sirius Corporation plc - control through freedom http://www.siriusit.co.uk/ || t: +44 870 608 0063 Debian Cyrus Team - https://alioth.debian.org/projects/pkg-cyrus-imapd/ From ravi.raju at gmail.com Tue Sep 29 23:10:32 2009 From: ravi.raju at gmail.com (ravi raju) Date: Tue, 29 Sep 2009 23:10:32 -0400 Subject: saslauthd w/postfix smtp only works the first time Message-ID: Folks, I set up cyrus sasl2 to work with postfix smtp server. I am able to send e-mail by authenticating via sasl the first time after I start the saslauthd process. When I send another e-mail, it doesn't work. I looked through different logs, here is what I find: 1. Start saslauthd. I checked the status, several pids start up. Starting saslauthd [ OK ] Creating hardlink from /var/lib/sasl2/mux to /var/spool/postfix/var/lib/sasl2/ saslauthd (pid 29638 29636 29634 29628 29627) is running... 2. Send e-mail. Everything works. 3. I checked the /var/log/syslog to make sure the connection was terminated after it was first opened when sending e-mail. 4. Try sending another e-mail from the same box. E-mail is not sent. 5. I see most of the saslauthd process are shutdown at this point. Status only shows one process running saslauthd (pid 29627) is running... 6. After I force restart saslauthd, I can send another e-mail. At any point in time, I am able to only send one e-mail. Anyone has a clue what is going on? I appreciate your time and help with this. Thanks Ravi -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090929/22fd5e02/attachment.html From brong at fastmail.fm Wed Sep 30 00:15:18 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 30 Sep 2009 14:15:18 +1000 Subject: Statistics on message sizes, folders, etc Message-ID: <20090930041518.GA5556@brong.net> Well, I figure this is interesting anyway - rough statistics on how many folders users have, how big their messages are, how much "churn" the folders have. Source code attached (including a copy of IndexFile.pm which it requires to read the index files) I've tried to make it compatible with different hashing layouts and mailbox setups, but no guarantees, it's pretty untested! Here's a dump for the store that my mailbox is on. 300Gb total mail storage space, sitting at about 85% full. Note that the output is totally unsorted because I'm lazy. Some notes: * we autocreate INBOX, INBOX.Drafts, INBOX.Trash and 'INBOX.Sent Items' - hence the huge '4' spike in folders per user. * lots of empty folders, but they don't explain all the 'NONE' for gappyness, because it checks for a gap between the end and 'LastUid' as well, so they'll actually end up with 'ALL' if there was ever a message in the folder. * don't ask me what that 12 byte "message" is. Obviously some imap client puked it into a folder. I'm not going searching for it! * if you want different buckets, the format is pretty simple, and the specification is at the top of the file! Enjoy. Bron ( yes, I'd love to see some other people's stats. Thanks for asking. I'm planning to feed these stats into my massively realistic IMAP server stresstester... ) STATS for /etc/imapd-slots10a1p3-master.conf Partitions: 1 Users: 3237 Folders: 25619 Folders per user: 10 - 30 273 100 - 300 19 3 1 30 - 100 70 300 - 1000 1 4 1974 5 491 6 190 7 89 8 71 9 57 > 1000 1 Messages per folder: 0 8856 1 2055 1,000 - 3,000 343 10 - 30 3368 10,000 - 30,000 34 100 - 300 1548 2 1476 20,000 - 100,000 15 3 958 3,000 - 10,000 142 30 - 100 2885 300 - 1,000 853 4 759 5 595 6 535 7 444 8 421 9 330 > 100,000 2 Message sizes: 1 - 3 KB 1094127 1 - 3 MB 22340 10 - 30 KB 1073445 100 - 300 B 40 100 - 300 KB 89560 12 1 3 - 10 KB 1555633 3 - 10 MB 13849 30 - 100 KB 307347 300 B - 1 KB 124574 300 KB - 1 MB 46155 > 10 MB 2335 Ratio of gaps between UIDs: (expunge tracking) ALL 4143 HIGH 3494 LOW 3399 MEDIUM 2056 NONE 12527 -------------- next part -------------- A non-text attachment was scrubbed... Name: IndexFile.pm Type: text/x-perl Size: 28737 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090930/0fa67a3d/attachment-0002.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: cyr_stats.pl Type: text/x-perl Size: 4834 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090930/0fa67a3d/attachment-0003.bin From bawood at umich.edu Wed Sep 30 01:01:28 2009 From: bawood at umich.edu (Brian Awood) Date: Wed, 30 Sep 2009 01:01:28 -0400 Subject: Implement Cyrus IMAPD in High Load Enviromment In-Reply-To: <20090929224129.GB2567@brong.net> References: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> <200909290919.14057.bawood@umich.edu> <20090929224129.GB2567@brong.net> Message-ID: <200909300101.32402.bawood@umich.edu> On Tuesday 29 September 2009 @ 18:41, Bron Gondwana wrote: > > Possibly the secret is that we use IPAddr2 from linux-ha to force > ARP flushes, and we transfer the primary IP address between > machines, so nothing else needs to know - we just shut down one end > and bring up the other with the IP and it's all good. Our primaries and replicas are located in different data centers, and since we have not control over how the network is subdivided it's impossible for them to take the same IPs. > > Our process is: > > a) check there are less than 10kb of files in $conf/sync/ - else > abort b) shut down the master (host A) > c) run sync_client -f $file on each file in $conf/sync (if any) > c2) (if any sync fails, restart the master (host A)) > d) shut down the replica (host B) > e) update the database with the new master location > f) start up the replica (host A) > g) start up the master (host B) > > This means replication starts immediately, because the replica is > already there when the master starts. So you just immediately start replicating back to a host (or site) that just failed? How does that work? We have a third level of machines that we sync to, in an out of band process, but the data is stored exactly the same way so we can start replicating to them immediately. So even if a entire data center failed, we can still be running a fully replicated service with almost no downtime visible to users. Brian From brong at fastmail.fm Wed Sep 30 02:30:05 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 30 Sep 2009 16:30:05 +1000 Subject: Implement Cyrus IMAPD in High Load Enviromment In-Reply-To: <200909300101.32402.bawood@umich.edu> References: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> <200909290919.14057.bawood@umich.edu> <20090929224129.GB2567@brong.net> <200909300101.32402.bawood@umich.edu> Message-ID: <20090930063005.GA6580@brong.net> On Wed, Sep 30, 2009 at 01:01:28AM -0400, Brian Awood wrote: > On Tuesday 29 September 2009 @ 18:41, Bron Gondwana wrote: > > > > Possibly the secret is that we use IPAddr2 from linux-ha to force > > ARP flushes, and we transfer the primary IP address between > > machines, so nothing else needs to know - we just shut down one end > > and bring up the other with the IP and it's all good. > > Our primaries and replicas are located in different data centers, and > since we have not control over how the network is subdivided it's > impossible for them to take the same IPs. Yeah, fair enough. > > Our process is: > > > > a) check there are less than 10kb of files in $conf/sync/ - else > > abort b) shut down the master (host A) > > c) run sync_client -f $file on each file in $conf/sync (if any) > > c2) (if any sync fails, restart the master (host A)) > > d) shut down the replica (host B) > > e) update the database with the new master location > > f) start up the replica (host A) > > g) start up the master (host B) > > > > This means replication starts immediately, because the replica is > > already there when the master starts. > > So you just immediately start replicating back to a host (or site) > that just failed? How does that work? We don't usually "fail" as such. We're transferring the master role to a different machine. Generally you have some advance warning something bad is happening (like, a single drive in a RAIDset fails) and transition the master to the less-risky location until the RAID has rebuilt. Or you're doing maintainence on the machine that had the master role. Sure - in a host death situation there's a "force" mode which just does the "host B" parts. Then you have to figure out what needs fixing semi-manually afterwards. We don't have a cleaner-upperer yet. We do have "checkreplication" though, which does a pretty good job of finding what's wrong between the two machines. > We have a third level of machines that we sync to, in an out of band > process, but the data is stored exactly the same way so we can start > replicating to them immediately. So even if a entire data center > failed, we can still be running a fully replicated service with > almost no downtime visible to users. Yeah, that would be nice. We don't have a second datacentre at the moment. We're planning to get one running at some point, at least for the higher-level of paying customers! Bron. From bernd at firmix.at Wed Sep 30 07:13:45 2009 From: bernd at firmix.at (Bernd Petrovitsch) Date: Wed, 30 Sep 2009 13:13:45 +0200 Subject: Implement Cyrus IMAPD in High Load Enviromment In-Reply-To: <1c2a7feede45971fd5408d949b1b0e7f.squirrel@webmail.bi.corp.invoca.ch> References: <916415860909280845u2ef16f68h53cbcde3dc3b40c4@mail.gmail.com> <4AC0DD6F.9090309@ucdavis.edu> <20090928220831.GA7260@brong.net> <4AC139C8.3070108@ucdavis.edu> <20090929103943.GA3570@brong.net> <1c2a7feede45971fd5408d949b1b0e7f.squirrel@webmail.bi.corp.invoca.ch> Message-ID: <1254309225.22653.23.camel@tara.firmix.at> An embedded and charset-unspecified text was scrubbed... Name: not available Url: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090930/9b5069b2/attachment.ksh From D.Piethan at web.de Wed Sep 30 08:31:26 2009 From: D.Piethan at web.de (Dirk Piethan) Date: Wed, 30 Sep 2009 14:31:26 +0200 Subject: unrecognized options: --with-auth Message-ID: <1839383693@web.de> Hi, I can't configure cyrus-imapd-2.3.15. Every time I get the same error. I build it on an AMD64 machine with debian-lenny. I use the Howto www.postfix-howto.de (german translated page) Openssl will compiled with this options: ./config --prefix=/usr And SASL build with follow options: ./configure \ --enable-anon \ --enable-plain \ --enable-login \ --enable-sql \ --enable-pam \ --disable-krb4 \ --disable-otp \ --disable-cram \ --disable-digest \ --with-saslauthd=/var/run/saslauthd \ --with-mysql=/usr/local/mysql \ --with-dblib=berkeley \ --with-bdb-libdir=/usr/local/bdb/lib \ --with-bdb-incdir=/usr/local/bdb/include \ --with-openssl=/usr/ssl \ --with-plugindir=/usr/local/lib/sasl2 Cyrus-imapd will be configured by follow attributes: ./configure --with-sasl=/usr/local/lib --with-perl --with-auth=unix --with-dbdir=/usr/local/bdb --with-bdb-libdir=/usr/local/bdb/lib --with-bdb-incdir=/usr/local/bdb/include --with-openssl=/usr/ssl/ --without-ucdsnmp >/tmp/conflog.log configure: WARNING: unrecognized options: --with-auth configure: WARNING: No Kerberos V4 found config.status: WARNING: 'man/Makefile.in' seems to ignore the --datarootdir setting config.status: WARNING: 'Makefile.in' seems to ignore the --datarootdir setting configure: WARNING: unrecognized options: --with-auth Which way I have to go? Thanks @ ALL Dirk ________________________________________________________________ Neu: WEB.DE Doppel-FLAT mit Internet-Flatrate + Telefon-Flatrate f?r nur 19,99 Euro/mtl.!* http://produkte.web.de/go/02/ From boutilpj at ednet.ns.ca Wed Sep 30 09:22:10 2009 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Wed, 30 Sep 2009 10:22:10 -0300 Subject: unrecognized options: --with-auth In-Reply-To: <1839383693@web.de> References: <1839383693@web.de> Message-ID: <4AC35B82.8090403@ednet.ns.ca> On 09/30/2009 09:31 AM, Dirk Piethan wrote: > Hi, > > I can't configure cyrus-imapd-2.3.15. Every time I get the same error. I build it on an AMD64 machine with debian-lenny. > I use the Howto www.postfix-howto.de (german translated page) > > Openssl will compiled with this options: > ./config --prefix=/usr > > And SASL build with follow options: > > ./configure \ > --enable-anon \ > --enable-plain \ > --enable-login \ > --enable-sql \ > --enable-pam \ > --disable-krb4 \ > --disable-otp \ > --disable-cram \ > --disable-digest \ > --with-saslauthd=/var/run/saslauthd \ > --with-mysql=/usr/local/mysql \ > --with-dblib=berkeley \ > --with-bdb-libdir=/usr/local/bdb/lib \ > --with-bdb-incdir=/usr/local/bdb/include \ > --with-openssl=/usr/ssl \ > --with-plugindir=/usr/local/lib/sasl2 > > > Cyrus-imapd will be configured by follow attributes: > > ./configure --with-sasl=/usr/local/lib --with-perl --with-auth=unix --with-dbdir=/usr/local/bdb --with-bdb-libdir=/usr/local/bdb/lib --with-bdb-incdir=/usr/local/bdb/include --with-openssl=/usr/ssl/ --without-ucdsnmp>/tmp/conflog.log > > configure: WARNING: unrecognized options: --with-auth > configure: WARNING: No Kerberos V4 found > config.status: WARNING: 'man/Makefile.in' seems to ignore the --datarootdir setting > config.status: WARNING: 'Makefile.in' seems to ignore the --datarootdir setting > configure: WARNING: unrecognized options: --with-auth > > > Which way I have to go? Have you tried to remove "--with-auth=unix" from your configure statement? > > Thanks @ ALL > Dirk > ________________________________________________________________ > Neu: WEB.DE Doppel-FLAT mit Internet-Flatrate + Telefon-Flatrate > f?r nur 19,99 Euro/mtl.!* http://produkte.web.de/go/02/ > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -------------- next part -------------- A non-text attachment was scrubbed... Name: boutilpj.vcf Type: text/x-vcard Size: 286 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090930/62aab3c3/attachment.vcf From dwhite at olp.net Wed Sep 30 09:23:41 2009 From: dwhite at olp.net (Dan White) Date: Wed, 30 Sep 2009 08:23:41 -0500 Subject: saslauthd w/postfix smtp only works the first time In-Reply-To: References: Message-ID: <20090930132341.GA5130@dan.olp.net> On 29/09/09?23:10?-0400, ravi raju wrote: >Folks, >I set up cyrus sasl2 to work with postfix smtp server. I am able to send >e-mail by authenticating via sasl the first time after I start the saslauthd >process. When I send another e-mail, it doesn't work. I looked through >different logs, here is what I find: > >1. Start saslauthd. I checked the status, several pids start up. >Starting saslauthd [ OK ] >Creating hardlink from /var/lib/sasl2/mux to >/var/spool/postfix/var/lib/sasl2/ > >saslauthd (pid 29638 29636 29634 29628 29627) is running... > >2. Send e-mail. Everything works. > >3. I checked the /var/log/syslog to make sure the connection was terminated >after it was first opened when sending e-mail. > >4. Try sending another e-mail from the same box. E-mail is not sent. > >5. I see most of the saslauthd process are shutdown at this point. Status >only shows one process running > >saslauthd (pid 29627) is running... > >6. After I force restart saslauthd, I can send another e-mail. At any point >in time, I am able to only send one e-mail. > >Anyone has a clue what is going on? I appreciate your time and help with >this. Is postifix authenticating to LMTP? or is your mail client authenticating to Postfix? Either way, I recommend using lmtptest or smtptest to trouble shoot. You can find them in the cyrus-imapd distribution. To further trouble shoot with us, please provide (sanitized) copies of the following: postfix syslog of a good and bad email delivery attempt any corresponding imapd/lmtpd syslog entries any corresponding auth syslog entries (cyrus sasl) Also, please provide your imapd.conf configuration, or at least the output of 'grep sasl /etc/imapd.conf', your postfix sasl configuration if appropriate (the contents of /etc/postfix/sasl/*), and your postfix lmtp/deliver configuration. -- Dan White