From D.J.Mayo at bath.ac.uk Thu Oct 1 06:01:14 2009 From: D.J.Mayo at bath.ac.uk (David Mayo) Date: Thu, 01 Oct 2009 11:01:14 +0100 Subject: Clients sporadically dropping connections in Murder environment Message-ID: <4AC47DEA.5030602@bath.ac.uk> In August we introduced a Cyrus 2.2 IMAP proxy server in front of our Cyrus 2.2 backend server. This is the first part of the process to move our users' mailboxes to a Cyrus 2.3 backend server. Since the upgrade we have had sporadic reports of users' clients disconnecting from the IMAP server. One user (a fellow sysadmin) using Mulberry 4 reports that occasionally when they send an email, they get a message, "The draft has been sent successfully, but an error has occurred whilst doing post-processing." The message is not saved to their IMAP sent-mail folder, as configured in Mulberry. We enabled telemetry logging on the user's account. There was no attempt to APPEND to the folder, and there were no BAD or NO responses during their session. No errors in Cyrus logs (although we're only logging upto 'info' levels on the production service). The proxyd daemon hasn't crashed[1]. Their client checks for new messages every 2 minutes so presumably this keeps the connection open. People using the same client haven't noticed this problem. Pine, Alpine and Evolution users have apparently also reported similar symptoms although I don't have any first-hand reports. No reports of problems from Outlook, Thunderbird or our Horde Webmail client which are our main supported clients. I've looked through the list archives and can't see this being discussed. Has anyone else using a Murder environment seen issues like this? We're running Solaris 10. Our front-end server is 2.2.13. Our back-end server is 2.2.12. The servers use GSSAPI to authenticate to each other and we refresh the KerberosV tickets every hour from a key tab. We aren't running IMAPS on the back-end server. Any thoughts on this gratefully received! Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK [1] proxyd has crashed a couple of times a week on the proxy server but that's a different matter! We will properly investigate it if the problems are still present when we upgrade the proxy server to Cyrus 2.3 From vladimir at urtext.ru Thu Oct 1 06:51:43 2009 From: vladimir at urtext.ru (Vladimir Zorin) Date: Thu, 01 Oct 2009 14:51:43 +0400 Subject: sql backend question Message-ID: G'day to everyone. As it's written in the cyrus changelog, since Cyrus 2.3.12 we can specify different sql-backends for cyrusdb. I was wondering if it is possible to specify sql-backend for the mailboxes.db? I failed to find any tips on that in docs and mans. Best regards, Vladimir Zorin From listacc at gmx.de Fri Oct 2 08:29:49 2009 From: listacc at gmx.de (listacc at gmx.de) Date: Fri, 02 Oct 2009 14:29:49 +0200 Subject: Folder rights problem Message-ID: <20091002122949.300610@gmx.net> Hello, one of our employees complies that in his thunderbird changed two normal folders to greyed-out folders, e.g. it?s not possible to put messages inside. If I access this account with my own thunderbird, I see the same. But we have no acl?s set set for this account, and my cyradm says for this folder: cyrus lrswipcda Also strange that after I deleted a shared folder tree that was embedded in this account, the two greyed-out folders became writable again. But - this morning they have changed back to be greyed-out! Probably there is something wrong with one of cyrus? databases? But with which one, and how to fix it? And we have no error messages in our logfiles. We use OpenSuSE 10.3 with really old Cyrus 2.2.13-24. I?ll be very glad for any help! Andreas -- GRATIS f?r alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01 From hans.moser at ofd-sth.niedersachsen.de Fri Oct 2 08:56:01 2009 From: hans.moser at ofd-sth.niedersachsen.de (Marc Patermann) Date: Fri, 02 Oct 2009 14:56:01 +0200 Subject: Folder rights problem In-Reply-To: <20091002122949.300610@gmx.net> References: <20091002122949.300610@gmx.net> Message-ID: <4AC5F861.2010607@ofd-sth.niedersachsen.de> Hi, listacc at gmx.de schrieb: > one of our employees complies that in his thunderbird changed two > normal folders to greyed-out folders, e.g. it?s not possible to put > messages inside. > > If I access this account with my own thunderbird, I see the same. But > we have no acl?s set set for this account, and my cyradm says for > this folder: > > cyrus lrswipcda So your employee has no rights on this folder? But on underlaying folders he has? Otherwise the folder may not be grey/shown. I have seen cases where users with enough rights moved shared folders into their own account, which made their INBOX show up in other users folder list (grey) because the shared folder was now beyond this INBOX. :) > Probably there is something wrong with one of cyrus? databases? But > with which one, and how to fix it? And we have no error messages in > our logfiles. Maybe it's PEBKAC. :) Marc From listacc at gmx.de Fri Oct 2 10:56:47 2009 From: listacc at gmx.de (listacc at gmx.de) Date: Fri, 02 Oct 2009 16:56:47 +0200 Subject: [RESOLVED:] Folder rights problem In-Reply-To: <4AC5F861.2010607@ofd-sth.niedersachsen.de> References: <20091002122949.300610@gmx.net> <4AC5F861.2010607@ofd-sth.niedersachsen.de> Message-ID: <20091002145647.231430@gmx.net> -------- Original-Nachricht -------- > Datum: Fri, 02 Oct 2009 14:56:01 +0200 > Von: Marc Patermann > An: listacc at gmx.de > CC: info-cyrus at lists.andrew.cmu.edu > Betreff: Re: Folder rights problem > Hi, > > listacc at gmx.de schrieb: > > > one of our employees complies that in his thunderbird changed two > > normal folders to greyed-out folders, e.g. it?s not possible to put > > messages inside. > > > > If I access this account with my own thunderbird, I see the same. But > > we have no acl?s set set for this account, and my cyradm says for > > this folder: > > > > cyrus lrswipcda > So your employee has no rights on this folder? But on underlaying > folders he has? Otherwise the folder may not be grey/shown. > > I have seen cases where users with enough rights moved shared folders > into their own account, which made their INBOX show up in other users > folder list (grey) because the shared folder was now beyond this INBOX. :) > > > Probably there is something wrong with one of cyrus? databases? But > > with which one, and how to fix it? And we have no error messages in > > our logfiles. > Maybe it's PEBKAC. :) > > > Marc Hi Marc, thanks a lot for the superquick response! And the best is: you hit it :-) It really was PEBKAC, I wondered of acl?s set in mailboxes.db for this special user but I didn?t gave it the right sense. I think it was a problem caused by us admins when giving new acl?s: seemed we interchanged the mailboxes :-/ Again, thanks a lot for my now very relaxed weekend :-) Andreas -- GRATIS f?r alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01 From anebi at iguanait.com Sat Oct 3 16:17:48 2009 From: anebi at iguanait.com (Ali Nebi) Date: Sat, 03 Oct 2009 22:17:48 +0200 Subject: Why cannot create a user with name user@mydomain.tld ? Message-ID: <20091003221748.ozuz1x3u4o0k4sk8@hermod.iguanait.com> Hi, i have installed ispman and it is a frontend for cyrus-imap. When i try to create a user, the ispman tries to create it in this format user.user at mydomain.tld and i get this error: IMAP::Admin [ create ]: couldn't create user.user at domain.tld : try NO Permission denied Does cyrus allows to create a user in this format? if i try to create it this way: user_mydomain_tld, then everything is ok. Is there any workaround for this or i need to use something different than user.user at mydomain.tld? ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. From reinaldoc at gmail.com Sat Oct 3 16:29:53 2009 From: reinaldoc at gmail.com (Reinaldo de Carvalho) Date: Sat, 3 Oct 2009 17:29:53 -0300 Subject: Why cannot create a user with name user@mydomain.tld ? In-Reply-To: <20091003221748.ozuz1x3u4o0k4sk8@hermod.iguanait.com> References: <20091003221748.ozuz1x3u4o0k4sk8@hermod.iguanait.com> Message-ID: <4a5881460910031329t246a009dh98442147c1599104@mail.gmail.com> On Sat, Oct 3, 2009 at 5:17 PM, Ali Nebi wrote: > Hi, > > i have installed ispman and it is a frontend for cyrus-imap. > > When i try to create a user, the ispman tries to create it in this format > > user.user at mydomain.tld > > and i get this error: > > IMAP::Admin [ create ]: couldn't create user.user at domain.tld : try NO > Permission denied > You aren't global admin. Try Korreio frontend (korreio.sf.net) > Does cyrus allows to create a user in this format? > Yes > if i try to create it this way: user_mydomain_tld, then everything is ok. > > Is there any workaround for this or i need to use something different > than user.user at mydomain.tld? > -- Reinaldo de Carvalho http://korreio.sf.net http://python-cyrus.sf.net "Don't try to adapt the software to the way you work, but rather yourself to the way the software works" (myself) From ErikQ at andrew.cmu.edu Sat Oct 3 17:31:31 2009 From: ErikQ at andrew.cmu.edu (Erik Quaeghebeur) Date: Sat, 3 Oct 2009 17:31:31 -0400 (EDT) Subject: problems encountered with CMU's imap referrals using Alpine MUA Message-ID: Hi, I'm using Alpine 2.0 (Kubuntu) as my MUA. I recently arrived at CMU as a visiting scholar. When setting up my cyrus account, I encountered the following problem: cyrus.andrew.cmu.edu works, but often (non-consistent across mail sessions) I get re-asked for my password, but for mail6.andrew.cmu.edu. Authentication does not work then, and I can't access the mailbox I wanted to open (for that session, perhaps the next session I have better luck). I know (from conversations with the CMU computing helpdesk) that mail6 is the backend storage murder server for cyrus on which my mailboxes reside (but I have no clue about the technicalities). After asking help on the Alpine usenet group comp.mail.pine, Mark Crispin suggested I contact people knowledgeable about Cyrus/murder (at CMU) to resolve my problem, which he suggested was related to imap referral handling issues. To this end, I'm writing to this list, with, in attachment a log of a(n artificially short) Alpine session that should contain the info necessary to start diagnosing my problem. It corresponds to me opening my CMU collection, which works, and then trying to open the 'Drafts' folder, which does not work, because Alpine starts talking to mail6 directly(?); i.e., I'm asked to authenticate to mail6, which fails. (N.B.: the Kerberos error/warning appearing in this log has already been resolved.) Thanks in advance for any help, Erik -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: alpine-imap-debug.txt Url: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20091003/566789b3/attachment.txt From ravi.raju at gmail.com Sat Oct 3 18:37:14 2009 From: ravi.raju at gmail.com (Ravi Raju) Date: Sat, 3 Oct 2009 18:37:14 -0400 Subject: Unsubscribe Message-ID: <0B120E11-9C2D-4FB3-93C7-0530C7933BD3@gmail.com> Thanks Ravi From anthony-list at tibbs.ca Sun Oct 4 12:19:34 2009 From: anthony-list at tibbs.ca (Anthony Tibbs) Date: Sun, 04 Oct 2009 12:19:34 -0400 Subject: ctl_cyrusdb I/O requirements Message-ID: <4AC8CB16.6080900@tibbs.ca> Hi everyone, I'm running Cyrus 2.3.14 on a VPS (Gentoo), and have been for a number of years now. All was well until last night when performance went down the tubes and I started seeing high I/O wait times. I'm pretty sure this is a problem on the hosting end, but in trying to diagnose the source of the I/O congestion, the closest I could come to answer was that ctl_cyrusdb is reading/writing about 15-20 4096b blocks (from the /var/imap/db/* files) per second. This amounts to read/write throughput in the 250kb/second range. Is this "normal" behaviour? I never noticed before, but it's the only thing I see that is constantly generating I/O... Take care, Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20091004/2d9c2a70/attachment.html From hmh at debian.org Sun Oct 4 12:59:58 2009 From: hmh at debian.org (Henrique de Moraes Holschuh) Date: Sun, 4 Oct 2009 13:59:58 -0300 Subject: ctl_cyrusdb I/O requirements In-Reply-To: <4AC8CB16.6080900@tibbs.ca> References: <4AC8CB16.6080900@tibbs.ca> Message-ID: <20091004165958.GE10505@khazad-dum.debian.net> On Sun, 04 Oct 2009, Anthony Tibbs wrote: > number of years now. All was well until last night when performance > went down the tubes and I started seeing high I/O wait times. I'm > pretty sure this is a problem on the hosting end, but in trying to > diagnose the source of the I/O congestion, the closest I could come Rebuilding RAID arrays? "sharing" the spindles with some other virtual server whose workload seeks the disk around like crazy? > to answer was that ctl_cyrusdb is reading/writing about 15-20 4096b > blocks (from the /var/imap/db/* files) per second. This amounts > to read/write throughput in the 250kb/second range. Even a laptop HD can do > 200 blocks/s. If you're *sure* it is not the storage/disk going bonkers, or your provider giving you the shaft on the sharing of IOPS to the real storage for your virtual server, it means that box (all virtual servers AND the hypervisor) needs a switft boot to the head because the IO scheduler went insane. -- "One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh From gmane-2006-04-16 at jt-socal.com Sun Oct 4 13:03:42 2009 From: gmane-2006-04-16 at jt-socal.com (John Thomas) Date: Sun, 04 Oct 2009 10:03:42 -0700 Subject: Fastmail on Slashdot Message-ID: <4AC8D56E.5060502@jt-socal.com> This may increase the load on the Cyrus download servers. Hopefully it increases the cash flow to Fastmail so they can keep up their awesome work. http://tech.slashdot.org/story/09/10/03/2131247/Interview-With-Jeremy-Howard-of-FastMailfm -- Sincerely, John Thomas From brong at fastmail.fm Sun Oct 4 16:10:19 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Mon, 05 Oct 2009 07:10:19 +1100 Subject: Fastmail on Slashdot In-Reply-To: <4AC8D56E.5060502@jt-socal.com> References: <4AC8D56E.5060502@jt-socal.com> Message-ID: <1254687019.391.1337998011@webmail.messagingengine.com> On Sun, 04 Oct 2009 10:03 -0700, "John Thomas" wrote: > This may increase the load on the Cyrus download servers. Hopefully it > increases the cash flow to Fastmail so they can keep up their awesome > work. > http://tech.slashdot.org/story/09/10/03/2131247/Interview-With-Jeremy-Howard-of-FastMailfm As I posted somewhere in that thread. "I haven't been paged yet (fingers crossed)" :) Bron. -- Bron Gondwana brong at fastmail.fm From awilliam at whitemice.org Mon Oct 5 09:41:56 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Mon, 05 Oct 2009 09:41:56 -0400 Subject: Fastmail on Slashdot In-Reply-To: <4AC8D56E.5060502@jt-socal.com> References: <4AC8D56E.5060502@jt-socal.com> Message-ID: <1254750116.5264.18.camel@linux-m3mt> On Sun, 2009-10-04 at 10:03 -0700, John Thomas wrote: > This may increase the load on the Cyrus download servers. Hopefully it > increases the cash flow to Fastmail so they can keep up their awesome work. > http://tech.slashdot.org/story/09/10/03/2131247/Interview-With-Jeremy-Howard-of-FastMailfm Sweet, good to see both Cyrus get some air-time [it is seriously under appreciated] and Slashdot actually covering something FOSS related. From selsky at columbia.edu Mon Oct 5 10:33:37 2009 From: selsky at columbia.edu (Matt Selsky) Date: Mon, 5 Oct 2009 10:33:37 -0400 Subject: Statistics on message sizes, folders, etc In-Reply-To: <20090930041518.GA5556@brong.net> References: <20090930041518.GA5556@brong.net> Message-ID: <40C1E541-6D12-46C0-B8C6-493C9C40C1A1@columbia.edu> Here's the output for the store my mailbox is on (6 x 500GB partitions, most are 40% full). Let me know if it would be useful to have output from the rest of the stores. -Matt STATS for /etc/imapd.conf Partitions: 7 Users: 12570 Folders: 110469 Folders per user: 1 1607 10 - 30 2177 100 - 300 51 2 2840 3 1417 30 - 100 509 300 - 1000 9 4 1343 5 668 6 781 7 431 8 490 9 245 > 1000 2 Messages per folder: 0 24283 1 9293 1,000 - 3,000 3781 10 - 30 17276 10,000 - 30,000 178 100 - 300 9791 2 4843 20,000 - 100,000 7 3 3743 3,000 - 10,000 1446 30 - 100 16006 300 - 1,000 6848 4 2987 5 2508 6 2213 7 1935 8 1780 9 1551 Message sizes: 0 254 1 - 3 KB 7006825 1 - 3 MB 132156 10 - 30 KB 3304247 100 - 300 B 3642 100 - 300 KB 689501 2 1 3 - 10 KB 8620348 3 - 10 MB 70074 30 - 100 KB 1938822 300 B - 1 KB 1131644 300 KB - 1 MB 357495 60 4 62 2 66 6 67 1 68 1 69 3 78 1 79 2 82 1 83 1 84 1 90 1 92 3 93 3 94 3 95 3 98 1 > 10 MB 13663 Ratio of gaps between UIDs: (expunge tracking) ALL 6374 HIGH 6815 LOW 13375 MEDIUM 5308 NONE 78597 From luca at wetron.es Mon Oct 5 12:53:25 2009 From: luca at wetron.es (Luca Olivetti) Date: Mon, 05 Oct 2009 18:53:25 +0200 Subject: login doesn't fully work, authenticate does Message-ID: <4ACA2485.3030306@wetron.es> Hello, I just upgraded my cyrus-imapd server from 2.2.12 to 2.3.15. Squirrelmail stopped working with the error 'SELECT "INBOX" - mailbox does not exist". I found out that neither SELECT nor STATUS work if authenticating with login, while they work with AUTHENTICATE (which I cannot use with squirrelmail and my setup) $ imtest -m login localhost S: * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID STARTTLS AUTH=PLAIN AUTH=CRAM-MD5 SASL-IR COMPRESS=DEFLATE] mail.wetron.es Cyrus IMAP v2.3.15-Mandriva-RPM-2.3.15-1.1.100mdk server ready Please enter your password: C: L01 LOGIN luca {8} S: + go ahead C: S: L01 OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID STARTTLS AUTH=PLAIN AUTH=CRAM-MD5 SASL-IR COMPRESS=DEFLATE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE CATENATE CONDSTORE SCAN IDLE LISTEXT LIST-SUBSCRIBED X-NETSCAPE URLAUTH] User logged in Authenticated. Security strength factor: 0 x SELECT INBOX x NO Mailbox does not exist x LOGOUT * BYE LOGOUT received x OK Completed Connection closed. $ imtest -m plain localhost S: * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID STARTTLS AUTH=PLAIN AUTH=CRAM-MD5 SASL-IR COMPRESS=DEFLATE] mail.wetron.es Cyrus IMAP v2.3.15-Mandriva-RPM-2.3.15-1.1.100mdk server ready Please enter your password: C: A01 AUTHENTICATE PLAIN xxxxxxxxxxxxxxxxxxxxx S: A01 OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID LOGINDISABLED COMPRESS=DEFLATE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE CATENATE CONDSTORE SCAN IDLE LISTEXT LIST-SUBSCRIBED X-NETSCAPE URLAUTH] Success (no protection) Authenticated. Security strength factor: 0 x SELECT INBOX * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $Forwarded $MDNSent NonJunk Junk $Label1 $Label2 $Label3 $Label4 $Label5) * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $Forwarded $MDNSent NonJunk Junk $Label1 $Label2 $Label3 $Label4 $Label5 \*)] * 3060 EXISTS * 0 RECENT * OK [UIDVALIDITY 904828784] * OK [UIDNEXT 45691] * OK [NOMODSEQ] Sorry, modsequences have not been enabled on this mailbox * OK [URLMECH INTERNAL] x OK [READ-WRITE] Completed x LOGOUT * BYE LOGOUT received x OK Completed Connection closed. It worked either way with 2.2.12 and I couldn't find a configuration option controlling this behavior. Is there one? Bye -- Luca Olivetti Wetron Automatizaci?n S.A. http://www.wetron.es/ Tel. +34 93 5883004 (Ext.133) Fax +34 93 5883007 From mingram at cbnco.com Mon Oct 5 14:52:14 2009 From: mingram at cbnco.com (Matt Ingram) Date: Mon, 05 Oct 2009 14:52:14 -0400 Subject: Cyrus Mailboxes Message-ID: <4ACA405E.1030607@cbnco.com> Hi All. I have a user who wants to give another employee access to his mailbox, without sharing his password. I believe Cyrus can do this, but I'm not sure on the best practice and exactly how to do it - a link to some good documentation on this would be great.. I've found a few documents but not highlighting exactly what I'm trying to do. I'm using cyrus-imapd-2.2.12-27.8 on SLES 10.1 Thanks in advance for any help. Matt. -- Matt Ingram Intermediate Unix Administrator, IS Canadian Bank Note Company, Limited \m/ From awilliam at whitemice.org Mon Oct 5 15:20:56 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Mon, 05 Oct 2009 15:20:56 -0400 Subject: Cyrus Mailboxes In-Reply-To: <4ACA405E.1030607@cbnco.com> References: <4ACA405E.1030607@cbnco.com> Message-ID: <1254770456.5338.8.camel@linux-m3mt> On Mon, 2009-10-05 at 14:52 -0400, Matt Ingram wrote: > Hi All. > I have a user who wants to give another employee access to his mailbox, > without sharing his password. I believe Cyrus can do this, but I'm not > sure on the best practice and exactly how to do it - a link to some good > documentation on this would be great.. I've found a few documents but > not highlighting exactly what I'm trying to do. > I'm using cyrus-imapd-2.2.12-27.8 on SLES 10.1 cyradm is the tool you want. See the Cyrus IMAPd chapter of WMOGAG -- OpenGroupware developer: awilliam at whitemice.org OpenGroupare & Cyrus IMAPd documenation @ From reinaldoc at gmail.com Mon Oct 5 16:04:30 2009 From: reinaldoc at gmail.com (Reinaldo de Carvalho) Date: Mon, 5 Oct 2009 17:04:30 -0300 Subject: Cyrus Mailboxes In-Reply-To: <4ACA405E.1030607@cbnco.com> References: <4ACA405E.1030607@cbnco.com> Message-ID: <4a5881460910051304x47a0fe72hba2bfe17cdb51e81@mail.gmail.com> On Mon, Oct 5, 2009 at 3:52 PM, Matt Ingram wrote: > Hi All. > > I have a user who wants to give another employee access to his mailbox, > without sharing his password. I believe Cyrus can do this, but I'm not > sure on the best practice and exactly how to do it - a link to some good > documentation on this would be great.. I've found a few documents but > not highlighting exactly what I'm trying to do. > > I'm using cyrus-imapd-2.2.12-27.8 on SLES 10.1 > > Thanks in advance for any help. > http://python-cyrus.sf.net http://korreio.sf.net Cyrus Manager on your Desktop: http://sourceforge.net/project/screenshots.php?group_id=206408&ssid=104818 -- Reinaldo de Carvalho "Don't try to adapt the software to the way you work, but rather yourself to the way the software works" (myself) From luca at wetron.es Tue Oct 6 05:36:43 2009 From: luca at wetron.es (Luca Olivetti) Date: Tue, 06 Oct 2009 11:36:43 +0200 Subject: login doesn't fully work, authenticate does In-Reply-To: <4ACA2485.3030306@wetron.es> References: <4ACA2485.3030306@wetron.es> Message-ID: <4ACB0FAB.6050100@wetron.es> En/na Luca Olivetti ha escrit: > Hello, > > I just upgraded my cyrus-imapd server from 2.2.12 to 2.3.15. > Squirrelmail stopped working with the error 'SELECT "INBOX" - mailbox > does not exist". > I found out that neither SELECT nor STATUS work if authenticating with > login, while they work with AUTHENTICATE (which I cannot use with > squirrelmail and my setup) It turns out that the comment in cmd_login (imap/imapd.c) that says /* authstate already created by mysasl_proxy_policy() */ doesn't apply here (maybe my sasl libraries are too old and don't work as intended?). Anyway, I applied this workaround --- cyrus-imapd-2.3.15.orig/imap/imapd.c 2009-10-06 11:31:51.456950883 +0200 +++ cyrus-imapd-2.3.15/imap/imapd.c 2009-10-06 11:33:22.107293040 +0200 @@ -2199,6 +2199,8 @@ } /* authstate already created by mysasl_proxy_policy() */ + if (!imapd_authstate) + imapd_authstate = auth_newstate(imapd_userid); imapd_userisadmin = global_authisa(imapd_authstate, IMAPOPT_ADMINS); prot_printf(imapd_out, "%s OK [CAPABILITY ", tag); Bye -- Luca Olivetti Wetron Automatizaci?n S.A. http://www.wetron.es/ Tel. +34 93 5883004 (Ext.133) Fax +34 93 5883007 From Eric.Luyten at vub.ac.be Tue Oct 6 10:19:59 2009 From: Eric.Luyten at vub.ac.be (Eric Luyten) Date: Tue, 6 Oct 2009 16:19:59 +0200 (CEST) Subject: Need for pop3d processes on Solaris 10 to scan /etc/passwd and /etc/group, twice ? Message-ID: <64099.134.184.15.103.1254838799.squirrel@nuts.vub.ac.be> Folks, We migrated a single-server Cyrus from Solaris 9 to Solaris 10 early last week, jumping from 2.2.13 to 2.3.15 in the process. All runs pretty well, save for a huge number of authentication failures when the system is under less-than-trivial load. cyrus-sasl-2.1.23 was compiled with -D_REENTRANT compiler flag and started with '-a shadow' authentication mechanism. When truss-ing ("tracing") one of the pop3 processes, we can observe two scans of /etc/passwd and /etc/group Question : at which stage would a Cyrus pop3d process need to obtain information from the /etc/passwd and /etc/group files, since it does not need to set euid or egid nor perform authen- tication by its own, since that's handled by saslauthd (easily verifiable by halting the running saslauthd, which makes all POP and IMAP authentication attempts fail) ? All hints very welcome, Eric Luyten, Computing Centre VUB/ULB. From simon.matter at invoca.ch Tue Oct 6 10:59:00 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Tue, 6 Oct 2009 16:59:00 +0200 Subject: Need for pop3d processes on Solaris 10 to scan /etc/passwd and /etc/group, twice ? In-Reply-To: <64099.134.184.15.103.1254838799.squirrel@nuts.vub.ac.be> References: <64099.134.184.15.103.1254838799.squirrel@nuts.vub.ac.be> Message-ID: > Folks, > > > We migrated a single-server Cyrus from Solaris 9 to Solaris 10 > early last week, jumping from 2.2.13 to 2.3.15 in the process. > > All runs pretty well, save for a huge number of authentication > failures when the system is under less-than-trivial load. > > cyrus-sasl-2.1.23 was compiled with -D_REENTRANT compiler flag > and started with '-a shadow' authentication mechanism. > > When truss-ing ("tracing") one of the pop3 processes, we can > observe two scans of /etc/passwd and /etc/group > > Question : at which stage would a Cyrus pop3d process need to > obtain information from the /etc/passwd and /etc/group files, > since it does not need to set euid or egid nor perform authen- > tication by its own, since that's handled by saslauthd (easily > verifiable by halting the running saslauthd, which makes all > POP and IMAP authentication attempts fail) ? > > > All hints very welcome, Doesn't it use /etc/passwd and /etc/group for doing unix-style authorization like checking ACL's on (shared) folders? Saslauthd is only used for authentication, isn't it? Regards, Simon From atarallo at acm.org Wed Oct 7 08:26:13 2009 From: atarallo at acm.org (Andres Tarallo) Date: Wed, 7 Oct 2009 09:26:13 -0300 Subject: Lockers ? Message-ID: <3fdbc0030910070526n354b7a59kc55da501b16e97e4@mail.gmail.com> I have a cyrus-imap server that receives mail from a postfix server via LMTP (network, they are separated physical servers). Every day we receive several messages from mailer daemon that have the following errors: host 192.168.84.184[192.168.84.184] said: 308 lockers (in reply to end of DATA command). How can we solve this? Thanks Andres From selsky at columbia.edu Wed Oct 7 08:44:28 2009 From: selsky at columbia.edu (Matt Selsky) Date: Wed, 7 Oct 2009 08:44:28 -0400 Subject: Lockers ? In-Reply-To: <3fdbc0030910070526n354b7a59kc55da501b16e97e4@mail.gmail.com> References: <3fdbc0030910070526n354b7a59kc55da501b16e97e4@mail.gmail.com> Message-ID: <2CF0A9C8-020F-494A-B1EC-ED85B5AD5C3A@columbia.edu> On Oct 7, 2009, at 8:26 AM, Andres Tarallo wrote: > I have a cyrus-imap server that receives mail from a postfix server > via LMTP (network, they are separated physical servers). > > Every day we receive several messages from mailer daemon that have the > following errors: host 192.168.84.184[192.168.84.184] said: 308 > lockers (in reply to end of DATA command). How can we solve this? What version of Cyrus are you using? What database formats? It sounds like you should switch from bdb to skiplist... -- Matt From selsky at columbia.edu Wed Oct 7 09:05:04 2009 From: selsky at columbia.edu (Matt Selsky) Date: Wed, 7 Oct 2009 09:05:04 -0400 Subject: Lockers ? In-Reply-To: <3fdbc0030910070557q632d5a3ax244964c1417dfed3@mail.gmail.com> References: <3fdbc0030910070526n354b7a59kc55da501b16e97e4@mail.gmail.com> <2CF0A9C8-020F-494A-B1EC-ED85B5AD5C3A@columbia.edu> <3fdbc0030910070557q632d5a3ax244964c1417dfed3@mail.gmail.com> Message-ID: <9433AFF8-0243-4A9D-A11C-13FF500A357C@columbia.edu> [please keep list traffic on the list] On Oct 7, 2009, at 8:57 AM, Andres Tarallo wrote: >> What version of Cyrus are you using? > cyrus-imapd-2.2.12, it's RPMs in SuSE 10.1 > >> What database formats? It sounds like >> you should switch from bdb to skiplist... >> > Which database? Postfix uses OpenLDAP for almost anything. The Cyrus internal databases. Eg: $ grep _db /etc/imapd.conf ptscache_db: skiplist duplicate_db: skiplist statuscache_db: skiplist tlscache_db: skiplist You likely also want to upgrade to a more recent version of Cyrus. Cheers, -- Matt From Leena.Heino at uta.fi Wed Oct 7 09:18:26 2009 From: Leena.Heino at uta.fi (Leena Heino) Date: Wed, 7 Oct 2009 16:18:26 +0300 (FLE Daylight Time) Subject: Lockers ? In-Reply-To: <2CF0A9C8-020F-494A-B1EC-ED85B5AD5C3A@columbia.edu> References: <3fdbc0030910070526n354b7a59kc55da501b16e97e4@mail.gmail.com> <2CF0A9C8-020F-494A-B1EC-ED85B5AD5C3A@columbia.edu> Message-ID: On Wed, 7 Oct 2009, Matt Selsky wrote: > On Oct 7, 2009, at 8:26 AM, Andres Tarallo wrote: > >> I have a cyrus-imap server that receives mail from a postfix server >> via LMTP (network, they are separated physical servers). >> >> Every day we receive several messages from mailer daemon that have the >> following errors: host 192.168.84.184[192.168.84.184] said: 308 >> lockers (in reply to end of DATA command). How can we solve this? > > What version of Cyrus are you using? What database formats? It > sounds like you should switch from bdb to skiplist... This bug was fixed in 2005 with this change to Cyrus IMAPD: https://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/lib/cyrusdb_berkeley.c.diff?r1=1.9;r2=1.11 Bugzilla entry for this bug is: https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=2699 -- Leena Heino University of Tampere / Computer Centre ( liinu at uta.fi ) ( http://www.uta.fi/laitokset/tkk ) From Eric.Luyten at vub.ac.be Wed Oct 7 11:58:48 2009 From: Eric.Luyten at vub.ac.be (Eric Luyten) Date: Wed, 7 Oct 2009 17:58:48 +0200 (CEST) Subject: Need for pop3d processes on Solaris 10 to scan /etc/passwd and /etc/group, twice ? In-Reply-To: References: <64099.134.184.15.103.1254838799.squirrel@nuts.vub.ac.be> Message-ID: <52741.134.184.15.103.1254931128.squirrel@nuts.vub.ac.be> On Tue, October 6, 2009 4:59 pm, Simon Matter wrote: [me] >> We migrated a single-server Cyrus from Solaris 9 to Solaris 10 >> early last week, jumping from 2.2.13 to 2.3.15 in the process. >> >> All runs pretty well, save for a huge number of authentication >> failures when the system is under less-than-trivial load. >> >> cyrus-sasl-2.1.23 was compiled with -D_REENTRANT compiler flag and started >> with '-a shadow' authentication mechanism. >> >> When truss-ing ("tracing") one of the pop3 processes, we can >> observe two scans of /etc/passwd and /etc/group >> >> Question : at which stage would a Cyrus pop3d process need to >> obtain information from the /etc/passwd and /etc/group files, since it does >> not need to set euid or egid nor perform authen- tication by its own, since >> that's handled by saslauthd (easily verifiable by halting the running >> saslauthd, which makes all POP and IMAP authentication attempts fail) ? > > Doesn't it use /etc/passwd and /etc/group for doing unix-style > authorization like checking ACL's on (shared) folders? Simon, Copy that. This is the 'unix_group_enable' in imapd.conf, which defaults to '1' = 'on' > Saslauthd is only used for authentication, isn't it? Correct. Eric. From gavin.gray at ed.ac.uk Thu Oct 8 09:04:57 2009 From: gavin.gray at ed.ac.uk (Gavin Gray) Date: Thu, 08 Oct 2009 14:04:57 +0100 Subject: Synchronisation two cyrus-imapd servers In-Reply-To: <1253191330.1801.14.camel@support.spectrum.ru> References: <1253191330.1801.14.camel@support.spectrum.ru> Message-ID: <20091008140457.ioc7iat55wwskwwo@www.staffmail.ed.ac.uk> Hi there, I also have this problem. I can't get the sync_server to run under 2.3.15. Here is a snippet fom our logs: Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 143423 local6.error] DBERROR: reading /var/imap/db/skipstamp, assuming the worst: No such file or directory Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 518349 local6.debug] executed Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 301543 local6.info] skiplist: checkpointed /var/imap/mailboxes.db (0 records, 144 bytes) in 0 seconds Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 301543 local6.info] skiplist: checkpointed /var/imap/annotations.db (0 records, 144 bytes) in 0 seconds Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 921384 local6.debug] accepted connection Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 177842 local6.debug] cmdloop(): startup Oct 8 12:42:36 mailbe9r last message repeated 1 time Oct 8 12:42:38 mailbe9r master[25640]: [ID 970914 local6.error] process 25647 exited, signaled to death by 11 Oct 8 12:42:38 mailbe9r master[25640]: [ID 621917 local6.debug] service syncserver pid 25647 in BUSY state: terminated abnormally I've had a go at trying to debug the problem, but had no success. Does anyone have any ideas? regards, Gavin Gray Quoting Alexander Demin : > Hello. > > I have problem with synchronisation two cyrus-imapd servers. > > ******* Start "Replica" host configuration ******* > OS: FreeBSD 7.2-STABLE i386 > cyrus-imapd-2.3.15 WITH_BDB=true WITH_REPLICATION=true > cyrus-sasl-2.1.23 WITH_AUTHDAEMOND=true WITH_LOGIN=true WITH_PLAIN=true > WITH_CRAM=true WITH_DIGEST=true > cyrus-sasl-saslauthd-2.1.23 > All soft installed from ports. > > Cyrus configuration: > /usr/local/etc/cyrus.conf > START { > recover cmd="ctl_cyrusdb -r" > } > > SERVICES { > imap cmd="imapd" listen="imap" prefork=0 > imaps cmd="imapd -s" listen="imaps" prefork=0 > pop3 cmd="pop3d" listen="pop3" prefork=0 > pop3s cmd="pop3d -s" listen="pop3s" prefork=0 > sieve cmd="timsieved" listen="sieve" prefork=0 > lmtpunix cmd="lmtpd" listen="/data/imap/socket/lmtp" prefork=0 > smmap cmd="smmapd" listen="/data/imap/socket/smmap" prefork=1 > syncserver cmd="sync_server" listen="csync" prefork=1 > } > > EVENTS { > checkpoint cmd="ctl_cyrusdb -c" period=30 > delprune cmd="cyr_expire -E 3" at=0400 > tlsprune cmd="tls_prune" at=0400 > } > > /usr/local/etc/imapd.conf > configdirectory: /backup/imap > partition-default: /backup/spool/imap > unixhierarchysep: no > altnamespace: yes > allowanonymouslogin: no > allowplaintext: yes > imapidresponse: yes > admins: cyrus > munge8bit: 0 > rfc2046_strict: 0 > sievedir: /backup/imap/sieve > sendmail: /usr/sbin/sendmail > postmaster: postmaster > annotation_db: skiplist > duplicate_db: berkeley-nosync > mboxlist_db: skiplist > ptscache_db: berkeley > seenstate_db: skiplist > subscription_db: flat > sasl_pwcheck_method: auxprop > sasl_auxprop_plugin: sasldb > sasl_log_level: 7 > sasl_mech_list: plain cram-md5 digest-md5 login > lmtpsocket: /backup/imap/socket/lmtp > virtdomains: userid > lmtp_downcase_rcpt: 1 > > # > # EOF > > /etc/services > csync 2005/tcp > > /etc/rc.conf (show only cyrus/sasl params) > cyrus_imapd_enable="YES" > saslauthd_enable="YES" > saslauthd_flags="-a sasldb" > ******* End "Replica" host configuration ******* > > ******* Start "Master" host configuration ******* > OS: FreeBSD 7.2-STABLE amd64 > cyrus-imapd-2.3.15 WITH_BDB=true WITH_REPLICATION=true > cyrus-sasl-2.1.23 WITH_AUTHDAEMOND=true WITH_LOGIN=true WITH_PLAIN=true > WITH_CRAM=true WITH_DIGEST=true > cyrus-sasl-saslauthd-2.1.23 > All soft installed from ports. > > Cyrus configuration: > /usr/local/etc/cyrus.conf > START { > recover cmd="ctl_cyrusdb -r" > } > > SERVICES { > imap cmd="imapd" listen="imap" prefork=0 > imaps cmd="imapd -s" listen="imaps" prefork=0 > pop3 cmd="pop3d" listen="pop3" prefork=0 > pop3s cmd="pop3d -s" listen="pop3s" prefork=0 > sieve cmd="timsieved" listen="sieve" prefork=0 > lmtpunix cmd="lmtpd" listen="/data/imap/socket/lmtp" prefork=0 > smmap cmd="smmapd" listen="/data/imap/socket/smmap" prefork=1 > syncclient cmd="sync_client -r" listen="csync" prefork=1 > } > > EVENTS { > checkpoint cmd="ctl_cyrusdb -c" period=30 > delprune cmd="cyr_expire -E 3" at=0400 > tlsprune cmd="tls_prune" at=0400 > } > > /usr/local/etc/imapd.conf > configdirectory: /data/imap > partition-default: /data/spool/imap > unixhierarchysep: no > altnamespace: yes > allowanonymouslogin: no > allowplaintext: yes > imapidresponse: yes > admins: cyrus cyrus at spectrum.ru > munge8bit: 0 > rfc2046_strict: 0 > sievedir: /data/imap/sieve > sendmail: /usr/sbin/sendmail > postmaster: postmaster > annotation_db: skiplist > duplicate_db: berkeley-nosync > mboxlist_db: skiplist > ptscache_db: berkeley > seenstate_db: skiplist > subscription_db: flat > sasl_pwcheck_method: auxprop > sasl_auxprop_plugin: sasldb > sasl_log_level: 7 > sasl_mech_list: plain cram-md5 digest-md5 login > tls_cert_file: /etc/ssl/imapserver.pem > tls_key_file: /etc/ssl/imapserver.pem > tls_ca_file: /etc/ssl/imapserver.pem > tls_session_timeout: 0 > lmtpsocket: /data/imap/socket/lmtp > virtdomains: userid > lmtp_downcase_rcpt: 1 > sync_repeat_interval: 10 > sync_host: support.spectrum.ru > sync_authname: cyrus > sync_password: *********** > sync_log: 1 > > # > # EOF > > /etc/services > csync 2005/tcp > > /etc/rc.conf (show only cyrus/sasl params) > cyrus_imapd_enable="YES" > saslauthd_enable="YES" > saslauthd_flags="-a sasldb" > > "Master" - it's production mail server of my company. All services > worked is fine. > ******* End "Master" host configuration ******* > > Step-by-step: > "Replica" host > 1. /usr/local/etc/rc.d/imapd start > 2. imtest -a cyrus localhost - has passed successfully > 3. synctest -u cyrus localhost - failed > S: * SASL LOGIN PLAIN DIGEST-MD5 CRAM-MD5 > S: * OK support.spectrum.ru Cyrus sync server v2.3.15 > C: AUTHENTICATE DIGEST-MD5 > failure: prot layer failure > 4. ps -ax | grep sync_server > 65257 ?? I 0:00,00 sync_server > 65617 ?? I 0:00,00 sync_server > 5. grep sync /var/log/all.log > Sep 17 15:59:24 support syncserver[65589]: accepted connection > Sep 17 15:59:24 support master[65616]: about to > exec /usr/local/cyrus/bin/sync_server > Sep 17 15:59:24 support kernel: pid 65589 (sync_server), uid 60: exited > on signal 11 > Sep 17 15:59:24 support syncserver[65589]: cmdloop(): startup > Sep 17 15:59:24 support syncserver[65616]: executed > Sep 17 15:59:24 support master[65253]: service syncserver pid 65589 in > BUSY state: terminated abnormally > Sep 17 15:59:24 support syncserver[65616]: accepted connection > Sep 17 15:59:24 support master[65617]: about to > exec /usr/local/cyrus/bin/sync_server > Sep 17 15:59:24 support kernel: pid 65616 (sync_server), uid 60: exited > on signal 11 > Sep 17 15:59:24 support syncserver[65616]: cmdloop(): startup > Sep 17 15:59:24 support syncserver[65617]: executed > Sep 17 15:59:24 support master[65253]: service syncserver pid 65616 in > BUSY state: terminated abnormally > > "Master" host > 1. /usr/local/etc/rc.d/imapd restart > 2. imtest -a cyrus localhost - has passed successfully > 3. ps -ax | grep sync_client > 63196 ?? S 0:00,01 sync_client -r > 63197 ?? S 0:00,01 sync_client -r > 4. grep sync /var/log/all.log > Sep 17 16:24:18 mail sync_client[63196]: couldn't authenticate to > backend server: generic failure > Sep 17 16:24:18 mail sync_client[63197]: couldn't authenticate to > backend server: generic failure > Sep 17 16:25:18 mail sync_client[63196]: couldn't authenticate to > backend server: generic failure > Sep 17 16:25:18 mail sync_client[63197]: couldn't authenticate to > backend server: generic failure > > Did i make something not correctly? > Help me, please, to find the mistake and understand this problem. > > Thanks. > > -- > Demin Alexander / Network Administrator > Group of companies Spectrum / tel. (+7 495) 995-8999 > Russia, Moscow, 103009, Strastnoy blvr. 8 > Web: http://www.spectrum.ru/ > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > > -- Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.gray at ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From steffo76 at gmx.de Thu Oct 8 09:41:05 2009 From: steffo76 at gmx.de (steffo76 at gmx.de) Date: Thu, 08 Oct 2009 15:41:05 +0200 Subject: Synchronisation two cyrus-imapd servers In-Reply-To: <20091008140457.ioc7iat55wwskwwo@www.staffmail.ed.ac.uk> References: <1253191330.1801.14.camel@support.spectrum.ru> <20091008140457.ioc7iat55wwskwwo@www.staffmail.ed.ac.uk> Message-ID: <20091008134105.7700@gmx.net> Hi, you are probably missing 'defaultpartition' in imapd.conf: https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3170 Regards, Stephan -------- Original-Nachricht -------- > Datum: Thu, 08 Oct 2009 14:04:57 +0100 > Von: Gavin Gray > An: info-cyrus at lists.andrew.cmu.edu > Betreff: Re: Synchronisation two cyrus-imapd servers > Hi there, > I also have this problem. I can't get the sync_server to run > under 2.3.15. Here is a snippet fom our logs: > > Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 143423 local6.error] > DBERROR: reading /var/imap/db/skipstamp, assuming the worst: No such > file or directory > Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 518349 local6.debug] > executed > Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 301543 local6.info] > skiplist: checkpointed /var/imap/mailboxes.db (0 records, 144 bytes) > in 0 seconds > Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 301543 local6.info] > skiplist: checkpointed /var/imap/annotations.db (0 records, 144 bytes) > in 0 seconds > Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 921384 local6.debug] > accepted connection > Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 177842 local6.debug] > cmdloop(): startup > Oct 8 12:42:36 mailbe9r last message repeated 1 time > Oct 8 12:42:38 mailbe9r master[25640]: [ID 970914 local6.error] > process 25647 exited, signaled to death by 11 > Oct 8 12:42:38 mailbe9r master[25640]: [ID 621917 local6.debug] > service syncserver pid 25647 in BUSY state: terminated abnormally > > I've had a go at trying to debug the problem, but had no success. > > Does anyone have any ideas? > > regards, > > Gavin Gray > > > Quoting Alexander Demin : > > > Hello. > > > > I have problem with synchronisation two cyrus-imapd servers. > > > > ******* Start "Replica" host configuration ******* > > OS: FreeBSD 7.2-STABLE i386 > > cyrus-imapd-2.3.15 WITH_BDB=true WITH_REPLICATION=true > > cyrus-sasl-2.1.23 WITH_AUTHDAEMOND=true WITH_LOGIN=true WITH_PLAIN=true > > WITH_CRAM=true WITH_DIGEST=true > > cyrus-sasl-saslauthd-2.1.23 > > All soft installed from ports. > > > > Cyrus configuration: > > /usr/local/etc/cyrus.conf > > START { > > recover cmd="ctl_cyrusdb -r" > > } > > > > SERVICES { > > imap cmd="imapd" listen="imap" prefork=0 > > imaps cmd="imapd -s" listen="imaps" prefork=0 > > pop3 cmd="pop3d" listen="pop3" prefork=0 > > pop3s cmd="pop3d -s" listen="pop3s" prefork=0 > > sieve cmd="timsieved" listen="sieve" prefork=0 > > lmtpunix cmd="lmtpd" listen="/data/imap/socket/lmtp" prefork=0 > > smmap cmd="smmapd" listen="/data/imap/socket/smmap" prefork=1 > > syncserver cmd="sync_server" listen="csync" prefork=1 > > } > > > > EVENTS { > > checkpoint cmd="ctl_cyrusdb -c" period=30 > > delprune cmd="cyr_expire -E 3" at=0400 > > tlsprune cmd="tls_prune" at=0400 > > } > > > > /usr/local/etc/imapd.conf > > configdirectory: /backup/imap > > partition-default: /backup/spool/imap > > unixhierarchysep: no > > altnamespace: yes > > allowanonymouslogin: no > > allowplaintext: yes > > imapidresponse: yes > > admins: cyrus > > munge8bit: 0 > > rfc2046_strict: 0 > > sievedir: /backup/imap/sieve > > sendmail: /usr/sbin/sendmail > > postmaster: postmaster > > annotation_db: skiplist > > duplicate_db: berkeley-nosync > > mboxlist_db: skiplist > > ptscache_db: berkeley > > seenstate_db: skiplist > > subscription_db: flat > > sasl_pwcheck_method: auxprop > > sasl_auxprop_plugin: sasldb > > sasl_log_level: 7 > > sasl_mech_list: plain cram-md5 digest-md5 login > > lmtpsocket: /backup/imap/socket/lmtp > > virtdomains: userid > > lmtp_downcase_rcpt: 1 > > > > # > > # EOF > > > > /etc/services > > csync 2005/tcp > > > > /etc/rc.conf (show only cyrus/sasl params) > > cyrus_imapd_enable="YES" > > saslauthd_enable="YES" > > saslauthd_flags="-a sasldb" > > ******* End "Replica" host configuration ******* > > > > ******* Start "Master" host configuration ******* > > OS: FreeBSD 7.2-STABLE amd64 > > cyrus-imapd-2.3.15 WITH_BDB=true WITH_REPLICATION=true > > cyrus-sasl-2.1.23 WITH_AUTHDAEMOND=true WITH_LOGIN=true WITH_PLAIN=true > > WITH_CRAM=true WITH_DIGEST=true > > cyrus-sasl-saslauthd-2.1.23 > > All soft installed from ports. > > > > Cyrus configuration: > > /usr/local/etc/cyrus.conf > > START { > > recover cmd="ctl_cyrusdb -r" > > } > > > > SERVICES { > > imap cmd="imapd" listen="imap" prefork=0 > > imaps cmd="imapd -s" listen="imaps" prefork=0 > > pop3 cmd="pop3d" listen="pop3" prefork=0 > > pop3s cmd="pop3d -s" listen="pop3s" prefork=0 > > sieve cmd="timsieved" listen="sieve" prefork=0 > > lmtpunix cmd="lmtpd" listen="/data/imap/socket/lmtp" prefork=0 > > smmap cmd="smmapd" listen="/data/imap/socket/smmap" prefork=1 > > syncclient cmd="sync_client -r" listen="csync" prefork=1 > > } > > > > EVENTS { > > checkpoint cmd="ctl_cyrusdb -c" period=30 > > delprune cmd="cyr_expire -E 3" at=0400 > > tlsprune cmd="tls_prune" at=0400 > > } > > > > /usr/local/etc/imapd.conf > > configdirectory: /data/imap > > partition-default: /data/spool/imap > > unixhierarchysep: no > > altnamespace: yes > > allowanonymouslogin: no > > allowplaintext: yes > > imapidresponse: yes > > admins: cyrus cyrus at spectrum.ru > > munge8bit: 0 > > rfc2046_strict: 0 > > sievedir: /data/imap/sieve > > sendmail: /usr/sbin/sendmail > > postmaster: postmaster > > annotation_db: skiplist > > duplicate_db: berkeley-nosync > > mboxlist_db: skiplist > > ptscache_db: berkeley > > seenstate_db: skiplist > > subscription_db: flat > > sasl_pwcheck_method: auxprop > > sasl_auxprop_plugin: sasldb > > sasl_log_level: 7 > > sasl_mech_list: plain cram-md5 digest-md5 login > > tls_cert_file: /etc/ssl/imapserver.pem > > tls_key_file: /etc/ssl/imapserver.pem > > tls_ca_file: /etc/ssl/imapserver.pem > > tls_session_timeout: 0 > > lmtpsocket: /data/imap/socket/lmtp > > virtdomains: userid > > lmtp_downcase_rcpt: 1 > > sync_repeat_interval: 10 > > sync_host: support.spectrum.ru > > sync_authname: cyrus > > sync_password: *********** > > sync_log: 1 > > > > # > > # EOF > > > > /etc/services > > csync 2005/tcp > > > > /etc/rc.conf (show only cyrus/sasl params) > > cyrus_imapd_enable="YES" > > saslauthd_enable="YES" > > saslauthd_flags="-a sasldb" > > > > "Master" - it's production mail server of my company. All services > > worked is fine. > > ******* End "Master" host configuration ******* > > > > Step-by-step: > > "Replica" host > > 1. /usr/local/etc/rc.d/imapd start > > 2. imtest -a cyrus localhost - has passed successfully > > 3. synctest -u cyrus localhost - failed > > S: * SASL LOGIN PLAIN DIGEST-MD5 CRAM-MD5 > > S: * OK support.spectrum.ru Cyrus sync server v2.3.15 > > C: AUTHENTICATE DIGEST-MD5 > > failure: prot layer failure > > 4. ps -ax | grep sync_server > > 65257 ?? I 0:00,00 sync_server > > 65617 ?? I 0:00,00 sync_server > > 5. grep sync /var/log/all.log > > Sep 17 15:59:24 support syncserver[65589]: accepted connection > > Sep 17 15:59:24 support master[65616]: about to > > exec /usr/local/cyrus/bin/sync_server > > Sep 17 15:59:24 support kernel: pid 65589 (sync_server), uid 60: exited > > on signal 11 > > Sep 17 15:59:24 support syncserver[65589]: cmdloop(): startup > > Sep 17 15:59:24 support syncserver[65616]: executed > > Sep 17 15:59:24 support master[65253]: service syncserver pid 65589 in > > BUSY state: terminated abnormally > > Sep 17 15:59:24 support syncserver[65616]: accepted connection > > Sep 17 15:59:24 support master[65617]: about to > > exec /usr/local/cyrus/bin/sync_server > > Sep 17 15:59:24 support kernel: pid 65616 (sync_server), uid 60: exited > > on signal 11 > > Sep 17 15:59:24 support syncserver[65616]: cmdloop(): startup > > Sep 17 15:59:24 support syncserver[65617]: executed > > Sep 17 15:59:24 support master[65253]: service syncserver pid 65616 in > > BUSY state: terminated abnormally > > > > "Master" host > > 1. /usr/local/etc/rc.d/imapd restart > > 2. imtest -a cyrus localhost - has passed successfully > > 3. ps -ax | grep sync_client > > 63196 ?? S 0:00,01 sync_client -r > > 63197 ?? S 0:00,01 sync_client -r > > 4. grep sync /var/log/all.log > > Sep 17 16:24:18 mail sync_client[63196]: couldn't authenticate to > > backend server: generic failure > > Sep 17 16:24:18 mail sync_client[63197]: couldn't authenticate to > > backend server: generic failure > > Sep 17 16:25:18 mail sync_client[63196]: couldn't authenticate to > > backend server: generic failure > > Sep 17 16:25:18 mail sync_client[63197]: couldn't authenticate to > > backend server: generic failure > > > > Did i make something not correctly? > > Help me, please, to find the mistake and understand this problem. > > > > Thanks. > > > > -- > > Demin Alexander / Network Administrator > > Group of companies Spectrum / tel. (+7 495) 995-8999 > > Russia, Moscow, 103009, Strastnoy blvr. 8 > > Web: http://www.spectrum.ru/ > > ---- > > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > > > > > > > > -- > Gavin Gray > Edinburgh University Information Services > Rm 2013 JCMB > Kings Buildings > Edinburgh > EH9 3JZ > UK > tel +44 (0)131 650 5987 > email gavin.gray at ed.ac.uk > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- GRATIS f?r alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01 From gavin.gray at ed.ac.uk Thu Oct 8 09:55:07 2009 From: gavin.gray at ed.ac.uk (Gavin Gray) Date: Thu, 08 Oct 2009 14:55:07 +0100 Subject: Synchronisation two cyrus-imapd servers In-Reply-To: <20091008134105.7700@gmx.net> References: <1253191330.1801.14.camel@support.spectrum.ru> <20091008140457.ioc7iat55wwskwwo@www.staffmail.ed.ac.uk> <20091008134105.7700@gmx.net> Message-ID: <20091008145507.5r7m15ffb44ogs84@www.staffmail.ed.ac.uk> Hi there, Yes you're right. If I add a defaultpartition and corresponding partition- to imapd.conf all is well, many thanks, Gavin. Quoting steffo76 at gmx.de: > Hi, > > you are probably missing 'defaultpartition' in imapd.conf: > https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3170 > > Regards, > Stephan > > -------- Original-Nachricht -------- >> Datum: Thu, 08 Oct 2009 14:04:57 +0100 >> Von: Gavin Gray >> An: info-cyrus at lists.andrew.cmu.edu >> Betreff: Re: Synchronisation two cyrus-imapd servers > >> Hi there, >> I also have this problem. I can't get the sync_server to run >> under 2.3.15. Here is a snippet fom our logs: >> >> Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 143423 local6.error] >> DBERROR: reading /var/imap/db/skipstamp, assuming the worst: No such >> file or directory >> Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 518349 local6.debug] >> executed >> Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 301543 local6.info] >> skiplist: checkpointed /var/imap/mailboxes.db (0 records, 144 bytes) >> in 0 seconds >> Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 301543 local6.info] >> skiplist: checkpointed /var/imap/annotations.db (0 records, 144 bytes) >> in 0 seconds >> Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 921384 local6.debug] >> accepted connection >> Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 177842 local6.debug] >> cmdloop(): startup >> Oct 8 12:42:36 mailbe9r last message repeated 1 time >> Oct 8 12:42:38 mailbe9r master[25640]: [ID 970914 local6.error] >> process 25647 exited, signaled to death by 11 >> Oct 8 12:42:38 mailbe9r master[25640]: [ID 621917 local6.debug] >> service syncserver pid 25647 in BUSY state: terminated abnormally >> >> I've had a go at trying to debug the problem, but had no success. >> >> Does anyone have any ideas? >> >> regards, >> >> Gavin Gray >> >> >> Quoting Alexander Demin : >> >> > Hello. >> > >> > I have problem with synchronisation two cyrus-imapd servers. >> > >> > ******* Start "Replica" host configuration ******* >> > OS: FreeBSD 7.2-STABLE i386 >> > cyrus-imapd-2.3.15 WITH_BDB=true WITH_REPLICATION=true >> > cyrus-sasl-2.1.23 WITH_AUTHDAEMOND=true WITH_LOGIN=true WITH_PLAIN=true >> > WITH_CRAM=true WITH_DIGEST=true >> > cyrus-sasl-saslauthd-2.1.23 >> > All soft installed from ports. >> > >> > Cyrus configuration: >> > /usr/local/etc/cyrus.conf >> > START { >> > recover cmd="ctl_cyrusdb -r" >> > } >> > >> > SERVICES { >> > imap cmd="imapd" listen="imap" prefork=0 >> > imaps cmd="imapd -s" listen="imaps" prefork=0 >> > pop3 cmd="pop3d" listen="pop3" prefork=0 >> > pop3s cmd="pop3d -s" listen="pop3s" prefork=0 >> > sieve cmd="timsieved" listen="sieve" prefork=0 >> > lmtpunix cmd="lmtpd" listen="/data/imap/socket/lmtp" prefork=0 >> > smmap cmd="smmapd" listen="/data/imap/socket/smmap" prefork=1 >> > syncserver cmd="sync_server" listen="csync" prefork=1 >> > } >> > >> > EVENTS { >> > checkpoint cmd="ctl_cyrusdb -c" period=30 >> > delprune cmd="cyr_expire -E 3" at=0400 >> > tlsprune cmd="tls_prune" at=0400 >> > } >> > >> > /usr/local/etc/imapd.conf >> > configdirectory: /backup/imap >> > partition-default: /backup/spool/imap >> > unixhierarchysep: no >> > altnamespace: yes >> > allowanonymouslogin: no >> > allowplaintext: yes >> > imapidresponse: yes >> > admins: cyrus >> > munge8bit: 0 >> > rfc2046_strict: 0 >> > sievedir: /backup/imap/sieve >> > sendmail: /usr/sbin/sendmail >> > postmaster: postmaster >> > annotation_db: skiplist >> > duplicate_db: berkeley-nosync >> > mboxlist_db: skiplist >> > ptscache_db: berkeley >> > seenstate_db: skiplist >> > subscription_db: flat >> > sasl_pwcheck_method: auxprop >> > sasl_auxprop_plugin: sasldb >> > sasl_log_level: 7 >> > sasl_mech_list: plain cram-md5 digest-md5 login >> > lmtpsocket: /backup/imap/socket/lmtp >> > virtdomains: userid >> > lmtp_downcase_rcpt: 1 >> > >> > # >> > # EOF >> > >> > /etc/services >> > csync 2005/tcp >> > >> > /etc/rc.conf (show only cyrus/sasl params) >> > cyrus_imapd_enable="YES" >> > saslauthd_enable="YES" >> > saslauthd_flags="-a sasldb" >> > ******* End "Replica" host configuration ******* >> > >> > ******* Start "Master" host configuration ******* >> > OS: FreeBSD 7.2-STABLE amd64 >> > cyrus-imapd-2.3.15 WITH_BDB=true WITH_REPLICATION=true >> > cyrus-sasl-2.1.23 WITH_AUTHDAEMOND=true WITH_LOGIN=true WITH_PLAIN=true >> > WITH_CRAM=true WITH_DIGEST=true >> > cyrus-sasl-saslauthd-2.1.23 >> > All soft installed from ports. >> > >> > Cyrus configuration: >> > /usr/local/etc/cyrus.conf >> > START { >> > recover cmd="ctl_cyrusdb -r" >> > } >> > >> > SERVICES { >> > imap cmd="imapd" listen="imap" prefork=0 >> > imaps cmd="imapd -s" listen="imaps" prefork=0 >> > pop3 cmd="pop3d" listen="pop3" prefork=0 >> > pop3s cmd="pop3d -s" listen="pop3s" prefork=0 >> > sieve cmd="timsieved" listen="sieve" prefork=0 >> > lmtpunix cmd="lmtpd" listen="/data/imap/socket/lmtp" prefork=0 >> > smmap cmd="smmapd" listen="/data/imap/socket/smmap" prefork=1 >> > syncclient cmd="sync_client -r" listen="csync" prefork=1 >> > } >> > >> > EVENTS { >> > checkpoint cmd="ctl_cyrusdb -c" period=30 >> > delprune cmd="cyr_expire -E 3" at=0400 >> > tlsprune cmd="tls_prune" at=0400 >> > } >> > >> > /usr/local/etc/imapd.conf >> > configdirectory: /data/imap >> > partition-default: /data/spool/imap >> > unixhierarchysep: no >> > altnamespace: yes >> > allowanonymouslogin: no >> > allowplaintext: yes >> > imapidresponse: yes >> > admins: cyrus cyrus at spectrum.ru >> > munge8bit: 0 >> > rfc2046_strict: 0 >> > sievedir: /data/imap/sieve >> > sendmail: /usr/sbin/sendmail >> > postmaster: postmaster >> > annotation_db: skiplist >> > duplicate_db: berkeley-nosync >> > mboxlist_db: skiplist >> > ptscache_db: berkeley >> > seenstate_db: skiplist >> > subscription_db: flat >> > sasl_pwcheck_method: auxprop >> > sasl_auxprop_plugin: sasldb >> > sasl_log_level: 7 >> > sasl_mech_list: plain cram-md5 digest-md5 login >> > tls_cert_file: /etc/ssl/imapserver.pem >> > tls_key_file: /etc/ssl/imapserver.pem >> > tls_ca_file: /etc/ssl/imapserver.pem >> > tls_session_timeout: 0 >> > lmtpsocket: /data/imap/socket/lmtp >> > virtdomains: userid >> > lmtp_downcase_rcpt: 1 >> > sync_repeat_interval: 10 >> > sync_host: support.spectrum.ru >> > sync_authname: cyrus >> > sync_password: *********** >> > sync_log: 1 >> > >> > # >> > # EOF >> > >> > /etc/services >> > csync 2005/tcp >> > >> > /etc/rc.conf (show only cyrus/sasl params) >> > cyrus_imapd_enable="YES" >> > saslauthd_enable="YES" >> > saslauthd_flags="-a sasldb" >> > >> > "Master" - it's production mail server of my company. All services >> > worked is fine. >> > ******* End "Master" host configuration ******* >> > >> > Step-by-step: >> > "Replica" host >> > 1. /usr/local/etc/rc.d/imapd start >> > 2. imtest -a cyrus localhost - has passed successfully >> > 3. synctest -u cyrus localhost - failed >> > S: * SASL LOGIN PLAIN DIGEST-MD5 CRAM-MD5 >> > S: * OK support.spectrum.ru Cyrus sync server v2.3.15 >> > C: AUTHENTICATE DIGEST-MD5 >> > failure: prot layer failure >> > 4. ps -ax | grep sync_server >> > 65257 ?? I 0:00,00 sync_server >> > 65617 ?? I 0:00,00 sync_server >> > 5. grep sync /var/log/all.log >> > Sep 17 15:59:24 support syncserver[65589]: accepted connection >> > Sep 17 15:59:24 support master[65616]: about to >> > exec /usr/local/cyrus/bin/sync_server >> > Sep 17 15:59:24 support kernel: pid 65589 (sync_server), uid 60: exited >> > on signal 11 >> > Sep 17 15:59:24 support syncserver[65589]: cmdloop(): startup >> > Sep 17 15:59:24 support syncserver[65616]: executed >> > Sep 17 15:59:24 support master[65253]: service syncserver pid 65589 in >> > BUSY state: terminated abnormally >> > Sep 17 15:59:24 support syncserver[65616]: accepted connection >> > Sep 17 15:59:24 support master[65617]: about to >> > exec /usr/local/cyrus/bin/sync_server >> > Sep 17 15:59:24 support kernel: pid 65616 (sync_server), uid 60: exited >> > on signal 11 >> > Sep 17 15:59:24 support syncserver[65616]: cmdloop(): startup >> > Sep 17 15:59:24 support syncserver[65617]: executed >> > Sep 17 15:59:24 support master[65253]: service syncserver pid 65616 in >> > BUSY state: terminated abnormally >> > >> > "Master" host >> > 1. /usr/local/etc/rc.d/imapd restart >> > 2. imtest -a cyrus localhost - has passed successfully >> > 3. ps -ax | grep sync_client >> > 63196 ?? S 0:00,01 sync_client -r >> > 63197 ?? S 0:00,01 sync_client -r >> > 4. grep sync /var/log/all.log >> > Sep 17 16:24:18 mail sync_client[63196]: couldn't authenticate to >> > backend server: generic failure >> > Sep 17 16:24:18 mail sync_client[63197]: couldn't authenticate to >> > backend server: generic failure >> > Sep 17 16:25:18 mail sync_client[63196]: couldn't authenticate to >> > backend server: generic failure >> > Sep 17 16:25:18 mail sync_client[63197]: couldn't authenticate to >> > backend server: generic failure >> > >> > Did i make something not correctly? >> > Help me, please, to find the mistake and understand this problem. >> > >> > Thanks. >> > >> > -- >> > Demin Alexander / Network Administrator >> > Group of companies Spectrum / tel. (+7 495) 995-8999 >> > Russia, Moscow, 103009, Strastnoy blvr. 8 >> > Web: http://www.spectrum.ru/ >> > ---- >> > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ >> > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki >> > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html >> > >> > >> >> >> >> -- >> Gavin Gray >> Edinburgh University Information Services >> Rm 2013 JCMB >> Kings Buildings >> Edinburgh >> EH9 3JZ >> UK >> tel +44 (0)131 650 5987 >> email gavin.gray at ed.ac.uk >> >> -- >> The University of Edinburgh is a charitable body, registered in >> Scotland, with registration number SC005336. >> >> -- >> The University of Edinburgh is a charitable body, registered in >> Scotland, with registration number SC005336. >> >> >> ---- >> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ >> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki >> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > > -- > GRATIS f?r alle GMX-Mitglieder: Die maxdome Movie-FLAT! > Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01 > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > > -- Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.gray at ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From gmludo at gmail.com Fri Oct 9 09:03:14 2009 From: gmludo at gmail.com (Ludovic Gasc) Date: Fri, 9 Oct 2009 15:03:14 +0200 Subject: No e-mail notification with sieve, Thunderbird and Cyrus-imap Message-ID: <9954532b0910090603r2679718dg4e8c1fd1ca5096f3@mail.gmail.com> Hi everybody, We're using Cyrus-imap during some time, it's a good tool for us. We've a strange behaviour (bug)? with sieve, Thunderbird and Cyrus-imap. I want to listen your opinions, because I'm not sure to understand correctly the problem. We use some sieve scripts to filter the e-mails in the sub-folders of INBOX. With Thunderbird, we don't have the notification when an new e-mail arrives in a sub-folder filtered by a sieve script. We must click on the folder to receive new e-mails. With some other e-mail clients (Kmail, Claws-mail), we receive correctly the new e-mails. After some searches, I've found: * Thunderbird uses IMAP IDLE when it's possible to receive the new e-mails, unlike others open all folders each time. * Cyrus seems to notify the new e-mails only in INBOX with IMAP IDLE. I've found the bug report for Thunderbird: https://bugzilla.mozilla.org/show_bug.cgi?id=483859 With courier-imap, they've an option to choose to behaviour of IMAP IDLE: IMAP_CHECK_ALL_FOLDERS: http://forums.mozillazine.org/viewtopic.php?f=31&t=99252&start=0 At the beginning, I thought the problem was in Thunderbird, but now I'm not sure. In the cyrus-imap documentation: http://cyrusimap.web.cmu.edu/imapd/install-compile.html Do you think --with-idle=idled could resolve our problem ? Thank you very much for your answer. -- Ludovic Gasc From Pascal.Gienger at uni-konstanz.de Fri Oct 9 09:08:33 2009 From: Pascal.Gienger at uni-konstanz.de (Pascal Gienger) Date: Fri, 09 Oct 2009 15:08:33 +0200 Subject: No e-mail notification with sieve, Thunderbird and Cyrus-imap In-Reply-To: <9954532b0910090603r2679718dg4e8c1fd1ca5096f3@mail.gmail.com> References: <9954532b0910090603r2679718dg4e8c1fd1ca5096f3@mail.gmail.com> Message-ID: <4ACF35D1.20305@uni-konstanz.de> Ludovic Gasc schrieb: > Hi everybody, > > We're using Cyrus-imap during some time, it's a good tool for us. > > We've a strange behaviour (bug)? with sieve, Thunderbird and Cyrus-imap. > I want to listen your opinions, because I'm not sure to understand > correctly the problem. > > We use some sieve scripts to filter the e-mails in the sub-folders of INBOX. I never had this problem. Be sure to mark every subfolder you need with "Check for new messages" (right click on the folder you want to be checked, then click on "Properties"). Thunderbird opens a new IMAP connection for each folder. For each folder marked with "Check for new message" ("Auf neue Nachrichten ?berpr?fen" in my case, I have a german localized Thunderbird) it will issue an "IDLE" command (easily traceable). Pascal From cvizitiu at gbif.org Fri Oct 9 09:12:39 2009 From: cvizitiu at gbif.org (Ciprian Marius Vizitiu (GBIF)) Date: Fri, 09 Oct 2009 15:12:39 +0200 Subject: No e-mail notification with sieve, Thunderbird and Cyrus-imap In-Reply-To: <9954532b0910090603r2679718dg4e8c1fd1ca5096f3@mail.gmail.com> References: <9954532b0910090603r2679718dg4e8c1fd1ca5096f3@mail.gmail.com> Message-ID: <4ACF36C7.7010200@gbif.org> Ludovic Gasc wrote: > We've a strange behaviour (bug)? with sieve, Thunderbird and Cyrus-imap. > > We use some sieve scripts to filter the e-mails in the sub-folders of INBOX. > > With Thunderbird, we don't have the notification when an new e-mail > arrives in a sub-folder filtered by a sieve script. > We must click on the folder to receive new e-mails. > With some other e-mail clients (Kmail, Claws-mail), we receive > correctly the new e-mails. > I suppose you've already set Thunderbird to check for new mails in all IMAP subfolders? "Preferences->Advanced->Config Editor->mail.check_all_imap_folders_for_new"? From gmludo at gmail.com Fri Oct 9 09:19:04 2009 From: gmludo at gmail.com (Ludovic Gasc) Date: Fri, 9 Oct 2009 15:19:04 +0200 Subject: No e-mail notification with sieve, Thunderbird and Cyrus-imap In-Reply-To: <4ACF36C7.7010200@gbif.org> References: <9954532b0910090603r2679718dg4e8c1fd1ca5096f3@mail.gmail.com> <4ACF36C7.7010200@gbif.org> Message-ID: <9954532b0910090619r6962b098s670a7bdee6777f2@mail.gmail.com> On Fri, Oct 9, 2009 at 3:12 PM, Ciprian Marius Vizitiu (GBIF) < cvizitiu at gbif.org> wrote: > Ludovic Gasc wrote: > > We've a strange behaviour (bug)? with sieve, Thunderbird and Cyrus-imap. > > > > We use some sieve scripts to filter the e-mails in the sub-folders of > INBOX. > > > > With Thunderbird, we don't have the notification when an new e-mail > > arrives in a sub-folder filtered by a sieve script. > > We must click on the folder to receive new e-mails. > > With some other e-mail clients (Kmail, Claws-mail), we receive > > correctly the new e-mails. > > > > I suppose you've already set Thunderbird to check for new mails in all > IMAP subfolders? "Preferences->Advanced->Config > Editor->mail.check_all_imap_folders_for_new"? > Yes, I've already changed these values, and I've tested with Thunderbird 3 nightly builds. -- Ludovic Gasc -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20091009/0c61b069/attachment.html From gmludo at gmail.com Fri Oct 9 09:25:02 2009 From: gmludo at gmail.com (Ludovic Gasc) Date: Fri, 9 Oct 2009 15:25:02 +0200 Subject: No e-mail notification with sieve, Thunderbird and Cyrus-imap In-Reply-To: <4ACF35D1.20305@uni-konstanz.de> References: <9954532b0910090603r2679718dg4e8c1fd1ca5096f3@mail.gmail.com> <4ACF35D1.20305@uni-konstanz.de> Message-ID: <9954532b0910090625u3cc62548v134ee97f674da297@mail.gmail.com> On Fri, Oct 9, 2009 at 3:08 PM, Pascal Gienger wrote: > > Ludovic Gasc schrieb: >> >> Hi everybody, >> >> We're using Cyrus-imap during some time, it's a good tool for us. >> >> We've a strange behaviour (bug)? with sieve, Thunderbird and Cyrus-imap. >> I want to listen your opinions, because I'm not sure to understand >> correctly the problem. >> >> We use some sieve scripts to filter the e-mails in the sub-folders of INBOX. > > I never had this problem. Be sure to mark every subfolder you need with "Check for new messages" (right click on the folder you want to be checked, then click on "Properties"). > > Thunderbird opens a new IMAP connection for each folder. For each folder marked with "Check for new message" ("Auf neue Nachrichten ?berpr?fen" in my case, I have a german localized Thunderbird) it will issue an "IDLE" command (easily traceable). Yes, I've marked each folder with "Check for new messages". What is your version of Cyrus ? The compile options or the package you've used ? I use cyrus-imapd 2.2.13 on Debian Lenny: http://packages.debian.org/lenny/cyrus-imapd-2.2 I'm very interested by your feedback. Yours. -- Ludovic Gasc From nodens2099 at gmail.com Sat Oct 10 06:34:17 2009 From: nodens2099 at gmail.com (Clement Hermann (nodens)) Date: Sat, 10 Oct 2009 12:34:17 +0200 Subject: No e-mail notification with sieve, Thunderbird and Cyrus-imap In-Reply-To: <9954532b0910090625u3cc62548v134ee97f674da297@mail.gmail.com> References: <9954532b0910090603r2679718dg4e8c1fd1ca5096f3@mail.gmail.com> <4ACF35D1.20305@uni-konstanz.de> <9954532b0910090625u3cc62548v134ee97f674da297@mail.gmail.com> Message-ID: <4AD06329.9030702@gmail.com> Ludovic Gasc a ?crit : > On Fri, Oct 9, 2009 at 3:08 PM, Pascal Gienger > wrote: >> Ludovic Gasc schrieb: >>> Hi everybody, >>> >>> We're using Cyrus-imap during some time, it's a good tool for us. >>> >>> We've a strange behaviour (bug)? with sieve, Thunderbird and Cyrus-imap. >>> I want to listen your opinions, because I'm not sure to understand >>> correctly the problem. >>> >>> We use some sieve scripts to filter the e-mails in the sub-folders of INBOX. >> I never had this problem. Be sure to mark every subfolder you need with "Check for new messages" (right click on the folder you want to be checked, then click on "Properties"). >> >> Thunderbird opens a new IMAP connection for each folder. For each folder marked with "Check for new message" ("Auf neue Nachrichten ?berpr?fen" in my case, I have a german localized Thunderbird) it will issue an "IDLE" command (easily traceable). > > Yes, I've marked each folder with "Check for new messages". > > What is your version of Cyrus ? The compile options or the package you've used ? > > I use cyrus-imapd 2.2.13 on Debian Lenny: > http://packages.debian.org/lenny/cyrus-imapd-2.2 > > I'm very interested by your feedback. > We use Debian Lenny's cyrus-imapd as well, and don't have this problem either (using idled). We recompiled the package to build ptloader, but I don't think this has any incidence. Regards, -- Clement Hermann (nodens) - "L'air pur ? c'est pas en RL, ?a ? c'est pas hors charte ?" Jean in L'Histoire des Pingouins, http://tnemeth.free.fr/fmbl/linuxsf/ Vous trouverez ma clef publique sur le serveur public pgp.mit.edu. Please find my public key on the public keyserver pgp.mit.edu. From gmludo at gmail.com Mon Oct 12 06:11:23 2009 From: gmludo at gmail.com (Ludovic Gasc) Date: Mon, 12 Oct 2009 12:11:23 +0200 Subject: No e-mail notification with sieve, Thunderbird and Cyrus-imap In-Reply-To: <4AD06329.9030702@gmail.com> References: <9954532b0910090603r2679718dg4e8c1fd1ca5096f3@mail.gmail.com> <4ACF35D1.20305@uni-konstanz.de> <9954532b0910090625u3cc62548v134ee97f674da297@mail.gmail.com> <4AD06329.9030702@gmail.com> Message-ID: <9954532b0910120311p57d3bd5akb967be28429cf11d@mail.gmail.com> On Sat, Oct 10, 2009 at 12:34 PM, Clement Hermann (nodens) wrote: > We use Debian Lenny's cyrus-imapd as well, and don't have this problem > either (using idled). Thank you very much for your feedback, I've enabled idled in Cyrus-imap, and now it works. For other people who could have the same problem, you must uncomment this line in /etc/cyrus.conf: idled cmd="idled" And you must change in /etc/imapd.conf: idlemethod: idled Information from README.Debian.gz: 4. Configurable idled support. Cyrus IMAPd supports three options of using IDLE in IMAP sessions. The first option is not to support IDLE at all. The second is to use internal polling in the IMAP daemon. The third option is to use an external daemon, idled. Upstream only supports configuration of this during compilation, Debian however includes a patch which makes this runtime-configurable. Please set the 'idlemethod' imapd.conf option according to your needs and enable idled in cyrus.conf if you want to use it. -- Ludovic Gasc From champ at umbc.edu Mon Oct 12 13:49:52 2009 From: champ at umbc.edu (Tim Champ) Date: Mon, 12 Oct 2009 13:49:52 -0400 Subject: Strange reconstruct problem with version migration Message-ID: <4AD36C40.1060005@umbc.edu> Hello all. We've setup a new cyrus backend server with 2.3.15. (as a part of our cyrus mail setup here) Our current version is a patched up 2.3.8. When we xfer users from an "old" backend server to another "old" server and then reconstruct, we have no problems. It works as we'd expect - purges expunged mails, etc. When we do this from an "old" backend server to the "new" one, it always sees the expunge file as not valid. The message: "Unable to verify header - deleting: $MAILBOX_PATH/cyrus.expunge" The access to the mailbox, etc, all seems to work fine, except that any mail that was previously deleted and not removed via cyr_expire (we do delayed delete) returns to the mailbox. This concerns us as we're not sure why it wouldn't understand the expunge files. Although we could not reconstruct post-move (we've done it for cleanup purposes historically), it worries us that we'd place this version of cyrus on an existing mail server and have this issue the first time a reconstruct was needed. (undead mail, as we've termed it) Any ideas? Still doing testing and trying things, but at a loss right now. If more information is needed, I'm happy to provide it. We've googled around, and searched the mail archives, read documenation, but to no avail. Thanks! Tim From michael.menge at zdv.uni-tuebingen.de Mon Oct 12 14:31:36 2009 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Mon, 12 Oct 2009 20:31:36 +0200 Subject: Strange reconstruct problem with version migration In-Reply-To: <4AD36C40.1060005@umbc.edu> References: <4AD36C40.1060005@umbc.edu> Message-ID: <20091012203136.1066790erh0de9s8@webmail.uni-tuebingen.de> Quoting Tim Champ : > Hello all. > > We've setup a new cyrus backend server with 2.3.15. (as a part of our > cyrus mail setup here) Our current version is a patched up 2.3.8. When > we xfer users from an "old" backend server to another "old" server and > then reconstruct, we have no problems. It works as we'd expect - purges > expunged mails, etc. > > When we do this from an "old" backend server to the "new" one, it always > sees the expunge file as not valid. The message: > > "Unable to verify header - deleting: $MAILBOX_PATH/cyrus.expunge" > using xfer you don't need to run reconstruct. But it should not harm. > The access to the mailbox, etc, all seems to work fine, except that any > mail that was previously deleted and not removed via cyr_expire (we do > delayed delete) returns to the mailbox. This concerns us as we're not > sure why it wouldn't understand the expunge files. > reconstruct has a new option -k, to keep delayed delete mails. > Although we could not reconstruct post-move (we've done it for cleanup > purposes historically), it worries us that we'd place this version of > cyrus on an existing mail server and have this issue the first time a > reconstruct was needed. (undead mail, as we've termed it) > > Any ideas? Still doing testing and trying things, but at a loss right > now. If more information is needed, I'm happy to provide it. We've > googled around, and searched the mail archives, read documenation, but > to no avail. > > Thanks! > > Tim > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5339 bytes Desc: S/MIME krytographische Unterschrift Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20091012/2c205d4c/attachment.bin From champ at umbc.edu Mon Oct 12 14:39:26 2009 From: champ at umbc.edu (Tim Champ) Date: Mon, 12 Oct 2009 14:39:26 -0400 Subject: Strange reconstruct problem with version migration In-Reply-To: <20091012203136.1066790erh0de9s8@webmail.uni-tuebingen.de> References: <4AD36C40.1060005@umbc.edu> <20091012203136.1066790erh0de9s8@webmail.uni-tuebingen.de> Message-ID: <4AD377DE.20900@umbc.edu> > >> The access to the mailbox, etc, all seems to work fine, except that any >> mail that was previously deleted and not removed via cyr_expire (we do >> delayed delete) returns to the mailbox. This concerns us as we're not >> sure why it wouldn't understand the expunge files. >> > > reconstruct has a new option -k, to keep delayed delete mails. I saw this, but in this case its not that reconstruct is deleting the "delayed delete" mails, its that its deleting the cyrus.expunge file (saying it has a bad header). Once you do that, all the deleted mail shows back up in the mailbox again. I'm fine with deleting the "delayed delete" mails, its the return of the deleted mail to the mailbox that causes us the consternation. Thanks! Tim From david.lang at digitalinsight.com Tue Oct 13 18:29:30 2009 From: david.lang at digitalinsight.com (David Lang) Date: Tue, 13 Oct 2009 15:29:30 -0700 (PDT) Subject: server-side signature verification through IMAP? Message-ID: the attached messages were posted to the mulberry mailing list short version, in order to do s/mime verification the client must retreive the entire message to do the verification client-side. is there any way to do this server-side? David Lang -------------- next part -------------- An embedded message was scrubbed... From: "John C Klensin" Subject: [Mulberry-discuss] S/MIME signed messages and Mulberry Date: Tue, 13 Oct 2009 14:03:40 -0700 Size: 10677 Url: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20091013/5c3b2381/attachment.mht -------------- next part -------------- An embedded message was scrubbed... From: "Kenneth Porter" Subject: Re: [Mulberry-discuss] S/MIME signed messages and Mulberry Date: Tue, 13 Oct 2009 15:27:10 -0700 Size: 5677 Url: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20091013/5c3b2381/attachment-0001.mht From broder at MIT.EDU Wed Oct 14 15:28:57 2009 From: broder at MIT.EDU (Evan Broder) Date: Wed, 14 Oct 2009 15:28:57 -0400 Subject: Cyrus::IMAP with SSL Message-ID: <358_1255548544_n9EJT3Jr031539_4AD62679.9000707@mit.edu> Hi - I'm working on migrating MIT's mitmailutils scripts, which use Cyrus::IMAP, to our new GSSAPI-enabled mail configuration (finally!). However, I seem to be running into bugs when I try to use imclient's GSSAPI handling for both authentication and encryption, which it does by default if you connect over a non-SSL connection. I'd like to be using SSL anyway, but I can't figure out any way to open an SSL connection with Cyrus::IMAP. Is this possible at all? Is there example code anywhere I can steal? Thanks, - Evan From jukka.huhta at helsinki.fi Thu Oct 15 07:51:10 2009 From: jukka.huhta at helsinki.fi (Jukka Huhta) Date: Thu, 15 Oct 2009 14:51:10 +0300 (EEST) Subject: Sieve scripts in global namespace and replication In-Reply-To: <20090319114634.GB28796@helsinki.fi> References: <20090319114634.GB28796@helsinki.fi> Message-ID: On Thu, 19 Mar 2009, Janne Peltonen wrote: > It appears that my sieve scripts that I have in the global namespace and > that are annotated to certain Cyrus bulletin boards don't get > replicated. Apparently, only user sieve scripts ever get replicated > (when something generates a USER replication event). How do I > (rollingly) replicate the sieve scripts in the global namespace? > > Thanks for any advice. I'd like to ask again this question. Is it possible to replicate the global sieve scripts? How? (It would make sense to start taking backups from the replicas instead of primary servers, but there seems to be a problem with these global sieve scripts.) Thanks, -Jukka Huhta From mehmood67 at yahoo.com Thu Oct 15 09:52:55 2009 From: mehmood67 at yahoo.com (Khalid Mehmood) Date: Thu, 15 Oct 2009 06:52:55 -0700 (PDT) Subject: Recovering a folder from DELETED hierarchy Message-ID: <268348.17820.qm@web30801.mail.mud.yahoo.com> Hello everyone! I'm facing a problem, and don't know how to fix this. I have tried the list and googled but couldn't find any solution. One of our user deleted a folder and the folder name contains spaces e.g (New, Folder). Is there any possibility of recovering such a folder? The imapd.conf uses "delete_mode: delayed". Thanks. Regards From boutilpj at ednet.ns.ca Thu Oct 15 09:59:35 2009 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Thu, 15 Oct 2009 10:59:35 -0300 Subject: Recovering a folder from DELETED hierarchy In-Reply-To: <268348.17820.qm@web30801.mail.mud.yahoo.com> References: <268348.17820.qm@web30801.mail.mud.yahoo.com> Message-ID: <29273_1255615178_n9FDxbJv010882_4AD72AC7.7010800@ednet.ns.ca> On 10/15/2009 10:52 AM, Khalid Mehmood wrote: > Hello everyone! > > I'm facing a problem, and don't know how to fix this. I have tried the list and googled but couldn't find any solution. One of our user deleted a folder and the folder name contains spaces e.g (New, Folder). Is there any possibility of recovering such a folder? The imapd.conf uses "delete_mode: delayed". > Are you using cyradm? You should be able to put single quotes around the DELETED mailbox name. > Thanks. > > Regards > > > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html From Hagedorn at uni-koeln.de Thu Oct 15 10:00:55 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Thu, 15 Oct 2009 16:00:55 +0200 Subject: Recovering a folder from DELETED hierarchy In-Reply-To: <268348.17820.qm@web30801.mail.mud.yahoo.com> References: <268348.17820.qm@web30801.mail.mud.yahoo.com> Message-ID: <6A278A14396D7AF46DED8094@tyrion.rrz.uni-koeln.de> --On 15. Oktober 2009 06:52:55 -0700 Khalid Mehmood wrote: > I'm facing a problem, and don't know how to fix this. I have tried the > list and googled but couldn't find any solution. One of our user deleted > a folder and the folder name contains spaces e.g (New, Folder). Is there > any possibility of recovering such a folder? The imapd.conf uses > "delete_mode: delayed". Try this from within cyradm: lm DELETED/user/USERNAME/* Find the mailbox you want to restore, then e.g.: rename 'DELETED/user/xxx/bla bla/48883148' user/xxx/restored -- .:.Sebastian Hagedorn - RZKR-R1 (Geb?ude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universit?t zu K?ln / Cologne University - ? +49-221-478-5587.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pkcs7-signature Size: 5292 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20091015/b3b9b0cc/attachment.bin From mhlavink at redhat.com Thu Oct 15 10:40:17 2009 From: mhlavink at redhat.com (Michal Hlavinka) Date: Thu, 15 Oct 2009 16:40:17 +0200 Subject: Excessive logging of inflate/deflate messages Message-ID: <200910151640.17189.mhlavink@redhat.com> Hi, one Fedora user has reported their imap server has logged 3GB of messages like: Oct 8 19:43:01 mail imaps[6149]: deflate(4096 bytes, level=-1, flush=SYNC) Oct 8 19:43:01 mail imaps[6149]: => compressed to 325 bytes Oct 8 19:43:01 mail imaps[6149]: deflate(4096 bytes, level=-1, flush=SYNC) Oct 8 19:43:01 mail imaps[6149]: => compressed to 320 bytes Oct 8 19:43:01 mail imaps[6149]: deflate(4096 bytes, level=-1, flush=SYNC) Oct 8 19:43:01 mail imaps[6149]: => compressed to 231 bytes Oct 8 19:43:01 mail imaps[6149]: deflate(4096 bytes, level=-1, flush=SYNC) Oct 8 19:43:01 mail imaps[6149]: => compressed to 451 bytes Oct 8 19:43:01 mail imaps[5932]: inflate(14 bytes) Oct 8 19:43:01 mail imaps[5932]: => decompressed to 32 bytes Oct 8 19:43:02 mail imaps[6202]: deflate(4096 bytes, level=-1, flush=SYNC) Oct 8 19:43:02 mail imaps[6202]: => compressed to 783 bytes Would it be possible to lower number of these messages (they are new in 2.3.15) or (preferably) add new option for enabling/disabling LOG_DEBUG messages? It should be quite easy, just call setlogmask(x) with appropriate argument depending on config option. I can write patch if required. Regards, Michal Hlavinka From champ at umbc.edu Thu Oct 15 10:59:41 2009 From: champ at umbc.edu (Tim Champ) Date: Thu, 15 Oct 2009 10:59:41 -0400 Subject: Strange reconstruct problem with version migration In-Reply-To: <4AD377DE.20900@umbc.edu> References: <4AD36C40.1060005@umbc.edu> <20091012203136.1066790erh0de9s8@webmail.uni-tuebingen.de> <4AD377DE.20900@umbc.edu> Message-ID: <4AD738DD.60109@umbc.edu> Tim Champ wrote: >>> The access to the mailbox, etc, all seems to work fine, except that any >>> mail that was previously deleted and not removed via cyr_expire (we do >>> delayed delete) returns to the mailbox. This concerns us as we're not >>> sure why it wouldn't understand the expunge files. >>> >> reconstruct has a new option -k, to keep delayed delete mails. > > I saw this, but in this case its not that reconstruct is deleting the > "delayed delete" mails, its that its deleting the cyrus.expunge file > (saying it has a bad header). > > Once you do that, all the deleted mail shows back up in the mailbox > again. I'm fine with deleting the "delayed delete" mails, its the > return of the deleted mail to the mailbox that causes us the consternation. > Hello all again. I hadn't heard back from anyone, but I wanted to throw out some additional work I'd done. The problem is still not resolved, though. As part of testing, I deleted mail in a mailbox, moved it to the new (2.3.15) server (without the reconstruct) and then ran cyr_expire. Although it claimed to run successfully, it didn't delete the mails that were marked for delayed delete. After this, I ran reconstruct, and ran into the same problem with "Unable to verify header" for all the cyrus.expunge files in the users mailboxes. Also, now when trying to access the inbox through thunderbird I get "The current command did not succeed. The mail server responded: Invalid sequence in Uid". So, after that pops up a few times, I can then go through the mail, but it doesn't remember seen state it appears. Subsequent accesses of the mailbox show the "undead" mail returned, and strange random seen states on mail (although it looks to all be there, at least). With these problems, I can't exactly roll out the software on the production mail servers. Any ideas? Anyone? If you need any information, please don't hesitate to ask - offers of free food and drinks are on the table for help in remedying the situation! Thanks! Tim Champ From david at touzeau.eu Sun Oct 18 19:24:05 2009 From: david at touzeau.eu (David Touzeau) Date: Mon, 19 Oct 2009 01:24:05 +0200 Subject: cyrus replication : master to replica and replica to master Message-ID: <1255908245.21631.6.camel@dtouzeau-laptop> Dear I have set cyrus-imap with master and replica. This configuration is a kind of cluster Active/passive I would like to know if it is possible to SET the replica has the master too in order to replicate new mail saved on the replica to the master and vis versa In this case it should be turn to active/active.. If it is possible, did someone build this kind of configuration ? best regards From baconm at email.unc.edu Mon Oct 19 16:38:52 2009 From: baconm at email.unc.edu (Michael Bacon) Date: Mon, 19 Oct 2009 16:38:52 -0400 Subject: painful mupdate syncs between front-ends and database server Message-ID: <1482D6C9665A7BA237C6D53C@trophic.its.unc.edu> Hello, list, Today we're enjoying our first full work day of independence from the old monolithic cyrus server installed in 1999 (Sun 6800 -- it's had new CPU boards since then, but that's it), and on our new shiny cluster of T5220's that are mostly happily operating as a murder. I say mostly because while most of the times the thing handles our 80,000 users and 14,000+ simultaneous connections like a champ, some of the time, we get some extreme pain, mostly due to syncs between the MUPDATE master and the front-end servers. When we spec'ed out our servers, we didn't put much I/O capacity into the front-end servers -- just a pair of mirrored 10k disks doing the OS, the logging, the mailboxes.db, and all the webmail action going on in another solaris zone on the same hardware. We thought this was sufficient given the fact that no real permanent data lives on these servers, but it turns out that while most of thie time it's fine, if the mupdate processes ever decide they need to re-sync with the master, we've got 6 minutes of trouble ahead while it downloads and stores the 800k entries in the mailboxes.db. During these sync periods, we see two negative impacts. The first is lockup on the mailboxes.db on the front-end servers, which slows down both accepting new IMAP/POP connections and the reception of incoming messages. (The front-ends also accept LMTP connections from a separate pair of queueing hosts, then proxy those to the back-ends.) The second is that, because the front-ends go into a It's awfully frustrating that a system that, as my boss says, performs like a Camaro most of the times until you hit a little rock in the road, and it suddenly turns into a Pinto. It's also frustrating that this seems like one of the less complicated aspects of the system -- publishing replicas of a read-only database to a few worker boxes. I suppose this is Fastmail and others ripped out the proxyd's and replaced them with nginx or perdition. Currently we still support GSSAPI as an auth mechanism, which kept me from going that direction, but given the problems we're seeing, I'd be open to architectural suggestions on either how to tie perdition or nginx to the MUPDATE master (because we don't have the back-ends split along any discernable lines at this point), or suggestions on how to make the master-to-frontend propagation faster or less painful. Sorry for the long message, but it's not a simple problem we're fighting. Michael Bacon UNC Chapel Hill From dwhite at olp.net Mon Oct 19 17:06:39 2009 From: dwhite at olp.net (Dan White) Date: Mon, 19 Oct 2009 16:06:39 -0500 Subject: painful mupdate syncs between front-ends and database server In-Reply-To: <1482D6C9665A7BA237C6D53C@trophic.its.unc.edu> References: <1482D6C9665A7BA237C6D53C@trophic.its.unc.edu> Message-ID: <20091019210639.GE4942@dan.olp.net> On 19/10/09?16:38?-0400, Michael Bacon wrote: >I say mostly because while most of the times the thing handles our 80,000 >users and 14,000+ simultaneous connections like a champ, some of the time, >we get some extreme pain, mostly due to syncs between the MUPDATE master >and the front-end servers. What database type are you using for mailboxes.db? This might provide some optimization tips, if you haven't already parsed it: http://cyrusimap.web.cmu.edu/imapd/install-perf.html -- Dan White From morgan at orst.edu Mon Oct 19 17:13:03 2009 From: morgan at orst.edu (Andrew Morgan) Date: Mon, 19 Oct 2009 14:13:03 -0700 (PDT) Subject: painful mupdate syncs between front-ends and database server In-Reply-To: <1482D6C9665A7BA237C6D53C@trophic.its.unc.edu> References: <1482D6C9665A7BA237C6D53C@trophic.its.unc.edu> Message-ID: On Mon, 19 Oct 2009, Michael Bacon wrote: > Hello, list, > > Today we're enjoying our first full work day of independence from the old > monolithic cyrus server installed in 1999 (Sun 6800 -- it's had new CPU > boards since then, but that's it), and on our new shiny cluster of T5220's > that are mostly happily operating as a murder. > > I say mostly because while most of the times the thing handles our 80,000 > users and 14,000+ simultaneous connections like a champ, some of the time, > we get some extreme pain, mostly due to syncs between the MUPDATE master > and the front-end servers. > > When we spec'ed out our servers, we didn't put much I/O capacity into the > front-end servers -- just a pair of mirrored 10k disks doing the OS, the > logging, the mailboxes.db, and all the webmail action going on in another > solaris zone on the same hardware. We thought this was sufficient given > the fact that no real permanent data lives on these servers, but it turns > out that while most of thie time it's fine, if the mupdate processes ever > decide they need to re-sync with the master, we've got 6 minutes of trouble > ahead while it downloads and stores the 800k entries in the mailboxes.db. What is causing a (re)sync of the frontends? Normally this should only happen when you start Cyrus on a frontend, right? > During these sync periods, we see two negative impacts. The first is > lockup on the mailboxes.db on the front-end servers, which slows down both > accepting new IMAP/POP connections and the reception of incoming messages. > (The front-ends also accept LMTP connections from a separate pair of > queueing hosts, then proxy those to the back-ends.) The second is that, > because the front-ends go into a A part of this paragraph was chopped off. What else did you have to say? I ran some tests back in 2006: > However, I just performed an interesting test comparing skiplist versus > berkeley. > > skiplist - approx 20-25mins > berkeley - 3mins > > Those are the times it took to push the entire mailbox list from our test > server to the mupdate master (146382 mailboxes). This was a test run populating the mupdate master mailboxes.db from a single backend server. However, I think it illustrates the differences between skiplist and berkeley database formats. In our case, we still went with skiplist, but it might be something to consider. Andy From baconm at email.unc.edu Mon Oct 19 17:37:24 2009 From: baconm at email.unc.edu (Michael Bacon) Date: Mon, 19 Oct 2009 17:37:24 -0400 Subject: painful mupdate syncs between front-ends and database server In-Reply-To: References: <1482D6C9665A7BA237C6D53C@trophic.its.unc.edu> Message-ID: --On October 19, 2009 2:13:03 PM -0700 Andrew Morgan wrote: > What is causing a (re)sync of the frontends? Normally this should only > happen when you start Cyrus on a frontend, right? I am not entirely sure. I think what may be happening is that the slave mupdate requests get some kind of timeout, and end up disconnecting. As soon as they reconnect, they want to re-sync. I've upped the "mupdate_retry_timeout" to 10 minutes, so most of the time, they'll only timeout once, then the next retry will be successful. This solved a constant re-sync issue we had early on, but apparently hasn't solved the problem entirely. > On Mon, 19 Oct 2009, Michael Bacon wrote: >> During these sync periods, we see two negative impacts. The first is >> lockup on the mailboxes.db on the front-end servers, which slows down >> both accepting new IMAP/POP connections and the reception of incoming >> messages. (The front-ends also accept LMTP connections from a separate >> pair of queueing hosts, then proxy those to the back-ends.) The second >> is that, because the front-ends go into a > > A part of this paragraph was chopped off. What else did you have to say? Sorry, must have blanked on that. The front-ends go into a sync cycle, which ties up the MUPDATE server while they download the database (which can take up over two minutes). This causes a similar halt on anything that was responding to a mupdate "kick" on the clients, which appears to stop up a decent amount of inbound mail. > > I ran some tests back in 2006: > >> However, I just performed an interesting test comparing skiplist versus >> berkeley. >> >> skiplist - approx 20-25mins >> berkeley - 3mins >> >> Those are the times it took to push the entire mailbox list from our test >> server to the mupdate master (146382 mailboxes). > > This was a test run populating the mupdate master mailboxes.db from a > single backend server. However, I think it illustrates the differences > between skiplist and berkeley database formats. In our case, we still > went with skiplist, but it might be something to consider. Interesting. We're running skiplist everywhere, after some nasty experiences I've had with bdb, but that's a pretty astonishing performance difference. I'm pretty sure we can solve the problem by adding additional I/O capacity to the mailboxes.db on the front-ends, but it's kind of frustrating that we have to. I've considered putting those in a swap-mounted file system, but that makes me a bit nervous. -Michael From bawood at umich.edu Mon Oct 19 18:27:15 2009 From: bawood at umich.edu (Brian Awood) Date: Mon, 19 Oct 2009 18:27:15 -0400 Subject: painful mupdate syncs between front-ends and database server In-Reply-To: <1482D6C9665A7BA237C6D53C@trophic.its.unc.edu> References: <1482D6C9665A7BA237C6D53C@trophic.its.unc.edu> Message-ID: <200910191827.17192.bawood@umich.edu> On Monday 19 October 2009 @ 16:38, Michael Bacon wrote: > > I say mostly because while most of the times the thing handles our > 80,000 users and 14,000+ simultaneous connections like a champ, > some of the time, we get some extreme pain, mostly due to syncs > between the MUPDATE master and the front-end servers. We have similar usage levels, but don't have any issues with proxies and MUPDATE. What is the load on your mupdate master like? When a proxy mupdate first connects it should get a dump of the database then sit and wait for any changes to be sent to it. If the load on your mupdate master is high, have you tried setting; mupdate_config: unified on your proxies? > I suppose this is Fastmail and others ripped out the proxyd's and > replaced them with nginx or perdition. Currently we still support > GSSAPI as an auth mechanism, which kept me from going that > direction, but given the problems we're seeing, I'd be open to > architectural suggestions on either how to tie perdition or nginx > to the MUPDATE master (because we don't have the back-ends split > along any discernable lines at this point), or suggestions on how > to make the master-to-frontend propagation faster or less painful. > For comparison, we also support GSSAPI. We have 3 IPs behind our mail hostname, each IP is a hardware load balancer with 2 machines behind it. The system load on each of the 6 machines is generally not much more than 1. Before the load balancers, when we just used a muti-A record we would tend to have some proxyd related load issues because mail clients tend to prefer the lowest IP in a set. -Brian From morgan at orst.edu Mon Oct 19 19:03:52 2009 From: morgan at orst.edu (Andrew Morgan) Date: Mon, 19 Oct 2009 16:03:52 -0700 (PDT) Subject: painful mupdate syncs between front-ends and database server In-Reply-To: References: <1482D6C9665A7BA237C6D53C@trophic.its.unc.edu> Message-ID: On Mon, 19 Oct 2009, Michael Bacon wrote: > --On October 19, 2009 2:13:03 PM -0700 Andrew Morgan wrote: > >> What is causing a (re)sync of the frontends? Normally this should only >> happen when you start Cyrus on a frontend, right? > > I am not entirely sure. I think what may be happening is that the slave > mupdate requests get some kind of timeout, and end up disconnecting. As soon > as they reconnect, they want to re-sync. I've upped the > "mupdate_retry_timeout" to 10 minutes, so most of the time, they'll only > timeout once, then the next retry will be successful. This solved a constant > re-sync issue we had early on, but apparently hasn't solved the problem > entirely. >>> During these sync periods, we see two negative impacts. The first is >>> lockup on the mailboxes.db on the front-end servers, which slows down >>> both accepting new IMAP/POP connections and the reception of incoming >>> messages. (The front-ends also accept LMTP connections from a separate >>> pair of queueing hosts, then proxy those to the back-ends.) The second >>> is that, because the front-ends go into a >> >> A part of this paragraph was chopped off. What else did you have to say? > > Sorry, must have blanked on that. The front-ends go into a sync cycle, which > ties up the MUPDATE server while they download the database (which can take > up over two minutes). This causes a similar halt on anything that was > responding to a mupdate "kick" on the clients, which appears to stop up a > decent amount of inbound mail. Yeah, normally I take a frontend out of rotation (hardware load balancer) before I restart cyrus, for this very reason. > Interesting. We're running skiplist everywhere, after some nasty experiences > I've had with bdb, but that's a pretty astonishing performance difference. We went with skiplist to avoid the hassle of Berkeley DB upgrades. > I'm pretty sure we can solve the problem by adding additional I/O capacity to > the mailboxes.db on the front-ends, but it's kind of frustrating that we have > to. I've considered putting those in a swap-mounted file system, but that > makes me a bit nervous. I think it would be more useful to understand why your frontends need to resync outside of a restart. Anything else is just a work-around. Andy From wes at umich.edu Mon Oct 19 21:37:57 2009 From: wes at umich.edu (Wesley Craig) Date: Mon, 19 Oct 2009 21:37:57 -0400 Subject: painful mupdate syncs between front-ends and database server In-Reply-To: References: <1482D6C9665A7BA237C6D53C@trophic.its.unc.edu> Message-ID: On 19 Oct 2009, at 17:37, Michael Bacon wrote: > --On October 19, 2009 2:13:03 PM -0700 Andrew Morgan > wrote: >> What is causing a (re)sync of the frontends? Normally this should >> only >> happen when you start Cyrus on a frontend, right? > > I am not entirely sure. I think what may be happening is that the > slave > mupdate requests get some kind of timeout, and end up > disconnecting. As > soon as they reconnect, they want to re-sync. I've upped the > "mupdate_retry_timeout" to 10 minutes, so most of the time, they'll > only > timeout once, then the next retry will be successful. This solved a > constant re-sync issue we had early on, but apparently hasn't > solved the > problem entirely. How are your frontend mupdate processes authenticating to your mupdate master? And what version of Kerberos are you using (anticipating the answer to your first question)? I suspect that you're getting a GSSAPI expired context. > The front-ends go into a sync cycle, > which ties up the MUPDATE server while they download the database > (which > can take up over two minutes). This causes a similar halt on > anything that > was responding to a mupdate "kick" on the clients, which appears to > stop up > a decent amount of inbound mail. lmtpd on your proxies is talking directly to the mupdate master? It's possible for lmtpd on your proxies to only check the local mailboxes.db. :wes From brong at fastmail.fm Mon Oct 19 21:54:45 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 20 Oct 2009 12:54:45 +1100 Subject: painful mupdate syncs between front-ends and database server In-Reply-To: <1482D6C9665A7BA237C6D53C@trophic.its.unc.edu> References: <1482D6C9665A7BA237C6D53C@trophic.its.unc.edu> Message-ID: <1256003685.9839.1340931677@webmail.messagingengine.com> On Mon, 19 Oct 2009 16:38 -0400, "Michael Bacon" wrote: > When we spec'ed out our servers, we didn't put much I/O capacity into the > front-end servers -- just a pair of mirrored 10k disks doing the OS, the > logging, the mailboxes.db, and all the webmail action going on in another > solaris zone on the same hardware. We thought this was sufficient given > the fact that no real permanent data lives on these servers, but it turns > out that while most of thie time it's fine, if the mupdate processes ever > decide they need to re-sync with the master, we've got 6 minutes of > trouble > ahead while it downloads and stores the 800k entries in the mailboxes.db. Have you checked if it's actually IO limited? Reading the code, it appears to do the entire sync in a single transaction, which is bad because it locks the entire mailboxes.db for the entire time. > During these sync periods, we see two negative impacts. The first is > lockup on the mailboxes.db on the front-end servers, which slows down > both > accepting new IMAP/POP connections and the reception of incoming > messages. > (The front-ends also accept LMTP connections from a separate pair of > queueing hosts, then proxy those to the back-ends.) The second is that, > because the front-ends go into a Lost you there - I'm assuming it causes a nasty load spike when it finishes too. Makes sense. > I suppose this is Fastmail and others ripped out the proxyd's and > replaced > them with nginx or perdition. Currently we still support GSSAPI as an > auth > mechanism, which kept me from going that direction, but given the > problems > we're seeing, I'd be open to architectural suggestions on either how to > tie > perdition or nginx to the MUPDATE master (because we don't have the > back-ends split along any discernable lines at this point), or > suggestions > on how to make the master-to-frontend propagation faster or less painful. We didn't ever go with murder. All our backends are totally independent. > Sorry for the long message, but it's not a simple problem we're fighting. No - it's not! I wonder if a better approach would be to batch the mailboxes.db updates into groups of no more than (say) 256. Arrgh - stupid, stupid, stupid. Layers of abstraction mean we have a nice fast "foreach" going on, and then throw away the data and dataptr fields, followed by which we fetch the data field again. It's very inefficient. I wonder what percentage of the time is just reading stuff from the mailboxes.db? Anyway - the bit that's actually going to be blocking you will be the mailboxes.db transactions. I've attached a patch. Advance warning - I don't use murder, so I haven't done more than compile test it! It SHOULD be safe though, it just commits to the mailboxes.db every 256 changes and then closes the transaction, which means that things that were queued waiting for the lock should get a chance to run before you update the next 256 records. The patch is against current CVS (well, against my git clone of current CVS anyway) Bron. -- Bron Gondwana brong at fastmail.fm -------------- next part -------------- A non-text attachment was scrubbed... Name: mupdate_transctions.diff Type: application/octet-stream Size: 2253 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20091020/de380d87/attachment.obj From elfejoyeux at gmail.com Tue Oct 20 06:13:05 2009 From: elfejoyeux at gmail.com (Cyril Servant) Date: Tue, 20 Oct 2009 12:13:05 +0200 Subject: painful mupdate syncs between front-ends and database server In-Reply-To: <1482D6C9665A7BA237C6D53C@trophic.its.unc.edu> References: <1482D6C9665A7BA237C6D53C@trophic.its.unc.edu> Message-ID: <8c895a0b0910200313m159ed2e9n164077233f968ad5@mail.gmail.com> Hello, 2009/10/19 Michael Bacon : > Hello, list, > > Today we're enjoying our first full work day of independence from the old > monolithic cyrus server installed in 1999 (Sun 6800 -- it's had new CPU > boards since then, but that's it), and on our new shiny cluster of T5220's > that are mostly happily operating as a murder. > > I say mostly because while most of the times the thing handles our 80,000 > users and 14,000+ simultaneous connections like a champ, some of the time, > we get some extreme pain, mostly due to syncs between the MUPDATE master > and the front-end servers. > > When we spec'ed out our servers, we didn't put much I/O capacity into the > front-end servers -- just a pair of mirrored 10k disks doing the OS, the > logging, the mailboxes.db, and all the webmail action going on in another > solaris zone on the same hardware. ?We thought this was sufficient given > the fact that no real permanent data lives on these servers, but it turns > out that while most of thie time it's fine, if the mupdate processes ever > decide they need to re-sync with the master, we've got 6 minutes of trouble > ahead while it downloads and stores the 800k entries in the mailboxes.db. > > During these sync periods, we see two negative impacts. ?The first is > lockup on the mailboxes.db on the front-end servers, which slows down both > accepting new IMAP/POP connections and the reception of incoming messages. > (The front-ends also accept LMTP connections from a separate pair of > queueing hosts, then proxy those to the back-ends.) ?The second is that, > because the front-ends go into a > > It's awfully frustrating that a system that, as my boss says, performs like > a Camaro most of the times until you hit a little rock in the road, and it > suddenly turns into a Pinto. ?It's also frustrating that this seems like > one of the less complicated aspects of the system -- publishing replicas of > a read-only database to a few worker boxes. > > I suppose this is Fastmail and others ripped out the proxyd's and replaced > them with nginx or perdition. ?Currently we still support GSSAPI as an auth > mechanism, which kept me from going that direction, but given the problems > we're seeing, I'd be open to architectural suggestions on either how to tie > perdition or nginx to the MUPDATE master (because we don't have the > back-ends split along any discernable lines at this point), or suggestions > on how to make the master-to-frontend propagation faster or less painful. > > Sorry for the long message, but it's not a simple problem we're fighting. > > Michael Bacon > UNC Chapel Hill > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > Here we had a similar situation : more than a million mailboxes, and each MUPDATE sync was veeeeery long (when it succeeded). Now, we bypass the problem : we get rid of the MUPDATE (and the skiplist mailboxes.db). We use a home made mysql backend for mailboxes. We added write and read filters to this backend so front-end and back-end servers get the right value from mysql. With this configuration, we're no more in murder mode, we just use front-end cyrus (proxys), back-end cyrus, and mysql. We don't need MUPDATE any more, so we have no sync problems. Cyrus restarts are fast. -- Cyril From xavier.bestel at free.fr Tue Oct 20 13:36:24 2009 From: xavier.bestel at free.fr (Xavier Bestel) Date: Tue, 20 Oct 2009 19:36:24 +0200 Subject: Exec'ing a script from Cyrus when imapd has a client Message-ID: <1256060184.2946.53.camel@skunk> Hi, I have a small install with cyrus-imapd 2.3.14, which reads some of its mails with fetchmail. To limit the delay in mail delivery, fetchmail awakes each minute to get mails. What I would like is let fetchmail do that only when there's a client actually reading its mails, i.e. an MUA actually connected to imapd. So, my question: how to hook a script each time a client connects/disconnects from imapd ? Thanks, Xav From woods-cyrus at weird.com Tue Oct 20 15:58:04 2009 From: woods-cyrus at weird.com (Greg A. Woods) Date: Tue, 20 Oct 2009 15:58:04 -0400 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: <1256060184.2946.53.camel@skunk> References: <1256060184.2946.53.camel@skunk> Message-ID: At Tue, 20 Oct 2009 19:36:24 +0200, Xavier Bestel wrote: Subject: Exec'ing a script from Cyrus when imapd has a client > > I have a small install with cyrus-imapd 2.3.14, which reads some of its > mails with fetchmail. To limit the delay in mail delivery, fetchmail > awakes each minute to get mails. > What I would like is let fetchmail do that only when there's a client > actually reading its mails, i.e. an MUA actually connected to imapd. I don't get it. Are you saying you are using fetchmail to inject messages into a locally running Cyrus install which you then connect to with a locally running IMAP MUA? Maybe you should just eliminate fetchmail from the picture and then see if things make more sense. Just point your MUA at the originating IMAP server and eliminate everything in between. If you're using an IMAP capable client, and you're using IMAP to read your e-mail, then you don't need or want fetchmail -- it's just a rather nasty old hack for the days before IMAP. Personally I think the last usable copy of fetchmail should be archived away on an ancient 9-track magnetic tape reel and put behind a little glass door which has "Break glass in case of emergency" written on it. Everything is ever so much easier and simpler if you can just eliminate all the unnecessary complexities, such as fetchmail. (if you want to keep a copy of your messages on your local machine, then you can and should probably just use your MUA to do that) -- Greg A. Woods +1 416 218-0098 VE3TCP RoboHack Planix, Inc. Secrets of the Weird From david.lang at digitalinsight.com Tue Oct 20 16:00:23 2009 From: david.lang at digitalinsight.com (David Lang) Date: Tue, 20 Oct 2009 13:00:23 -0700 (PDT) Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: References: <1256060184.2946.53.camel@skunk> Message-ID: On Tue, 20 Oct 2009, Greg A. Woods wrote: > At Tue, 20 Oct 2009 19:36:24 +0200, Xavier Bestel wrote: > Subject: Exec'ing a script from Cyrus when imapd has a client >> >> I have a small install with cyrus-imapd 2.3.14, which reads some of its >> mails with fetchmail. To limit the delay in mail delivery, fetchmail >> awakes each minute to get mails. >> What I would like is let fetchmail do that only when there's a client >> actually reading its mails, i.e. an MUA actually connected to imapd. > > I don't get it. Are you saying you are using fetchmail to inject > messages into a locally running Cyrus install which you then connect to > with a locally running IMAP MUA? I think what he is saying is that he does not have a MTA. he uses fetchmail to download mail from elsewhere and put it in cyrus. currently he crons fetchmail to run once a min so that when people are logged in they see new mail with low latencies. however, if nobody is logged in to Cyrus, this is a waste of time, and he would be better off running fetchmail less frequently (or not at all). so he is asking if there is a way to tell if anyone is connected to Cyrus or not, so that if not he can skip the fetchmail run. David Lang > Maybe you should just eliminate fetchmail from the picture and then see > if things make more sense. Just point your MUA at the originating IMAP > server and eliminate everything in between. > > If you're using an IMAP capable client, and you're using IMAP to read > your e-mail, then you don't need or want fetchmail -- it's just a rather > nasty old hack for the days before IMAP. Personally I think the last > usable copy of fetchmail should be archived away on an ancient 9-track > magnetic tape reel and put behind a little glass door which has "Break > glass in case of emergency" written on it. Everything is ever so much > easier and simpler if you can just eliminate all the unnecessary > complexities, such as fetchmail. > > (if you want to keep a copy of your messages on your local machine, then > you can and should probably just use your MUA to do that) > > From steffo76 at gmx.de Tue Oct 20 16:22:32 2009 From: steffo76 at gmx.de (steffo76 at gmx.de) Date: Tue, 20 Oct 2009 22:22:32 +0200 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: <1256060184.2946.53.camel@skunk> References: <1256060184.2946.53.camel@skunk> Message-ID: <200910202222.32607.steffo76@gmx.de> Hi there, maybe logsurfer (or something similar) would be of help, you could monitor your cyrus logs with it and run fetchmail accordingly: http://www.crypt.gen.nz/logsurfer/ http://sourceforge.net/projects/logsurfer/ Regards, Stephan Am Dienstag 20 Oktober 2009 19:36:24 schrieb Xavier Bestel: > Hi, > > I have a small install with cyrus-imapd 2.3.14, which reads some of its > mails with fetchmail. To limit the delay in mail delivery, fetchmail > awakes each minute to get mails. > What I would like is let fetchmail do that only when there's a client > actually reading its mails, i.e. an MUA actually connected to imapd. > > So, my question: how to hook a script each time a client > connects/disconnects from imapd ? > > Thanks, > Xav > > > > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > -- If a train station is a place where a train stops, what's a workstation? From xavier.bestel at free.fr Tue Oct 20 16:54:12 2009 From: xavier.bestel at free.fr (Xavier Bestel) Date: Tue, 20 Oct 2009 22:54:12 +0200 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: References: <1256060184.2946.53.camel@skunk> Message-ID: <1256072052.3739.4.camel@badjo> Le mardi 20 octobre 2009 ? 13:00 -0700, David Lang a ?crit : > On Tue, 20 Oct 2009, Greg A. Woods wrote: > > > At Tue, 20 Oct 2009 19:36:24 +0200, Xavier Bestel wrote: > > Subject: Exec'ing a script from Cyrus when imapd has a client > >> > >> I have a small install with cyrus-imapd 2.3.14, which reads some of its > >> mails with fetchmail. To limit the delay in mail delivery, fetchmail > >> awakes each minute to get mails. > >> What I would like is let fetchmail do that only when there's a client > >> actually reading its mails, i.e. an MUA actually connected to imapd. > > > > I don't get it. Are you saying you are using fetchmail to inject > > messages into a locally running Cyrus install which you then connect to > > with a locally running IMAP MUA? > > I think what he is saying is that he does not have a MTA. he uses fetchmail to > download mail from elsewhere and put it in cyrus. > > currently he crons fetchmail to run once a min so that when people are logged in > they see new mail with low latencies. > > however, if nobody is logged in to Cyrus, this is a waste of time, and he would > be better off running fetchmail less frequently (or not at all). > > so he is asking if there is a way to tell if anyone is connected to Cyrus or > not, so that if not he can skip the fetchmail run. Yes, that's it, precisely. Xav From xavier.bestel at free.fr Tue Oct 20 16:56:42 2009 From: xavier.bestel at free.fr (Xavier Bestel) Date: Tue, 20 Oct 2009 22:56:42 +0200 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: <200910202222.32607.steffo76@gmx.de> References: <1256060184.2946.53.camel@skunk> <200910202222.32607.steffo76@gmx.de> Message-ID: <1256072203.3739.6.camel@badjo> Hi, Le mardi 20 octobre 2009 ? 22:22 +0200, steffo76 at gmx.de a ?crit : > maybe logsurfer (or something similar) would be of help, you could > monitor your cyrus logs with it and run fetchmail accordingly: > > http://www.crypt.gen.nz/logsurfer/ > http://sourceforge.net/projects/logsurfer/ That may be an answer, yes. However I'd have preferred if the signal came from Cyrus directly (I think it'sa more robust solution). Thanks, Xav From brong at fastmail.fm Tue Oct 20 19:34:47 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 21 Oct 2009 10:34:47 +1100 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: <1256072203.3739.6.camel@badjo> References: <1256060184.2946.53.camel@skunk> <200910202222.32607.steffo76@gmx.de> <1256072203.3739.6.camel@badjo> Message-ID: <20091020233447.GA17677@brong.net> On Tue, Oct 20, 2009 at 10:56:42PM +0200, Xavier Bestel wrote: > Hi, > > Le mardi 20 octobre 2009 ? 22:22 +0200, steffo76 at gmx.de a ?crit : > > maybe logsurfer (or something similar) would be of help, you could > > monitor your cyrus logs with it and run fetchmail accordingly: > > > > http://www.crypt.gen.nz/logsurfer/ > > http://sourceforge.net/projects/logsurfer/ > > That may be an answer, yes. However I'd have preferred if the signal > came from Cyrus directly (I think it'sa more robust solution). Log watching is actually pretty good - but another option is to monitor the $conf/proc/ directory. When someone is connected there will be a file in there with their username and what folder they currently have selected. Your cron job could just grep that folder and run if there was a file with the required username present. [brong at imap1 proc]$ grep brong * 31327:web6.internal [10.202.2.215] brong user.brong Getting Cyrus to actually trigger something is much more complex - but you could easily do something with an inotify on the directory if you want smaller than a minute resolution. Bron. From adrieder at sbox.tugraz.at Wed Oct 21 11:54:20 2009 From: adrieder at sbox.tugraz.at (Dietmar Rieder) Date: Wed, 21 Oct 2009 17:54:20 +0200 Subject: ACL question Message-ID: <4ADF2EAC.9040906@sbox.tugraz.at> Hi, is there a possibility to set an acl to a folder outside the users INBOX hierarchy such as a user can not delete it but at the same time it should be possible for her/him to create and delete subfolders in that folder. e.g. The users INBOX is : user.testuser The folder outside is: archive.testuser With the following acl: localhost> lam archive.testuser testuser lrswipkxtecd the user can delete the folder archive.testuser and its subfolders that were created by the user. But I'd like to prevent that the user can delete the archive.testuser With the following acl: localhost> lam archive.testuser testuser lrswipktec the user can not delete the folder archive.testuser and its subfolders that were created by the user. But I'd like to allow that the user can delete the subfolders of archive.testuser Thanks for any hint... Didi From dwhite at olp.net Wed Oct 21 11:59:54 2009 From: dwhite at olp.net (Dan White) Date: Wed, 21 Oct 2009 10:59:54 -0500 Subject: ACL question In-Reply-To: <4ADF2EAC.9040906@sbox.tugraz.at> References: <4ADF2EAC.9040906@sbox.tugraz.at> Message-ID: <20091021155954.GE4976@dan.olp.net> On 21/10/09?17:54?+0200, Dietmar Rieder wrote: >Hi, > >is there a possibility to set an acl to a folder outside the users INBOX >hierarchy such as a user can not delete it but at the same time it >should be possible for her/him to create and delete subfolders in that >folder. > >e.g. > >The users INBOX is : user.testuser >The folder outside is: archive.testuser > >With the following acl: >localhost> lam archive.testuser >testuser lrswipkxtecd Dietmar, See RFC 4314 for an explanation of the acl flags. -- Dan White From adrieder at sbox.tugraz.at Wed Oct 21 12:05:03 2009 From: adrieder at sbox.tugraz.at (Dietmar Rieder) Date: Wed, 21 Oct 2009 18:05:03 +0200 Subject: ACL question In-Reply-To: <20091021155954.GE4976@dan.olp.net> References: <4ADF2EAC.9040906@sbox.tugraz.at> <20091021155954.GE4976@dan.olp.net> Message-ID: <4ADF312F.7000501@sbox.tugraz.at> Dan White wrote: > On 21/10/09 17:54 +0200, Dietmar Rieder wrote: >> Hi, >> >> is there a possibility to set an acl to a folder outside the users >> INBOX hierarchy such as a user can not delete it but at the same time >> it should be possible for her/him to create and delete subfolders in >> that folder. >> >> e.g. >> >> The users INBOX is : user.testuser >> The folder outside is: archive.testuser >> >> With the following acl: >> localhost> lam archive.testuser >> testuser lrswipkxtecd > > Dietmar, > > See RFC 4314 for an explanation of the acl flags. > Dan, thanks for your hint. I did that already but (maybe I'm to stupid) I couldn't figure out a set of flags, that would meet my needs... Didi From dwhite at olp.net Wed Oct 21 12:10:01 2009 From: dwhite at olp.net (Dan White) Date: Wed, 21 Oct 2009 11:10:01 -0500 Subject: ACL question In-Reply-To: <4ADF312F.7000501@sbox.tugraz.at> References: <4ADF2EAC.9040906@sbox.tugraz.at> <20091021155954.GE4976@dan.olp.net> <4ADF312F.7000501@sbox.tugraz.at> Message-ID: <20091021161001.GF4976@dan.olp.net> On 21/10/09?18:05?+0200, Dietmar Rieder wrote: >> Dietmar, >> >> See RFC 4314 for an explanation of the acl flags. >> > > Dan, > > thanks for your hint. I did that already but (maybe I'm to stupid) I > couldn't figure out a set of flags, that would meet my needs... If I'm reading right: "Mailbox management: CREATE - "k" right on a nearest existing parent mailbox. When a new mailbox is created, it SHOULD inherit the ACL from the parent mailbox (if one exists) in the defined hierarchy." ... it appears the sub mailbox will always get the same ACL as the parent mailbox. You might have to modify the ACL of the sub mailbox after its created. -- Dan White From adrieder at sbox.tugraz.at Wed Oct 21 12:20:25 2009 From: adrieder at sbox.tugraz.at (Dietmar Rieder) Date: Wed, 21 Oct 2009 18:20:25 +0200 Subject: ACL question In-Reply-To: <20091021161001.GF4976@dan.olp.net> References: <4ADF2EAC.9040906@sbox.tugraz.at> <20091021155954.GE4976@dan.olp.net> <4ADF312F.7000501@sbox.tugraz.at> <20091021161001.GF4976@dan.olp.net> Message-ID: <4ADF34C9.5020805@sbox.tugraz.at> Dan White wrote: > On 21/10/09 18:05 +0200, Dietmar Rieder wrote: >>> Dietmar, >>> >>> See RFC 4314 for an explanation of the acl flags. >>> >> >> Dan, >> >> thanks for your hint. I did that already but (maybe I'm to stupid) I >> couldn't figure out a set of flags, that would meet my needs... > > If I'm reading right: > > "Mailbox management: > CREATE - "k" right on a nearest existing parent mailbox. When a > new mailbox is created, it SHOULD inherit the ACL from the parent > mailbox (if one exists) in the defined hierarchy." > > ... it appears the sub mailbox will always get the same ACL as the parent > mailbox. You might have to modify the ACL of the sub mailbox after its > created. Ok, thanks, that's what I've overlooked, so basically it is not possible the way I'd liked it - at least not "automagically" thanks anyway! Didi From david at touzeau.eu Wed Oct 21 14:45:11 2009 From: david at touzeau.eu (David Touzeau) Date: Wed, 21 Oct 2009 20:45:11 +0200 Subject: cyrus replication : master to replica and replica to master In-Reply-To: <1255908245.21631.6.camel@dtouzeau-laptop> References: <1255908245.21631.6.camel@dtouzeau-laptop> Message-ID: <1256150711.21117.2.camel@dtouzeau-laptop> Dear I have set cyrus-imap with master and replica. This configuration is a kind of cluster Active/passive I would like to know if it is possible to SET the replica has the master too in order to replicate new mail saved on the replica to the master and vis versa In this case it should be turn to active/active.. If it is possible, did someone build this kind of configuration ? best regards ---- Any updates ??? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20091021/83b296d9/attachment.html From brong at fastmail.fm Wed Oct 21 18:31:20 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 22 Oct 2009 09:31:20 +1100 Subject: cyrus replication : master to replica and replica to master In-Reply-To: <1256150711.21117.2.camel@dtouzeau-laptop> References: <1255908245.21631.6.camel@dtouzeau-laptop> <1256150711.21117.2.camel@dtouzeau-laptop> Message-ID: <20091021223120.GA3466@brong.net> On Wed, Oct 21, 2009 at 08:45:11PM +0200, David Touzeau wrote: > Dear > > I have set cyrus-imap with master and replica. > This configuration is a kind of cluster Active/passive > > I would like to know if it is possible to SET the replica has the master > too > > in order to replicate new mail saved on the replica to the master and > vis versa > > In this case it should be turn to active/active.. > > If it is possible, did someone build this kind of configuration ? > > best regards > > > ---- > > Any updates ??? Well - it's theoretically possible. But I don't know anyone who's done it, and it has the potential to get ugly if you're delivering to the same mailboxes at each end. There's nothing I can see that would actually stop it working. Bron. From robm at fastmail.fm Thu Oct 22 00:20:09 2009 From: robm at fastmail.fm (Rob Mueller) Date: Thu, 22 Oct 2009 15:20:09 +1100 Subject: cyrus replication : master to replica and replica to master In-Reply-To: <20091021223120.GA3466@brong.net> References: <1255908245.21631.6.camel@dtouzeau-laptop><1256150711.21117.2.camel@dtouzeau-laptop> <20091021223120.GA3466@brong.net> Message-ID: > Well - it's theoretically possible. But I don't know anyone who's done > it, and it has the potential to get ugly if you're delivering to the > same mailboxes at each end. There's nothing I can see that would > actually stop it working. I think Bron failed to put sufficiently large warning signs here :) The difference between "in theory this would work" and the practice of actually doing it are huge. Basically it works only if you are 100% sure that only one side is ever being accessed at a time. eg. IMAP/POP/LMTP/etc. If somehow you get an LTMP deliver to different ends, then one side will end up overwriting the other. In other words, DON'T DO THIS. Rob From david at touzeau.eu Thu Oct 22 02:08:16 2009 From: david at touzeau.eu (David Touzeau) Date: Thu, 22 Oct 2009 08:08:16 +0200 Subject: cyrus replication : master to replica and replica to master In-Reply-To: References: <1255908245.21631.6.camel@dtouzeau-laptop> <1256150711.21117.2.camel@dtouzeau-laptop> <20091021223120.GA3466@brong.net> Message-ID: <1256191696.17299.10.camel@dtouzeau-laptop> Sujet: Re: cyrus replication : master to replica and replica to master Date: Thu, 22 Oct 2009 15:20:09 +1100me > Well - it's theoretically possible. But I don't know anyone who's done > it, and it has the potential to get ugly if you're delivering to the > same mailboxes at each end. There's nothing I can see that would > actually stop it working. >I think Bron failed to put sufficiently large warning signs here :) >The difference between "in theory this would work" and the practice of >actually doing it are huge. Basically it works only if you are 100% sure >that only one side is ever being accessed at a time. eg. IMAP/POP/LMTP/etc. >If somehow you get an LTMP deliver to different ends, then one side will end >up overwriting the other. >In other words, DON'T DO THIS. >Rob Thanks rob i'm very surprised that there is not really official point from cyrus-imap dev team against using cyrus in cluster active/active mode Since serverals years the messaging service become very important and the clustering system is the right way to provide a real fail over system After googlize i had read that the only best method is to using drbd but there no really wikis and documentations about implementing this kind of structure. I would like to restart this topic and discuss. David -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20091022/508efab5/attachment.html From robm at fastmail.fm Thu Oct 22 03:37:48 2009 From: robm at fastmail.fm (Rob Mueller) Date: Thu, 22 Oct 2009 18:37:48 +1100 Subject: cyrus replication : master to replica and replica to master References: <1255908245.21631.6.camel@dtouzeau-laptop><1256150711.21117.2.camel@dtouzeau-laptop><20091021223120.GA3466@brong.net> <1256191696.17299.10.camel@dtouzeau-laptop> Message-ID: <4F91C264B2794E1282F0B2E3B7FAF310@jem> > i'm very surprised that there is not really official point from cyrus-imap > dev team against using cyrus in cluster active/active mode I can't comment, but I guess they're busy. > Since serverals years the messaging service become very important and the > clustering system is the right way to provide a real fail over system > After googlize i had read that the only best method is to using drbd but > there no really wikis and documentations about implementing this kind of > structure. drbd doesn't allow a master/master mode either, unless you use some clustered filesystem, and take the performance hit due to locked access to meta data (eg mailboxes.db) that would be happening. cyrus replication does what drbd does, but has the added advantage that it's "content aware", so it only has to replicate the actual needed data, it can do checksumming of email data on both sides, and you won't lose everything if you get some os level file system corruption on your master. Basically they're similar solutions (master -> slave replication) but at different levels (block level vs content level). Rob From jonforthewin at gmail.com Thu Oct 22 03:56:03 2009 From: jonforthewin at gmail.com (Jon .) Date: Thu, 22 Oct 2009 00:56:03 -0700 Subject: cyrus replication : master to replica and replica to master In-Reply-To: References: <1255908245.21631.6.camel@dtouzeau-laptop> <1256150711.21117.2.camel@dtouzeau-laptop> <20091021223120.GA3466@brong.net> Message-ID: <5801465c0910220056x68d4d51dpfe63e1aa607fa227@mail.gmail.com> On Wed, Oct 21, 2009 at 9:20 PM, Rob Mueller wrote: ... > The difference between "in theory this would work" and the practice of > actually doing it are huge. Basically it works only if you are 100% sure > that only one side is ever being accessed at a time. eg. IMAP/POP/LMTP/etc. ... > In other words, DON'T DO THIS. > > Rob What are the particular bits that could conflict and have undesirable results? Metadata, messages, entire mailboxes? In this hypothetical active/active configuration, what exactly what could an IMAP client potentially do to create undesirable results? Would it be a huge undertaking to timestamp data that is to be replicated to another Cyrus daemon for the receiving Cyrus daemon to know which conflicting pieces of data to drop in favor of newer data? Right now I have a client who needs 130 or so users on a private mail server and has two cheap 1U Dell servers to work with. Ideally they are to be put in physically distanced data-centers for redundancy to one another. Combined with the hypothetical replication of timestamped data describe above, wouldn't setting the fqdn imap.example.com to resolve two IP addresses so users' IMAP clients can fall back should an IMAP storage server be unavailable (with at least the most recent data replication of any kind is able to provide) make for a much simpler and more elegant solution than DRBD, clustered filesystems, or introducing more machines just for load balancing / resolving to an available IMAP daemon? Also, wouldn't timestamps also hypothetically resolve the inevitable split-brain situations clients would create? From brong at fastmail.fm Thu Oct 22 05:07:04 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 22 Oct 2009 20:07:04 +1100 Subject: cyrus replication : master to replica and replica to master In-Reply-To: <5801465c0910220056x68d4d51dpfe63e1aa607fa227@mail.gmail.com> References: <1255908245.21631.6.camel@dtouzeau-laptop> <1256150711.21117.2.camel@dtouzeau-laptop> <20091021223120.GA3466@brong.net> <5801465c0910220056x68d4d51dpfe63e1aa607fa227@mail.gmail.com> Message-ID: <20091022090704.GA1042@brong.net> On Thu, Oct 22, 2009 at 12:56:03AM -0700, Jon . wrote: > On Wed, Oct 21, 2009 at 9:20 PM, Rob Mueller wrote: > ... > > > The difference between "in theory this would work" and the practice of > > actually doing it are huge. Basically it works only if you are 100% sure > > that only one side is ever being accessed at a time. eg. IMAP/POP/LMTP/etc. Pretty much. With appropriate fencing, non-local bind and a service IP address that's feasible. But Rob won't let me do it. Fair enough too, it's pretty messy. > ... > > > In other words, DON'T DO THIS. > > > > Rob Yeah, yeah. I know. I could have worded it a bit more strongly. "Nobody's ever done it because it's really tricky to get right and you'll lose data for sure if you don't know what you are doing". > What are the particular bits that could conflict and have undesirable > results? Metadata, messages, entire mailboxes? In this hypothetical > active/active configuration, what exactly what could an IMAP client > potentially do to create undesirable results? Yes. Those things. Any and/or all. Try thinking about a folder rename at one end and a copy/expunge cycle between folders at the other end and resolving the resultant mess. Basically this is tricky stuff that nobody does particularly well. Generic sync is a hard problem[tm], and the Cyrus code doesn't even try. In particular, it doesn't track deltas. To get even halfway good tracking of changes, you need three things: 1) current state of A 2) current state of B 3) state last time A and B were in sync even better is knowing the changes that were made and resolving them. But without even this much information, consider the following. A: UID 5 is SEEN b: UID 5 is UNSEEN what should be the result? > Would it be a huge undertaking to timestamp data that is to be > replicated to another Cyrus daemon for the receiving Cyrus daemon to > know which conflicting pieces of data to drop in favor of newer data? Timestamp each piece of metadata individually, yes - it would be a huge undertaking. > Right now I have a client who needs 130 or so users on a private mail > server and has two cheap 1U Dell servers to work with. Ideally they > are to be put in physically distanced data-centers for redundancy to > one another. > > Combined with the hypothetical replication of timestamped data > describe above, wouldn't setting the fqdn imap.example.com to resolve > two IP addresses so users' IMAP clients can fall back should an IMAP > storage server be unavailable (with at least the most recent data > replication of any kind is able to provide) make for a much simpler > and more elegant solution than DRBD, clustered filesystems, or > introducing more machines just for load balancing / resolving to an > available IMAP daemon? Also, wouldn't timestamps also hypothetically > resolve the inevitable split-brain situations clients would create? I assume they don't like losing messages. If you really, REALLY want to go down that path I would at least take FastMail's patch that checks the GUID if the same message exists on both ends and refuses to overwrite if the message contents differ. This is half a solution, you then need to resolve the issue (we LOCALDELETE the original message at both ends so it doesn't even wind up in the .expunge file, then we append BOTH messages with brand new UIDs and set the flags they used to have on the master - finally syncing the resulting mailbox again so both messages are on both ends - but the code for that isn't in our Cyrus patches - it's a standalone script) And that's just for the split brain that results when a machine dies for whatever reason (it happened last night incidentally - one of our external RAID units had an "episode" and decided to stop talking to the server. It looks like a couple of timeouts on a failing drive tickled a firmware bug and resulted in the inbuilt OS locking up. Software, you have to love it. Embedded at all layers. So many firmwares to keep up-to-date!) - we don't have multi-directional replication. Our approach to utilisation handling has been documented here plenty of times. Basically we run multiple instances of Cyrus on each machine, so every server has both masters and replicas. We can shut down any one machine just by switching roles (a shutdown and restart of each end with new configs) Regards, Bron. From robm at fastmail.fm Thu Oct 22 05:18:18 2009 From: robm at fastmail.fm (Robert Mueller (web)) Date: Thu, 22 Oct 2009 20:18:18 +1100 Subject: cyrus replication : master to replica and replica to master In-Reply-To: <5801465c0910220056x68d4d51dpfe63e1aa607fa227@mail.gmail.com> References: <1255908245.21631.6.camel@dtouzeau-laptop> <1256150711.21117.2.camel@dtouzeau-laptop> <20091021223120.GA3466@brong.net> <5801465c0910220056x68d4d51dpfe63e1aa607fa227@mail.gmail.com> Message-ID: <1256203098.8629.1341361091@webmail.messagingengine.com> > What are the particular bits that could conflict and have undesirable > results? Metadata, messages, entire mailboxes? In this hypothetical > active/active configuration, what exactly what could an IMAP client > potentially do to create undesirable results? Simple. Client A: upload message to Inbox, gets UID 100 At the same time, Client B: upload message to Inbox, gets UID 100 You can't have two messages with the same UID. There's 3 solutions I can see: 1. Mysql solves this by having interleving id's on separate servers (eg. auto-increment column on server A is odd numbers, on server B it's even numbers). I guess you could in theory do the same with IMAP (though I'd have to double check the spec), but it would create really annoying UID lists because you basically lose the ability to use things like 30:50. One other option would be to alternate in 100's or something like that (eg. 1-100 on s1, 101-200 on s2, etc) 2. Use global locking so anything allocating UIDs gets a cross-server lock, allocates the UIDs, and keeps the global UID counter somewhere. This gets tricky to deal with the case where one server goes down, you need to handle that case well (eg the locking server has to know the difference between "down" vs "unreachable" so you don't get split brain) 3. Use some conflict resolution strategy. If some client uploads UID 100 on s1, and another uploads UID 100 on s2, then when the conflict is noticed, both sides have to delete + expunge the message (because different IMAP clients might have different ideas on what message UID 100 is) and create new UIDs 101 and 102 with the two messages. This can be messy because if a POP client is connected, you can't alter the mailbox at all because the message list isn't allowed to change under the POP clients feet, so connected POP clients could cause nasty locking issues. > Would it be a huge undertaking to timestamp data that is to be > replicated to another Cyrus daemon for the receiving Cyrus daemon to > know which conflicting pieces of data to drop in favor of newer data? It's certainly not trivial, and getting every edge case and race condition right is going to be hard. > Right now I have a client who needs 130 or so users on a private mail > server and has two cheap 1U Dell servers to work with. Ideally they > are to be put in physically distanced data-centers for redundancy to > one another. That's what replication is for, but you will have to use some manual failover strategy > Combined with the hypothetical replication of timestamped data > describe above, wouldn't setting the fqdn imap.example.com to resolve > two IP addresses so users' IMAP clients can fall back should an IMAP > storage server be unavailable (with at least the most recent data > replication of any kind is able to provide) make for a much simpler > and more elegant solution than DRBD, clustered filesystems, or > introducing more machines just for load balancing / resolving to an > available IMAP daemon? Also, wouldn't timestamps also hypothetically > resolve the inevitable split-brain situations clients would create? Wonderful in theory, but hard to implement in practice. Rob From eddy.beliveau at hec.ca Thu Oct 22 10:26:03 2009 From: eddy.beliveau at hec.ca (Eddy Beliveau) Date: Thu, 22 Oct 2009 10:26:03 -0400 Subject: cyrus - sieve returning some text with EVERY email received Message-ID: <4AE06B7B.7080607@hec.ca> Hi! On our academic server, we are using cyrus 2.2.12 with CMU Sieve 2.2 and it works perfectly. Thanks. :-) One of our user would like to return some text, for *every* email, that he received like: "Hi, we just received your email. It will be process shortly. Thanks." He tries to implement it with sieve 'vacation :days 1 ...' but this only replies once per day at original sender. He would like to reply to *all* messages Can it be done ? Any workaround ? Thanks in advance Eddy -- Eddy Beliveau HEC Montreal Montreal (Quebec) Canada From jmok at attglobal.net Thu Oct 22 10:38:49 2009 From: jmok at attglobal.net (John Mok) Date: Thu, 22 Oct 2009 22:38:49 +0800 Subject: Cyrus IMAP GSSAPI for multiple AD domains Message-ID: <4AE06E79.9080202@attglobal.net> Hi, I have successfully setup Cyrus IMAP 2.2.12 with GSSAPI / Kerberos as authentication for an AD domain "grt.citizen.co.jp", which is the default domain in /etc/imapd.conf. However, when I tried to add another AD domain "go.citizen.co.jp" other the default domain. The AD users in the latter domain, i.e. "go.citizen.co.jp", failed to authenticate from the e-mail client (e.g. Thunderbird). The error message on the server log :- Oct 22 15:35:02 imapsv01 cyrus/imap[19466]: badlogin: John.sml.citizen.co.jp [10.144.1.192] GSSAPI [SASL(-13): authentication failure: user komatsuj at go.citizen.co.jp is not allowed to proxy] I checked with imtest and it passed successfully :- >imtest -m GSSAPI imapsv01.grt.citizen.co.jp The IMAP config. /etc/imapd.conf follows :- .... virtdomains: yes defaultdomain: grt.citizen.co.jp sasl_pwcheck_method: saslauthd .... I hope someone could advise how I could make the IMAP to authenticate users from two or more AD domains. Thanks a lot. John Mok From dwhite at olp.net Thu Oct 22 11:58:18 2009 From: dwhite at olp.net (Dan White) Date: Thu, 22 Oct 2009 10:58:18 -0500 Subject: Cyrus IMAP GSSAPI for multiple AD domains In-Reply-To: <4AE06E79.9080202@attglobal.net> References: <4AE06E79.9080202@attglobal.net> Message-ID: <20091022155818.GE4948@dan.olp.net> On 22/10/09?22:38?+0800, John Mok wrote: >Oct 22 15:35:02 imapsv01 cyrus/imap[19466]: badlogin: >John.sml.citizen.co.jp [10.144.1.192] GSSAPI [SASL(-13): authentication >failure: user komatsuj at go.citizen.co.jp is not allowed to proxy] > >I checked with imtest and it passed successfully :- > > >imtest -m GSSAPI imapsv01.grt.citizen.co.jp > >The IMAP config. /etc/imapd.conf follows :- > >.... >virtdomains: yes >defaultdomain: grt.citizen.co.jp >sasl_pwcheck_method: saslauthd The "...not allowed to proxy" would seem to indicate that the client is sending an authorization identity, and that it does not match the authentication identity derived from GSSAPI. What does your 'loginrealms:' entry look like in imapd.conf? Are you specifying a(n authorization) username within the email client? If so, try including go.citizen.co.jp in your loginrealms config, and configuring 'komatsuj at go.citizen.co.jp' as your authorization identity in your client, or perhaps not specify it at all. -- Dan White From david at touzeau.eu Thu Oct 22 17:28:04 2009 From: david at touzeau.eu (David Touzeau) Date: Thu, 22 Oct 2009 23:28:04 +0200 Subject: cyrus replication : master to replica and replica to master In-Reply-To: <20091022090704.GA1042@brong.net> References: <1255908245.21631.6.camel@dtouzeau-laptop> <1256150711.21117.2.camel@dtouzeau-laptop> <20091021223120.GA3466@brong.net> <5801465c0910220056x68d4d51dpfe63e1aa607fa227@mail.gmail.com> <20091022090704.GA1042@brong.net> Message-ID: <1256246884.17168.12.camel@dtouzeau-laptop> On Thu, Oct 22, 2009 at 12:56:03AM -0700, Jon . wrote: > On Wed, Oct 21, 2009 at 9:20 PM, Rob Mueller wrote: > ... > > > The difference between "in theory this would work" and the practice of > > actually doing it are huge. Basically it works only if you are 100% sure > > that only one side is ever being accessed at a time. eg. IMAP/POP/LMTP/etc. Pretty much. With appropriate fencing, non-local bind and a service IP address that's feasible. But Rob won't let me do it. Fair enough too, it's pretty messy. > ... > > > In other words, DON'T DO THIS. > > > > Rob Yeah, yeah. I know. I could have worded it a bit more strongly. "Nobody's ever done it because it's really tricky to get right and you'll lose data for sure if you don't know what you are doing". > What are the particular bits that could conflict and have undesirable > results? Metadata, messages, entire mailboxes? In this hypothetical > active/active configuration, what exactly what could an IMAP client > potentially do to create undesirable results? Yes. Those things. Any and/or all. Try thinking about a folder rename at one end and a copy/expunge cycle between folders at the other end and resolving the resultant mess. Basically this is tricky stuff that nobody does particularly well. Generic sync is a hard problem[tm], and the Cyrus code doesn't even try. In particular, it doesn't track deltas. To get even halfway good tracking of changes, you need three things: 1) current state of A 2) current state of B 3) state last time A and B were in sync even better is knowing the changes that were made and resolving them. But without even this much information, consider the following. A: UID 5 is SEEN b: UID 5 is UNSEEN what should be the result? > Would it be a huge undertaking to timestamp data that is to be > replicated to another Cyrus daemon for the receiving Cyrus daemon to > know which conflicting pieces of data to drop in favor of newer data? Timestamp each piece of metadata individually, yes - it would be a huge undertaking. > Right now I have a client who needs 130 or so users on a private mail > server and has two cheap 1U Dell servers to work with. Ideally they > are to be put in physically distanced data-centers for redundancy to > one another. > > Combined with the hypothetical replication of timestamped data > describe above, wouldn't setting the fqdn imap.example.com to resolve > two IP addresses so users' IMAP clients can fall back should an IMAP > storage server be unavailable (with at least the most recent data > replication of any kind is able to provide) make for a much simpler > and more elegant solution than DRBD, clustered filesystems, or > introducing more machines just for load balancing / resolving to an > available IMAP daemon? Also, wouldn't timestamps also hypothetically > resolve the inevitable split-brain situations clients would create? > I assume they don't like losing messages. If you really, REALLY want > to go down that path I would at least take FastMail's patch that checks > the GUID if the same message exists on both ends and refuses to overwrite > if the message contents differ. This is half a solution, you then need > to resolve the issue (we LOCALDELETE the original message at both ends so > it doesn't even wind up in the .expunge file, then we append BOTH messages > with brand new UIDs and set the flags they used to have on the master - > finally syncing the resulting mailbox again so both messages are on both > ends - but the code for that isn't in our Cyrus patches - it's a standalone > script) > And that's just for the split brain that results when a machine dies for > whatever reason (it happened last night incidentally - one of our external > RAID units had an "episode" and decided to stop talking to the server. It > looks like a couple of timeouts on a failing drive tickled a firmware bug > and resulted in the inbuilt OS locking up. Software, you have to love it. > Embedded at all layers. So many firmwares to keep up-to-date!) - we don't > have multi-directional replication. ------------------------------------------------------------------------------------------------------------------------ Our approach to utilisation handling has been documented here plenty of times. Basically we run multiple instances of Cyrus on each machine, so every server has both masters and replicas. We can shut down any one machine just by switching roles (a shutdown and restart of each end with new configs) ------------------------------------------------------------------------------------------------------------------------ Dear Bron... have you some configuration examples somewhere about hin kind of structure ?? From david.lang at digitalinsight.com Thu Oct 22 17:43:35 2009 From: david.lang at digitalinsight.com (David Lang) Date: Thu, 22 Oct 2009 14:43:35 -0700 (PDT) Subject: cyrus replication : master to replica and replica to master In-Reply-To: <1256246884.17168.12.camel@dtouzeau-laptop> References: <1255908245.21631.6.camel@dtouzeau-laptop><1256150711.21117.2.camel@dtouzeau-laptop><20091021223120.GA3466@brong.net><5801465c0910220056x68d4d51dpfe63e1aa607fa227@mail.gmail.com><20091022090704.GA1042@brong.net> <1256246884.17168.12.camel@dtouzeau-laptop> Message-ID: On Thu, 22 Oct 2009, David Touzeau wrote: > On Thu, Oct 22, 2009 at 12:56:03AM -0700, Jon . wrote: >> On Wed, Oct 21, 2009 at 9:20 PM, Rob Mueller wrote: >> ... >> >>> The difference between "in theory this would work" and the practice > of >>> actually doing it are huge. Basically it works only if you are 100% > sure >>> that only one side is ever being accessed at a time. eg. > IMAP/POP/LMTP/etc. > > Pretty much. With appropriate fencing, non-local bind and a service IP > address that's feasible. But Rob won't let me do it. Fair enough too, > it's pretty messy. implementing this should not be that hard allow non-local bind in /etc/sysctl heartbeat (linux-ha.org) can handle moving the service IP and fencing (up to and including turning a box off if the cluster decides that it has failed) I've been doing this (without going to the extent of turning the failed box off) on my firewalls for years. it sounds more complicated than it is. David Lang From brong at fastmail.fm Thu Oct 22 18:39:39 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 23 Oct 2009 09:39:39 +1100 Subject: cyrus replication : master to replica and replica to master In-Reply-To: References: <1255908245.21631.6.camel@dtouzeau-laptop> <1256150711.21117.2.camel@dtouzeau-laptop> <20091021223120.GA3466@brong.net> <5801465c0910220056x68d4d51dpfe63e1aa607fa227@mail.gmail.com> <20091022090704.GA1042@brong.net> <1256246884.17168.12.camel@dtouzeau-laptop> Message-ID: <20091022223939.GA28379@brong.net> On Thu, Oct 22, 2009 at 02:43:35PM -0700, David Lang wrote: > implementing this should not be that hard > > allow non-local bind in /etc/sysctl > > heartbeat (linux-ha.org) can handle moving the service IP and fencing (up to and > including turning a box off if the cluster decides that it has failed) > > I've been doing this (without going to the extent of turning the failed box off) > on my firewalls for years. it sounds more complicated than it is. I've seen heartbeat get split brain before. We gave up on it. We do all our fencing via humans now! Check the KVM, kick the box, manually run the failover script. Bron. From robm at fastmail.fm Thu Oct 22 20:35:25 2009 From: robm at fastmail.fm (Rob Mueller) Date: Fri, 23 Oct 2009 11:35:25 +1100 Subject: cyrus replication : master to replica and replica to master In-Reply-To: <1256203098.8629.1341361091@webmail.messagingengine.com> References: <1255908245.21631.6.camel@dtouzeau-laptop><1256150711.21117.2.camel@dtouzeau-laptop><20091021223120.GA3466@brong.net><5801465c0910220056x68d4d51dpfe63e1aa607fa227@mail.gmail.com> <1256203098.8629.1341361091@webmail.messagingengine.com> Message-ID: <20D4F59CCB554B75BC411A463BE848E1@Atticus> > Client A: upload message to Inbox, gets UID 100 > At the same time, Client B: upload message to Inbox, gets UID 100 > > You can't have two messages with the same UID. > > There's 3 solutions I can see: > > 1. Mysql solves this by having interleving id's on separate servers (eg. > auto-increment column on server A is odd numbers, on server B it's > even numbers). I guess you could in theory do the same with IMAP > (though I'd have to double check the spec), but it would create > really annoying UID lists because you basically lose the ability to > use things like 30:50. One other option would be to alternate in > 100's or something like that (eg. 1-100 on s1, 101-200 on s2, etc) I realised another potential problem with this. The IMAP spec says UIDs must be incremental. So if you upload a message to s1 and it gets UID 100, and upload on s2, and it gets UID 200, then when you upload the next message on s1, it has to get UID 300. So you have to make sure that if any UIDs are allocated in a "higher range", you have to jump to the next range. This could cause you to run out of UIDs quite quickly in pathalogical back and forth cases (eg 2 IMAP clients connected separately to s1 & s2 both uploading a bunch of messages to the same folder). > 3. Use some conflict resolution strategy. If some client uploads UID 100 > on s1, and another uploads UID 100 on s2, then when the conflict is > noticed, both sides have to delete + expunge the message (because > different IMAP clients might have different ideas on what message UID > 100 is) and create new UIDs 101 and 102 with the two messages. This > can be messy because if a POP client is connected, you can't alter > the mailbox at all because the message list isn't allowed to change > under the POP clients feet, so connected POP clients could cause > nasty locking issues. As an FYI, this is basically what we currently do with active/passive replication if we have to do a "hard" failover, but we have to do it all by hand. In a controlled failover, we make sure all the sync logs are correctly played before switching roles, so we know master/replica are in sync when we change roles. However if a server hard fails (eg kernel panic or some other OS lockup), then we may have to switch roles, without knowing if the logs have been cleanly played. When we get the dead server back online, it may be that a message with the same UID exists on both ends. In the current cyrus code, the master wins, and overwrites the message on the replica. That's dangerous, and might end up destroying messages. We have a patch for cyrus that checks for this case, and compares the GUID of the messages (basically SHA1 of message content), and if they're different, it refuses to overwrite the message, and instead just logs a message to syslog. We have another script that notices those syslog messages, and emails us. We then have yet another script, that lets us inspect the offending message on both sides (eg master and replica), to see what the situation is. In most cases, it's a message that was clearly delivered to one side, but not replicated to the other before the machine crashed. We can then give a flag to the script that makes it delete the UID on both sides, and then re-appends both messages to the current master server, causing both messages to get new UIDs. Ideally, this whole process would be automated, and at some point we probably will make our scripts do it automatically, but it would probably be nice if this was eventually pushed down into the cyrus replication protocol somehow. As Bron noted, there would still be lots of other things to make active-active replication work properly (thinking about folder renames while someone else has a folder selected on the other server really scares me), but at least this would deal with the most common active-passive case where a non-graceful failover occured. Rob From woods-cyrus at weird.com Thu Oct 22 21:15:33 2009 From: woods-cyrus at weird.com (Greg A. Woods) Date: Thu, 22 Oct 2009 21:15:33 -0400 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: <1256072052.3739.4.camel@badjo> References: <1256060184.2946.53.camel@skunk> <1256072052.3739.4.camel@badjo> Message-ID: At Tue, 20 Oct 2009 22:54:12 +0200, Xavier Bestel wrote: Subject: Re: Exec'ing a script from Cyrus when imapd has a client > > Le mardi 20 octobre 2009 ? 13:00 -0700, David Lang a ?crit : > > On Tue, 20 Oct 2009, Greg A. Woods wrote: > > > > > At Tue, 20 Oct 2009 19:36:24 +0200, Xavier Bestel wrote: > > > Subject: Exec'ing a script from Cyrus when imapd has a client > > >> > > >> I have a small install with cyrus-imapd 2.3.14, which reads some of its > > >> mails with fetchmail. To limit the delay in mail delivery, fetchmail > > >> awakes each minute to get mails. > > >> What I would like is let fetchmail do that only when there's a client > > >> actually reading its mails, i.e. an MUA actually connected to imapd. > > > > > > I don't get it. Are you saying you are using fetchmail to inject > > > messages into a locally running Cyrus install which you then connect to > > > with a locally running IMAP MUA? > > > > I think what he is saying is that he does not have a MTA. he uses fetchmail to > > download mail from elsewhere and put it in cyrus. > > > > currently he crons fetchmail to run once a min so that when people are logged in > > they see new mail with low latencies. > > > > however, if nobody is logged in to Cyrus, this is a waste of time, and he would > > be better off running fetchmail less frequently (or not at all). > > > > so he is asking if there is a way to tell if anyone is connected to Cyrus or > > not, so that if not he can skip the fetchmail run. > > Yes, that's it, precisely. OK, that's just the kind of crazy overkill and complexity I was talking about. If you don't have an MTA (but you do have an IMAP-capable MUA), then you really do not need or want Cyrus, and you certainly do not want fetchmail either. Just use your IMAP MUA directly for goodness sake! Why all this madness of many unnecessary layers that are just causing way too much confusion? That said, there are a million hacks that could be used, but all they do is increase the complexity ever more. Cyrus IMAP itself doesn't directly support this kind of thing though. -- Greg A. Woods +1 416 218-0098 VE3TCP RoboHack Planix, Inc. Secrets of the Weird From woods-cyrus at weird.com Thu Oct 22 21:17:30 2009 From: woods-cyrus at weird.com (Greg A. Woods) Date: Thu, 22 Oct 2009 21:17:30 -0400 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: <1256072203.3739.6.camel@badjo> References: <1256060184.2946.53.camel@skunk> <200910202222.32607.steffo76@gmx.de> <1256072203.3739.6.camel@badjo> Message-ID: At Tue, 20 Oct 2009 22:56:42 +0200, Xavier Bestel wrote: Subject: Re: Exec'ing a script from Cyrus when imapd has a client > > That may be an answer, yes. However I'd have preferred if the signal > came from Cyrus directly (I think it'sa more robust solution). Not using Cyrus or fetchmail at all in this scenario is pretty much infinitely more robust. :-) -- Greg A. Woods +1 416 218-0098 VE3TCP RoboHack Planix, Inc. Secrets of the Weird From woods-cyrus at weird.com Thu Oct 22 21:23:34 2009 From: woods-cyrus at weird.com (Greg A. Woods) Date: Thu, 22 Oct 2009 21:23:34 -0400 Subject: cyrus - sieve returning some text with EVERY email received In-Reply-To: <4AE06B7B.7080607@hec.ca> References: <4AE06B7B.7080607@hec.ca> Message-ID: At Thu, 22 Oct 2009 10:26:03 -0400, Eddy Beliveau wrote: Subject: cyrus - sieve returning some text with EVERY email received > > One of our user would like to return some text, for *every* email, that > he received > like: > "Hi, we just received your email. It will be process shortly. Thanks." This is an MUA "problem" -- not something that should ever be dealt with in any way whatsoever by the mail transport and delivery mechanisms. Actually what your user is asking for is not something that should ever be done in the first place. I.e. never implement automated tools that can reply to messages without building in and carefully testing significant safeguards. Reliable tools that do reply automatically to mail will only send one reply per week or so to a given address because to do otherwise invites abuse and allows for denial of service attacks. Teach your users to use e-mail responsibly. -- Greg A. Woods +1 416 218-0098 VE3TCP RoboHack Planix, Inc. Secrets of the Weird From david.lang at digitalinsight.com Thu Oct 22 21:43:41 2009 From: david.lang at digitalinsight.com (David Lang) Date: Thu, 22 Oct 2009 18:43:41 -0700 (PDT) Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: References: <1256060184.2946.53.camel@skunk> <1256072052.3739.4.camel@badjo> Message-ID: On Thu, 22 Oct 2009, Greg A. Woods wrote: > At Tue, 20 Oct 2009 22:54:12 +0200, Xavier Bestel wrote: > Subject: Re: Exec'ing a script from Cyrus when imapd has a client >> >> Le mardi 20 octobre 2009 ? 13:00 -0700, David Lang a ?crit : >>> On Tue, 20 Oct 2009, Greg A. Woods wrote: >>> >>>> At Tue, 20 Oct 2009 19:36:24 +0200, Xavier Bestel wrote: >>>> Subject: Exec'ing a script from Cyrus when imapd has a client >>>>> >>>>> I have a small install with cyrus-imapd 2.3.14, which reads some of its >>>>> mails with fetchmail. To limit the delay in mail delivery, fetchmail >>>>> awakes each minute to get mails. >>>>> What I would like is let fetchmail do that only when there's a client >>>>> actually reading its mails, i.e. an MUA actually connected to imapd. >>>> >>>> I don't get it. Are you saying you are using fetchmail to inject >>>> messages into a locally running Cyrus install which you then connect to >>>> with a locally running IMAP MUA? >>> >>> I think what he is saying is that he does not have a MTA. he uses fetchmail to >>> download mail from elsewhere and put it in cyrus. >>> >>> currently he crons fetchmail to run once a min so that when people are logged in >>> they see new mail with low latencies. >>> >>> however, if nobody is logged in to Cyrus, this is a waste of time, and he would >>> be better off running fetchmail less frequently (or not at all). >>> >>> so he is asking if there is a way to tell if anyone is connected to Cyrus or >>> not, so that if not he can skip the fetchmail run. >> >> Yes, that's it, precisely. > > > OK, that's just the kind of crazy overkill and complexity I was talking about. > > If you don't have an MTA (but you do have an IMAP-capable MUA), then you > really do not need or want Cyrus, and you certainly do not want > fetchmail either. > > Just use your IMAP MUA directly for goodness sake! there can be cases where you are providing mail services for several people, or have multiple machines you use yourself where having an IMAP server is worthwhile. and if you are going to setup an IMAP server, why use anything less than the best? ;-) now, it's unusual to use something like this without having a full MTA, but it's not unheard of. David Lang From awilliam at whitemice.org Fri Oct 23 09:39:35 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Fri, 23 Oct 2009 09:39:35 -0400 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: <20091020233447.GA17677@brong.net> References: <1256060184.2946.53.camel@skunk> <200910202222.32607.steffo76@gmx.de> <1256072203.3739.6.camel@badjo> <20091020233447.GA17677@brong.net> Message-ID: <1256305175.5237.9.camel@linux-m3mt> > Getting Cyrus to actually trigger something is much more complex - but > you could easily do something with an inotify on the directory if you > want smaller than a minute resolution. -- openSUSE Linux for human beings who need to get things done. From dpc22 at cam.ac.uk Fri Oct 23 10:42:08 2009 From: dpc22 at cam.ac.uk (David Carter) Date: Fri, 23 Oct 2009 15:42:08 +0100 (BST) Subject: cyrus replication : master to replica and replica to master In-Reply-To: <20091022223939.GA28379@brong.net> References: <1255908245.21631.6.camel@dtouzeau-laptop> <1256150711.21117.2.camel@dtouzeau-laptop> <20091021223120.GA3466@brong.net> <5801465c0910220056x68d4d51dpfe63e1aa607fa227@mail.gmail.com> <20091022090704.GA1042@brong.net> <1256246884.17168.12.camel@dtouzeau-laptop> <20091022223939.GA28379@brong.net> Message-ID: On Fri, 23 Oct 2009, Bron Gondwana wrote: > I've seen heartbeat get split brain before. We gave up on it. We do > all our fencing via humans now! Check the KVM, kick the box, manually > run the failover script. Some of my colleagues have had a lot of grief with Heartbeat going split brain. It seems to really be designed for a pair of machines sitting next to each other in a rack with a serial link for the heartbeat, rather servers installed in a pair of machine rooms three miles apart. We do manual failover with our Cyrus mailstores: I would rather 1/8th of my users had an outage of a couple of hours (and typically just a few minutes) than end up with a split brain. On the one occasion in five years that we did end up with a Cyrus split brain (replication failed because of a memory DIMM error and then the entire master failed a few minutes later) it was easy enough to fish missing messages out of the dead system the following day and reinject them using LMTP. Certainly easier than reengineering the entire Cyrus mailstore to allow active/active replication. On Wed, Oct 21, 2009 at 08:45:11PM +0200, David Touzeau wrote: > I would like to know if it is possible to SET the replica has the master > too in order to replicate new mail saved on the replica to the master > and vis versa In this case it should be turn to active/active.. We do this to a limited degree: the set of active users on a pair of mailstores can be partitioned and bounced back and forth between the two servers in a pair. This is mostly useful for load balancing between our two machine rooms, or migrating all the users off a master so that we can patch and reboot without any user visible downtime. However this is using my own replication code rather than the branch which was rewritten into Cyrus by Ken. I have additional safeguards to stop sync_client from overwriting the master data in a pair (which has only ever happened because of stupidity on my part when testing). I've never used the standard replication code in Cyrus other than to backport (sideport?) additional features such as CONDSTORE and GUID support. Given the grief Fastmail had with the early Cyrus replication code I think that I'm rather glad about this. Every once in a while I think about moving to standard Cyrus replication. Unfortunately there are a lot of warts that I really don't like. It is much easier to just drop my own replication code onto new versions of Cyrus (typically < 5 minutes work each time). That was one of my original design objectives. -- David Carter Email: David.Carter at ucs.cam.ac.uk University Computing Service, Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. From woods-cyrus at weird.com Fri Oct 23 15:57:38 2009 From: woods-cyrus at weird.com (Greg A. Woods) Date: Fri, 23 Oct 2009 15:57:38 -0400 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: References: <1256060184.2946.53.camel@skunk> <1256072052.3739.4.camel@badjo> Message-ID: At Thu, 22 Oct 2009 18:43:41 -0700 (PDT), David Lang wrote: Subject: Re: Exec'ing a script from Cyrus when imapd has a client > > there can be cases where you are providing mail services for several people, or > have multiple machines you use yourself where having an IMAP server is > worthwhile. Neither of those things make any real sense whatsoever. They certainly don't define any clear requirements that make sense in this context. Every modern and useful IMAP-capable MUA can collect e-mail from any combination of many IMAP servers anywhere and everywhere all at once. If "fetchmail" can fetch the mail from an IMAP server, then so can any MUA. Just get rid of all the unnecessary complexity in the middle and just use the MUA for what it's designed to be used for! > now, it's unusual to use something like this without having a full MTA, but it's > not unheard of. It's not unusual for people to create all kinds of crazy complicated setups that have no real purpose, in every domain in life. I'm sure I make my own life more complicated than it needs to be in some ways. However things do not _need_ to be made more complicated than necessary, Here the OP's question provides a perfect clue showing that something is far more complicated than it needs to be because we see that it will even have to get more complex (and even less robust) before it begins to work the way it would actually work without any of this unnecessary complexity in the middle in the first place. -- Greg A. Woods +1 416 218-0098 VE3TCP RoboHack Planix, Inc. Secrets of the Weird From david.lang at digitalinsight.com Fri Oct 23 16:00:34 2009 From: david.lang at digitalinsight.com (David Lang) Date: Fri, 23 Oct 2009 13:00:34 -0700 (PDT) Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: References: <1256060184.2946.53.camel@skunk> <1256072052.3739.4.camel@badjo> Message-ID: On Fri, 23 Oct 2009, Greg A. Woods wrote: > At Thu, 22 Oct 2009 18:43:41 -0700 (PDT), David Lang wrote: > Subject: Re: Exec'ing a script from Cyrus when imapd has a client >> >> there can be cases where you are providing mail services for several people, or >> have multiple machines you use yourself where having an IMAP server is >> worthwhile. > > Neither of those things make any real sense whatsoever. They certainly > don't define any clear requirements that make sense in this context. > > Every modern and useful IMAP-capable MUA can collect e-mail from any > combination of many IMAP servers anywhere and everywhere all at once. > > If "fetchmail" can fetch the mail from an IMAP server, then so can any > MUA. > > Just get rid of all the unnecessary complexity in the middle and just > use the MUA for what it's designed to be used for! as long as you are willing to limit yourself to a single MUA on a single desktop/laptop. if you want to be able to access your mail from different devices you need a mail server, not just a MUA David Lang > > >> now, it's unusual to use something like this without having a full MTA, but it's >> not unheard of. > > It's not unusual for people to create all kinds of crazy complicated > setups that have no real purpose, in every domain in life. > > I'm sure I make my own life more complicated than it needs to be in some ways. > > However things do not _need_ to be made more complicated than necessary, > > Here the OP's question provides a perfect clue showing that something is > far more complicated than it needs to be because we see that it will > even have to get more complex (and even less robust) before it begins to > work the way it would actually work without any of this unnecessary > complexity in the middle in the first place. > > From woods-cyrus at weird.com Fri Oct 23 16:30:55 2009 From: woods-cyrus at weird.com (Greg A. Woods) Date: Fri, 23 Oct 2009 16:30:55 -0400 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: References: <1256060184.2946.53.camel@skunk> <1256072052.3739.4.camel@badjo> Message-ID: At Fri, 23 Oct 2009 13:00:34 -0700 (PDT), David Lang wrote: Subject: Re: Exec'ing a script from Cyrus when imapd has a client > > as long as you are willing to limit yourself to a single MUA on a single > desktop/laptop. > > if you want to be able to access your mail from different devices you need a > mail server, not just a MUA Huh? What in the heck are you talking about? I run multiple IMAP clients (some on the same computer, some on different computers) all _simultaneously_, all accessing a half-dozen different mail accounts on different servers around the Internet. All my MUAs access the same folders and same messages directly, and simultaneously. I certainly don't need yet another IMAP server and a whole bunch of unnecessary complexity with things like fetchmail just to do this. What I would really like to learn is why anyone would falsely believe that they do somehow need their own IMAP server for this reason. There must be some false conception or expectation permeating some parts of the ether out there. -- Greg A. Woods +1 416 218-0098 VE3TCP RoboHack Planix, Inc. Secrets of the Weird From carson at taltos.org Fri Oct 23 16:32:52 2009 From: carson at taltos.org (Carson Gaspar) Date: Fri, 23 Oct 2009 13:32:52 -0700 Subject: cyrus replication : master to replica and replica to master In-Reply-To: References: <1255908245.21631.6.camel@dtouzeau-laptop> <1256150711.21117.2.camel@dtouzeau-laptop> <20091021223120.GA3466@brong.net> <5801465c0910220056x68d4d51dpfe63e1aa607fa227@mail.gmail.com> <20091022090704.GA1042@brong.net> <1256246884.17168.12.camel@dtouzeau-laptop> <20091022223939.GA28379@brong.net> Message-ID: <4AE212F4.8010907@taltos.org> On 10/23/09 7:42 AM, David Carter wrote: > On Fri, 23 Oct 2009, Bron Gondwana wrote: > >> I've seen heartbeat get split brain before. We gave up on it. We do >> all our fencing via humans now! Check the KVM, kick the box, manually >> run the failover script. > > Some of my colleagues have had a lot of grief with Heartbeat going split > brain. It seems to really be designed for a pair of machines sitting next > to each other in a rack with a serial link for the heartbeat, rather > servers installed in a pair of machine rooms three miles apart. To be fair to heartbeat, if you're getting unexpected split brain, then you have configured it incorrectly. A 2 node cluster without _extremely_ reliable communication and fencing between the nodes requires a tie-breaker service. This is true of any clustering technology I have ever seen. Heartbeat provides a light-weight quorum service for just this purpose. Of course if you only have 2 sites, and the site with the "extra" vote goes down, you lose the service. Anything else requires 3 sites, or a "meatware" failover decision. -- Carson From david.lang at digitalinsight.com Fri Oct 23 16:37:30 2009 From: david.lang at digitalinsight.com (David Lang) Date: Fri, 23 Oct 2009 13:37:30 -0700 (PDT) Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: References: <1256060184.2946.53.camel@skunk> <1256072052.3739.4.camel@badjo> Message-ID: On Fri, 23 Oct 2009, Greg A. Woods wrote: > At Fri, 23 Oct 2009 13:00:34 -0700 (PDT), David Lang wrote: > Subject: Re: Exec'ing a script from Cyrus when imapd has a client >> >> as long as you are willing to limit yourself to a single MUA on a single >> desktop/laptop. >> >> if you want to be able to access your mail from different devices you need a >> mail server, not just a MUA > > Huh? What in the heck are you talking about? > > I run multiple IMAP clients (some on the same computer, some on > different computers) all _simultaneously_, all accessing a half-dozen > different mail accounts on different servers around the Internet. > > All my MUAs access the same folders and same messages directly, and > simultaneously. > > I certainly don't need yet another IMAP server and a whole bunch of > unnecessary complexity with things like fetchmail just to do this. I possibly missed it, but I didn't see anything that said that fetchmail was grabbing things via IMAP. if the remote account is POP then doing something like this has value if you have intermittent/expensive-per-min internet connectivity doing something like this has value. I did something similar to this several years ago where a non-profit only had dial-up internet. I ran a local cyrus server and then had a process to bring up the internet connection as-needed. In that case I just polled for mail every half hour and people were willing to live with that at that time. In this case I actually used UUCP through a fixed mail server to do the routing instead of fetchmail, but the basic concept is the same. another reason to run your own server is just to be free from quotas. many ISPs have small mail quotas. David Lang > What I would really like to learn is why anyone would falsely believe > that they do somehow need their own IMAP server for this reason. There > must be some false conception or expectation permeating some parts of > the ether out there. > > From rudy.gevaert at ugent.be Mon Oct 26 03:48:59 2009 From: rudy.gevaert at ugent.be (Rudy Gevaert) Date: Mon, 26 Oct 2009 08:48:59 +0100 Subject: lmtp trickery In-Reply-To: References: <1255908245.21631.6.camel@dtouzeau-laptop> <1256150711.21117.2.camel@dtouzeau-laptop> <20091021223120.GA3466@brong.net> <5801465c0910220056x68d4d51dpfe63e1aa607fa227@mail.gmail.com> <20091022090704.GA1042@brong.net> <1256246884.17168.12.camel@dtouzeau-laptop> <20091022223939.GA28379@brong.net> Message-ID: <20091026084859.41905tw7sc9mwd4b@langoest.ugent.be> Citeren David Carter : > On the one occasion in five years that we did end up with a Cyrus split > brain (replication failed because of a memory DIMM error and then the > entire master failed a few minutes later) it was easy enough to fish > missing messages out of the dead system the following day and reinject > them using LMTP. Certainly easier than reengineering the entire Cyrus > mailstore to allow active/active replication. Hi David, I was wondering how you did that. a) the fishing part: I don't see any way how I can map a log entry (message id) to a file b) injecting through lmtp: piping messages via stdin? Thanks!! -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Rudy Gevaert Rudy.Gevaert at UGent.be tel:+32 9 264 4734 Directie ICT, afd. Infrastructuur Direction ICT, Infrastructure dept. Groep Systemen Systems group Universiteit Gent Ghent University Krijgslaan 281, gebouw S9, 9000 Gent, Belgie www.UGent.be -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- From brong at fastmail.fm Mon Oct 26 04:01:46 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Mon, 26 Oct 2009 19:01:46 +1100 Subject: lmtp trickery In-Reply-To: <20091026084859.41905tw7sc9mwd4b@langoest.ugent.be> References: <1255908245.21631.6.camel@dtouzeau-laptop><1256150711.21117.2.camel@dtouzeau-laptop><20091021223120.GA3466@brong.net><5801465c0910220056x68d4d51dpfe63e1aa607fa227@mail.gmail.com><20091022090704.GA1042@brong.net><1256246884.17168.12.camel@dtouzeau-laptop><20091022223939.GA28379@brong.net> <20091026084859.41905tw7sc9mwd4b@langoest.ugent.be> Message-ID: <1256544106.24643.1341957791@webmail.messagingengine.com> On Mon, 26 Oct 2009 08:48 +0100, "Rudy Gevaert" wrote: > > Citeren David Carter : > > > > On the one occasion in five years that we did end up with a Cyrus split > > brain (replication failed because of a memory DIMM error and then the > > entire master failed a few minutes later) it was easy enough to fish > > missing messages out of the dead system the following day and reinject > > them using LMTP. Certainly easier than reengineering the entire Cyrus > > mailstore to allow active/active replication. > > Hi David, > > I was wondering how you did that. > > a) the fishing part: I don't see any way how I can map a log entry > (message id) to a file > b) injecting through lmtp: piping messages via stdin? Our system just makes IMAP connections as 'admin' to both ends and injects the message via IMAP. As for mapping messages to files... well, we have the auditlog patch, which helps considerably :) Bron. -- Bron Gondwana brong at fastmail.fm From woods-cyrus at weird.com Mon Oct 26 12:36:21 2009 From: woods-cyrus at weird.com (Greg A. Woods) Date: Mon, 26 Oct 2009 12:36:21 -0400 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: References: <1256060184.2946.53.camel@skunk> <1256072052.3739.4.camel@badjo> Message-ID: At Fri, 23 Oct 2009 13:37:30 -0700 (PDT), David Lang wrote: Subject: Re: Exec'ing a script from Cyrus when imapd has a client > > I possibly missed it, but I didn't see anything that said that fetchmail was > grabbing things via IMAP. Yup, I think you missed it. > if you have intermittent/expensive-per-min internet connectivity doing something > like this has value. Nope, not really. All modern useful IMAP clients can work offline too. All another IMAP server is doing is adding to the complexity _and_ decreasing, i.e. lowering, the robustness of the overall solution. > another reason to run your own server is just to be free from quotas. many ISPs > have small mail quotas. All modern useful IMAP clients can also store message locally -- moving them from server to server, or server to local (or back), is as simple as selecting and saving/dragging messages between folders. -- Greg A. Woods +1 416 218-0098 VE3TCP RoboHack Planix, Inc. Secrets of the Weird From xavier.bestel at free.fr Mon Oct 26 12:59:21 2009 From: xavier.bestel at free.fr (Xavier Bestel) Date: Mon, 26 Oct 2009 17:59:21 +0100 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: References: <1256060184.2946.53.camel@skunk> <1256072052.3739.4.camel@badjo> Message-ID: <1256576361.10148.42.camel@skunk> On Mon, 2009-10-26 at 12:36 -0400, Greg A. Woods wrote: > At Fri, 23 Oct 2009 13:37:30 -0700 (PDT), David Lang wrote: > Subject: Re: Exec'ing a script from Cyrus when imapd has a client > > > > I possibly missed it, but I didn't see anything that said that fetchmail was > > grabbing things via IMAP. > > Yup, I think you missed it. > > > if you have intermittent/expensive-per-min internet connectivity doing something > > like this has value. > > Nope, not really. All modern useful IMAP clients can work offline too. > > All another IMAP server is doing is adding to the complexity _and_ > decreasing, i.e. lowering, the robustness of the overall solution. > > > another reason to run your own server is just to be free from quotas. many ISPs > > have small mail quotas. > > All modern useful IMAP clients can also store message locally -- moving > them from server to server, or server to local (or back), is as simple > as selecting and saving/dragging messages between folders. - not all mail providers do IMAP. - not all mail providers doing IMAP guarantee unlimited storage or lifelong mail availability. - some users have accumulated mail providers and want to centralize everything in a trusted server That's what I do with Cyrus: fetching mail from various POP3 sources, and having all where we can access it, from several IMAP clients. That you may find this solution not optimal isn't the question (nor really my problem, in fact), I just wanted my server to rest a bit when unused (spinning off the drives array has measurable power gains), so I wanted to be able to know when Cyrus has connected users. Thanks, Xav -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20091026/a5cf5f43/attachment.html From david.lang at digitalinsight.com Mon Oct 26 13:07:13 2009 From: david.lang at digitalinsight.com (David Lang) Date: Mon, 26 Oct 2009 10:07:13 -0700 (PDT) Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: References: <1256060184.2946.53.camel@skunk> <1256072052.3739.4.camel@badjo> Message-ID: On Mon, 26 Oct 2009, Greg A. Woods wrote: > At Fri, 23 Oct 2009 13:37:30 -0700 (PDT), David Lang wrote: > Subject: Re: Exec'ing a script from Cyrus when imapd has a client >> >> I possibly missed it, but I didn't see anything that said that fetchmail was >> grabbing things via IMAP. > > Yup, I think you missed it. > >> if you have intermittent/expensive-per-min internet connectivity doing something >> like this has value. > > Nope, not really. All modern useful IMAP clients can work offline too. > > All another IMAP server is doing is adding to the complexity _and_ > decreasing, i.e. lowering, the robustness of the overall solution. > >> another reason to run your own server is just to be free from quotas. many ISPs >> have small mail quotas. > > All modern useful IMAP clients can also store message locally -- moving > them from server to server, or server to local (or back), is as simple > as selecting and saving/dragging messages between folders. in my mind, having the IMAP client copy all messages to the local drive goes a long way to defeating the benifits of using IMAP in the first place. what do you consider a 'modern IMAP client' that is actually reasonably efficiant to use? there are a lot of 'IMAP clients' out there that treat IMAP as if it was POP (downloading everything and then working on it locally, taking _no_ advantage of the server capabilities) I am interested in finding such a client because at the moment I am using pine and mulberry, both of which are very good at using the server, but not exactly 'modern'. David Lang From xavier.bestel at free.fr Mon Oct 26 13:37:58 2009 From: xavier.bestel at free.fr (Xavier Bestel) Date: Mon, 26 Oct 2009 18:37:58 +0100 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: References: <1256060184.2946.53.camel@skunk> <1256072052.3739.4.camel@badjo> Message-ID: <1256578678.10148.47.camel@skunk> On Mon, 2009-10-26 at 10:07 -0700, David Lang wrote: > On Mon, 26 Oct 2009, Greg A. Woods wrote: > > > At Fri, 23 Oct 2009 13:37:30 -0700 (PDT), David Lang wrote: > > Subject: Re: Exec'ing a script from Cyrus when imapd has a client > >> > >> I possibly missed it, but I didn't see anything that said that fetchmail was > >> grabbing things via IMAP. > > > > Yup, I think you missed it. > > > >> if you have intermittent/expensive-per-min internet connectivity doing something > >> like this has value. > > > > Nope, not really. All modern useful IMAP clients can work offline too. > > > > All another IMAP server is doing is adding to the complexity _and_ > > decreasing, i.e. lowering, the robustness of the overall solution. > > > >> another reason to run your own server is just to be free from quotas. many ISPs > >> have small mail quotas. > > > > All modern useful IMAP clients can also store message locally -- moving > > them from server to server, or server to local (or back), is as simple > > as selecting and saving/dragging messages between folders. > > in my mind, having the IMAP client copy all messages to the local drive goes a > long way to defeating the benifits of using IMAP in the first place. The drive is not exactly local, it's on a separate server (which does mainly mail and file server), which is accessed remotely or not, depending on who uses it and when. > what do you consider a 'modern IMAP client' that is actually reasonably > efficiant to use? there are a lot of 'IMAP clients' out there that treat IMAP as > if it was POP (downloading everything and then working on it locally, taking > _no_ advantage of the server capabilities) I am interested in finding such a > client because at the moment I am using pine and mulberry, both of which are > very good at using the server, but not exactly 'modern'. I admit I have yet to find the ideal IMAP client, efficiency-wise. But that's another problem. Xav From david.lang at digitalinsight.com Mon Oct 26 13:36:05 2009 From: david.lang at digitalinsight.com (David Lang) Date: Mon, 26 Oct 2009 10:36:05 -0700 (PDT) Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: <1256578678.10148.47.camel@skunk> References: <1256060184.2946.53.camel@skunk> <1256072052.3739.4.camel@badjo> <1256578678.10148.47.camel@skunk> Message-ID: On Mon, 26 Oct 2009, Xavier Bestel wrote: > On Mon, 2009-10-26 at 10:07 -0700, David Lang wrote: >> On Mon, 26 Oct 2009, Greg A. Woods wrote: >> >>> At Fri, 23 Oct 2009 13:37:30 -0700 (PDT), David Lang wrote: >>> Subject: Re: Exec'ing a script from Cyrus when imapd has a client >>>> >>>> I possibly missed it, but I didn't see anything that said that fetchmail was >>>> grabbing things via IMAP. >>> >>> Yup, I think you missed it. >>> >>>> if you have intermittent/expensive-per-min internet connectivity doing something >>>> like this has value. >>> >>> Nope, not really. All modern useful IMAP clients can work offline too. >>> >>> All another IMAP server is doing is adding to the complexity _and_ >>> decreasing, i.e. lowering, the robustness of the overall solution. >>> >>>> another reason to run your own server is just to be free from quotas. many ISPs >>>> have small mail quotas. >>> >>> All modern useful IMAP clients can also store message locally -- moving >>> them from server to server, or server to local (or back), is as simple >>> as selecting and saving/dragging messages between folders. >> >> in my mind, having the IMAP client copy all messages to the local drive goes a >> long way to defeating the benifits of using IMAP in the first place. > > The drive is not exactly local, it's on a separate server (which does > mainly mail and file server), which is accessed remotely or not, > depending on who uses it and when. I was responding to Greg, who was saying that all modern IMAP clients will copy the mail to the local drive so that they can work offline. David Lang >> what do you consider a 'modern IMAP client' that is actually reasonably >> efficiant to use? there are a lot of 'IMAP clients' out there that treat IMAP as >> if it was POP (downloading everything and then working on it locally, taking >> _no_ advantage of the server capabilities) I am interested in finding such a >> client because at the moment I am using pine and mulberry, both of which are >> very good at using the server, but not exactly 'modern'. > > I admit I have yet to find the ideal IMAP client, efficiency-wise. But > that's another problem. > > Xav > > > > From simon.matter at invoca.ch Mon Oct 26 15:21:38 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Mon, 26 Oct 2009 20:21:38 +0100 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: References: <1256060184.2946.53.camel@skunk> <1256072052.3739.4.camel@badjo> <1256578678.10148.47.camel@skunk> Message-ID: <388439939a070f65115b3b07f8b2bc57.squirrel@webmail.bi.corp.invoca.ch> > On Mon, 26 Oct 2009, Xavier Bestel wrote: > >> On Mon, 2009-10-26 at 10:07 -0700, David Lang wrote: >>> On Mon, 26 Oct 2009, Greg A. Woods wrote: >>> >>>> At Fri, 23 Oct 2009 13:37:30 -0700 (PDT), David Lang >>>> wrote: >>>> Subject: Re: Exec'ing a script from Cyrus when imapd has a client >>>>> >>>>> I possibly missed it, but I didn't see anything that said that >>>>> fetchmail was >>>>> grabbing things via IMAP. >>>> >>>> Yup, I think you missed it. >>>> >>>>> if you have intermittent/expensive-per-min internet connectivity >>>>> doing something >>>>> like this has value. >>>> >>>> Nope, not really. All modern useful IMAP clients can work offline >>>> too. >>>> >>>> All another IMAP server is doing is adding to the complexity _and_ >>>> decreasing, i.e. lowering, the robustness of the overall solution. >>>> >>>>> another reason to run your own server is just to be free from quotas. >>>>> many ISPs >>>>> have small mail quotas. >>>> >>>> All modern useful IMAP clients can also store message locally -- >>>> moving >>>> them from server to server, or server to local (or back), is as simple >>>> as selecting and saving/dragging messages between folders. >>> >>> in my mind, having the IMAP client copy all messages to the local drive >>> goes a >>> long way to defeating the benifits of using IMAP in the first place. >> >> The drive is not exactly local, it's on a separate server (which does >> mainly mail and file server), which is accessed remotely or not, >> depending on who uses it and when. > > I was responding to Greg, who was saying that all modern IMAP clients will > copy > the mail to the local drive so that they can work offline. There is another point which can be very important in the corporate world: even if the mail clients are so wonderful that they can connect to multiple servers and copy mail between them and you can configure everything so nicely in a hundred config windows and tabs, maybe you simply don't want to configure it on the client side but prefer to do as much as possible on the server side. Then you can use tools like fetchmail, cronjobs, scripts to glue things together. That way you are also free to use whatever client you want, you don't depend so much on the clients features. Regards, Simon From woods-cyrus at weird.com Tue Oct 27 00:15:08 2009 From: woods-cyrus at weird.com (Greg A. Woods) Date: Tue, 27 Oct 2009 00:15:08 -0400 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: <1256576361.10148.42.camel@skunk> References: <1256060184.2946.53.camel@skunk> <1256072052.3739.4.camel@badjo> <1256576361.10148.42.camel@skunk> Message-ID: At Mon, 26 Oct 2009 17:59:21 +0100, Xavier Bestel wrote: Subject: Re: Exec'ing a script from Cyrus when imapd has a client > > - not all mail providers do IMAP. Too bad, so sad. Seriously. If you really want to use e-mail address at some antique domain where IMAP access is still not available, and where e-mail forwarding is also not available, then you can still just set up your MUA so you can easily move messages from those antiquated POP folders over to suitable IMAP folders on some other system. You do not _need_ fetchmail to do this for you, and you certainly don't need yet another IMAP server to do this for you. Personally I would try desperately to get a forwarding alias set up at any domain I needed to use which didn't offer IMAP access (and probably even if it did offer IMAP). > - not all mail providers doing IMAP guarantee unlimited storage or > lifelong mail availability. Quota limits are easily avoided with any useful modern IMAP client which can move selected messages between folders, including between systems, and including to the "local" system (wherever the MUA runs). > - some users have accumulated mail providers and want to centralize > everything in a trusted server Normally we use forwarding aliases to do that. :-) NEVER EVER fetchmail. I have e-mail addresses at a dozen or so domains, and I could probably have many more if I wished, but I have only two main IMAP accounts (and a few test IMAP accounts at sites where I manage systems and software). All my scattered e-mail addresses forward to one of my main IMAP accounts. I don't need or would I ever want any e-mail address where I cannot set up a forwarding alias. > That you may find this solution not optimal isn't the question (nor > really my problem, in fact), I just wanted my server to rest a bit when > unused (spinning off the drives array has measurable power gains), so I > wanted to be able to know when Cyrus has connected users. Indeed it's really not a Cyrus problem in the first place. You have created the situation for yourself within which you must now either choose to exist, or to change. :-) If you really want a truly robust central e-mail server then you should probably be hosting it at a dedicated co-location provider and then its power consumption will likely be the least of your worries. Ideally you would set it up to run SMTP as well and then use forwarding aliases at any other domains to collect all your e-mail at this one server. You'd still need to do off-site backups too of course. Ignoring the questionable sanity of using fetchail to feed an IMAP server, what you originally asked is really a systems programming question, one more appropriate for a forum related to systems programming for kind of OS you're running your server on. There are potentially dozens of various ways the same end result could be achieved on any one given OS and system configuration. -- Greg A. Woods +1 416 218-0098 VE3TCP RoboHack Planix, Inc. Secrets of the Weird From rudy.gevaert at ugent.be Tue Oct 27 03:18:17 2009 From: rudy.gevaert at ugent.be (Rudy Gevaert) Date: Tue, 27 Oct 2009 08:18:17 +0100 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: References: <1256060184.2946.53.camel@skunk> <1256072052.3739.4.camel@badjo> Message-ID: <20091027081817.19251tt99xark989@langoest.ugent.be> Citeren David Lang : > what do you consider a 'modern IMAP client' that is actually reasonably > efficiant to use? I can't help you answer that question. But I can share my setup. I'm using the offlineimap client so sync my IMAP (Cyrus of course) accounts (2 in fact). I then use mutt to read the maildir. This way I use a normal IMAP client when I'm online (or have a fast connection). When I'm offline or on 3G (or worse) I use the offlineimap tool to sync my mailbox now and then. To me using offlineimap to sync my mailbox I much faster than doing IMAP over slow links. It also gives me a backup of my mailbox very easily. -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Rudy Gevaert Rudy.Gevaert at UGent.be tel:+32 9 264 4734 Directie ICT, afd. Infrastructuur Direction ICT, Infrastructure dept. Groep Systemen Systems group Universiteit Gent Ghent University Krijgslaan 281, gebouw S9, 9000 Gent, Belgie www.UGent.be -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- From brong at fastmail.fm Tue Oct 27 07:26:27 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 27 Oct 2009 22:26:27 +1100 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: References: <1256072052.3739.4.camel@badjo> <1256576361.10148.42.camel@skunk> Message-ID: <20091027112627.GB9747@brong.net> On Tue, Oct 27, 2009 at 12:15:08AM -0400, Greg A. Woods wrote: > Too bad, so sad. Seriously. If you really want to use e-mail address > at some antique domain where IMAP access is still not available, and > where e-mail forwarding is also not available, then you can still just > set up your MUA so you can easily move messages from those antiquated > POP folders over to suitable IMAP folders on some other system. > > You do not _need_ fetchmail to do this for you, and you certainly don't > need yet another IMAP server to do this for you. Ahh, I've worked with people like this. It's not an architecturally "pure" enough idea, so we'll come up with a bunch of client-side hacks instead of solving the actual problem, which I will state my understanding of: "I want a local IMAP server that I can connect to with multiple different clients if necessary, back up sanely, etc. Due to circumstances, I need it to pull mail in rather than be pushed to. I would _prefer_ to not pull all the time, but only when there's a client - any client - connected" That's a viable problem statement. That's a REAL WORLD problem statement. The solutions suggested on this list have basically been "your problem isn't viable, you should do things differently because our perfect world doesn't include this architecture". You know what - I would write a system pretty much just like this if I was stuck at the far end of a slow link. I would run a server and run backups on that, treating the copy on the client as a disposable copy rather than trying to back that up. Honestly, please stop shooting down the architecture. This isn't the place to ask how to do the whole problem, but it is the place to ask "how can I get the triggering data from Cyrus that I want". Here are the answers we have come up with: 1) parse the log file (this can be made reliable, we use it at FastMail for some things) 2) scan the $conf/proc/ directory 3) inotify on the meta directory (actually, this is somewhat bogus if you don't have atime turned on) Oh, here's a clever one: 4) write a custom saslauthd and trigger from it. 1 and 4 will get you instant notification (near enough) 2 will be cheap, especially if you do like we do and put $conf/proc on tmpfs. 3 just sucks. Most complex and least useful. Bron. From brong at fastmail.fm Tue Oct 27 07:30:12 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 27 Oct 2009 22:30:12 +1100 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: <20091027081817.19251tt99xark989@langoest.ugent.be> References: <1256072052.3739.4.camel@badjo> <20091027081817.19251tt99xark989@langoest.ugent.be> Message-ID: <20091027113012.GC9747@brong.net> On Tue, Oct 27, 2009 at 08:18:17AM +0100, Rudy Gevaert wrote: > > Citeren David Lang : > > > what do you consider a 'modern IMAP client' that is actually reasonably > > efficiant to use? > > I can't help you answer that question. But I can share my setup. > > I'm using the offlineimap client so sync my IMAP (Cyrus of course) > accounts (2 in fact). > > I then use mutt to read the maildir. Heh. You are me. > This way I use a normal IMAP client when I'm online (or have a fast > connection). When I'm offline or on 3G (or worse) I use the > offlineimap tool to sync my mailbox now and then. I use mutt and offlineimap or the FastMail web interface. I don't tend to bother with a separate IMAP client - just run offlineimap and then use mutt. > To me using offlineimap to sync my mailbox I much faster than doing > IMAP over slow links. It also gives me a backup of my mailbox very > easily. Offlineimap is slow too - it syncs all the flags and internaldates on every message, every time. You might want this patch: http://github.com/brong/brong-offlineimap/commit/7846ab83c5c45911749ab1cb42569702363a619b You'll need a pretty current Cyrus of course, but then you get COMPRESS=DEFLATE support built in to offlineimap. I find about 80% bandwidth savings when there are few actual changes on a big account. Bron. From rudy.gevaert at ugent.be Tue Oct 27 08:15:55 2009 From: rudy.gevaert at ugent.be (Rudy Gevaert) Date: Tue, 27 Oct 2009 13:15:55 +0100 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: <20091027113012.GC9747@brong.net> References: <1256072052.3739.4.camel@badjo> <20091027081817.19251tt99xark989@langoest.ugent.be> <20091027113012.GC9747@brong.net> Message-ID: <20091027131555.18573lyke255mmij@langoest.ugent.be> Citeren Bron Gondwana : > You might want this patch: > > http://github.com/brong/brong-offlineimap/commit/7846ab83c5c45911749ab1cb42569702363a619b > > You'll need a pretty current Cyrus of course, but then you get > COMPRESS=DEFLATE support built in to offlineimap. I find about > 80% bandwidth savings when there are few actual changes on a big > account. Thanks. When we upgrade to the latest cyrus I'll enable compress and try it! Rudy -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Rudy Gevaert Rudy.Gevaert at UGent.be tel:+32 9 264 4734 Directie ICT, afd. Infrastructuur Direction ICT, Infrastructure dept. Groep Systemen Systems group Universiteit Gent Ghent University Krijgslaan 281, gebouw S9, 9000 Gent, Belgie www.UGent.be -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- From awilliam at whitemice.org Tue Oct 27 08:58:36 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Tue, 27 Oct 2009 08:58:36 -0400 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: <20091027081817.19251tt99xark989@langoest.ugent.be> References: <1256060184.2946.53.camel@skunk> <1256072052.3739.4.camel@badjo> <20091027081817.19251tt99xark989@langoest.ugent.be> Message-ID: <1256648316.5264.8.camel@linux-m3mt> On Tue, 2009-10-27 at 08:18 +0100, Rudy Gevaert wrote: > Citeren David Lang : > > what do you consider a 'modern IMAP client' that is actually reasonably > > efficiant to use? > I can't help you answer that question. But I can share my setup. I use Evolution on my laptop, I don't know if it is "efficient" but it works for online/offline very well [and with "Copy folder content locally for offline operations" I can search mail - and attachments - along with all other documents - with Beagle]. I'm a big fan. > I'm using the offlineimap client so sync my IMAP (Cyrus of course) > accounts (2 in fact). My server is, of course, Cyrus. While I don't use fetchmail to aggregate mail currently I have in the past and don't see anything unreasonable about doing so. Sometimes arraigning for inbound SMTP is either not feasible or just expensive. > I then use mutt to read the maildir. > This way I use a normal IMAP client when I'm online (or have a fast > connection). When I'm offline or on 3G (or worse) I use the > offlineimap tool to sync my mailbox now and then. Yep, I used fetchmail to pull mail into a Cyrus server on a network that only had Inet connectivity over a cell connection [that required a 30ft. tower and high-gain antenna in order to get even that]. > To me using offlineimap to sync my mailbox I much faster than doing > IMAP over slow links. It also gives me a backup of my mailbox very > easily. I'm thinking about using imapsync for a backup solution. I've moved by Cyrus server to a hosted VM at linnode and I'm currently just rsync'ing to a local physcial box for backup. But that kind of stinks. From xavier.bestel at free.fr Tue Oct 27 09:56:57 2009 From: xavier.bestel at free.fr (Xavier Bestel) Date: Tue, 27 Oct 2009 14:56:57 +0100 Subject: Exec'ing a script from Cyrus when imapd has a client In-Reply-To: <1256648316.5264.8.camel@linux-m3mt> References: <1256060184.2946.53.camel@skunk> <1256072052.3739.4.camel@badjo> <20091027081817.19251tt99xark989@langoest.ugent.be> <1256648316.5264.8.camel@linux-m3mt> Message-ID: <1256651817.10148.70.camel@skunk> On Tue, 2009-10-27 at 08:58 -0400, Adam Tauno Williams wrote: > On Tue, 2009-10-27 at 08:18 +0100, Rudy Gevaert wrote: > > To me using offlineimap to sync my mailbox I much faster than doing > > IMAP over slow links. It also gives me a backup of my mailbox very > > easily. > > I'm thinking about using imapsync for a backup solution. I've moved by > Cyrus server to a hosted VM at linnode and I'm currently just rsync'ing > to a local physcial box for backup. But that kind of stinks. I used imapsync to upgrade cyrus (from 1.5 to 2.3), it seems like a pretty solid piece of software. At least it worked like a charm in my case. Xav From jmadden at ivytech.edu Tue Oct 27 15:26:13 2009 From: jmadden at ivytech.edu (John Madden) Date: Tue, 27 Oct 2009 15:26:13 -0400 Subject: 2.3.15, murder, skiplist aborts Message-ID: <4AE74955.3080209@ivytech.edu> Looking through the archives it looks like Bron produced a patch that puts this abort in to prevent corruption (yay) and instead print aborts, but I now have a situation where things appear to be hung and I'm trying to find out why and what I should do to fix it. (This all happened while batch-creating about 300k mailboxes but I only got a couple thousand in before the borking.) The master and backends appear to be ok, but the frontend, where the creates are going, sees this: Oct 27 13:43:34 imap ctl_cyrusdb[20242]: checkpointing cyrus databases Oct 27 13:43:34 imap ctl_cyrusdb[20242]: done checkpointing cyrus databases Oct 27 13:44:00 imap mupdate[3134]: skiplist: checkpointed /var/imap/mailboxes.db (15607 records, 1183084 bytes) in 0 seconds Oct 27 13:44:12 imap imap[20158]: Fatal error: Internal error: assertion failed: cyrusdb_skiplist.c: 727: db->current_txn == NULL Oct 27 13:44:12 imap imap[20158]: skiplist: closed while still locked Oct 27 13:46:42 imap imap[20323]: auxpropfunc error invalid parameter supplied Oct 27 13:46:47 imap imap[20205]: login: localhost.localdomain [127.0.0.1] cyrus PLAIN User logged in Oct 27 13:47:26 imap mupdate[3134]: skiplist: checkpointed /var/imap/mailboxes.db (19188 records, 1454316 bytes) in 1 second Oct 27 13:48:34 imap ctl_cyrusdb[20332]: checkpointing cyrus databases Oct 27 13:48:34 imap ctl_cyrusdb[20332]: done checkpointing cyrus databases Oct 27 13:49:31 imap imap[18754]: skiplist: closed while still locked Oct 27 13:51:11 imap imap[20206]: skiplist: closed while still locked Oct 27 13:53:08 imap imap[18753]: skiplist: closed while still locked Oct 27 13:53:34 imap ctl_cyrusdb[20381]: checkpointing cyrus databases Oct 27 13:53:34 imap ctl_cyrusdb[20381]: done checkpointing cyrus databases Oct 27 13:54:37 imap imap[18758]: skiplist: closed while still locked Does any of that need to be worried about? While running (a multi-threaded "go make a ton of mailboxes as quickly as you can,") I noticed that the connection some of the backends stops. If it's just that the imapd generating the abort is the one responsible, that's fine, but how do we prevent this situation to begin with? John -- John Madden Sr UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden at ivytech.edu From schweizer.martin at gmail.com Tue Oct 27 16:20:03 2009 From: schweizer.martin at gmail.com (Martin Schweizer) Date: Tue, 27 Oct 2009 21:20:03 +0100 Subject: Strange Outlook 2007 mail did not arrive Message-ID: <380ccfd60910271320l6e12636h9d3372b201c1343e@mail.gmail.com> Hello I have a strange issue. My setup is FreeBSD 7.2 amd64 and the newest Cyrus Imapd (.15).: In the maillog from sendmail I see that the mail will transfered to Cyrus but it never arrived in the addressed mailbox (of course I tried different addresses...). I find out that, if I re-send a specific mail the mail did not arrived but if I write the mail new (same text and subject) it arrived. The only difference I find, was the message id field in the header. (sorry for wrapping the lines) The bad one: msgid= The good one: msgid=<000401ca56ea$6babfd10$4303f730$@ch> Any ideas are welcome. Kind regards, -- Martin Schweizer schweizer.martin at gmail.com Tel.: +41 32 512 48 54 (VoIP) Fax: +1 619 3300587 From simon.matter at invoca.ch Tue Oct 27 16:31:24 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Tue, 27 Oct 2009 21:31:24 +0100 Subject: Strange Outlook 2007 mail did not arrive In-Reply-To: <380ccfd60910271320l6e12636h9d3372b201c1343e@mail.gmail.com> References: <380ccfd60910271320l6e12636h9d3372b201c1343e@mail.gmail.com> Message-ID: <5b5afe62feb164c16e632223ee9b22c0.squirrel@webmail.bi.corp.invoca.ch> > Hello > > I have a strange issue. My setup is FreeBSD 7.2 amd64 and the newest > Cyrus Imapd (.15).: > In the maillog from sendmail I see that the mail will transfered to > Cyrus but it never arrived in the addressed mailbox (of course I tried > different addresses...). I find out that, if I re-send a specific mail > the mail did not arrived but if I write the mail new (same text and > subject) it arrived. The only difference I find, was the message id > field in the header. Outlook is broken. If you re-send a mail it is a new mail but Outlook insists on using the same message id again. You can find an KB entry at Microsoft which describes the bug. The "solution" Microsoft provided was to force Exchange to always generate a new message id :) Regards, Simon > > (sorry for wrapping the lines) > > The bad one: > msgid= > > The good one: > msgid=<000401ca56ea$6babfd10$4303f730$@ch> > > Any ideas are welcome. > > Kind regards, > -- > Martin Schweizer > schweizer.martin at gmail.com > Tel.: +41 32 512 48 54 (VoIP) > Fax: +1 619 3300587 > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > From maria at shadlen.org Wed Oct 28 03:47:42 2009 From: maria at shadlen.org (Maria McKinley) Date: Wed, 28 Oct 2009 00:47:42 -0700 Subject: authentication and/or sieve problem? Message-ID: <4AE7F71E.3050909@shadlen.org> Greetings, I am running cyrus/tls/ldap. The imaps connection is not working, but the imap and smtp connections are: ella:/var/log# testsaslauthd -u "test" -p "xxx" -s smtp 0: OK "Success." ella:/var/log# testsaslauthd -u "test" -p "xxx" -s imaps 0: NO "authentication failed" ella:/var/log# testsaslauthd -u "test" -p "xxx" -s imap 0: OK "Success." I can't figure out why this would be. TLS seems to work just fine for smtp: Oct 28 00:13:21 ella postfix/smtpd[5794]: initializing the server-side TLS engine Oct 28 00:13:21 ella postfix/smtpd[5794]: connect from c-76-28-239-89.hsd1.wa.comcast.net[76.28.239.89] Oct 28 00:13:21 ella postfix/smtpd[5794]: setting up TLS connection from c-76-28-239-89.hsd1.wa.comcast.net[76.28.239.89] ... But I get tls errors regarding imaps: Oct 26 06:36:35 ella cyrus/imaps[18356]: Fatal error: tls_start_servertls() failed I'm not entirely sure how big of a deal this is, since we use ssl over imaps to check mail, but it does seem to be causing a problem with filters/sieve. When someone attempts to change filters using squirrelmail, the connection times out, and the logs fill with imaps tls errors. Oct 28 00:37:45 ella cyrus/sieve[7080]: starttls: TLSv1 with cipher AES256-SHA (256/256 bits new) no authentication Oct 28 00:37:48 ella cyrus/imaps[7082]: imaps TLS negotiation failed: [10.208.108.93] Oct 28 00:37:48 ella cyrus/imaps[7082]: Fatal error: tls_start_servertls() failed But that could be unrelated, since only that first line is the sieve. Anybody have an idea what could be going on? In the meantime, I continue to google and check config files... thanks, maria From D.J.Mayo at bath.ac.uk Wed Oct 28 09:43:31 2009 From: D.J.Mayo at bath.ac.uk (David Mayo) Date: Wed, 28 Oct 2009 13:43:31 +0000 Subject: XFER when changing database formats Message-ID: <4AE84A83.8040301@bath.ac.uk> Earlier this year we introduced a Cyrus 2.2 IMAP proxy server in front of our Cyrus 2.2 back-end server. This was the first part of the process to move our users' mailboxes to a Cyrus 2.3 back-end server. I am now looking at transferring our users to the new back-end server. The XFER command seems to do the job perfectly, except for the seen and subscription databases because we are changing formats from flat to skplist. Looking at the telemetry, it appears the XFER command just dumps everything raw and the other end slurps it in without interpretation. We get lots of errors in the logs[1] and unfortunately the sync client doesn't take too kindly to this either, and bails out! Our users won't accept losing their seen state. I can convert the seen state before/after the XFER with a bit of work, however does anyone knew of a cleaner solution that saves me having to run two different processes? We're running Solaris 10. Our old back-end server is 2.2.12. The new back-end server is running 2.3.13. Our front-end server is 2.2.13. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK [1] Oct 28 13:17:05 sauber.bath.ac.uk imap[7925]: [ID 639319 mail.error] skiplist: invalid magic header: /opt/etc/imapd/user/b/bucsuser.seen.7925 Oct 28 13:17:05 sauber.bath.ac.uk imap[7925]: [ID 558109 mail.error] skiplist: closed while still locked Oct 28 13:17:06 sauber.bath.ac.uk sync_client[7931]: [ID 284028 mail.error] USER received NO response: IMAP_MAILBOX_NONEXISTENT Failed to access inbox for bucsuser: Mailbox does not exist Oct 28 13:17:06 sauber.bath.ac.uk sync_client[7931]: [ID 639319 mail.error] skiplist: invalid magic header: /opt/etc/imapd/user/b/bucsuser.sub Oct 28 13:17:06 sauber.bath.ac.uk sync_client[7931]: [ID 558109 mail.error] skiplist: closed while still locked Oct 28 13:17:06 sauber.bath.ac.uk sync_client[7931]: [ID 404664 mail.error] IOERROR: System I/O error Oct 28 13:17:08 sauber.bath.ac.uk sync_client[7931]: [ID 639319 mail.error] skiplist: invalid magic header: /opt/etc/imapd/user/b/bucsuser.sub Oct 28 13:17:08 sauber.bath.ac.uk sync_client[7931]: [ID 558109 mail.error] skiplist: closed while still locked Oct 28 13:17:08 sauber.bath.ac.uk sync_client[7931]: [ID 404664 mail.error] IOERROR: System I/O error Oct 28 13:17:12 sauber.bath.ac.uk sync_client[7931]: [ID 639319 mail.error] skiplist: invalid magic header: /opt/etc/imapd/user/b/bucsuser.sub Oct 28 13:17:12 sauber.bath.ac.uk sync_client[7931]: [ID 558109 mail.error] skiplist: closed while still locked Oct 28 13:17:12 sauber.bath.ac.uk sync_client[7931]: [ID 404664 mail.error] IOERROR: System I/O error Oct 28 13:17:12 sauber.bath.ac.uk sync_client[7931]: [ID 912450 mail.error] Error in do_sync(): bailing out! From dwhite at olp.net Wed Oct 28 10:54:58 2009 From: dwhite at olp.net (Dan White) Date: Wed, 28 Oct 2009 09:54:58 -0500 Subject: authentication and/or sieve problem? In-Reply-To: <4AE7F71E.3050909@shadlen.org> References: <4AE7F71E.3050909@shadlen.org> Message-ID: <20091028145458.GC4901@dan.olp.net> On 28/10/09?00:47?-0700, Maria McKinley wrote: >ella:/var/log# testsaslauthd -u "test" -p "xxx" -s smtp >0: OK "Success." >ella:/var/log# testsaslauthd -u "test" -p "xxx" -s imaps >0: NO "authentication failed" >ella:/var/log# testsaslauthd -u "test" -p "xxx" -s imap >0: OK "Success." Can you provide sanitized copies of the following?: Your saslauthd startup options (e.g. /etc/default/saslauthd) Your saslauthd.conf if it exists your PAM configuration for smtp, imaps and imap if appropriate >TLS seems to work just fine for smtp: > >Oct 28 00:13:21 ella postfix/smtpd[5794]: initializing the server-side >TLS engine >Oct 28 00:13:21 ella postfix/smtpd[5794]: connect from >c-76-28-239-89.hsd1.wa.comcast.net[76.28.239.89] >Oct 28 00:13:21 ella postfix/smtpd[5794]: setting up TLS connection from >c-76-28-239-89.hsd1.wa.comcast.net[76.28.239.89] >... > >But I get tls errors regarding imaps: > >Oct 26 06:36:35 ella cyrus/imaps[18356]: Fatal error: >tls_start_servertls() failed Permissions problem? Can your cyrus user read the TLS files you've specified in imapd.conf? If not, please include sanitised copies of your imapd.conf and cyrus.conf. >I'm not entirely sure how big of a deal this is, since we use ssl over >imaps to check mail, but it does seem to be causing a problem with >filters/sieve. When someone attempts to change filters using >squirrelmail, the connection times out, and the logs fill with imaps tls >errors. > >Oct 28 00:37:45 ella cyrus/sieve[7080]: starttls: TLSv1 with cipher >AES256-SHA (256/256 bits new) no authentication >Oct 28 00:37:48 ella cyrus/imaps[7082]: imaps TLS negotiation failed: >[10.208.108.93] >Oct 28 00:37:48 ella cyrus/imaps[7082]: Fatal error: >tls_start_servertls() failed What does your sieve entry look like in cyrus.conf? What's your squirrelmail sieve (avelsieve?) configuration look like? -- Dan White From woods-cyrus at weird.com Wed Oct 28 17:55:18 2009 From: woods-cyrus at weird.com (Greg A. Woods) Date: Wed, 28 Oct 2009 17:55:18 -0400 Subject: sieve question: archive script In-Reply-To: <4AE795A1.4010100@realss.com> References: <4AE6B164.1070203@realss.com> <20091027113240.GD9747@brong.net> <4AE795A1.4010100@realss.com> Message-ID: At Wed, 28 Oct 2009 08:51:45 +0800, Zhang Weiwu wrote: Subject: Re: sieve question: archive script > > Bron Gondwana wrote: > > Sieve only runs during the LMTP phase. It doesn't move messages. > > > > I'd say go Perl + IMAP. It's pretty easy. > > > Thanks. I guess I should start learning perl anyway. Try Python instead -- it's _MUCH_ more productive! :-) -- Greg A. Woods +1 416 218-0098 VE3TCP RoboHack Planix, Inc. Secrets of the Weird From wes at umich.edu Wed Oct 28 21:13:08 2009 From: wes at umich.edu (Wesley Craig) Date: Wed, 28 Oct 2009 21:13:08 -0400 Subject: XFER when changing database formats In-Reply-To: <4AE84A83.8040301@bath.ac.uk> References: <4AE84A83.8040301@bath.ac.uk> Message-ID: <2D9666E0-FE19-434E-9F55-83AE06474D75@umich.edu> On 28 Oct 2009, at 09:43, David Mayo wrote: > I am now looking at transferring our users to the new back-end server. > The XFER command seems to do the job perfectly, except for the seen > and > subscription databases because we are changing formats from flat to > skplist. > Looking at the telemetry, it appears the XFER command just dumps > everything raw and the other end slurps it in without > interpretation. We > get lots of errors in the logs[1] and unfortunately the sync client > doesn't take too kindly to this either, and bails out! That's correct. > Our users won't accept losing their seen state. I can convert the seen > state before/after the XFER with a bit of work, however does anyone > knew > of a cleaner solution that saves me having to run two different > processes? I'm not aware of an off-the-shelf solution. It might be possible for undump_mailbox() to examine the xfer'd data and convert it before instantiating it. flat doesn't have a magic number, but skiplist does. Presumably BDB does too, tho Cyrus doesn't know about it directly. A general solution would probably be hard and/or ugly. If it were me, I'd probably keep the file formats the same until XFER is complete and convert after the fact. :wes From john at sml.citizen.co.jp Wed Oct 28 21:52:37 2009 From: john at sml.citizen.co.jp (John/SML) Date: Thu, 29 Oct 2009 09:52:37 +0800 Subject: cyradm > lm returns empty list but mailboxes are accessible via Message-ID: Hi, I have similar problem as been posted :- http://lists.andrew.cmu.edu/pipermail/info-cyrus/2006-July/022899.html I am using Cyrus IMAP 2.2 packages from Ubuntu 6.06.2 LTS, and I used GSSAPI / Kerberos as authentication :- >kinit john at GRT.CITIZEN.CO.JP >cyradm --user john imapsv01.grt.citizen.co.jp imap At the beginning, console command lm showed a list of mailboxes on the IMAP server. After adding over 90 mailboxes, suddenly lm showed empty list, but the mailboxes are accessible from MTA (Postfix) and from mail client (Thunderbird) via GSSAPI / Kerberos :- Is there anyone knows what went wrong ? Please help. Thanks a lot. John Mok ========================= Sunciti Manufacturers Ltd. Direct: +852 27976403 Mobile (HK): +852 81011325 Mobile (CN): +86 15017608946 Facsmile: +852 22601701 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20091029/98237144/attachment.html From maria at shadlen.org Thu Oct 29 03:37:49 2009 From: maria at shadlen.org (Maria McKinley) Date: Thu, 29 Oct 2009 00:37:49 -0700 Subject: authentication and/or sieve problem? In-Reply-To: <20091028145458.GC4901@dan.olp.net> References: <4AE7F71E.3050909@shadlen.org> <20091028145458.GC4901@dan.olp.net> Message-ID: <4AE9464D.1010608@shadlen.org> Dan White wrote: > On 28/10/09 00:47 -0700, Maria McKinley wrote: >> ella:/var/log# testsaslauthd -u "test" -p "xxx" -s smtp >> 0: OK "Success." >> ella:/var/log# testsaslauthd -u "test" -p "xxx" -s imaps >> 0: NO "authentication failed" >> ella:/var/log# testsaslauthd -u "test" -p "xxx" -s imap >> 0: OK "Success." > > Can you provide sanitized copies of the following?: > > Your saslauthd startup options (e.g. /etc/default/saslauthd) So, while copying file contents, I stumbled upon a missing file. I had looked in pam.d a bunch of times, but until you asked me for the actual contents, I had somehow overlooked that there was no imaps file. I copied the smtp file to the imaps file, and now testsaslauthd works for all three services. Yay! But, then I started going back through the logs, and realized that tls with imaps had been working all along. It was just one particular ip that was failing, it was just failing a lot, and there were very few scattered here and there that were successful. So, not sure why that was working (should I worry that it was?), but I am now pretty sure my problems with sieve are not related to authenticating. > > What does your sieve entry look like in cyrus.conf? What's your > squirrelmail sieve (avelsieve?) configuration look like? > from cyrus.conf: sieve cmd="timsieved" listen="sieve" prefork=0 maxchild=100 avelsieve-config.php. got rid of some comments in the interest of space Hadn't noticed this debugging option before. Would it send logs to syslog? /** * Debug Mode. Enable this (change to 1) if you need to send a bug report. */ define('AVELSIEVE_DEBUG', 0); /* =================== IMAP Server / SIEVE Setup ========================= */ /* Backend to use */ global $avelsieve_backend; $avelsieve_backend = 'ManageSieve'; /* =================== ManageSieve Backend Options ======================== */ /* Port where timsieved listens on the Cyrus IMAP server. Default is 2000. */ global $sieveport; $sieveport = 2000; /** * @var string Space separated list of preferred SASL mechanisms for the * authentication to timsieved. e.g. "PLAIN DIGEST-MD5";*/ global $sieve_preferred_sasl_mech; $sieve_preferred_sasl_mech = 'PLAIN'; /* ====== Implementation- and Server-Specific Options ==================== */ global $avelsieve_oldcyrus; $avelsieve_oldcyrus = true; global $avelsieve_enable_envelope_auth; $avelsieve_enable_envelope_auth = true; global $avelsieve_custom_sieve_implementation; $avelsieve_custom_sieve_implementation = ''; global $avelsieve_hardcoded_capabilities; $avelsieve_hardcoded_capabilities = array( 'envelope', 'fileinto', 'copy', 'vacation', 'comparator-i;ascii-numeric' ); global $avelsieve_imapproxymode, $avelsieve_imapproxyserv; $avelsieve_imapproxymode = false; $avelsieve_imapproxyserv = array( 'localhost' => 'imap.example.org' ); /** @var boolean Ldapuserdata mode: Gets user's email addresses (including * mailAlternate & mailAuthorized) from LDAP Prefs Backend plugin's cache */ global $avelsieve_ldapuserdatamode; $avelsieve_ldapuserdatamode = false; /** @var array Map of cyrus administrator users, for proxy authentication */ global $avelsieve_cyrusadmins_map; $avelsieve_cyrusadmins_map = array( 'cyrusimap' => 'cyrussieve' ); /* =============== Avelsieve Interface / Behavior Setup ================== */ /* Be conservative to our updates on the SIEVE server? If true, a button * entitled "Save Changes" will appear, which will give the user the * functionality to register her changes. 'false' is recommended. */ $conservative = false; /* Use images for the move up / down, delete rule buttons and STOP? */ $useimages = true; /* Translate the messages returned by the "Reject" and "Vacation" actions? The * default behaviour since 0.9 is not to translate them. Change to true if in * an intranet environment or in a same-language environment. */ global $translate_return_msgs; $translate_return_msgs = false; $imagetheme = 'bluecurve_24x24'; //$imagetheme = 'bluecurve_16x16'; /* Number of items to display _initially_, when displaying the header match * rule */ $startitems = 3; /* Maximum number of items to allow in one header match rule. */ $maxitems = 10; /* Headers to display in listbox widget, when adding a new header rule. */ $headers = array( 'From', 'To', 'Cc', 'Bcc', 'Subject', 'Reply-To', 'Sender', 'List-Id', 'MailingList', 'Mailing-List', 'X-ML-Name', 'X-List', 'X-List-Name', 'X-MailingList', 'Resent-From', 'Resent-To', 'X-Mailer', 'X-Mailing-List', /* debian and ubuntu flags */ 'X-PTS-Package', 'X-Loop', 'X-Debian-PR-Message', 'X-Debian-PR-Package', 'X-Debian-PR-Keywords', 'X-Debian-PR-Source', 'X-PTS-Keyword', 'X-Debian', 'X-Debian-Package', 'X-Launchpad-Bug', 'X-Launchpad-Bug-Private', 'X-Launchpad-Bug-Security-Vulnerability', 'X-Launchpad-Message-Rationale', 'X-Generated-By', /* debian and ubuntu flags end */ 'X-Spam-Flag', 'X-Spam-Status', 'X-Priority', 'Importance', 'X-MSMail-Priority', 'Precedence', 'Return-Path', 'Received', 'Auto-Submitted' ); /* Available :method's for the :notify extension (if applicable) */ global $notifymethods; $notifymethods = array( 'mailto', 'sms' ); /* use the value "false" if you want to provide a simple input box so that * users can edit the method themselves : */ //$notifymethods = false; // $disable_avelsieve_capabilities = array("notify"); global $disable_avelsieve_capabilities; $disable_avelsieve_capabilities = array(); /* Display Filters link in the top Squirrelmail header? */ global $avelsieveheaderlink; $avelsieveheaderlink = true; /* Default rules table display mode, one of 'verbose' or 'terse' */ global $avelsieve_default_mode; $avelsieve_default_mode = 'terse'; /* ========================= Custom rules Configuration =================== */ $spamrule_enable = false; $spamrule_score_max = 100; $spamrule_score_default = 80; $spamrule_score_header = 'X-Spam-Score'; $spamrule_tests_ldap = false; /* Try to ask Sendmail's LDAP Configuration */ $spamrule_tests = array( 'Open.Relay.DataBase' => "Open Relay Database", 'Spamhaus.Block.List' => "Spamhaus Block List", 'SpamCop' => "SpamCop", 'Composite.Blocking.List' => "Composite Blocking List", 'FORGED' => "Forged Header" ); $spamrule_tests_header = 'X-Spam-Tests'; $spamrule_action_default = 'trash'; /* Please keep the following setting false; it is alpha + needs Squirrelmail * to be patched in three or four places. */ $avelsieve_spam_highlight_enable = false; ?> Here is config.php 'xx.xxx.xxx.xx', 'base' => 'ou=people,dc=myorg,dc=org', 'maxrows' => 50 ); $abook_global_file = ''; $abook_global_file_writeable = false; $abook_global_file_listing = true; $abook_file_line_length = 2048; $addrbook_dsn = ''; $addrbook_table = 'address'; $prefs_dsn = ''; $prefs_table = 'userprefs'; $prefs_user_field = 'user'; $prefs_key_field = 'prefkey'; $prefs_val_field = 'prefval'; $addrbook_global_dsn = ''; $addrbook_global_table = 'global_abook'; $addrbook_global_writeable = false; $addrbook_global_listing = false; $no_list_for_subscribe = false; $smtp_auth_mech = 'none'; $imap_auth_mech = 'login'; $smtp_sitewide_user = ''; $smtp_sitewide_pass = ''; $use_imap_tls = false; $use_smtp_tls = false; $session_name = 'SQMSESSID'; $only_secure_cookies = true; $config_location_base = ''; @include SM_PATH . 'config/config_local.php'; nothing in config_local.php thanks, maria From simon.matter at invoca.ch Thu Oct 29 04:06:02 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Thu, 29 Oct 2009 09:06:02 +0100 Subject: authentication and/or sieve problem? In-Reply-To: <4AE9464D.1010608@shadlen.org> References: <4AE7F71E.3050909@shadlen.org> <20091028145458.GC4901@dan.olp.net> <4AE9464D.1010608@shadlen.org> Message-ID: > Dan White wrote: >> On 28/10/09 00:47 -0700, Maria McKinley wrote: >>> ella:/var/log# testsaslauthd -u "test" -p "xxx" -s smtp >>> 0: OK "Success." >>> ella:/var/log# testsaslauthd -u "test" -p "xxx" -s imaps >>> 0: NO "authentication failed" >>> ella:/var/log# testsaslauthd -u "test" -p "xxx" -s imap >>> 0: OK "Success." >> >> Can you provide sanitized copies of the following?: >> >> Your saslauthd startup options (e.g. /etc/default/saslauthd) > > So, while copying file contents, I stumbled upon a missing file. I had > looked in pam.d a bunch of times, but until you asked me for the actual > contents, I had somehow overlooked that there was no imaps file. I I'm a bit confused because I never had a imaps config for PAM. Doesn't cyrus-imapd always use "imap" as service name for imap + imaps? If that's the case then it's no error that you testsaslauthd call mentioned above didn't succeed. > copied the smtp file to the imaps file, and now testsaslauthd works for > all three services. Yay! But, then I started going back through the > logs, and realized that tls with imaps had been working all along. It Yes, that makes sense now, doesn't it? > was just one particular ip that was failing, it was just failing a lot, > and there were very few scattered here and there that were successful. > So, not sure why that was working (should I worry that it was?), but I > am now pretty sure my problems with sieve are not related to > authenticating. I'm using squirrelmail-1.4.19 with avelsieve-1.9.8 against cyrus-imapd-2.3.15 both on PHP4 and PHP5 but with plaintext. Older versions had some issues when using the wrong combination IIRC. What are your versions used? Regards, Simon > >> >> What does your sieve entry look like in cyrus.conf? What's your >> squirrelmail sieve (avelsieve?) configuration look like? >> > > from cyrus.conf: > > sieve cmd="timsieved" listen="sieve" prefork=0 maxchild=100 > > > avelsieve-config.php. got rid of some comments in the interest of space > > Hadn't noticed this debugging option before. Would it send logs to syslog? > > /** > * Debug Mode. Enable this (change to 1) if you need to send a bug > report. > */ > define('AVELSIEVE_DEBUG', 0); > > /* =================== IMAP Server / SIEVE Setup > ========================= */ > > /* Backend to use */ > global $avelsieve_backend; > $avelsieve_backend = 'ManageSieve'; > > /* =================== ManageSieve Backend Options > ======================== */ > > /* Port where timsieved listens on the Cyrus IMAP server. Default is > 2000. */ > > global $sieveport; > $sieveport = 2000; > > /** > * @var string Space separated list of preferred SASL mechanisms for the > * authentication to timsieved. e.g. "PLAIN DIGEST-MD5";*/ > > global $sieve_preferred_sasl_mech; > $sieve_preferred_sasl_mech = 'PLAIN'; > > > /* ====== Implementation- and Server-Specific Options > ==================== */ > > global $avelsieve_oldcyrus; > $avelsieve_oldcyrus = true; > > global $avelsieve_enable_envelope_auth; > $avelsieve_enable_envelope_auth = true; > > global $avelsieve_custom_sieve_implementation; > $avelsieve_custom_sieve_implementation = ''; > > global $avelsieve_hardcoded_capabilities; > $avelsieve_hardcoded_capabilities = array( > 'envelope', 'fileinto', 'copy', 'vacation', > 'comparator-i;ascii-numeric' > ); > > global $avelsieve_imapproxymode, $avelsieve_imapproxyserv; > $avelsieve_imapproxymode = false; > $avelsieve_imapproxyserv = array( > 'localhost' => 'imap.example.org' > ); > > /** @var boolean Ldapuserdata mode: Gets user's email addresses (including > * mailAlternate & mailAuthorized) from LDAP Prefs Backend plugin's > cache */ > > global $avelsieve_ldapuserdatamode; > $avelsieve_ldapuserdatamode = false; > > /** @var array Map of cyrus administrator users, for proxy authentication > */ > > global $avelsieve_cyrusadmins_map; > $avelsieve_cyrusadmins_map = array( > 'cyrusimap' => 'cyrussieve' > ); > > > /* =============== Avelsieve Interface / Behavior Setup > ================== */ > > /* Be conservative to our updates on the SIEVE server? If true, a button > * entitled "Save Changes" will appear, which will give the user the > * functionality to register her changes. 'false' is recommended. */ > $conservative = false; > > /* Use images for the move up / down, delete rule buttons and STOP? */ > > $useimages = true; > > /* Translate the messages returned by the "Reject" and "Vacation" > actions? The > * default behaviour since 0.9 is not to translate them. Change to true > if in > * an intranet environment or in a same-language environment. */ > > global $translate_return_msgs; > $translate_return_msgs = false; > > $imagetheme = 'bluecurve_24x24'; > //$imagetheme = 'bluecurve_16x16'; > > /* Number of items to display _initially_, when displaying the header > match > * rule */ > > $startitems = 3; > > /* Maximum number of items to allow in one header match rule. */ > > $maxitems = 10; > > /* Headers to display in listbox widget, when adding a new header rule. */ > > $headers = array( > 'From', 'To', 'Cc', 'Bcc', 'Subject', 'Reply-To', 'Sender', 'List-Id', > 'MailingList', 'Mailing-List', 'X-ML-Name', 'X-List', 'X-List-Name', > 'X-MailingList', > 'Resent-From', 'Resent-To', 'X-Mailer', 'X-Mailing-List', > /* debian and ubuntu flags */ > 'X-PTS-Package', 'X-Loop', 'X-Debian-PR-Message', 'X-Debian-PR-Package', > 'X-Debian-PR-Keywords', 'X-Debian-PR-Source', 'X-PTS-Keyword', > 'X-Debian', 'X-Debian-Package', > 'X-Launchpad-Bug', 'X-Launchpad-Bug-Private', > 'X-Launchpad-Bug-Security-Vulnerability', > 'X-Launchpad-Message-Rationale', 'X-Generated-By', > /* debian and ubuntu flags end */ > 'X-Spam-Flag', 'X-Spam-Status', > 'X-Priority', 'Importance', 'X-MSMail-Priority', 'Precedence', > 'Return-Path', 'Received', 'Auto-Submitted' > ); > > /* Available :method's for the :notify extension (if applicable) */ > global $notifymethods; > $notifymethods = array( > 'mailto', 'sms' > ); > /* use the value "false" if you want to provide a simple input box so that > * users can edit the method themselves : */ > //$notifymethods = false; > > // $disable_avelsieve_capabilities = array("notify"); > global $disable_avelsieve_capabilities; > $disable_avelsieve_capabilities = array(); > > /* Display Filters link in the top Squirrelmail header? */ > > global $avelsieveheaderlink; > $avelsieveheaderlink = true; > > /* Default rules table display mode, one of 'verbose' or 'terse' */ > global $avelsieve_default_mode; > $avelsieve_default_mode = 'terse'; > > /* ========================= Custom rules Configuration > =================== */ > > $spamrule_enable = false; > $spamrule_score_max = 100; > $spamrule_score_default = 80; > $spamrule_score_header = 'X-Spam-Score'; > $spamrule_tests_ldap = false; /* Try to ask Sendmail's LDAP Configuration > */ > $spamrule_tests = array( > 'Open.Relay.DataBase' => "Open Relay Database", > 'Spamhaus.Block.List' => "Spamhaus Block List", > 'SpamCop' => "SpamCop", > 'Composite.Blocking.List' => "Composite Blocking List", > 'FORGED' => "Forged Header" > ); > $spamrule_tests_header = 'X-Spam-Tests'; > $spamrule_action_default = 'trash'; > > /* Please keep the following setting false; it is alpha + needs > Squirrelmail > * to be patched in three or four places. */ > > $avelsieve_spam_highlight_enable = false; > ?> > > Here is config.php > > > /** > * SquirrelMail Configuration File > * Created using the configure script, conf.pl > */ > > global $version; > $config_version = '1.4.0'; > $config_use_color = 1; > > $org_name = "SquirrelMail"; > $org_logo = SM_PATH . 'images/sm_logo.png'; > $org_logo_width = '308'; > $org_logo_height = '111'; > $org_title = "SquirrelMail $version"; > $signout_page = ''; > $frame_top = '_top'; > > $provider_uri = 'http://www.squirrelmail.org/'; > > $provider_name = 'SquirrelMail'; > > $motd = ""; > > $squirrelmail_default_language = 'en_US'; > $default_charset = 'iso-8859-1'; > $lossy_encoding = false; > > $domain = 'myorg.org'; > $imapServerAddress = 'localhost'; > $imapPort = 143; > $useSendmail = false; > $smtpServerAddress = 'smtp.myorg.org'; > $smtpPort = 25; > $sendmail_path = '/usr/sbin/sendmail'; > $sendmail_args = '-i -t'; > $pop_before_smtp = false; > $imap_server_type = 'cyrus'; > $invert_time = false; > $optional_delimiter = 'detect'; > $encode_header_key = ''; > > $default_folder_prefix = ''; > $trash_folder = 'INBOX.Trash'; > $sent_folder = 'INBOX.Sent'; > $draft_folder = 'INBOX.Drafts'; > $default_move_to_trash = true; > $default_move_to_sent = true; > $default_save_as_draft = true; > $show_prefix_option = false; > $list_special_folders_first = true; > $use_special_folder_color = true; > $auto_expunge = true; > $default_sub_of_inbox = true; > $show_contain_subfolders_option = false; > $default_unseen_notify = 2; > $default_unseen_type = 1; > $auto_create_special = true; > $delete_folder = false; > $noselect_fix_enable = false; > > $data_dir = '/var/lib/squirrelmail/data/'; > $attachment_dir = '/var/spool/squirrelmail/attach/'; > $dir_hash_level = 0; > $default_left_size = '150'; > $force_username_lowercase = false; > $default_use_priority = true; > $hide_sm_attributions = false; > $default_use_mdn = true; > $edit_identity = true; > $edit_name = true; > $hide_auth_header = false; > $allow_thread_sort = false; > $allow_server_sort = false; > $allow_charset_search = true; > $uid_support = true; > $plugins[0] = 'calendar'; > $plugins[1] = 'delete_move_next'; > $plugins[2] = 'abook_take'; > $plugins[3] = 'message_details'; > $plugins[4] = 'preview_pane'; > $plugins[5] = 'avelsieve'; > $plugins[6] = 'squirrel_logger'; > > $theme_css = ''; > $theme_default = 0; > $theme[0]['PATH'] = SM_PATH . 'themes/'; > > **a bunch more theme stuff that I'm guessing don't matter... > > > $default_use_javascript_addr_book = false; > $ldap_server[0] = array( > 'host' => 'xx.xxx.xxx.xx', > 'base' => 'ou=people,dc=myorg,dc=org', > 'maxrows' => 50 > ); > > $abook_global_file = ''; > $abook_global_file_writeable = false; > $abook_global_file_listing = true; > $abook_file_line_length = 2048; > > $addrbook_dsn = ''; > $addrbook_table = 'address'; > > $prefs_dsn = ''; > $prefs_table = 'userprefs'; > $prefs_user_field = 'user'; > $prefs_key_field = 'prefkey'; > $prefs_val_field = 'prefval'; > $addrbook_global_dsn = ''; > $addrbook_global_table = 'global_abook'; > $addrbook_global_writeable = false; > $addrbook_global_listing = false; > > $no_list_for_subscribe = false; > $smtp_auth_mech = 'none'; > $imap_auth_mech = 'login'; > $smtp_sitewide_user = ''; > $smtp_sitewide_pass = ''; > $use_imap_tls = false; > $use_smtp_tls = false; > $session_name = 'SQMSESSID'; > $only_secure_cookies = true; > > $config_location_base = ''; > > @include SM_PATH . 'config/config_local.php'; > > nothing in config_local.php > > thanks, > maria > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > From maria at shadlen.org Thu Oct 29 04:52:14 2009 From: maria at shadlen.org (Maria McKinley) Date: Thu, 29 Oct 2009 01:52:14 -0700 Subject: authentication and/or sieve problem? In-Reply-To: References: <4AE7F71E.3050909@shadlen.org> <20091028145458.GC4901@dan.olp.net> <4AE9464D.1010608@shadlen.org> Message-ID: <4AE957BE.1000502@shadlen.org> Simon Matter wrote: >> Dan White wrote: >>> On 28/10/09 00:47 -0700, Maria McKinley wrote: >>>> ella:/var/log# testsaslauthd -u "test" -p "xxx" -s smtp >>>> 0: OK "Success." >>>> ella:/var/log# testsaslauthd -u "test" -p "xxx" -s imaps >>>> 0: NO "authentication failed" >>>> ella:/var/log# testsaslauthd -u "test" -p "xxx" -s imap >>>> 0: OK "Success." >>> Can you provide sanitized copies of the following?: >>> >>> Your saslauthd startup options (e.g. /etc/default/saslauthd) >> So, while copying file contents, I stumbled upon a missing file. I had >> looked in pam.d a bunch of times, but until you asked me for the actual >> contents, I had somehow overlooked that there was no imaps file. I > > I'm a bit confused because I never had a imaps config for PAM. Doesn't > cyrus-imapd always use "imap" as service name for imap + imaps? If that's > the case then it's no error that you testsaslauthd call mentioned above > didn't succeed. > >> copied the smtp file to the imaps file, and now testsaslauthd works for >> all three services. Yay! But, then I started going back through the >> logs, and realized that tls with imaps had been working all along. It > > Yes, that makes sense now, doesn't it? > Yup, I understand now. Should have looked at the logs more closely in the first place, instead of jumping on all of the tls failures... >> was just one particular ip that was failing, it was just failing a lot, >> and there were very few scattered here and there that were successful. >> So, not sure why that was working (should I worry that it was?), but I >> am now pretty sure my problems with sieve are not related to >> authenticating. > > I'm using squirrelmail-1.4.19 with avelsieve-1.9.8 against > cyrus-imapd-2.3.15 both on PHP4 and PHP5 but with plaintext. Older > versions had some issues when using the wrong combination IIRC. What are > your versions used? > squirrelmail 2:1.4.15-4+lenny2 avelsieve 1.9.7-6+lenny1 cyrus-imapd- 2.2.13-14+lenny3 looks like php5 I guess one more bit of info that may be helpful is this is a mail server that got moved from an old 32 bit machine, and I used this procedure to migrate: http://www.mail-archive.com/info-cyrus at lists.andrew.cmu.edu/msg38092.html which worked great, at least for the mail part, but maybe sieve stuff didn't transfer so well? cheers, maria > Regards, > Simon > From dwhite at olp.net Thu Oct 29 09:33:17 2009 From: dwhite at olp.net (Dan White) Date: Thu, 29 Oct 2009 08:33:17 -0500 Subject: cyradm > lm returns empty list but mailboxes are accessible via In-Reply-To: References: Message-ID: <20091029133317.GA5019@dan.olp.net> On 29/10/09?09:52?+0800, John/SML wrote: >>kinit john at GRT.CITIZEN.CO.JP >>cyradm --user john imapsv01.grt.citizen.co.jp imap > >At the beginning, console command lm showed a list of mailboxes on the >IMAP server. After adding over 90 mailboxes, suddenly lm showed empty >list, but the mailboxes are accessible from MTA (Postfix) and from mail >client (Thunderbird) via GSSAPI / Kerberos :- Are you trying to connect as an administrator (john) to view all mailboxes? Or are you trying to just view john's mailboxes? Verify in your logs that you are authenticating as the user you are expecting to. If you have virtdomains enabled, see: http://cyrusimap.web.cmu.edu/imapd/install-virtdomains.html particularly the Administration section. I've had similar problems as you when not getting the admin config correct: If virtdomains are enabled and you are connecting as 'cyrus', you might need to add 'cyrus at my-default-domain.org'. Or in your case, john. -- Dan White From dwhite at olp.net Thu Oct 29 09:42:04 2009 From: dwhite at olp.net (Dan White) Date: Thu, 29 Oct 2009 08:42:04 -0500 Subject: cyradm > lm returns empty list but mailboxes are accessible via In-Reply-To: <20091029133317.GA5019@dan.olp.net> References: <20091029133317.GA5019@dan.olp.net> Message-ID: <20091029134204.GB5019@dan.olp.net> On 29/10/09?08:33?-0500, Dan White wrote: >If you have virtdomains enabled, see: > >http://cyrusimap.web.cmu.edu/imapd/install-virtdomains.html > >particularly the Administration section. I've had similar problems as you >when not getting the admin config correct: If virtdomains are enabled and >you are connecting as 'cyrus', you might need to add >'cyrus at my-default-domain.org'. Or in your case, john. I think I have that totally wrong. If you have virtdomains enabled, you must specify a default domain, and then specify an unqualified username in your admin config. -- Dan White From awilliam at whitemice.org Thu Oct 29 10:05:08 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Thu, 29 Oct 2009 10:05:08 -0400 Subject: cyradm > lm returns empty list but mailboxes are accessible via In-Reply-To: References: Message-ID: <1256825108.5291.2.camel@linux-m3mt> On Thu, 2009-10-29 at 09:52 +0800, John/SML wrote: > I have similar problem as been posted :- > http://lists.andrew.cmu.edu/pipermail/info-cyrus/2006-July/022899.html > I am using Cyrus IMAP 2.2 packages from Ubuntu 6.06.2 LTS, and I used > GSSAPI / Kerberos as authentication :- > >kinit john at GRT.CITIZEN.CO.JP > >cyradm --user john imapsv01.grt.citizen.co.jp imap > At the beginning, console command lm showed a list of mailboxes on the > IMAP server. I've seen this behavior before; after performing a bunch of operations cyradm seems to loose the ability to list folders. Exiting cyradmin and starting a new session and everything is fine. I haven't seen this since our latest update - 2.3.14-8 > After adding over 90 mailboxes, suddenly lm showed empty list, but the > mailboxes are accessible from MTA (Postfix) and from mail client Yep. From jmok at attglobal.net Thu Oct 29 11:37:40 2009 From: jmok at attglobal.net (John Mok) Date: Thu, 29 Oct 2009 23:37:40 +0800 Subject: cyradm > lm returns empty list but mailboxes are accessible via In-Reply-To: <20091029133317.GA5019@dan.olp.net> References: <20091029133317.GA5019@dan.olp.net> Message-ID: <4AE9B6C4.8080206@attglobal.net> Hi Dan, Thank you for your prompt reply. > particularly the Administration section. I've had similar problems as you > when not getting the admin config correct: If virtdomains are enabled and > you are connecting as 'cyrus', you might need to add > 'cyrus at my-default-domain.org'. Or in your case, john. My /etc/imapd.conf follows : admins: cyrus john virtdoamins: yes defaultdomain: grt.citizen.co.jp Then, I authenticate with kerberos and login to cyradm console :- >kinit john at GRT.CITIZEN.CO.JP >cyradm --user john imapsv01.grt.citizen.co.jp imap imapsv01.grt.citizen.co.jp>lm << empty list >> imapsv01.grt.citizen.co.jp> I checked the server log and it read that I passed GSSAPI login. The most interesting point is that cyradm seems get crashed after first failure :- >kinit john at GRT.CITIZEN.CO.JP >cyradm --user john imapsv01.grt.citizen.co.jp imap imapsv01.grt.citizen.co.jp>lm user/c* << a list of mailboxes starting with c* >> imapsv01.grt.citizen.co.jp>lam user/chkuk << show ACL of the mailbox chkuk >> imapsv01.grt.citizen.co.jp>lm << empty list >> imapsv01.grt.citizen.co.jp>lm user/c* << empty list >> imapsv01.grt.citizen.co.jp>quit >cyradm --user john imapsv01.grt.citizen.co.jp imap imapsv01.grt.citizen.co.jp>lm user/c* << a list of mailboxes starting with c* >> Is it a bug in the cyrus-admin-2.2 package on Ubuntu 6.06.2 LTS? Thanks a lot. John Mok From morgan at orst.edu Thu Oct 29 13:00:54 2009 From: morgan at orst.edu (Andrew Morgan) Date: Thu, 29 Oct 2009 10:00:54 -0700 (PDT) Subject: authentication and/or sieve problem? In-Reply-To: References: <4AE7F71E.3050909@shadlen.org> <20091028145458.GC4901@dan.olp.net> <4AE9464D.1010608@shadlen.org> Message-ID: On Thu, 29 Oct 2009, Simon Matter wrote: >> Dan White wrote: >>> On 28/10/09 00:47 -0700, Maria McKinley wrote: >>>> ella:/var/log# testsaslauthd -u "test" -p "xxx" -s smtp >>>> 0: OK "Success." >>>> ella:/var/log# testsaslauthd -u "test" -p "xxx" -s imaps >>>> 0: NO "authentication failed" >>>> ella:/var/log# testsaslauthd -u "test" -p "xxx" -s imap >>>> 0: OK "Success." >>> >>> Can you provide sanitized copies of the following?: >>> >>> Your saslauthd startup options (e.g. /etc/default/saslauthd) >> >> So, while copying file contents, I stumbled upon a missing file. I had >> looked in pam.d a bunch of times, but until you asked me for the actual >> contents, I had somehow overlooked that there was no imaps file. I > > I'm a bit confused because I never had a imaps config for PAM. Doesn't > cyrus-imapd always use "imap" as service name for imap + imaps? If that's > the case then it's no error that you testsaslauthd call mentioned above > didn't succeed. I always thought that it uses the service name from cyrus.conf (the first column on a service definition), but now that I look at my own systems I see that I am missing the /etc/pam.d/imaps file as well. Go figure! Andy From gombasg at sztaki.hu Thu Oct 29 15:18:55 2009 From: gombasg at sztaki.hu (Gabor Gombas) Date: Thu, 29 Oct 2009 20:18:55 +0100 Subject: authentication and/or sieve problem? In-Reply-To: References: <4AE7F71E.3050909@shadlen.org> <20091028145458.GC4901@dan.olp.net> <4AE9464D.1010608@shadlen.org> Message-ID: <20091029191855.GB30189@boogie.lpds.sztaki.hu> On Thu, Oct 29, 2009 at 10:00:54AM -0700, Andrew Morgan wrote: > I always thought that it uses the service name from cyrus.conf (the first > column on a service definition), but now that I look at my own systems I > see that I am missing the /etc/pam.d/imaps file as well. Go figure! ... and if you actually check the code, you find: if (sasl_server_new("imap", config_servername, NULL, NULL, NULL, NULL, 0, &imapd_saslconn) != SASL_OK) { So the service name is hard-coded as "imap". Gabor -- --------------------------------------------------------- MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences --------------------------------------------------------- From dwhite at olp.net Thu Oct 29 15:24:08 2009 From: dwhite at olp.net (Dan White) Date: Thu, 29 Oct 2009 14:24:08 -0500 Subject: cyradm > lm returns empty list but mailboxes are accessible via In-Reply-To: <4AE9B6C4.8080206@attglobal.net> References: <20091029133317.GA5019@dan.olp.net> <4AE9B6C4.8080206@attglobal.net> Message-ID: <20091029192408.GL5019@dan.olp.net> > I checked the server log and it read that I passed GSSAPI login. > > The most interesting point is that cyradm seems get crashed after first > failure :- > > Is it a bug in the cyrus-admin-2.2 package on Ubuntu 6.06.2 LTS? That rings a bell too. I don't recall what my resolution was. Does it happen when doing non GSSAPI authentication? -- Dan White From jmok at attglobal.net Fri Oct 30 11:20:57 2009 From: jmok at attglobal.net (John Mok) Date: Fri, 30 Oct 2009 23:20:57 +0800 Subject: cyradm > lm returns empty list but mailboxes are accessible via In-Reply-To: <20091029192408.GL5019@dan.olp.net> References: <20091029133317.GA5019@dan.olp.net> <4AE9B6C4.8080206@attglobal.net> <20091029192408.GL5019@dan.olp.net> Message-ID: <4AEB0459.1030207@attglobal.net> Hi Dan, I checked the /var/log/auth.log, and found the following error when cyradm>lm returned a empty list :- Oct 29 08:36:13 imapsv01 perl: encoded packet size too big (4156 > 4096) Does it remind you how to solve the problem? I googled the problem and the following message advised to patch the Cyrus SASL as follows :- http://www.irbs.net/internet/info-cyrus/0404/0226.html --- src/sasl/plugins/gssapi.c 2004/02/06 17:23:51 1.84 +++ src/sasl/plugins/gssapi.c 2004/04/12 16:36:21 1.85 @@ -1419,7 +1419,7 @@ if(oparams->mech_ssf) { /* xxx probably too large */ - oparams->maxoutbuf -= 50; + oparams->maxoutbuf -= 256; } gss_release_buffer(&min_stat, output_token); Is there anyone knows how to solve the problem? Thanks a lot. John Mok Dan White wrote: >> I checked the server log and it read that I passed GSSAPI login. >> >> The most interesting point is that cyradm seems get crashed after >> first failure :- >> >> Is it a bug in the cyrus-admin-2.2 package on Ubuntu 6.06.2 LTS? > > That rings a bell too. I don't recall what my resolution was. > > Does it happen when doing non GSSAPI authentication? > From schweizer.martin at gmail.com Fri Oct 30 13:20:38 2009 From: schweizer.martin at gmail.com (Martin Schweizer) Date: Fri, 30 Oct 2009 18:20:38 +0100 Subject: sendmail/cyrus deliver not to subdomain Message-ID: <380ccfd60910301020m6f5d4c08g8273924752eb3713@mail.gmail.com> Hello My setup is FreeBSD 7.2 amd64 and the newest Cyrus Imapd (.15). My cyrus server as the defaultdomain set to abc.ch and I set virtdomains: userid . All users works a expected. Now I want to receive additionaly mails for sub.abc.ch. In sendmail set all the necessary entries in access. I also set in mailtable sub.abc.ch cyrusv2:/usr/imap/var/imap/socket/lmtp In maillog I get: Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: n9UGguri011920: --- 050 ... Connecting to /usr/imap/var/imap/socket/lmtp via cyrusv2... Oct 30 17:42:56 acsvfbsd06 sm-mta[11920]: n9UGguri011920: --- 250 2.0.0 n9UGguri011920 Message accepted for delivery Oct 30 17:42:56 acsvfbsd06 sm-mta[11920]: STARTTLS=read, info: fds=8/4, err=2 Oct 30 17:42:56 acsvfbsd06 sm-mta[11920]: n9UGgurj011920: <-- QUIT Oct 30 17:42:56 acsvfbsd06 sm-mta[11920]: n9UGgurj011920: --- 221 2.0.0 acsvfbsd06.abc.ch closing connection Oct 30 17:42:56 acsvfbsd06 sm-mta[11920]: STARTTLS=server, SSL_shutdown not done Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: STARTTLS: CRLFile missing Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: STARTTLS=client, init=1 Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: AUTH=client, relay=localhost, mech=, bits=0 Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: n9UGguri011920: --- 550 5.1.1 ... User unknown (hold) Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: n9UGguri011920: to=, delay=00:00:00, xdelay=00:00:00, mailer=cyrusv2, pri=30344, relay=localhost, dsn=5.1.1, stat=User unknown Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: n9UGguri011920: --- 050 ... aliased to martin at abc.ch Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: n9UGguri011920: alias => martin at abc.ch Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: n9UGguri011920: n9UGgurh011922: DSN: User unknown Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: n9UGgurh011922: --- 050 martin at abc.ch... Using cached LMTP connection to localhost via cyrusv2... Is seems that abc at sub.abc.ch is unknown but if I lm *@sub.abc.ch I get user.abc at sub.abc.ch (\HasNoChildren). What Do I wrong? Or did I missunderstood something? Kind regards, -- Martin Schweizer schweizer.martin at gmail.com Tel.: +41 32 512 48 54 (VoIP) Fax: +1 619 3300587 From anfi at onet.eu Fri Oct 30 13:42:39 2009 From: anfi at onet.eu (Andrzej Adam Filip) Date: Fri, 30 Oct 2009 18:42:39 +0100 Subject: sendmail/cyrus deliver not to subdomain In-Reply-To: <380ccfd60910301020m6f5d4c08g8273924752eb3713@mail.gmail.com> (Martin Schweizer's message of "Fri, 30 Oct 2009 18:20:38 +0100") References: <380ccfd60910301020m6f5d4c08g8273924752eb3713@mail.gmail.com> Message-ID: Martin Schweizer wrote: > Hello > > My setup is FreeBSD 7.2 amd64 and the newest Cyrus Imapd (.15). > > My cyrus server as the defaultdomain set to abc.ch and I set virtdomains: userid > . All users works a expected. Now I want to receive additionaly mails > for sub.abc.ch. In sendmail set all the necessary entries in access. I > also set in mailtable > > sub.abc.ch cyrusv2:/usr/imap/var/imap/socket/lmtp > > In maillog I get: > > Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: n9UGguri011920: --- 050 > ... Connecting to /usr/imap/var/imap/socket/lmtp via > cyrusv2... > Oct 30 17:42:56 acsvfbsd06 sm-mta[11920]: n9UGguri011920: --- 250 > 2.0.0 n9UGguri011920 Message accepted for delivery > Oct 30 17:42:56 acsvfbsd06 sm-mta[11920]: STARTTLS=read, info: fds=8/4, err=2 > Oct 30 17:42:56 acsvfbsd06 sm-mta[11920]: n9UGgurj011920: <-- QUIT > Oct 30 17:42:56 acsvfbsd06 sm-mta[11920]: n9UGgurj011920: --- 221 > 2.0.0 acsvfbsd06.abc.ch closing connection > Oct 30 17:42:56 acsvfbsd06 sm-mta[11920]: STARTTLS=server, SSL_shutdown not done > Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: STARTTLS: CRLFile missing > Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: STARTTLS=client, init=1 > Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: AUTH=client, > relay=localhost, mech=, bits=0 > Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: n9UGguri011920: --- 550 > 5.1.1 ... User unknown (hold) > Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: n9UGguri011920: > to=, delay=00:00:00, xdelay=00:00:00, mailer=cyrusv2, > pri=30344, relay=localhost, dsn=5.1.1, stat=User unknown > Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: n9UGguri011920: --- 050 > ... aliased to martin at abc.ch > Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: n9UGguri011920: alias > => martin at abc.ch > Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: n9UGguri011920: > n9UGgurh011922: DSN: User unknown > Oct 30 17:42:56 acsvfbsd06 sm-mta[11922]: n9UGgurh011922: --- 050 > martin at abc.ch... Using cached LMTP connection to localhost via > cyrusv2... > > Is seems that abc at sub.abc.ch is unknown but if I lm *@sub.abc.ch I > get user.abc at sub.abc.ch (\HasNoChildren). > > What Do I wrong? Or did I missunderstood something? *IF* you use unmodified cyrusv2 mailer provided by sendmail.org *THEN* be warned that it "strips" domain part of the recipient address. You may test it by sending test mail in verbose mode as root - it should show you trace of LMTP session: (echo subject: test; echo)|sendmail -Am -v -- abc at sub.abc.ch BTW you can not set "lmtp socket path" via mailertable in the above mentioned mailer. URL(s): http://anfi.homeunix.org/sendmail/cyrusv2.html http://open-sendmail.sourceforge.net/rtcyrus3/ -- [pl>en: Andrew] Andrzej Adam Filip : anfi at onet.eu Thank goodness modern convenience is a thing of the remote future. -- Pogo, by Walt Kelly From baconm at email.unc.edu Fri Oct 30 15:24:25 2009 From: baconm at email.unc.edu (Michael Bacon) Date: Fri, 30 Oct 2009 15:24:25 -0400 Subject: painful mupdate syncs between front-ends and database server In-Reply-To: <1256003685.9839.1340931677@webmail.messagingengine.com> References: <1482D6C9665A7BA237C6D53C@trophic.its.unc.edu> <1256003685.9839.1340931677@webmail.messagingengine.com> Message-ID: <549E975613EEDF1D9EA407A2@trophic.its.unc.edu> I apologize for not responding sooner here. I've had my head down in the code and doing some tests, including playing with Bron's patch here. I haven't had the guts to roll the patched, CVS version into production as our primary mupdate server, but I did put it in on a test machine in replica mode. My measurement was on a clean server (no pre-existing mailboxes.db), and it didn't appear noticeably faster. I haven't measured hard numbers, but it was still well over 10 minutes to complete the sync and write it out to disk. The odd thing is that we see major performance differences depending on what disk the client is living on. For instance, if we put the mailboxes.db (and the whole metapartition) on superfast Hitachi disks over a 4 GB SAN connection, the sync will finish in just under three minutes. Still, even though we see that big difference, we don't see any kind of I/O contention in the iostat output. the k/sec figures are well within what the drives should be able to handle, and the % blocking stays in low single digits most of the time, while peeking up in the 15-25 range from time to time, but not staying there. It does make me wonder if what we're seeing is related to I/O latency. I haven't delved deep into the skiplist code, but I almost wonder if at least some of the slowness is the foreach iteration on the mupdate master in read mode. On all systems in the murder, we'll see instances where the mupdate process goes into a spin where, in truss, it's an endless repeat of fcntl, stat, fstat, fcntl, thousands of times over. These execute extremely quickly, but I do wonder if we're assuming that something that takes very little time takes an insignificant amount of time, when the time involved becomes significant on an 800k mailboxes database. Finally, as to how we get into this situation in the first place, it appears to happen when the mupdate master, in our environment and configuration, can handle having up to three replicas connected to it before it goes into a bad state during high load. I've never caught it at the point of actually going downhill, but my impression is that so many processes start demanding responses from the mupdate server that the persistent connections that the slave mupdates have to the master timeout and disconnect, then reconnect and try to re-sync. (At least that's what it looks like in the logs.) Incoming IMAP connections won't do it, but lmtpproxy connections seem to have a knack for it, since for whatever reason they appear to generate "kicks" at a pretty high rate. Still looking, but open to suggestions here. Michael Bacon UNC Chapel Hill --On October 20, 2009 12:54:45 PM +1100 Bron Gondwana wrote: > > > On Mon, 19 Oct 2009 16:38 -0400, "Michael Bacon" > wrote: >> When we spec'ed out our servers, we didn't put much I/O capacity into >> the front-end servers -- just a pair of mirrored 10k disks doing the >> OS, the logging, the mailboxes.db, and all the webmail action going on >> in another solaris zone on the same hardware. We thought this was >> sufficient given the fact that no real permanent data lives on these >> servers, but it turns out that while most of thie time it's fine, if >> the mupdate processes ever decide they need to re-sync with the master, >> we've got 6 minutes of trouble >> ahead while it downloads and stores the 800k entries in the mailboxes.db. > > Have you checked if it's actually IO limited? Reading the code, it > appears to do the entire sync in a single transaction, which is bad > because it locks the entire mailboxes.db for the entire time. > >> During these sync periods, we see two negative impacts. The first is >> lockup on the mailboxes.db on the front-end servers, which slows down >> both >> accepting new IMAP/POP connections and the reception of incoming >> messages. >> (The front-ends also accept LMTP connections from a separate pair of >> queueing hosts, then proxy those to the back-ends.) The second is that, >> because the front-ends go into a > > Lost you there - I'm assuming it causes a nasty load spike when it > finishes too. Makes sense. > >> I suppose this is Fastmail and others ripped out the proxyd's and >> replaced >> them with nginx or perdition. Currently we still support GSSAPI as an >> auth >> mechanism, which kept me from going that direction, but given the >> problems >> we're seeing, I'd be open to architectural suggestions on either how to >> tie >> perdition or nginx to the MUPDATE master (because we don't have the >> back-ends split along any discernable lines at this point), or >> suggestions >> on how to make the master-to-frontend propagation faster or less painful. > > We didn't ever go with murder. All our backends are totally independent. > >> Sorry for the long message, but it's not a simple problem we're fighting. > > No - it's not! I wonder if a better approach would be to batch the > mailboxes.db updates into groups of no more than (say) 256. > > Arrgh - stupid, stupid, stupid. Layers of abstraction mean we have a nice > fast "foreach" going on, and then throw away the data and dataptr fields, > followed by which we fetch the data field again. It's very inefficient. > I wonder what percentage of the time is just reading stuff from the > mailboxes.db? > > Anyway - the bit that's actually going to be blocking you will be the > mailboxes.db transactions. I've attached a patch. Advance warning - I > don't use murder, so I haven't done more than compile test it! It SHOULD > be safe though, it just commits to the mailboxes.db every 256 changes and > then closes the transaction, which means that things that were queued > waiting for the lock should get a chance to run before you update the > next 256 records. > > The patch is against current CVS (well, against my git clone of current > CVS anyway) > > Bron. > -- > Bron Gondwana > brong at fastmail.fm > From baconm at email.unc.edu Fri Oct 30 15:38:07 2009 From: baconm at email.unc.edu (Michael Bacon) Date: Fri, 30 Oct 2009 15:38:07 -0400 Subject: painful mupdate syncs between front-ends and database server In-Reply-To: References: <1482D6C9665A7BA237C6D53C@trophic.its.unc.edu> Message-ID: <293FEDFA8403A47B33375FE5@trophic.its.unc.edu> --On October 19, 2009 9:37:57 PM -0400 Wesley Craig wrote: > How are your frontend mupdate processes authenticating to your mupdate > master? And what version of Kerberos are you using (anticipating the > answer to your first question)? I suspect that you're getting a GSSAPI > expired context. We were originally doing GSSAPI, but ran into catastrophic performance with the auth_krb module. I think this relates to Sun's terrible, horrible, no good, very bad implementation of krb5 (it appears to have gone downhill with the arrival of the cryptoadm framework), but at the time I didn't have time to fight with it, so we ditched it in favor of PLAIN with a very long password. > lmtpd on your proxies is talking directly to the mupdate master? It's > possible for lmtpd on your proxies to only check the local mailboxes.db. I'm a little confused as to why this is happening too. Initially, when we configured inbound mail, we were relying on the lmtpproxyd's to return "User unknown" messages, and just telling sendmail to let it all in. Bad idea -- I quickly put a valid user database back into place, because that was absolutely destroying the mupdate server, with all of the kicks it was generating. At this point, we really shouldn't be having any bad users coming into the system, but because of various things (like user tables that aren't 100% up to date), we still get the occasional ones, but I don't think that's what's killing us. At this point, I'm not 100% sure on how to quantify exactly what's making connections to the mupdate master and what's not, so I could be very off on what's going on. Michael Bacon UNC Chapel Hill From baconm at email.unc.edu Fri Oct 30 15:44:59 2009 From: baconm at email.unc.edu (Michael Bacon) Date: Fri, 30 Oct 2009 15:44:59 -0400 Subject: painful mupdate syncs between front-ends and database server In-Reply-To: <8c895a0b0910200313m159ed2e9n164077233f968ad5@mail.gmail.com> References: <1482D6C9665A7BA237C6D53C@trophic.its.unc.edu> <8c895a0b0910200313m159ed2e9n164077233f968ad5@mail.gmail.com> Message-ID: <34A1810BF195586C457DDAC4@trophic.its.unc.edu> --On October 20, 2009 12:13:05 PM +0200 Cyril Servant wrote: > Hello, > > Here we had a similar situation : more than a million mailboxes, and > each MUPDATE sync was veeeeery long (when it succeeded). Now, we > bypass the problem : we get rid of the MUPDATE (and the skiplist > mailboxes.db). We use a home made mysql backend for mailboxes. We > added write and read filters to this backend so front-end and back-end > servers get the right value from mysql. > > With this configuration, we're no more in murder mode, we just use > front-end cyrus (proxys), back-end cyrus, and mysql. We don't need > MUPDATE any more, so we have no sync problems. Cyrus restarts are > fast. Thanks for sharing this. I have wondered a couple of times how much work it would be to rip out the master-to-slave MUPDATE code, and replace it with some kind of OpenLDAP replication that's a little more widely deployed, then just let MUPDATE be exclusively for conversations between the backends and master. It sounds like MySQL would be something similar. I know the FastMail guys have abandoned the murder architecture too, and I can kind of see why -- it seems like the Murder code spends a lot of time and energy avoiding situations which don't come up very often, such as duplicate mailbox names between back-ends. Right now, I think we need the murder's ability to deal with the fact that we have no logical sorting of users across the backends, but I didn't know someone had gone away from the murder mode but was still using Cyrus front-ends (and not perdition or nginx), which we still need for the GSSAPI client support. Michael Bacon UNC Chapel Hill From dwhite at olp.net Fri Oct 30 18:07:36 2009 From: dwhite at olp.net (Dan White) Date: Fri, 30 Oct 2009 17:07:36 -0500 Subject: cyradm > lm returns empty list but mailboxes are accessible via In-Reply-To: <4AEB0459.1030207@attglobal.net> References: <20091029133317.GA5019@dan.olp.net> <4AE9B6C4.8080206@attglobal.net> <20091029192408.GL5019@dan.olp.net> <4AEB0459.1030207@attglobal.net> Message-ID: <20091030220736.GD7245@dan.olp.net> On 30/10/09?23:20?+0800, John Mok wrote: > I checked the /var/log/auth.log, and found the following error when > cyradm>lm returned a empty list :- > > Oct 29 08:36:13 imapsv01 perl: encoded packet size too big (4156 > 4096) > > Does it remind you how to solve the problem? Yes. See: http://markmail.org/message/qvgd6gspvpx2cije I believe this problem went away when I changed or upgraded my Heimdal libraries (and recompiled sasl). I'm currently using Heimdal 1.1 libraries. I believe I had this problem on the older 0.7.x libraries. The problem appears to be that the client is advertising a max out buffer of 4096, but is sending a packet of data larger than that (incorrectly). It's probably due to a problem between cyrus sasl and which ever kerberos library you're using. OpenLDAP client utilities provide a -O to specify security-options (e.g. -O maxbufsize=4096), but I don't know of a way to do that with cyrus clients without recompiling the defaults in sasl. -- Dan White From brong at fastmail.fm Sat Oct 31 02:28:40 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sat, 31 Oct 2009 17:28:40 +1100 Subject: painful mupdate syncs between front-ends and database server In-Reply-To: <549E975613EEDF1D9EA407A2@trophic.its.unc.edu> References: <1482D6C9665A7BA237C6D53C@trophic.its.unc.edu> <1256003685.9839.1340931677@webmail.messagingengine.com> <549E975613EEDF1D9EA407A2@trophic.its.unc.edu> Message-ID: <20091031062840.GA26828@brong.net> On Fri, Oct 30, 2009 at 03:24:25PM -0400, Michael Bacon wrote: > I haven't had the guts to roll the patched, CVS version into > production as our primary mupdate server, but I did put it in on a > test machine in replica mode. My measurement was on a clean server > (no pre-existing mailboxes.db), and it didn't appear noticeably > faster. I haven't measured hard numbers, but it was still well over > 10 minutes to complete the sync and write it out to disk. Sorry - I probably didn't explain what the patch does very well! It doesn't actually make things run any faster - what it does it breaks the one big transaction into lots of small transactions so it doens't block everything else from happening while it runs. > The odd thing is that we see major performance differences depending > on what disk the client is living on. For instance, if we put the > mailboxes.db (and the whole metapartition) on superfast Hitachi > disks over a 4 GB SAN connection, the sync will finish in just under > three minutes. Still, even though we see that big difference, we > don't see any kind of I/O contention in the iostat output. the > k/sec figures are well within what the drives should be able to > handle, and the % blocking stays in low single digits most of the > time, while peeking up in the 15-25 range from time to time, but not > staying there. It does make me wonder if what we're seeing is > related to I/O latency. Hmm, yeah. > I haven't delved deep into the skiplist code, but I almost wonder if > at least some of the slowness is the foreach iteration on the > mupdate master in read mode. On all systems in the murder, we'll > see instances where the mupdate process goes into a spin where, in > truss, it's an endless repeat of fcntl, stat, fstat, fcntl, > thousands of times over. These execute extremely quickly, but I do > wonder if we're assuming that something that takes very little time > takes an insignificant amount of time, when the time involved > becomes significant on an 800k mailboxes database. Almost definitely. I didn't even look at that end of the operation, but I suspect this could be made a lot more efficient with transactional batching as well. Either read all 800k database into a linked list in memory, or do something even trickier. The even trickier bit will be pretty nasty though. Here's what I really want to add to the cyrus db layer: /* pseudocode */ db->next_record(char *key, int keylen, db_txn *txn); Which gets the next record AFTER the (possibly non-existant) record pointed to by key. This is what foreach uses internally - but by having it directly accessible you could implement a partial, restartable foreach. > Finally, as to how we get into this situation in the first place, it > appears to happen when the mupdate master, in our environment and > configuration, can handle having up to three replicas connected to > it before it goes into a bad state during high load. I've never > caught it at the point of actually going downhill, but my impression > is that so many processes start demanding responses from the mupdate > server that the persistent connections that the slave mupdates have > to the master timeout and disconnect, then reconnect and try to > re-sync. (At least that's what it looks like in the logs.) > Incoming IMAP connections won't do it, but lmtpproxy connections > seem to have a knack for it, since for whatever reason they appear > to generate "kicks" at a pretty high rate. > > Still looking, but open to suggestions here. I'll have a look at speeding up the mupdate reads. Bron. From dpc22 at cam.ac.uk Sat Oct 31 06:02:13 2009 From: dpc22 at cam.ac.uk (David Carter) Date: Sat, 31 Oct 2009 10:02:13 +0000 (GMT) Subject: painful mupdate syncs between front-ends and database server In-Reply-To: <549E975613EEDF1D9EA407A2@trophic.its.unc.edu> References: <1482D6C9665A7BA237C6D53C@trophic.its.unc.edu> <1256003685.9839.1340931677@webmail.messagingengine.com> <549E975613EEDF1D9EA407A2@trophic.its.unc.edu> Message-ID: On Fri, 30 Oct 2009, Michael Bacon wrote: > On all systems in the murder, we'll see instances where the mupdate > process goes into a spin where, in truss, it's an endless repeat of > fcntl, stat, fstat, fcntl, thousands of times over. These execute > extremely quickly, but I do wonder if we're assuming that something that > takes very little time takes an insignificant amount of time, when the > time involved becomes significant on an 800k mailboxes database. I agree that latency is probably your problem here. I'm wondering if fsync() latency on the frontends might be a factor given that you report little disk I/O on the mupdate master (IOPS are much more important than Kps, but I'm sure that you already know that). The update process will only be as fast as its weakest link, and you stated earlier: > When we spec'ed out our servers, we didn't put much I/O capacity into > the front-end servers -- just a pair of mirrored 10k disks doing the OS, > the logging, the mailboxes.db, and all the webmail action going on in > another solaris zone on the same hardware. No mention of battery backed write cache there, which tends to be fairly critical for anything involving fsync(). There is an easy way to find out: skiplist_unsafe: 0 If enabled, this option forces the skiplist cyrusdb backend to not sync writes to the disk. Enabling this option is NOT RECOMMENDED. You can ignore the scary warning (at least for test purposes) on murder frontends, given that it is just a readonly replica of the mupdate master. I hope that this isn't a complete red herring. It just struck me that it would be a really easy test to make. -- David Carter Email: David.Carter at ucs.cam.ac.uk University Computing Service, Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH.