From atif.ghaffar at gmail.com Mon Mar 2 19:22:06 2009 From: atif.ghaffar at gmail.com (Atif Ghaffar) Date: Tue, 3 Mar 2009 01:22:06 +0100 Subject: Separating master processes for different services to utilise multiple cores/cpus In-Reply-To: References: Message-ID: Hello all, Sorry to reply to myself, Has no-one else really experienced this? Sorry again for the re-post and best regads -- Atif On Wed, Feb 25, 2009 at 11:47 PM, Atif Ghaffar wrote: > Hello, > > I am a long time cyrus user but have not been hands on for a while. > At my current job we are running cyrus to manage all mailstore for our ISP. > > Recently I have migrate a couple of boxes to newer ones with more cores per > cpu. > > when running htop -u cyrus, i see that only one core is being used for > most of the work and that seems to be a waste of the rest of the cpus. > > We are using the following services (imap, pop3, lmtp, sieve) > > One thought that comes to mind is to run different master processes which > do one service at a time. > > I am thinking on consolidating into > > 1. cyrus-master-main (running imap/sieve) > 2. cyrus-master-pop3 (running pop3) > 3. cyrus-master-lmtp (running lmtp) > > Does this makes any sense? > Would this balance the usage of the CPUs. > > If yes, I will post the config files for inspection. > > > best regards > Atif > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090303/2553815e/attachment.html From support at harmsconsulting.com Mon Mar 2 19:22:22 2009 From: support at harmsconsulting.com (Harms Consulting IT support desk) Date: Tue, 03 Mar 2009 11:22:22 +1100 Subject: log reporting Message-ID: <49AC783E.5010408@harmsconsulting.com> Hello, I'm looking for a script or program to process cyrus log entries in /var/log/mail.log and produce a report. There is a reference to a logwatch script at http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/Logwatch but the attachment no longer seems to exist. I've also search this mail list for "log report". I guess for starters I'd like to know who is checking mail, but it would also be nice to know the popularity of shared folders. Is anyone on this list getting log reports and how are you doing it? Thanks, Josh -- IT Support Desk Harms Consulting Pty Ltd Suite 302, 5 Hunter Street, SYDNEY NSW 2000 Phone: +61 2 9221 2000, Fax: +61 2 9221 1600 From doconnor at gsoft.com.au Mon Mar 2 19:43:39 2009 From: doconnor at gsoft.com.au (Daniel O'Connor) Date: Tue, 3 Mar 2009 11:13:39 +1030 Subject: Separating master processes for different services to utilise multiple cores/cpus In-Reply-To: References: Message-ID: <14798_1236041039_n230huGD007108_200903031113.47104.doconnor@gsoft.com.au> On Tuesday 03 March 2009 10:52:06 Atif Ghaffar wrote: > Has no-one else really experienced this? Sounds like a problem with the scheduler rather than cyrus.. Since each connection is a separate process I imagine all the CPUs would get used. (Not that I have a dual core cyrus box) -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090303/5782c492/attachment.bin From rjaffey at artic.edu Mon Mar 2 20:13:25 2009 From: rjaffey at artic.edu (Raphael Jaffey) Date: Mon, 2 Mar 2009 19:13:25 -0600 Subject: Separating master processes for different services to utilise multiple cores/cpus In-Reply-To: References: Message-ID: <20090302191325.myvzatstusgw8kso@webmail-test.artic.edu> I've never seen this problem. I've been running cyrus for eight years now on dual P3 systems, dual Xeon (hyperthreaded), and now dual quad core systems. The load has always been distributed across all processors. Are you sure you're running an SMP kernel? Although I'd think htop would only report on one core if you weren't. Quoting Atif Ghaffar : > Hello all, > > Sorry to reply to myself, > > Has no-one else really experienced this? > > > Sorry again for the re-post and best regads > -- > Atif > > > On Wed, Feb 25, 2009 at 11:47 PM, Atif Ghaffar wrote: > >> Hello, >> >> I am a long time cyrus user but have not been hands on for a while. >> At my current job we are running cyrus to manage all mailstore for our ISP. >> >> Recently I have migrate a couple of boxes to newer ones with more cores per >> cpu. >> >> when running htop -u cyrus, i see that only one core is being used for >> most of the work and that seems to be a waste of the rest of the cpus. >> >> We are using the following services (imap, pop3, lmtp, sieve) >> >> One thought that comes to mind is to run different master processes which >> do one service at a time. >> >> I am thinking on consolidating into >> >> 1. cyrus-master-main (running imap/sieve) >> 2. cyrus-master-pop3 (running pop3) >> 3. cyrus-master-lmtp (running lmtp) >> >> Does this makes any sense? >> Would this balance the usage of the CPUs. >> >> If yes, I will post the config files for inspection. >> >> >> best regards >> Atif >> > From morgan at orst.edu Mon Mar 2 20:40:30 2009 From: morgan at orst.edu (Andrew Morgan) Date: Mon, 2 Mar 2009 17:40:30 -0800 (PST) Subject: Separating master processes for different services to utilise multiple cores/cpus In-Reply-To: References: Message-ID: > On Wed, Feb 25, 2009 at 11:47 PM, Atif Ghaffar wrote: > >> Hello, >> >> I am a long time cyrus user but have not been hands on for a while. >> At my current job we are running cyrus to manage all mailstore for our ISP. >> >> Recently I have migrate a couple of boxes to newer ones with more cores per >> cpu. >> >> when running htop -u cyrus, i see that only one core is being used for >> most of the work and that seems to be a waste of the rest of the cpus. >> >> We are using the following services (imap, pop3, lmtp, sieve) >> >> One thought that comes to mind is to run different master processes which >> do one service at a time. >> >> I am thinking on consolidating into >> >> 1. cyrus-master-main (running imap/sieve) >> 2. cyrus-master-pop3 (running pop3) >> 3. cyrus-master-lmtp (running lmtp) >> >> Does this makes any sense? >> Would this balance the usage of the CPUs. >> >> If yes, I will post the config files for inspection. Cyrus is already a multi-process program. Each new incoming connection spawns a separate imapd, pop3d, timsieved, or lmtpd process. You won't gain any additional efficiency by running multiple master processes. Most of the time, Cyrus is not limited by your cpu performance but it is often limited by your I/O (disk) performance. On some operating systems, all I/O operations are handled by one particular cpu. For example, I/O from a network card may always be processed by cpu #0. It is possible that is happening to you, although I don't know if Linux does that. Are you having a performance problem? Andy From boutilpj at ednet.ns.ca Tue Mar 3 08:11:03 2009 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Tue, 03 Mar 2009 09:11:03 -0400 Subject: Another cache bug! In-Reply-To: <4990376C.7080600@ednet.ns.ca> References: <1228865816.22867.1289197119@webmail.messagingengine.com> <498ECDA4.3050903@ednet.ns.ca> <20090208230629.GA6895@brong.net> <498F807B.3090009@ednet.ns.ca> <20090209015133.GA18309@brong.net> <31865_1234145527_n192C6X5012827_498F90F4.8000100@ednet.ns.ca> <4990376C.7080600@ednet.ns.ca> Message-ID: <49AD2C67.4010908@ednet.ns.ca> Patrick Boutilier wrote: > Patrick Boutilier wrote: >> Bron Gondwana wrote: >>> On Sun, Feb 08, 2009 at 09:01:47PM -0400, Patrick Boutilier wrote: >>>> Feb 8 13:33:59 student2 cyr_expire[29242]: IOERROR: reading cache >>>> record for user.wooddm: got bogus offset 0 for 68/3; try reconstruct >>>> Feb 8 13:33:59 student2 cyr_expire[29242]: failure expiring >>>> user.wooddm: System I/O error >>> OK - so apart from the "you need to reconstruct" bit, this message >>> actually sucks. >>> >>> The attached patch will make things much more useful! (but the core >>> message of "things are messed up in those there index files, just >>> reconstruct the sucker already) is the same. >>> >>> Bron ( going to commit this to CVS too ) >>> >> >> Patch applied and I will post log entries when the errors come back. >> >> Thanks. > > > Feb 9 03:31:21 student2 cyr_expire[4527]: IOERROR: reading cache record > for user.wwood: item 1 has bogus offset 1818891452 of 56252 for 2/8; > mailbox needs a reconstruct > > Feb 9 03:31:21 student2 cyr_expire[4527]: failure expiring user.wwood: > System I/O error > > > > Running reconstruct on the mailbox does fix the problem. I wrote a script to reconstruct any mailbox that showed corruption during cyr_expire. I had to reconstruct about 4000 accounts the first week and another 5000 or so the second week. It has been well over two weeks now since any further corruption. The only thing I can think of is that these mailboxes have been "corrupt" for quite a while (years maybe) but they only started showing problems once we enabled delayed expunge/delete . > >> >> >> ---- >> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ >> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki >> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -------------- next part -------------- A non-text attachment was scrubbed... Name: boutilpj.vcf Type: text/x-vcard Size: 286 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090303/3aacc1ac/attachment.vcf From adam at morrison-ind.com Tue Mar 3 08:54:54 2009 From: adam at morrison-ind.com (Adam Tauno Williams) Date: Tue, 03 Mar 2009 08:54:54 -0500 Subject: Separating master processes for different services to utilise multiple cores/cpus In-Reply-To: <20090302191325.myvzatstusgw8kso@webmail-test.artic.edu> References: <20090302191325.myvzatstusgw8kso@webmail-test.artic.edu> Message-ID: <20090303085454.8cs492in6s4ow8kk@www.mormail.com> > I've never seen this problem. I've been running cyrus for eight years > now on dual P3 systems, dual Xeon (hyperthreaded), and now dual quad > core systems. The load has always been distributed across all > processors. Ditto, I've never seen a hot-processor situation. Processes have always merrily bounced to the available CPU. > Are you sure you're running an SMP kernel? Although I'd > think htop would only report on one core if you weren't. Or your encountering some kind of filesystem bug or the like with is causing soemthing to hang for a spin-lock? But that seems very unlikely (don't use resierfs). > Quoting Atif Ghaffar : >> Has no-one else really experienced this? I have not; we've been running Cyrus on various systems since like 1.5.x From falko.zurell at idmedia.com Tue Mar 3 09:56:11 2009 From: falko.zurell at idmedia.com (Falko Zurell) Date: Tue, 3 Mar 2009 15:56:11 +0100 Subject: Problem with cyrus imapd and virtual domains Message-ID: <281F06C2-66A6-463A-9FFD-30E413FD657C@idmedia.com> Hi, I got a tricky problem with my cyrus imapd + sasldb authentication. This cyrus was configured without support for virtual domains. Everything works fine. Now I have to add another domain and like the cyrus to support virtual domains. So I added the following lines to my /etc/imapd.conf virtdomains: yes defaultdomain: club.mydomain.de unixhierarchysep: yes My whole imapd.conf looks like this: =============================== servername: club.mydomain.de # loginrealms: sode0-101 club.mydomain.de configdirectory: /var/lib/imap partition-default: /var/spool/imap admins: cyrus fzurell fzurell at sode0-101 fzurell at localdomain imap_admins: cyrus fzurell fzurell at sode0-101 sievedir: /var/lib/imap/sieve sendmail: /usr/sbin/sendmail hashimapspool: true #sasl_pwcheck_method: saslauthd sasl_pwcheck_method: auxprop saslauthd sasl_auxprop_plugin: sasldb sasl_mech_list: PLAIN LOGIN allowanonymouslogin: no allowplaintext: yes #tls_cert_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem #tls_key_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem #tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt # Support virtual Domains virtdomains: yes # The default domain for virtual domain support # If the domain of a user can't be taken from its login and it can't # be determined by doing a reverse lookup on the interface IP, this # domain is used. defaultdomain: club.mydomain.de unixhierarchysep: yes ========================= I disabled the loginrealms parameter as well. Then I restarted the imapd and tried to login. I can login with: cyradm --user cyrus localhost But when I try to list the existing mailboxes I got an empty list. There seems to be no user accounts in the server. I also can't create any new mailbox like this: cyradm --user cyrus localhost localhost.localdomain> cm user/user at reader-club.mydomain.de createmailbox: Invalid mailbox name localhost.localdomain> When I disable virtdomains and login in the very same way I can see all the existing mailboxes. I thinks it's a problem with the "view" of the cyrus admin account. But what makes me wonder is that I can't even create new mailboxes in this new view. I got these entries in my sasldb [root at sode0-101 ~]# sasldblistusers2 | grep cyrus cyrus at sode0-101: userPassword cyrus at club.mydomain.de: userPassword any Ideas on this? Falko Zurell ? /i-d media AG Berlin From pmx at comcelhaiti.com Tue Mar 3 13:48:04 2009 From: pmx at comcelhaiti.com (Tanis Patrick) Date: Tue, 3 Mar 2009 13:48:04 -0500 Subject: mailbox directory Message-ID: <4ED51B077EF943898F508CE8CA970427@LANADMIN> Greetings, I am using cyrus on redhat enterprise linux 5.3. Since i have upgraded to the last version, i notice the partition mailbox grow everyday. Any idea. Tanis Patrick -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090303/b9ce2676/attachment.html From awilliam at whitemice.org Tue Mar 3 16:23:15 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Tue, 03 Mar 2009 16:23:15 -0500 Subject: mailbox directory In-Reply-To: <4ED51B077EF943898F508CE8CA970427@LANADMIN> References: <4ED51B077EF943898F508CE8CA970427@LANADMIN> Message-ID: <1236115395.5644.18.camel@linux-m3mt> On Tue, 2009-03-03 at 13:48 -0500, Tanis Patrick wrote: > Greetings, > I am using cyrus on redhat enterprise linux 5.3. Since i have upgraded > to the last version, i notice the partition mailbox grow everyday. By "partition mailbox" do you mean the partition that contains the user's actual mailboxes (vs. the meta-data partition)? If so, isn't it normal for that partition to grow as new mail arrives? Do you have SQUAT enabled? Or delayed expunge? -- OpenGroupware developer: awilliam at whitemice.org From ram at netcore.co.in Wed Mar 4 03:52:09 2009 From: ram at netcore.co.in (ram) Date: Wed, 04 Mar 2009 14:22:09 +0530 Subject: Pop process hangs in the write() In-Reply-To: <49ABF82A.5040200@olp.net> References: <1235993623.4046.41.camel@darkstar.netcore.co.in> <49ABF82A.5040200@olp.net> Message-ID: <1236156729.6857.38.camel@darkstar.netcore.co.in> On Mon, 2009-03-02 at 09:15 -0600, Dan White wrote: > ram wrote: > > I have problem at a clients end exactly same as described in this > > > > http://marc.info/?l=info-cyrus&m=108967188821511&w=2 > > > > A pop process blocks at write() for any mail at random. And they start > > getting pop lock issues > > > > I have checked with the customer , there is no IDS on their network > > Is there any other way I can debug this issue > > > > > > You could enable telemetry logging, which should help to rule out a > problem with the client. > How do I do that ? > - Dan From selsky at columbia.edu Wed Mar 4 09:40:08 2009 From: selsky at columbia.edu (Matt Selsky) Date: Wed, 4 Mar 2009 09:40:08 -0500 Subject: Pop process hangs in the write() In-Reply-To: <1236156729.6857.38.camel@darkstar.netcore.co.in> References: <1235993623.4046.41.camel@darkstar.netcore.co.in> <49ABF82A.5040200@olp.net> <1236156729.6857.38.camel@darkstar.netcore.co.in> Message-ID: On Mar 4, 2009, at 3:52 AM, ram wrote: > On Mon, 2009-03-02 at 09:15 -0600, Dan White wrote: >> >> You could enable telemetry logging, which should help to rule out a >> problem with the client. > > How do I do that ? # mkdir /var/lib/imap/log/$user # chown cyrus /var/lib/imap/log/$user One file will be created in that directory for every process used by that user. -- Matt From dwhite at olp.net Wed Mar 4 09:40:59 2009 From: dwhite at olp.net (Dan White) Date: Wed, 04 Mar 2009 08:40:59 -0600 Subject: Pop process hangs in the write() In-Reply-To: <1236156729.6857.38.camel@darkstar.netcore.co.in> References: <1235993623.4046.41.camel@darkstar.netcore.co.in> <49ABF82A.5040200@olp.net> <1236156729.6857.38.camel@darkstar.netcore.co.in> Message-ID: <49AE92FB.60707@olp.net> ram wrote: > On Mon, 2009-03-02 at 09:15 -0600, Dan White wrote: > >> You could enable telemetry logging, which should help to rule out a >> problem with the client. >> >> > > How do I do that ? > > > See: http://cyrusimap.web.cmu.edu/imapd/overview.html You create a folder with the same name as the user connecting, underneath the cyrus log directory. Make sure you set appropriate permissions on the directory so that cyrus can write to it. Telemetry logging logs each interaction between the client and the server, except for authentication, which isn't logged. Also, if you're receiving a pop lock response, that probably does not get logged in telemetry logging either, so you should also monitor your syslog for connection attempts. You want to verify that the client is sending a 'quit' at the end of each telemetry log. - Dan From awilliam at whitemice.org Wed Mar 4 09:39:34 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Wed, 04 Mar 2009 09:39:34 -0500 Subject: mailbox directory In-Reply-To: <3E6A37CD653048858D8A5E235A6191C6@LANADMIN> References: <4ED51B077EF943898F508CE8CA970427@LANADMIN> <1236115395.5644.18.camel@linux-m3mt> <3E6A37CD653048858D8A5E235A6191C6@LANADMIN> Message-ID: <1236177574.5513.14.camel@linux-m3mt> On Tue, 2009-03-03 at 22:26 -0500, Tanis Patrick wrote: > Thanks for your answer. > I mean the folder that contains the user mailboxes. This folder grows > exponentially as soon as i upgrade to the last version. Can you tell me more > about SQUAT and delayed expunged? See WMOGAG For SQUAT, which is a full-text index of messages to enabled faster searching, see page 122 For delayed expunge, which keeps messages in the store even after the user expunges them, see page 124 Either of these can cause the mailstore to seem to grow unnaturally. And please keep messages on the list; reply to the list, not to the sender. > > On Tue, 2009-03-03 at 13:48 -0500, Tanis Patrick wrote: > >> Greetings, > >> I am using cyrus on redhat enterprise linux 5.3. Since i have upgraded > >> to the last version, i notice the partition mailbox grow everyday. > > By "partition mailbox" do you mean the partition that contains the > > user's actual mailboxes (vs. the meta-data partition)? If so, isn't it > > normal for that partition to grow as new mail arrives? > > Do you have SQUAT enabled? Or delayed expunge? -- OpenGroupware developer: awilliam at whitemice.org From schaarschmidt at zdv.uni-tuebingen.de Thu Mar 5 07:15:49 2009 From: schaarschmidt at zdv.uni-tuebingen.de (Gabriele Schaarschmidt) Date: Thu, 05 Mar 2009 13:15:49 +0100 Subject: rolling replication - sieve scripts Message-ID: <20090305131549.73455jovwwgb72b9@webmail.uni-tuebingen.de> Hello, we're testing Cyrus replication between 32- and 64-bit Systems, it works fine with cyrus-imapd-2.3.13. But "rolling" replication /usr/cyrus/bin/sync_client -r doesn't sync sieve scripts. So I have to do "sync_client -u -f " in an extra cronjob? Is this problem solved in next version cyrus-imapd-2.3.14? Best regards, Gabriele Schaarschmidt ----------------------------------------------------------------------------- Gabriele Schaarschmidt gabriele.schaarschmidt at uni-tuebingen.de Postmasterin Zentrum f?r Datenverarbeitung Universit?t T?bingen http://www.zdv.uni-tuebingen.de W?chterstr.76, 72074 T?bingen Tel. 07071-2970325 Fax 07071-295131 ----------------------------------------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5293 bytes Desc: S/MIME krytographische Unterschrift Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090305/93463bfb/attachment.bin From brong at fastmail.fm Thu Mar 5 18:56:19 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 06 Mar 2009 10:56:19 +1100 Subject: rolling replication - sieve scripts In-Reply-To: <20090305131549.73455jovwwgb72b9@webmail.uni-tuebingen.de> References: <20090305131549.73455jovwwgb72b9@webmail.uni-tuebingen.de> Message-ID: <1236297379.4350.1303902167@webmail.messagingengine.com> On Thu, 05 Mar 2009 13:15 +0100, "Gabriele Schaarschmidt" wrote: > Hello, > > we're testing Cyrus replication between 32- and 64-bit Systems, > it works fine with cyrus-imapd-2.3.13. > > But "rolling" replication > /usr/cyrus/bin/sync_client -r > doesn't sync sieve scripts. > > So I have to do "sync_client -u -f " in an extra > cronjob? > > Is this problem solved in next version cyrus-imapd-2.3.14? There is no 2.3.14 (yet!) How are you creating your sieve scripts? Replication works if you create a named sieve script and have "default" as a symlink pointing to it. It _DOESN'T_ work if you actually upload or insert a file called "default". We found that out the hard way back when we started using replication. Bron. -- Bron Gondwana brong at fastmail.fm From brong at fastmail.fm Thu Mar 5 19:12:11 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 6 Mar 2009 11:12:11 +1100 Subject: rolling replication - sieve scripts In-Reply-To: <1236297379.4350.1303902167@webmail.messagingengine.com> References: <20090305131549.73455jovwwgb72b9@webmail.uni-tuebingen.de> <1236297379.4350.1303902167@webmail.messagingengine.com> Message-ID: <20090306001211.GB27289@brong.net> On Fri, Mar 06, 2009 at 10:56:19AM +1100, Bron Gondwana wrote: > > Is this problem solved in next version cyrus-imapd-2.3.14? > > There is no 2.3.14 (yet!) > > How are you creating your sieve scripts? Replication works if you create a named sieve > script and have "default" as a symlink pointing to it. It _DOESN'T_ work if you actually > upload or insert a file called "default". We found that out the hard way back when we > started using replication. Actually, specifically it's defaultbc that you care about. Here's what mine looks like: [brong at imap3 brong]$ ls -la total 19 drwx------ 2 cyrus mail 144 Jan 28 01:25 . drwx------ 122 cyrus mail 3408 Feb 12 02:46 .. lrwxrwxrwx 1 cyrus mail 11 Dec 28 2007 defaultbc -> websieve.bc -rw------- 1 cyrus mail 6304 Jan 28 01:25 websieve.bc -rw------- 1 cyrus mail 4901 Jan 28 01:25 websieve.script Also - if you aren't using timsieved to update your sieve scripts, you'll need something else to insert the sync log records. Try cyr_synclog. I'm pretty sure that's in 2.3.13. /usr/cyrus/bin/cyr_synclog -v $USER done :) Bron. From schaarschmidt at zdv.uni-tuebingen.de Fri Mar 6 04:47:17 2009 From: schaarschmidt at zdv.uni-tuebingen.de (Gabriele Schaarschmidt) Date: Fri, 06 Mar 2009 10:47:17 +0100 Subject: rolling replication - sieve scripts In-Reply-To: <20090306001211.GB27289@brong.net> References: <20090305131549.73455jovwwgb72b9@webmail.uni-tuebingen.de> <1236297379.4350.1303902167@webmail.messagingengine.com> <20090306001211.GB27289@brong.net> Message-ID: <20090306104717.13845mfs7qcqui3p@webmail.uni-tuebingen.de> Thanks a lot Bron! It was my mistake - I changed sieve-scripts in my testenvironment with vi and sievec manually :-) Now I made some tests with sieveshell and timsieved and it works fine. In our running systems we always use timsieved to update the sieve scripts. Best regards,, Gabriele Schaarschmidt ------------------------------------------------------------------------ Gabriele Schaarschmidt schaarschmidt at zdv.uni-tuebingen.de Postmasterin Zentrum f?r Datenverarbeitung Universit?t T?bingen http://www.zdv.uni-tuebingen.de W?chterstr.76, 72074 T?bingen Tel. 07071-2970325 Fax 07071-295131 ------------------------------------------------------------------------ ----- Nachricht von brong at fastmail.fm --------- Datum: Fri, 6 Mar 2009 11:12:11 +1100 Von: Bron Gondwana Betreff: Re: rolling replication - sieve scripts An: Bron Gondwana Cc: Gabriele Schaarschmidt , Info Cyrus > On Fri, Mar 06, 2009 at 10:56:19AM +1100, Bron Gondwana wrote: >> > Is this problem solved in next version cyrus-imapd-2.3.14? >> >> There is no 2.3.14 (yet!) >> >> How are you creating your sieve scripts? Replication works if you >> create a named sieve >> script and have "default" as a symlink pointing to it. It >> _DOESN'T_ work if you actually >> upload or insert a file called "default". We found that out the >> hard way back when we >> started using replication. > > Actually, specifically it's defaultbc that you care about. Here's > what mine looks like: > > [brong at imap3 brong]$ ls -la > total 19 > drwx------ 2 cyrus mail 144 Jan 28 01:25 . > drwx------ 122 cyrus mail 3408 Feb 12 02:46 .. > lrwxrwxrwx 1 cyrus mail 11 Dec 28 2007 defaultbc -> websieve.bc > -rw------- 1 cyrus mail 6304 Jan 28 01:25 websieve.bc > -rw------- 1 cyrus mail 4901 Jan 28 01:25 websieve.script > > Also - if you aren't using timsieved to update your sieve scripts, > you'll need something else to insert the sync log records. Try > cyr_synclog. I'm pretty sure that's in 2.3.13. > > /usr/cyrus/bin/cyr_synclog -v $USER > > done :) > > Bron. > ----- Ende der Nachricht von brong at fastmail.fm ----- -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5293 bytes Desc: S/MIME krytographische Unterschrift Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090306/8a1ccd93/attachment.bin From ram at netcore.co.in Sat Mar 7 06:34:54 2009 From: ram at netcore.co.in (ram) Date: Sat, 07 Mar 2009 17:04:54 +0530 Subject: Not able to use smart-sieve Message-ID: <1236425694.2850.45.camel@darkstar.netcore.co.in> I have been using smart sieve for managing sieve scripts for a long time Now on my new cyrus 2.3.13 server smart sieve is not able to login at all. I get a login-failed message everytime I am still not sure wether it is a smartsieve issue or a cyrus issue sivtest script logs in fine I googled for smartsieve errors , most of them are regards to pear errors. I have all the pear modules installed already When I try to debug smartsieve , Mar 7 16:01:04 pop4 smartsieve[26136]: getCryptLib: using rc4 Mar 7 16:01:04 pop4 smartsieve[26136]: getCryptLib: using rc4 Mar 7 16:01:04 pop4 smartsieve[26136]: FAILED LOGIN: XXXXXXXXX [59.163.11.66] {example}: authenticate: authentication failure connecting to 127.0.0.1: data received How do I debug this further ?? From ram at netcore.co.in Sat Mar 7 06:53:45 2009 From: ram at netcore.co.in (ram) Date: Sat, 07 Mar 2009 17:23:45 +0530 Subject: Not able to use smart-sieve In-Reply-To: <1236425694.2850.45.camel@darkstar.netcore.co.in> References: <1236425694.2850.45.camel@darkstar.netcore.co.in> Message-ID: <1236426825.2850.48.camel@darkstar.netcore.co.in> On Sat, 2009-03-07 at 17:04 +0530, ram wrote: > I have been using smart sieve for managing sieve scripts for a long time > Now on my new cyrus 2.3.13 server smart sieve is not able to login at > all. I get a login-failed message everytime > > I am still not sure wether it is a smartsieve issue or a cyrus issue > sivtest script logs in fine > > I googled for smartsieve errors , most of them are regards to pear > errors. I have all the pear modules installed already > > When I try to debug smartsieve , > Mar 7 16:01:04 pop4 smartsieve[26136]: getCryptLib: using rc4 > Mar 7 16:01:04 pop4 smartsieve[26136]: getCryptLib: using rc4 > Mar 7 16:01:04 pop4 smartsieve[26136]: FAILED LOGIN: XXXXXXXXX > [59.163.11.66] {example}: authenticate: authentication failure > connecting to 127.0.0.1: data received > > > > How do I debug this further ?? > Oops , sorry for replying to my own mail. This was some certificates issue I disabled the certs in imapd.conf and now it is working fine > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html From awilliam at whitemice.org Mon Mar 9 16:21:22 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Mon, 09 Mar 2009 16:21:22 -0400 Subject: Intermittent mailbox not found In-Reply-To: References: Message-ID: <1236630082.5320.26.camel@linux-m3mt> > I have inherited a cyrus / postfix server, which is generally well-behaved. > However, there is an intermittent error when delivering messages to a certain > mailbox. It is infrequent, and generally the message is delievered on the > user's next attempt in his MUA. The log is something like: What format is your mailbox database in? Do you see any concomitant messages in the log? > Mar 9 14:29:07 captain postfix/lmtp[20644]: D557618536: > to=, > relay=captain.munged-domain.com[/var/run/cyrus/socket/lmtp], delay=0.1, > delays=0.02/0.01/0/0.08, dsn=5.1.1, status=bounced (host > captain.munged-domain.com[/var/run/cyrus/socket/lmtp] said: 550-Mailbox > unknown. > Either there is no mailbox associated with this 550-name or you do not have > authorization to see it. 550 5.1.1 User unknown (in reply to RCPT TO command)) > Mar 9 14:29:07 captain postfix/cleanup[20642]: ED3B418538: > message-id=<20090309142907.ED3B418538 at captain.munged-domain.com> > Does anyone have any idea how I can go about debugging this? Any chance you can increase the debug level of your MTA (postfix?) to see the entire conversation? -- OpenGroupware developer: awilliam at whitemice.org From ffarid at pragmatic-source.com Mon Mar 9 19:25:42 2009 From: ffarid at pragmatic-source.com (Farzad FARID) Date: Tue, 10 Mar 2009 00:25:42 +0100 Subject: Announcing cyrus-user-map, a local recipient map generator for Postfix Message-ID: <49B5A576.2050703@pragmatic-source.com> Hi, Having just deployed a Cyrus Imap server with Postfix, with only virtual users and no Unix or LDAP based imap accounts, I needed a way to generate a local recipient map for Postfix. This map is used by Postfix to identify existing email accounts and correctly reject incoming mail received for unknown recipients (as in http://www.postfix.org/LOCAL_RECIPIENT_README.html). So I wrote a little tool that generates and update this map automatically, with the help of cron. You can download the script and documentation from my website http://www.pragmatic-source.com/en/opensource/source-code/cyrus-user-map-generate-cyrus-user-map-postfix or from GitHub: http://github.com/Farzy/cyrus-user-map Regards -- Farzad FARID / Architecte Open Source - Associ? Pragmatic Source / http://www.pragmatic-source.com Tel : +33 9 53 19 21 90 / Mob : +33 6 03 70 65 46 From robm at fastmail.fm Mon Mar 9 19:59:58 2009 From: robm at fastmail.fm (Rob Mueller) Date: Tue, 10 Mar 2009 10:59:58 +1100 Subject: Intermittent mailbox not found In-Reply-To: References: Message-ID: > I have inherited a cyrus / postfix server, which is generally > well-behaved. > However, there is an intermittent error when delivering messages to a > certain > mailbox. It is infrequent, and generally the message is delievered on the > user's next attempt in his MUA. The log is something like: What version of cyrus and what version of postfix? Can you should the logs for a successful delivery to that account. Rob From brong at fastmail.fm Mon Mar 9 17:52:43 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 10 Mar 2009 08:52:43 +1100 Subject: Intermittent mailbox not found In-Reply-To: References: Message-ID: <20090309215242.GA6858@brong.net> On Mon, Mar 09, 2009 at 07:52:04PM +0000, Antony Gelberg wrote: > Mar 9 14:29:07 captain cyrus/lmtpunix[20645]: verify_user(user.sirius-john) > failed: Mailbox does not exits Where on earth is that error message from? I don't see "not exits" anywhere in the current Cyrus codebase... Bron. From reinaldoc at gmail.com Mon Mar 9 21:56:23 2009 From: reinaldoc at gmail.com (Reinaldo de Carvalho) Date: Mon, 9 Mar 2009 22:56:23 -0300 Subject: Announcing cyrus-user-map, a local recipient map generator for Postfix In-Reply-To: <49B5A576.2050703@pragmatic-source.com> References: <49B5A576.2050703@pragmatic-source.com> Message-ID: <4a5881460903091856s29f1d48av4031bff8bb36085a@mail.gmail.com> On Mon, Mar 9, 2009 at 8:25 PM, Farzad FARID wrote: > Hi, > > Having just deployed a Cyrus Imap server with Postfix, with only virtual > users and no Unix or LDAP based imap accounts, I needed a way to > generate a local recipient map for Postfix. This map is used by Postfix > to identify existing email accounts and correctly reject incoming mail > received for unknown recipients (as in > http://www.postfix.org/LOCAL_RECIPIENT_README.html). > > So I wrote a little tool that generates and update this map > automatically, with the help of cron. You can download the script and > documentation from my website > http://www.pragmatic-source.com/en/opensource/source-code/cyrus-user-map-generate-cyrus-user-map-postfix > or from GitHub: http://github.com/Farzy/cyrus-user-map > > Regards > Its not necessary use ruby (or another language) merely to call shell commands, do it on pure (ba|c|k)sh. -- Reinaldo de Carvalho http://korreio.sf.net http://python-cyrus.sf.net From adam at morrison-ind.com Mon Mar 9 22:09:24 2009 From: adam at morrison-ind.com (Adam Tauno Williams) Date: Mon, 09 Mar 2009 22:09:24 -0400 Subject: Intermittent mailbox not found In-Reply-To: <20090309215242.GA6858@brong.net> References: <20090309215242.GA6858@brong.net> Message-ID: <1236650964.11240.12.camel@linux-m3mt> On Tue, 2009-03-10 at 08:52 +1100, Bron Gondwana wrote: > On Mon, Mar 09, 2009 at 07:52:04PM +0000, Antony Gelberg wrote: > > Mar 9 14:29:07 captain cyrus/lmtpunix[20645]: verify_user(user.sirius-john) > > failed: Mailbox does not exits > Where on earth is that error message from? I don't see "not exits" > anywhere in the current Cyrus codebase... I'm thinking it is coming from Postfix's LMTP delivery agent. I wonder if some other error, like a connect error, occurs and Postfix falls back to mailbox-does-not-exist. -- OpenGroupware developer: awilliam at whitemice.org From brong at fastmail.fm Mon Mar 9 23:00:16 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 10 Mar 2009 14:00:16 +1100 Subject: Intermittent mailbox not found In-Reply-To: <1236650964.11240.12.camel@linux-m3mt> References: <20090309215242.GA6858@brong.net> <1236650964.11240.12.camel@linux-m3mt> Message-ID: <20090310030016.GB15110@brong.net> On Mon, Mar 09, 2009 at 10:09:24PM -0400, Adam Tauno Williams wrote: > On Tue, 2009-03-10 at 08:52 +1100, Bron Gondwana wrote: > > On Mon, Mar 09, 2009 at 07:52:04PM +0000, Antony Gelberg wrote: > > > Mar 9 14:29:07 captain cyrus/lmtpunix[20645]: verify_user(user.sirius-john) > > > failed: Mailbox does not exits > > Where on earth is that error message from? I don't see "not exits" > > anywhere in the current Cyrus codebase... > > I'm thinking it is coming from Postfix's LMTP delivery agent. I wonder > if some other error, like a connect error, occurs and Postfix falls back > to mailbox-does-not-exist. This old thread might help you: http://markmail.org/message/nqh3rrvhkt4ccs2r Bron ( that's just the first one I found on google ) From wes at umich.edu Mon Mar 9 23:03:10 2009 From: wes at umich.edu (Wesley Craig) Date: Mon, 9 Mar 2009 23:03:10 -0400 Subject: Intermittent mailbox not found In-Reply-To: <20090309215242.GA6858@brong.net> References: <20090309215242.GA6858@brong.net> Message-ID: <43465D7B-F9D7-409D-B604-397F4DE9DAE4@umich.edu> On 09 Mar 2009, at 17:52, Bron Gondwana wrote: > On Mon, Mar 09, 2009 at 07:52:04PM +0000, Antony Gelberg wrote: >> Mar 9 14:29:07 captain cyrus/lmtpunix[20645]: verify_user >> (user.sirius-john) >> failed: Mailbox does not exits > > Where on earth is that error message from? I don't see "not exits" > anywhere in the current Cyrus codebase... This part: verify_user(%s) failed: %s exists. I'd guess the other is a typo. verify_user mostly returns IMAP_MAILBOX_NONEXISTENT, masking a variety of other problems without logging. :wes From listreader at jsj.dyndns.org Tue Mar 10 04:59:21 2009 From: listreader at jsj.dyndns.org (Stefan Schmidt) Date: Tue, 10 Mar 2009 09:59:21 +0100 Subject: Announcing cyrus-user-map, a local recipient map generator for Postfix Message-ID: <200903100959.22299.listreader@jsj.dyndns.org> Hi Farzad, On Tuesday 10 March 2009 00:25:42 Farzad FARID wrote: > Having just deployed a Cyrus Imap server with Postfix, with only > virtual users and no Unix or LDAP based imap accounts, I needed a way > to generate a local recipient map for Postfix. This map is used by > Postfix to identify existing email accounts and correctly reject > incoming mail received for unknown recipients (as in > http://www.postfix.org/LOCAL_RECIPIENT_README.html). > > So I wrote a little tool that generates and update this map > automatically, with the help of cron. You can download the script and > documentation from my website > http://www.pragmatic-source.com/en/opensource/source-code/cyrus-user- >map-generate-cyrus-user-map-postfix or from GitHub: > http://github.com/Farzy/cyrus-user-map Nice thing, but just out of curiosity, why don't you use postfix' recipient verification mechanism? In smtp_recipient_restrictions add reject_unverified_recipient at a reasonable position in these restrictions. Everything else automagically happens. Greetings, Stefan -- Stefan Schmidt listreader at jsj.dyndns.org From brong at fastmail.fm Tue Mar 10 05:01:13 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 10 Mar 2009 20:01:13 +1100 Subject: Calling all regressions Message-ID: <1236675674.12318.1304575661@webmail.messagingengine.com> I was looking through the bug tracker, and one of the items was "skiplist issues in 2.2" - well, the ticket has morphed into that anyway. The question is - do we backport the skiplist bugfixes to Cyrus 2.2? For that matter, do we try to isolate just the actual bug fixes and backport them, or do we copy the whole cyrusdb_skiplist.c file (pretty self-contained, and the API is unchaged) back? What about other bugfixes in the 2.3 branch? There have been plenty, and not all related to new 2.3 branch features. So - I ask now. Does anybody know of any regressions in 2.3 that aren't present in the earlier branches? Even performance regressions are interesting to some sites. Is there any good reason for a site to continue running 2.2 or even 2.1? If there is - I want to know it, and I want to fix it, so we don't have to think about maintaining the old Cyrus. There aren't enough of us working on Cyrus to split the effort! Now - also of interest is the cost of having to change configurations when upgrading. 2.3.x is NOT a direct dropin for 2.[12].x because some defaults have changed. That's annoying. I don't know any good solution other than refusing to start or complaining very loudly if those keys are NOT present in the config file. I would make the error spit out: a) the backwards compatible value b) the new default value c) a BRIEF description of the benefits of using the new value. Along with a line to copy and paste into the config file for each of the above. Distro maintainers can of course avoid most of this by having clever upgrade scripts, but we need to make sure each site can find and fix the issues caused by an upgrade quickly. Ok, enough manifesto for me for one evening! Let me know about those regressions, or forever hold your peace as I push for deprecation of the old branches... -- Bron Gondwana brong at fastmail.fm From ffarid at pragmatic-source.com Tue Mar 10 05:58:36 2009 From: ffarid at pragmatic-source.com (Farzad FARID) Date: Tue, 10 Mar 2009 10:58:36 +0100 Subject: Announcing cyrus-user-map, a local recipient map generator for Postfix In-Reply-To: <200903100959.22299.listreader@jsj.dyndns.org> References: <200903100959.22299.listreader@jsj.dyndns.org> Message-ID: <49B639CC.5030603@pragmatic-source.com> Le 10.03.2009 09:59, Stefan Schmidt a ?crit : > Nice thing, but just out of curiosity, why don't you use postfix' > recipient verification mechanism? > In smtp_recipient_restrictions add reject_unverified_recipient at a > reasonable position in these restrictions. Everything else > automagically happens. > Thanks for asking. In the "Postfix Address Verification Howto" (http://www.postfix.org/ADDRESS_VERIFICATION_README.html) it says that: "/A Postfix MTA verifies a sender or recipient address by probing the nearest MTA for that address, without actually delivering mail. The nearest MTA could be the Postfix MTA itself, or it could be a remote MTA (SMTP interruptus). Probe messages are like normal mail, except that they are never delivered, deferred or bounced; probe messages are always discarded./" This is a very heavyweight process, especially if we have the Cyrus database at hand :) So I think that looking up an entry in a hash map is much faster and doesn't involve any SMTP/LMTP connection. And if the Postfix relay and the Cyrus database are not on the same physical computer, and someone's interested, I can extend my little tool to use a network connection to fetch the user database. Best regards -- Farzad FARID / Architecte Open Source - Associ? Pragmatic Source / http://www.pragmatic-source.com Tel : +33 9 53 19 21 90 / Mob : +33 6 03 70 65 46 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090310/d8e363c4/attachment.html From ffarid at pragmatic-source.com Tue Mar 10 06:09:37 2009 From: ffarid at pragmatic-source.com (Farzad FARID) Date: Tue, 10 Mar 2009 11:09:37 +0100 Subject: Announcing cyrus-user-map, a local recipient map generator for Postfix In-Reply-To: <4a5881460903091856s29f1d48av4031bff8bb36085a@mail.gmail.com> References: <49B5A576.2050703@pragmatic-source.com> <4a5881460903091856s29f1d48av4031bff8bb36085a@mail.gmail.com> Message-ID: <49B63C61.4050406@pragmatic-source.com> Hi, Le 10.03.2009 02:56, Reinaldo de Carvalho a ?crit : >> So I wrote a little tool that generates and update this map >> automatically, with the help of cron. You can download the script and >> documentation from my website >> http://www.pragmatic-source.com/en/opensource/source-code/cyrus-user-map-generate-cyrus-user-map-postfix >> or from GitHub: http://github.com/Farzy/cyrus-user-map >> > Its not necessary use ruby (or another language) merely to call shell > commands, do it on pure (ba|c|k)sh. > You're right! But I chose a somewhat cleaner and higher level language to have a more readable script. You may rewrite it in *sh if you're interested and I'll include it in my package or add a link to it. It's also a proactive choice for the reason that Ruby is gaining in popularity, is available on many server platforms (even Sun promotes Ruby) and I'm pushing the use of Ruby for agile sysadmin tasks :) This is off-topic but interested *french speaking* people can view my lightning talk in this subjet: http://tinyurl.com/avuebz Best regards, -- Farzad FARID / Architecte Open Source - Associ? Pragmatic Source / http://www.pragmatic-source.com Tel : +33 9 53 19 21 90 / Mob : +33 6 03 70 65 46 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090310/3769a84f/attachment.html From kleo+cyrus at netbox.cz Tue Mar 10 06:18:43 2009 From: kleo+cyrus at netbox.cz (Vladimir Klejch) Date: Tue, 10 Mar 2009 11:18:43 +0100 (CET) Subject: Announcing cyrus-user-map, a local recipient map generator for Postfix In-Reply-To: <49B639CC.5030603@pragmatic-source.com> References: <200903100959.22299.listreader@jsj.dyndns.org> <49B639CC.5030603@pragmatic-source.com> Message-ID: On Tue, 10 Mar 2009, Farzad FARID wrote: > Le 10.03.2009 09:59, Stefan Schmidt a ?crit : >> Nice thing, but just out of curiosity, why don't you use postfix' >> recipient verification mechanism? >> In smtp_recipient_restrictions add reject_unverified_recipient at a >> reasonable position in these restrictions. Everything else >> automagically happens. >> > Thanks for asking. In the "Postfix Address Verification Howto" > (http://www.postfix.org/ADDRESS_VERIFICATION_README.html) it says that: > > "/A Postfix MTA verifies a sender or recipient address by probing the > nearest MTA for that address, without actually delivering mail. The nearest > MTA could be the Postfix MTA itself, or it could be a remote MTA (SMTP > interruptus). Probe messages are like normal mail, except that they are never > delivered, deferred or bounced; probe messages are always discarded./" > > This is a very heavyweight process, especially if we have the Cyrus database > at hand :) So I think that looking up an entry in a hash map is much faster > and doesn't involve any SMTP/LMTP connection. > > And if the Postfix relay and the Cyrus database are not on the same physical > computer, and someone's interested, I can extend my little tool to use a > network connection to fetch the user database. > > Best regards > > what about something like "Postfix version of smmapd" : http://www.mail-archive.com/info-cyrus at lists.andrew.cmu.edu/msg31479.html ... it is good solution, it has test for existention of mailbox and overquota .. i have fidled with it, but have not working solution, because there is some problem with unixhierarchysep and altnamespace , i think ... :-(( its about 1.5 years ago as tryed it ... -- _____________________________________________________________ | You have moved the mouse. # | Windows must be restarted for the changes to take effect. # | # ##############################################################/ ~~ ~~ ~~ ~~ ~~ ~~ ~~ Vladimir `KLEO' Klejch Kleo'at'netbox.cz ... ... ... ... -------------- next part -------------- ---- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html From Eric.Luyten at vub.ac.be Tue Mar 10 06:35:17 2009 From: Eric.Luyten at vub.ac.be (Eric Luyten) Date: Tue, 10 Mar 2009 11:35:17 +0100 (CET) Subject: Announcing cyrus-user-map, a local recipient map generator for Postfix In-Reply-To: <49B639CC.5030603@pragmatic-source.com> References: <200903100959.22299.listreader@jsj.dyndns.org> <49B639CC.5030603@pragmatic-source.com> Message-ID: <58058.134.184.15.103.1236681317.squirrel@nuts.vub.ac.be> On Tue, March 10, 2009 10:58 am, Farzad FARID wrote: > Le 10.03.2009 09:59, Stefan Schmidt a ?crit : > >> Nice thing, but just out of curiosity, why don't you use postfix' >> recipient verification mechanism? In smtp_recipient_restrictions add >> reject_unverified_recipient at a reasonable position in these >> restrictions. Everything else automagically happens. >> > Thanks for asking. In the "Postfix Address Verification Howto" > (http://www.postfix.org/ADDRESS_VERIFICATION_README.html) it says that: > > > "/A Postfix MTA verifies a sender or recipient address by probing the > nearest MTA for that address, without actually delivering mail. The nearest > MTA could be the Postfix MTA itself, or it could be a remote MTA > (SMTP interruptus). Probe messages are like normal mail, except that > they are never delivered, deferred or bounced; probe messages are always > discarded./" > > This is a very heavyweight process, especially if we have the Cyrus > database at hand :) So I think that looking up an entry in a hash map is > much faster and doesn't involve any SMTP/LMTP connection. Correct, but those probes are pretty lightweight and the results can be cached by Postfix. We service dozens of our internal (!) mail domains using those Postfix probes and performance has never been an issue, even with small mail backends or small relays. Recipient address validation is a very useful Postfix feature. When we started operating IronPort appliances for the main mail domains we were forced to feed to complete local recipient address list to an LDAP server, so, yes, your method certainly has its uses and merits. Regards, Eric Luyten, Computing Centre VUB/ULB. From listreader at jsj.dyndns.org Tue Mar 10 06:50:07 2009 From: listreader at jsj.dyndns.org (Stefan Schmidt) Date: Tue, 10 Mar 2009 11:50:07 +0100 Subject: Announcing cyrus-user-map, a local recipient map generator for Postfix In-Reply-To: <49B639CC.5030603@pragmatic-source.com> References: <200903100959.22299.listreader@jsj.dyndns.org> <49B639CC.5030603@pragmatic-source.com> Message-ID: <200903101150.08237.listreader@jsj.dyndns.org> On Tuesday 10 March 2009 10:58:36 Farzad FARID wrote: > Thanks for asking. In the "Postfix Address Verification Howto" > (http://www.postfix.org/ADDRESS_VERIFICATION_README.html) it says > that: > : verification process > > This is a very heavyweight process, especially if we have the Cyrus > database at hand :) So I think that looking up an entry in a hash map > is much faster and doesn't involve any SMTP/LMTP connection. Postfix cashes to test result for a configurable time spam, with separately definable ranges for positive and negative test results. It is - after an initial learning period - basically a hash table lookup without the hazzles of manually generating the lookup table. > And if the Postfix relay and the Cyrus database are not on the same > physical computer, and someone's interested, I can extend my little > tool to use a network connection to fetch the user database. Still no problem, as postfix has to deliver the mail sto cyrus somehow, either smtp or lmtp. The verification simply works here. Greetings, Stefan -- Stefan Schmidt listreader at jsj.dyndns.org From michael.menge at zdv.uni-tuebingen.de Tue Mar 10 07:07:13 2009 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Tue, 10 Mar 2009 12:07:13 +0100 Subject: Calling all regressions In-Reply-To: <1236675674.12318.1304575661@webmail.messagingengine.com> References: <1236675674.12318.1304575661@webmail.messagingengine.com> Message-ID: <20090310120713.140117sn46lwgg69@webmail.uni-tuebingen.de> Quoting Bron Gondwana : > Is there any good reason for a site to continue running 2.2 or even 2.1? > Some Enterprise Distibutions still use 2.2.x as they maintain the same version for a very long time. E.g. Suse (SLES 10) uses Cyrus 2.2.12. These Distrutions patch the software with own patches and backports, but I don't know if SuSE backported changes from 2.3.x > If there is - I want to know it, and I want to fix it, so we don't > have to think about maintaining the old Cyrus. There aren't enough > of us working on Cyrus to split the effort! > > > Now - also of interest is the cost of having to change > configurations when upgrading. 2.3.x is NOT a direct dropin for > 2.[12].x because some defaults have changed. That's annoying. I > don't know any good solution other than refusing to start or > complaining very loudly if those keys are NOT present in the config > file. I would make the error spit out: > Is there a way to know if cyrus was updated? Regards Michael Menge -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5339 bytes Desc: S/MIME krytographische Unterschrift Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090310/a4ae9638/attachment.bin From ffarid at pragmatic-source.com Tue Mar 10 07:52:28 2009 From: ffarid at pragmatic-source.com (Farzad FARID) Date: Tue, 10 Mar 2009 12:52:28 +0100 Subject: Announcing cyrus-user-map, a local recipient map generator for Postfix In-Reply-To: <200903101150.08237.listreader@jsj.dyndns.org> References: <200903100959.22299.listreader@jsj.dyndns.org> <49B639CC.5030603@pragmatic-source.com> <200903101150.08237.listreader@jsj.dyndns.org> Message-ID: <49B6547C.9070807@pragmatic-source.com> Thanks to you and Eric Luyten for the information on the small overhead. So until someone ports smmapd to Postfix I agree that the recipient address validation setting is sufficient. Regards, Le 10.03.2009 11:50, Stefan Schmidt a ?crit : > > Postfix cashes to test result for a configurable time spam, with > separately definable ranges for positive and negative test results. > It is - after an initial learning period - basically a hash table lookup > without the hazzles of manually generating the lookup table. > -- Farzad FARID / Architecte Open Source - Associ? Pragmatic Source / http://www.pragmatic-source.com Tel : +33 9 53 19 21 90 / Mob : +33 6 03 70 65 46 From adam at morrison-ind.com Tue Mar 10 08:38:21 2009 From: adam at morrison-ind.com (Adam Tauno Williams) Date: Tue, 10 Mar 2009 08:38:21 -0400 Subject: Calling all regressions In-Reply-To: <20090310120713.140117sn46lwgg69@webmail.uni-tuebingen.de> References: <1236675674.12318.1304575661@webmail.messagingengine.com> <20090310120713.140117sn46lwgg69@webmail.uni-tuebingen.de> Message-ID: <1236688701.5616.17.camel@linux-m3mt> On Tue, 2009-03-10 at 12:07 +0100, Michael Menge wrote: > Quoting Bron Gondwana : > > Is there any good reason for a site to continue running 2.2 or even 2.1? > Some Enterprise Distibutions still use 2.2.x as they maintain the > same version for a very long time. E.g. Suse (SLES 10) uses Cyrus 2.2.12. > These Distrutions patch the software with own patches and backports, > but I don't know if SuSE backported changes from 2.3.x > > If there is - I want to know it, and I want to fix it, so we don't > > have to think about maintaining the old Cyrus. There aren't enough > > of us working on Cyrus to split the effort! > > Now - also of interest is the cost of having to change > > configurations when upgrading. 2.3.x is NOT a direct dropin for > > 2.[12].x because some defaults have changed. That's annoying. I > > don't know any good solution other than refusing to start or > > complaining very loudly if those keys are NOT present in the config > > file. I would make the error spit out: > Is there a way to know if cyrus was updated? Documentation is pretty miserable; best source for changes is either or to read the CVS commit maillist [or subscribe to it cyrus-cvs-subscribe at lists.andrew.cmu.edu ] -- OpenGroupware developer: awilliam at whitemice.org OpenGroupware & Cyrus IMAPd documentation: From michael.menge at zdv.uni-tuebingen.de Tue Mar 10 08:51:30 2009 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Tue, 10 Mar 2009 13:51:30 +0100 Subject: Calling all regressions In-Reply-To: <1236688701.5616.17.camel@linux-m3mt> References: <1236675674.12318.1304575661@webmail.messagingengine.com> <20090310120713.140117sn46lwgg69@webmail.uni-tuebingen.de> <1236688701.5616.17.camel@linux-m3mt> Message-ID: <20090310135130.156225x0tn2katzm@webmail.uni-tuebingen.de> Quoting Adam Tauno Williams : > On Tue, 2009-03-10 at 12:07 +0100, Michael Menge wrote: >> Quoting Bron Gondwana : >> > Is there any good reason for a site to continue running 2.2 or even 2.1? >> Some Enterprise Distibutions still use 2.2.x as they maintain the >> same version for a very long time. E.g. Suse (SLES 10) uses Cyrus 2.2.12. >> These Distrutions patch the software with own patches and backports, >> but I don't know if SuSE backported changes from 2.3.x >> > If there is - I want to know it, and I want to fix it, so we don't >> > have to think about maintaining the old Cyrus. There aren't enough >> > of us working on Cyrus to split the effort! >> > Now - also of interest is the cost of having to change >> > configurations when upgrading. 2.3.x is NOT a direct dropin for >> > 2.[12].x because some defaults have changed. That's annoying. I >> > don't know any good solution other than refusing to start or >> > complaining very loudly if those keys are NOT present in the config >> > file. I would make the error spit out: >> Is there a way to know if cyrus was updated? > > Documentation is pretty miserable; best source for changes is either > or to read the CVS > commit maillist [or > subscribe to it cyrus-cvs-subscribe at lists.andrew.cmu.edu ] I think i should rephrase my question. Is there a way for cyrus to know that it was updated since the last time it did run, so it can do the things suggested by Bron. -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5339 bytes Desc: S/MIME krytographische Unterschrift Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090310/c5b59cf4/attachment.bin From wes at umich.edu Tue Mar 10 13:33:00 2009 From: wes at umich.edu (Wesley Craig) Date: Tue, 10 Mar 2009 13:33:00 -0400 Subject: Calling all regressions In-Reply-To: <20090310135130.156225x0tn2katzm@webmail.uni-tuebingen.de> References: <1236675674.12318.1304575661@webmail.messagingengine.com> <20090310120713.140117sn46lwgg69@webmail.uni-tuebingen.de> <1236688701.5616.17.camel@linux-m3mt> <20090310135130.156225x0tn2katzm@webmail.uni-tuebingen.de> Message-ID: <26722814-9B2B-4203-AC1A-0554A9D2C190@umich.edu> On 10 Mar 2009, at 08:51, Michael Menge wrote: > Quoting Adam Tauno Williams : >> Documentation is pretty miserable; best source for changes is either >> or to read the CVS >> commit maillist >> [or >> subscribe to it cyrus-cvs-subscribe at lists.andrew.cmu.edu ] I'd probably say the "documentation" was miserable if all I could find was a web page that hadn't been updated since 2.3.1, too. A more nuanced bash might be that the web site is miserably out of date. In fact, changes.html from the distribution is current for each release. Of course, contributions to improve the miserable documentation are always welcome. > I think i should rephrase my question. > > Is there a way for cyrus to know that it was updated > since the last time it did run, so it can do the things > suggested by Bron. Perhaps we should add a config option with no default whose value must be the version of the software you're running. :wes From dgottsc at emory.edu Tue Mar 10 13:42:30 2009 From: dgottsc at emory.edu (Gottschalk, David) Date: Tue, 10 Mar 2009 13:42:30 -0400 Subject: Vacation Application Message-ID: Hi All, I have a question regarding a vacation application for Cyrus servers. I want to know what others are using to meet this need. Currently, we have a home grown application that allows users to enable/disable vacation messages for their Cyrus accounts. This application is old and outdated, and needs to be replaced. I found the Horde vacation application which looks like it will meet my needs, but I have one issue. We currently have three Cyrus servers with all the accounts spread out across those three servers. The problem I am running into with any vacation application, is how to identify to the vacation application which server a user is on without the user needing to know this information. Is anyone else in a similar situation? Thanks for any help. David Gottschalk UTS Email team david.gottschalk at emory.edu This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments). From nic at onlight.com Tue Mar 10 13:59:55 2009 From: nic at onlight.com (Nic Bernstein) Date: Tue, 10 Mar 2009 12:59:55 -0500 Subject: Vacation Application In-Reply-To: References: Message-ID: <49B6AA9B.4060905@onlight.com> On 03/10/2009 12:42 PM, Gottschalk, David wrote: > Hi All, > I have a question regarding a vacation application for Cyrus servers. I want to know what others are using to meet this need. Currently, we have a home grown application that allows users to enable/disable vacation messages for their Cyrus accounts. This application is old and outdated, and needs to be replaced. I found the Horde vacation application which looks like it will meet my needs, but I have one issue. We currently have three Cyrus servers with all the accounts spread out across those three servers. The problem I am running into with any vacation application, is how to identify to the vacation application which server a user is on without the user needing to know this information. Is anyone else in a similar situation? > Um, use a murder? Otherwise you may want to adapt something like the cyrus user validation script which was recently posted here. Your application could then perform lookups in the output of that script. Lastly, you could just let you application attempt authenticating against each server, in turn, stopping once it succeeds. Cheers, -nic -- Nic Bernstein nic at onlight.com Onlight llc. www.onlight.com 2266 North Prospect Avenue #610 v. 414.272.4477 Milwaukee, Wisconsin 53202-6306 f. 414.290.0335 From awilliam at whitemice.org Tue Mar 10 13:58:24 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Tue, 10 Mar 2009 13:58:24 -0400 Subject: Vacation Application In-Reply-To: References: Message-ID: <1236707904.5613.3.camel@linux-m3mt> On Tue, 2009-03-10 at 13:42 -0400, Gottschalk, David wrote: > Hi All, > I have a question regarding a vacation application for Cyrus servers. I want > to know what others are using to meet this need. Currently, we have a home > grown application that allows users to enable/disable vacation messages for > their Cyrus accounts. This application is old and outdated, and needs to be > replaced. I found the Horde vacation application which looks like it will > meet my needs, but I have one issue. We currently have three Cyrus servers We use Horde's Ingo application which I assume is what your referring to. It works very well and is relatively "pretty". > with all the accounts spread out across those three servers. The problem I > am running into with any vacation application, is how to identify to the > vacation application which server a user is on without the user needing to > know this information. Is anyone else in a similar situation? Could you use Horde hooks to this? Horde hooks are pretty easy and they can set values you then refer to in your sources configuration. -- OpenGroupware developer: awilliam at whitemice.org OpenGroupare & Cyrus IMAPd documenation @ From michael.menge at zdv.uni-tuebingen.de Tue Mar 10 14:55:13 2009 From: michael.menge at zdv.uni-tuebingen.de (Michael Menge) Date: Tue, 10 Mar 2009 19:55:13 +0100 Subject: Vacation Application In-Reply-To: References: Message-ID: <20090310195513.454943x1i0562pht@webmail.uni-tuebingen.de> Hi, Quoting "Gottschalk, David" : > Hi All, > I have a question regarding a vacation application for Cyrus > servers. I want to know what others are using to meet this need. > Currently, we have a home grown application that allows users to > enable/disable vacation messages for their Cyrus accounts. This > application is old and outdated, and needs to be replaced. I found > the Horde vacation application which looks like it will meet my > needs, but I have one issue. We currently have three Cyrus servers > with all the accounts spread out across those three servers. The > problem I am running into with any vacation application, is how to > identify to the vacation application which server a user is on > without the user needing to know this information. Is anyone else in > a similar situation? > We are in the same situation. We have our accounts spread across six servers. For each account there is a DNS-alias pointing to the mailserver the account is on. We use the Horde Framework as webmailer and pim server, and allow our users to change their sieve scripts. You can use the hooks in Horde to change the server, or you can run functions in your horde configuration file, to set the correct value for the servername. Editing the filter rules and Vacations is easy and sufficent for most users. But the webinterface does not allow all what would be possible with sieve. You should restrict the access to timsieved to horde, so that the sievescripts are consitent with the horde database. Regards Michael Menge -------------------------------------------------------------------------------- M.Menge Tel.: (49) 7071/29-70316 Universit?t T?bingen Fax.: (49) 7071/29-5912 Zentrum f?r Datenverarbeitung mail: michael.menge at zdv.uni-tuebingen.de W?chterstra?e 76 72074 T?bingen -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5339 bytes Desc: S/MIME krytographische Unterschrift Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090310/03f6b7b3/attachment.bin From dgottsc at emory.edu Tue Mar 10 15:00:39 2009 From: dgottsc at emory.edu (Gottschalk, David) Date: Tue, 10 Mar 2009 15:00:39 -0400 Subject: Vacation Application In-Reply-To: <49B6AA9B.4060905@onlight.com> References: <49B6AA9B.4060905@onlight.com> Message-ID: No murder here. Oldschool Cyrus setup. I want to upgrade it actually, but there are 30K + accounts on these 3 servers. The only problem I see with having it try each server, but failing how do you know if it really isn't that server, or there is just a network issue. David Gottschalk UTS Email team david.gottschalk at emory.edu -----Original Message----- From: Nic Bernstein [mailto:nic at onlight.com] Sent: Tuesday, March 10, 2009 2:00 PM To: Gottschalk, David Cc: info-cyrus at lists.andrew.cmu.edu Subject: Re: Vacation Application On 03/10/2009 12:42 PM, Gottschalk, David wrote: > Hi All, > I have a question regarding a vacation application for Cyrus servers. I want to know what others are using to meet this need. Currently, we have a home grown application that allows users to enable/disable vacation messages for their Cyrus accounts. This application is old and outdated, and needs to be replaced. I found the Horde vacation application which looks like it will meet my needs, but I have one issue. We currently have three Cyrus servers with all the accounts spread out across those three servers. The problem I am running into with any vacation application, is how to identify to the vacation application which server a user is on without the user needing to know this information. Is anyone else in a similar situation? > Um, use a murder? Otherwise you may want to adapt something like the cyrus user validation script which was recently posted here. Your application could then perform lookups in the output of that script. Lastly, you could just let you application attempt authenticating against each server, in turn, stopping once it succeeds. Cheers, -nic -- Nic Bernstein nic at onlight.com Onlight llc. www.onlight.com 2266 North Prospect Avenue #610 v. 414.272.4477 Milwaukee, Wisconsin 53202-6306 f. 414.290.0335 This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments). From dgottsc at emory.edu Tue Mar 10 15:03:10 2009 From: dgottsc at emory.edu (Gottschalk, David) Date: Tue, 10 Mar 2009 15:03:10 -0400 Subject: Vacation Application In-Reply-To: <1236707904.5613.3.camel@linux-m3mt> References: <1236707904.5613.3.camel@linux-m3mt> Message-ID: I'm actually using the Horde vacation app that is part of Sork utilities. David Gottschalk UTS Email team david.gottschalk at emory.edu 404.727.9744 -----Original Message----- From: info-cyrus-bounces+dgottsc=emory.edu at lists.andrew.cmu.edu [mailto:info-cyrus-bounces+dgottsc=emory.edu at lists.andrew.cmu.edu] On Behalf Of Adam Tauno Williams Sent: Tuesday, March 10, 2009 1:58 PM To: info-cyrus at lists.andrew.cmu.edu Subject: Re: Vacation Application On Tue, 2009-03-10 at 13:42 -0400, Gottschalk, David wrote: > Hi All, > I have a question regarding a vacation application for Cyrus servers. I want > to know what others are using to meet this need. Currently, we have a home > grown application that allows users to enable/disable vacation messages for > their Cyrus accounts. This application is old and outdated, and needs to be > replaced. I found the Horde vacation application which looks like it will > meet my needs, but I have one issue. We currently have three Cyrus servers We use Horde's Ingo application which I assume is what your referring to. It works very well and is relatively "pretty". > with all the accounts spread out across those three servers. The problem I > am running into with any vacation application, is how to identify to the > vacation application which server a user is on without the user needing to > know this information. Is anyone else in a similar situation? Could you use Horde hooks to this? Horde hooks are pretty easy and they can set values you then refer to in your sources configuration. -- OpenGroupware developer: awilliam at whitemice.org OpenGroupare & Cyrus IMAPd documenation @ ---- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments). From ktm at rice.edu Tue Mar 10 15:05:59 2009 From: ktm at rice.edu (Kenneth Marshall) Date: Tue, 10 Mar 2009 14:05:59 -0500 Subject: Vacation Application In-Reply-To: References: <49B6AA9B.4060905@onlight.com> Message-ID: <20090310190559.GT13289@it.is.rice.edu> We use SmartSieve (http://smartsieve.sourceforge.net/) to manage vacation/sieve scripts. We look up the appropriate backend in our LDAP directory, but a DNS lookup could be done just as easily. It is simple to setup an configure, 1.0RC2. Cheers, Ken On Tue, Mar 10, 2009 at 03:00:39PM -0400, Gottschalk, David wrote: > No murder here. > > Oldschool Cyrus setup. I want to upgrade it actually, but there are 30K + accounts on these 3 servers. > > The only problem I see with having it try each server, but failing how do you know if it really isn't that server, or there is just a network issue. > > David Gottschalk > UTS Email team > david.gottschalk at emory.edu > > > -----Original Message----- > From: Nic Bernstein [mailto:nic at onlight.com] > Sent: Tuesday, March 10, 2009 2:00 PM > To: Gottschalk, David > Cc: info-cyrus at lists.andrew.cmu.edu > Subject: Re: Vacation Application > > On 03/10/2009 12:42 PM, Gottschalk, David wrote: > > Hi All, > > I have a question regarding a vacation application for Cyrus servers. I want to know what others are using to meet this need. Currently, we have a home grown application that allows users to enable/disable vacation messages for their Cyrus accounts. This application is old and outdated, and needs to be replaced. I found the Horde vacation application which looks like it will meet my needs, but I have one issue. We currently have three Cyrus servers with all the accounts spread out across those three servers. The problem I am running into with any vacation application, is how to identify to the vacation application which server a user is on without the user needing to know this information. Is anyone else in a similar situation? > > > Um, use a murder? Otherwise you may want to adapt something like the > cyrus user validation script which was recently posted here. Your > application could then perform lookups in the output of that script. > Lastly, you could just let you application attempt authenticating > against each server, in turn, stopping once it succeeds. > > Cheers, > -nic > > -- > Nic Bernstein nic at onlight.com > Onlight llc. www.onlight.com > 2266 North Prospect Avenue #610 v. 414.272.4477 > Milwaukee, Wisconsin 53202-6306 f. 414.290.0335 > > > This e-mail message (including any attachments) is for the sole use of > the intended recipient(s) and may contain confidential and privileged > information. If the reader of this message is not the intended > recipient, you are hereby notified that any dissemination, distribution > or copying of this message (including any attachments) is strictly > prohibited. > > If you have received this message in error, please contact > the sender by reply e-mail message and destroy all copies of the > original message (including attachments). > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > From dgottsc at emory.edu Tue Mar 10 15:13:18 2009 From: dgottsc at emory.edu (Gottschalk, David) Date: Tue, 10 Mar 2009 15:13:18 -0400 Subject: Vacation Application In-Reply-To: <20090310190559.GT13289@it.is.rice.edu> References: <49B6AA9B.4060905@onlight.com> <20090310190559.GT13289@it.is.rice.edu> Message-ID: That looks great! I will check it out. Thanks for the link. David Gottschalk UTS Email team david.gottschalk at emory.edu -----Original Message----- From: Kenneth Marshall [mailto:ktm at rice.edu] Sent: Tuesday, March 10, 2009 3:06 PM To: Gottschalk, David Cc: Nic Bernstein; info-cyrus at lists.andrew.cmu.edu Subject: Re: Vacation Application We use SmartSieve (http://smartsieve.sourceforge.net/) to manage vacation/sieve scripts. We look up the appropriate backend in our LDAP directory, but a DNS lookup could be done just as easily. It is simple to setup an configure, 1.0RC2. Cheers, Ken On Tue, Mar 10, 2009 at 03:00:39PM -0400, Gottschalk, David wrote: > No murder here. > > Oldschool Cyrus setup. I want to upgrade it actually, but there are 30K + accounts on these 3 servers. > > The only problem I see with having it try each server, but failing how do you know if it really isn't that server, or there is just a network issue. > > David Gottschalk > UTS Email team > david.gottschalk at emory.edu > > > -----Original Message----- > From: Nic Bernstein [mailto:nic at onlight.com] > Sent: Tuesday, March 10, 2009 2:00 PM > To: Gottschalk, David > Cc: info-cyrus at lists.andrew.cmu.edu > Subject: Re: Vacation Application > > On 03/10/2009 12:42 PM, Gottschalk, David wrote: > > Hi All, > > I have a question regarding a vacation application for Cyrus servers. I want to know what others are using to meet this need. Currently, we have a home grown application that allows users to enable/disable vacation messages for their Cyrus accounts. This application is old and outdated, and needs to be replaced. I found the Horde vacation application which looks like it will meet my needs, but I have one issue. We currently have three Cyrus servers with all the accounts spread out across those three servers. The problem I am running into with any vacation application, is how to identify to the vacation application which server a user is on without the user needing to know this information. Is anyone else in a similar situation? > > > Um, use a murder? Otherwise you may want to adapt something like the > cyrus user validation script which was recently posted here. Your > application could then perform lookups in the output of that script. > Lastly, you could just let you application attempt authenticating > against each server, in turn, stopping once it succeeds. > > Cheers, > -nic > > -- > Nic Bernstein nic at onlight.com > Onlight llc. www.onlight.com > 2266 North Prospect Avenue #610 v. 414.272.4477 > Milwaukee, Wisconsin 53202-6306 f. 414.290.0335 > > > This e-mail message (including any attachments) is for the sole use of > the intended recipient(s) and may contain confidential and privileged > information. If the reader of this message is not the intended > recipient, you are hereby notified that any dissemination, distribution > or copying of this message (including any attachments) is strictly > prohibited. > > If you have received this message in error, please contact > the sender by reply e-mail message and destroy all copies of the > original message (including attachments). > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > From omniflux+lists at omniflux.com Tue Mar 10 18:44:19 2009 From: omniflux+lists at omniflux.com (Omni Flux) Date: Tue, 10 Mar 2009 16:44:19 -0600 Subject: Announcing cyrus-user-map, a local recipient map generator for Postfix In-Reply-To: <49B6547C.9070807@pragmatic-source.com> References: <200903100959.22299.listreader@jsj.dyndns.org> <49B639CC.5030603@pragmatic-source.com> <200903101150.08237.listreader@jsj.dyndns.org> <49B6547C.9070807@pragmatic-source.com> Message-ID: <49B6ED43.9070303@omniflux.com> I have a policy daemon which verifies quotas via smmapd. I have not looked at it in a long time, but maybe it can be easily extended to verify mailbox existence as well? It is available at http://www.omniflux.com/devel/ It does have some shortcomings such as not supporting virtual domains, but this is because I haven't needed that, not because it cannot be done. -- Omni Flux Farzad FARID wrote: > Thanks to you and Eric Luyten for the information on the small overhead. > So until someone ports smmapd to Postfix I agree that the recipient > address validation setting is sufficient. From gary at primeexalia.com Tue Mar 10 23:57:54 2009 From: gary at primeexalia.com (Gary W. Smith) Date: Tue, 10 Mar 2009 20:57:54 -0700 Subject: Replication question Message-ID: <417DD550405AF84F95040DA44520C8E50BCC4C@pxtbenexd02.pxt.primeexalia.com> A couple weeks back I asked about combining multiple stores into a single one and someone recommended replication. So I started playing around with it to populate a new store from an existing. This seems to be working fine. I still have a little more testing to do though. My question is about the multiple mail stores. Each mail store has unique names. I assume the only thing I need to do is configure all of the masters to replicate to the backend and then run sync_client (with a list of all of the unique mailbox names for that server) on each of the mail stores. Is this correct? Also, if they have sieve rules in place, do these copy over as well or is it just the mailbox? From brong at fastmail.fm Wed Mar 11 00:22:11 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 11 Mar 2009 15:22:11 +1100 Subject: Calling all regressions In-Reply-To: <26722814-9B2B-4203-AC1A-0554A9D2C190@umich.edu> References: <1236675674.12318.1304575661@webmail.messagingengine.com> <20090310120713.140117sn46lwgg69@webmail.uni-tuebingen.de> <1236688701.5616.17.camel@linux-m3mt> <20090310135130.156225x0tn2katzm@webmail.uni-tuebingen.de> <26722814-9B2B-4203-AC1A-0554A9D2C190@umich.edu> Message-ID: <20090311042211.GD8560@brong.net> On Tue, Mar 10, 2009 at 01:33:00PM -0400, Wesley Craig wrote: > On 10 Mar 2009, at 08:51, Michael Menge wrote: > > I think i should rephrase my question. > > > > Is there a way for cyrus to know that it was updated > > since the last time it did run, so it can do the things > > suggested by Bron. > > Perhaps we should add a config option with no default whose value > must be the version of the software you're running. Hmmm.... v2.3.13-fmsvn18309-5c1c69c4 ;) It makes inplace updates a cow, since cyrus detects the new file and automatically closes old processes and starts new ones. Pretty tricky. Though a configuration with no default that said "running Cyrus 2.3 series" would be reasonable. Actually, I think at this point I would recommend releasing Cyrus 2.4 based on our current stable plus this change, and require a config variable that said "I'm running Cyrus 2.4". At this point I would also recommend revisiting EVERY config option and making the default be the best value (in our humble opinions) rather than the backwards compatible one. I have written about this before - and I think it's a jump we should take pretty soon, to get all new users using the best possible options without having to maintain a giant config file. Bron. From simon.matter at invoca.ch Wed Mar 11 02:42:17 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Wed, 11 Mar 2009 07:42:17 +0100 (CET) Subject: Calling all regressions In-Reply-To: <20090311042211.GD8560@brong.net> References: <1236675674.12318.1304575661@webmail.messagingengine.com> <20090310120713.140117sn46lwgg69@webmail.uni-tuebingen.de> <1236688701.5616.17.camel@linux-m3mt> <20090310135130.156225x0tn2katzm@webmail.uni-tuebingen.de> <26722814-9B2B-4203-AC1A-0554A9D2C190@umich.edu> <20090311042211.GD8560@brong.net> Message-ID: <0754101f074cc5778d94e72233198371.squirrel@webmail.bi.corp.invoca.ch> > On Tue, Mar 10, 2009 at 01:33:00PM -0400, Wesley Craig wrote: >> On 10 Mar 2009, at 08:51, Michael Menge wrote: >> > I think i should rephrase my question. >> > >> > Is there a way for cyrus to know that it was updated >> > since the last time it did run, so it can do the things >> > suggested by Bron. >> >> Perhaps we should add a config option with no default whose value >> must be the version of the software you're running. > > Hmmm.... v2.3.13-fmsvn18309-5c1c69c4 ;) It makes inplace updates > a cow, since cyrus detects the new file and automatically closes > old processes and starts new ones. Pretty tricky. > > Though a configuration with no default that said "running Cyrus 2.3 > series" would be reasonable. > > Actually, I think at this point I would recommend releasing > Cyrus 2.4 based on our current stable plus this change, and > require a config variable that said "I'm running Cyrus 2.4". > > At this point I would also recommend revisiting EVERY config > option and making the default be the best value (in our humble > opinions) rather than the backwards compatible one. I have > written about this before - and I think it's a jump we should > take pretty soon, to get all new users using the best possible > options without having to maintain a giant config file. As a package maintainer I would be happy to see some more advanced options being the default. My rpms have different defaults but that requires patching not only the options file but also the docs to reflect the changes. Unfortunately it can still lead to confusion if people "forget" to read the manpage and other docs. About the config variable discussed above: Since my rpms do quite some magick to make upgrades work without any user intervention, I had this problem for a long time. One should keep in mind that having old config files is one thing, the other thing are old configdir/spooldirs. My solution for the latter was to maintain a db config file in the rpm package, and maintain another db config file in the configdir. That way the init script evaulates the two db configs and the imapd config, it they don't match it takes any action it needs and then writes the new config to the configdir. That way you can also transfer a configdir/spooldir from another server with different cyrus-imapd version to the current server and things will still work correctly. Or you can change imapd.conf and the changes will take effect on the configdir on startup. At the moment the db config includes the db version and the sieve version like this: annotation_db=skiplist duplicate_db=skiplist mboxkey_db=skiplist mboxlist_db=skiplist ptscache_db=skiplist quota_db=quotalegacy seenstate_db=skiplist sieve_version=2.2.3 statuscache_db=skiplist subscription_db=flat tlscache_db=skiplist My question is how can all this be implemented in cyrus-imapd without too many changes. What about having a function to read all current setting (something like postconf does in postfix) and have cyrus-imapd maintain a copy of this in the configdir? That way cyrus-imapd or the init script could check whether both configs match enough and they action or refuse to start if they don't match. The config file should not only include configurable options but also internal options like what I did with the sieve_version to reflect internal changes. Cyrus-imapd does handles a lot of those changes on the fly but not all, live upgrading sieve scripts, and those should be found in the configs. Regards, Simon From boutilpj at ednet.ns.ca Wed Mar 11 08:46:14 2009 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Wed, 11 Mar 2009 09:46:14 -0300 Subject: unexpunge segfaults with -l on some mailboxes Message-ID: <49B7B296.6000305@ednet.ns.ca> Cyrus 2.3.13: [cyrus at student2 ~]$ gdb /usr/local/cyrus/bin/unexpunge core.16094 GNU gdb Red Hat Linux (6.5-37.el5_2.2rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...(no debugging symbols found) Using host libthread_db library "/lib64/libthread_db.so.1". warning: Can't read pathname for load map: Input/output error. Reading symbols from /usr/local/lib64/libsasl2.so.2...done. Loaded symbols for /usr/local/lib64/libsasl2.so.2 Reading symbols from /lib64/libssl.so.6...done. Loaded symbols for /lib64/libssl.so.6 Reading symbols from /lib64/libcrypto.so.6...done. Loaded symbols for /lib64/libcrypto.so.6 Reading symbols from /lib64/libresolv.so.2...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libdb-4.3.so...done. Loaded symbols for /lib64/libdb-4.3.so Reading symbols from /lib64/libc.so.6...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/libdl.so.2...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done. Loaded symbols for /usr/lib64/libgssapi_krb5.so.2 Reading symbols from /usr/lib64/libkrb5.so.3...done. Loaded symbols for /usr/lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /usr/lib64/libk5crypto.so.3...done. Loaded symbols for /usr/lib64/libk5crypto.so.3 Reading symbols from /usr/lib64/libz.so.1...done. Loaded symbols for /usr/lib64/libz.so.1 Reading symbols from /lib64/libpthread.so.0...done. Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/ld-linux-x86-64.so.2...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib64/libkrb5support.so.0...done. Loaded symbols for /usr/lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libselinux.so.1...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/libsepol.so.1...done. Loaded symbols for /lib64/libsepol.so.1 Core was generated by `/usr/local/cyrus/bin/unexpunge -l user.boutilpj'. Program terminated with signal 11, Segmentation fault. #0 0x0000000000405d64 in list_expunged () (gdb) bt #0 0x0000000000405d64 in list_expunged () #1 0x00000000004073f1 in main () (gdb) -------------- next part -------------- A non-text attachment was scrubbed... Name: boutilpj.vcf Type: text/x-vcard Size: 286 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090311/3c58b340/attachment.vcf From awilliam at whitemice.org Wed Mar 11 08:53:01 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Wed, 11 Mar 2009 08:53:01 -0400 Subject: Intermittent mailbox not found In-Reply-To: References: <1236630082.5320.26.camel@linux-m3mt> Message-ID: <1236775981.5554.9.camel@linux-m3mt> On Tue, 2009-03-10 at 11:29 +0000, Antony Gelberg wrote: > On 2009-03-09, Adam Tauno Williams wrote: > >> I have inherited a cyrus / postfix server, which is generally well-behaved. > >> However, there is an intermittent error when delivering messages to a certain > >> mailbox. It is infrequent, and generally the message is delievered on the > >> user's next attempt in his MUA. The log is something like: > > What format is your mailbox database in? Do you see any concomitant > > messages in the log? > Not sure what format it's in - I'm new to Cyrus but I thought it had its own > format, similar to Maildir? Not the mail spool (/var/spool/imap?) but the metadata spool (/var/lib/imap?). There are multiple databases in the metadata spool, one of which is the mailbox database that lists the mailboxes, their ACLs, etc... These databases can be in a variety of formats: skiplist, berkley db, flat, etc... It is important to see if you have an DBERROR (berkley) or skiplist error messages in your log as that might relate to the problem. [root at sardine imap]# pwd /var/lib/imap [root at sardine imap]# file mailboxes.db mailboxes.db: Cyrus skiplist DB > I had to look up concomitant in the > dictionary, and I don't believe there are any concomitant messages in the log. > >> Mar 9 14:29:07 captain postfix/lmtp[20644]: D557618536: > >> to=, > >> relay=captain.munged-domain.com[/var/run/cyrus/socket/lmtp], delay=0.1, > >> delays=0.02/0.01/0/0.08, dsn=5.1.1, status=bounced (host > >> captain.munged-domain.com[/var/run/cyrus/socket/lmtp] said: 550-Mailbox > >> unknown. > >> Either there is no mailbox associated with this 550-name or you do not have > >> authorization to see it. 550 5.1.1 User unknown (in reply to RCPT TO command)) > >> Mar 9 14:29:07 captain postfix/cleanup[20642]: ED3B418538: > >> message-id=<20090309142907.ED3B418538 at captain.munged-domain.com> > >> Does anyone have any idea how I can go about debugging this? > > Any chance you can increase the debug level of your MTA (postfix?) to > > see the entire conversation? > Done that, but will have to wait for another failure. -- OpenGroupware developer: awilliam at whitemice.org OpenGroupare & Cyrus IMAPd documenation @ From boutilpj at ednet.ns.ca Wed Mar 11 11:40:28 2009 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Wed, 11 Mar 2009 12:40:28 -0300 Subject: unexpunge segfaults with -l on some mailboxes In-Reply-To: <49B7B296.6000305@ednet.ns.ca> References: <49B7B296.6000305@ednet.ns.ca> Message-ID: <49B7DB6C.9080602@ednet.ns.ca> Patrick Boutilier wrote: > Cyrus 2.3.13: > > > [cyrus at student2 ~]$ gdb /usr/local/cyrus/bin/unexpunge core.16094 > GNU gdb Red Hat Linux (6.5-37.el5_2.2rh) > Copyright (C) 2006 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you > are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu"...(no debugging > symbols found) > Using host libthread_db library "/lib64/libthread_db.so.1". > > > warning: Can't read pathname for load map: Input/output error. > Reading symbols from /usr/local/lib64/libsasl2.so.2...done. > Loaded symbols for /usr/local/lib64/libsasl2.so.2 > Reading symbols from /lib64/libssl.so.6...done. > Loaded symbols for /lib64/libssl.so.6 > Reading symbols from /lib64/libcrypto.so.6...done. > Loaded symbols for /lib64/libcrypto.so.6 > Reading symbols from /lib64/libresolv.so.2...done. > Loaded symbols for /lib64/libresolv.so.2 > Reading symbols from /lib64/libdb-4.3.so...done. > Loaded symbols for /lib64/libdb-4.3.so > Reading symbols from /lib64/libc.so.6...done. > Loaded symbols for /lib64/libc.so.6 > Reading symbols from /lib64/libdl.so.2...done. > Loaded symbols for /lib64/libdl.so.2 > Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done. > Loaded symbols for /usr/lib64/libgssapi_krb5.so.2 > Reading symbols from /usr/lib64/libkrb5.so.3...done. > Loaded symbols for /usr/lib64/libkrb5.so.3 > Reading symbols from /lib64/libcom_err.so.2...done. > Loaded symbols for /lib64/libcom_err.so.2 > Reading symbols from /usr/lib64/libk5crypto.so.3...done. > Loaded symbols for /usr/lib64/libk5crypto.so.3 > Reading symbols from /usr/lib64/libz.so.1...done. > Loaded symbols for /usr/lib64/libz.so.1 > Reading symbols from /lib64/libpthread.so.0...done. > Loaded symbols for /lib64/libpthread.so.0 > Reading symbols from /lib64/ld-linux-x86-64.so.2...done. > Loaded symbols for /lib64/ld-linux-x86-64.so.2 > Reading symbols from /usr/lib64/libkrb5support.so.0...done. > Loaded symbols for /usr/lib64/libkrb5support.so.0 > Reading symbols from /lib64/libkeyutils.so.1...done. > Loaded symbols for /lib64/libkeyutils.so.1 > Reading symbols from /lib64/libselinux.so.1...done. > Loaded symbols for /lib64/libselinux.so.1 > Reading symbols from /lib64/libsepol.so.1...done. > Loaded symbols for /lib64/libsepol.so.1 > Core was generated by `/usr/local/cyrus/bin/unexpunge -l user.boutilpj'. > Program terminated with signal 11, Segmentation fault. > #0 0x0000000000405d64 in list_expunged () > (gdb) bt > #0 0x0000000000405d64 in list_expunged () > #1 0x00000000004073f1 in main () > (gdb) Darn. This is caused by mailbox corruption again. My script to detect corruption wasn't working properly. Any idea how we can track down what is causing the corruption? Thanks. Mar 11 03:05:52 student2 cyr_expire[7604]: IOERROR: reading cache record for user.boutilpj: item 1 has bogus offset 1685634032 of 2049160 for 5020/6938; mailbox needs a reconstruct Mar 11 03:05:52 student2 cyr_expire[7604]: failure expiring user.boutilpj: System I/O error > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -------------- next part -------------- A non-text attachment was scrubbed... Name: boutilpj.vcf Type: text/x-vcard Size: 286 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090311/5df49d29/attachment.vcf From brong at fastmail.fm Wed Mar 11 20:07:06 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 12 Mar 2009 11:07:06 +1100 Subject: unexpunge segfaults with -l on some mailboxes In-Reply-To: <49B7DB6C.9080602@ednet.ns.ca> References: <49B7B296.6000305@ednet.ns.ca> <49B7DB6C.9080602@ednet.ns.ca> Message-ID: <20090312000706.GA7208@brong.net> On Wed, Mar 11, 2009 at 12:40:28PM -0300, Patrick Boutilier wrote: > Darn. This is caused by mailbox corruption again. My script to detect > corruption wasn't working properly. Any idea how we can track down what > is causing the corruption? Yeah, I was going to say. It's corrupted cache file offset pointers. Assuming you upgraded this from an earlier version of Cyrus at some point, cyrus.expunge and cyrus.index files had issues maintaining cache pointer consistency. You pretty much have to reconstruct all your mailboxes to guarantee consistency I think. I'm tempted to protect the code from crashing though... we don't use unexpunge at FastMail, which is probably why I haven't already done so. Something like the attached should do it. I'll test it more completely and commit it to CVS for 2.3.14 (since Ken hasn't cut a release candidate yet!) Bron. -------------- next part -------------- diff --git a/imap/mailbox.c b/imap/mailbox.c index 39896a4..4044b3c 100644 --- a/imap/mailbox.c +++ b/imap/mailbox.c @@ -297,19 +297,11 @@ unsigned mailbox_cached_header_inline(const char *text) return BIT32_MAX; } -unsigned long mailbox_cache_size(struct mailbox *mailbox, unsigned msgno) +unsigned long mailbox_cache_size_offset(struct mailbox *mailbox, unsigned cache_offset) { - const char *p; - unsigned long cache_offset; unsigned int cache_ent; const char *cacheitem, *cacheitembegin; - assert((msgno > 0) && (msgno <= mailbox->exists)); - - p = (mailbox->index_base + mailbox->start_offset + - ((msgno-1) * mailbox->record_size)); - - cache_offset = ntohl(*((bit32 *)(p+OFFSET_CACHE_OFFSET))); if (cache_offset > mailbox->cache_size) { return 0; } @@ -321,13 +313,27 @@ unsigned long mailbox_cache_size(struct mailbox *mailbox, unsigned msgno) for (cache_ent = 0; cache_ent < NUM_CACHE_FIELDS; cache_ent++) { cacheitem = CACHE_ITEM_NEXT(cacheitem); if (cacheitem < cacheitembegin || - cacheitem > cacheitembegin + mailbox->cache_size) { + cacheitem > mailbox->cache_base + mailbox->cache_size) { return 0; /* clearly bogus */ } } return (cacheitem - cacheitembegin); } +unsigned long mailbox_cache_size(struct mailbox *mailbox, unsigned msgno) +{ + const char *p; + + assert((msgno > 0) && (msgno <= mailbox->exists)); + + p = (mailbox->index_base + mailbox->start_offset + + ((msgno-1) * mailbox->record_size)); + + cache_offset = ntohl(*((bit32 *)(p+OFFSET_CACHE_OFFSET))); + + return mailbox_cache_size_offset(mailbox, cache_offset); +} + /* function to be used for notification of mailbox changes/updates */ static mailbox_notifyproc_t *updatenotifier = NULL; diff --git a/imap/mailbox.h b/imap/mailbox.h index 06b43b5..19c0662 100644 --- a/imap/mailbox.h +++ b/imap/mailbox.h @@ -293,6 +293,7 @@ enum { unsigned mailbox_cached_header(const char *s); unsigned mailbox_cached_header_inline(const char *text); +unsigned long mailbox_cache_size_offset(struct mailbox *mailbox, unsigned cache_offset); unsigned long mailbox_cache_size(struct mailbox *mailbox, unsigned msgno); typedef unsigned mailbox_decideproc_t(struct mailbox *mailbox, void *rock, diff --git a/imap/unexpunge.c b/imap/unexpunge.c index 37f62cc..b7816e2 100644 --- a/imap/unexpunge.c +++ b/imap/unexpunge.c @@ -123,7 +123,7 @@ void list_expunged(struct mailbox *mailbox, const char *expunge_index_base) { const char *rec; - unsigned msgno; + unsigned msgno, cache_ent; unsigned long uid, size, cache_offset; time_t internaldate, sentdate, last_updated; const char *cacheitem; @@ -146,6 +146,11 @@ void list_expunged(struct mailbox *mailbox, printf("\tRecv: %s", ctime(&internaldate)); printf("\tExpg: %s", ctime(&last_updated)); + if (!mailbox_cache_size_offset(mailbox, cache_offset)) { + printf("\tERROR: cache record missing or corrupt, not printing cache details\n"); + continue; + } + cacheitem = mailbox->cache_base + cache_offset; cacheitem = CACHE_ITEM_NEXT(cacheitem); /* skip envelope */ cacheitem = CACHE_ITEM_NEXT(cacheitem); /* skip body structure */ From wcooley at nakedape.cc Thu Mar 12 00:42:32 2009 From: wcooley at nakedape.cc (Wil Cooley) Date: Wed, 11 Mar 2009 21:42:32 -0700 Subject: Is user lock-out required for backend xfers? Message-ID: <1236832952.8538.13.camel@spartacus.nakedape.cc> Somewhere I read (or thought I read) that it was important to ensure that a user was not logged in when his mailbox was transferred between backends in a Murder setup, otherwise there was a risk of mailbox corruption. Is this true or have I been reading the tabloids again? Context: I have several shiny new backends with 2.3 that I have ready to replace our old 2.2 backends. We have something like 70,000 mailboxes to move, which will obviously be something done in nightly batches over a period of weeks. Our current obstacle is having to ensure that the mailbox is not open, because that requires: notifying the user that his e-mail will be unavailable for a period of time, locking the user's account (which locks him out of everything else unless we work up some Cyrus-specific solution, like some PAM magic), checking all the front-ends to verify that he's not logged in (this is iffy, because the proc/ files are all we've got and some of those are left-overs from crashes and lack of housekeeping), doing the transfer, and then undoing all of this. Alternately, if there is only a small risk of mailbox corruption, it may be better to just do the transfers late at night and accept having to do a handful of mailbox reconstructs. Is this what other people have done? Wil -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090311/2f8099b1/attachment.bin From brong at fastmail.fm Thu Mar 12 05:35:29 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 12 Mar 2009 20:35:29 +1100 Subject: Calling all regressions In-Reply-To: <0754101f074cc5778d94e72233198371.squirrel@webmail.bi.corp.invoca.ch> References: <1236675674.12318.1304575661@webmail.messagingengine.com> <20090310120713.140117sn46lwgg69@webmail.uni-tuebingen.de> <1236688701.5616.17.camel@linux-m3mt> <20090310135130.156225x0tn2katzm@webmail.uni-tuebingen.de> <26722814-9B2B-4203-AC1A-0554A9D2C190@umich.edu> <20090311042211.GD8560@brong.net> <0754101f074cc5778d94e72233198371.squirrel@webmail.bi.corp.invoca.ch> Message-ID: <20090312093529.GA10774@brong.net> Hi Simon, I was particularly interested in your feedback, since you're the most active package maintainer, and probably responsible for most of our "long tail" of users of the recent code. They're the people who are of most concern for things like configuration changes. Big sites tend to have more time to spend getting their config "perfect", but the little guys just need a turnkey solution that's "good enough"! On Wed, Mar 11, 2009 at 07:42:17AM +0100, Simon Matter wrote: > > At this point I would also recommend revisiting EVERY config > > option and making the default be the best value (in our humble > > opinions) rather than the backwards compatible one. I have > > written about this before - and I think it's a jump we should > > take pretty soon, to get all new users using the best possible > > options without having to maintain a giant config file. > > As a package maintainer I would be happy to see some more advanced options > being the default. My rpms have different defaults but that requires > patching not only the options file but also the docs to reflect the > changes. Unfortunately it can still lead to confusion if people "forget" > to read the manpage and other docs. Yes, much better if we all have the same defaults. Plus, anything that reduces the size of your diff is good. > About the config variable discussed above: > Since my rpms do quite some magick to make upgrades work without any user > intervention, I had this problem for a long time. One should keep in mind > that having old config files is one thing, the other thing are old > configdir/spooldirs. How old do we support back until? The index file upgrade code looks pretty backwards compatible for approximately ever. The main issue with spooldirs is hashing - but I think no hashing _is_ the sane default. Ext3 is limited to 32k subdirs, but if your site is that big you're big enough to read the documentation anyway - and newer filesystems are better about arbitrary limits too. > My solution for the latter was to maintain a db > config file in the rpm package, and maintain another db config file in the > configdir. That way the init script evaulates the two db configs and the > imapd config, it they don't match it takes any action it needs and then > writes the new config to the configdir. That way you can also transfer a > configdir/spooldir from another server with different cyrus-imapd version > to the current server and things will still work correctly. Or you can > change imapd.conf and the changes will take effect on the configdir on > startup. That's a good idea. I like it. Another thing to consider for 2.4! So basically - you have /etc/imapd.conf and /etc/cyrus.conf (or whatever) and $configdir/imapd.conf $configdir/cyrus.conf. Is that right? I really should have a look at your RPMs! > At the moment the db config includes the db version and the sieve version > like this: > annotation_db=skiplist > duplicate_db=skiplist > mboxkey_db=skiplist > mboxlist_db=skiplist > ptscache_db=skiplist > quota_db=quotalegacy > seenstate_db=skiplist > sieve_version=2.2.3 > statuscache_db=skiplist > subscription_db=flat > tlscache_db=skiplist > > My question is how can all this be implemented in cyrus-imapd without too > many changes. What about having a function to read all current setting > (something like postconf does in postfix) and have cyrus-imapd maintain a > copy of this in the configdir? AAA+++ to cyr_conf! I'll write it up :) I think we want it to have the following spec: * cyr_conf - output all configuration variables and their current value * cyr_conf -C $file - as above with the following config file * cyr_conf -n $name - all configuration variables for process $name (eg cyr_conf -n imapd => show imapd overrides where given) * cyr_conf -q - only show variables that are different than default - those three can be mixed and matched * cyr_conf -D - show all the DEFAULT variables. Obviously, ignores all other options! > That way cyrus-imapd or the init script > could check whether both configs match enough and they action or refuse to > start if they don't match. Cool. > The config file should not only include > configurable options but also internal options like what I did with the > sieve_version to reflect internal changes. Cyrus-imapd does handles a lot > of those changes on the fly but not all, live upgrading sieve scripts, and > those should be found in the configs. Yep, definitely. Store them all, and refuse to start unless: cyr_conf -C $oldfile is the same output as cyr_conf with the -C option passed to 'master'. Spit out an error message describing what's different and how to fix it. I like :) Bron. From carson at taltos.org Thu Mar 12 05:55:03 2009 From: carson at taltos.org (Carson Gaspar) Date: Thu, 12 Mar 2009 02:55:03 -0700 Subject: Calling all regressions In-Reply-To: <20090312093529.GA10774@brong.net> References: <1236675674.12318.1304575661@webmail.messagingengine.com> <20090310120713.140117sn46lwgg69@webmail.uni-tuebingen.de> <1236688701.5616.17.camel@linux-m3mt> <20090310135130.156225x0tn2katzm@webmail.uni-tuebingen.de> <26722814-9B2B-4203-AC1A-0554A9D2C190@umich.edu> <20090311042211.GD8560@brong.net> <0754101f074cc5778d94e72233198371.squirrel@webmail.bi.corp.invoca.ch> <20090312093529.GA10774@brong.net> Message-ID: <49B8DBF7.5040403@taltos.org> Bron Gondwana wrote: > AAA+++ to cyr_conf! I'll write it up :) I think we want it to have the > following spec: > > * cyr_conf - output all configuration variables and their current value > * cyr_conf -C $file - as above with the following config file > * cyr_conf -n $name - all configuration variables for process $name > (eg cyr_conf -n imapd => show imapd overrides where given) > * cyr_conf -q - only show variables that are different than default > > - those three can be mixed and matched > > * cyr_conf -D - show all the DEFAULT variables. Obviously, ignores all > other options! Please include one of the most useful options that postconf has (postconf -n) - emit only those config key/value pairs that are not set to their default values. More often useful in real life than just emitting all config key/value pairs. -- Carson From brong at fastmail.fm Thu Mar 12 07:42:34 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 12 Mar 2009 22:42:34 +1100 Subject: Calling all regressions In-Reply-To: <49B8DBF7.5040403@taltos.org> References: <1236675674.12318.1304575661@webmail.messagingengine.com> <20090310120713.140117sn46lwgg69@webmail.uni-tuebingen.de> <1236688701.5616.17.camel@linux-m3mt> <20090310135130.156225x0tn2katzm@webmail.uni-tuebingen.de> <26722814-9B2B-4203-AC1A-0554A9D2C190@umich.edu> <20090311042211.GD8560@brong.net> <0754101f074cc5778d94e72233198371.squirrel@webmail.bi.corp.invoca.ch> <20090312093529.GA10774@brong.net> <49B8DBF7.5040403@taltos.org> Message-ID: <20090312114234.GA20450@brong.net> On Thu, Mar 12, 2009 at 02:55:03AM -0700, Carson Gaspar wrote: > Bron Gondwana wrote: > > > AAA+++ to cyr_conf! I'll write it up :) I think we want it to have the > > following spec: > > > > * cyr_conf - output all configuration variables and their current value > > * cyr_conf -C $file - as above with the following config file > > * cyr_conf -n $name - all configuration variables for process $name > > (eg cyr_conf -n imapd => show imapd overrides where given) > > * cyr_conf -q - only show variables that are different than default > > > > - those three can be mixed and matched > > > > * cyr_conf -D - show all the DEFAULT variables. Obviously, ignores all > > other options! > > Please include one of the most useful options that postconf has > (postconf -n) - emit only those config key/value pairs that are not set > to their default values. More often useful in real life than just > emitting all config key/value pairs. Yeah, I called that -q. If we make it -n then we'll have to make the named process config a different character. The spec isn't nailed down yet, I haven't even written anything! Bron ( been busy with cache stuff ) From boutilpj at ednet.ns.ca Thu Mar 12 08:08:47 2009 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Thu, 12 Mar 2009 09:08:47 -0300 Subject: unexpunge segfaults with -l on some mailboxes In-Reply-To: <20090312000706.GA7208@brong.net> References: <49B7B296.6000305@ednet.ns.ca> <49B7DB6C.9080602@ednet.ns.ca> <20090312000706.GA7208@brong.net> Message-ID: <49B8FB4F.9050806@ednet.ns.ca> Bron Gondwana wrote: > On Wed, Mar 11, 2009 at 12:40:28PM -0300, Patrick Boutilier wrote: >> Darn. This is caused by mailbox corruption again. My script to detect >> corruption wasn't working properly. Any idea how we can track down what >> is causing the corruption? > > Yeah, I was going to say. > > It's corrupted cache file offset pointers. Assuming you upgraded > this from an earlier version of Cyrus at some point, I believe we started at 1.5.19 and then migrated up through the 2.0 , 2.2 series. Our last major update was from 2.2.13 to 2.3.11. Currently on 2.3.13 cyrus.expunge > and cyrus.index files had issues maintaining cache pointer > consistency. You pretty much have to reconstruct all your mailboxes > to guarantee consistency I think. This latest list of corrupted mailboxes includes many that were correupted a couple of weeks ago (including mine) so this is the second time they have been reconstructed. > > I'm tempted to protect the code from crashing though... we don't > use unexpunge at FastMail, which is probably why I haven't already > done so. > > Something like the attached should do it. I'll test it more > completely and commit it to CVS for 2.3.14 (since Ken hasn't > cut a release candidate yet!) Thanks for the patch. Would ipurge be causing the corruption? We currently purge e-mails older than 31 days on a weekly basis. I will turn that off for a bit (since disk space is not as much of an issue as it used to be) and see if the corruption returns. > > Bron. > -------------- next part -------------- A non-text attachment was scrubbed... Name: boutilpj.vcf Type: text/x-vcard Size: 286 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090312/ac9ce9f5/attachment.vcf From list at joreybump.com Thu Mar 12 08:32:56 2009 From: list at joreybump.com (Jorey Bump) Date: Thu, 12 Mar 2009 08:32:56 -0400 Subject: Calling all regressions In-Reply-To: <20090312114234.GA20450@brong.net> References: <1236675674.12318.1304575661@webmail.messagingengine.com> <20090310120713.140117sn46lwgg69@webmail.uni-tuebingen.de> <1236688701.5616.17.camel@linux-m3mt> <20090310135130.156225x0tn2katzm@webmail.uni-tuebingen.de> <26722814-9B2B-4203-AC1A-0554A9D2C190@umich.edu> <20090311042211.GD8560@brong.net> <0754101f074cc5778d94e72233198371.squirrel@webmail.bi.corp.invoca.ch> <20090312093529.GA10774@brong.net> <49B8DBF7.5040403@taltos.org> <20090312114234.GA20450@brong.net> Message-ID: <49B900F8.1030701@joreybump.com> Bron Gondwana wrote, at 03/12/2009 07:42 AM: > On Thu, Mar 12, 2009 at 02:55:03AM -0700, Carson Gaspar wrote: >> Bron Gondwana wrote: >> >>> AAA+++ to cyr_conf! I'll write it up :) I think we want it to have the >>> following spec: >>> >>> * cyr_conf - output all configuration variables and their current value >>> * cyr_conf -C $file - as above with the following config file >>> * cyr_conf -n $name - all configuration variables for process $name >>> (eg cyr_conf -n imapd => show imapd overrides where given) >>> * cyr_conf -q - only show variables that are different than default >>> >>> - those three can be mixed and matched >>> >>> * cyr_conf -D - show all the DEFAULT variables. Obviously, ignores all >>> other options! >> Please include one of the most useful options that postconf has >> (postconf -n) - emit only those config key/value pairs that are not set >> to their default values. More often useful in real life than just >> emitting all config key/value pairs. > > Yeah, I called that -q. If we make it -n then we'll have to make the > named process config a different character. The spec isn't nailed down > yet, I haven't even written anything! Actually, postconf -n shows parameters that have been explicitly set, even if they are the same as the default. This is a subtle (and IMHO very useful) distinction. Although many simply grep the output of postconf because they can't remember the exact name of a parameter, another very useful feature of postconf is the ability to directly query the value of one or more parameters using postconf [parameter...]: postconf mynetworks or: postconf mynetworks inet_interfaces I realize that a variety of MTAs are used in conjunction with Cyrus IMAPd, but Postfix is an extremely well thought-out application. As a Postfix administrator, I'd be thrilled to see a utility like cyr_conf modeled as closely after postconf as possible, if only to simplify any new tricks this old dog has to learn. From simon.matter at invoca.ch Thu Mar 12 09:04:17 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Thu, 12 Mar 2009 14:04:17 +0100 (CET) Subject: Calling all regressions In-Reply-To: <20090312093529.GA10774@brong.net> References: <1236675674.12318.1304575661@webmail.messagingengine.com> <20090310120713.140117sn46lwgg69@webmail.uni-tuebingen.de> <1236688701.5616.17.camel@linux-m3mt> <20090310135130.156225x0tn2katzm@webmail.uni-tuebingen.de> <26722814-9B2B-4203-AC1A-0554A9D2C190@umich.edu> <20090311042211.GD8560@brong.net> <0754101f074cc5778d94e72233198371.squirrel@webmail.bi.corp.invoca.ch> <20090312093529.GA10774@brong.net> Message-ID: <4f861a9b6f180919f2d881d80dbf03ca.squirrel@webmail.bi.corp.invoca.ch> > AAA+++ to cyr_conf! I'll write it up :) I think we want it to have the > following spec: > > * cyr_conf - output all configuration variables and their current value > * cyr_conf -C $file - as above with the following config file > * cyr_conf -n $name - all configuration variables for process $name > (eg cyr_conf -n imapd => show imapd overrides where given) > * cyr_conf -q - only show variables that are different than default > > - those three can be mixed and matched > > * cyr_conf -D - show all the DEFAULT variables. Obviously, ignores all > other options! > >> That way cyrus-imapd or the init script >> could check whether both configs match enough and they action or refuse >> to >> start if they don't match. > > Cool. > >> The config file should not only include >> configurable options but also internal options like what I did with the >> sieve_version to reflect internal changes. Cyrus-imapd does handles a >> lot >> of those changes on the fly but not all, live upgrading sieve scripts, >> and >> those should be found in the configs. > > Yep, definitely. Store them all, and refuse to start unless: > cyr_conf -C $oldfile is the same output as cyr_conf with the > -C option passed to 'master'. Spit out an error message > describing what's different and how to fix it. Hi Bron, cyr_conf sounds good :) For the "refuse to start", do you think it should be integrated into master or being kept in the init script? What would be very cool is a detect feature which detects things like database config from the configdir. That way it could also override configuration settings from the config files on startup and log about it. It sounds a bit dangerous, what do you think? Simon From awilliam at whitemice.org Thu Mar 12 09:49:40 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Thu, 12 Mar 2009 09:49:40 -0400 Subject: cyrus events off by an hour Message-ID: <1236865780.5675.18.camel@linux-m3mt> I have - delprune cmd="cyr_expire -E 3 -D 120 -X 120" at=0400 - so cyr_expire should run at 4:00, right? But it runs at 5:00. The time on the server is correct. Could daylight savings time change effect this? -- OpenGroupware developer: awilliam at whitemice.org OpenGroupare & Cyrus IMAPd documenation @ From morgan at orst.edu Thu Mar 12 12:27:27 2009 From: morgan at orst.edu (Andrew Morgan) Date: Thu, 12 Mar 2009 09:27:27 -0700 (PDT) Subject: cyrus events off by an hour In-Reply-To: <1236865780.5675.18.camel@linux-m3mt> References: <1236865780.5675.18.camel@linux-m3mt> Message-ID: On Thu, 12 Mar 2009, Adam Tauno Williams wrote: > I have - > delprune cmd="cyr_expire -E 3 -D 120 -X 120" at=0400 > - so cyr_expire should run at 4:00, right? > > But it runs at 5:00. The time on the server is correct. Could daylight > savings time change effect this? Yes, all of your "at=xyz" events will be off an hour until you restart cyrus. I found this bug a long time ago and basically ignored it. :) Andy From morgan at orst.edu Thu Mar 12 12:29:56 2009 From: morgan at orst.edu (Andrew Morgan) Date: Thu, 12 Mar 2009 09:29:56 -0700 (PDT) Subject: Is user lock-out required for backend xfers? In-Reply-To: <1236832952.8538.13.camel@spartacus.nakedape.cc> References: <1236832952.8538.13.camel@spartacus.nakedape.cc> Message-ID: On Wed, 11 Mar 2009, Wil Cooley wrote: > Somewhere I read (or thought I read) that it was important to ensure > that a user was not logged in when his mailbox was transferred between > backends in a Murder setup, otherwise there was a risk of mailbox > corruption. Is this true or have I been reading the tabloids again? > > Context: I have several shiny new backends with 2.3 that I have ready to > replace our old 2.2 backends. We have something like 70,000 mailboxes to > move, which will obviously be something done in nightly batches over a > period of weeks. > > Our current obstacle is having to ensure that the mailbox is not open, > because that requires: notifying the user that his e-mail will be > unavailable for a period of time, locking the user's account (which > locks him out of everything else unless we work up some Cyrus-specific > solution, like some PAM magic), checking all the front-ends to verify > that he's not logged in (this is iffy, because the proc/ files > are all we've got and some of those are left-overs from crashes and lack > of housekeeping), doing the transfer, and then undoing all of this. > > Alternately, if there is only a small risk of mailbox corruption, it may > be better to just do the transfers late at night and accept having to do > a handful of mailbox reconstructs. > > Is this what other people have done? When we went from a standalone server to a Murder setup, we moved half of our mailboxes from cyrus-be1 to cyrus-be2. We did them in batches overnight, without any special care taken to prevent users from accessing them. I'm not aware of any problems caused by this. This was back in Cyrus 2.1 or 2.2. So no promises, but it worked fine for us. Andy From dave64 at andrew.cmu.edu Thu Mar 12 13:03:08 2009 From: dave64 at andrew.cmu.edu (Dave McMurtrie) Date: Thu, 12 Mar 2009 13:03:08 -0400 Subject: Is user lock-out required for backend xfers? In-Reply-To: References: <1236832952.8538.13.camel@spartacus.nakedape.cc> Message-ID: <49B9404C.5050103@andrew.cmu.edu> Andrew Morgan wrote: > On Wed, 11 Mar 2009, Wil Cooley wrote: > >> Somewhere I read (or thought I read) that it was important to ensure >> that a user was not logged in when his mailbox was transferred between >> backends in a Murder setup, otherwise there was a risk of mailbox >> corruption. Is this true or have I been reading the tabloids again? >> >> Context: I have several shiny new backends with 2.3 that I have ready to >> replace our old 2.2 backends. We have something like 70,000 mailboxes to >> move, which will obviously be something done in nightly batches over a >> period of weeks. >> >> Our current obstacle is having to ensure that the mailbox is not open, >> because that requires: notifying the user that his e-mail will be >> unavailable for a period of time, locking the user's account (which >> locks him out of everything else unless we work up some Cyrus-specific >> solution, like some PAM magic), checking all the front-ends to verify >> that he's not logged in (this is iffy, because the proc/ files >> are all we've got and some of those are left-overs from crashes and lack >> of housekeeping), doing the transfer, and then undoing all of this. >> >> Alternately, if there is only a small risk of mailbox corruption, it may >> be better to just do the transfers late at night and accept having to do >> a handful of mailbox reconstructs. >> >> Is this what other people have done? > > When we went from a standalone server to a Murder setup, we moved half of > our mailboxes from cyrus-be1 to cyrus-be2. We did them in batches > overnight, without any special care taken to prevent users from accessing > them. I'm not aware of any problems caused by this. This was back in > Cyrus 2.1 or 2.2. > > So no promises, but it worked fine for us. You should not need to do anything to prevent client access to a mailbox when it's being moved and there should never be any resulting corruption. Cyrus locks the mailbox while it's being moved. I will mention, however, that some clients get confused if they're connected while a mailbox is being moved and I've never actually figured out why. The confusion is usually a matter of a user's mailbox suddenly appearing to be empty which is resolved by having them restart their mail client. The most extreme confusion I've seen so far was with some version of Outlook where the user actually had to delete and recreate their account configuration to convince Outlook that the mailbox was not empty. Thanks, Dave -- Dave McMurtrie, SPE Email Systems Team Leader Carnegie Mellon University, Computing Services From carson at taltos.org Thu Mar 12 14:13:08 2009 From: carson at taltos.org (Carson Gaspar) Date: Thu, 12 Mar 2009 11:13:08 -0700 Subject: Calling all regressions In-Reply-To: <20090312114234.GA20450@brong.net> References: <1236675674.12318.1304575661@webmail.messagingengine.com> <20090310120713.140117sn46lwgg69@webmail.uni-tuebingen.de> <1236688701.5616.17.camel@linux-m3mt> <20090310135130.156225x0tn2katzm@webmail.uni-tuebingen.de> <26722814-9B2B-4203-AC1A-0554A9D2C190@umich.edu> <20090311042211.GD8560@brong.net> <0754101f074cc5778d94e72233198371.squirrel@webmail.bi.corp.invoca.ch> <20090312093529.GA10774@brong.net> <49B8DBF7.5040403@taltos.org> <20090312114234.GA20450@brong.net> Message-ID: <49B950B4.7080301@taltos.org> Bron Gondwana wrote: > On Thu, Mar 12, 2009 at 02:55:03AM -0700, Carson Gaspar wrote: >> Bron Gondwana wrote: ... >>> * cyr_conf -q - only show variables that are different than default ... >> Please include one of the most useful options that postconf has >> (postconf -n) - emit only those config key/value pairs that are not set ... > Yeah, I called that -q. If we make it -n then we'll have to make the > named process config a different character. The spec isn't nailed down > yet, I haven't even written anything! I clearly suffered a fault in my "reading" subsystem... my apologies, and consider this a +1 for the whole idea. Which letter does what I really don't care about. -- Carson From mhlavink at redhat.com Thu Mar 12 14:38:38 2009 From: mhlavink at redhat.com (Michal Hlavinka) Date: Thu, 12 Mar 2009 19:38:38 +0100 Subject: ctl_cyrusdb questions, looking for answers Message-ID: <200903121938.38709.mhlavink@redhat.com> Hi, does anybody know, how ctl_cyrusdb works (how it should be used)? I've seen ctl_cyrusdb's man pages, but there is just few notes and nothing more. I assume ctl_cyrusdb is used (in /etc/cyrus.conf) to generate "checkpoints" of the Cyrus databases, presumably those in /var/lib/imap. The information that man does not provide is 1) Listing of database files "checkpointed" 2) Description of checkpoint format a) "ls /var/lib/imap" suggest two sets of checkpoint database are retained. b) Description of checkpoint configuration options, if any, to control checkpoint sets 3) Checkpointed image backup requirements 4) Appropriate cross-references to utilities used to determine database corrution requiring recovery procedure 5) Appropriate recovery procedure when invocation with "-r" vs. "-r -x' is required. It'll be really appreciated if someone could explain at least some of these questions. Thanks Regards, Michal Hlavinka From brong at fastmail.fm Thu Mar 12 16:53:02 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 13 Mar 2009 07:53:02 +1100 Subject: Calling all regressions In-Reply-To: <4f861a9b6f180919f2d881d80dbf03ca.squirrel@webmail.bi.corp.invoca.ch> References: <1236675674.12318.1304575661@webmail.messagingengine.com> <20090310120713.140117sn46lwgg69@webmail.uni-tuebingen.de> <1236688701.5616.17.camel@linux-m3mt> <20090310135130.156225x0tn2katzm@webmail.uni-tuebingen.de> <26722814-9B2B-4203-AC1A-0554A9D2C190@umich.edu> <20090311042211.GD8560@brong.net> <0754101f074cc5778d94e72233198371.squirrel@webmail.bi.corp.invoca.ch> <20090312093529.GA10774@brong.net> <4f861a9b6f180919f2d881d80dbf03ca.squirrel@webmail.bi.corp.invoca.ch> Message-ID: <20090312205302.GB7655@brong.net> On Thu, Mar 12, 2009 at 02:04:17PM +0100, Simon Matter wrote: > > Yep, definitely. Store them all, and refuse to start unless: > > cyr_conf -C $oldfile is the same output as cyr_conf with the > > -C option passed to 'master'. Spit out an error message > > describing what's different and how to fix it. > > Hi Bron, > > cyr_conf sounds good :) > > For the "refuse to start", do you think it should be integrated into > master or being kept in the init script? Init scripts. All this "config manglement" is supporting infrastructure, not core operations. The init script would do all the initial config setup into $configdir, and launch master with that config file and a cyrus.conf as well. > What would be very cool is a detect feature which detects things like > database config from the configdir. That way it could also override > configuration settings from the config files on startup and log about it. > It sounds a bit dangerous, what do you think? Quite doable. It could probably just use "file" to figure out what sort of database file was in use. Skiplist is certainly magic-checkable (the first 20 bytes contain all the details), and I imagine BDB is similar. I'm _not_ so sure about berkeley-hash vs berkeley-btree, and I am pretty certain it can't auto-detect berkeley-nosync or skiplist-unsafe! I'd rather have it detect the wrong type and abort than detect and log on startup. This is a daemon that deals with files that people care about, after all! Bron. From brong at fastmail.fm Thu Mar 12 17:39:33 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 13 Mar 2009 08:39:33 +1100 Subject: unexpunge segfaults with -l on some mailboxes In-Reply-To: <49B8FB4F.9050806@ednet.ns.ca> References: <49B7B296.6000305@ednet.ns.ca> <49B7DB6C.9080602@ednet.ns.ca> <20090312000706.GA7208@brong.net> <49B8FB4F.9050806@ednet.ns.ca> Message-ID: <20090312213933.GC7655@brong.net> On Thu, Mar 12, 2009 at 09:08:47AM -0300, Patrick Boutilier wrote: > Bron Gondwana wrote: >> On Wed, Mar 11, 2009 at 12:40:28PM -0300, Patrick Boutilier wrote: >>> Darn. This is caused by mailbox corruption again. My script to detect >>> corruption wasn't working properly. Any idea how we can track down >>> what is causing the corruption? >> >> Yeah, I was going to say. >> >> It's corrupted cache file offset pointers. Assuming you upgraded >> this from an earlier version of Cyrus at some point, > > I believe we started at 1.5.19 and then migrated up through the 2.0 , > 2.2 series. Our last major update was from 2.2.13 to 2.3.11. Currently > on 2.3.13 > > cyrus.expunge >> and cyrus.index files had issues maintaining cache pointer >> consistency. You pretty much have to reconstruct all your mailboxes >> to guarantee consistency I think. > > This latest list of corrupted mailboxes includes many that were > correupted a couple of weeks ago (including mine) so this is the second > time they have been reconstructed. Not so good. Crap. And this is on 2.3.13? I don't see any changes touching that code in the post 2.3.13 changelogs... >> I'm tempted to protect the code from crashing though... we don't >> use unexpunge at FastMail, which is probably why I haven't already >> done so. >> >> Something like the attached should do it. I'll test it more >> completely and commit it to CVS for 2.3.14 (since Ken hasn't >> cut a release candidate yet!) > > Thanks for the patch. Hey, don't use it though - it doesn't even compile! It was a first draft. I'll give you a real patch soon... been working on doing it _properly_ :) > Would ipurge be causing the corruption? We currently purge e-mails older > than 31 days on a weekly basis. I will turn that off for a bit (since > disk space is not as much of an issue as it used to be) and see if the > corruption returns. Oooh... maybe. I don't use ipurge. Let me know what you find with turning it off. I've never even _looked_ at that code. Bron. From brong at fastmail.fm Thu Mar 12 17:47:26 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 13 Mar 2009 08:47:26 +1100 Subject: cyrus events off by an hour In-Reply-To: References: <1236865780.5675.18.camel@linux-m3mt> Message-ID: <20090312214726.GD7655@brong.net> On Thu, Mar 12, 2009 at 09:27:27AM -0700, Andrew Morgan wrote: > On Thu, 12 Mar 2009, Adam Tauno Williams wrote: > > > I have - > > delprune cmd="cyr_expire -E 3 -D 120 -X 120" at=0400 > > - so cyr_expire should run at 4:00, right? > > > > But it runs at 5:00. The time on the server is correct. Could daylight > > savings time change effect this? > > Yes, all of your "at=xyz" events will be off an hour until you restart > cyrus. I found this bug a long time ago and basically ignored it. :) Annoying! Doesn't look too hard to fix thankfully. And it's only once per day per service :) I'll look into that... Bron ( have you considered raising a bugzilla? ) From simon.matter at invoca.ch Thu Mar 12 18:18:07 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Thu, 12 Mar 2009 23:18:07 +0100 (CET) Subject: Calling all regressions In-Reply-To: <20090312205302.GB7655@brong.net> References: <1236675674.12318.1304575661@webmail.messagingengine.com> <20090310120713.140117sn46lwgg69@webmail.uni-tuebingen.de> <1236688701.5616.17.camel@linux-m3mt> <20090310135130.156225x0tn2katzm@webmail.uni-tuebingen.de> <26722814-9B2B-4203-AC1A-0554A9D2C190@umich.edu> <20090311042211.GD8560@brong.net> <0754101f074cc5778d94e72233198371.squirrel@webmail.bi.corp.invoca.ch> <20090312093529.GA10774@brong.net> <4f861a9b6f180919f2d881d80dbf03ca.squirrel@webmail.bi.corp.invoca.ch> <20090312205302.GB7655@brong.net> Message-ID: <88eb6870059a6118737f83c66b852678.squirrel@webmail.bi.corp.invoca.ch> > On Thu, Mar 12, 2009 at 02:04:17PM +0100, Simon Matter wrote: >> > Yep, definitely. Store them all, and refuse to start unless: >> > cyr_conf -C $oldfile is the same output as cyr_conf with the >> > -C option passed to 'master'. Spit out an error message >> > describing what's different and how to fix it. >> >> Hi Bron, >> >> cyr_conf sounds good :) >> >> For the "refuse to start", do you think it should be integrated into >> master or being kept in the init script? > > Init scripts. All this "config manglement" is supporting infrastructure, > not core operations. The init script would do all the initial config > setup into $configdir, and launch master with that config file and a > cyrus.conf as well. Very good, that's how I prefer it. > >> What would be very cool is a detect feature which detects things like >> database config from the configdir. That way it could also override >> configuration settings from the config files on startup and log about >> it. >> It sounds a bit dangerous, what do you think? > > Quite doable. It could probably just use "file" to figure out what sort > of database file was in use. Skiplist is certainly magic-checkable (the > first 20 bytes contain all the details), and I imagine BDB is similar. > > I'm _not_ so sure about berkeley-hash vs berkeley-btree, and I am pretty > certain it can't auto-detect berkeley-nosync or skiplist-unsafe! > > I'd rather have it detect the wrong type and abort than detect and log > on startup. This is a daemon that deals with files that people care > about, after all! OK, my point is that I have to care about upgrades of a whole distribution or people who move mailspool to a newer box. To make the spool portable the init scripts convert all BDB databases to skiplist. At startup it first converts all databases back to the configured version and then fires master. That way you can move a mailspool which is using an old BDB version to a newer box without requiring support for the old BDB on the new box. It all started as a dirty hack but seems to work quite well for a long time now. With the help of cyr_conf I'll be able to improve the mechanism and make things a bit more clean. Thanks for looking into it, it will be of great help. Regards, Simon From boutilpj at ednet.ns.ca Thu Mar 12 18:48:05 2009 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Thu, 12 Mar 2009 19:48:05 -0300 Subject: unexpunge segfaults with -l on some mailboxes In-Reply-To: <20090312213933.GC7655@brong.net> References: <49B7B296.6000305@ednet.ns.ca> <49B7DB6C.9080602@ednet.ns.ca> <20090312000706.GA7208@brong.net> <49B8FB4F.9050806@ednet.ns.ca> <20090312213933.GC7655@brong.net> Message-ID: <10173_1236898088_n2CMm7nQ017919_49B99125.9020508@ednet.ns.ca> Bron Gondwana wrote: > Not so good. Crap. And this is on 2.3.13? I don't see any changes > touching that code in the post 2.3.13 changelogs... Correct. We have been running 2.3.13 pretty much since it was released, 2.3.12 before that, and 2.3.11 before that. > >>> I'm tempted to protect the code from crashing though... we don't >>> use unexpunge at FastMail, which is probably why I haven't already >>> done so. >>> >>> Something like the attached should do it. I'll test it more >>> completely and commit it to CVS for 2.3.14 (since Ken hasn't >>> cut a release candidate yet!) >> Thanks for the patch. > > Hey, don't use it though - it doesn't even compile! It was a first > draft. I'll give you a real patch soon... been working on doing it > _properly_ :) Ok. :-) > >> Would ipurge be causing the corruption? We currently purge e-mails older >> than 31 days on a weekly basis. I will turn that off for a bit (since >> disk space is not as much of an issue as it used to be) and see if the >> corruption returns. > > Oooh... maybe. I don't use ipurge. Let me know what you find with > turning it off. I've never even _looked_ at that code. > The more I think of it, the more I believe that ipurge will be the source of the problem. When I was manually checking for corruption (and reconstructing the mailboxes that had problems) there would be no corruption for 6 days. On the 7th day (always Sunday morning if I recall correctly) corruption would reappear in many mailboxes. As it turns out, our weekly ipurge ran on Saturday morning. This will be real easy to test though. I will just run ipurge on a subfolder of my mailbox and see if it corrupts it. :-) > Bron. From brong at fastmail.fm Thu Mar 12 18:24:54 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 13 Mar 2009 09:24:54 +1100 Subject: cyrus events off by an hour In-Reply-To: References: <1236865780.5675.18.camel@linux-m3mt> Message-ID: <20090312222454.GA8431@brong.net> On Thu, Mar 12, 2009 at 09:27:27AM -0700, Andrew Morgan wrote: > On Thu, 12 Mar 2009, Adam Tauno Williams wrote: > > > I have - > > delprune cmd="cyr_expire -E 3 -D 120 -X 120" at=0400 > > - so cyr_expire should run at 4:00, right? > > > > But it runs at 5:00. The time on the server is correct. Could daylight > > savings time change effect this? > > Yes, all of your "at=xyz" events will be off an hour until you restart > cyrus. I found this bug a long time ago and basically ignored it. :) Something like this should fix it. I'll test it and commit... Bron. -------------- next part -------------- A non-text attachment was scrubbed... Name: runat.diff Type: text/x-diff Size: 1502 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090313/ee988d1d/attachment.bin From awilliam at whitemice.org Thu Mar 12 20:25:22 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Thu, 12 Mar 2009 20:25:22 -0400 Subject: ctl_cyrusdb questions, looking for answers In-Reply-To: <200903121938.38709.mhlavink@redhat.com> References: <200903121938.38709.mhlavink@redhat.com> Message-ID: <1236903922.9159.5.camel@linux-m3mt> > does anybody know, how ctl_cyrusdb works (how it should be used)? I've never felt very compelled to mess with the default: checkpoint cmd="ctl_cyrusdb -c" period=30 > I've > seen ctl_cyrusdb's man pages, but there is just few notes and nothing more. > I assume ctl_cyrusdb is used (in /etc/cyrus.conf) to generate > "checkpoints" of the Cyrus databases, presumably those in /var/lib/imap. Yep. > The information that man does not provide is > 1) Listing of database files "checkpointed" I've always assumed all-of-them > 2) Description of checkpoint format > a) "ls /var/lib/imap" suggest two sets of checkpoint database are > retained. ? Not following you here. It checkpoints the databases, not certain what "two sets" of what your seeing. > b) Description of checkpoint configuration options, if any, to control > checkpoint sets > 3) Checkpointed image backup requirements I've always been very impressed at Cyrus' durability. I think you could restore a Cyrus database from a bit of rusty barbed wire. > 4) Appropriate cross-references to utilities used to determine database > corrution requiring recovery procedure > 5) Appropriate recovery procedure when invocation with "-r" vs. "-r -x' > is required. Cyrus runs a -r when it starts to verify the database; if that fails it abends with log messages. Most manual recovery/rebuilding is done using the reconstruct command. > It'll be really appreciated if someone could explain at least some of > these questions. Thanks -- OpenGroupware developer: awilliam at whitemice.org OpenGroupare & Cyrus IMAPd documenation @ From simon.matter at invoca.ch Fri Mar 13 04:00:33 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Fri, 13 Mar 2009 09:00:33 +0100 (CET) Subject: ctl_cyrusdb questions, looking for answers In-Reply-To: <200903121938.38709.mhlavink@redhat.com> References: <200903121938.38709.mhlavink@redhat.com> Message-ID: <30d2d300a56d2bf0d7f172b66b577aec.squirrel@webmail.bi.corp.invoca.ch> > Hi, > > does anybody know, how ctl_cyrusdb works (how it should be used)? I've > seen ctl_cyrusdb's man pages, but there is just few notes and nothing > more. > > I assume ctl_cyrusdb is used (in /etc/cyrus.conf) to generate > "checkpoints" of the Cyrus databases, presumably those in /var/lib/imap. > > The information that man does not provide is Hi Michal, At this time it seems to better read the source in imap/ctl_cyrusdb.c. Manpage writers are warmly welcomed :) > > 1) Listing of database files "checkpointed" > 2) Description of checkpoint format > a) "ls /var/lib/imap" suggest two sets of checkpoint database are > retained. I'm not really sure here. Cyrus-imapd checkpoints databases and archives some of them (mboxlist_db, quota_db, annotation_db). Archives are put and rotated into the db.backup[1,2] dirs. > b) Description of checkpoint configuration options, if any, to control > checkpoint sets > 3) Checkpointed image backup requirements Not exactly sure what you mean here. I think you only need the database in db.backup? and of course clean the environment before restoring such a db. > 4) Appropriate cross-references to utilities used to determine database > corrution requiring recovery procedure I'm not sure there really exists such a thing despite the list archives. > 5) Appropriate recovery procedure when invocation with "-r" vs. "-r -x' > is required. I don't remember to have used -x in the last 10 years using cyrus-imapd. I really don't know what the cleanup part means. Is it cleaning up unused space in a db file or does it mean handling of open transactions? Regards, Simon From Leena.Heino at uta.fi Fri Mar 13 04:44:44 2009 From: Leena.Heino at uta.fi (Leena Heino) Date: Fri, 13 Mar 2009 10:44:44 +0200 (FLE Standard Time) Subject: Skiplist problems, locking problems with seen files Message-ID: Some of our users have reported that they have been locked out of their accounts. When I examined one such occurence it seemed that one of the imapd processes had acquired a write lock to user's seen file and doing absolutely nothing to that seen file just holding the writelock. When this process was killed user was able to access his mail again. Now this problem seems to be spreading. Imapd processes just hanging on to their writelocks to seen files doing nothing. What could cause such a problem? Current version is: version : v2.3.12p2 2008/04/24 16:09:27 os : SunOS os-version : 5.10 environment: Built w/Cyrus SASL 2.1.22 Built w/Sleepycat Software: Berkeley DB 4.4.20: (January 10, 2006) lock = fcntl nonblock = fcntl -- Leena Heino University of Tampere / Computer Centre ( liinu at uta.fi ) ( http://www.uta.fi/laitokset/tkk ) From Leena.Heino at uta.fi Fri Mar 13 05:03:16 2009 From: Leena.Heino at uta.fi (Leena Heino) Date: Fri, 13 Mar 2009 11:03:16 +0200 (FLE Standard Time) Subject: Update installation to new features: metadata partition, delayed delete Message-ID: Meta data partition: If I want to update our cyrus installation to use metadata partition, then how is this done. Do I manually create the similar directory structure to the new metadata partition as I have now in the conventional mixed data partition? Do I manually copy over the cyrus.* metadata files from the conventional mixed data partition to metadata partition? After those are done, then just change the configuration settings in the imapd.conf to use the new the meta data partition? Delayed delete: If I want to use this feature do I have to do more than just change the delete_mode configuration setting to delayed and adjust cyr_expire event in the cyrus.conf accordingly? -- Leena Heino University of Tampere / Computer Centre ( liinu at uta.fi ) ( http://www.uta.fi/laitokset/tkk ) From boutilpj at ednet.ns.ca Fri Mar 13 09:18:53 2009 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Fri, 13 Mar 2009 10:18:53 -0300 Subject: unexpunge segfaults with -l on some mailboxes In-Reply-To: <10173_1236898088_n2CMm7nQ017919_49B99125.9020508@ednet.ns.ca> References: <49B7B296.6000305@ednet.ns.ca> <49B7DB6C.9080602@ednet.ns.ca> <20090312000706.GA7208@brong.net> <49B8FB4F.9050806@ednet.ns.ca> <20090312213933.GC7655@brong.net> <10173_1236898088_n2CMm7nQ017919_49B99125.9020508@ednet.ns.ca> Message-ID: <49BA5D3D.4080509@ednet.ns.ca> Patrick Boutilier wrote: > Bron Gondwana wrote: > > >> Not so good. Crap. And this is on 2.3.13? I don't see any changes >> touching that code in the post 2.3.13 changelogs... > > > Correct. We have been running 2.3.13 pretty much since it was released, > 2.3.12 before that, and 2.3.11 before that. > >>>> I'm tempted to protect the code from crashing though... we don't >>>> use unexpunge at FastMail, which is probably why I haven't already >>>> done so. >>>> >>>> Something like the attached should do it. I'll test it more >>>> completely and commit it to CVS for 2.3.14 (since Ken hasn't >>>> cut a release candidate yet!) >>> Thanks for the patch. >> Hey, don't use it though - it doesn't even compile! It was a first >> draft. I'll give you a real patch soon... been working on doing it >> _properly_ :) > > Ok. :-) > >>> Would ipurge be causing the corruption? We currently purge e-mails older >>> than 31 days on a weekly basis. I will turn that off for a bit (since >>> disk space is not as much of an issue as it used to be) and see if the >>> corruption returns. >> Oooh... maybe. I don't use ipurge. Let me know what you find with >> turning it off. I've never even _looked_ at that code. >> > > The more I think of it, the more I believe that ipurge will be the > source of the problem. When I was manually checking for corruption (and > reconstructing the mailboxes that had problems) there would be no > corruption for 6 days. On the 7th day (always Sunday morning if I recall > correctly) corruption would reappear in many mailboxes. As it turns out, > our weekly ipurge ran on Saturday morning. > > This will be real easy to test though. I will just run ipurge on a > subfolder of my mailbox and see if it corrupts it. :-) I can duplicate the corruption by following steps similar to this report: http://archives.devshed.com/forums/networking-100/cyr-expire-sigsegv-2146878.html Current directory here is /var/imap/metadata/G/user/boutilpj/Testmail Clean mailbox: [cyrus at student2 Testmail]$ ll total 60 -rw------- 1 cyrus cyrus 49612 Mar 13 10:03 cyrus.cache -rw------- 1 cyrus cyrus 189 Mar 13 10:03 cyrus.header -rw------- 1 cyrus cyrus 3792 Mar 13 10:03 cyrus.index Quick way to show number of messages in the folder: [cyrus at student2 Testmail]$ cat cyrus.cache |grep -a Prec|wc -l 42 Delete 11 messages through IMAP client: [cyrus at student2 Testmail]$ ll total 64 -rw------- 1 cyrus cyrus 49612 Mar 13 10:03 cyrus.cache -rw------- 1 cyrus cyrus 1064 Mar 13 10:05 cyrus.expunge -rw------- 1 cyrus cyrus 189 Mar 13 10:03 cyrus.header -rw------- 1 cyrus cyrus 2824 Mar 13 10:05 cyrus.index Run ipurge with a number high enough not to delete anything. I purge only sees 31 messages and removes them from cyrus.cache: [cyrus at student2 Testmail]$ /usr/local/cyrus/bin/ipurge -f -d 2333 user.boutilpj.Testmail Working on user.boutilpj.Testmail... total messages 31 total bytes 539400 Deleted messages 0 Deleted bytes 0 Remaining messages 31 Remaining bytes 539400 [cyrus at student2 Testmail]$ ll total 48 -rw------- 1 cyrus cyrus 36652 Mar 13 10:08 cyrus.cache -rw------- 1 cyrus cyrus 1064 Mar 13 10:05 cyrus.expunge -rw------- 1 cyrus cyrus 189 Mar 13 10:03 cyrus.header -rw------- 1 cyrus cyrus 2824 Mar 13 10:08 cyrus.index [cyrus at student2 Testmail]$ cat cyrus.cache |grep -a Prec|wc -l 31 Run cyr_expire and corruption occurs: [cyrus at student2 Testmail]$ /usr/local/cyrus/bin/cyr_expire -E 333 -X 7 -p user.boutilpj.Testmail -v Expunging deleted messages in mailboxes older than 7 days Expunged 0 out of 33 messages from 1 mailboxes [cyrus at student2 Testmail]$ ll total 96 -rw------- 1 cyrus cyrus 36652 Mar 13 10:08 cyrus.cache -rw------- 1 cyrus cyrus 37852 Mar 13 10:15 cyrus.cache.NEW -rw------- 1 cyrus cyrus 1064 Mar 13 10:05 cyrus.expunge -rw------- 1 cyrus cyrus 272 Mar 13 10:15 cyrus.expunge.NEW -rw------- 1 cyrus cyrus 189 Mar 13 10:03 cyrus.header -rw------- 1 cyrus cyrus 2824 Mar 13 10:08 cyrus.index -rw------- 1 cyrus cyrus 2824 Mar 13 10:15 cyrus.index.NEW Mar 13 10:15:40 student2 cyr_expire[8725]: IOERROR: reading cache record for user.boutilpj.Testmail: item 1 has bogus offset 576450680 of 36652 for 2/11; mailbox needs a reconstruct > >> Bron. > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -------------- next part -------------- A non-text attachment was scrubbed... Name: boutilpj.vcf Type: text/x-vcard Size: 286 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090313/13e7c468/attachment.vcf From dwhite at olp.net Fri Mar 13 12:53:25 2009 From: dwhite at olp.net (Dan White) Date: Fri, 13 Mar 2009 11:53:25 -0500 Subject: Update installation to new features: metadata partition, delayed delete In-Reply-To: References: Message-ID: <49BA8F85.4020305@olp.net> Leena Heino wrote: > Meta data partition: > If I want to update our cyrus installation to use metadata partition, > then how is this done. > > Do I manually create the similar directory structure to the new metadata > partition as I have now in the conventional mixed data partition? > > Do I manually copy over the cyrus.* metadata files from the conventional > mixed data partition to metadata partition? > > After those are done, then just change the configuration > settings in the imapd.conf to use the new the meta data partition? > > Delayed delete: > If I want to use this feature do I have to do more than just change the > delete_mode configuration setting to delayed and adjust cyr_expire event > in the cyrus.conf accordingly? > > Leena, With regards to the metadata partition, there is documentation located in the doc/install-upgrade.html file located in the source tarball release. - Dan From Leena.Heino at uta.fi Fri Mar 13 14:15:37 2009 From: Leena.Heino at uta.fi (Leena Heino) Date: Fri, 13 Mar 2009 20:15:37 +0200 (FLE Standard Time) Subject: Update installation to new features: metadata partition, delayed delete In-Reply-To: <49BA8F85.4020305@olp.net> References: <49BA8F85.4020305@olp.net> Message-ID: On Fri, 13 Mar 2009, Dan White wrote: > With regards to the metadata partition, there is documentation located in the > doc/install-upgrade.html file located in the source tarball release. Thanks Dan. You are absolutely right and the metadata upgrade process was mentioned in the doc/install-upgrade.html file. Silly me, for not noticing it. I thought I had read the document, but obviously I had missed that part. One reason could be that it was mentioned in the "Upgrading from 2.2.x or earlier" section, and I had just read the sections about the changes in the 2.3.x series. Though, I still could not find any information about delayed delete. -- Leena Heino University of Tampere / Computer Centre ( liinu at uta.fi ) ( http://www.uta.fi/laitokset/tkk ) From morgan at orst.edu Fri Mar 13 15:34:42 2009 From: morgan at orst.edu (Andrew Morgan) Date: Fri, 13 Mar 2009 12:34:42 -0700 (PDT) Subject: Update installation to new features: metadata partition, delayed delete In-Reply-To: References: <49BA8F85.4020305@olp.net> Message-ID: On Fri, 13 Mar 2009, Leena Heino wrote: > Though, I still could not find any information about delayed delete. You were correct. Just update the config file to enable delayed delete, then modify your cyr_expire event in cyrus.conf. Andy From wes at umich.edu Fri Mar 13 22:18:46 2009 From: wes at umich.edu (Wesley Craig) Date: Fri, 13 Mar 2009 22:18:46 -0400 Subject: Is user lock-out required for backend xfers? In-Reply-To: <49B9404C.5050103@andrew.cmu.edu> References: <1236832952.8538.13.camel@spartacus.nakedape.cc> <49B9404C.5050103@andrew.cmu.edu> Message-ID: On 12 Mar 2009, at 13:03, Dave McMurtrie wrote: > You should not need to do anything to prevent client access to a > mailbox > when it's being moved and there should never be any resulting > corruption. Cyrus locks the mailbox while it's being moved. And any result different from the above should be considered a bug :) > I will mention, however, that some clients get confused if they're > connected while a mailbox is being moved and I've never actually > figured > out why. The confusion is usually a matter of a user's mailbox > suddenly > appearing to be empty which is resolved by having them restart their > mail client. The most extreme confusion I've seen so far was with > some > version of Outlook where the user actually had to delete and recreate > their account configuration to convince Outlook that the mailbox > was not > empty. I'm pretty sure this is because neither the frontends nor the clients are able to intelligently handle the kinds of transient errors that are returned. Frontend improvements might be possible. Client improvements are Sisyphean. :wes From damm at yazzy.org Sat Mar 14 01:15:16 2009 From: damm at yazzy.org (Scott M. Likens) Date: Fri, 13 Mar 2009 22:15:16 -0700 Subject: Intermittent mailbox not found In-Reply-To: References: <20090309215242.GA6858@brong.net> <1236650964.11240.12.camel@linux-m3mt> <20090310030016.GB15110@brong.net> Message-ID: <15A616AB-E2CE-4A83-8FD1-A356918C1B0B@yazzy.org> Hi, Not to interject through the pain of this. This issue only happens when multiple users on the same domain receive an email? If I'm wrong, never mind. On Mar 10, 2009, at 4:45 AM, Antony Gelberg wrote: > On 2009-03-10, Bron Gondwana wrote: >> On Mon, Mar 09, 2009 at 10:09:24PM -0400, Adam Tauno Williams wrote: >>> On Tue, 2009-03-10 at 08:52 +1100, Bron Gondwana wrote: >>>> On Mon, Mar 09, 2009 at 07:52:04PM +0000, Antony Gelberg wrote: >>>>> Mar 9 14:29:07 captain cyrus/lmtpunix[20645]: >>>>> verify_user(user.sirius-john) >>>>> failed: Mailbox does not exits >>>> Where on earth is that error message from? I don't see "not exits" >>>> anywhere in the current Cyrus codebase... >>> >>> I'm thinking it is coming from Postfix's LMTP delivery agent. I >>> wonder >>> if some other error, like a connect error, occurs and Postfix >>> falls back >>> to mailbox-does-not-exist. >> >> This old thread might help you: >> >> http://markmail.org/message/nqh3rrvhkt4ccs2r >> >> Bron ( that's just the first one I found on google ) >> > > Thanks - I had searched the web with little concrete advice coming up. > > I only found advice in the above thread and others, to use lmtp not > the cyrus > transport. But postfix is already doing so: > virtual_transport = lmtp:unix:/var/run/cyrus/socket/lmtp > > I found it interesting that the OP in that thread said "Everything > works > besides sending emails to multiple addresses in the same domain. " > I think > the problem email went to multiple addresses in the same domain, but > I resent > one just now to the same addresses with no issue. I'll ask the > client if > every one of these bounces over the last few months fits with that > scenario. > > -- > Antony > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > > > !DSPAM:49b653a3215661804284693! > > From wcooley at nakedape.cc Sat Mar 14 20:59:38 2009 From: wcooley at nakedape.cc (Wil Cooley) Date: Sat, 14 Mar 2009 17:59:38 -0700 Subject: Is user lock-out required for backend xfers? In-Reply-To: References: <1236832952.8538.13.camel@spartacus.nakedape.cc> <49B9404C.5050103@andrew.cmu.edu> Message-ID: <49BC52FA.6010703@nakedape.cc> Thanks everyone; I'm not sure where I got that idea. Wil -- Wil Cooley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090314/8a453aaf/attachment.bin From atarallo at acm.org Mon Mar 16 08:42:00 2009 From: atarallo at acm.org (Andres Tarallo) Date: Mon, 16 Mar 2009 09:42:00 -0300 Subject: Bulk installation of sieve filters Message-ID: <3fdbc0030903160542u1194ad47k7cc3866ec4f1cc7e@mail.gmail.com> I have an installation with about 2000 mailboxes. Now we want to send all the mails tagged as SPAM (by spamassassin) to the spam folder. I'm trying to automate the process of doing that job. I know how to create in bulk the spam folder. But I can't find if it's possible to do such work with sieve. Thanks in advance. Apologize poor English Andr?s Tarallo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090316/d39f8554/attachment.html From reinaldoc at gmail.com Mon Mar 16 09:18:52 2009 From: reinaldoc at gmail.com (Reinaldo de Carvalho) Date: Mon, 16 Mar 2009 10:18:52 -0300 Subject: Bulk installation of sieve filters In-Reply-To: <3fdbc0030903160542u1194ad47k7cc3866ec4f1cc7e@mail.gmail.com> References: <3fdbc0030903160542u1194ad47k7cc3866ec4f1cc7e@mail.gmail.com> Message-ID: <4a5881460903160618o7037b0d9s1b83474481eb0517@mail.gmail.com> 2009/3/16 Andres Tarallo : > I have an installation with about 2000 mailboxes. Now we want to send all > the mails tagged as SPAM (by spamassassin) to the spam folder. > > I'm trying to automate the process of doing that job. I know how to create > in bulk the spam folder. But I can't find if it's possible to do such work > with sieve. > > Andr?s Tarallo > You can use Korreio (korreio.sf.net) or write your program with sievelib of python-cyrus package (python-cyrus.sf.net). -- Reinaldo de Carvalho http://korreio.sf.net http://python-cyrus.sf.net From Barry.R.RIbbeck at rice.edu Mon Mar 16 10:16:04 2009 From: Barry.R.RIbbeck at rice.edu (Barry R. Ribbeck) Date: Mon, 16 Mar 2009 09:16:04 -0500 Subject: unsubscribe Message-ID: <49BE5F24.3030802@rice.edu> -------------- next part -------------- A non-text attachment was scrubbed... Name: Barry_R_RIbbeck.vcf Type: text/x-vcard Size: 334 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090316/34486c47/attachment.vcf From choeger at open-xchange.com Mon Mar 16 12:22:26 2009 From: choeger at open-xchange.com (Carsten Hoeger) Date: Mon, 16 Mar 2009 17:22:26 +0100 Subject: Bulk installation of sieve filters In-Reply-To: <3fdbc0030903160542u1194ad47k7cc3866ec4f1cc7e@mail.gmail.com> References: <3fdbc0030903160542u1194ad47k7cc3866ec4f1cc7e@mail.gmail.com> Message-ID: <20090316162226.GQ3856@open-xchange.com> On Mon, Mar 16, Andres Tarallo wrote: > I have an installation with about 2000 mailboxes. Now we want to send all > the mails tagged as SPAM (by spamassassin) to the spam folder. > > I'm trying to automate the process of doing that job. I know how to create > in bulk the spam folder. But I can't find if it's possible to do such work > with sieve. > > Thanks in advance. Apologize poor English 1. write you sieve script and test/upload it for one mailbox. 2. copy the script sourcecode into the subdirectory /var/lib/sieve/[...] (depends on distribution which is in use) 3. run masssievec to compile a bunch of scrips into bytecode -- With best regards, Carsten Hoeger -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090316/324588b9/attachment.bin From mcnutt at buffalo.edu Mon Mar 16 13:52:32 2009 From: mcnutt at buffalo.edu (James M McNutt) Date: Mon, 16 Mar 2009 13:52:32 -0400 (EDT) Subject: Solaris10/ZFS In-Reply-To: Message-ID: We are currently running Solaris 9 with VxVM/VxFS and looking to move to Solaris 10 with ZFS. I was looking for some feedback from those using ZFS. what type of system? what type of storage? how large? compression? replication? problems? current configuration: 48,000 user accounts with 350,000 mailboxes spread across: 4 Sun V880 server attached to a SAN 3 spools per server (for a total of 12 spools) each spool is using about 250GB Thanks, -jim University at Buffalo mcnutt at buffalo.edu From daleg at elemental.org Mon Mar 16 14:01:16 2009 From: daleg at elemental.org (Dale Ghent) Date: Mon, 16 Mar 2009 14:01:16 -0400 Subject: Solaris10/ZFS In-Reply-To: References: Message-ID: <839684B4-A74E-4DEC-8B7A-7091363C37E5@elemental.org> On Mar 16, 2009, at 1:52 PM, James M McNutt wrote: > We are currently running Solaris 9 with VxVM/VxFS > and looking to move to Solaris 10 with ZFS. > current configuration: > 48,000 user accounts with 350,000 mailboxes spread across: > 4 Sun V880 server attached to a SAN > 3 spools per server (for a total of 12 spools) > each spool is using about 250GB Would you be looking to reuse your current SAN-based storage with this? /dale From Leena.Heino at uta.fi Mon Mar 16 14:02:08 2009 From: Leena.Heino at uta.fi (Leena Heino) Date: Mon, 16 Mar 2009 20:02:08 +0200 (FLE Standard Time) Subject: Solaris10/ZFS In-Reply-To: References: Message-ID: On Mon, 16 Mar 2009, James M McNutt wrote: > We are currently running Solaris 9 with VxVM/VxFS > and looking to move to Solaris 10 with ZFS. > > I was looking for some feedback from those using ZFS. > > what type of system? 20000 accounts, lots of mailboxes > what type of storage? EMC Clariion CX3 SAN > how large? about 3 TB > compression? none > replication? none > problems? None so far. Or we just haven't noticed them, yet. -- Leena Heino University of Tampere / Computer Centre ( liinu at uta.fi ) ( http://www.uta.fi/laitokset/tkk ) From mcnutt at buffalo.edu Mon Mar 16 14:22:38 2009 From: mcnutt at buffalo.edu (James M McNutt) Date: Mon, 16 Mar 2009 14:22:38 -0400 (EDT) Subject: Solaris10/ZFS In-Reply-To: <839684B4-A74E-4DEC-8B7A-7091363C37E5@elemental.org> Message-ID: move from Sun StorEdge 9960 to Sun StorEdge 9990 On Mon, 16 Mar 2009, Dale Ghent wrote: > On Mar 16, 2009, at 1:52 PM, James M McNutt wrote: > > > We are currently running Solaris 9 with VxVM/VxFS > > and looking to move to Solaris 10 with ZFS. > > > > > current configuration: > > 48,000 user accounts with 350,000 mailboxes spread across: > > 4 Sun V880 server attached to a SAN > > 3 spools per server (for a total of 12 spools) > > each spool is using about 250GB > > Would you be looking to reuse your current SAN-based storage with this? > > /dale > From Pascal.Gienger at uni-konstanz.de Mon Mar 16 14:49:26 2009 From: Pascal.Gienger at uni-konstanz.de (Pascal Gienger) Date: Mon, 16 Mar 2009 19:49:26 +0100 Subject: Solaris10/ZFS In-Reply-To: References: Message-ID: <49BE9F36.80207@uni-konstanz.de> James M McNutt schrieb: > We are currently running Solaris 9 with VxVM/VxFS > and looking to move to Solaris 10 with ZFS. > > I was looking for some feedback from those using ZFS. > > what type of system? 2x X4200 with 32 GB RAM each > what type of storage? 2x FiberChannel Storage Systems in two different locations, mirrored via zfs. > how large? 2x8 TB mirror > compression? gzip > replication? zfs mirror > problems? at the beginning (2006) many, nowadays X patches and kernels after, none left. Pascal From Darin.Perusich at cognigencorp.com Mon Mar 16 14:56:14 2009 From: Darin.Perusich at cognigencorp.com (Darin Perusich) Date: Mon, 16 Mar 2009 14:56:14 -0400 Subject: Solaris10/ZFS In-Reply-To: References: Message-ID: <49BEA0CE.5060900@cognigencorp.com> You'll likely need to disable cache flushes or you're going to see poor performance to any mailspools on the SAN because of the array NVRAM. You can either disable this on the SAN or by setting a kernel parameter. See http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Cache_Flushes for details. James M McNutt wrote: > move from Sun StorEdge 9960 to Sun StorEdge 9990 > > On Mon, 16 Mar 2009, Dale Ghent wrote: > >> On Mar 16, 2009, at 1:52 PM, James M McNutt wrote: >> >>> We are currently running Solaris 9 with VxVM/VxFS >>> and looking to move to Solaris 10 with ZFS. >> >> >>> current configuration: >>> 48,000 user accounts with 350,000 mailboxes spread across: >>> 4 Sun V880 server attached to a SAN >>> 3 spools per server (for a total of 12 spools) >>> each spool is using about 250GB >> Would you be looking to reuse your current SAN-based storage with this? >> >> /dale >> -- Darin Perusich Unix Systems Administrator Cognigen Corporation 395 Youngs Rd. Williamsville, NY 14221 Phone: 716-633-3463 Email: darinper at cognigencorp.com From Leena.Heino at uta.fi Tue Mar 17 07:26:44 2009 From: Leena.Heino at uta.fi (Leena Heino) Date: Tue, 17 Mar 2009 13:26:44 +0200 (EET) Subject: Delayed delete, restoring deleted mailboxes Message-ID: What is the recommended process to restore deleted messages or mailboxes when delayed delete is in use? -- Leena Heino University of Tampere / Computer Centre ( liinu at uta.fi ) ( http://www.uta.fi/laitokset/tkk ) From awilliam at whitemice.org Tue Mar 17 07:57:04 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Tue, 17 Mar 2009 07:57:04 -0400 Subject: Delayed delete, restoring deleted mailboxes In-Reply-To: References: Message-ID: <1237291024.13273.29.camel@linux-m3mt> On Tue, 2009-03-17 at 13:26 +0200, Leena Heino wrote: > What is the recommended process to restore deleted messages or mailboxes > when delayed delete is in use? Use the unexpunge command; the man page isn't bad, and I have examples and some notes in the Cyrus chapter of WMOGAG . -- OpenGroupware developer: awilliam at whitemice.org OpenGroupare & Cyrus IMAPd documenation @ From brennan at columbia.edu Tue Mar 17 09:05:43 2009 From: brennan at columbia.edu (Joseph Brennan) Date: Tue, 17 Mar 2009 09:05:43 -0400 Subject: Delayed delete, restoring deleted mailboxes In-Reply-To: <1237291024.13273.29.camel@linux-m3mt> References: <1237291024.13273.29.camel@linux-m3mt> Message-ID: <79C0D89C6881C5B9CEA97FA3@sodor.cc.columbia.edu> --On Tuesday, March 17, 2009 7:57 -0400 Adam Tauno Williams wrote: > On Tue, 2009-03-17 at 13:26 +0200, Leena Heino wrote: >> What is the recommended process to restore deleted messages or mailboxes >> when delayed delete is in use? > > Use the unexpunge command; the man page isn't bad, and I have examples > and some notes in the Cyrus chapter of WMOGAG > . Unexpunge for messages. For entire mailboxes, use rename. You can find the name by doing a "lm *string*" in cyradm. It will start "DELETED." and end with an extra string. Joseph Brennan Lead Email Systems Engineer Columbia University Information Technology From Leena.Heino at uta.fi Tue Mar 17 09:06:15 2009 From: Leena.Heino at uta.fi (Leena Heino) Date: Tue, 17 Mar 2009 15:06:15 +0200 (EET) Subject: Delayed delete, restoring deleted mailboxes In-Reply-To: <1237291024.13273.29.camel@linux-m3mt> References: <1237291024.13273.29.camel@linux-m3mt> Message-ID: On Tue, 17 Mar 2009, Adam Tauno Williams wrote: > On Tue, 2009-03-17 at 13:26 +0200, Leena Heino wrote: >> What is the recommended process to restore deleted messages or mailboxes >> when delayed delete is in use? > > Use the unexpunge command; the man page isn't bad, and I have examples > and some notes in the Cyrus chapter of WMOGAG > . The unexpunge command was the first thing I tried and I read it's man page. But what puzzles me is that it seems that the deleted mailboxes are not shown in unexpunge commands output. The mailbox meta files and mailbox's messages are shown in the spool dir under the DELETED prefix eg. in the directory: /cyrus_spool/u/DELETED/user/foo/sent-mail/HEXSTRING/ What is the correct procedure the restore this mailbox so that user is able see the messages and access it once again. -- Leena Heino University of Tampere / Computer Centre ( liinu at uta.fi ) ( http://www.uta.fi/laitokset/tkk ) From ktm at rice.edu Tue Mar 17 09:10:13 2009 From: ktm at rice.edu (Kenneth Marshall) Date: Tue, 17 Mar 2009 08:10:13 -0500 Subject: Delayed delete, restoring deleted mailboxes In-Reply-To: References: <1237291024.13273.29.camel@linux-m3mt> Message-ID: <20090317131013.GD2464@it.is.rice.edu> On Tue, Mar 17, 2009 at 03:06:15PM +0200, Leena Heino wrote: > On Tue, 17 Mar 2009, Adam Tauno Williams wrote: > > > On Tue, 2009-03-17 at 13:26 +0200, Leena Heino wrote: > >> What is the recommended process to restore deleted messages or mailboxes > >> when delayed delete is in use? > > > > Use the unexpunge command; the man page isn't bad, and I have examples > > and some notes in the Cyrus chapter of WMOGAG > > . > > The unexpunge command was the first thing I tried and I read it's man > page. But what puzzles me is that it seems that the deleted mailboxes > are not shown in unexpunge commands output. > > The mailbox meta files and mailbox's messages are shown in the spool dir > under the DELETED prefix eg. in the directory: > /cyrus_spool/u/DELETED/user/foo/sent-mail/HEXSTRING/ > > What is the correct procedure the restore this mailbox so that user is > able see the messages and access it once again. > You will need to move the mailbox back to the original location in the user's tree. unexpunge works on messages not entire folders. Cheers, Ken From mhlavink at redhat.com Tue Mar 17 09:31:09 2009 From: mhlavink at redhat.com (Michal Hlavinka) Date: Tue, 17 Mar 2009 14:31:09 +0100 Subject: ctl_cyrusdb questions, looking for answers In-Reply-To: <1236903922.9159.5.camel@linux-m3mt> References: <200903121938.38709.mhlavink@redhat.com> <1236903922.9159.5.camel@linux-m3mt> Message-ID: <200903171431.10067.mhlavink@redhat.com> Thanks for the answers! > ... > > > The information that man does not provide is > > 1) Listing of database files "checkpointed" > > I've always assumed all-of-them excerpt from the source: ... /* if we need to archive this db, add it to the list */ if (dblist[i].archive) { ...dblist[i] is added to archive_files list... } 2 = (*(dblist[i].env))->sync(); ....for some dbs (afaik for those who has .archive=1) this is empty function r2 = (*(dblist[i].env))->archive((const char**) archive_files, backup1); ... but dbs with .archive=0 are not in archive_files list ... this dbs have .archive=1 : mboxlist_db, quota_db, annotation_db other have .archive=0 > > > 2) Description of checkpoint format > > a) "ls /var/lib/imap" suggest two sets of checkpoint database are > > retained. > > ? Not following you here. It checkpoints the databases, not certain > what "two sets" of what your seeing. it's related to prev. question. The archive function (used only for some dbs) copies them to db.backupX directory. Other dbs are maybe somehow checkpointed by the sync call (that is non-empty function for them). So it seems not all dbs are checkpointed or maybe not all of them are in db.backupX ? From mhlavink at redhat.com Tue Mar 17 09:31:08 2009 From: mhlavink at redhat.com (Michal Hlavinka) Date: Tue, 17 Mar 2009 14:31:08 +0100 Subject: ctl_cyrusdb questions, looking for answers In-Reply-To: <30d2d300a56d2bf0d7f172b66b577aec.squirrel@webmail.bi.corp.invoca.ch> References: <200903121938.38709.mhlavink@redhat.com> <30d2d300a56d2bf0d7f172b66b577aec.squirrel@webmail.bi.corp.invoca.ch> Message-ID: <200903171431.08328.mhlavink@redhat.com> > ... > > 1) Listing of database files "checkpointed" > > 2) Description of checkpoint format > > a) "ls /var/lib/imap" suggest two sets of checkpoint database are > > retained. > > I'm not really sure here. Cyrus-imapd checkpoints databases and archives > some of them (mboxlist_db, quota_db, annotation_db). Archives are put and > rotated into the db.backup[1,2] dirs. so are other databases (other then mboxlist_db, quota_db, annotation_db) checkpointed or not? > > > b) Description of checkpoint configuration options, if any, to control > > checkpoint sets > > 3) Checkpointed image backup requirements > > Not exactly sure what you mean here. I think you only need the database in > db.backup? and of course clean the environment before restoring such a db. AFAIK only that three databases are copied to the db.backupX directory. So if there are other checkpointed dbs they stay in the original place and I need to backup them? > > 5) Appropriate recovery procedure when invocation with "-r" vs. "-r -x' > > is required. > > I don't remember to have used -x in the last 10 years using cyrus-imapd. I > really don't know what the cleanup part means. Is it cleaning up unused > space in a db file or does it mean handling of open transactions? thats what I'm wondering :) > > Regards, > Simon Thanks for the answers! From awilliam at whitemice.org Tue Mar 17 09:37:37 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Tue, 17 Mar 2009 09:37:37 -0400 Subject: ctl_cyrusdb questions, looking for answers In-Reply-To: <200903171431.08328.mhlavink@redhat.com> References: <200903121938.38709.mhlavink@redhat.com> <30d2d300a56d2bf0d7f172b66b577aec.squirrel@webmail.bi.corp.invoca.ch> <200903171431.08328.mhlavink@redhat.com> Message-ID: <1237297057.7794.2.camel@linux-m3mt> > > Not exactly sure what you mean here. I think you only need the database in > > db.backup? and of course clean the environment before restoring such a db. > AFAIK only that three databases are copied to the db.backupX directory. So if > there are other checkpointed dbs they stay in the original place and I need to > backup them? As I understand it: there is no reason to backup deliver.db, statuscache.db, or tls_sessions.db. You can dump those without consequence and they will be rebuilt on the fly. -- OpenGroupware developer: awilliam at whitemice.org OpenGroupare & Cyrus IMAPd documenation @ From selsky at columbia.edu Tue Mar 17 11:26:54 2009 From: selsky at columbia.edu (Matt Selsky) Date: Tue, 17 Mar 2009 11:26:54 -0400 Subject: Is user lock-out required for backend xfers? In-Reply-To: References: <1236832952.8538.13.camel@spartacus.nakedape.cc> <49B9404C.5050103@andrew.cmu.edu> Message-ID: <20090317112654.8ptbukbyv4wskgkk@cubmailtest.cc.columbia.edu> Quoting Wesley Craig : > I'm pretty sure this is because neither the frontends nor the clients > are able to intelligently handle the kinds of transient errors that > are returned. Frontend improvements might be possible. Client > improvements are Sisyphean. What could the frontends do that would be better? -- Matt From brong at fastmail.fm Tue Mar 17 19:26:06 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 18 Mar 2009 10:26:06 +1100 Subject: ctl_cyrusdb questions, looking for answers In-Reply-To: <1237297057.7794.2.camel@linux-m3mt> References: <200903121938.38709.mhlavink@redhat.com> <30d2d300a56d2bf0d7f172b66b577aec.squirrel@webmail.bi.corp.invoca.ch> <200903171431.08328.mhlavink@redhat.com> <1237297057.7794.2.camel@linux-m3mt> Message-ID: <20090317232606.GA31648@brong.net> On Tue, Mar 17, 2009 at 09:37:37AM -0400, Adam Tauno Williams wrote: > > > Not exactly sure what you mean here. I think you only need the database in > > > db.backup? and of course clean the environment before restoring such a db. > > AFAIK only that three databases are copied to the db.backupX directory. So if > > there are other checkpointed dbs they stay in the original place and I need to > > backup them? > > As I understand it: there is no reason to backup deliver.db, > statuscache.db, or tls_sessions.db. You can dump those without > consequence and they will be rebuilt on the fly. Yes - we delete these as part of our startup scripts (prefer not to have any BDB files lying around after a restart) Hey "checkpoint" is a massively overloaded term. In skiplist, a "checkpoint" is when the file gets rewritten in order. It's a complete rewrite of the file. Otherwise, skiplist is append only, so deleted or replaced records take up space. The algorithm for checkpoints is currently: tid->logend > (2 * db->logstart + SKIPLIST_MINREWRITE) Which basically means checkpoint when the file is over half garbage. I think that's actually a little too rare, especially with SKIPLIST_MINREWRITE = 16834, it means .seen files have a lot of junk in them! The "checkpoint" that ctl_cyrusdb does on the other hand, is totally different. It's just copying those files into the backup directories. It doesn't actually run a db checkpoint, at least not on skiplist. There's no way to trigger one externally. I have patches in the FastMail set that tune SKIPLIST_MINREWRITE lower, and the amount of junk required lower. They also do a checkpoint every time we run recovery (recovery is basically going through and recreating all the links between records, rolling back any uncompleted transaction as well). I think the "checkpoint on recovery" code will go into 2.3.15 - it makes sense to rewrite the file when it's hot. Bron. From brong at fastmail.fm Tue Mar 17 19:27:59 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 18 Mar 2009 10:27:59 +1100 Subject: Delayed delete, restoring deleted mailboxes In-Reply-To: References: <1237291024.13273.29.camel@linux-m3mt> Message-ID: <20090317232759.GB31648@brong.net> On Tue, Mar 17, 2009 at 03:06:15PM +0200, Leena Heino wrote: > On Tue, 17 Mar 2009, Adam Tauno Williams wrote: > > > On Tue, 2009-03-17 at 13:26 +0200, Leena Heino wrote: > >> What is the recommended process to restore deleted messages or mailboxes > >> when delayed delete is in use? > > > > Use the unexpunge command; the man page isn't bad, and I have examples > > and some notes in the Cyrus chapter of WMOGAG > > . > > The unexpunge command was the first thing I tried and I read it's man > page. But what puzzles me is that it seems that the deleted mailboxes > are not shown in unexpunge commands output. > > The mailbox meta files and mailbox's messages are shown in the spool dir > under the DELETED prefix eg. in the directory: > /cyrus_spool/u/DELETED/user/foo/sent-mail/HEXSTRING/ (that HEXSTRING is a hexified time_t from when it was deleted by the way) > What is the correct procedure the restore this mailbox so that user is > able see the messages and access it once again. Yeah, this is a missing feature, for sure. We use custom in-house stuff to restore backups, which can also fetch files out of the DELETED namespace - but the cyrus tools need to be extended to support it too. Bron. From wes at umich.edu Tue Mar 17 20:40:21 2009 From: wes at umich.edu (Wesley Craig) Date: Tue, 17 Mar 2009 20:40:21 -0400 Subject: Is user lock-out required for backend xfers? In-Reply-To: <20090317112654.8ptbukbyv4wskgkk@cubmailtest.cc.columbia.edu> References: <1236832952.8538.13.camel@spartacus.nakedape.cc> <49B9404C.5050103@andrew.cmu.edu> <20090317112654.8ptbukbyv4wskgkk@cubmailtest.cc.columbia.edu> Message-ID: <3351B3DF-1DD2-4566-AC2C-30A7AC953A67@umich.edu> On 17 Mar 2009, at 11:26, Matt Selsky wrote: > Quoting Wesley Craig : >> I'm pretty sure this is because neither the frontends nor the clients >> are able to intelligently handle the kinds of transient errors that >> are returned. Frontend improvements might be possible. Client >> improvements are Sisyphean. > > What could the frontends do that would be better? Lie? Try again? Not having any reason to look closely at the problem lately, I can't say for sure what the proper solution might be. But the frontend *can* tell what's happening, i.e., there are some distinct states that the xfer'ing mailbox progresses through. For each of those cases, the frontend can probably do something smart, e.g., retry, make the MUA wait until the mailbox is again in an accessible state, something else that might become apparent looking at the code. :wes From reinaldoc at gmail.com Wed Mar 18 15:02:52 2009 From: reinaldoc at gmail.com (Reinaldo de Carvalho) Date: Wed, 18 Mar 2009 16:02:52 -0300 Subject: Log message deletion Message-ID: <4a5881460903181202q3421308cr3ac133e923ee0118@mail.gmail.com> Hi, If is possible, how to do cyrus log when user delete a message? -- Reinaldo de Carvalho http://korreio.sf.net http://python-cyrus.sf.net From mattg at assembly.state.ny.us Wed Mar 18 17:54:34 2009 From: mattg at assembly.state.ny.us (Matt Garretson) Date: Wed, 18 Mar 2009 17:54:34 -0400 Subject: vacation :from parameter -- setting display name? Message-ID: <49C16D9A.9080001@assembly.state.ny.us> Hello... With cyrus-imap 2.3.13, I was hoping to use the :from parameter of the vacation statement to set the display-name portion of the sender address in the generated auto-reply. For example: vacation :from "My Name " The sieve draft document says that the argument must be a valid rfc2822 mailbox-list, which I believe does include the optional display-name. However, sieve seems to be treating the entire string as an addr-spec, resulting in the following header in the auto-reply: From: " Is this intended behavior? Is there a better way for me to accomplish what I want? Thanks in advance, -Matt From brong at fastmail.fm Wed Mar 18 20:50:18 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 19 Mar 2009 11:50:18 +1100 Subject: Log message deletion In-Reply-To: <4a5881460903181202q3421308cr3ac133e923ee0118@mail.gmail.com> References: <4a5881460903181202q3421308cr3ac133e923ee0118@mail.gmail.com> Message-ID: <20090319005018.GA10445@brong.net> On Wed, Mar 18, 2009 at 04:02:52PM -0300, Reinaldo de Carvalho wrote: > Hi, > > If is possible, how to do cyrus log when user delete a message? At FastMail we have the an "auditlog" patch that does this, but it's not in core cyrus. It also logs all message creation. Bron. From reinaldoc at gmail.com Wed Mar 18 20:57:07 2009 From: reinaldoc at gmail.com (Reinaldo de Carvalho) Date: Wed, 18 Mar 2009 21:57:07 -0300 Subject: Log message deletion In-Reply-To: <20090319005018.GA10445@brong.net> References: <4a5881460903181202q3421308cr3ac133e923ee0118@mail.gmail.com> <20090319005018.GA10445@brong.net> Message-ID: <4a5881460903181757o1bfa0f17la218b15d49a6f7fa@mail.gmail.com> On Wed, Mar 18, 2009 at 9:50 PM, Bron Gondwana wrote: > On Wed, Mar 18, 2009 at 04:02:52PM -0300, Reinaldo de Carvalho wrote: >> Hi, >> >> If is possible, how to do cyrus log when user delete a message? > > At FastMail we have the an "auditlog" patch that does this, but it's > not in core cyrus. ?It also logs all message creation. > > Bron. > Hi Bron, please try post the patch to developer list to be commited. Is important code read the configuration like loglevel to be possible enable/disable. -- Reinaldo de Carvalho http://korreio.sf.net http://python-cyrus.sf.net From brong at fastmail.fm Wed Mar 18 21:25:50 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 19 Mar 2009 12:25:50 +1100 Subject: Log message deletion In-Reply-To: <4a5881460903181757o1bfa0f17la218b15d49a6f7fa@mail.gmail.com> References: <4a5881460903181202q3421308cr3ac133e923ee0118@mail.gmail.com><20090319005018.GA10445@brong.net> <4a5881460903181757o1bfa0f17la218b15d49a6f7fa@mail.gmail.com> Message-ID: <1237425950.29490.1306169565@webmail.messagingengine.com> On Wed, 18 Mar 2009 21:57 -0300, "Reinaldo de Carvalho" wrote: > On Wed, Mar 18, 2009 at 9:50 PM, Bron Gondwana wrote: > > On Wed, Mar 18, 2009 at 04:02:52PM -0300, Reinaldo de Carvalho wrote: > >> Hi, > >> > >> If is possible, how to do cyrus log when user delete a message? > > > > At FastMail we have the an "auditlog" patch that does this, but it's > > not in core cyrus. ?It also logs all message creation. > > > > Bron. > > > > Hi Bron, > > please try post the patch to developer list to be commited. Is > important code read the configuration like loglevel to be possible > enable/disable. I'm planning to polish the code up for 2.3.15 hopefully. I'm going to be dumping a LOT more into 2.3.15. But I wanted to keep this one relatively small and stable :) -- Bron Gondwana brong at fastmail.fm From paulino.calderon at gmail.com Wed Mar 18 23:14:36 2009 From: paulino.calderon at gmail.com (Paulino Calderon) Date: Wed, 18 Mar 2009 21:14:36 -0600 Subject: Question about CYRUS-IMAP and FETCH BODY[] Message-ID: <49C1B89C.2010706@gmail.com> Hey guys, We recently upgraded cyrus-imapd in one of our production servers (debian btw) to 2.2.13-14+b3. After that, one of our programs stopped working complaining about not being able to fetch the messages, so I took a look at the traffic between our imap server and the program and I saw this: http://calder0n.com/cyrus-imap-traffic.png As you can see, the connection is being established succesfully but our program ( it was running OK for almost 2 years btw ) is sending a: IMAP FETCH BODY[1] And the server is only responding: IMAP Ok Completed (0.000 sec) But imap is not actually sending the body's content. Now since this issue came to surface after our upgrade I suspect that it might have something to do with our program's imap library not being fully RFC complaint. Has anyone seen this before? Any hints or ideas on what might be causing this? Possible workarounds? Thanks for your help in advance. From Hagedorn at uni-koeln.de Thu Mar 19 04:43:11 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Thu, 19 Mar 2009 09:43:11 +0100 Subject: Question about CYRUS-IMAP and FETCH BODY[] In-Reply-To: <49C1B89C.2010706@gmail.com> References: <49C1B89C.2010706@gmail.com> Message-ID: Hi, --On 18. M?rz 2009 21:14:36 -0600 Paulino Calderon wrote: > Hey guys, > We recently upgraded cyrus-imapd in one of our production servers > (debian btw) to 2.2.13-14+b3. After that, one of our programs stopped > working complaining about not being able to fetch the messages, so I > took a look at the traffic between our imap server and the program and I > saw this: > > http://calder0n.com/cyrus-imap-traffic.png > > As you can see, the connection is being established succesfully but our > program ( it was running OK for almost 2 years btw ) is sending a: > > IMAP FETCH BODY[1] actually that's not true. It's sending a *UID* FETCH, which is something quite different. > And the server is only responding: > IMAP Ok Completed (0.000 sec) That's quite correct if no message with UID 2 exists ... you've given no indication that it *does* exist. -- .:.Sebastian Hagedorn - RZKR-R1 (Geb?ude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universit?t zu K?ln / Cologne University - ? +49-221-478-5587.:. .:.:.:.Skype: shagedorn.:.:.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090319/daaf0a8e/attachment.bin From ml at awinkelmann.de Thu Mar 19 06:28:13 2009 From: ml at awinkelmann.de (Andreas Winkelmann) Date: Thu, 19 Mar 2009 11:28:13 +0100 (CET) Subject: ctl_cyrusdb questions, looking for answers In-Reply-To: <200903121938.38709.mhlavink@redhat.com> References: <200903121938.38709.mhlavink@redhat.com> Message-ID: <73f91df729ac7830f5cc746399125640.squirrel@a-angels.ath.cx> > does anybody know, how ctl_cyrusdb works (how it should be used)? I've > seen ctl_cyrusdb's man pages, but there is just few notes and nothing > more. ctl_cyrusdb has two modes to operate. One is checkpointing and archiving the other one is recovering. There can only run one action at a time. Checkpointing and Archiving: > ctl_cyrusdb [ -C config-file ] -c ctl_cyrusdb executes checkpointing for all Databases (mboxlist, quota, annotation, duplicate, tlscache, ptscache and statuscache) but only the Berkeley DB-Engine supports checkpointing at the moment. So at the end checkpointing is done for all BDB Databases. So for BDB Databases the Memory will be flushed to disk, the checkpoint is written to the Transaction Log and the Log is flushed to the DB File(s). Archiving is basicially copying the DB Files to the $configdirectory/db.backup1 Directory. Before copying starts the existing $configdirectory/db.backup1 Dir will be moved to $configdirectory/db.backup2. There are three DBs which will be archived (mboxlist, quota and annotation). Only the SQL DB-Backend dos not support archiving. Recovering: > ctl_cyrusdb [ -C config-file ] -r [ -x ] ctl_cyrusdb starts recovering for all Databases. Only BDB and skiplist do support recovering. Unless -x is specified recovering includes deleting of Mailbox reservations. Mailbox reservations are done in the process of Mailbox creation. > I assume ctl_cyrusdb is used (in /etc/cyrus.conf) to generate > "checkpoints" of the Cyrus databases, presumably those in /var/lib/imap. > > The information that man does not provide is > > 1) Listing of database files "checkpointed" see above > 2) Description of checkpoint format > a) "ls /var/lib/imap" suggest two sets of checkpoint database are > retained. db.backup1/2 is the backup Directory for archiving. Not related to checkpointing. > b) Description of checkpoint configuration options, if any, to control > checkpoint sets Nothing in Cyrus-IMAP I am aware of. > 3) Checkpointed image backup requirements ?? > 4) Appropriate cross-references to utilities used to determine database > corrution requiring recovery procedure If Cyrus finds DB Corruption you will see it in the Logs. > 5) Appropriate recovery procedure when invocation with "-r" vs. "-r -x' > is required. ?? > It'll be really appreciated if someone could explain at least some of > these questions. Thanks -- Andreas From ml at awinkelmann.de Thu Mar 19 06:44:54 2009 From: ml at awinkelmann.de (Andreas Winkelmann) Date: Thu, 19 Mar 2009 11:44:54 +0100 (CET) Subject: Question about CYRUS-IMAP and FETCH BODY[] In-Reply-To: <49C1B89C.2010706@gmail.com> References: <49C1B89C.2010706@gmail.com> Message-ID: > We recently upgraded cyrus-imapd in one of our production servers > (debian btw) to 2.2.13-14+b3. After that, one of our programs stopped > working complaining about not being able to fetch the messages, so I > took a look at the traffic between our imap server and the program and I > saw this: > > http://calder0n.com/cyrus-imap-traffic.png > > As you can see, the connection is being established succesfully but our > program ( it was running OK for almost 2 years btw ) is sending a: > > IMAP FETCH BODY[1] > And the server is only responding: > IMAP Ok Completed (0.000 sec) > > But imap is not actually sending the body's content. > Now since this issue came to surface after our upgrade I suspect that it > might have something to do with our program's imap library not being > fully RFC complaint. Has anyone seen this before? > > Any hints or ideas on what might be causing this? Possible workarounds? > > Thanks for your help in advance. Hmm, I am missing the complete answer for the SELECT INBOX in your dump. IMAP003 is not completed. Check you Logs. Maybe there is something wrong with the Mailbox. Try a reconstruct. -- Andreas From janne.peltonen at helsinki.fi Thu Mar 19 07:46:35 2009 From: janne.peltonen at helsinki.fi (Janne Peltonen) Date: Thu, 19 Mar 2009 13:46:35 +0200 Subject: Sieve scripts in global namespace and replication Message-ID: <20090319114634.GB28796@helsinki.fi> Hi! It appears that my sieve scripts that I have in the global namespace and that are annotated to certain Cyrus bulletin boards don't get replicated. Apparently, only user sieve scripts ever get replicated (when something generates a USER replication event). How do I (rollingly) replicate the sieve scripts in the global namespace? Thanks for any advice. --Janne Peltonen University of Helsinki -- Janne Peltonen PGP Key ID: 0x9CFAC88B Please consider membership of the Hospitality Club (http://www.hospitalityclub.org) From paulino.calderon at gmail.com Thu Mar 19 13:40:53 2009 From: paulino.calderon at gmail.com (Paulino Calderon) Date: Thu, 19 Mar 2009 11:40:53 -0600 Subject: Question about CYRUS-IMAP and FETCH BODY[] In-Reply-To: References: <49C1B89C.2010706@gmail.com> Message-ID: <49C283A5.7050401@gmail.com> Sebastian Hagedorn wrote: > Hi, > > --On 18. M?rz 2009 21:14:36 -0600 Paulino Calderon > wrote: > >> Hey guys, >> We recently upgraded cyrus-imapd in one of our production servers >> (debian btw) to 2.2.13-14+b3. After that, one of our programs stopped >> working complaining about not being able to fetch the messages, so I >> took a look at the traffic between our imap server and the program and I >> saw this: >> >> http://calder0n.com/cyrus-imap-traffic.png >> >> As you can see, the connection is being established succesfully but our >> program ( it was running OK for almost 2 years btw ) is sending a: >> >> IMAP FETCH BODY[1] > > actually that's not true. It's sending a *UID* FETCH, which is > something quite different. > >> And the server is only responding: >> IMAP Ok Completed (0.000 sec) > > That's quite correct if no message with UID 2 exists ... you've given > no indication that it *does* exist. Umm, well after a: IMAP SELECT INBOX The server is responding: http://calder0n.com/cyrus-imap-selectinbox-response.png I ran reconstruct but the problem persists, also there are no errors in the logs that could point me to the right direction. Thanks. From Hagedorn at uni-koeln.de Thu Mar 19 13:48:25 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Thu, 19 Mar 2009 18:48:25 +0100 Subject: Question about CYRUS-IMAP and FETCH BODY[] In-Reply-To: <49C283A5.7050401@gmail.com> References: <49C1B89C.2010706@gmail.com> <49C283A5.7050401@gmail.com> Message-ID: <027FCF725984627CA6CEBF1A@G5.local> -- Paulino Calderon is rumored to have mumbled on 19. M?rz 2009 11:40:53 -0600 regarding Re: Question about CYRUS-IMAP and FETCH BODY[]: >>> As you can see, the connection is being established succesfully but our >>> program ( it was running OK for almost 2 years btw ) is sending a: >>> >>> IMAP FETCH BODY[1] >> >> actually that's not true. It's sending a *UID* FETCH, which is >> something quite different. >> >>> And the server is only responding: >>> IMAP Ok Completed (0.000 sec) >> >> That's quite correct if no message with UID 2 exists ... you've given >> no indication that it *does* exist. > Umm, well after a: > IMAP SELECT INBOX > The server is responding: > http://calder0n.com/cyrus-imap-selectinbox-response.png So? I still see no evidence that a message with *UID* 2 exists. The response only shows that a message with *sequence number* 2 exists! That's why I said that the actual command is quite different from what you claimed it was. This command will give you a response whenever there are at least two messages: 10 FETCH 2 BODY[1] This command won't give you a response even if there are 100,000 messages in the mailbox if none of them has UID 2: 10 UID FETCH 2 BODY[1] -- Sebastian Hagedorn - RZKR-R1 (Flachbau), Zi. 18, Robert-Koch-Str. 10 Regionales Rechenzentrum (RRZK) Universit?t zu K?ln / Cologne University - Tel. +49-221-478-5587 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090319/85c4f27b/attachment.bin From paulino.calderon at gmail.com Thu Mar 19 20:20:49 2009 From: paulino.calderon at gmail.com (Paulino Calderon) Date: Thu, 19 Mar 2009 18:20:49 -0600 Subject: Question about CYRUS-IMAP and FETCH BODY[] In-Reply-To: <027FCF725984627CA6CEBF1A@G5.local> References: <49C1B89C.2010706@gmail.com> <49C283A5.7050401@gmail.com> <027FCF725984627CA6CEBF1A@G5.local> Message-ID: <49C2E161.3070204@gmail.com> Sebastian Hagedorn wrote: > -- Paulino Calderon is rumored to have > mumbled on 19. M?rz 2009 11:40:53 -0600 regarding Re: Question about > CYRUS-IMAP and FETCH BODY[]: > >>>> As you can see, the connection is being established succesfully but >>>> our >>>> program ( it was running OK for almost 2 years btw ) is sending a: >>>> >>>> IMAP FETCH BODY[1] >>> >>> actually that's not true. It's sending a *UID* FETCH, which is >>> something quite different. >>> >>>> And the server is only responding: >>>> IMAP Ok Completed (0.000 sec) >>> >>> That's quite correct if no message with UID 2 exists ... you've given >>> no indication that it *does* exist. >> Umm, well after a: >> IMAP SELECT INBOX >> The server is responding: >> http://calder0n.com/cyrus-imap-selectinbox-response.png > > So? I still see no evidence that a message with *UID* 2 exists. The > response only shows that a message with *sequence number* 2 exists! > That's why I said that the actual command is quite different from what > you claimed it was. This command will give you a response whenever > there are at least two messages: Sorry about that. What command should I send to retrieve the uids of the unseen messages? The output in the image shows the traffic between the program it used to work and the server, i didnt send those commands manually. So "FETCH <#> BODY[1]" was working before. I assumed that 2 ( on the screenshot) was from the unseen response after a "select inbox". Thanks again. > > 10 FETCH 2 BODY[1] > > This command won't give you a response even if there are 100,000 > messages in the mailbox if none of them has UID 2: > > 10 UID FETCH 2 BODY[1] > -- > Sebastian Hagedorn - RZKR-R1 (Flachbau), Zi. 18, Robert-Koch-Str. 10 > Regionales Rechenzentrum (RRZK) > Universit?t zu K?ln / Cologne University - Tel. +49-221-478-5587 From brong at fastmail.fm Thu Mar 19 23:42:01 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 20 Mar 2009 14:42:01 +1100 Subject: Question about CYRUS-IMAP and FETCH BODY[] In-Reply-To: <49C2E161.3070204@gmail.com> References: <49C1B89C.2010706@gmail.com> <49C283A5.7050401@gmail.com> <027FCF725984627CA6CEBF1A@G5.local> <49C2E161.3070204@gmail.com> Message-ID: <20090320034201.GA4643@brong.net> On Thu, Mar 19, 2009 at 06:20:49PM -0600, Paulino Calderon wrote: > Sebastian Hagedorn wrote: > > -- Paulino Calderon is rumored to have > > mumbled on 19. M?rz 2009 11:40:53 -0600 regarding Re: Question about > > CYRUS-IMAP and FETCH BODY[]: > > > >>>> As you can see, the connection is being established succesfully but > >>>> our > >>>> program ( it was running OK for almost 2 years btw ) is sending a: > >>>> > >>>> IMAP FETCH BODY[1] > >>> > >>> actually that's not true. It's sending a *UID* FETCH, which is > >>> something quite different. > >>> > >>>> And the server is only responding: > >>>> IMAP Ok Completed (0.000 sec) > >>> > >>> That's quite correct if no message with UID 2 exists ... you've given > >>> no indication that it *does* exist. > >> Umm, well after a: > >> IMAP SELECT INBOX > >> The server is responding: > >> http://calder0n.com/cyrus-imap-selectinbox-response.png > > > > So? I still see no evidence that a message with *UID* 2 exists. The > > response only shows that a message with *sequence number* 2 exists! > > That's why I said that the actual command is quite different from what > > you claimed it was. This command will give you a response whenever > > there are at least two messages: > Sorry about that. What command should I send to retrieve the uids of the > unseen messages? The output in the image shows the traffic between the > program it used to work and the server, i didnt send those commands > manually. So "FETCH <#> BODY[1]" was working before. I assumed that 2 ( > on the screenshot) was from the unseen response after a "select inbox". > > Thanks again. TAG UID SEARCH UNSEEN (e.g. this on my INBOX) ---------------------------- . select inbox * FLAGS (\Answered \Flagged \Draft \Deleted \Seen hasnoatt hasatt * selected medeleted NonJunk receipt-handled) * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen hasnoatt * hasatt selected medeleted NonJunk receipt-handled \*)] * 887 EXISTS * 1 RECENT * OK [UNSEEN 882] * OK [UIDVALIDITY 1148523981] * OK [UIDNEXT 379340] * OK [NOMODSEQ] Sorry, modsequences have not been enabled on this * mailbox * OK [URLMECH INTERNAL] . OK [READ-WRITE] Completed TAG UID SEARCH UNSEEN * SEARCH 379334 379335 379336 379337 379338 379339 TAG OK Completed (6 msgs in 0.010 secs) TAG SEARCH UNSEEN * SEARCH 882 883 884 885 886 887 TAG OK Completed (6 msgs in 0.000 secs) ---------------------------- The second one being sequence numbers rather than UIDs. It's the "UID" in the command that makes the difference. Bron. From paulino.calderon at gmail.com Fri Mar 20 00:43:43 2009 From: paulino.calderon at gmail.com (Paulino Calderon) Date: Thu, 19 Mar 2009 22:43:43 -0600 Subject: Question about CYRUS-IMAP and FETCH BODY[] In-Reply-To: <20090320034201.GA4643@brong.net> References: <49C1B89C.2010706@gmail.com> <49C283A5.7050401@gmail.com> <027FCF725984627CA6CEBF1A@G5.local> <49C2E161.3070204@gmail.com> <20090320034201.GA4643@brong.net> Message-ID: <49C31EFF.8090208@gmail.com> Bron Gondwana wrote: > On Thu, Mar 19, 2009 at 06:20:49PM -0600, Paulino Calderon wrote: > >> Sebastian Hagedorn wrote: >> >>> -- Paulino Calderon is rumored to have >>> mumbled on 19. M?rz 2009 11:40:53 -0600 regarding Re: Question about >>> CYRUS-IMAP and FETCH BODY[]: >>> >>> >>>>>> As you can see, the connection is being established succesfully but >>>>>> our >>>>>> program ( it was running OK for almost 2 years btw ) is sending a: >>>>>> >>>>>> IMAP FETCH BODY[1] >>>>>> >>>>> actually that's not true. It's sending a *UID* FETCH, which is >>>>> something quite different. >>>>> >>>>> >>>>>> And the server is only responding: >>>>>> IMAP Ok Completed (0.000 sec) >>>>>> >>>>> That's quite correct if no message with UID 2 exists ... you've given >>>>> no indication that it *does* exist. >>>>> >>>> Umm, well after a: >>>> IMAP SELECT INBOX >>>> The server is responding: >>>> http://calder0n.com/cyrus-imap-selectinbox-response.png >>>> >>> So? I still see no evidence that a message with *UID* 2 exists. The >>> response only shows that a message with *sequence number* 2 exists! >>> That's why I said that the actual command is quite different from what >>> you claimed it was. This command will give you a response whenever >>> there are at least two messages: >>> >> Sorry about that. What command should I send to retrieve the uids of the >> unseen messages? The output in the image shows the traffic between the >> program it used to work and the server, i didnt send those commands >> manually. So "FETCH <#> BODY[1]" was working before. I assumed that 2 ( >> on the screenshot) was from the unseen response after a "select inbox". >> >> Thanks again. >> > > TAG UID SEARCH UNSEEN > > (e.g. this on my INBOX) > > ---------------------------- > > . select inbox > * FLAGS (\Answered \Flagged \Draft \Deleted \Seen hasnoatt hasatt > * selected medeleted NonJunk receipt-handled) > * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen hasnoatt > * hasatt selected medeleted NonJunk receipt-handled \*)] > * 887 EXISTS > * 1 RECENT > * OK [UNSEEN 882] > * OK [UIDVALIDITY 1148523981] > * OK [UIDNEXT 379340] > * OK [NOMODSEQ] Sorry, modsequences have not been enabled on this > * mailbox > * OK [URLMECH INTERNAL] > . OK [READ-WRITE] Completed > TAG UID SEARCH UNSEEN > * SEARCH 379334 379335 379336 379337 379338 379339 > TAG OK Completed (6 msgs in 0.010 secs) > TAG SEARCH UNSEEN > * SEARCH 882 883 884 885 886 887 > TAG OK Completed (6 msgs in 0.000 secs) > > ---------------------------- > > The second one being sequence numbers rather than UIDs. It's the > "UID" in the command that makes the difference. > > Bron. > Yeah I've realized it would be for the best to replace that old library. It's going to fail again if we leave it there. Anyway, I'm still curious why it broke after the upgrade, what do you guys think? Is it just not following a newer RFC as it's supposed to? The library we're using is: http://www.codeproject.com/KB/IP/imaplibrary.aspx From Hagedorn at uni-koeln.de Fri Mar 20 07:32:27 2009 From: Hagedorn at uni-koeln.de (Sebastian Hagedorn) Date: Fri, 20 Mar 2009 12:32:27 +0100 Subject: Question about CYRUS-IMAP and FETCH BODY[] In-Reply-To: <49C31EFF.8090208@gmail.com> References: <49C1B89C.2010706@gmail.com> <49C283A5.7050401@gmail.com> <027FCF725984627CA6CEBF1A@G5.local> <49C2E161.3070204@gmail.com> <20090320034201.GA4643@brong.net> <49C31EFF.8090208@gmail.com> Message-ID: Hi, --On 19. M?rz 2009 22:43:43 -0600 Paulino Calderon wrote: > Yeah I've realized it would be for the best to replace that old library. > It's going to fail again if we leave it there. Anyway, I'm still curious > why it broke after the upgrade, what do you guys think? Is it just not > following a newer RFC as it's supposed to? no, nothing has changed in that respect. > The library we're using is: > > http://www.codeproject.com/KB/IP/imaplibrary.aspx Looking at the documentation it strikes me that there appears to be no way to actually list the contents of a mailbox. If those commands are really the only ones that exist, the only way to use it would appear to be a SearchMessage followed by a FetchMessage. If you don't do a SeachMessage, you have no way of knowing what you can fetch. -- .:.Sebastian Hagedorn - RZKR-R1 (Geb?ude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universit?t zu K?ln / Cologne University - ? +49-221-478-5587.:. .:.:.:.Skype: shagedorn.:.:.:. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090320/f83a202f/attachment-0001.bin From adam at morrison-ind.com Fri Mar 20 08:25:42 2009 From: adam at morrison-ind.com (Adam Tauno Williams) Date: Fri, 20 Mar 2009 08:25:42 -0400 Subject: Question about CYRUS-IMAP and FETCH BODY[] In-Reply-To: References: <49C1B89C.2010706@gmail.com> <49C283A5.7050401@gmail.com> <027FCF725984627CA6CEBF1A@G5.local> <49C2E161.3070204@gmail.com> <20090320034201.GA4643@brong.net> <49C31EFF.8090208@gmail.com> Message-ID: <1237551942.5660.9.camel@linux-m3mt> On Fri, 2009-03-20 at 12:32 +0100, Sebastian Hagedorn wrote: > > The library we're using is: > > http://www.codeproject.com/KB/IP/imaplibrary.aspx > Looking at the documentation it strikes me that there appears to be no way > to actually list the contents of a mailbox. If those commands are really > the only ones that exist, the only way to use it would appear to be a > SearchMessage followed by a FetchMessage. If you don't do a SeachMessage, > you have no way of knowing what you can fetch. I tried the CodeProject IMAP code awhile back; it was pretty bad. If you need to access IMAP from .NET [which, oddly, support seems lacking for such a common protocol] the LumiSoft.NET library is much better and works reliably (even on Mono). Unfortunately even the ongoing support of that assembly *feels* dodgy; for one, it is pretty darn hard to find. I'd strongly recommend wrapping your IMAP server access is some class or interface so you can replace the IMAP protocol provider without too much modification to your app. -- OpenGroupware developer: awilliam at whitemice.org OpenGroupare & Cyrus IMAPd documenation @ From freebsd at abv.bg Sun Mar 22 05:36:36 2009 From: freebsd at abv.bg (Mario Pavlov) Date: Sun, 22 Mar 2009 11:36:36 +0200 (EET) Subject: sendmail and cyrus-imap - delivery problem Message-ID: <1762486541.144931.1237714596819.JavaMail.apache@mail51.abv.bg> Hi, I've setup sendmail 8.14.3 to work with cyrus-imapd 2.3.13 on FreeBSD 7-STABLE but I have a delivery issue of course outgoing e-mail is OK but incoming is not ... it was OK before I switched to the cyrus mailer so if we assume my domain is yo.com here's my imapd.conf # grep -v "^#" /usr/local/etc/imapd.conf configdirectory: /var/imap partition-default: /var/imap/spool unixhierarchysep: yes admins: cyrus admin at yo.com sieveusehomedir: false sievedir: /var/imap/sieve sasl_pwcheck_method: auxprop defaultdomain: yo.com loginrealms: yo.com gg.com and cyrus.conf # grep -v "#" /usr/local/etc/cyrus.conf START { recover cmd="ctl_cyrusdb -r" } SERVICES { imap cmd="imapd" listen="imap" prefork=0 imaps cmd="imapd -s" listen="imaps" prefork=0 pop3 cmd="pop3d" listen="pop3" prefork=0 pop3s cmd="pop3d -s" listen="pop3s" prefork=0 sieve cmd="timsieved" listen="sieve" prefork=0 lmtpunix cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0 } EVENTS { checkpoint cmd="ctl_cyrusdb -c" period=30 delprune cmd="cyr_expire -E 3" at=0400 tlsprune cmd="tls_prune" at=0400 } and yo.com.mc assuming mu hostname is yo.com VERSIONID(`$FreeBSD: src/etc/sendmail/freebsd.mc,v 1.34.2.3 2008/08/31 18:26:27 gshapiro Exp $') OSTYPE(freebsd6) DOMAIN(generic) FEATURE(access_db, `hash -o -T /etc/mail/access') FEATURE(blacklist_recipients) FEATURE(local_lmtp) FEATURE(mailertable, `hash -o /etc/mail/mailertable') FEATURE(virtusertable, `hash -o /etc/mail/virtusertable') DAEMON_OPTIONS(`Name=IPv4, Family=inet') define(`confBIND_OPTS', `WorkAroundBrokenAAAA') define(`confNO_RCPT_ACTION', `add-to-undisclosed') define(`confPRIVACY_FLAGS', `authwarnings,noexpn,novrfy') TRUST_AUTH_MECH(`GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN') define(`confAUTH_MECHANISMS', `GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN') define(`confLOCAL_MAILER', `cyrusv2') MAILER(local) MAILER(smtp) MAILER(`cyrusv2') my users and boxes # sasldblistusers2 boo at yo.com: userPassword cyrus at yo.com: userPassword admin at yo.com: userPassword # cyradm --user cyrus localhost Password: localhost> lm user/boo at yo.com (\HasNoChildren) having this configuration I can see the following in the maillog if someone in internet decides to send me an e-mail: Mar 21 09:55:57 yo sm-mta[3426]: n2L9tvmO003426: from=, size=1599, class=0, nrcpts=1, msgid=<9cddded00903210255v1a8a4148yb5923d7547493b6a at mail.gmail.com>, proto=ESMTP, daemon=IPv4, relay=fg-out-1718.google.com [72.14.220.154] Mar 21 09:55:57 yo sm-mta[3465]: n2L9tvmO003426: to=, delay=00:00:00, xdelay=00:00:00, mailer=cyrusv2, pri=31599, relay=localhost, dsn=5.1.1, stat=User unknown Mar 21 09:55:57 yo sm-mta[3465]: n2L9tvmO003426: n2L9tvmO003465: DSN: User unknown I suspect sendmail and cyrus have to somehow agree which socket they will use there is a socket specified in cyrus.conf maybe I should somehow tell sendmail to use that too ...or may be sendmail and cyrus have to somehow agree on usernames... cyrus knows about user/boo at yo.com but does sendmail know about it ? or it thinks it's just boo at yo.com I've tried to add this define(`CYRUS_LMTP_SOCKET',`/var/imap/socket/lmtp') to yo.com.mc but it made no difference I even don't understand where the problem is what do you think could be the problem ? thank you. From duncan.gibb at siriusit.co.uk Sun Mar 22 06:51:15 2009 From: duncan.gibb at siriusit.co.uk (Duncan Gibb) Date: Sun, 22 Mar 2009 10:51:15 +0000 Subject: sendmail and cyrus-imap - delivery problem In-Reply-To: <1762486541.144931.1237714596819.JavaMail.apache@mail51.abv.bg> References: <1762486541.144931.1237714596819.JavaMail.apache@mail51.abv.bg> Message-ID: <49C61823.7010802@siriusit.co.uk> Mario Pavlov wrote: MP> I've setup sendmail 8.14.3 to work with cyrus-imapd 2.3.13 on MP> FreeBSD 7-STABLE but I have a delivery issue > localhost> lm > user/boo at yo.com (\HasNoChildren) > Mar 21 09:55:57 yo sm-mta[3465]: n2L9tvmO003426: to=, > delay=00:00:00, xdelay=00:00:00, mailer=cyrusv2, pri=31599, > relay=localhost, dsn=5.1.1, stat=User unknown MP> cyrus knows about user/boo at yo.com MP> but does sendmail know about it ? MP> or it thinks it's just boo at yo.com MP> what do you think could be the problem ? I think your problem is that for Cyrus virtdomains to work the MTA needs to send the _full_ address in the LMTP transaction. When Sendmail speaks to Cyrus lmtpd, it needs to say RCPT TO: rather than just RCPT TO: Sendmail's definition of the cyrusv2 mailer (/usr/share/sendmail/cf/mailer/cyrusv2.m4 on Debian, possibly elsewhere on FreeBSD) sets "R=EnvToL/HdrToL", but for virtdomains you want "R=EnvToSMTP". It's also recommended that you change "S=EnvFromSMTP/HdrFromL" to "S=EnvFromSMTP/HdrFromSMTP". As a quick hack, you can change those in situ; or - more gracefully - you can define a new mailer which works this way and use it instead. The Open-Sendmail project has bells-and-whistles implementation, cyrusv3.m4, within RTCyrus3: http://open-sendmail.sourceforge.net/rtcyrus3/ but it is quite complicated to install. > FEATURE(local_lmtp) > MAILER(local) It's not clear to me what the above lines are achieving when you also have the below: > define(`confLOCAL_MAILER', `cyrusv2') > MAILER(`cyrusv2') Cheers Duncan -- Duncan Gibb - Technical Director Sirius Corporation plc - control through freedom http://www.siriusit.co.uk/ || t: +44 870 608 0063 From matthew.seaman at thebunker.net Sun Mar 22 06:56:40 2009 From: matthew.seaman at thebunker.net (Matthew Seaman) Date: Sun, 22 Mar 2009 10:56:40 +0000 Subject: sendmail and cyrus-imap - delivery problem In-Reply-To: <1762486541.144931.1237714596819.JavaMail.apache@mail51.abv.bg> References: <1762486541.144931.1237714596819.JavaMail.apache@mail51.abv.bg> Message-ID: <49C61968.9030906@thebunker.net> Mario Pavlov wrote: > Hi, > I've setup sendmail 8.14.3 to work with cyrus-imapd 2.3.13 on FreeBSD 7-STABLE > but I have a delivery issue > of course outgoing e-mail is OK > but incoming is not ... it was OK before I switched to the cyrus mailer This sounds like a job for smmapd. > so if we assume my domain is yo.com > here's my imapd.conf > > # grep -v "^#" /usr/local/etc/imapd.conf > configdirectory: /var/imap > partition-default: /var/imap/spool > unixhierarchysep: yes > admins: cyrus admin at yo.com > sieveusehomedir: false > sievedir: /var/imap/sieve > sasl_pwcheck_method: auxprop > defaultdomain: yo.com > loginrealms: yo.com gg.com > > and cyrus.conf > > # grep -v "#" /usr/local/etc/cyrus.conf > START { > recover cmd="ctl_cyrusdb -r" > } > SERVICES { > imap cmd="imapd" listen="imap" prefork=0 > imaps cmd="imapd -s" listen="imaps" prefork=0 > pop3 cmd="pop3d" listen="pop3" prefork=0 > pop3s cmd="pop3d -s" listen="pop3s" prefork=0 > sieve cmd="timsieved" listen="sieve" prefork=0 > lmtpunix cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0 > } > EVENTS { > checkpoint cmd="ctl_cyrusdb -c" period=30 > delprune cmd="cyr_expire -E 3" at=0400 > tlsprune cmd="tls_prune" at=0400 > } Add: # Sendmail socket map daemon smmapdunix cmd="smmapd" listen="/var/imap/socket/smmapd" prefork=1 in the SERVICES section > and yo.com.mc assuming mu hostname is yo.com > > VERSIONID(`$FreeBSD: src/etc/sendmail/freebsd.mc,v 1.34.2.3 2008/08/31 18:26:27 gshapiro Exp $') > OSTYPE(freebsd6) > DOMAIN(generic) > FEATURE(access_db, `hash -o -T /etc/mail/access') > FEATURE(blacklist_recipients) > FEATURE(local_lmtp) > FEATURE(mailertable, `hash -o /etc/mail/mailertable') > FEATURE(virtusertable, `hash -o /etc/mail/virtusertable') > DAEMON_OPTIONS(`Name=IPv4, Family=inet') > define(`confBIND_OPTS', `WorkAroundBrokenAAAA') > define(`confNO_RCPT_ACTION', `add-to-undisclosed') > define(`confPRIVACY_FLAGS', `authwarnings,noexpn,novrfy') > TRUST_AUTH_MECH(`GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN') > define(`confAUTH_MECHANISMS', `GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN') > define(`confLOCAL_MAILER', `cyrusv2') > MAILER(local) > MAILER(smtp) > MAILER(`cyrusv2') Add: FEATURE(`ckuser_cyrus', ,`local:/var/imap/socket/smmapd')dnl with the rest of the FEATUREs > > my users and boxes > > # sasldblistusers2 > boo at yo.com: userPassword > cyrus at yo.com: userPassword > admin at yo.com: userPassword > > # cyradm --user cyrus localhost > Password: > localhost> lm > user/boo at yo.com (\HasNoChildren) > > having this configuration I can see the following in the maillog if someone in internet decides to send me an e-mail: > > Mar 21 09:55:57 yo sm-mta[3426]: n2L9tvmO003426: from=, size=1599, class=0, nrcpts=1, msgid=<9cddded00903210255v1a8a4148yb5923d7547493b6a at mail.gmail.com>, proto=ESMTP, daemon=IPv4, relay=fg-out-1718.google.com [72.14.220.154] > Mar 21 09:55:57 yo sm-mta[3465]: n2L9tvmO003426: to=, delay=00:00:00, xdelay=00:00:00, mailer=cyrusv2, pri=31599, relay=localhost, dsn=5.1.1, stat=User unknown > Mar 21 09:55:57 yo sm-mta[3465]: n2L9tvmO003426: n2L9tvmO003465: DSN: User unknown > > I suspect sendmail and cyrus have to somehow agree which socket they will use > there is a socket specified in cyrus.conf maybe I should somehow tell sendmail to use that too > ...or may be sendmail and cyrus have to somehow agree on usernames... > cyrus knows about user/boo at yo.com > but does sendmail know about it ? > or it thinks it's just boo at yo.com You will need to compile a version of sendmail with SASL support. You can make the stock system sendmail do that by adding settings to /etc/make.conf, but then you end up with a base system that depends on shlibs outside the base system. It's probably cleaner if you install sendmail with appropriate tweaks from ports. Add the following to /etc/make.conf: WITH_BDB_VER= 46 WITH_SASL= yes WITH_SENDMAIL_PORT= yes SENDMAIL_CF_DIR= /usr/local/share/sendmail/cf MAKEMAP= /usr/local/sbin/makemap .if ${.CURDIR:M*/mail/sendmail*} SENDMAIL_WITHOUT_NIS= yes SENDMAIL_WITH_CYRUSLOOKUP= yes SENDMAIL_WITH_SOCKETMAP= yes SENDMAIL_WITH_TLS= yes SENDMAIL_WITH_SASL2= yes SENDMAIL_WITHOUT_LDAP= yes SENDMAIL_WITH_BERKELEYDB= yes SENDMAIL_WITH_PICKY_HELO_CHECK= yes SENDMAIL_WITH_SHARED_MILTER= yes .endif Then add this to /etc/src.conf to stop the system installing the base sendmail: WITHOUT_SENDMAIL=yes Build and install the mail/sendmail port. The new sendmail binary should link against libsasl2: % ldd /usr/local/sbin/sendmail /usr/local/sbin/sendmail: libwrap.so.5 => /usr/lib/libwrap.so.5 (0x8006cf000) libsasl2.so.2 => /usr/local/lib/libsasl2.so.2 (0x8007d8000) <<<---*** libssl.so.5 => /usr/local/lib/libssl.so.5 (0x8008f0000) libcrypto.so.5 => /usr/local/lib/libcrypto.so.5 (0x800a3a000) libdb-4.6.so.0 => /usr/local/lib/libdb-4.6.so.0 (0x800ca5000) libutil.so.7 => /lib/libutil.so.7 (0x800ed5000) libc.so.7 => /lib/libc.so.7 (0x800fe3000) libthr.so.3 => /lib/libthr.so.3 (0x8011fe000) Then edit /etc/mail/mailer.conf to look like this: # $FreeBSD: src/etc/mail/mailer.conf,v 1.3.32.1 2008/11/25 02:59:29 kensmith Exp $ # # Execute the "real" sendmail program, named /usr/libexec/sendmail/sendmail # sendmail /usr/local/sbin/sendmail send-mail /usr/local/sbin/sendmail mailq /usr/local/sbin/sendmail newaliases /usr/local/sbin/sendmail hoststat /usr/local/sbin/sendmail purgestat /usr/local/sbin/sendmail Now remove any previous DB files and rebuild using the new sendmail's version of makemap: # cd /etc/mail # rm -i *.db # make all install Fire up the newly built sendmail and test: # make start > I've tried to add this > > define(`CYRUS_LMTP_SOCKET',`/var/imap/socket/lmtp') > > to yo.com.mc > but it made no difference That's already the default setting. > I even don't understand where the problem is > > what do you think could be the problem ? In short: you need to compile in some extra functionality to sendmail in order to support SASL and sendmail socket-map. Cheers, Matthew -- Dr Matthew Seaman The Bunker, Ash Radar Station PGP: 0x60AE908C on servers Marshborough Rd Tel: +44 1304 814890 Sandwich Fax: +44 1304 814899 Kent, CT13 0PL, UK -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 259 bytes Desc: OpenPGP digital signature Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090322/20196253/attachment.bin From reinaldoc at gmail.com Sun Mar 22 08:02:57 2009 From: reinaldoc at gmail.com (Reinaldo de Carvalho) Date: Sun, 22 Mar 2009 09:02:57 -0300 Subject: sendmail and cyrus-imap - delivery problem In-Reply-To: <1762486541.144931.1237714596819.JavaMail.apache@mail51.abv.bg> References: <1762486541.144931.1237714596819.JavaMail.apache@mail51.abv.bg> Message-ID: <4a5881460903220502o7ba507f7y2e695be035f2ac2f@mail.gmail.com> On Sun, Mar 22, 2009 at 6:36 AM, Mario Pavlov wrote: > > # grep -v "^#" /usr/local/etc/imapd.conf > configdirectory: /var/imap > partition-default: /var/imap/spool > unixhierarchysep: yes > admins: cyrus admin at yo.com if virtdomains is "no" then logon is by user, like "cyrus". else virtdomains is "yes" then logon is by email, like "admin at yo.com" > sieveusehomedir: false > sievedir: /var/imap/sieve > sasl_pwcheck_method: auxprop > defaultdomain: yo.com > loginrealms: yo.com gg.com > In any case you should create mailbox with "cm user/boo" because domain is detected by admin logon. -- Reinaldo de Carvalho http://korreio.sf.net http://python-cyrus.sf.net From dick at nagual.nl Sun Mar 22 08:19:17 2009 From: dick at nagual.nl (dick hoogendijk) Date: Sun, 22 Mar 2009 13:19:17 +0100 Subject: sendmail and cyrus-imap - delivery problem In-Reply-To: <1762486541.144931.1237714596819.JavaMail.apache@mail51.abv.bg> References: <1762486541.144931.1237714596819.JavaMail.apache@mail51.abv.bg> Message-ID: <20090322131917.481c3e42.dick@nagual.nl> On Sun, 22 Mar 2009 11:36:36 +0200 (EET) Mario Pavlov wrote: > MAILER(local) > MAILER(smtp) > MAILER(`cyrusv2') Is this the correct order? Does it make a difference if you switch (smtp) <-> (cyrusv2) ? -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS sxce snv109 ++ + All that's really worth doing is what we do for others (Lewis Carrol) From freebsd at abv.bg Sun Mar 22 12:37:32 2009 From: freebsd at abv.bg (Mario Pavlov) Date: Sun, 22 Mar 2009 18:37:32 +0200 (EET) Subject: sendmail and cyrus-imap - delivery problem Message-ID: <1945400308.152765.1237739852087.JavaMail.apache@mail52.abv.bg> Hi, thanks for the response. I did what you suggested I defined new mailer like this: # cat /usr/share/sendmail/cf/mailer/cyrusv2vd.m4 PUSHDIVERT(-1) # # Copyright (c) 2002 Sendmail, Inc. and its suppliers. # All rights reserved. # # By using this file, you agree to the terms and conditions set # forth in the LICENSE file which can be found at the top level of # the sendmail distribution. # # Contributed by Kenneth Murchison. # _DEFIFNOT(`_DEF_CYRUSV2_MAILER_FLAGS', `lsDFMnqXz') _DEFIFNOT(`CYRUSV2_MAILER_FLAGS', `A@/:|m') ifdef(`CYRUSV2_MAILER_ARGS',, `define(`CYRUSV2_MAILER_ARGS', `FILE /var/imap/socket/lmtp')') define(`_CYRUSV2_QGRP', `ifelse(defn(`CYRUSV2_MAILER_QGRP'),`',`', ` Q=CYRUSV2_MAILER_QGRP,')')dnl POPDIVERT ######################################### ### Cyrus V2 Mailer specification ### ######################################### VERSIONID(`$Id: cyrusv2vd.m4,v 1.1 2002/06/01 21:14:57 ca Exp $') Mcyrusv2vd, P=[IPC], F=_MODMF_(CONCAT(_DEF_CYRUSV2_MAILER_FLAGS, CYRUSV2_MAILER_FLAGS), `CYRUSV2'), S=EnvFromSMTP/HdrFromSMTP, R=EnvToSMTP, E=\r\n, _OPTINS(`CYRUSV2_MAILER_MAXMSGS', `m=', `, ')_OPTINS(`CYRUSV2_MAILER_MAXRCPTS', `r=', `, ')_OPTINS(`CYRUSV2_MAILER_CHARSET', `C=', `, ')T=DNS/RFC822/SMTP,_CYRUSV2_QGRP A=CYRUSV2_MAILER_ARGS which is of course copy of cyrusv2.m4 with the modifications you suggested and in my yo.com.mc file I just replaced all occurances of cyrusv2 with cyrusv2vd then I rebuilt sendmail.cf and restarted sendmail it seemed to accept the new mailer however the result is exactly the same Mar 22 16:28:22 yo sm-mta[97935]: n2MGSLe6097935: from=, size=1575, class=0, nrcpts=1, msgid=<9cddded00903220928i7a2a8fcfkc2fd206491021847 at mail.gmail.com>, proto=ESMTP, daemon=IPv4, relay=mail-fx0-f163.google.com [209.85.220.163] Mar 22 16:28:22 yo sm-mta[97937]: n2MGSLe6097935: to=, delay=00:00:00, xdelay=00:00:00, mailer=cyrusv2vd, pri=31575, relay=localhost, dsn=5.1.1, stat=User unknown Mar 22 16:28:22 yo sm-mta[97937]: n2MGSLe6097935: n2MGSMe6097937: DSN: User unknown I tried this: # ( echo subject: test ; echo testmsg ) | sendmail -v -Am boo at yo.com pesho at e-soul.org... Connecting to /var/imap/socket/lmtp via cyrusv2vd... 220 e-soul.org Cyrus LMTP v2.3.13 server ready >>> LHLO yo.com 250-yo.com 250-8BITMIME 250-ENHANCEDSTATUSCODES 250-PIPELINING 250-SIZE 250-AUTH EXTERNAL 250 IGNOREQUOTA >>> MAIL From: SIZE=22 AUTH=moo at yo.com 250 2.1.0 ok >>> RCPT To: >>> DATA 550-Mailbox unknown. Either there is no mailbox associated with this 550-name or you do not have authorization to see it. 550 5.1.1 User unknown 503 5.5.1 No recipients >>> RSET 250 2.0.0 ok >>> RSET 250 2.0.0 ok mgp... Using cached LMTP connection to localhost via cyrusv2vd... >>> MAIL From:<> SIZE=1046 250 2.1.0 ok >>> RCPT To: >>> DATA 550-Mailbox unknown. Either there is no mailbox associated with this 550-name or you do not have authorization to see it. 550 5.1.1 User unknown 503 5.5.1 No recipients >>> RSET 250 2.0.0 ok >>> RSET 250 2.0.0 ok postmaster... Using cached LMTP connection to localhost via cyrusv2vd... >>> MAIL From:<> SIZE=2070 250 2.1.0 ok >>> RCPT To: >>> DATA 550-Mailbox unknown. Either there is no mailbox associated with this 550-name or you do not have authorization to see it. 550 5.1.1 User unknown 503 5.5.1 No recipients >>> RSET 250 2.0.0 ok Closing connection to localhost >>> QUIT 221 2.0.0 bye it now seems like sendmail doesn't strip the domain but cyrus still can't "find" the mailbox yes, the mailbox is there I check this over and over and it's always there any quick suggestions before I try rtcyrus3 ? thank you. >Mario Pavlov wrote: > >MP> I've setup sendmail 8.14.3 to work with cyrus-imapd 2.3.13 on >MP> FreeBSD 7-STABLE but I have a delivery issue > >> localhost> lm >> user/boo at yo.com (\HasNoChildren) > >> Mar 21 09:55:57 yo sm-mta[3465]: n2L9tvmO003426: to=, >> delay=00:00:00, xdelay=00:00:00, mailer=cyrusv2, pri=31599, >> relay=localhost, dsn=5.1.1, stat=User unknown > >MP> cyrus knows about user/boo at yo.com >MP> but does sendmail know about it ? >MP> or it thinks it's just boo at yo.com > >MP> what do you think could be the problem ? > >I think your problem is that for Cyrus virtdomains to work the MTA needs >to send the _full_ address in the LMTP transaction. When Sendmail >speaks to Cyrus lmtpd, it needs to say > > RCPT TO: > >rather than just > > RCPT TO: > >Sendmail's definition of the cyrusv2 mailer >(/usr/share/sendmail/cf/mailer/cyrusv2.m4 on Debian, possibly elsewhere >on FreeBSD) sets "R=EnvToL/HdrToL", but for virtdomains you want >"R=EnvToSMTP". It's also recommended that you change >"S=EnvFromSMTP/HdrFromL" to "S=EnvFromSMTP/HdrFromSMTP". > >As a quick hack, you can change those in situ; or - more gracefully - >you can define a new mailer which works this way and use it instead. >The Open-Sendmail project has bells-and-whistles implementation, >cyrusv3.m4, within RTCyrus3: > > http://open-sendmail.sourceforge.net/rtcyrus3/ > >but it is quite complicated to install. > > > >> FEATURE(local_lmtp) >> MAILER(local) > >It's not clear to me what the above lines are achieving when you also >have the below: > >> define(`confLOCAL_MAILER', `cyrusv2') >> MAILER(`cyrusv2') > > >Cheers > > >Duncan > >-- >Duncan Gibb - Technical Director >Sirius Corporation plc - control through freedom >http://www.siriusit.co.uk/ || t: +44 870 608 0063 > From anfi at onet.eu Sun Mar 22 13:23:53 2009 From: anfi at onet.eu (Andrzej Adam Filip) Date: Sun, 22 Mar 2009 18:23:53 +0100 Subject: sendmail and cyrus-imap - delivery problem In-Reply-To: <1945400308.152765.1237739852087.JavaMail.apache@mail52.abv.bg> (Mario Pavlov's message of "Sun\, 22 Mar 2009 18\:37\:32 +0200 \(EET\)") References: <1945400308.152765.1237739852087.JavaMail.apache@mail52.abv.bg> Message-ID: Mario Pavlov wrote: > Hi, > thanks for the response. > > I did what you suggested > I defined new mailer like this: > > # cat /usr/share/sendmail/cf/mailer/cyrusv2vd.m4 > PUSHDIVERT(-1) > # > # Copyright (c) 2002 Sendmail, Inc. and its suppliers. > # All rights reserved. > # > # By using this file, you agree to the terms and conditions set > # forth in the LICENSE file which can be found at the top level of > # the sendmail distribution. > # > # Contributed by Kenneth Murchison. > # > > _DEFIFNOT(`_DEF_CYRUSV2_MAILER_FLAGS', `lsDFMnqXz') > _DEFIFNOT(`CYRUSV2_MAILER_FLAGS', `A@/:|m') > ifdef(`CYRUSV2_MAILER_ARGS',, `define(`CYRUSV2_MAILER_ARGS', `FILE /var/imap/socket/lmtp')') > define(`_CYRUSV2_QGRP', `ifelse(defn(`CYRUSV2_MAILER_QGRP'),`',`', ` Q=CYRUSV2_MAILER_QGRP,')')dnl > > POPDIVERT > > ######################################### > ### Cyrus V2 Mailer specification ### > ######################################### > > VERSIONID(`$Id: cyrusv2vd.m4,v 1.1 2002/06/01 21:14:57 ca Exp $') > > Mcyrusv2vd, P=[IPC], F=_MODMF_(CONCAT(_DEF_CYRUSV2_MAILER_FLAGS, CYRUSV2_MAILER_FLAGS), `CYRUSV2'), > S=EnvFromSMTP/HdrFromSMTP, R=EnvToSMTP, E=\r\n, > _OPTINS(`CYRUSV2_MAILER_MAXMSGS', `m=', `, ')_OPTINS(`CYRUSV2_MAILER_MAXRCPTS', `r=', `, ')_OPTINS(`CYRUSV2_MAILER_CHARSET', `C=', `, ')T=DNS/RFC822/SMTP,_CYRUSV2_QGRP > A=CYRUSV2_MAILER_ARGS > > > which is of course copy of cyrusv2.m4 with the modifications you suggested > and in my yo.com.mc file I just replaced all occurances of cyrusv2 with cyrusv2vd > then I rebuilt sendmail.cf and restarted sendmail > it seemed to accept the new mailer > however the result is exactly the same > > Mar 22 16:28:22 yo sm-mta[97935]: n2MGSLe6097935: from=, size=1575, class=0, nrcpts=1, msgid=<9cddded00903220928i7a2a8fcfkc2fd206491021847 at mail.gmail.com>, proto=ESMTP, daemon=IPv4, relay=mail-fx0-f163.google.com [209.85.220.163] > Mar 22 16:28:22 yo sm-mta[97937]: n2MGSLe6097935: to=, delay=00:00:00, xdelay=00:00:00, mailer=cyrusv2vd, pri=31575, relay=localhost, dsn=5.1.1, stat=User unknown > Mar 22 16:28:22 yo sm-mta[97937]: n2MGSLe6097935: n2MGSMe6097937: DSN: User unknown > > I tried this: > # ( echo subject: test ; echo testmsg ) | sendmail -v -Am boo at yo.com > pesho at e-soul.org... Connecting to /var/imap/socket/lmtp via cyrusv2vd... > 220 e-soul.org Cyrus LMTP v2.3.13 server ready >>>> LHLO yo.com > 250-yo.com > 250-8BITMIME > 250-ENHANCEDSTATUSCODES > 250-PIPELINING > 250-SIZE > 250-AUTH EXTERNAL > 250 IGNOREQUOTA >>>> MAIL From: SIZE=22 AUTH=moo at yo.com > 250 2.1.0 ok >>>> RCPT To: >>>> DATA > 550-Mailbox unknown. Either there is no mailbox associated with this > 550-name or you do not have authorization to see it. > 550 5.1.1 User unknown > 503 5.5.1 No recipients >>>> RSET > 250 2.0.0 ok >>>> RSET > 250 2.0.0 ok > mgp... Using cached LMTP connection to localhost via cyrusv2vd... >>>> MAIL From:<> SIZE=1046 > 250 2.1.0 ok >>>> RCPT To: >>>> DATA > 550-Mailbox unknown. Either there is no mailbox associated with this > 550-name or you do not have authorization to see it. > 550 5.1.1 User unknown > 503 5.5.1 No recipients >>>> RSET > 250 2.0.0 ok >>>> RSET > 250 2.0.0 ok > postmaster... Using cached LMTP connection to localhost via cyrusv2vd... >>>> MAIL From:<> SIZE=2070 > 250 2.1.0 ok >>>> RCPT To: >>>> DATA > 550-Mailbox unknown. Either there is no mailbox associated with this > 550-name or you do not have authorization to see it. > 550 5.1.1 User unknown > 503 5.5.1 No recipients >>>> RSET > 250 2.0.0 ok > Closing connection to localhost >>>> QUIT > 221 2.0.0 bye > > it now seems like sendmail doesn't strip the domain > but cyrus still can't "find" the mailbox It seems that sendmail strips yo.com domain and appends yoo.com domain. Sendmail strips domain before selecting local mailer or its substitute. You have defined "with domain" rules in you mailer, so sendmail appends "this host email name" ($j) as the domain to "domainless recipients". > yes, the mailbox is there > I check this over and over > and it's always there > > any quick suggestions before I try rtcyrus3 ? > > thank you. > [...] -- [pl>en: Andrew] Andrzej Adam Filip : anfi at onet.eu What's love but a second-hand emotion? -- Tina Turner From freebsd at abv.bg Sun Mar 22 13:58:04 2009 From: freebsd at abv.bg (Mario Pavlov) Date: Sun, 22 Mar 2009 19:58:04 +0200 (EET) Subject: sendmail and cyrus-imap - delivery problem Message-ID: <1919513193.152548.1237744684791.JavaMail.apache@mail53.abv.bg> Hi, please excuse me I totally messed up this output...because of my pathetic try to substitute yo.com with my real domain this output should be read as follows: # ( echo subject: test ; echo testmsg ) | sendmail -v -Am boo at yo.com boo at yo.com... Connecting to /var/imap/socket/lmtp via cyrusv2vd... 220 yo.com Cyrus LMTP v2.3.13 server ready >>> LHLO yo.com 250-yo.com 250-8BITMIME 250-ENHANCEDSTATUSCODES 250-PIPELINING 250-SIZE 250-AUTH EXTERNAL 250 IGNOREQUOTA >>> MAIL From: SIZE=22 AUTH=moo at yo.com 250 2.1.0 ok >>> RCPT To: >>> DATA 550-Mailbox unknown. Either there is no mailbox associated with this 550-name or you do not have authorization to see it. 550 5.1.1 User unknown 503 5.5.1 No recipients >>> RSET 250 2.0.0 ok >>> RSET 250 2.0.0 ok mgp... Using cached LMTP connection to localhost via cyrusv2vd... >>> MAIL From:<> SIZE=1046 250 2.1.0 ok >>> RCPT To: >>> DATA 550-Mailbox unknown. Either there is no mailbox associated with this 550-name or you do not have authorization to see it. 550 5.1.1 User unknown 503 5.5.1 No recipients >>> RSET 250 2.0.0 ok >>> RSET 250 2.0.0 ok postmaster... Using cached LMTP connection to localhost via cyrusv2vd... >>> MAIL From:<> SIZE=2070 250 2.1.0 ok >>> RCPT To: >>> DATA 550-Mailbox unknown. Either there is no mailbox associated with this 550-name or you do not have authorization to see it. 550 5.1.1 User unknown 503 5.5.1 No recipients >>> RSET 250 2.0.0 ok Closing connection to localhost >>> QUIT 221 2.0.0 bye sorry ... again thank you. >Mario Pavlov wrote: > >> Hi, >> thanks for the response. >> >> I did what you suggested >> I defined new mailer like this: >> >> # cat /usr/share/sendmail/cf/mailer/cyrusv2vd.m4 >> PUSHDIVERT(-1) >> # >> # Copyright (c) 2002 Sendmail, Inc. and its suppliers. >> # All rights reserved. >> # >> # By using this file, you agree to the terms and conditions set >> # forth in the LICENSE file which can be found at the top level of >> # the sendmail distribution. >> # >> # Contributed by Kenneth Murchison. >> # >> >> _DEFIFNOT(`_DEF_CYRUSV2_MAILER_FLAGS', `lsDFMnqXz') >> _DEFIFNOT(`CYRUSV2_MAILER_FLAGS', `A@/:|m') >> ifdef(`CYRUSV2_MAILER_ARGS',, `define(`CYRUSV2_MAILER_ARGS', `FILE /var/imap/socket/lmtp')') >> define(`_CYRUSV2_QGRP', `ifelse(defn(`CYRUSV2_MAILER_QGRP'),`',`', ` Q=CYRUSV2_MAILER_QGRP,')')dnl >> >> POPDIVERT >> >> ######################################### >> ### Cyrus V2 Mailer specification ### >> ######################################### >> >> VERSIONID(`$Id: cyrusv2vd.m4,v 1.1 2002/06/01 21:14:57 ca Exp $') >> >> Mcyrusv2vd, P=[IPC], F=_MODMF_(CONCAT(_DEF_CYRUSV2_MAILER_FLAGS, CYRUSV2_MAILER_FLAGS), `CYRUSV2'), >> S=EnvFromSMTP/HdrFromSMTP, R=EnvToSMTP, E=\r\n, >> _OPTINS(`CYRUSV2_MAILER_MAXMSGS', `m=', `, ')_OPTINS(`CYRUSV2_MAILER_MAXRCPTS', `r=', `, ')_OPTINS(`CYRUSV2_MAILER_CHARSET', `C=', `, ')T=DNS/RFC822/SMTP,_CYRUSV2_QGRP >> A=CYRUSV2_MAILER_ARGS >> >> >> which is of course copy of cyrusv2.m4 with the modifications you suggested >> and in my yo.com.mc file I just replaced all occurances of cyrusv2 with cyrusv2vd >> then I rebuilt sendmail.cf and restarted sendmail >> it seemed to accept the new mailer >> however the result is exactly the same >> >> Mar 22 16:28:22 yo sm-mta[97935]: n2MGSLe6097935: from=, size=1575, class=0, nrcpts=1, msgid=, proto=ESMTP, daemon=IPv4, relay=mail-fx0-f163.google.com [209.85.220.163] >> Mar 22 16:28:22 yo sm-mta[97937]: n2MGSLe6097935: to=, delay=00:00:00, xdelay=00:00:00, mailer=cyrusv2vd, pri=31575, relay=localhost, dsn=5.1.1, stat=User unknown >> Mar 22 16:28:22 yo sm-mta[97937]: n2MGSLe6097935: n2MGSMe6097937: DSN: User unknown >> >> I tried this: >> # ( echo subject: test ; echo testmsg ) | sendmail -v -Am boo at yo.com >> pesho at e-soul.org... Connecting to /var/imap/socket/lmtp via cyrusv2vd... >> 220 e-soul.org Cyrus LMTP v2.3.13 server ready >>>>> LHLO yo.com >> 250-yo.com >> 250-8BITMIME >> 250-ENHANCEDSTATUSCODES >> 250-PIPELINING >> 250-SIZE >> 250-AUTH EXTERNAL >> 250 IGNOREQUOTA >>>>> MAIL From: SIZE=22 AUTH=moo at yo.com >> 250 2.1.0 ok >>>>> RCPT To: >>>>> DATA >> 550-Mailbox unknown. Either there is no mailbox associated with this >> 550-name or you do not have authorization to see it. >> 550 5.1.1 User unknown >> 503 5.5.1 No recipients >>>>> RSET >> 250 2.0.0 ok >>>>> RSET >> 250 2.0.0 ok >> mgp... Using cached LMTP connection to localhost via cyrusv2vd... >>>>> MAIL From: SIZE=1046 >> 250 2.1.0 ok >>>>> RCPT To: >>>>> DATA >> 550-Mailbox unknown. Either there is no mailbox associated with this >> 550-name or you do not have authorization to see it. >> 550 5.1.1 User unknown >> 503 5.5.1 No recipients >>>>> RSET >> 250 2.0.0 ok >>>>> RSET >> 250 2.0.0 ok >> postmaster... Using cached LMTP connection to localhost via cyrusv2vd... >>>>> MAIL From: SIZE=2070 >> 250 2.1.0 ok >>>>> RCPT To: >>>>> DATA >> 550-Mailbox unknown. Either there is no mailbox associated with this >> 550-name or you do not have authorization to see it. >> 550 5.1.1 User unknown >> 503 5.5.1 No recipients >>>>> RSET >> 250 2.0.0 ok >> Closing connection to localhost >>>>> QUIT >> 221 2.0.0 bye >> >> it now seems like sendmail doesn't strip the domain >> but cyrus still can't "find" the mailbox > >It seems that sendmail strips yo.com domain and appends yoo.com domain. >Sendmail strips domain before selecting local mailer or its substitute. >You have defined "with domain" rules in you mailer, so sendmail appends >"this host email name" ($j) as the domain to "domainless recipients". > >> yes, the mailbox is there >> I check this over and over >> and it's always there >> >> any quick suggestions before I try rtcyrus3 ? >> >> thank you. >> [...] > >-- >[pl>en: Andrew] Andrzej Adam Filip : anfi at onet.eu >What's love but a second-hand emotion? > -- Tina Turner >---- >Cyrus Home Page: http://cyrusimap.web.cmu.edu/ >Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki >List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > From freebsd at abv.bg Mon Mar 23 03:12:56 2009 From: freebsd at abv.bg (Mario Pavlov) Date: Mon, 23 Mar 2009 09:12:56 +0200 (EET) Subject: sendmail and cyrus-imap - delivery problem Message-ID: <913847928.160387.1237792376616.JavaMail.apache@mail52.abv.bg> Hi, thanks again to all of you for your time I've tried all that you suggested I've tried ckuser_cyrus as Matthew Seaman suggested...it made no difference then I've tried rtcyrus3 as Andrzej Adam Filip suggested...it didn't work either...and the logging output in maillog stopped...but I still get this e-mail in gmail.com =========================================================================================================== This is an automatically generated Delivery Status Notification Delivery to the following recipient failed permanently: boo at yo.com Technical details of permanent failure: Google tried to deliver your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 553 553 5.1.2 ... Unrecognized host name gmail.com. (state 13). ========================================================================================================= I'm sure it's all my fault I'm doing something wrong...or somethings wrong only I don't know what it is :) so this is my last yo.com.mc ========================================================================================================= VERSIONID(`$FreeBSD: src/etc/sendmail/freebsd.mc,v 1.34.2.3 2008/08/31 18:26:27 gshapiro Exp $') OSTYPE(freebsd6) DOMAIN(generic) FEATURE(access_db, `hash -o -T /etc/mail/access') FEATURE(blacklist_recipients) FEATURE(local_lmtp) FEATURE(mailertable, `hash -o /etc/mail/mailertable') FEATURE(virtusertable, `hash -o /etc/mail/virtusertable') FEATURE(`ckuser_cyrus')dnl FEATURE(`preserve_local_plus_detail')dnl DAEMON_OPTIONS(`Name=IPv4, Family=inet') define(`confBIND_OPTS', `WorkAroundBrokenAAAA') define(`confNO_RCPT_ACTION', `add-to-undisclosed') define(`confPRIVACY_FLAGS', `authwarnings,noexpn,novrfy') TRUST_AUTH_MECH(`GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN')dnl define(`confAUTH_MECHANISMS', `GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN')dnl dnl $Id: MC.rtcyrus3,v 1.1 2007/09/18 20:39:10 anfi Exp $ LOCAL_CONFIG # list of virtual cyrus domains C{VCyrusDomains} yo.com gl.com divert(-1) dnl # socket map socket location serviced by cyrus dnl define(`SMMAP_SOCKET',`/var/imap/socket/smmapd')dnl dnl # lmtp socket location serviced by cyrus dnl define(`CYRUS_LMTP_SOCKET',`/var/imap/socket/lmtp')dnl FEATURE(`anfi/vcyrus')dnl MAILER(local) MAILER(smtp) MAILER(`anfi/cyrusv3')dnl ========================================================================================================= I've added smmap in cyrus services sendmail is compiled with socketmap # sendmail -d0.1 Version 8.14.3 Compiled with: DNSMAP LOG MAP_REGEX MATCHGECOS MILTER MIME7TO8 MIME8TO7 NAMED_BIND NETINET NETINET6 NETUNIX NEWDB NIS PIPELINING SASLv2 SCANF SOCKETMAP STARTTLS TCPWRAPPERS USERDB XDEBUG I wanted cyrus logging but I couldn't configure it... I've added: local6.debug /var/log/imapd.log auth.debug /var/log/auth.log to syslog.conf but cyrus doesn't write to these files (privileges are correct) do I need to do something else to make cyrus log ? I know I'm hopeless :) I don't want to waste mroe your time with that stuff, you already helped me a lot but still...if you feel like digging in that more I'll be grateful thank you. >Mario Pavlov wrote: > >> Hi, >> thanks for your response >> you're right >> here's the output: >> >> # (echo subject: test; echo) | sendmail -v -Am boo at yo.com >> pesho at e-soul.org... Connecting to /var/imap/socket/lmtp via cyrusv2... >> 220 yo.com Cyrus LMTP v2.3.13 server ready >>>>> LHLO yo.com >> 250-yo.com >> 250-8BITMIME >> 250-ENHANCEDSTATUSCODES >> 250-PIPELINING >> 250-SIZE >> 250-AUTH EXTERNAL >> 250 IGNOREQUOTA >>>>> MAIL From: SIZE=15 AUTH=moo at yo.com >> 250 2.1.0 ok >>>>> RCPT To: >>>>> DATA >> 550-Mailbox unknown. Either there is no mailbox associated with this >> 550-name or you do not have authorization to see it. >> 550 5.1.1 User unknown >> 503 5.5.1 No recipients >>>>> RSET >> 250 2.0.0 ok >> >> what can I do to make sendmail not strip the domain ? >> >> thank you. > >I suggest using RTCyrus3 recipe. It is not the simplest approach but it >allows to make sendmail reject messages to non existing cyrus mailboxes >in reply to "RCPT TO:" => your sendmail does not send "bounce" messages >in reply to messages/spam sent to non exiting mailboxes. > >RTCyrus3 requires patching cf/m4/proto.m4 and generating new sendmail.cf but >no recompilation of sendmail binaries if they are compiled with socket >map support. >See "compiled with" section produced by the test command below to find >out if your sendmail is compiled with socketmap support >sendmail -bv root -d0.1 | grep -i socketmap > >http://open-sendmail.sourceforge.net/rtcyrus3/ > >-- >[pl>en: Andrew] Andrzej Adam Filip : anfi at onet.eu >I want to buy a husband who, every week when I sit down to watch "St. >Elsewhere", won't scream, "Forget it, Blanche... It's time for Hee-Haw!" > -- Berke Breathed, "Bloom County" > From paul at vandervlis.nl Mon Mar 23 09:31:29 2009 From: paul at vandervlis.nl (Paul van der Vlis) Date: Mon, 23 Mar 2009 14:31:29 +0100 Subject: autocreatequota Message-ID: <49C78F31.7010102@vandervlis.nl> Hello, I read in "The book of IMAP" the following about the autocreatequota option in imapd.conf: ----- If you enter a value other than 0, Cyrus will automatically create mailboxes for new users when they first log in, and then limit de memory available to these mailboxes to the quota value you specify here. (...) ----- I have tried that, but no mailbox was autocreated ;-( In de man-page of imapd.conf I see the following: ----- autocreatequota: 0 If nonzero, normal users may create their own IMAP accounts by creating the mailbox INBOX. The user's quota is set to the value if it is positive, otherwise the user has unlimited quota. ------ Does autocreation of mailboxes only work with the autocreate-patch? I am using Cyrus 2.2.13. With regards, Paul van der Vlis. -- http://www.vandervlis.nl/ From simon.matter at invoca.ch Mon Mar 23 09:34:47 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Mon, 23 Mar 2009 14:34:47 +0100 (CET) Subject: autocreatequota In-Reply-To: <49C78F31.7010102@vandervlis.nl> References: <49C78F31.7010102@vandervlis.nl> Message-ID: <246b4688649d279678511de033eba5eb.squirrel@webmail.bi.corp.invoca.ch> > Hello, > > I read in "The book of IMAP" the following about the autocreatequota > option in imapd.conf: > ----- > If you enter a value other than 0, Cyrus will automatically create > mailboxes for new users when they first log in, and then limit de memory > available to these mailboxes to the quota value you specify here. > (...) > ----- > > I have tried that, but no mailbox was autocreated ;-( > > In de man-page of imapd.conf I see the following: > ----- > autocreatequota: 0 > If nonzero, normal users may create their own IMAP accounts by creating > the mailbox INBOX. The user's quota is set to the value if it is > positive, otherwise the user has unlimited quota. > ------ > > Does autocreation of mailboxes only work with the autocreate-patch? Yes, without the autocreate patch the functionality is not there. Simon > I am using Cyrus 2.2.13. > > With regards, > Paul van der Vlis. > > > > > -- > http://www.vandervlis.nl/ > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > From anfi at onet.eu Mon Mar 23 17:48:09 2009 From: anfi at onet.eu (Andrzej Adam Filip) Date: Mon, 23 Mar 2009 22:48:09 +0100 Subject: sendmail and cyrus-imap - delivery problem In-Reply-To: <913847928.160387.1237792376616.JavaMail.apache@mail52.abv.bg> (Mario Pavlov's message of "Mon\, 23 Mar 2009 09\:12\:56 +0200 \(EET\)") References: <913847928.160387.1237792376616.JavaMail.apache@mail52.abv.bg> Message-ID: Mario Pavlov wrote: > thanks again to all of you for your time I've tried all that you > suggested I've tried ckuser_cyrus as Matthew Seaman suggested...it > made no difference then I've tried rtcyrus3 as Andrzej Adam Filip > suggested...it didn't work either...and the logging output in maillog > stopped.. [...] First: Use cyradm to recheck that cyrus reports mailbox named as in "RCPT TO:" of LMTP session. As I understand you have achieved that sendmail uses correct "mailbox name with domain" in LMTP session but Cyrus rejects the recipient. -- [pl>en: Andrew] Andrzej Adam Filip : anfi at onet.eu Life without caffeine is stimulating enough. -- Sanka Ad From dbosso+lists.cyrus at lsit.ucsb.edu Mon Mar 23 19:08:47 2009 From: dbosso+lists.cyrus at lsit.ucsb.edu (David R Bosso) Date: Mon, 23 Mar 2009 16:08:47 -0700 Subject: unexpunge segfaults with -l on some mailboxes In-Reply-To: <49BA5D3D.4080509@ednet.ns.ca> References: <49B7B296.6000305@ednet.ns.ca> <49B7DB6C.9080602@ednet.ns.ca> <20090312000706.GA7208@brong.net> <49B8FB4F.9050806@ednet.ns.ca> <20090312213933.GC7655@brong.net> <10173_1236898088_n2CMm7nQ017919_49B99125.9020508@ednet.ns.ca> <49BA5D3D.4080509@ednet.ns.ca> Message-ID: --On March 13, 2009 10:18:53 AM -0300 Patrick Boutilier wrote: >> This will be real easy to test though. I will just run ipurge on a >> subfolder of my mailbox and see if it corrupts it. :-) FWIW we see the same corruption here with ipurge. We use ipurge to clean out old messages from a spam box for each user. We've worked around it currently by reconstructing the box after the ipurge which adds a lot of work to our nightly maintenance. I'd love to see a fix for this. -David > I can duplicate the corruption by following steps similar to this report: > > http://archives.devshed.com/forums/networking-100/cyr-expire-sigsegv-2146 > 878.html > > > Current directory here is /var/imap/metadata/G/user/boutilpj/Testmail > > > Clean mailbox: > > > [cyrus at student2 Testmail]$ ll > total 60 > -rw------- 1 cyrus cyrus 49612 Mar 13 10:03 cyrus.cache > -rw------- 1 cyrus cyrus 189 Mar 13 10:03 cyrus.header > -rw------- 1 cyrus cyrus 3792 Mar 13 10:03 cyrus.index > > > Quick way to show number of messages in the folder: > > > > [cyrus at student2 Testmail]$ cat cyrus.cache |grep -a Prec|wc -l > 42 > > > Delete 11 messages through IMAP client: > > [cyrus at student2 Testmail]$ ll > total 64 > -rw------- 1 cyrus cyrus 49612 Mar 13 10:03 cyrus.cache > -rw------- 1 cyrus cyrus 1064 Mar 13 10:05 cyrus.expunge > -rw------- 1 cyrus cyrus 189 Mar 13 10:03 cyrus.header > -rw------- 1 cyrus cyrus 2824 Mar 13 10:05 cyrus.index > > > Run ipurge with a number high enough not to delete anything. I purge > only sees 31 messages and removes them from cyrus.cache: > > > [cyrus at student2 Testmail]$ /usr/local/cyrus/bin/ipurge -f -d 2333 > user.boutilpj.Testmail > Working on user.boutilpj.Testmail... > total messages 31 > total bytes 539400 > Deleted messages 0 > Deleted bytes 0 > Remaining messages 31 > Remaining bytes 539400 > > [cyrus at student2 Testmail]$ ll > total 48 > -rw------- 1 cyrus cyrus 36652 Mar 13 10:08 cyrus.cache > -rw------- 1 cyrus cyrus 1064 Mar 13 10:05 cyrus.expunge > -rw------- 1 cyrus cyrus 189 Mar 13 10:03 cyrus.header > -rw------- 1 cyrus cyrus 2824 Mar 13 10:08 cyrus.index > > [cyrus at student2 Testmail]$ cat cyrus.cache |grep -a Prec|wc -l > 31 > > > Run cyr_expire and corruption occurs: > > [cyrus at student2 Testmail]$ /usr/local/cyrus/bin/cyr_expire -E 333 -X 7 -p > user.boutilpj.Testmail -v > Expunging deleted messages in mailboxes older than 7 days > > Expunged 0 out of 33 messages from 1 mailboxes > [cyrus at student2 Testmail]$ ll > total 96 > -rw------- 1 cyrus cyrus 36652 Mar 13 10:08 cyrus.cache > -rw------- 1 cyrus cyrus 37852 Mar 13 10:15 cyrus.cache.NEW > -rw------- 1 cyrus cyrus 1064 Mar 13 10:05 cyrus.expunge > -rw------- 1 cyrus cyrus 272 Mar 13 10:15 cyrus.expunge.NEW > -rw------- 1 cyrus cyrus 189 Mar 13 10:03 cyrus.header > -rw------- 1 cyrus cyrus 2824 Mar 13 10:08 cyrus.index > -rw------- 1 cyrus cyrus 2824 Mar 13 10:15 cyrus.index.NEW > > > Mar 13 10:15:40 student2 cyr_expire[8725]: IOERROR: reading cache record > for user.boutilpj.Testmail: item 1 has bogus offset 576450680 of 36652 > for 2/11; mailbox needs a reconstruct > > > > >> >>> Bron. >> >> ---- >> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ >> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki >> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > From brong at fastmail.fm Tue Mar 24 01:34:28 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 24 Mar 2009 16:34:28 +1100 Subject: unexpunge segfaults with -l on some mailboxes In-Reply-To: References: <49B7B296.6000305@ednet.ns.ca> <49B7DB6C.9080602@ednet.ns.ca><20090312000706.GA7208@brong.net> <49B8FB4F.9050806@ednet.ns.ca><20090312213933.GC7655@brong.net><10173_1236898088_n2CMm7nQ017919_49B99125.9020508@ednet.ns.ca><49BA5D3D.4080509@ednet.ns.ca> Message-ID: <1237872868.30591.1306977233@webmail.messagingengine.com> https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3145 (just copied and pasted what's below) And I've assigned it to myself so I don't forget to look at it! (this may wind up depending on all my other cache work, because I don't want to create too many conflicts, and besides - it's much nicer to work in the refactored APIs) On Mon, 23 Mar 2009 16:08 -0700, "David R Bosso" wrote: > --On March 13, 2009 10:18:53 AM -0300 Patrick Boutilier > wrote: > > > > >> This will be real easy to test though. I will just run ipurge on a > >> subfolder of my mailbox and see if it corrupts it. :-) > > FWIW we see the same corruption here with ipurge. We use ipurge to clean > out old messages from a spam box for each user. We've worked around it > currently by reconstructing the box after the ipurge which adds a lot of > work to our nightly maintenance. > > I'd love to see a fix for this. > > -David > > > I can duplicate the corruption by following steps similar to this report: > > > > http://archives.devshed.com/forums/networking-100/cyr-expire-sigsegv-2146 > > 878.html > > > > > > Current directory here is /var/imap/metadata/G/user/boutilpj/Testmail > > > > > > Clean mailbox: > > > > > > [cyrus at student2 Testmail]$ ll > > total 60 > > -rw------- 1 cyrus cyrus 49612 Mar 13 10:03 cyrus.cache > > -rw------- 1 cyrus cyrus 189 Mar 13 10:03 cyrus.header > > -rw------- 1 cyrus cyrus 3792 Mar 13 10:03 cyrus.index > > > > > > Quick way to show number of messages in the folder: > > > > > > > > [cyrus at student2 Testmail]$ cat cyrus.cache |grep -a Prec|wc -l > > 42 > > > > > > Delete 11 messages through IMAP client: > > > > [cyrus at student2 Testmail]$ ll > > total 64 > > -rw------- 1 cyrus cyrus 49612 Mar 13 10:03 cyrus.cache > > -rw------- 1 cyrus cyrus 1064 Mar 13 10:05 cyrus.expunge > > -rw------- 1 cyrus cyrus 189 Mar 13 10:03 cyrus.header > > -rw------- 1 cyrus cyrus 2824 Mar 13 10:05 cyrus.index > > > > > > Run ipurge with a number high enough not to delete anything. I purge > > only sees 31 messages and removes them from cyrus.cache: > > > > > > [cyrus at student2 Testmail]$ /usr/local/cyrus/bin/ipurge -f -d 2333 > > user.boutilpj.Testmail > > Working on user.boutilpj.Testmail... > > total messages 31 > > total bytes 539400 > > Deleted messages 0 > > Deleted bytes 0 > > Remaining messages 31 > > Remaining bytes 539400 > > > > [cyrus at student2 Testmail]$ ll > > total 48 > > -rw------- 1 cyrus cyrus 36652 Mar 13 10:08 cyrus.cache > > -rw------- 1 cyrus cyrus 1064 Mar 13 10:05 cyrus.expunge > > -rw------- 1 cyrus cyrus 189 Mar 13 10:03 cyrus.header > > -rw------- 1 cyrus cyrus 2824 Mar 13 10:08 cyrus.index > > > > [cyrus at student2 Testmail]$ cat cyrus.cache |grep -a Prec|wc -l > > 31 > > > > > > Run cyr_expire and corruption occurs: > > > > [cyrus at student2 Testmail]$ /usr/local/cyrus/bin/cyr_expire -E 333 -X 7 -p > > user.boutilpj.Testmail -v > > Expunging deleted messages in mailboxes older than 7 days > > > > Expunged 0 out of 33 messages from 1 mailboxes > > [cyrus at student2 Testmail]$ ll > > total 96 > > -rw------- 1 cyrus cyrus 36652 Mar 13 10:08 cyrus.cache > > -rw------- 1 cyrus cyrus 37852 Mar 13 10:15 cyrus.cache.NEW > > -rw------- 1 cyrus cyrus 1064 Mar 13 10:05 cyrus.expunge > > -rw------- 1 cyrus cyrus 272 Mar 13 10:15 cyrus.expunge.NEW > > -rw------- 1 cyrus cyrus 189 Mar 13 10:03 cyrus.header > > -rw------- 1 cyrus cyrus 2824 Mar 13 10:08 cyrus.index > > -rw------- 1 cyrus cyrus 2824 Mar 13 10:15 cyrus.index.NEW > > > > > > Mar 13 10:15:40 student2 cyr_expire[8725]: IOERROR: reading cache record > > for user.boutilpj.Testmail: item 1 has bogus offset 576450680 of 36652 > > for 2/11; mailbox needs a reconstruct > > > > > > > > > >> > >>> Bron. > >> > >> ---- > >> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > >> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > >> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > > > > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- Bron Gondwana brong at fastmail.fm From mhlavink at redhat.com Tue Mar 24 12:18:06 2009 From: mhlavink at redhat.com (Michal Hlavinka) Date: Tue, 24 Mar 2009 17:18:06 +0100 Subject: ctl_cyrusdb questions, looking for answers In-Reply-To: <200903121938.38709.mhlavink@redhat.com> References: <200903121938.38709.mhlavink@redhat.com> Message-ID: <200903241718.06147.mhlavink@redhat.com> Hi, thanks all for the answers. Some summary what I know (please correct me if I'm wrong): 1) checkpointing "Checkpointed" are this databases: quotas.db tls_sessions.db deliver.db statuschache.db annotations.db mailboxes.db ptclient/ptscache.db but only DBs with Berkeley DB-Engine it does something. So really checkpointed are: tls_sessions.db deliver.db statuschache.db ptclient/ptscache.db And checkpointing here only means sync and commit changes if there is a lot of them. 2)backuping Backuped are only this databases: mailboxes.db annotations.db quotas.db there is no need to backup deliver.db, statuscache.db, or tls_sessions.db. These can be rebuilt with no data loss. But what about ptclient/ptscache.db ? 3) "real" backup when I want to create "real" backup (for example for hdd failure) I need to backup only mailboxes.db annotations.db quotas.db from /var/lib/imap (or from /var/lib/imap/db.backup1 ?) or something else? 4)check corrupted db There is no tool I can use to check if database is corrupted. I can only check logs for errors (will cyrus-imapd terminate or will it work with corrupted database - so I can easily miss the error in the logs?). Can I use chk_cyrus for this somehow? 5) ctl_cyrusdb -r I'm not completely sure what is effect of ctl_cyrusdb -r (it's executed always when cyrus-imap is starting) This means that not committed changes (everything after last ctl_cyrusdb -c ) are thrown away? Or it's just check "everything is ok" ? 6) ctl_cyrusdb -r VS. ctl_cyrusdb -r -x When -x is specified it does nothing? See ctl_cyrusdb.c source: > switch (op) { > case RECOVER: > break; ... > if(op == RECOVER && reserve_flag) > recover_reserved(); reserve_flag = 1 only when there is no -x But there are also cyrus_init(alt_config, "ctl_cyrusdb", 0); in the beginning and cyrus_done(); at the end, so maybe they are doing something? Regards, Michal From awilliam at whitemice.org Tue Mar 24 12:46:09 2009 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Tue, 24 Mar 2009 12:46:09 -0400 Subject: ctl_cyrusdb questions, looking for answers In-Reply-To: <200903241718.06147.mhlavink@redhat.com> References: <200903121938.38709.mhlavink@redhat.com> <200903241718.06147.mhlavink@redhat.com> Message-ID: <1237913169.5731.10.camel@linux-m3mt> On Tue, 2009-03-24 at 17:18 +0100, Michal Hlavinka wrote: > Hi, > thanks all for the answers. Some summary what I know (please correct me if I'm > wrong): > 1) checkpointing > "Checkpointed" are this databases: > but only DBs with Berkeley DB-Engine it does something. So really checkpointed > are: > tls_sessions.db > deliver.db > statuschache.db > ptclient/ptscache.db Incorrect, the Berkeley DBs will be checkpointed. What databases are in which format is a site configuration issue. > And checkpointing here only means sync and commit changes if there is a lot of > them. For Berkley DBs generally yes - for details see the Berkeley DB documentation. Skiplist files do checkpoint [rewrite themselves] but not related to the checkpoint interval in cyrus.conf > 2)backuping > Backuped are only this databases: > mailboxes.db > annotations.db > quotas.db > there is no need to backup deliver.db, statuscache.db, or tls_sessions.db. > These can be rebuilt with no data loss. > But what about ptclient/ptscache.db ? ptscache.db can be discarded. It is only a "cache". > 3) "real" backup > when I want to create "real" backup (for example for hdd failure) I need to > backup only > mailboxes.db > annotations.db > quotas.db > from /var/lib/imap (or from /var/lib/imap/db.backup1 ?) or something else? I'd assume you also want user's SIEVE scripts. > 4)check corrupted db > There is no tool I can use to check if database is corrupted. I can only check > logs for errors (will cyrus-imapd terminate or will it work with corrupted > database - so I can easily miss the error in the logs?). Can I use chk_cyrus > for this somehow? You can export a database to a file format like flat. That is useful for consistency checking and for backups. > 5) ctl_cyrusdb -r > I'm not completely sure what is effect of ctl_cyrusdb -r (it's executed always > when cyrus-imap is starting) > This means that not committed changes (everything after last ctl_cyrusdb -c ) > are thrown away? Or it's just check "everything is ok" ? I believe this relates to Berkeley DB databases primarily. From freebsd at abv.bg Tue Mar 24 13:59:55 2009 From: freebsd at abv.bg (Mario Pavlov) Date: Tue, 24 Mar 2009 19:59:55 +0200 (EET) Subject: sendmail and cyrus-imap - delivery problem Message-ID: <65668074.212869.1237917595714.JavaMail.apache@mail54.abv.bg> Hi, I've discovered something odd (at least to me) when I first configured cyrus-imap I was unable to create mailboxes I kept getting this: localhost> cm user.boo at yo.com createmailbox: Permission denied then I read somewhere that I should set "unixhierarchysep" to "yes" and it will work so I did and it worked...I was able to create mailboxes localhost> cm user/boo at yo.com localhost> lm user/boo at yo.com (\HasNoChildren) and while I was struggling with my delivery problem these days I decided to set "unixhierarchysep" back to "no" just to see what happens so it was the same...I was unable to create mailboxes because of that "Permission denied" error but when I wrote "lm" I saw that: localhost> lm user.boo at yo^com (\HasNoChildren) you see the difference ? There is this sign "^" in the domain name instead of dot "." I checked: localhost> info user.boo at yo.com // <--- dot "." localhost> info user.boo at yo^com // <--- shift + 6 "^" {user.boo at yo^com}: condstore: false duplicatedeliver: false lastpop: lastupdate: 21-Mar-2009 08:56:34 +0000 partition: default sharedseen: false size: 0 so turns out "user.boo at yo.com" is not an existing mailbox, is it ? could this be my delivery problem ? is it possible cyrus not matching mailboxes because of this ? thank you. >Mario Pavlov wrote: >> thanks again to all of you for your time I've tried all that you >> suggested I've tried ckuser_cyrus as Matthew Seaman suggested...it >> made no difference then I've tried rtcyrus3 as Andrzej Adam Filip >> suggested...it didn't work either...and the logging output in maillog >> stopped.. [...] > >First: Use cyradm to recheck that cyrus reports mailbox named as in >"RCPT TO:" of LMTP session. > >As I understand you have achieved that sendmail uses correct >"mailbox name with domain" in LMTP session but Cyrus rejects >the recipient. > >-- >[pl>en: Andrew] Andrzej Adam Filip : anfi at onet.eu >Life without caffeine is stimulating enough. > -- Sanka Ad >---- >Cyrus Home Page: http://cyrusimap.web.cmu.edu/ >Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki >List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > From davidk at cs.umass.edu Tue Mar 24 20:55:35 2009 From: davidk at cs.umass.edu (David Korpiewski) Date: Tue, 24 Mar 2009 20:55:35 -0400 Subject: EMERGENCY - Mail reset! Message-ID: <49C98107.5010903@cs.umass.edu> I had a situation where I shut down the cyrus system and did a database reset (ran these steps) to fix a problem with LMPTD (it worked btw). # serveradmin stop mail $ sudo -u cyrusimap bash $ cd /var/imap/ $ rm db/* $ rm db.backup?/* $ rm deliver.db $ rm tls_sessions.db $ /usr/bin/cyrus/bin/ctl_mboxlist -d > mailboxes.txt $ mv mailboxes.db mailboxes.db.old $ /usr/bin/cyrus/bin/ctl_mboxlist -d < mailboxes.txt # serveradmin start mail If you noticed, the second to last step I left the "-d" instead of running "-u". This instead created a blank mailboxes.db file. The problem is that mail kept coming in between then and the five minutes that it took me to realize that I had screwed up. The problem is that everyone who received mail had their inbox reset so that their email numbers start back at "1.". The bigger problem is that they can't read their email that was in their inbox before the email number got set to 1. and some of these people had THOUSANDS of emails. Can anyone help me figure out how to fix this problem? It affects all my faculty in my department and they are pretty pissed off. Thanks David -- =========================================== David Korpiewski Software Specialist I CSCF - Computer Science Computing Facility Department of Computer Science Phone: 413-545-4319 Fax: 413-577-2285 =========================================== From davidk at cs.umass.edu Tue Mar 24 21:23:52 2009 From: davidk at cs.umass.edu (David Korpiewski) Date: Tue, 24 Mar 2009 21:23:52 -0400 Subject: EMERGENCY - Mail reset! In-Reply-To: <20090325010329.GB181@mail.me.com> References: <49C98107.5010903@cs.umass.edu> <20090325010329.GB181@mail.me.com> Message-ID: <49C987A8.1030306@cs.umass.edu> Actually it seems that running "/usr/bin/cyrus/bin/reconstruct -r -f user/username" bring back the mail but the seen information is gone. At least it is a step in the right direction... Mike Eggleston wrote: > On Tue, 24 Mar 2009, David Korpiewski might have said: > >> I had a situation where I shut down the cyrus system and did a database >> reset (ran these steps) to fix a problem with LMPTD (it worked btw). >> >> >> # serveradmin stop mail >> $ sudo -u cyrusimap bash >> $ cd /var/imap/ >> $ rm db/* >> $ rm db.backup?/* >> $ rm deliver.db >> $ rm tls_sessions.db >> $ /usr/bin/cyrus/bin/ctl_mboxlist -d > mailboxes.txt >> $ mv mailboxes.db mailboxes.db.old >> $ /usr/bin/cyrus/bin/ctl_mboxlist -d < mailboxes.txt >> # serveradmin start mail >> >> >> If you noticed, the second to last step I left the "-d" instead of >> running "-u". This instead created a blank mailboxes.db file. The >> problem is that mail kept coming in between then and the five minutes >> that it took me to realize that I had screwed up. >> >> The problem is that everyone who received mail had their inbox reset so >> that their email numbers start back at "1.". The bigger problem is that >> they can't read their email that was in their inbox before the email >> number got set to 1. and some of these people had THOUSANDS of emails. >> >> Can anyone help me figure out how to fix this problem? It affects all >> my faculty in my department and they are pretty pissed off. >> >> Thanks >> David > > Morning David, > > Would a ctl_cyrusdb -r to reconstruct the mailboxes fix the issue? > > Mike -- =========================================== David Korpiewski Software Specialist I CSCF - Computer Science Computing Facility Department of Computer Science Phone: 413-545-4319 Fax: 413-577-2285 =========================================== From rspell at bates.edu Wed Mar 25 08:40:54 2009 From: rspell at bates.edu (Robert Spellman) Date: Wed, 25 Mar 2009 08:40:54 -0400 Subject: What should be listening on tcp port 12345? Message-ID: <49CA2656.7010604@bates.edu> I'm in the process of adding another front end server to our cyrus email cluster. The new server is running RedHat ES5, and rpm shows the version of cyrus is cyrus-imapd-2.3.7. Our front ends provide imaps, smtp (via sendmail) and lmtp. Sendmail uses smmap to verify the recipients addrsess, using the process outlined within the RTCyrus3 project. lmtp then handles the delivery of the email to our backend servers. The backend servers are running cyrus-imapd-2.2.12. During testing of the new server, I noticed a significant delay during the delivery of email of up to 10 minutes. The delay seems to be caused by the smmap process while it attempts to connect to tcp port 12345 on the backend server. iptables on the backend server was configured to ignore attempts to connect on tcp port 12345. Changing iptables to reject connections to tcp port 12345 sped up the process considerably. What process is supposed to be listening on that port? Here's output from /var/log/maillog. mailstore03 is the correct backend server for my mailbox. Mar 24 14:41:25 postoffice04 smmapd[31605]: executed Mar 24 14:42:41 postoffice04 smmapd[31605]: accepted connection Mar 24 14:42:41 postoffice04 smmapd[31605]: verify_user(user.rspell) proxying to host mailstore03.bates.edu Mar 24 14:42:41 postoffice04 smmapd[31605]: verify_user(user.rspell) failed: can't connect to mailstore03.bates.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: rspell.vcf Type: text/x-vcard Size: 404 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090325/dcecccfb/attachment-0001.vcf -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 5751 bytes Desc: S/MIME Cryptographic Signature Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090325/dcecccfb/attachment-0001.bin From murch at andrew.cmu.edu Wed Mar 25 09:04:05 2009 From: murch at andrew.cmu.edu (Ken Murchison) Date: Wed, 25 Mar 2009 09:04:05 -0400 Subject: Cyrus IMAPd 2.3.14 Released Message-ID: <49CA2BC5.10902@andrew.cmu.edu> I am pleased to announce the release of Cyrus IMAPd 2.3.14. This release should be considered production quality. This is mostly a bugfix release. For full details, please see doc/changes.html and doc/install-upgrade.html which are included in the distribution. URLs for this release: ftp://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.3.14.tar.gz or http://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.3.14.tar.gz Questions and comments can be directed to info-cyrus at lists.andrew.cmu.edu (public list), or cyrus-bugs at andrew.cmu.edu. -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University From simon.matter at invoca.ch Wed Mar 25 10:40:55 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Wed, 25 Mar 2009 15:40:55 +0100 (CET) Subject: Cyrus IMAPd 2.3.14 Released In-Reply-To: <49CA2BC5.10902@andrew.cmu.edu> References: <49CA2BC5.10902@andrew.cmu.edu> Message-ID: <3f3206b0cd9c759173c5f5c3ed9e7ad1.squirrel@webmail.bi.corp.invoca.ch> > I am pleased to announce the release of Cyrus IMAPd 2.3.14. This > release should be considered production quality. This is mostly a > bugfix release. For full details, please see doc/changes.html and > doc/install-upgrade.html which are included in the distribution. > > URLs for this release: > ftp://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.3.14.tar.gz > or > http://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.3.14.tar.gz Thanks for the new release! My question goes mainly to Bron: I've included some of your fastmail.fm patches in my RPMS in the past. They have always been on http://cyrus.brong.fastmail.fm/ Now things have changed and those patches are no longer maintained there. Is there a way for me to get the same patches but for 2.3.14 and newer releases (for example the cyrus-clean-shutdown-2.3.13.diff patch)? I'm quite sure the code is somewhere on http://github.com/brong/cyrus-imapd/ but I don't know how to extract the patches from there? Any help is much appreciated. Regards, Simon From boutilpj at ednet.ns.ca Wed Mar 25 10:55:09 2009 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Wed, 25 Mar 2009 11:55:09 -0300 Subject: unexpunge segfaults with -l on some mailboxes In-Reply-To: References: <49B7B296.6000305@ednet.ns.ca> <49B7DB6C.9080602@ednet.ns.ca> <20090312000706.GA7208@brong.net> <49B8FB4F.9050806@ednet.ns.ca> <20090312213933.GC7655@brong.net> <10173_1236898088_n2CMm7nQ017919_49B99125.9020508@ednet.ns.ca> <49BA5D3D.4080509@ednet.ns.ca> Message-ID: <49CA45CD.40501@ednet.ns.ca> David R Bosso wrote: > --On March 13, 2009 10:18:53 AM -0300 Patrick Boutilier > wrote: > > > >>> This will be real easy to test though. I will just run ipurge on a >>> subfolder of my mailbox and see if it corrupts it. :-) > > FWIW we see the same corruption here with ipurge. We use ipurge to clean > out old messages from a spam box for each user. We've worked around it > currently by reconstructing the box after the ipurge which adds a lot of > work to our nightly maintenance. Plus you lose all the messages that are in "delayed expunge" state after running a reconstruct. :-( > > I'd love to see a fix for this. > > -David > -------------- next part -------------- A non-text attachment was scrubbed... Name: boutilpj.vcf Type: text/x-vcard Size: 286 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090325/79efd096/attachment.vcf From Pascal.Gienger at uni-konstanz.de Wed Mar 25 10:58:58 2009 From: Pascal.Gienger at uni-konstanz.de (Pascal Gienger) Date: Wed, 25 Mar 2009 15:58:58 +0100 Subject: unexpunge segfaults with -l on some mailboxes In-Reply-To: <49CA45CD.40501@ednet.ns.ca> References: <49B7B296.6000305@ednet.ns.ca> <49B7DB6C.9080602@ednet.ns.ca> <20090312000706.GA7208@brong.net> <49B8FB4F.9050806@ednet.ns.ca> <20090312213933.GC7655@brong.net> <10173_1236898088_n2CMm7nQ017919_49B99125.9020508@ednet.ns.ca> <49BA5D3D.4080509@ednet.ns.ca> <49CA45CD.40501@ednet.ns.ca> Message-ID: <49CA46B2.2080707@uni-konstanz.de> Patrick Boutilier schrieb: > Plus you lose all the messages that are in "delayed expunge" state after > running a reconstruct. :-( Just delete cyrus.expunge in the appropriate mailboxes meta directory before running reconstruct. From rjaffey at artic.edu Wed Mar 25 11:28:45 2009 From: rjaffey at artic.edu (Raphael Jaffey) Date: Wed, 25 Mar 2009 10:28:45 -0500 Subject: unexpunge segfaults with -l on some mailboxes In-Reply-To: <49CA45CD.40501@ednet.ns.ca> References: <49B7B296.6000305@ednet.ns.ca> <49B7DB6C.9080602@ednet.ns.ca> <20090312000706.GA7208@brong.net> <49B8FB4F.9050806@ednet.ns.ca> <20090312213933.GC7655@brong.net> <10173_1236898088_n2CMm7nQ017919_49B99125.9020508@ednet.ns.ca> <49BA5D3D.4080509@ednet.ns.ca> <49CA45CD.40501@ednet.ns.ca> Message-ID: <20090325102845.zxxczxll40gw08k0@webmail-test.artic.edu> Quoting Patrick Boutilier : > David R Bosso wrote: >> --On March 13, 2009 10:18:53 AM -0300 Patrick Boutilier >> wrote: >> >> >> >>>> This will be real easy to test though. I will just run ipurge on a >>>> subfolder of my mailbox and see if it corrupts it. :-) >> >> FWIW we see the same corruption here with ipurge. We use ipurge to >> clean out old messages from a spam box for each user. We've >> worked around it currently by reconstructing the box after the >> ipurge which adds a lot of work to our nightly maintenance. > > Plus you lose all the messages that are in "delayed expunge" state > after running a reconstruct. :-( Have you tried using '-k' when running reconstruct? > >> >> I'd love to see a fix for this. >> >> -David >> > From gavin.gray at ed.ac.uk Wed Mar 25 11:52:40 2009 From: gavin.gray at ed.ac.uk (Gavin Gray) Date: Wed, 25 Mar 2009 15:52:40 +0000 Subject: Cyrus IMAP Murder and (SUN)Solaris Cluster Message-ID: <20090325155240.4047h4f74so0w4ww@www.staffmail.ed.ac.uk> Hi there, We are running a cyrus IMAP murder for thousands of users with around 3TB of mail. We have a number of backends that each have attached storage for the mail store. We are currently using solaris 10 to run the murder on. We want to try and improve the resilience of our setup and are looking at a number of options to provide High Availability: 1. Using the latest version of cyrus to implement backend replication so that we have stand by spares for the murder if one of the backends go down. 2. Use Sun cluster to provide automatic failover for the backends somehow or another. 3. Some other solution we haven't thought of yet If anyone out there has though this kind of thing through or has experience of implementing Cyrus IMAP in a SUN Cluster environment, we'd love to hear from you, regards, Gavin. -- Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.gray at ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From boutilpj at ednet.ns.ca Wed Mar 25 11:54:21 2009 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Wed, 25 Mar 2009 12:54:21 -0300 Subject: unexpunge segfaults with -l on some mailboxes In-Reply-To: <20090325102845.zxxczxll40gw08k0@webmail-test.artic.edu> References: <49B7B296.6000305@ednet.ns.ca> <49B7DB6C.9080602@ednet.ns.ca> <20090312000706.GA7208@brong.net> <49B8FB4F.9050806@ednet.ns.ca> <20090312213933.GC7655@brong.net> <10173_1236898088_n2CMm7nQ017919_49B99125.9020508@ednet.ns.ca> <49BA5D3D.4080509@ednet.ns.ca> <49CA45CD.40501@ednet.ns.ca> <20090325102845.zxxczxll40gw08k0@webmail-test.artic.edu> Message-ID: <49CA53AD.7060903@ednet.ns.ca> Raphael Jaffey wrote: > Quoting Patrick Boutilier : > >> David R Bosso wrote: >>> --On March 13, 2009 10:18:53 AM -0300 Patrick Boutilier >>> wrote: >>> >>> >>> >>>>> This will be real easy to test though. I will just run ipurge on a >>>>> subfolder of my mailbox and see if it corrupts it. :-) >>> FWIW we see the same corruption here with ipurge. We use ipurge to >>> clean out old messages from a spam box for each user. We've >>> worked around it currently by reconstructing the box after the >>> ipurge which adds a lot of work to our nightly maintenance. >> Plus you lose all the messages that are in "delayed expunge" state >> after running a reconstruct. :-( > > Have you tried using '-k' when running reconstruct? Uh, no. :-( But I will from now on. :-) -k Preserve the cyrus.expunge file and the corresponding message files instead of deleting them. > >>> I'd love to see a fix for this. >>> >>> -David >>> >> > > > > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -------------- next part -------------- A non-text attachment was scrubbed... Name: boutilpj.vcf Type: text/x-vcard Size: 286 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090325/cab9879b/attachment.vcf From ml at awinkelmann.de Wed Mar 25 12:39:49 2009 From: ml at awinkelmann.de (Andreas Winkelmann) Date: Wed, 25 Mar 2009 17:39:49 +0100 (CET) Subject: What should be listening on tcp port 12345? In-Reply-To: <49CA2656.7010604@bates.edu> References: <49CA2656.7010604@bates.edu> Message-ID: <983873a8a8dec80b036c949f7d54f87a.squirrel@a-angels.ath.cx> > I'm in the process of adding another front end server to our cyrus email > cluster. The new server is running RedHat ES5, and rpm shows the > version of cyrus is cyrus-imapd-2.3.7. Our front ends provide imaps, > smtp (via sendmail) and lmtp. Sendmail uses smmap to verify the > recipients addrsess, using the process outlined within the RTCyrus3 > project. lmtp then handles the delivery of the email to our backend > servers. The backend servers are running cyrus-imapd-2.2.12. > > During testing of the new server, I noticed a significant delay during > the delivery of email of up to 10 minutes. The delay seems to be caused > by the smmap process while it attempts to connect to tcp port 12345 on > the backend server. iptables on the backend server was configured to > ignore attempts to connect on tcp port 12345. Changing iptables to > reject connections to tcp port 12345 sped up the process considerably. > > What process is supposed to be listening on that port? smmapd. It needs to check for Quota infos on the Backend. At least this is noted in the Sourcecode of smmapd. Unfortunately the Port is hardcoded. > Here's output from /var/log/maillog. mailstore03 is the correct backend > server for my mailbox. > > Mar 24 14:41:25 postoffice04 smmapd[31605]: executed > Mar 24 14:42:41 postoffice04 smmapd[31605]: accepted connection > Mar 24 14:42:41 postoffice04 smmapd[31605]: verify_user(user.rspell) > proxying to host mailstore03.bates.edu > Mar 24 14:42:41 postoffice04 smmapd[31605]: verify_user(user.rspell) > failed: can't connect to mailstore03.bates.edu -- Andreas From freebsd at abv.bg Wed Mar 25 15:42:29 2009 From: freebsd at abv.bg (Mario Pavlov) Date: Wed, 25 Mar 2009 21:42:29 +0200 (EET) Subject: sendmail and cyrus-imap - delivery problem Message-ID: <1318866588.245533.1238010149826.JavaMail.apache@mail51.abv.bg> Hi, I've tried the new version of cyrus-imapd (2.3.14) ... it works like the previous version for me but in addition I got a strange problem with cyradm when I do: cyradm --user cyrus localhost I have to wait about a minute before I'm prompted for password could you tell me how can I set up detailed (debug) logging so I can try to see what's wrong with my mailbox matching... I tried to add these: local6.debug /var/log/imapd.log auth.debug /var/log/auth.log to syslog.conf but I get no logging messages thank you. >Hi, >I've discovered something odd (at least to me) >when I first configured cyrus-imap I was unable to create mailboxes >I kept getting this: > >localhost> cm user.boo at yo.com >createmailbox: Permission denied > >then I read somewhere that I should set "unixhierarchysep" to "yes" and it will work >so I did and it worked...I was able to create mailboxes > >localhost> cm user/boo at yo.com >localhost> lm >user/boo at yo.com (\HasNoChildren) > >and while I was struggling with my delivery problem these days I decided to set "unixhierarchysep" back to "no" just to see what happens >so it was the same...I was unable to create mailboxes because of that "Permission denied" error >but when I wrote "lm" I saw that: > >localhost> lm >user.boo at yo^com (\HasNoChildren) > >you see the difference ? There is this sign "^" in the domain name instead of dot "." >I checked: > >localhost> info user.boo at yo.com // info user.boo at yo^com // Mario Pavlov wrote: > >> thanks again to all of you for your time I've tried all that you > >> suggested I've tried ckuser_cyrus as Matthew Seaman suggested...it > >> made no difference then I've tried rtcyrus3 as Andrzej Adam Filip > >> suggested...it didn't work either...and the logging output in maillog > >> stopped.. [...] > > > >First: Use cyradm to recheck that cyrus reports mailbox named as in > >"RCPT TO:" of LMTP session. > > > >As I understand you have achieved that sendmail uses correct > >"mailbox name with domain" in LMTP session but Cyrus rejects > >the recipient. > > > >-- > >[pl>en: Andrew] Andrzej Adam Filip : anfi at onet.eu > >Life without caffeine is stimulating enough. > > -- Sanka Ad > >---- > >Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > >Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > >List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > > > From brong at fastmail.fm Wed Mar 25 18:12:46 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 26 Mar 2009 09:12:46 +1100 Subject: Cyrus IMAPd 2.3.14 Released In-Reply-To: <3f3206b0cd9c759173c5f5c3ed9e7ad1.squirrel@webmail.bi.corp.invoca.ch> References: <49CA2BC5.10902@andrew.cmu.edu> <3f3206b0cd9c759173c5f5c3ed9e7ad1.squirrel@webmail.bi.corp.invoca.ch> Message-ID: <20090325221246.GB7271@brong.net> On Wed, Mar 25, 2009 at 03:40:55PM +0100, Simon Matter wrote: > > I am pleased to announce the release of Cyrus IMAPd 2.3.14. This > > release should be considered production quality. This is mostly a > > bugfix release. For full details, please see doc/changes.html and > > doc/install-upgrade.html which are included in the distribution. > > > > URLs for this release: > > ftp://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.3.14.tar.gz > > or > > http://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.3.14.tar.gz > > Thanks for the new release! > > My question goes mainly to Bron: > I've included some of your fastmail.fm patches in my RPMS in the past. > They have always been on http://cyrus.brong.fastmail.fm/ > Now things have changed and those patches are no longer maintained there. > Is there a way for me to get the same patches but for 2.3.14 and newer > releases (for example the cyrus-clean-shutdown-2.3.13.diff patch)? > I'm quite sure the code is somewhere on > http://github.com/brong/cyrus-imapd/ but I don't know how to extract the > patches from there? > Any help is much appreciated. Yeah, I never quite got around to finishing the "rebuild my patches page from git" code :( Sorry. I'm tempted to post them to the mailing list or something - then they'll be in various versions in caches somewhere. Git is a great development platform for me, but it does mean that I rewrite the patches frequently as I reorder them and stuff - so it's a pain for people tracking them. Bron. From brong at fastmail.fm Wed Mar 25 18:19:11 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 26 Mar 2009 09:19:11 +1100 Subject: unexpunge segfaults with -l on some mailboxes In-Reply-To: References: <49B7B296.6000305@ednet.ns.ca> <49B7DB6C.9080602@ednet.ns.ca> <20090312000706.GA7208@brong.net> <49B8FB4F.9050806@ednet.ns.ca> <20090312213933.GC7655@brong.net> <10173_1236898088_n2CMm7nQ017919_49B99125.9020508@ednet.ns.ca> <49BA5D3D.4080509@ednet.ns.ca> Message-ID: <20090325221911.GA8408@brong.net> On Mon, Mar 23, 2009 at 04:08:47PM -0700, David R Bosso wrote: > I'd love to see a fix for this. Here's one: >From 8448c2f304bd61fe073b1d98a0d614b69b0ee184 Subject: [PATCH] Don't immediate expunge in ipurge This has two benefits. 1) you can unexpunge 2) it won't corrupt the cache pointers in the expunge file NOTE: this only applies if you have delayed expunge enabled in your configuration, otherwise the behaviour is unchanged. --- imap/ipurge.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/imap/ipurge.c b/imap/ipurge.c index a8c6228..1a76ede 100644 --- a/imap/ipurge.c +++ b/imap/ipurge.c @@ -268,7 +268,7 @@ int purge_me(char *name, int matchlen __attribute__((unused)), (void) mailbox_lock_index(&the_box); the_box.index_lock_count = 1; - mailbox_expunge(&the_box, purge_check, &stats, EXPUNGE_FORCE); + mailbox_expunge(&the_box, purge_check, &stats, 0); sync_log_mailbox(the_box.name); mailbox_close(&the_box); -- 1.5.6.3 From brong at fastmail.fm Wed Mar 25 18:10:23 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 26 Mar 2009 09:10:23 +1100 Subject: unexpunge segfaults with -l on some mailboxes In-Reply-To: <49CA45CD.40501@ednet.ns.ca> References: <49B7B296.6000305@ednet.ns.ca> <49B7DB6C.9080602@ednet.ns.ca> <20090312000706.GA7208@brong.net> <49B8FB4F.9050806@ednet.ns.ca> <20090312213933.GC7655@brong.net> <10173_1236898088_n2CMm7nQ017919_49B99125.9020508@ednet.ns.ca> <49BA5D3D.4080509@ednet.ns.ca> <49CA45CD.40501@ednet.ns.ca> Message-ID: <20090325221023.GA7271@brong.net> On Wed, Mar 25, 2009 at 11:55:09AM -0300, Patrick Boutilier wrote: > David R Bosso wrote: >> --On March 13, 2009 10:18:53 AM -0300 Patrick Boutilier >> wrote: >> >> >> >>>> This will be real easy to test though. I will just run ipurge on a >>>> subfolder of my mailbox and see if it corrupts it. :-) >> >> FWIW we see the same corruption here with ipurge. We use ipurge to >> clean out old messages from a spam box for each user. We've worked >> around it currently by reconstructing the box after the ipurge which >> adds a lot of work to our nightly maintenance. You know - I can't see, from an immediate code reading, how this is happening. It's just calling "mailbox_expunge" with an immediate FORCE and a decideproc. Hmm.. of course, that rewrites the .cache file, but doesn't rewrite the .expunge file. DUH! Colour me an idiot. Yep, bug fully understood :) While I'm sitting here on the train typing in fact! I'll have a think about the best way to handle this. It's 99% totally un-necessary, because the cache pointers in the cyrus.expunge file aren't really needed anyway (rebuilding the cache record on unexpunge would not be the end of the world), but the current state of affairs is clearly bogus. > Plus you lose all the messages that are in "delayed expunge" state after > running a reconstruct. :-( RFTM -k Bron ( the simple and safe solution would be to not pass EXPUNGE_FORCE during ipurge, in which case the disk would be cleaned up by your regularly scheduled cyr_expire, and you could even unexpunge ipurged messages ) From boutilpj at ednet.ns.ca Wed Mar 25 19:43:10 2009 From: boutilpj at ednet.ns.ca (Patrick Boutilier) Date: Wed, 25 Mar 2009 20:43:10 -0300 Subject: unexpunge segfaults with -l on some mailboxes In-Reply-To: <20090325221911.GA8408@brong.net> References: <49B7B296.6000305@ednet.ns.ca> <49B7DB6C.9080602@ednet.ns.ca> <20090312000706.GA7208@brong.net> <49B8FB4F.9050806@ednet.ns.ca> <20090312213933.GC7655@brong.net> <10173_1236898088_n2CMm7nQ017919_49B99125.9020508@ednet.ns.ca> <49BA5D3D.4080509@ednet.ns.ca> <20090325221911.GA8408@brong.net> Message-ID: <2927_1238024593_n2PNhC36007865_49CAC18E.2080208@ednet.ns.ca> Bron Gondwana wrote: > On Mon, Mar 23, 2009 at 04:08:47PM -0700, David R Bosso wrote: >> I'd love to see a fix for this. > > Here's one: > >>From 8448c2f304bd61fe073b1d98a0d614b69b0ee184 > Subject: [PATCH] Don't immediate expunge in ipurge > > This has two benefits. > > 1) you can unexpunge > 2) it won't corrupt the cache pointers in the expunge file > > NOTE: this only applies if you have delayed expunge enabled > in your configuration, otherwise the behaviour is unchanged. > --- > imap/ipurge.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/imap/ipurge.c b/imap/ipurge.c > index a8c6228..1a76ede 100644 > --- a/imap/ipurge.c > +++ b/imap/ipurge.c > @@ -268,7 +268,7 @@ int purge_me(char *name, int matchlen __attribute__((unused)), > (void) mailbox_lock_index(&the_box); > the_box.index_lock_count = 1; > > - mailbox_expunge(&the_box, purge_check, &stats, EXPUNGE_FORCE); > + mailbox_expunge(&the_box, purge_check, &stats, 0); > > sync_log_mailbox(the_box.name); > mailbox_close(&the_box); Looks good. Passes my earlier test. And you are correct. You can unexpunge messages that ipurge purges. Slick :-) From brong at fastmail.fm Wed Mar 25 20:14:40 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 26 Mar 2009 11:14:40 +1100 Subject: unexpunge segfaults with -l on some mailboxes In-Reply-To: <2927_1238024593_n2PNhC36007865_49CAC18E.2080208@ednet.ns.ca> References: <49B7B296.6000305@ednet.ns.ca> <49B7DB6C.9080602@ednet.ns.ca> <20090312000706.GA7208@brong.net> <49B8FB4F.9050806@ednet.ns.ca> <20090312213933.GC7655@brong.net> <10173_1236898088_n2CMm7nQ017919_49B99125.9020508@ednet.ns.ca> <49BA5D3D.4080509@ednet.ns.ca> <20090325221911.GA8408@brong.net> <2927_1238024593_n2PNhC36007865_49CAC18E.2080208@ednet.ns.ca> Message-ID: <20090326001440.GB19392@brong.net> On Wed, Mar 25, 2009 at 08:43:10PM -0300, Patrick Boutilier wrote: > Bron Gondwana wrote: > > On Mon, Mar 23, 2009 at 04:08:47PM -0700, David R Bosso wrote: > >> I'd love to see a fix for this. > > > > Here's one: > > > >>From 8448c2f304bd61fe073b1d98a0d614b69b0ee184 > > Subject: [PATCH] Don't immediate expunge in ipurge > > > > This has two benefits. > > > > 1) you can unexpunge > > 2) it won't corrupt the cache pointers in the expunge file > > > > NOTE: this only applies if you have delayed expunge enabled > > in your configuration, otherwise the behaviour is unchanged. > > --- > > imap/ipurge.c | 2 +- > > 1 files changed, 1 insertions(+), 1 deletions(-) > > > > diff --git a/imap/ipurge.c b/imap/ipurge.c > > index a8c6228..1a76ede 100644 > > --- a/imap/ipurge.c > > +++ b/imap/ipurge.c > > @@ -268,7 +268,7 @@ int purge_me(char *name, int matchlen __attribute__((unused)), > > (void) mailbox_lock_index(&the_box); > > the_box.index_lock_count = 1; > > > > - mailbox_expunge(&the_box, purge_check, &stats, EXPUNGE_FORCE); > > + mailbox_expunge(&the_box, purge_check, &stats, 0); > > > > sync_log_mailbox(the_box.name); > > mailbox_close(&the_box); > > Looks good. Passes my earlier test. And you are correct. You can > unexpunge messages that ipurge purges. Slick :-) Committed to CVS. Trivially correct :) I still want to go fix up mailbox_expunge. It should NEVER be possible to pass a combination of flags that causes mailbox corruption.... but that's a different kettle of fish :) Bron ( nntp will still break at the moment, but I don't know if any sites run nntp and delayed expunge. Anyway, it's not forgotten yet... ) From simon.matter at invoca.ch Thu Mar 26 03:51:11 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Thu, 26 Mar 2009 08:51:11 +0100 (CET) Subject: Cyrus IMAPd 2.3.14 Released In-Reply-To: <20090325221246.GB7271@brong.net> References: <49CA2BC5.10902@andrew.cmu.edu> <3f3206b0cd9c759173c5f5c3ed9e7ad1.squirrel@webmail.bi.corp.invoca.ch> <20090325221246.GB7271@brong.net> Message-ID: <51747df094228c6252495c46adf087e2.squirrel@webmail.bi.corp.invoca.ch> > On Wed, Mar 25, 2009 at 03:40:55PM +0100, Simon Matter wrote: >> > I am pleased to announce the release of Cyrus IMAPd 2.3.14. This >> > release should be considered production quality. This is mostly a >> > bugfix release. For full details, please see doc/changes.html and >> > doc/install-upgrade.html which are included in the distribution. >> > >> > URLs for this release: >> > ftp://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.3.14.tar.gz >> > or >> > http://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.3.14.tar.gz >> >> Thanks for the new release! >> >> My question goes mainly to Bron: >> I've included some of your fastmail.fm patches in my RPMS in the past. >> They have always been on http://cyrus.brong.fastmail.fm/ >> Now things have changed and those patches are no longer maintained >> there. >> Is there a way for me to get the same patches but for 2.3.14 and newer >> releases (for example the cyrus-clean-shutdown-2.3.13.diff patch)? >> I'm quite sure the code is somewhere on >> http://github.com/brong/cyrus-imapd/ but I don't know how to extract the >> patches from there? >> Any help is much appreciated. > > Yeah, I never quite got around to finishing the "rebuild my patches > page from git" code :( Sorry. > > I'm tempted to post them to the mailing list or something - then > they'll be in various versions in caches somewhere. Git is a > great development platform for me, but it does mean that I > rewrite the patches frequently as I reorder them and stuff - so > it's a pain for people tracking them. While we are at it, wouldn't it make sense for the clean-shutdown patch to go into CVS? Simon From brong at fastmail.fm Thu Mar 26 05:37:53 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 26 Mar 2009 20:37:53 +1100 Subject: Cyrus IMAPd 2.3.14 Released In-Reply-To: <51747df094228c6252495c46adf087e2.squirrel@webmail.bi.corp.invoca.ch> References: <49CA2BC5.10902@andrew.cmu.edu> <3f3206b0cd9c759173c5f5c3ed9e7ad1.squirrel@webmail.bi.corp.invoca.ch> <20090325221246.GB7271@brong.net> <51747df094228c6252495c46adf087e2.squirrel@webmail.bi.corp.invoca.ch> Message-ID: <20090326093753.GB7954@brong.net> On Thu, Mar 26, 2009 at 08:51:11AM +0100, Simon Matter wrote: > >> My question goes mainly to Bron: > >> I've included some of your fastmail.fm patches in my RPMS in the past. > >> They have always been on http://cyrus.brong.fastmail.fm/ > >> Now things have changed and those patches are no longer maintained > >> there. > >> Is there a way for me to get the same patches but for 2.3.14 and newer > >> releases (for example the cyrus-clean-shutdown-2.3.13.diff patch)? > >> I'm quite sure the code is somewhere on > >> http://github.com/brong/cyrus-imapd/ but I don't know how to extract the > >> patches from there? > >> Any help is much appreciated. > > > > Yeah, I never quite got around to finishing the "rebuild my patches > > page from git" code :( Sorry. > > > > I'm tempted to post them to the mailing list or something - then > > they'll be in various versions in caches somewhere. Git is a > > great development platform for me, but it does mean that I > > rewrite the patches frequently as I reorder them and stuff - so > > it's a pain for people tracking them. > > While we are at it, wouldn't it make sense for the clean-shutdown patch to > go into CVS? Hmm... probably. Though I'm not 100% convinced that it's perfect. We still get the occasional mismatch after a shutdown - things that were committed enough and yet didn't get synclogged. I'm very tempted to add some sort of pre-log to the sync system. It would be a bit of a pain, but it would avoid all the shutdown issues, and it would give us a list of things to clean up on startup as well! In particular I'm thinking things like where an email file was created on disk and moved into the directory, but the append never finished. There's a spare file that will be found by a reconstruct even though it was never technically _delivered_. Bron. From brong at fastmail.fm Thu Mar 26 08:32:48 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 26 Mar 2009 23:32:48 +1100 Subject: Cyrus IMAPd 2.3.14 Released In-Reply-To: <3f3206b0cd9c759173c5f5c3ed9e7ad1.squirrel@webmail.bi.corp.invoca.ch> References: <49CA2BC5.10902@andrew.cmu.edu> <3f3206b0cd9c759173c5f5c3ed9e7ad1.squirrel@webmail.bi.corp.invoca.ch> Message-ID: <20090326123248.GC7954@brong.net> On Wed, Mar 25, 2009 at 03:40:55PM +0100, Simon Matter wrote: > > I am pleased to announce the release of Cyrus IMAPd 2.3.14. This > > release should be considered production quality. This is mostly a > > bugfix release. For full details, please see doc/changes.html and > > doc/install-upgrade.html which are included in the distribution. > > > > URLs for this release: > > ftp://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.3.14.tar.gz > > or > > http://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.3.14.tar.gz > > Thanks for the new release! > > My question goes mainly to Bron: > I've included some of your fastmail.fm patches in my RPMS in the past. > They have always been on http://cyrus.brong.fastmail.fm/ > Now things have changed and those patches are no longer maintained there. > Is there a way for me to get the same patches but for 2.3.14 and newer > releases (for example the cyrus-clean-shutdown-2.3.13.diff patch)? > I'm quite sure the code is somewhere on > http://github.com/brong/cyrus-imapd/ but I don't know how to extract the > patches from there? > Any help is much appreciated. Ok - http://cyrus.brong.fastmail.fm/ now contains a copy of the diffs between the "master" and "fastmail" branches on github (I have to run a script by hand though, it's not automatically updating at the moment...) Diff file format has changed somewhat, since they're generated with git-format-patch rather than quilt! Warning - there's some funky shit in there. In reality we're only running this on about 1.5% of our users - but we'll be rolling it out to the rest of them probably tomorrow. It's exciting times in there - LOTS of changes! Also advance warning that I do rebase this tree quite frequently, reordering and reworking patches. The files may change over time... That said, Enjoy! Bron ( who will hopefully stabilise things a bit more soon - have to start looking at some sysadmin tasks that have been backing up. I'm not really a _full_time_ Cyrus programmer ;) From simon.matter at invoca.ch Thu Mar 26 09:55:14 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Thu, 26 Mar 2009 14:55:14 +0100 (CET) Subject: Cyrus IMAPd 2.3.14 Released In-Reply-To: <20090326123248.GC7954@brong.net> References: <49CA2BC5.10902@andrew.cmu.edu> <3f3206b0cd9c759173c5f5c3ed9e7ad1.squirrel@webmail.bi.corp.invoca.ch> <20090326123248.GC7954@brong.net> Message-ID: > On Wed, Mar 25, 2009 at 03:40:55PM +0100, Simon Matter wrote: >> > I am pleased to announce the release of Cyrus IMAPd 2.3.14. This >> > release should be considered production quality. This is mostly a >> > bugfix release. For full details, please see doc/changes.html and >> > doc/install-upgrade.html which are included in the distribution. >> > >> > URLs for this release: >> > ftp://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.3.14.tar.gz >> > or >> > http://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.3.14.tar.gz >> >> Thanks for the new release! >> >> My question goes mainly to Bron: >> I've included some of your fastmail.fm patches in my RPMS in the past. >> They have always been on http://cyrus.brong.fastmail.fm/ >> Now things have changed and those patches are no longer maintained >> there. >> Is there a way for me to get the same patches but for 2.3.14 and newer >> releases (for example the cyrus-clean-shutdown-2.3.13.diff patch)? >> I'm quite sure the code is somewhere on >> http://github.com/brong/cyrus-imapd/ but I don't know how to extract the >> patches from there? >> Any help is much appreciated. > > Ok - http://cyrus.brong.fastmail.fm/ now contains a copy of > the diffs between the "master" and "fastmail" branches on > github (I have to run a script by hand though, it's not > automatically updating at the moment...) Thanks alot, that's exactly what I was looking for! I'll just test the new RPM builds. Regards, Simon From weigel at dfbnet.org Thu Mar 26 10:30:21 2009 From: weigel at dfbnet.org (Helmut Weigel) Date: Thu, 26 Mar 2009 15:30:21 +0100 Subject: quota on partitions Message-ID: <20090326153021812.00000003484@donald> Hi all! I use partitions defined in imapd.conf for my 32.000 something mailboxes. I wanted to run the quota command and found that only the boxes in the default partition are shown I tried the -d switch but that didn't work as I thought. what can i do to show quota usage for boxes in non-default partitions? Thanks in advance Helmut Weigel DFB-Medien GmbH & Co. KG Otto-Fleck-Schneise 6 60528 Frankfurt fon: +49 (69) 6788-319 fax: +49 (69) 6788-343 pcfax: +49 (511) 760770 76 - 259 email: helmut.weigel at dfbnet.de Homepage: www.dfb-medien.de -------------------------------------------- Hermann-Neuberger-Haus | Otto-Fleck-Schneise 6 | 60528 Frankfurt | DFB Medien GmbH & Co. KG | Gesch?ftsf?hrung: DFB Medien Verwaltungs-GmbH, deren Gesch?ftsf?hrer: Kurt G?rtner, Tilman Walk |Vorsitzender des Aufsichtsrates: Dr. Theo Zwanziger | HRA 30550 | Registergericht: Frankfurt From jblaine at kickflop.net Thu Mar 26 12:03:21 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Thu, 26 Mar 2009 12:03:21 -0400 Subject: Scripting admin stuff? Message-ID: <49CBA749.8010705@kickflop.net> In 2000, I wrote a simple script that was fed to cyradm to set all users quota to some value. It appears today that the only option to do something like this is to learn the Cyrus::* Perl modules. Is that correct? From pfleming at siumed.edu Thu Mar 26 12:18:08 2009 From: pfleming at siumed.edu (Paul M Fleming) Date: Thu, 26 Mar 2009 11:18:08 -0500 Subject: Scripting admin stuff? In-Reply-To: <49CBA749.8010705@kickflop.net> References: <49CBA749.8010705@kickflop.net> Message-ID: <49CBAAC0.7060701@siumed.edu> Save you the work -- had to do the same thing myself. Modify as needed - for example, i use this with Kerberos auth so no username / password is used. #!/usr/bin/perl use Cyrus::IMAP::Admin; use Cyrus::IMAP; $default_quota = 200000; my $client = Cyrus::IMAP::Admin->new("server",143); $client->authenticate; @mailboxes = $client->list('%', 'user.'); foreach $mbx ( @mailboxes ) { @m = @$mbx; ($root, %quota) = $client->quotaroot($m[0]); $cur_usage = $quota{"STORAGE"}[0]; $cur_quota = $quota{"STORAGE"}[1]; if ( defined $cur_quota ) { # quota defined if ( $cur_quota < $default_quota ) { print "$m[0] : below default increasing\n"; $client->setquota($m[0],"STORAGE",$default_quota); } if ( $cur_quota > $default_quota ) { print "$m[0] : over default: $cur_quota ($cur_usage / $cur_quota)\n"; } } else { print "$m[0] : NO QUOTA $cur_usage\n"; } } On 3/26/2009 11:03 AM, Jeff Blaine wrote: > In 2000, I wrote a simple script that was fed to cyradm > to set all users quota to some value. > > It appears today that the only option to do something like > this is to learn the Cyrus::* Perl modules. > > Is that correct? > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html From jblaine at kickflop.net Thu Mar 26 12:37:34 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Thu, 26 Mar 2009 12:37:34 -0400 Subject: Scripting admin stuff? In-Reply-To: <49CBAAC0.7060701@siumed.edu> References: <49CBA749.8010705@kickflop.net> <49CBAAC0.7060701@siumed.edu> Message-ID: <49CBAF4E.1030907@kickflop.net> Thanks. I shortened it to the following. For those using this, it needs to run as cyrus (or whatever your cyrus user is). #!/linus/mail/cyrus/bin/perl $default_quota = 400000; use Cyrus::IMAP::Admin; use Cyrus::IMAP; my $client = Cyrus::IMAP::Admin->new("YOUR_SERVER",143); $client->authenticate; @mailboxes = $client->list('%', 'user.'); foreach $mbx ( @mailboxes ) { @m = @$mbx; $client->setquota($m[0],"STORAGE",$default_quota); } Paul M Fleming wrote: > Save you the work -- had to do the same thing myself. > > Modify as needed - for example, i use this with Kerberos auth so no > username / password is used. > > > #!/usr/bin/perl > > use Cyrus::IMAP::Admin; > use Cyrus::IMAP; > $default_quota = 200000; > > my $client = Cyrus::IMAP::Admin->new("server",143); > $client->authenticate; > @mailboxes = $client->list('%', 'user.'); > foreach $mbx ( @mailboxes ) > { > > @m = @$mbx; > > ($root, %quota) = $client->quotaroot($m[0]); > > $cur_usage = $quota{"STORAGE"}[0]; > $cur_quota = $quota{"STORAGE"}[1]; > > if ( defined $cur_quota ) > { > # quota defined > if ( $cur_quota < $default_quota ) > { > print "$m[0] : below default increasing\n"; > $client->setquota($m[0],"STORAGE",$default_quota); > } > if ( $cur_quota > $default_quota ) > { > print "$m[0] : over default: $cur_quota > ($cur_usage / $cur_quota)\n"; > } > } > else > { > print "$m[0] : NO QUOTA $cur_usage\n"; > } > > > > } > > > On 3/26/2009 11:03 AM, Jeff Blaine wrote: >> In 2000, I wrote a simple script that was fed to cyradm >> to set all users quota to some value. >> >> It appears today that the only option to do something like >> this is to learn the Cyrus::* Perl modules. >> >> Is that correct? >> ---- >> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ >> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki >> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > From Jean-Christophe.Delaye at eurecom.fr Thu Mar 26 13:25:30 2009 From: Jean-Christophe.Delaye at eurecom.fr (Jean-Christophe Delaye) Date: Thu, 26 Mar 2009 18:25:30 +0100 Subject: Cyrus IMAP Murder and (SUN)Solaris Cluster In-Reply-To: <20090325155240.4047h4f74so0w4ww@www.staffmail.ed.ac.uk> References: <20090325155240.4047h4f74so0w4ww@www.staffmail.ed.ac.uk> Message-ID: <49CBBA8A.8020107@eurecom.fr> Gavin Gray wrote: > > We want to try and improve the resilience of our setup and are looking > at a number of > options to provide High Availability: ..... > 2. Use Sun cluster to provide automatic failover for the backends > somehow or another. ..... We do use Sun cluster at EURECOM. We are running Sun Cluster 3.2 on 2 nodes Solaris10 connected on San network with EMC Clariion Storage. Our Sun Cluster ressources groups are based on : - logical hosts (with IPMP) - HA storage zfs with zpool on EMC Lun. - GDS application hold the cyrus sasl and imap services. We are using separate ressources groups for backends, murder and frontend (a total of five at this time). We recently upgrade the configuration and introduce Solaris containers on each cluster node. The benefit is the possibility to run backends in non global zone within different subnet (for example in DMZ using IP Exclusive zone). Of course, this specific type of ressources groups can switch in non global zones on each cluster node, providing high availability. It works fine for us. Jean-Christophe. From freebsd at abv.bg Thu Mar 26 15:37:20 2009 From: freebsd at abv.bg (Mario Pavlov) Date: Thu, 26 Mar 2009 21:37:20 +0200 (EET) Subject: sendmail and cyrus-imap - delivery problem Message-ID: <1980593677.279164.1238096240540.JavaMail.apache@mail52.abv.bg> Hi, I've got some more verbose logging I've created new mailbox (just for fun) localhost> cm user/jane at yo.com localhost> lm user/boo at yo.com (\HasNoChildren) user/jane at yo.com (\HasNoChildren) I've increased the loglevel of sendmail to 16 and I've sent an e-mail from gmail to jane at yo.com and I got this in sendmail's output: Mar 26 19:15:56 yo sendmail[23789]: NOQUEUE: connect from fg-out-1718.google.com [72.14.220.154] Mar 26 19:15:56 yo sendmail[23789]: AUTH: available mech=NTLM LOGIN ANONYMOUS PLAIN GSSAPI OTP DIGEST-MD5 CRAM-MD5, allowed mech=GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN Mar 26 19:15:56 yo sendmail[23789]: n2QJFuHM023789: Milter: no active filter Mar 26 19:15:56 yo sendmail[23789]: n2QJFuHM023789: --- 220 yo.com ESMTP Sendmail 8.14.3/8.14.3; Thu, 26 Mar 2009 19:15:56 GMT Mar 26 19:15:56 yo sendmail[23789]: n2QJFuHM023789: <-- EHLO fg-out-1718.google.com Mar 26 19:15:56 yo sendmail[23789]: n2QJFuHM023789: --- 250-yo.com Hello fg-out-1718.google.com [72.14.220.154], pleased to meet you Mar 26 19:15:56 yo sendmail[23789]: n2QJFuHM023789: --- 250-ENHANCEDSTATUSCODES Mar 26 19:15:56 yo sendmail[23789]: n2QJFuHM023789: --- 250-PIPELINING Mar 26 19:15:56 yo sendmail[23789]: n2QJFuHM023789: --- 250-8BITMIME Mar 26 19:15:56 yo sendmail[23789]: n2QJFuHM023789: --- 250-SIZE Mar 26 19:15:56 yo sendmail[23789]: n2QJFuHM023789: --- 250-DSN Mar 26 19:15:56 yo sendmail[23789]: n2QJFuHM023789: --- 250-ETRN Mar 26 19:15:56 yo sendmail[23789]: n2QJFuHM023789: --- 250-AUTH GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN Mar 26 19:15:56 yo sendmail[23789]: n2QJFuHM023789: --- 250-DELIVERBY Mar 26 19:15:56 yo sendmail[23789]: n2QJFuHM023789: --- 250 HELP Mar 26 19:15:56 yo sendmail[23789]: n2QJFuHM023789: <-- MAIL FROM: Mar 26 19:15:56 yo sendmail[23789]: n2QJFuHM023789: --- 250 2.1.0 ... Sender ok Mar 26 19:15:57 yo sendmail[23789]: n2QJFuHM023789: <-- RCPT TO: Mar 26 19:15:57 yo sendmail[23789]: n2QJFuHM023789: --- 250 2.1.5 ... Recipient ok Mar 26 19:15:57 yo sendmail[23789]: n2QJFuHM023789: <-- DATA Mar 26 19:15:57 yo sendmail[23789]: n2QJFuHM023789: --- 354 Enter mail, end with "." on a line by itself Mar 26 19:15:57 yo sendmail[23789]: n2QJFuHM023789: from=, size=1592, class=0, nrcpts=1, msgid=<9cddded00903261215i77936b6fs8dbc476326cf47f5 at mail.gmail.com>, proto=ESMTP, daemon=IPv4, relay=fg-out-1718.google.com [72.14.220.154] Mar 26 19:15:57 yo sendmail[23792]: n2QJFuHM023789: --- 050 ... Connecting to /var/imap/socket/lmtp via cyrusv3d... Mar 26 19:15:57 yo sendmail[23789]: n2QJFuHM023789: --- 250 2.0.0 n2QJFuHM023789 Message accepted for delivery Mar 26 19:15:57 yo sendmail[23792]: AUTH=client, relay=localhost, mech=, bits=0 Mar 26 19:15:57 yo sendmail[23792]: n2QJFuHM023789: --- 550 5.1.1 ... User unknown (hold) Mar 26 19:15:57 yo sendmail[23792]: n2QJFuHM023789: to=, delay=00:00:00, xdelay=00:00:00, mailer=cyrusv3d, pri=31592, relay=localhost, dsn=5.1.1, stat=User unknown Mar 26 19:15:57 yo sendmail[23792]: n2QJFuHM023789: n2QJFvHM023792: DSN: User unknown Mar 26 19:15:57 yo sendmail[23792]: n2QJFvHM023792: --- 050 ... Connecting to gmail-smtp-in.l.google.com. via esmtp... Mar 26 19:15:57 yo sendmail[23792]: n2QJFvHM023792: SMTP outgoing connect on reverse.yo.com Mar 26 19:15:58 yo sendmail[23792]: n2QJFvHM023792: --- 050 ... Sent (OK 1238094956 3si369136fxm.108) Mar 26 19:15:58 yo sendmail[23792]: n2QJFvHM023792: to=, delay=00:00:01, xdelay=00:00:01, mailer=esmtp, pri=32616, relay=gmail-smtp-in.l.google.com. [209.85.220.27], dsn=2.0.0, stat=Sent (OK 1238094956 3si369136fxm.108) Mar 26 19:15:58 yo sendmail[23792]: n2QJFvHM023792: done; delay=00:00:01, ntries=1 Mar 26 19:15:58 yo sendmail[23792]: n2QJFuHM023789: done; delay=00:00:01, ntries=1 Mar 26 19:15:58 yo sendmail[23792]: NOQUEUE: --- 050 Closing connection to gmail-smtp-in.l.google.com. Mar 26 19:15:58 yo sendmail[23792]: NOQUEUE: --- 050 Closing connection to localhost and I also got this in cyrus' output: Mar 26 19:15:57 yo smmapd[23780]: accepted connection Mar 26 19:15:57 yo master[23790]: about to exec /usr/local/cyrus/bin/smmapd Mar 26 19:15:57 yo smmapd[23790]: executed Mar 26 19:15:57 yo master[23793]: about to exec /usr/local/cyrus/bin/lmtpd Mar 26 19:15:57 yo lmtpunix[23793]: executed Mar 26 19:15:57 yo lmtpunix[23793]: accepted connection Mar 26 19:15:57 yo lmtpunix[23793]: lmtp connection preauth'd as postman Mar 26 19:15:57 yo lmtpunix[23793]: verify_user(user.jane) failed: Mailbox does not exist you can see that the domain is stripped "verify_user(user.jane)" but sendmail seems to pass the user with the domain...is this normal ? here's what happens if I send an e-mail to non-existing mailbox sendmail: Mar 26 19:30:58 yo sendmail[23827]: NOQUEUE: connect from mail-fx0-f163.google.com [209.85.220.163] Mar 26 19:30:58 yo sendmail[23827]: AUTH: available mech=NTLM LOGIN ANONYMOUS PLAIN GSSAPI OTP DIGEST-MD5 CRAM-MD5, allowed mech=GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN Mar 26 19:30:58 yo sendmail[23827]: n2QJUwmk023827: Milter: no active filter Mar 26 19:30:58 yo sendmail[23827]: n2QJUwmk023827: --- 220 yo.com ESMTP Sendmail 8.14.3/8.14.3; Thu, 26 Mar 2009 19:30:58 GMT Mar 26 19:30:58 yo sendmail[23827]: n2QJUwmk023827: <-- EHLO mail-fx0-f163.google.com Mar 26 19:30:58 yo sendmail[23827]: n2QJUwmk023827: --- 250-yo.com Hello mail-fx0-f163.google.com [209.85.220.163], pleased to meet you Mar 26 19:30:58 yo sendmail[23827]: n2QJUwmk023827: --- 250-ENHANCEDSTATUSCODES Mar 26 19:30:58 yo sendmail[23827]: n2QJUwmk023827: --- 250-PIPELINING Mar 26 19:30:58 yo sendmail[23827]: n2QJUwmk023827: --- 250-8BITMIME Mar 26 19:30:58 yo sendmail[23827]: n2QJUwmk023827: --- 250-SIZE Mar 26 19:30:58 yo sendmail[23827]: n2QJUwmk023827: --- 250-DSN Mar 26 19:30:58 yo sendmail[23827]: n2QJUwmk023827: --- 250-ETRN Mar 26 19:30:58 yo sendmail[23827]: n2QJUwmk023827: --- 250-AUTH GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN Mar 26 19:30:58 yo sendmail[23827]: n2QJUwmk023827: --- 250-DELIVERBY Mar 26 19:30:58 yo sendmail[23827]: n2QJUwmk023827: --- 250 HELP Mar 26 19:30:58 yo sendmail[23827]: n2QJUwmk023827: <-- MAIL FROM: Mar 26 19:30:58 yo sendmail[23827]: n2QJUwmk023827: --- 250 2.1.0 ... Sender ok Mar 26 19:30:59 yo sendmail[23827]: n2QJUwmk023827: <-- RCPT TO: Mar 26 19:30:59 yo sendmail[23827]: n2QJUwmk023827: --- 550 5.1.1 ... Mailbox is not available. Mar 26 19:30:59 yo sendmail[23827]: n2QJUwmk023827: ... Mailbox is not available. Mar 26 19:30:59 yo sendmail[23827]: n2QJUwmk023827: <-- QUIT Mar 26 19:30:59 yo sendmail[23827]: n2QJUwmk023827: --- 221 2.0.0 yo.com closing connection Mar 26 19:30:59 yo sendmail[23827]: n2QJUwmk023827: from=, size=0, class=0, nrcpts=0, proto=ESMTP, daemon=IPv4, relay=mail-fx0-f163.google.com [209.85.220.163] cyrus: Mar 26 19:30:59 yo smmapd[23817]: accepted connection Mar 26 19:30:59 yo smmapd[23817]: verify_user(user.josh at yo^com) failed: Mailbox does not exist I can see sendmail says the mailbox is not available so does cyrus only this time the domain is present "verify_user(user.josh at yo^com)" and it's with that sign "^" instead of dot...is this normal ? thank you. >Hi, >I've tried the new version of cyrus-imapd (2.3.14) ... it works like the previous version for me >but in addition I got a strange problem with cyradm >when I do: >cyradm --user cyrus localhost > >I have to wait about a minute before I'm prompted for password > >could you tell me how can I set up detailed (debug) logging so I can try to see what's wrong with my mailbox matching... > >I tried to add these: >local6.debug /var/log/imapd.log >auth.debug /var/log/auth.log > >to syslog.conf but I get no logging messages > >thank you. > > > >Hi, > >I've discovered something odd (at least to me) > >when I first configured cyrus-imap I was unable to create mailboxes > >I kept getting this: > > > >localhost> cm user.boo at yo.com > >createmailbox: Permission denied > > > >then I read somewhere that I should set "unixhierarchysep" to "yes" and it will work > >so I did and it worked...I was able to create mailboxes > > > >localhost> cm user/boo at yo.com > >localhost> lm > >user/boo at yo.com (\HasNoChildren) > > > >and while I was struggling with my delivery problem these days I decided to set "unixhierarchysep" back to "no" just to see what happens > >so it was the same...I was unable to create mailboxes because of that "Permission denied" error > >but when I wrote "lm" I saw that: > > > >localhost> lm > >user.boo at yo^com (\HasNoChildren) > > > >you see the difference ? There is this sign "^" in the domain name instead of dot "." > >I checked: > > > >localhost> info user.boo at yo.com // info user.boo at yo^com // Mario Pavlov wrote: > > >> thanks again to all of you for your time I've tried all that you > > >> suggested I've tried ckuser_cyrus as Matthew Seaman suggested...it > > >> made no difference then I've tried rtcyrus3 as Andrzej Adam Filip > > >> suggested...it didn't work either...and the logging output in maillog > > >> stopped.. [...] > > > > > >First: Use cyradm to recheck that cyrus reports mailbox named as in > > >"RCPT TO:" of LMTP session. > > > > > >As I understand you have achieved that sendmail uses correct > > >"mailbox name with domain" in LMTP session but Cyrus rejects > > >the recipient. > > > > > >-- > > >[pl>en: Andrew] Andrzej Adam Filip : anfi at onet.eu > > >Life without caffeine is stimulating enough. > > > -- Sanka Ad > > >---- > > >Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > > >Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > > >List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > > > > > > From florin at andrei.myip.org Thu Mar 26 19:59:07 2009 From: florin at andrei.myip.org (Florin Andrei) Date: Thu, 26 Mar 2009 16:59:07 -0700 Subject: thoughts on running an IMAP-over-SSL server exposed to the Internet? Message-ID: <49CC16CB.7020001@andrei.myip.org> I want to read my email on the iPhone. To do that, I have 2 options: 1. VPN 2. IMAP-over-SSL #1 is a bit convoluted, I already run a VPN server, with OpenVPN, but the iPhone doesn't have an OpenVPN client. Running *two* VPN networks seems excessive for a small personal server - not that the machine cannot handle it, but it just feels too complicated for the task at hand. #2 would be easy to implement, just poke a hole in the firewall for the imaps port. But then there's the issue of security, of course. I am running cyrus-imapd-2.3.7 on CentOS 5.x How comfortable y'all are with exposing Cyrus IMAPd's imaps port to the big wild Internet? Do you see the SELinux confinement as a must-have in this context, or are you okay with running it without any such MAC protections? -- Florin Andrei http://florin.myip.org/ From craigwhite at azapple.com Thu Mar 26 20:40:32 2009 From: craigwhite at azapple.com (Craig White) Date: Thu, 26 Mar 2009 17:40:32 -0700 Subject: thoughts on running an IMAP-over-SSL server exposed to the Internet? In-Reply-To: <49CC16CB.7020001@andrei.myip.org> References: <49CC16CB.7020001@andrei.myip.org> Message-ID: <1238114432.18255.43.camel@lin-workstation.azapple.com> On Thu, 2009-03-26 at 16:59 -0700, Florin Andrei wrote: > I want to read my email on the iPhone. To do that, I have 2 options: > 1. VPN > 2. IMAP-over-SSL > > #1 is a bit convoluted, I already run a VPN server, with OpenVPN, but > the iPhone doesn't have an OpenVPN client. Running *two* VPN networks > seems excessive for a small personal server - not that the machine > cannot handle it, but it just feels too complicated for the task at hand. > > #2 would be easy to implement, just poke a hole in the firewall for the > imaps port. But then there's the issue of security, of course. > > I am running cyrus-imapd-2.3.7 on CentOS 5.x > > How comfortable y'all are with exposing Cyrus IMAPd's imaps port to the > big wild Internet? > Do you see the SELinux confinement as a must-have in this context, or > are you okay with running it without any such MAC protections? ---- I expect it to be safe because I too have opened IMAPS ports for the various clients that I have who want to use their iPhone's and Blackberry's, etc. That also means that I have had to implement SMTP auth so that they can send e-mail too. I have faith that these are daemons (cyrus and postfix) that can withstand attacks but every port you open is another attack vector on your system. Craig From zmully at smartbrief.com Fri Mar 27 09:46:08 2009 From: zmully at smartbrief.com (Zachariah Mully) Date: Fri, 27 Mar 2009 09:46:08 -0400 Subject: thoughts on running an IMAP-over-SSL server exposed to the Internet? In-Reply-To: <49CC16CB.7020001@andrei.myip.org> References: <49CC16CB.7020001@andrei.myip.org> Message-ID: <1238161568.26672.38.camel@zmully.smartbrief.com> On Thu, 2009-03-26 at 16:59 -0700, Florin Andrei wrote: > I want to read my email on the iPhone. To do that, I have 2 options: > 1. VPN > 2. IMAP-over-SSL > > #1 is a bit convoluted, I already run a VPN server, with OpenVPN, but > the iPhone doesn't have an OpenVPN client. Running *two* VPN networks > seems excessive for a small personal server - not that the machine > cannot handle it, but it just feels too complicated for the task at hand. > > #2 would be easy to implement, just poke a hole in the firewall for the > imaps port. But then there's the issue of security, of course. > > I am running cyrus-imapd-2.3.7 on CentOS 5.x > > How comfortable y'all are with exposing Cyrus IMAPd's imaps port to the > big wild Internet? > Do you see the SELinux confinement as a must-have in this context, or > are you okay with running it without any such MAC protections? > I went to a talk by Dam Kaminsky of this past summers DNS exploit fame. If you want to be scared sh*tless about the potential security vulnerabilities of DNS, read up on his work. SSL does nothing. But on the more practical side. What exactly are you worried about? Someone rooting your machine through IMAP/Cyrus (never seen/heard of that done with any IMAP server, but please correct me if I'm wrong)? Getting access to your email? What? The biggest security problem I see (daily) is users. I'd love to deploy two-factor auth, but that's not possible right now. Z From brennan at columbia.edu Fri Mar 27 09:53:56 2009 From: brennan at columbia.edu (Joseph Brennan) Date: Fri, 27 Mar 2009 09:53:56 -0400 Subject: thoughts on running an IMAP-over-SSL server exposed to the Internet? In-Reply-To: <1238161568.26672.38.camel@zmully.smartbrief.com> References: <49CC16CB.7020001@andrei.myip.org> <1238161568.26672.38.camel@zmully.smartbrief.com> Message-ID: <18D7ED71729064DB849B9147@sodor.cc.columbia.edu> --On Friday, March 27, 2009 9:46 -0400 Zachariah Mully wrote: >> How comfortable y'all are with exposing Cyrus IMAPd's imaps port to the >> big wild Internet? Not much point running it if you can't connect to it, is there? It's totally standard. Actually you need only plain imap with tls required, but imaps helps some clients work right. Joseph Brennan Lead Email Systems Engineer Columbia University Information Technology From florin at andrei.myip.org Fri Mar 27 14:06:27 2009 From: florin at andrei.myip.org (Florin Andrei) Date: Fri, 27 Mar 2009 11:06:27 -0700 Subject: thoughts on running an IMAP-over-SSL server exposed to the Internet? In-Reply-To: <1238161568.26672.38.camel@zmully.smartbrief.com> References: <49CC16CB.7020001@andrei.myip.org> <1238161568.26672.38.camel@zmully.smartbrief.com> Message-ID: <49CD15A3.4080806@andrei.myip.org> Zachariah Mully wrote: > > I went to a talk by Dam Kaminsky of this past summers DNS exploit fame. > If you want to be scared sh*tless about the potential security > vulnerabilities of DNS, read up on his work. SSL does nothing. Well, we're all gonna die of something, aren't we? There are many attacks out there. You address what you can, do not address what you cannot, cross your fingers and hope for the best. > But on the more practical side. What exactly are you worried about? > Someone rooting your machine through IMAP/Cyrus (never seen/heard of > that done with any IMAP server, but please correct me if I'm wrong)? > Getting access to your email? What? The thing worrying me at this time is some stupid buffer overflow in the IMAP server code. I have no idea what's the security history of this server, even though I've been using it for quite a while, because it was always in tightly controlled environments. Exposing it to the Internet changes the game. The reason why I'm not immediately jumping for the VPN solution is that I already have a VPN in place, just not compatible with the iPhone. Running two VPNs seems just silly. But maybe it is the right solution after all. > The biggest security problem I see (daily) is users. In this case, there are only a couple users and I'm one of them, so I'm not worried. (or maybe I should? heh heh) -- Florin Andrei http://florin.myip.org/ From raymond at sundland.com Fri Mar 27 14:14:15 2009 From: raymond at sundland.com (Raymond T. Sundland) Date: Fri, 27 Mar 2009 14:14:15 -0400 Subject: thoughts on running an IMAP-over-SSL server exposed to the Internet? In-Reply-To: <49CD15A3.4080806@andrei.myip.org> References: <49CC16CB.7020001@andrei.myip.org> <1238161568.26672.38.camel@zmully.smartbrief.com> <49CD15A3.4080806@andrei.myip.org> Message-ID: <49CD1777.5060903@sundland.com> Whenever you open services to the internet, you're taking a chance. I've been running Cyrus IMAP open to the Internet for years and have never had any issues, but I may have just been lucky. There are plenty of sources available for looking for the history of vulnerabilities for various software packages, including the change log for Cyrus. If you can run selinux to protect it a bit more, why not? If you can jail/chroot the processes, why not? Anything you can do to limit the exposure to future problems puts you one step ahead of the rest. Florin Andrei wrote: > Zachariah Mully wrote: > >> I went to a talk by Dam Kaminsky of this past summers DNS exploit fame. >> If you want to be scared sh*tless about the potential security >> vulnerabilities of DNS, read up on his work. SSL does nothing. >> > > Well, we're all gonna die of something, aren't we? > > There are many attacks out there. You address what you can, do not > address what you cannot, cross your fingers and hope for the best. > > >> But on the more practical side. What exactly are you worried about? >> Someone rooting your machine through IMAP/Cyrus (never seen/heard of >> that done with any IMAP server, but please correct me if I'm wrong)? >> Getting access to your email? What? >> > > The thing worrying me at this time is some stupid buffer overflow in the > IMAP server code. I have no idea what's the security history of this > server, even though I've been using it for quite a while, because it was > always in tightly controlled environments. Exposing it to the Internet > changes the game. > > The reason why I'm not immediately jumping for the VPN solution is that > I already have a VPN in place, just not compatible with the iPhone. > Running two VPNs seems just silly. But maybe it is the right solution > after all. > > >> The biggest security problem I see (daily) is users. >> > > In this case, there are only a couple users and I'm one of them, so I'm > not worried. (or maybe I should? heh heh) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090327/f5ce8eb6/attachment.html From zmully at smartbrief.com Fri Mar 27 14:17:29 2009 From: zmully at smartbrief.com (Zachariah Mully) Date: Fri, 27 Mar 2009 14:17:29 -0400 Subject: thoughts on running an IMAP-over-SSL server exposed to the Internet? In-Reply-To: <49CD15A3.4080806@andrei.myip.org> References: <49CC16CB.7020001@andrei.myip.org> <1238161568.26672.38.camel@zmully.smartbrief.com> <49CD15A3.4080806@andrei.myip.org> Message-ID: <1238177849.26672.242.camel@zmully.smartbrief.com> On Fri, 2009-03-27 at 11:06 -0700, Florin Andrei wrote: > The thing worrying me at this time is some stupid buffer overflow in the > IMAP server code. I have no idea what's the security history of this > server, even though I've been using it for quite a while, because it was > always in tightly controlled environments. Exposing it to the Internet > changes the game. > > In this case, there are only a couple users and I'm one of them, so I'm > not worried. (or maybe I should? heh heh) You've said nothing about the risks of a breach of your system, nor about what exactly it is you are worried about being comprimised. Once you figure out those, you can make an educated decision. Until then, you're just pissing into the wind. Z From wes at umich.edu Fri Mar 27 19:34:52 2009 From: wes at umich.edu (Wesley Craig) Date: Fri, 27 Mar 2009 19:34:52 -0400 Subject: thoughts on running an IMAP-over-SSL server exposed to the Internet? In-Reply-To: <49CD15A3.4080806@andrei.myip.org> References: <49CC16CB.7020001@andrei.myip.org> <1238161568.26672.38.camel@zmully.smartbrief.com> <49CD15A3.4080806@andrei.myip.org> Message-ID: On 27 Mar 2009, at 14:06, Florin Andrei wrote: > The thing worrying me at this time is some stupid buffer overflow > in the > IMAP server code. I have no idea what's the security history of this > server, even though I've been using it for quite a while, because > it was > always in tightly controlled environments. Exposing it to the Internet > changes the game. It's a very popular open source IMAP server, typically run exposed to the Internet at large. That certainly far from a guarantee that it's bug free, but intruders have had ample opportunity probe the code over the years. :wes From brong at fastmail.fm Fri Mar 27 20:01:43 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sat, 28 Mar 2009 11:01:43 +1100 Subject: thoughts on running an IMAP-over-SSL server exposed to the Internet? In-Reply-To: <49CC16CB.7020001@andrei.myip.org> References: <49CC16CB.7020001@andrei.myip.org> Message-ID: <20090328000143.GA11970@brong.net> On Thu, Mar 26, 2009 at 04:59:07PM -0700, Florin Andrei wrote: > I want to read my email on the iPhone. To do that, I have 2 options: > 1. VPN > 2. IMAP-over-SSL > > #1 is a bit convoluted, I already run a VPN server, with OpenVPN, but > the iPhone doesn't have an OpenVPN client. Running *two* VPN networks > seems excessive for a small personal server - not that the machine > cannot handle it, but it just feels too complicated for the task at hand. > > #2 would be easy to implement, just poke a hole in the firewall for the > imaps port. But then there's the issue of security, of course. > > I am running cyrus-imapd-2.3.7 on CentOS 5.x > > How comfortable y'all are with exposing Cyrus IMAPd's imaps port to the > big wild Internet? > Do you see the SELinux confinement as a must-have in this context, or > are you okay with running it without any such MAC protections? We don't actually use SSL directly within Cyrus, instead using nginx with SSL on our frontend servers, proxying to the backends. This is mainly for load balancing, but it does also mean that the nginx server can be run with zero privileges for anything else. It doesn't give any protection from authenticated users (once the login is finished, the traffic is just directly proxied to the backend), but it does mean unauthenticated users don't have direct access to the cyrus imapds. If you're paranoid, that might be worth doing! That said, like everyone else has mentioned - Cyrus has been around for a long time, and has a good security track record. Bron. From scrappy at hub.org Sat Mar 28 01:59:28 2009 From: scrappy at hub.org (Marc G. Fournier) Date: Sat, 28 Mar 2009 02:59:28 -0300 (ADT) Subject: Outlook + Cyrus IMAPd == Drop'd connection Message-ID: <20090328025604.G47574@hub.org> Running the latest version of 2.3.x, I'm having an issue where I can't seem to get Outlook to connect to the server ... I get a bunch of errors about: master[22775]: process 1486 exited, signaled to death by 10 Now, if I try and connect using Outlook Express, with the exact same connection information, without making any changes to the server, all works fine ... in fact, at the same time as Outlook is failing, an attempt to connect using horde works fine also ... This is trying Outlook on several different computers, from several different locations ... they all generate the exact same error above ... Something I'm overlooking? Problem with the versoin of Outlook itself, maybe? ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email . scrappy at hub.org MSN . scrappy at hub.org Yahoo . yscrappy Skype: hub.org ICQ . 7615664 From brong at fastmail.fm Sat Mar 28 03:24:35 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sat, 28 Mar 2009 18:24:35 +1100 Subject: Outlook + Cyrus IMAPd == Drop'd connection In-Reply-To: <20090328025604.G47574@hub.org> References: <20090328025604.G47574@hub.org> Message-ID: <20090328072435.GA7623@brong.net> On Sat, Mar 28, 2009 at 02:59:28AM -0300, Marc G. Fournier wrote: > Running the latest version of 2.3.x, I'm having an issue where I can't > seem to get Outlook to connect to the server ... I get a bunch of errors > about: > > master[22775]: process 1486 exited, signaled to death by 10 That's USR1 on my linux machine. It gets sent by idled. If idled still thinks you're idling, but your imapd doesn't, then it's not trapping USR1 any more. That's potentially a bug. > Now, if I try and connect using Outlook Express, with the exact same > connection information, without making any changes to the server, all > works fine ... in fact, at the same time as Outlook is failing, an attempt > to connect using horde works fine also ... > > This is trying Outlook on several different computers, from several > different locations ... they all generate the exact same error above ... > > Something I'm overlooking? Problem with the versoin of Outlook itself, > maybe? Possibly. It would be interesting to see the telemetry logs. I suspect that Outlook is doing something funky with idle, but assuming that it's changed recently, we might also be doing something odd with idle in Cyrus that's causing it to break. If you are able to get telemetry logs (create a directory $confdir/log/$user/ - i.e. /var/imap/log/brong in my case, and make sure it's owned by your cyrus user) that would be a great help. Obviously remove any identifying data you're worried about being public, and just send it to me if you want. I'd like to see what it's doing! Thanks, Bron. From scrappy at hub.org Sat Mar 28 08:38:42 2009 From: scrappy at hub.org (Marc G. Fournier) Date: Sat, 28 Mar 2009 09:38:42 -0300 (ADT) Subject: Outlook + Cyrus IMAPd == Drop'd connection In-Reply-To: <20090328072435.GA7623@brong.net> References: <20090328025604.G47574@hub.org> <20090328072435.GA7623@brong.net> Message-ID: <20090328092947.E47574@hub.org> On Sat, 28 Mar 2009, Bron Gondwana wrote: > If you are able to get telemetry logs (create a directory > $confdir/log/$user/ - i.e. /var/imap/log/brong in my case, and make sure > it's owned by your cyrus user) that would be a great help. Obviously > remove any identifying data you're worried about being public, and just > send it to me if you want. I'd like to see what it's doing! 'k, tried this ... I assumed this isn't something that I need to restart imapd to take effect, is it? I've tried regardless, and created both the userid and userid at domain (virtual hosts) and when I connect, its not logging anything ... when I use webmail (horde) to connect, though, I am getting telemetry, so the logging functionality is working, just not when connecting with Outlook ... But, carrying your thought through about idled, I just rebuild the port without idled, and the same error is being generated ... ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email . scrappy at hub.org MSN . scrappy at hub.org Yahoo . yscrappy Skype: hub.org ICQ . 7615664 From baconm at email.unc.edu Sat Mar 28 09:37:56 2009 From: baconm at email.unc.edu (Michael Bacon) Date: Sat, 28 Mar 2009 09:37:56 -0400 Subject: possible self-deadlock in idle signal handler Message-ID: We're experiencing some problems, particularly with a small number of users, which manifest themselves in the dreaded "one deadlocked, hundreds waiting" process logjam. The keystone process appears to be an imapd deadlocked on itself in this manner (this is Solaris 9): -> pstack 19090 19090: imapd febc5994 lwp_park (0, 0, 0) febc206c slow_lock (fecc05a8, feba0000, 0, fecbc000, 14, 0) + 58 fec46e70 malloc (c, 0, 13d668, 13d66c, 28cc, 13d790) + 18 00078ac0 xmalloc (c, 13d790, 0, 0, 0, 0) + 4 00074a64 lock_or_refresh (13d660, 1364b4, 107400, 0, 0, 0) + 10c 00074d50 myfetch (13d660, 1bbe58, 10, ffbfb25c, ffbfb254, 1364b4) + 44 00060d74 seen_readit (1364a0, ffbfb2ec, ffbfb2e8, 1252bc, ffbfb2e4, 1) + 60 0003d0c4 index_checkseen (123a00, 0, 0, 603, 1e5a4c, 87fd0) + 4c 0003e298 index_check (123a00, 0, 1, 125000, ffbfc370, 125000) + 234 0002c574 idle_update (3, 0, 0, 0, 0, 0) + 24 0005f7cc idle_handler (e, 0, ffbfcb20, 0, 0, 0) + 5c febc5bac __sighndlr (e, 0, ffbfcb20, 5f770, 0, 0) + c febbf804 call_user_handler (e, 0, ffbfcb20, 0, 0, 0) + 234 febbf9b4 sigacthandler (e, 0, ffbfcb20, 8, 1bd7c0, 0) + 64 --- called from signal handler with signal 14 (SIGALRM) --- fec470d4 _malloc_unlocked (64, 0, 0, fecbc000, 0, 0) + 240 fec46e78 malloc (64, ff0a07d0, a3, 1c4d0d, db, 6d) + 20 fefc5820 default_malloc_ex (64, ff0b17b0, ca, ca, 0, ffe43088) + 20 fefc61e4 CRYPTO_malloc (0, ff0b17b0, ca, 1bcff0, 1bcf78, 1bcf78) + 84 ff036efc EVP_DigestInit_ex (ffbfd150, ff0dfbb0, 0, fffffff8, 0, ffbfd1fd) + 13c fefdabec HMAC_Init_ex (ffbfd13c, ffbfd150, ffbfd048, ff0dfbb0, 0, 0) + cc ff160b70 tls1_mac (1bea88, ffbfd288, 0, 20, 0, 1) + 90 ff15cfa4 ssl3_read_bytes (1bea88, 17, ffbfd288, 8c, 1c4d03, 0) + 524 ff15a9c4 ssl3_read (1bea88, 13aef0, 1000, 0, 378, 0) + 44 ff16a30c SSL_read (0, 13aef0, 1000, 0, ffbfd5bc, ffbfd5b1) + 6c 0006bd5c prot_fill (13ae78, 0, 0, 0, ffbfd5bc, ffbfd428) + ec 0005e564 getword (13ae78, 125108, 1, 1a9e0, 2c8dc, 125000) + ac 0002c8f0 cmd_idle (13d358, 7dc00, 0, 0, 730061, 0) + 2e8 0002ea6c cmdloop (0, 1360d8, 8bc60, 8bc60, 123c00, 125000) + df0 00030d34 service_main (123c00, 132080, ffbffc2c, 0, 1aa50, 11a800) + 180 0001aaf8 main (ffbff2b4, 7c000, fa, 27667, 2602e4, 49c71400) + 640 0001a2ec _start (0, 0, 0, 0, 0, 0) + 5c From looking online, what looks to be the problem is that the SSL stack was in the middle of a malloc() call when the SIGALRM went off, causing the process to try to open the seen file, which resulted in another malloc. The second malloc requests a mutex on malloc for the process (part of Solaris's thread internals), but that mutex is held by the first call, and hence the mutex lock will never return and the process is permanently hung, holding the lock for the mailbox. Would anyone happen to have any tips on getting out from under this? Thanks, Michael Bacon ITS Messaging UNC Chapel Hill From wes at umich.edu Sat Mar 28 15:36:05 2009 From: wes at umich.edu (Wesley Craig) Date: Sat, 28 Mar 2009 15:36:05 -0400 Subject: possible self-deadlock in idle signal handler In-Reply-To: References: Message-ID: <34F0881B-F62B-41F1-8350-2160F421AAD1@umich.edu> See here: https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3100 The solution is to rewrite the signal handler to do much less. :wes On 28 Mar 2009, at 09:37, Michael Bacon wrote: > We're experiencing some problems, particularly with a small number of > users, which manifest themselves in the dreaded "one deadlocked, > hundreds waiting" process logjam. The keystone process appears to be > an imapd deadlocked on itself in this manner (this is Solaris 9): > > -> pstack 19090 > 19090: imapd > febc5994 lwp_park (0, 0, 0) > febc206c slow_lock (fecc05a8, feba0000, 0, fecbc000, 14, 0) + 58 > fec46e70 malloc (c, 0, 13d668, 13d66c, 28cc, 13d790) + 18 > 00078ac0 xmalloc (c, 13d790, 0, 0, 0, 0) + 4 > 00074a64 lock_or_refresh (13d660, 1364b4, 107400, 0, 0, 0) + 10c > 00074d50 myfetch (13d660, 1bbe58, 10, ffbfb25c, ffbfb254, > 1364b4) + 44 > 00060d74 seen_readit (1364a0, ffbfb2ec, ffbfb2e8, 1252bc, > ffbfb2e4, 1) > + 60 > 0003d0c4 index_checkseen (123a00, 0, 0, 603, 1e5a4c, 87fd0) + 4c > 0003e298 index_check (123a00, 0, 1, 125000, ffbfc370, 125000) + 234 > 0002c574 idle_update (3, 0, 0, 0, 0, 0) + 24 > 0005f7cc idle_handler (e, 0, ffbfcb20, 0, 0, 0) + 5c > febc5bac __sighndlr (e, 0, ffbfcb20, 5f770, 0, 0) + c > febbf804 call_user_handler (e, 0, ffbfcb20, 0, 0, 0) + 234 > febbf9b4 sigacthandler (e, 0, ffbfcb20, 8, 1bd7c0, 0) + 64 > --- called from signal handler with signal 14 (SIGALRM) --- > fec470d4 _malloc_unlocked (64, 0, 0, fecbc000, 0, 0) + 240 > fec46e78 malloc (64, ff0a07d0, a3, 1c4d0d, db, 6d) + 20 > fefc5820 default_malloc_ex (64, ff0b17b0, ca, ca, 0, ffe43088) + 20 > fefc61e4 CRYPTO_malloc (0, ff0b17b0, ca, 1bcff0, 1bcf78, 1bcf78) > + 84 > ff036efc EVP_DigestInit_ex (ffbfd150, ff0dfbb0, 0, fffffff8, 0, > ffbfd1fd) + 13c > fefdabec HMAC_Init_ex (ffbfd13c, ffbfd150, ffbfd048, ff0dfbb0, 0, > 0) + > cc > ff160b70 tls1_mac (1bea88, ffbfd288, 0, 20, 0, 1) + 90 > ff15cfa4 ssl3_read_bytes (1bea88, 17, ffbfd288, 8c, 1c4d03, 0) + 524 > ff15a9c4 ssl3_read (1bea88, 13aef0, 1000, 0, 378, 0) + 44 > ff16a30c SSL_read (0, 13aef0, 1000, 0, ffbfd5bc, ffbfd5b1) + 6c > 0006bd5c prot_fill (13ae78, 0, 0, 0, ffbfd5bc, ffbfd428) + ec > 0005e564 getword (13ae78, 125108, 1, 1a9e0, 2c8dc, 125000) + ac > 0002c8f0 cmd_idle (13d358, 7dc00, 0, 0, 730061, 0) + 2e8 > 0002ea6c cmdloop (0, 1360d8, 8bc60, 8bc60, 123c00, 125000) + df0 > 00030d34 service_main (123c00, 132080, ffbffc2c, 0, 1aa50, 11a800) + > 180 > 0001aaf8 main (ffbff2b4, 7c000, fa, 27667, 2602e4, 49c71400) > + 640 > 0001a2ec _start (0, 0, 0, 0, 0, 0) + 5c > > From looking online, what looks to be the problem is that the SSL > stack > was in the middle of a malloc() call when the SIGALRM went off, > causing > the process to try to open the seen file, which resulted in another > malloc. The second malloc requests a mutex on malloc for the process > (part of Solaris's thread internals), but that mutex is held by the > first call, and hence the mutex lock will never return and the process > is permanently hung, holding the lock for the mailbox. > > Would anyone happen to have any tips on getting out from under this? From baconm at email.unc.edu Sat Mar 28 23:27:06 2009 From: baconm at email.unc.edu (baconm at email.unc.edu) Date: Sat, 28 Mar 2009 23:27:06 -0400 Subject: possible self-deadlock in idle signal handler In-Reply-To: <34F0881B-F62B-41F1-8350-2160F421AAD1@umich.edu> References: <34F0881B-F62B-41F1-8350-2160F421AAD1@umich.edu> Message-ID: <20090328232706.rjr4ndq084skwwk0@webmail3.isis.unc.edu> Thanks for the link. That certainly would cover the general case of what I'm seeing. This looks like something much more than "well, let me fiddle with it a bit and submit a patch." Has anyone with the requisite design perspective taken a hack at solving it? I'm assuming that running idled doesn't ameliorate the problem. -Michael Quoting Wesley Craig : > See here: > > https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3100 > > The solution is to rewrite the signal handler to do much less. > > :wes > > On 28 Mar 2009, at 09:37, Michael Bacon wrote: >> We're experiencing some problems, particularly with a small number of >> users, which manifest themselves in the dreaded "one deadlocked, >> hundreds waiting" process logjam. The keystone process appears to be >> an imapd deadlocked on itself in this manner (this is Solaris 9): >> >> -> pstack 19090 >> 19090: imapd >> febc5994 lwp_park (0, 0, 0) >> febc206c slow_lock (fecc05a8, feba0000, 0, fecbc000, 14, 0) + 58 >> fec46e70 malloc (c, 0, 13d668, 13d66c, 28cc, 13d790) + 18 >> 00078ac0 xmalloc (c, 13d790, 0, 0, 0, 0) + 4 >> 00074a64 lock_or_refresh (13d660, 1364b4, 107400, 0, 0, 0) + 10c >> 00074d50 myfetch (13d660, 1bbe58, 10, ffbfb25c, ffbfb254, 1364b4) + 44 >> 00060d74 seen_readit (1364a0, ffbfb2ec, ffbfb2e8, 1252bc, ffbfb2e4, 1) >> + 60 >> 0003d0c4 index_checkseen (123a00, 0, 0, 603, 1e5a4c, 87fd0) + 4c >> 0003e298 index_check (123a00, 0, 1, 125000, ffbfc370, 125000) + 234 >> 0002c574 idle_update (3, 0, 0, 0, 0, 0) + 24 >> 0005f7cc idle_handler (e, 0, ffbfcb20, 0, 0, 0) + 5c >> febc5bac __sighndlr (e, 0, ffbfcb20, 5f770, 0, 0) + c >> febbf804 call_user_handler (e, 0, ffbfcb20, 0, 0, 0) + 234 >> febbf9b4 sigacthandler (e, 0, ffbfcb20, 8, 1bd7c0, 0) + 64 >> --- called from signal handler with signal 14 (SIGALRM) --- >> fec470d4 _malloc_unlocked (64, 0, 0, fecbc000, 0, 0) + 240 >> fec46e78 malloc (64, ff0a07d0, a3, 1c4d0d, db, 6d) + 20 >> fefc5820 default_malloc_ex (64, ff0b17b0, ca, ca, 0, ffe43088) + 20 >> fefc61e4 CRYPTO_malloc (0, ff0b17b0, ca, 1bcff0, 1bcf78, 1bcf78) + 84 >> ff036efc EVP_DigestInit_ex (ffbfd150, ff0dfbb0, 0, fffffff8, 0, >> ffbfd1fd) + 13c >> fefdabec HMAC_Init_ex (ffbfd13c, ffbfd150, ffbfd048, ff0dfbb0, 0, 0) + >> cc >> ff160b70 tls1_mac (1bea88, ffbfd288, 0, 20, 0, 1) + 90 >> ff15cfa4 ssl3_read_bytes (1bea88, 17, ffbfd288, 8c, 1c4d03, 0) + 524 >> ff15a9c4 ssl3_read (1bea88, 13aef0, 1000, 0, 378, 0) + 44 >> ff16a30c SSL_read (0, 13aef0, 1000, 0, ffbfd5bc, ffbfd5b1) + 6c >> 0006bd5c prot_fill (13ae78, 0, 0, 0, ffbfd5bc, ffbfd428) + ec >> 0005e564 getword (13ae78, 125108, 1, 1a9e0, 2c8dc, 125000) + ac >> 0002c8f0 cmd_idle (13d358, 7dc00, 0, 0, 730061, 0) + 2e8 >> 0002ea6c cmdloop (0, 1360d8, 8bc60, 8bc60, 123c00, 125000) + df0 >> 00030d34 service_main (123c00, 132080, ffbffc2c, 0, 1aa50, 11a800) + >> 180 >> 0001aaf8 main (ffbff2b4, 7c000, fa, 27667, 2602e4, 49c71400) + 640 >> 0001a2ec _start (0, 0, 0, 0, 0, 0) + 5c >> >> From looking online, what looks to be the problem is that the SSL stack >> was in the middle of a malloc() call when the SIGALRM went off, causing >> the process to try to open the seen file, which resulted in another >> malloc. The second malloc requests a mutex on malloc for the process >> (part of Solaris's thread internals), but that mutex is held by the >> first call, and hence the mutex lock will never return and the process >> is permanently hung, holding the lock for the mailbox. >> >> Would anyone happen to have any tips on getting out from under this? > From wes at umich.edu Sun Mar 29 21:12:55 2009 From: wes at umich.edu (Wesley Craig) Date: Sun, 29 Mar 2009 21:12:55 -0400 Subject: possible self-deadlock in idle signal handler In-Reply-To: <20090328232706.rjr4ndq084skwwk0@webmail3.isis.unc.edu> References: <34F0881B-F62B-41F1-8350-2160F421AAD1@umich.edu> <20090328232706.rjr4ndq084skwwk0@webmail3.isis.unc.edu> Message-ID: On 28 Mar 2009, at 23:27, baconm at email.unc.edu wrote: > This looks like something much more than "well, let me fiddle with > it a bit and submit a patch." Has anyone with the requisite design > perspective taken a hack at solving it? I'm assuming that running > idled doesn't ameliorate the problem. Running idled probably makes the problem worse, not better. I don't know that anyone besides me has even reviewed this code. And you can see how many people rushed to comment on the issue :) :wes From jross at broad.mit.edu Mon Mar 30 12:23:57 2009 From: jross at broad.mit.edu (Jesse Ross) Date: Mon, 30 Mar 2009 12:23:57 -0400 Subject: cyrus won't recognize new partition Message-ID: Hi all, I'm having trouble getting my cyrus imapd server (version 2.3.10) to load a new partition I've created. I created the partition and the stores./ and user/ directories, and set their permissions and ownership to exactly match an already working partition, as shown below. Here imap06 is the working partition and imap07 is the new one: lysine:~ # ls -l /var/spool/imap06 /var/spool/imap07 /var/spool/imap06: total 32 drwxr-xr-x 5 cyrus mail 4096 Jan 6 15:06 . drwxr-xr-x 24 root root 4096 Mar 26 17:21 .. drwx------ 2 cyrus mail 16384 Dec 18 14:31 lost+found drwxr-xr-x 2 cyrus mail 4096 Mar 30 12:14 stage. drwx------ 56 cyrus mail 4096 Mar 26 18:50 user /var/spool/imap07: total 32 drwxr-xr-x 5 cyrus mail 4096 Mar 26 16:00 . drwxr-xr-x 24 root root 4096 Mar 26 17:21 .. drwx------ 2 cyrus mail 16384 Mar 26 15:53 lost+found drwxr-xr-x 2 cyrus mail 4096 Mar 26 16:00 stage. drwx------ 2 cyrus mail 4096 Mar 26 16:53 user Then I added a line to the imapd.conf: partition-imap06: /var/spool/imap06 partition-imap07: /var/spool/imap07 and did a full stop/start on the server to ensure there weren't any listeners lying around with the old config. However, when I try to create a new mailbox in that partition, or move one to it, I get errors: localhost> cm --partition imap07 user.foo createmailbox: Unknown/invalid partition localhost> renm --partition imap07 user.sschmoss user.sschmoss renamemailbox: Operation is not supported on mailbox Note that the second error message is misleading: I can move the user.sschmoss mailbox to any partition other than the new one. Anyone have any ideas? What am I missing here? Thanks very much, Jesse Ross Systems Administrator Broad Institute of MIT and Harvard -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090330/3e1d097e/attachment.html From craigwhite at azapple.com Mon Mar 30 14:34:56 2009 From: craigwhite at azapple.com (Craig White) Date: Mon, 30 Mar 2009 11:34:56 -0700 Subject: upgrade from 2.2.x to 2.3.x Message-ID: <1238438096.5672.170.camel@lin-workstation.azapple.com> Planning on upgrading my server from... db4-4.2.52-7.3.el4 cyrus-imapd-2.2.12-9.RHEL4 to db4-4.3.29-9.fc6 cyrus-imapd-2.3.14-1 obviously, I can move the mail files but I'm also concerned with the various seen.db, mailboxes.db, annotations.db Do I just copy them over and use berkeley utilities to update them? Are there any web pages that cover the upgrade process? Craig From brong at fastmail.fm Mon Mar 30 17:46:59 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 31 Mar 2009 08:46:59 +1100 Subject: cyrus won't recognize new partition In-Reply-To: References: Message-ID: <20090330214659.GA8544@brong.net> On Mon, Mar 30, 2009 at 12:23:57PM -0400, Jesse Ross wrote: > I'm having trouble getting my cyrus imapd server (version 2.3.10) to load a > new partition I've created. > > Anyone have any ideas? What am I missing here? Have you tried running 'mkimap'? We always run it after every change, it will make any directories that it needs. Bron. From brong at fastmail.fm Mon Mar 30 18:38:58 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 31 Mar 2009 09:38:58 +1100 Subject: possible self-deadlock in idle signal handler In-Reply-To: References: <34F0881B-F62B-41F1-8350-2160F421AAD1@umich.edu> <20090328232706.rjr4ndq084skwwk0@webmail3.isis.unc.edu> Message-ID: <20090330223858.GA7270@brong.net> On Sun, Mar 29, 2009 at 09:12:55PM -0400, Wesley Craig wrote: > On 28 Mar 2009, at 23:27, baconm at email.unc.edu wrote: > > This looks like something much more than "well, let me fiddle with > > it a bit and submit a patch." Has anyone with the requisite design > > perspective taken a hack at solving it? I'm assuming that running > > idled doesn't ameliorate the problem. > > Running idled probably makes the problem worse, not better. I don't > know that anyone besides me has even reviewed this code. And you can > see how many people rushed to comment on the issue :) Yep ;) Bron ( taking a week off Cyrus to work on some other stuff, mostly ) From simon.matter at invoca.ch Tue Mar 31 03:38:02 2009 From: simon.matter at invoca.ch (Simon Matter) Date: Tue, 31 Mar 2009 09:38:02 +0200 (CEST) Subject: upgrade from 2.2.x to 2.3.x In-Reply-To: <1238438096.5672.170.camel@lin-workstation.azapple.com> References: <1238438096.5672.170.camel@lin-workstation.azapple.com> Message-ID: > Planning on upgrading my server from... > > db4-4.2.52-7.3.el4 > cyrus-imapd-2.2.12-9.RHEL4 > > to > > db4-4.3.29-9.fc6 > cyrus-imapd-2.3.14-1 Looks like you are going from RHEL4 to RHEL5? > > obviously, I can move the mail files but I'm also concerned with the > various seen.db, mailboxes.db, annotations.db > > Do I just copy them over and use berkeley utilities to update them? Looks like you use my rpms on RHEL5 right? Then you may want to first update cyrus-imapd on RHEL4 to cyrus-imapd-2.3.14-1. Then start and stop cyrus-imapd on RHEL4 and transfer all cyrus data to RHEL5 and start it up. The RPM handles this by converting all berkeley DB's to skiplist in shutdown and converting back as needed on startup. (I think the 2.2.12 rpm already does this on RHEL4 but I'm not sure so just stop it and check the db files.) Regards, Simon From jross at broad.mit.edu Tue Mar 31 10:33:28 2009 From: jross at broad.mit.edu (Jesse Ross) Date: Tue, 31 Mar 2009 10:33:28 -0400 Subject: cyrus won't recognize new partition In-Reply-To: <20090330214659.GA8544@brong.net> References: <20090330214659.GA8544@brong.net> Message-ID: Hi Bron, Thanks for the suggestion. The only thing that the mkimap script does differently from the way I did it was to create a directory /var/spool/imap07/sync./. However, I still can't get cyrus to recognize the partition! The symptoms are the same; ownership of directories set to cyrus, appropriate permissions as copied from working partitions, configuration same as for the working partitions. Is there perhaps a way to get cyrus to log what it thinks the problem with the partition is? thanks, jesse On Mon, Mar 30, 2009 at 5:46 PM, Bron Gondwana wrote: > On Mon, Mar 30, 2009 at 12:23:57PM -0400, Jesse Ross wrote: > > I'm having trouble getting my cyrus imapd server (version 2.3.10) to load > a > > new partition I've created. > > > > Anyone have any ideas? What am I missing here? > > Have you tried running 'mkimap'? We always run it after > every change, it will make any directories that it needs. > > Bron. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090331/08a02aeb/attachment.html From jblaine at kickflop.net Tue Mar 31 12:09:32 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Tue, 31 Mar 2009 12:09:32 -0400 Subject: Weekly/Monthly record-keeping / maintenance? Message-ID: <49D2403C.50908@kickflop.net> Every year or so, a user of ours reports a discrepancy between on-disk usage for their spool compared to what 'FETCH' is reporting (as implemented via imapdu.pl) Compact/Expunge via the client app buys nothing. Clearly we are failing to do something as admins in order to keep our Cyrus instance working up to snuff. Can anyone shed some light on what we're doing wrong? What are the regular maintenance tasks we should be performing to keep everything working as precisely as possible? I found nothing useful in the Wiki regarding this topic. # our imapd.conf with auth options censored configdirectory: /var/imap defaultpartition: default partition-default: /var/spool/imap unix_group_enable: 0 sieveusehomedir: true autocreatequota: 200000 duplicate_db: skiplist annotation_db: skiplist mboxkey_db: skiplist mboxlist_db: skiplist # our cyrus.conf START { recover cmd="ctl_cyrusdb -r" } SERVICES { imap cmd="imapd" listen="imap" prefork=5 proto=tcp4 imaps cmd="imapd -s" listen="imaps" prefork=1 proto=tcp4 lmtpunix cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=1 } EVENTS { checkpoint cmd="ctl_cyrusdb -c" period=10 delprune cmd="cyr_expire -E 3" at=0400 tlsprune cmd="tls_prune" at=0400 } From baconm at email.unc.edu Tue Mar 31 12:22:07 2009 From: baconm at email.unc.edu (Michael Bacon) Date: Tue, 31 Mar 2009 12:22:07 -0400 Subject: possible self-deadlock in idle signal handler In-Reply-To: <20090330223858.GA7270@brong.net> References: <34F0881B-F62B-41F1-8350-2160F421AAD1@umich.edu> <20090328232706.rjr4ndq084skwwk0@webmail3.isis.unc.edu> <20090330223858.GA7270@brong.net> Message-ID: <2B3FF962C805FAFF940C7742@ITSBACONM.depts.unc.edu> Just as a follow-up on this, I think I've found a way to at least decrease the incidence of the problem with a stupid conf file trick. The problem appears to emerge when the client decides to do something simultaneously with the idle alarm going off. Since most clients (in the case of this user, it was Thunderbird) do things in multiples of minutes, I changed the imapidlepoll setting from 60 to 65. (67 might be even better, come to think of it.) I'll have to let it run a few weeks before I know if it made any difference, but it may have the effect of de-syncing activity on the client and server ends, which would probably help. -Michael --On Tuesday, March 31, 2009 9:38 AM +1100 Bron Gondwana wrote: > On Sun, Mar 29, 2009 at 09:12:55PM -0400, Wesley Craig wrote: >> On 28 Mar 2009, at 23:27, baconm at email.unc.edu wrote: >> > This looks like something much more than "well, let me fiddle with >> > it a bit and submit a patch." Has anyone with the requisite design >> > perspective taken a hack at solving it? I'm assuming that running >> > idled doesn't ameliorate the problem. >> >> Running idled probably makes the problem worse, not better. I don't >> know that anyone besides me has even reviewed this code. And you can >> see how many people rushed to comment on the issue :) > > Yep ;) > > Bron ( taking a week off Cyrus to work on some other stuff, mostly ) From ml at awinkelmann.de Tue Mar 31 16:34:14 2009 From: ml at awinkelmann.de (Andreas Winkelmann) Date: Tue, 31 Mar 2009 22:34:14 +0200 Subject: cyrus won't recognize new partition In-Reply-To: References: Message-ID: <200903312234.14517.ml@awinkelmann.de> Am Montag 30 M?rz 2009 18:23:57 schrieb Jesse Ross: > I'm having trouble getting my cyrus imapd server (version 2.3.10) to load a > new partition I've created. > > I created the partition and the stores./ and user/ directories, and set > their permissions and ownership to exactly match an already working > partition, as shown below. Here imap06 is the working partition and imap07 > is the new one: > Then I added a line to the imapd.conf: > > partition-imap06: /var/spool/imap06 > partition-imap07: /var/spool/imap07 Please check for invalid characters around the second line (imap07) in your imapd.conf. Maybe show a hexdump of the config file. > and did a full stop/start on the server to ensure there weren't any > listeners lying around with the old config. > > However, when I try to create a new mailbox in that partition, or move one > to it, I get errors: > > localhost> cm --partition imap07 user.foo > createmailbox: Unknown/invalid partition > localhost> renm --partition imap07 user.sschmoss user.sschmoss > renamemailbox: Operation is not supported on mailbox > > Note that the second error message is misleading: I can move the > user.sschmoss mailbox to any partition other than the new one. > > Anyone have any ideas? What am I missing here? -- Andreas From ml at awinkelmann.de Tue Mar 31 16:43:18 2009 From: ml at awinkelmann.de (Andreas Winkelmann) Date: Tue, 31 Mar 2009 22:43:18 +0200 Subject: Weekly/Monthly record-keeping / maintenance? In-Reply-To: <49D2403C.50908@kickflop.net> References: <49D2403C.50908@kickflop.net> Message-ID: <200903312243.18875.ml@awinkelmann.de> Am Dienstag 31 M?rz 2009 18:09:32 schrieb Jeff Blaine: > Every year or so, a user of ours reports a discrepancy > between on-disk usage for their spool compared to what > 'FETCH' is reporting (as implemented via imapdu.pl) What means "on-disk usage for spool" ? Do you really compare the size of the Directory on the disk with the Size of each Mail-Message in this Directory? > Compact/Expunge via the client app buys nothing. > > Clearly we are failing to do something as admins in order > to keep our Cyrus instance working up to snuff. > > Can anyone shed some light on what we're doing wrong? > > What are the regular maintenance tasks we should be > performing to keep everything working as precisely as > possible? > > I found nothing useful in the Wiki regarding this topic. > > # our imapd.conf with auth options censored > configdirectory: /var/imap > defaultpartition: default > partition-default: /var/spool/imap > unix_group_enable: 0 > sieveusehomedir: true > autocreatequota: 200000 > duplicate_db: skiplist > annotation_db: skiplist > mboxkey_db: skiplist > mboxlist_db: skiplist > > # our cyrus.conf > START { > recover cmd="ctl_cyrusdb -r" > } > > SERVICES { > imap cmd="imapd" listen="imap" prefork=5 proto=tcp4 > imaps cmd="imapd -s" listen="imaps" prefork=1 proto=tcp4 > lmtpunix cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=1 > } > > EVENTS { > checkpoint cmd="ctl_cyrusdb -c" period=10 > delprune cmd="cyr_expire -E 3" at=0400 > tlsprune cmd="tls_prune" at=0400 > } > ---- > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- -- Andreas From jblaine at kickflop.net Tue Mar 31 16:50:07 2009 From: jblaine at kickflop.net (Jeff Blaine) Date: Tue, 31 Mar 2009 16:50:07 -0400 Subject: Weekly/Monthly record-keeping / maintenance? In-Reply-To: <200903312243.18875.ml@awinkelmann.de> References: <49D2403C.50908@kickflop.net> <200903312243.18875.ml@awinkelmann.de> Message-ID: <49D281FF.90305@kickflop.net> Andreas Winkelmann wrote: > Am Dienstag 31 M?rz 2009 18:09:32 schrieb Jeff Blaine: > >> Every year or so, a user of ours reports a discrepancy >> between on-disk usage for their spool compared to what >> 'FETCH' is reporting (as implemented via imapdu.pl) > > What means "on-disk usage for spool" ? What I mean, is, for one example -- a user is currently reporting that 'FETCH' (via the imapdu command) is showing 142 messages totalling 640KB in a folder that is actually completely empty on disk (except for cyrus.* files). reconstruct -r user.hername did not change what is reported via 'FETCH'. It did update the cyrus.* files though. From craigwhite at azapple.com Tue Mar 31 18:04:02 2009 From: craigwhite at azapple.com (Craig White) Date: Tue, 31 Mar 2009 15:04:02 -0700 Subject: upgrade from 2.2.x to 2.3.x In-Reply-To: References: <1238438096.5672.170.camel@lin-workstation.azapple.com> Message-ID: <1238537042.3680.26.camel@lin-workstation.azapple.com> On Tue, 2009-03-31 at 09:38 +0200, Simon Matter wrote: > > Planning on upgrading my server from... > > > > db4-4.2.52-7.3.el4 > > cyrus-imapd-2.2.12-9.RHEL4 > > > > to > > > > db4-4.3.29-9.fc6 > > cyrus-imapd-2.3.14-1 > > Looks like you are going from RHEL4 to RHEL5? > > > > > obviously, I can move the mail files but I'm also concerned with the > > various seen.db, mailboxes.db, annotations.db > > > > Do I just copy them over and use berkeley utilities to update them? > > Looks like you use my rpms on RHEL5 right? Then you may want to first > update cyrus-imapd on RHEL4 to cyrus-imapd-2.3.14-1. Then start and stop > cyrus-imapd on RHEL4 and transfer all cyrus data to RHEL5 and start it up. > The RPM handles this by converting all berkeley DB's to skiplist in > shutdown and converting back as needed on startup. (I think the 2.2.12 rpm > already does this on RHEL4 but I'm not sure so just stop it and check the > db files.) ---- yes - yes - yes Yes, from CentOS 4 to CentOS 5 Yes, from CentOS 4 cyrus-imapd package to Simon's package on CentOS 5 Yes, shut it down, did 'rsync -ar' everything from /var/spool/imap and /var/lib/imap, /etc/imapd.conf and /etc/cyrus.conf and generated new certs. No - it all failed at startup. Did chown cyrus:mail on both directories recursively and it started up and seems to be running fine. Did a 'reconstruct -fr user.*' just to be safe... Thanks for the src packages and thanks for the reply but I didn't bother installing 2.3.x on my RHEL 4 and it doesn't seem to have mattered. I think I've almost done everything necessary in migrating my server to where I can shut down the old server now. Craig From dbosso+lists.cyrus at lsit.ucsb.edu Tue Mar 31 18:51:54 2009 From: dbosso+lists.cyrus at lsit.ucsb.edu (David R Bosso) Date: Tue, 31 Mar 2009 15:51:54 -0700 Subject: Another cache bug! In-Reply-To: <1228865816.22867.1289197119@webmail.messagingengine.com> References: <1228865816.22867.1289197119@webmail.messagingengine.com> Message-ID: <083C2140202CD43473EC0F91@bossland.lsit.ucsb.edu> --On December 10, 2008 10:36:56 AM +1100 Bron Gondwana wrote: > Wow, this is the thanks I get for doing sanity checks on files, find more > bugs! > > This one is due to delayed expunge, plain and simple. Cyrus decides what > cache records to copy during an IMAP COPY command by reading the cache > offsets for msgno and msgno+1 (or the end of the cache file if it's the > last msgno). > > Obviously if some intervening messages have already been deleted from the > cyrus.index then it will be copying all those cache records as well to the > destination folder. Oops. > > The attached patch reworks mailbox_cache_size into two functions, the > second being mailbox_cache_size_detail that takes memory locations for > the cache mmap rather than a mailbox object (because imapd doesn't update > the locations in the object correctly according to my testing, *sigh*. > Gotta love global variables) > > It then uses mailbox_cache_size_detail to calculate the ACTUAL record > length for this single cache item rather than blindly copying everything > up to the next index record's pointer. > > Also note: in the event of cache corruption, mailbox_cache_size_detail > returns zero bytes, which correctly makes append_copy re-parse the > message file. It's all shiny :) > > Wes/Ken - please apply to CVS :) > > Thanks, > > Bron. > -- > Bron Gondwana > brong at fastmail.fm Going through my list of 2.3.13 patches to put together a local 2.3.14 build, I ran across this one. It doesn't appear to have made it into 2.3.14. Oversight, or is there something wrong with it? Thanks. -David -------------- next part -------------- A non-text attachment was scrubbed... Name: cyrus-copy-cachelen-2.3.13.diff Type: text/x-patch Size: 3530 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090331/a5f0f146/attachment-0001.bin From brong at fastmail.fm Tue Mar 31 20:18:24 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 01 Apr 2009 11:18:24 +1100 Subject: Another cache bug! In-Reply-To: <083C2140202CD43473EC0F91@bossland.lsit.ucsb.edu> References: <1228865816.22867.1289197119@webmail.messagingengine.com> <083C2140202CD43473EC0F91@bossland.lsit.ucsb.edu> Message-ID: <1238545104.25500.1308325875@webmail.messagingengine.com> On Tue, 31 Mar 2009 15:51 -0700, "David R Bosso" wrote: > --On December 10, 2008 10:36:56 AM +1100 Bron Gondwana > > wrote: > > > Wow, this is the thanks I get for doing sanity checks on files, find more > > bugs! > > > > This one is due to delayed expunge, plain and simple. Cyrus decides what > > cache records to copy during an IMAP COPY command by reading the cache > > offsets for msgno and msgno+1 (or the end of the cache file if it's the > > last msgno). > > > > Obviously if some intervening messages have already been deleted from the > > cyrus.index then it will be copying all those cache records as well to the > > destination folder. Oops. > > > > The attached patch reworks mailbox_cache_size into two functions, the > > second being mailbox_cache_size_detail that takes memory locations for > > the cache mmap rather than a mailbox object (because imapd doesn't update > > the locations in the object correctly according to my testing, *sigh*. > > Gotta love global variables) > > > > It then uses mailbox_cache_size_detail to calculate the ACTUAL record > > length for this single cache item rather than blindly copying everything > > up to the next index record's pointer. > > > > Also note: in the event of cache corruption, mailbox_cache_size_detail > > returns zero bytes, which correctly makes append_copy re-parse the > > message file. It's all shiny :) > > > > Wes/Ken - please apply to CVS :) > > > > Thanks, > > > > Bron. > > -- > > Bron Gondwana > > brong at fastmail.fm > > Going through my list of 2.3.13 patches to put together a local 2.3.14 > build, I ran across this one. It doesn't appear to have made it into > 2.3.14. Oversight, or is there something wrong with it? It's being reworked! In particular, I'm working on delayed cache file loading. There's also stuff about cache version number and locking all sort of floating around... For the immediate fix, I would recommend the attached patch. It applies directly on top of CVS. Not so sure if it depends on anything that's not in 2.3.14... I don't _think_ so. NOTE - this patch actually touches quite a lot of code. This is the only bit that's actually in production at FM at the moment - the delayed loading is still being tested. Still - this puts ALL cache accesses through a single, simple API :) -- Bron Gondwana brong at fastmail.fm -------------- next part -------------- A non-text attachment was scrubbed... Name: cache.patch Type: text/x-patch Size: 50990 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090401/c67c086b/attachment-0001.bin