From elfejoyeux at gmail.com Wed Nov 4 05:30:20 2009 From: elfejoyeux at gmail.com (Cyril Servant) Date: Wed, 4 Nov 2009 11:30:20 +0100 Subject: statuscache and fatal Message-ID: <8c895a0b0911040230g2ffba9d2k659c0d48460f9a3@mail.gmail.com> Hello, I wonder why is a "fatal" function called when the statuscache backend can't be opened. This makes the process terminate (then fork again, then terminate, then...). This is not logical, because statuscache is... a cache ! Cyrus should work fine without (and actually, it does, with the patch attached). As we use an sql backend for statuscache, when the sql server is not available, we prefer that cyrus-imapd works without cache than a total breakdown. -- Cyril -------------- next part -------------- A non-text attachment was scrubbed... Name: sc_degraded.patch Type: text/x-diff Size: 385 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20091104/fb31c729/attachment.bin From brong at fastmail.fm Wed Nov 4 07:39:20 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 4 Nov 2009 23:39:20 +1100 Subject: statuscache and fatal In-Reply-To: <8c895a0b0911040230g2ffba9d2k659c0d48460f9a3@mail.gmail.com> References: <8c895a0b0911040230g2ffba9d2k659c0d48460f9a3@mail.gmail.com> Message-ID: <20091104123920.GA24855@brong.net> On Wed, Nov 04, 2009 at 11:30:20AM +0100, Cyril Servant wrote: > Hello, > > I wonder why is a "fatal" function called when the statuscache backend > can't be opened. This makes the process terminate (then fork again, > then terminate, then...). This is not logical, because statuscache > is... a cache ! Cyrus should work fine without (and actually, it does, > with the patch attached). > > As we use an sql backend for statuscache, when the sql server is not > available, we prefer that cyrus-imapd works without cache than a total > breakdown. You have a good point :) I've committed a patch for it to CVS. I also protected the rest of the functions by not setting statuscache_dbopen and by testing for it at the start of each entry point. Bron. From brong at fastmail.fm Thu Nov 5 22:23:12 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 6 Nov 2009 14:23:12 +1100 Subject: XLIST extention Message-ID: <20091106032312.GA4273@brong.net> So, GMAIL has an XLIST command which names a few special folders by adding extra flags. Things like \Inbox, \Trash, \Drafts, \Sent. You can read more about it here: http://www.ietf.org/mail-archive/web/morg/current/msg00212.html http://groups.google.com/group/Gmail-Help-POP-and-IMAP-en/browse_thread/thread/a154105c54f020fb?pli=1 https://bugzilla.mozilla.org/show_bug.cgi?id=476260#c17 It means that Thunderbird can know what you folders are even if the names don't match its default, display the right icons and copy messages into the right places. Well, I thought that might come in handy, so this. Bron. -------------- next part -------------- A non-text attachment was scrubbed... Name: xlist.diff Type: text/x-diff Size: 7214 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20091106/0c5622ca/attachment.bin From awilliam at opengroupware.us Fri Nov 6 06:14:39 2009 From: awilliam at opengroupware.us (Adam Tauno Williams) Date: Fri, 06 Nov 2009 06:14:39 -0500 Subject: XLIST extention In-Reply-To: <20091106032312.GA4273@brong.net> References: <20091106032312.GA4273@brong.net> Message-ID: <1257506079.5575.4.camel@linux-m3mt> On Fri, 2009-11-06 at 14:23 +1100, Bron Gondwana wrote: > So, GMAIL has an XLIST command which names a few special folders by > adding extra flags. Things like \Inbox, \Trash, \Drafts, \Sent. > You can read more about it here: > http://www.ietf.org/mail-archive/web/morg/current/msg00212.html > http://groups.google.com/group/Gmail-Help-POP-and-IMAP-en/browse_thread/thread/a154105c54f020fb?pli=1 > https://bugzilla.mozilla.org/show_bug.cgi?id=476260#c17 > It means that Thunderbird can know what you folders are even if > the names don't match its default, display the right icons and > copy messages into the right places. > Well, I thought that might come in handy, so this. Sweet. Can these folders be flagged as Trash, Drafts, Sent via cyradmin somehow? [Maybe that is obvious in the patch, but I can't see it]. If mail clients would support DNS SRV we could get to am almost zero-config scenario [like XMPP clients]. But I guess that is a system administrator's pipe-deam. -- OpenGroupware developer: awilliam at whitemice.org OpenGroupare & Cyrus IMAPd documenation @ From brong at fastmail.fm Mon Nov 9 17:46:44 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 10 Nov 2009 09:46:44 +1100 Subject: XLIST extention In-Reply-To: <1257506079.5575.4.camel@linux-m3mt> References: <20091106032312.GA4273@brong.net> <1257506079.5575.4.camel@linux-m3mt> Message-ID: <20091109224643.GA2891@brong.net> On Fri, Nov 06, 2009 at 06:14:39AM -0500, Adam Tauno Williams wrote: > On Fri, 2009-11-06 at 14:23 +1100, Bron Gondwana wrote: > > So, GMAIL has an XLIST command which names a few special folders by > > adding extra flags. Things like \Inbox, \Trash, \Drafts, \Sent. > > You can read more about it here: > > http://www.ietf.org/mail-archive/web/morg/current/msg00212.html > > http://groups.google.com/group/Gmail-Help-POP-and-IMAP-en/browse_thread/thread/a154105c54f020fb?pli=1 > > https://bugzilla.mozilla.org/show_bug.cgi?id=476260#c17 > > It means that Thunderbird can know what you folders are even if > > the names don't match its default, display the right icons and > > copy messages into the right places. > > Well, I thought that might come in handy, so this. > > Sweet. Can these folders be flagged as Trash, Drafts, Sent via cyradmin > somehow? [Maybe that is obvious in the patch, but I can't see it]. Yeah, it really needs to be, doesn't it. I think I'll use one of the "spare" fields in the index header as "XLIST_FOLDER" or something. Make it an enum type, and make it settable as an annotation (I'm separately working on implementing the METADATA RFC rather than the SETANNOTATION early draft that Cyrus currently supports) > If mail clients would support DNS SRV we could get to am almost > zero-config scenario [like XMPP clients]. But I guess that is a system > administrator's pipe-deam. Yeah - the Thunderbird developers are wary of DNS cache poisoning being used to hack account details. Can sort of understand that. But zeroconfig sure would be nice. What about ACAP again? ;) Bron. From prlw1 at cam.ac.uk Fri Nov 13 10:10:53 2009 From: prlw1 at cam.ac.uk (Patrick Welche) Date: Fri, 13 Nov 2009 15:10:53 +0000 Subject: mmap cache file trouble Message-ID: <20091113151053.GC6107@quartz.inf.phy.cam.ac.uk> With cyrus-imap (and mutt) code of 11 November, I just got: IOERROR: mapping cache file for user.prlw1.http: Cannot allocate memory Fatal error: failed to mmap cache file I think that cyrus can work on systems with no mmap, so if mmap fails, could mmap-avoiding methods be used instead (read/write)? (Haven't checked the code...) But there is another side to this which puzzles me. After upgrading the system, I essentially did reconstruct user/prlw1 opened INBOX (about 233,000 messages) using mutt tagged 9651 of them copied the tagged messages to an empty, freshly created user/prlw1/http During this phase, the imap server failed to mmap the new mailbox's cache file. INBOX with all its messages has -rw------- 1 cyrus mail 347M Nov 13 05:23 cyrus.cache -rw------- 1 cyrus mail 155B Nov 12 10:16 cyrus.header -rw------- 1 cyrus mail 20M Nov 13 05:23 cyrus.index # ls -1 | wc -l 233797 yet new http folder, which should contain far fewer messages has -rw------- 1 cyrus mail 804M Nov 12 21:19 cyrus.cache -rw------- 1 cyrus mail 155B Nov 12 18:56 cyrus.header -rw------- 1 cyrus mail 41M Nov 12 21:19 cyrus.index # ls -1 | wc -l 491568 So what are all these files? # ls -liS 1167478 -rw------- 4 cyrus mail 28342472 Jan 28 2008 113844. 1167478 -rw------- 4 cyrus mail 28342472 Jan 28 2008 260251. 1167478 -rw------- 4 cyrus mail 28342472 Jan 28 2008 383849. 1149545 -rw------- 4 cyrus mail 21925261 Sep 24 2007 245262. 1149545 -rw------- 4 cyrus mail 21925261 Sep 24 2007 368860. 1149545 -rw------- 4 cyrus mail 21925261 Sep 24 2007 98855. 1233627 -rw------- 5 cyrus mail 21795046 Jun 12 16:41 160927. 1233627 -rw------- 5 cyrus mail 21795046 Jun 12 16:41 307334. 1233627 -rw------- 5 cyrus mail 21795046 Jun 12 16:41 430932. 1233627 -rw------- 5 cyrus mail 21795046 Jun 12 16:41 474643. 1172884 -rw------- 4 cyrus mail 20400877 Mar 3 2008 118751. 1172884 -rw------- 4 cyrus mail 20400877 Mar 3 2008 265158. 1172884 -rw------- 4 cyrus mail 20400877 Mar 3 2008 388756. Somehow multiple copies of the same file are being written with different filenames, yet same inode number. Any guesses on what is happening? (I have the telemetry log) (NetBSD-current/i386, ffs+wapbl/raidframe 1 filesystem) Cheers, Patrick From jlar310 at gmail.com Sat Nov 14 23:55:19 2009 From: jlar310 at gmail.com (Jeff) Date: Sat, 14 Nov 2009 22:55:19 -0600 Subject: Changing message filename convention Message-ID: Without getting into the details of why I want to do this, is there a quick way to hack the source to change the file naming convention for message files. I have reason to not want the trailing dot. I am working with 2.2.12 SRPM from CentOS 4.8. I have found one instance in imap/message.c in function mailbox_message_get_fname where the message uid has the dot appended to create a file name, but there must be others because that one change didn't work. I don't get any errors in the logs, message files are created with the naming convention I want, but the mail client can't pull down the content of the message and the subject and sender are blank in the message list. Thanks, Jeff From dpc22 at cam.ac.uk Sun Nov 15 03:51:02 2009 From: dpc22 at cam.ac.uk (David Carter) Date: Sun, 15 Nov 2009 08:51:02 +0000 (GMT) Subject: Changing message filename convention In-Reply-To: References: Message-ID: On Sat, 14 Nov 2009, Jeff wrote: > Without getting into the details of why I want to do this, is there a > quick way to hack the source to change the file naming convention for > message files. I have reason to not want the trailing dot. The trailing dot is there to generate unique names, so you don't have any conflict with the mailbox namespace. For example a mailbox named "42" or "cyrus.index". I really wouldn't recommend this. -- David Carter Email: David.Carter at ucs.cam.ac.uk University Computing Service, Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. From brong at fastmail.fm Sun Nov 15 05:52:33 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sun, 15 Nov 2009 21:52:33 +1100 Subject: Changing message filename convention In-Reply-To: References: Message-ID: <20091115105233.GA3745@brong.net> On Sun, Nov 15, 2009 at 08:51:02AM +0000, David Carter wrote: > On Sat, 14 Nov 2009, Jeff wrote: > > >Without getting into the details of why I want to do this, is > >there a quick way to hack the source to change the file naming > >convention for message files. I have reason to not want the > >trailing dot. > > The trailing dot is there to generate unique names, so you don't > have any conflict with the mailbox namespace. For example a mailbox > named "42" or "cyrus.index". I really wouldn't recommend this. I would recomment 12345.eml if you were going to do this, rather than just strip the dot. http://filext.com/file-extension/EML It shouldn't be too hard to add, but it's a pain to have to convert on a live instance of course. Also - I suspect there's a pile of code that depends on the \d+\. pattern existing. Bron. Bron. From jlar310 at gmail.com Sun Nov 15 09:09:02 2009 From: jlar310 at gmail.com (Jeff) Date: Sun, 15 Nov 2009 08:09:02 -0600 Subject: Changing message filename convention In-Reply-To: <20091115105233.GA3745@brong.net> References: <20091115105233.GA3745@brong.net> Message-ID: On Sun, Nov 15, 2009 at 4:52 AM, Bron Gondwana wrote: > On Sun, Nov 15, 2009 at 08:51:02AM +0000, David Carter wrote: >> On Sat, 14 Nov 2009, Jeff wrote: >> >> >Without getting into the details of why I want to do this, is >> >there a quick way to hack the source to change the file naming >> >convention for message files. I have reason to not want the >> >trailing dot. >> >> The trailing dot is there to generate unique names, so you don't >> have any conflict with the mailbox namespace. For example a mailbox >> named "42" or "cyrus.index". I really wouldn't recommend this. > > I would recomment 12345.eml if you were going to do this, rather than > just strip the dot. > > http://filext.com/file-extension/EML > > It shouldn't be too hard to add, but it's a pain to have to convert > on a live instance of course. > > Also - I suspect there's a pile of code that depends on the \d+\. > pattern existing. Yes, .eml was my plan, but changing only mailbox.c breaks things. I was looking for some pointers on which other source files made assumptions about the expected file name. Jeff From brong at fastmail.fm Sun Nov 15 15:34:18 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Mon, 16 Nov 2009 07:34:18 +1100 Subject: Changing message filename convention In-Reply-To: References: <20091115105233.GA3745@brong.net> Message-ID: <20091115203418.GA7243@brong.net> On Sun, Nov 15, 2009 at 08:09:02AM -0600, Jeff wrote: > On Sun, Nov 15, 2009 at 4:52 AM, Bron Gondwana wrote: > > On Sun, Nov 15, 2009 at 08:51:02AM +0000, David Carter wrote: > >> On Sat, 14 Nov 2009, Jeff wrote: > >> > >> >Without getting into the details of why I want to do this, is > >> >there a quick way to hack the source to change the file naming > >> >convention for message files. I have reason to not want the > >> >trailing dot. > >> > >> The trailing dot is there to generate unique names, so you don't > >> have any conflict with the mailbox namespace. For example a mailbox > >> named "42" or "cyrus.index". I really wouldn't recommend this. > > > > I would recomment 12345.eml if you were going to do this, rather than > > just strip the dot. > > > > http://filext.com/file-extension/EML > > > > It shouldn't be too hard to add, but it's a pain to have to convert > > on a live instance of course. > > > > Also - I suspect there's a pile of code that depends on the \d+\. > > pattern existing. > > Yes, .eml was my plan, but changing only mailbox.c breaks things. I > was looking for some pointers on which other source files made > assumptions about the expected file name. There's no guarantee I've got them all, and they're totally untested, but here's a couple of patches. I'm planning to apply the first one (once it's tested obviously) but not necessarily the second one! Putting all the filename creating logic in one place is sane for sure. Basically the pattern "%lu." in a sprintf is a likely candidate. I haven't audited all the other possibilities yet though! Bron. -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Add-and-use-mailbox_message_fname-implementation.patch Type: text/x-diff Size: 16910 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20091116/a61561b7/attachment.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-rename-mailbox-options.patch Type: text/x-diff Size: 1600 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20091116/a61561b7/attachment-0001.bin From brong at fastmail.fm Mon Nov 16 00:32:11 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Mon, 16 Nov 2009 16:32:11 +1100 Subject: Changing message filename convention In-Reply-To: <20091115203418.GA7243@brong.net> References: <20091115105233.GA3745@brong.net> <20091115203418.GA7243@brong.net> Message-ID: <20091116053210.GA30946@brong.net> On Mon, Nov 16, 2009 at 07:34:18AM +1100, Bron Gondwana wrote: > There's no guarantee I've got them all, and they're totally untested, > but here's a couple of patches. I'm planning to apply the first one > (once it's tested obviously) but not necessarily the second one! > Putting all the filename creating logic in one place is sane for sure. Here's a less broken version of that patch - at least it compiles! ALSO - grep the sources for "skip non-message files". There might be other places too, but that's two of them. Bron. -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Add-and-use-mailbox_message_fname-implementation.patch Type: text/x-diff Size: 17090 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20091116/17bab1e9/attachment-0001.bin From dilyan.palauzov at aegee.org Mon Nov 16 07:13:45 2009 From: dilyan.palauzov at aegee.org (=?UTF-8?B?0JTQuNC70Y/QvSDQn9Cw0LvQsNGD0LfQvtCy?=) Date: Mon, 16 Nov 2009 13:13:45 +0100 Subject: Policy for Accepting Patches Message-ID: <4B0141F9.4090903@aegee.org> Hello, Can somebody enlighten me what is the policy for accepting patches in cyrus/imap? I mean particularly bugs #3054 and 3113. Will they be integrated in future releases, who can tell me when will this happen? Thanks in advance for the feedback, ????? From dave64 at andrew.cmu.edu Mon Nov 16 07:40:14 2009 From: dave64 at andrew.cmu.edu (Dave McMurtrie) Date: Mon, 16 Nov 2009 07:40:14 -0500 Subject: Policy for Accepting Patches In-Reply-To: <4B0141F9.4090903@aegee.org> References: <4B0141F9.4090903@aegee.org> Message-ID: <4B01482E.7000207@andrew.cmu.edu> ????? ???????? wrote: > Hello, > > Can somebody enlighten me what is the policy for accepting patches in > cyrus/imap? I mean particularly bugs #3054 and 3113. Will they be > integrated in future releases, who can tell me when will this happen? Good morning, Officially, http://cyrusimap.web.cmu.edu/bylaws.html Since that page isn't completely up to date, there are currently two people who do the lion's share of CVS commits. Ken Murchison who works here at Carnegie Mellon is the lead Cyrus developer and Bron Gondwana from fastmail.fm. Basically, as time permits, they review patches and apply them. In addition, a small number of other people that you probably see posting to info-cyrus and cyrus-devel also have CVS access. Thanks, Dave -- Dave McMurtrie, SPE Email Systems Team Leader Carnegie Mellon University, Computing Services From jlar310 at gmail.com Mon Nov 16 09:07:37 2009 From: jlar310 at gmail.com (Jeff) Date: Mon, 16 Nov 2009 08:07:37 -0600 Subject: Changing message filename convention In-Reply-To: <20091116053210.GA30946@brong.net> References: <20091115105233.GA3745@brong.net> <20091115203418.GA7243@brong.net> <20091116053210.GA30946@brong.net> Message-ID: On Sun, Nov 15, 2009 at 11:32 PM, Bron Gondwana wrote: > On Mon, Nov 16, 2009 at 07:34:18AM +1100, Bron Gondwana wrote: >> There's no guarantee I've got them all, and they're totally untested, >> but here's a couple of patches. ?I'm planning to apply the first one >> (once it's tested obviously) but not necessarily the second one! >> Putting all the filename creating logic in one place is sane for sure. > > > Here's a less broken version of that patch - at least it compiles! > > ALSO - grep the sources for "skip non-message files". ?There might > be other places too, but that's two of them. Wow. Above and beyond the call of duty. I'm glad I could be a catalyst for common sense in the source code. I wan't to change as little as possible on our mail servers running an older version, but the patches give me a good strong push in the right direction even though I can't apply them directly. Curiously, the latest cyrus-imapd has many more hard-coded file name instances that 2.2.12 Thanks, Jeff From brong at fastmail.fm Mon Nov 16 17:50:15 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 17 Nov 2009 09:50:15 +1100 Subject: Changing message filename convention In-Reply-To: References: <20091115105233.GA3745@brong.net> <20091115203418.GA7243@brong.net> <20091116053210.GA30946@brong.net> Message-ID: <20091116225014.GA2111@brong.net> On Mon, Nov 16, 2009 at 08:07:37AM -0600, Jeff wrote: > On Sun, Nov 15, 2009 at 11:32 PM, Bron Gondwana wrote: > > On Mon, Nov 16, 2009 at 07:34:18AM +1100, Bron Gondwana wrote: > >> There's no guarantee I've got them all, and they're totally untested, > >> but here's a couple of patches. ?I'm planning to apply the first one > >> (once it's tested obviously) but not necessarily the second one! > >> Putting all the filename creating logic in one place is sane for sure. > > > > > > Here's a less broken version of that patch - at least it compiles! > > > > ALSO - grep the sources for "skip non-message files". ?There might > > be other places too, but that's two of them. > > Wow. Above and beyond the call of duty. I'm glad I could be a catalyst > for common sense in the source code. I wan't to change as little as > possible on our mail servers running an older version, but the patches > give me a good strong push in the right direction even though I can't > apply them directly. Curiously, the latest cyrus-imapd has many more > hard-coded file name instances that 2.2.12 You'll find a lot of that is related to the replication code - but yes, there's plenty of stuff that's ripe for refactoring in there! Interestingly, the API definition in mailbox.h already existed, but there was no implementation yet - so I wrote an implementation :) I'll do a bunch more testing and just checking that this makes sense, then roll it back into Cyrus once it's run on the FastMail servers for a bit for testing. It's always good to tidy code up when you get chance. I've found at least one other bug just by reading through the code as well, so I'll push that fix back too! Bron. From brong at fastmail.fm Mon Nov 16 21:45:23 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 17 Nov 2009 13:45:23 +1100 Subject: Policy for Accepting Patches In-Reply-To: <4B0141F9.4090903@aegee.org> References: <4B0141F9.4090903@aegee.org> Message-ID: <20091117024523.GA3553@brong.net> On Mon, Nov 16, 2009 at 01:13:45PM +0100, ????? ???????? wrote: > Hello, > > Can somebody enlighten me what is the policy for accepting patches > in cyrus/imap? I mean particularly bugs #3054 and 3113. Will they > be integrated in future releases, who can tell me when will this > happen? 3113 was committed to CVS back in March (I made a typo in the commit message and called it 3133, sorry) It should be in the current release. 3054 - I think the issue is that you haven't actually made a case for the change. What is the benefit to doing it your way? Also, I get confused by the diff - it has ! rather than + and it doesn't look like you're freeing str2. Bron. From wes at umich.edu Mon Nov 16 21:51:15 2009 From: wes at umich.edu (Wesley Craig) Date: Mon, 16 Nov 2009 21:51:15 -0500 Subject: Policy for Accepting Patches In-Reply-To: <4B01482E.7000207@andrew.cmu.edu> References: <4B0141F9.4090903@aegee.org> <4B01482E.7000207@andrew.cmu.edu> Message-ID: On 16 Nov 2009, at 07:40, Dave McMurtrie wrote: > ????? ???????? wrote: >> Can somebody enlighten me what is the policy for accepting patches >> in cyrus/imap? I mean particularly bugs #3054 and 3113. Will >> they be integrated in future releases, who can tell me when will >> this happen? > > Officially, http://cyrusimap.web.cmu.edu/bylaws.html > > Since that page isn't completely up to date, there are currently > two people who do the lion's share of CVS commits. Ken Murchison > who works here at Carnegie Mellon is the lead Cyrus developer and > Bron Gondwana from fastmail.fm. Basically, as time permits, they > review patches and apply them. As far as they apply to the question (not very much), the bylaws seem pretty up to date to me. There's no place in the bylaws for noting current committers, just initial committers. Perhaps that's an oversight. Bron was invited to be a committer some time ago. None of that really answers ?????'s question, tho. The way to get your patches integrated is to get a committer interested. In the case of 3054, the patch is backwards, but it seems like Ken has accepted the bug. So, the answer there is probably "bother Ken" or if you give up on Ken, find someone else who considers the bug (if it is one) important enough to make the small effort. For 3113, Bron apparently committed a fix in March... :wes From brong at fastmail.fm Mon Nov 16 23:12:17 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 17 Nov 2009 15:12:17 +1100 Subject: Charset changes pushed to CVS Message-ID: <20091117041217.GA4907@brong.net> Hi All, I've pushed the characterset handling changes to CVS. They solve all my remaining bugs in bugzilla, and are a joy to work with :) They've been running on the Fastmail servers for about 6 months happily. This brings full UTF8 support to sieve scripts, charset decoding of fields for comparision, and much better handling of whitespace in search terms. You will need to do a reconstruct before these changes work with search. In particular, a search for "foo bar" will fail to find it without the reconstruct. A workaround is to search for 'or "foobar" "foo bara'" which will find both old instances where spaces were squashed AND new instances where they weren't! I've also committed the user_folder_limit option, and a couple of minor bugfixes. Still pending is the CRC32 changeset. We've been very happy with it in production on our machines. The only downside is that openssl is required. I'd like to have our own CRC32 implementation embedded in the code as well, but haven't got around to massaging the public domain one to suit yet! Bron. From brong at fastmail.fm Tue Nov 17 07:28:49 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 17 Nov 2009 23:28:49 +1100 Subject: ANNOTATEMORE => METADATA and rfc 5464 Message-ID: <20091117122849.GA4503@brong.net> I'm in the process of implementing rfc 5464, which is what the ANNOTATEMORE drafts turned into. Unfortunately, Cyrus' support is an early draft, before the paths to everything were changed and the commands were renamed. It would be great to be complient, and there is software out there like Kolab which would benefit from it. Also, the database format is pretty nasty - complete with nulls embedded in keys and other fun stuff (like platform dependent type lengths codified in the format, ick) So I'm thinking: create a new metadata.db, require a conversion on upgrade from annotations.db. I had a look, and none of our servers had ANY annotations until I added a /comment to my INBOX for testing. Does anybody out there use annotations much? Does anybody know any code that would be broken by changing the way annotations are done? Thanks, Bron. From thomas.jarosch at intra2net.com Tue Nov 17 07:36:38 2009 From: thomas.jarosch at intra2net.com (Thomas Jarosch) Date: Tue, 17 Nov 2009 13:36:38 +0100 Subject: ANNOTATEMORE => METADATA and rfc 5464 In-Reply-To: <20091117122849.GA4503@brong.net> References: <20091117122849.GA4503@brong.net> Message-ID: <200911171336.38813.thomas.jarosch@intra2net.com> Hello Bron, On Tuesday, 17. November 2009 13:28:49 Bron Gondwana wrote: > Does anybody out there use annotations much? Does anybody know any code > that would be broken by changing the way annotations are done? The Kolab project and the horde framework counterparts use it to store the type of a groupware folder like "contact", "calendar" etc. This information is pretty vital :) Cheers, Thomas From martin.konold at erfrakon.de Tue Nov 17 07:37:47 2009 From: martin.konold at erfrakon.de (Martin Konold) Date: Tue, 17 Nov 2009 13:37:47 +0100 Subject: ANNOTATEMORE => METADATA and rfc 5464 In-Reply-To: <20091117122849.GA4503@brong.net> References: <20091117122849.GA4503@brong.net> Message-ID: <200911171337.47671.martin.konold@erfrakon.de> On Tuesday 17 November 2009 13:28:49 Bron Gondwana wrote: Hi, > to everything were changed and the commands were renamed. It would > be great to be complient, and there is software out there like Kolab > which would benefit from it. > > Also, the database format is pretty nasty - complete with nulls > embedded in keys and other fun stuff (like platform dependent type > lengths codified in the format, ick) > > So I'm thinking: create a new metadata.db, require a conversion on upgrade > from annotations.db. I had a look, and none of our servers had ANY > annotations until I added a /comment to my INBOX for testing. > > Does anybody out there use annotations much? Does anybody know any code > that would be broken by changing the way annotations are done? Yes, Kolab makes heavy/critical use of annotations. Provided there is a suitable upgrade path I am fully in favour of moving towards rfc 5464. Yours, -- martin From murch at andrew.cmu.edu Tue Nov 17 09:03:11 2009 From: murch at andrew.cmu.edu (Ken Murchison) Date: Tue, 17 Nov 2009 09:03:11 -0500 Subject: ANNOTATEMORE => METADATA and rfc 5464 In-Reply-To: <20091117122849.GA4503@brong.net> References: <20091117122849.GA4503@brong.net> Message-ID: <4B02AD1F.9030606@andrew.cmu.edu> What is your new format proposal? Bron Gondwana wrote: > I'm in the process of implementing rfc 5464, which is what the > ANNOTATEMORE drafts turned into. > > Unfortunately, Cyrus' support is an early draft, before the paths > to everything were changed and the commands were renamed. It would > be great to be complient, and there is software out there like Kolab > which would benefit from it. > > Also, the database format is pretty nasty - complete with nulls > embedded in keys and other fun stuff (like platform dependent type > lengths codified in the format, ick) > > So I'm thinking: create a new metadata.db, require a conversion on upgrade > from annotations.db. I had a look, and none of our servers had ANY > annotations until I added a /comment to my INBOX for testing. > > Does anybody out there use annotations much? Does anybody know any code > that would be broken by changing the way annotations are done? > > Thanks, > > Bron. > -- Kenneth Murchison Systems Programmer Carnegie Mellon University From brong at fastmail.fm Tue Nov 17 16:01:35 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 18 Nov 2009 08:01:35 +1100 Subject: ANNOTATEMORE => METADATA and rfc 5464 In-Reply-To: <4B02AD1F.9030606@andrew.cmu.edu> References: <20091117122849.GA4503@brong.net> <4B02AD1F.9030606@andrew.cmu.edu> Message-ID: <20091117210135.GA5302@brong.net> On Tue, Nov 17, 2009 at 09:03:11AM -0500, Ken Murchison wrote: > What is your new format proposal? I'll see :) Not sure yet - but mainly not sizeof(unsigned long)! Bron. From murch at andrew.cmu.edu Tue Nov 17 16:17:51 2009 From: murch at andrew.cmu.edu (Ken Murchison) Date: Tue, 17 Nov 2009 16:17:51 -0500 Subject: ANNOTATEMORE => METADATA and rfc 5464 In-Reply-To: <20091117210135.GA5302@brong.net> References: <20091117122849.GA4503@brong.net> <4B02AD1F.9030606@andrew.cmu.edu> <20091117210135.GA5302@brong.net> Message-ID: <4B0312FF.9080302@andrew.cmu.edu> Bron Gondwana wrote: > On Tue, Nov 17, 2009 at 09:03:11AM -0500, Ken Murchison wrote: >> What is your new format proposal? > > I'll see :) Not sure yet - but mainly not sizeof(unsigned long)! If we make a wholesale change to the database, perhaps this might be something we put in the 2.4 branch. It already has some partial/complete extensions like QRESYNC, LIST-EXTENDED, URLAUTH=BINARY and COMPRESS (which I backported to 2.3). I was also thinking that although the charset changes have been fully tested at Fastmail that it too might be a candidate for 2.4. -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University From Rudy.Gevaert at UGent.be Tue Nov 17 17:47:49 2009 From: Rudy.Gevaert at UGent.be (Rudy Gevaert) Date: Tue, 17 Nov 2009 23:47:49 +0100 Subject: ANNOTATEMORE => METADATA and rfc 5464 In-Reply-To: <20091117122849.GA4503@brong.net> References: <20091117122849.GA4503@brong.net> Message-ID: <20091117224749.GB13535@ugent.be> On Tue, Nov 17, 2009 at 11:28:49PM +1100, Bron Gondwana wrote: > > Does anybody out there use annotations much? Does anybody know any code > that would be broken by changing the way annotations are done? I'm the only one who uses it here ;) -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Rudy Gevaert Rudy.Gevaert at UGent.be tel:+32 9 264 4734 Directie ICT, afd. Infrastructuur Direction ICT, Infrastructure dept. Groep Systemen Systems group Universiteit Gent Ghent University Krijgslaan 281, gebouw S9, 9000 Gent, Belgie www.UGent.be -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- From brong at fastmail.fm Tue Nov 17 18:27:46 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 18 Nov 2009 10:27:46 +1100 Subject: ANNOTATEMORE => METADATA and rfc 5464 In-Reply-To: <4B0312FF.9080302@andrew.cmu.edu> References: <20091117122849.GA4503@brong.net> <4B02AD1F.9030606@andrew.cmu.edu> <20091117210135.GA5302@brong.net> <4B0312FF.9080302@andrew.cmu.edu> Message-ID: <20091117232746.GA6069@brong.net> On Tue, Nov 17, 2009 at 04:17:51PM -0500, Ken Murchison wrote: > Bron Gondwana wrote: > >On Tue, Nov 17, 2009 at 09:03:11AM -0500, Ken Murchison wrote: > >>What is your new format proposal? > > > >I'll see :) Not sure yet - but mainly not sizeof(unsigned long)! > > If we make a wholesale change to the database, perhaps this might be > something we put in the 2.4 branch. It already has some > partial/complete extensions like QRESYNC, LIST-EXTENDED, > URLAUTH=BINARY and COMPRESS (which I backported to 2.3). > > I was also thinking that although the charset changes have been > fully tested at Fastmail that it too might be a candidate for 2.4. Yeah, fair enough! I did commit them to CVS, but it's easy enough to back them out and commit to a branch instead. Do we have a roadmap for what else people want on the 2.4 branch? I'd be happy to put a bit more effort into polishing up those features that are there so we can ship a 2.4 soonish. Say by April next year, which gives us 6 months to prepare. Bron. From murch at andrew.cmu.edu Tue Nov 17 18:37:13 2009 From: murch at andrew.cmu.edu (Ken Murchison) Date: Tue, 17 Nov 2009 18:37:13 -0500 Subject: ANNOTATEMORE => METADATA and rfc 5464 In-Reply-To: <20091117232746.GA6069@brong.net> References: <20091117122849.GA4503@brong.net> <4B02AD1F.9030606@andrew.cmu.edu> <20091117210135.GA5302@brong.net> <4B0312FF.9080302@andrew.cmu.edu> <20091117232746.GA6069@brong.net> Message-ID: <4B0333A9.50901@andrew.cmu.edu> Bron Gondwana wrote: > On Tue, Nov 17, 2009 at 04:17:51PM -0500, Ken Murchison wrote: >> Bron Gondwana wrote: >>> On Tue, Nov 17, 2009 at 09:03:11AM -0500, Ken Murchison wrote: >>>> What is your new format proposal? >>> I'll see :) Not sure yet - but mainly not sizeof(unsigned long)! >> If we make a wholesale change to the database, perhaps this might be >> something we put in the 2.4 branch. It already has some >> partial/complete extensions like QRESYNC, LIST-EXTENDED, >> URLAUTH=BINARY and COMPRESS (which I backported to 2.3). >> >> I was also thinking that although the charset changes have been >> fully tested at Fastmail that it too might be a candidate for 2.4. > > Yeah, fair enough! I did commit them to CVS, but it's easy enough to > back them out and commit to a branch instead. > > Do we have a roadmap for what else people want on the 2.4 branch? > I'd be happy to put a bit more effort into polishing up those features > that are there so we can ship a 2.4 soonish. Say by April next year, > which gives us 6 months to prepare. My original vision for 2.4 was to be compliant with the LEMONADE v2 profile. At this point is can morph into anything we want. Some of the 2.4 features required changes that I felt were too in depth to put into a relatively stable 2.3. I'm pretty close to having the time to dive back into the 2.4 code. The first thing that needs to be done is to merge all of the new 2.3 stuff into 2.4. -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University From brong at fastmail.fm Wed Nov 18 06:34:44 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 18 Nov 2009 22:34:44 +1100 Subject: ANNOTATEMORE => METADATA and rfc 5464 In-Reply-To: <4B0333A9.50901@andrew.cmu.edu> References: <20091117122849.GA4503@brong.net> <4B02AD1F.9030606@andrew.cmu.edu> <20091117210135.GA5302@brong.net> <4B0312FF.9080302@andrew.cmu.edu> <20091117232746.GA6069@brong.net> <4B0333A9.50901@andrew.cmu.edu> Message-ID: <20091118113444.GA18502@brong.net> On Tue, Nov 17, 2009 at 06:37:13PM -0500, Ken Murchison wrote: > Bron Gondwana wrote: > >On Tue, Nov 17, 2009 at 04:17:51PM -0500, Ken Murchison wrote: > >>Bron Gondwana wrote: > >>>On Tue, Nov 17, 2009 at 09:03:11AM -0500, Ken Murchison wrote: > >>>>What is your new format proposal? > >>>I'll see :) Not sure yet - but mainly not sizeof(unsigned long)! > >>If we make a wholesale change to the database, perhaps this might be > >>something we put in the 2.4 branch. It already has some > >>partial/complete extensions like QRESYNC, LIST-EXTENDED, > >>URLAUTH=BINARY and COMPRESS (which I backported to 2.3). > >> > >>I was also thinking that although the charset changes have been > >>fully tested at Fastmail that it too might be a candidate for 2.4. > > > >Yeah, fair enough! I did commit them to CVS, but it's easy enough to > >back them out and commit to a branch instead. > > > >Do we have a roadmap for what else people want on the 2.4 branch? > >I'd be happy to put a bit more effort into polishing up those features > >that are there so we can ship a 2.4 soonish. Say by April next year, > >which gives us 6 months to prepare. > > My original vision for 2.4 was to be compliant with the LEMONADE v2 profile. Sounds like a good plan :) > At this point is can morph into anything we want. Some of the 2.4 > features required changes that I felt were too in depth to put into > a relatively stable 2.3. > > I'm pretty close to having the time to dive back into the 2.4 code. > The first thing that needs to be done is to merge all of the new 2.3 > stuff into 2.4. Sure. I'm happy to put the more unstable stuff (even including the charset changes) into 2.4. I just want to have some idea that they won't get stuck waiting for some lemonade scented towlettes forever. If we commit to something like "2.4 will ship with whatever features we have ready in April 2010" then we have a decent timeline to figure out what we'll actually have time to support, and focus on getting that stable. In particular, that's a good time for all the format changes to land all at once, and potentially new defaults for a bunch of config values that have occured over the years. In particular things like collation order for the mailboxes.db should just be fixed. Dump and restore your DB and do it this way! In general I'd like to simplify configuration where possible even at the expense of backwards compatibility. We could have a "config_version: 2.4" key which needs to be changed over major version differences, and keep configs compatible within the major versions. (2.x that is) Bron. From lists at egidy.de Wed Nov 18 07:21:07 2009 From: lists at egidy.de (Gerd v. Egidy) Date: Wed, 18 Nov 2009 13:21:07 +0100 Subject: ANNOTATEMORE => METADATA and rfc 5464 In-Reply-To: <20091117122849.GA4503@brong.net> References: <20091117122849.GA4503@brong.net> Message-ID: <200911181321.07900.lists@egidy.de> Hi Bron, thanks for looking into the annotation/metadata stuff. > So I'm thinking: create a new metadata.db, require a conversion on upgrade > from annotations.db. I had a look, and none of our servers had ANY > annotations until I added a /comment to my INBOX for testing. Just to be sure: do you plan to change the current annotatemore-code so that it will access the new database and an old client still using annotatemore will still work? > Does anybody out there use annotations much? Does anybody know any code > that would be broken by changing the way annotations are done? Given that there is code to convert the old annotations.db to metadata.db I don't see any problems for us. Our backup code will probably need some tweaking, but when the new db format is more sane than the current mess I don't see any problems with that. Kind regards, Gerd -- Address (better: trap) for people I really don't want to get mail from: jonas at cactusamerica.com From awilliam at opengroupware.us Wed Nov 18 08:42:01 2009 From: awilliam at opengroupware.us (Adam Tauno Williams) Date: Wed, 18 Nov 2009 08:42:01 -0500 Subject: ANNOTATEMORE => METADATA and rfc 5464 In-Reply-To: <20091118113444.GA18502@brong.net> References: <20091117122849.GA4503@brong.net> <4B02AD1F.9030606@andrew.cmu.edu> <20091117210135.GA5302@brong.net> <4B0312FF.9080302@andrew.cmu.edu> <20091117232746.GA6069@brong.net> <4B0333A9.50901@andrew.cmu.edu> <20091118113444.GA18502@brong.net> Message-ID: <1258551721.11136.6.camel@linux-m3mt> > > >Do we have a roadmap for what else people want on the 2.4 branch? > > >I'd be happy to put a bit more effort into polishing up those features > > >that are there so we can ship a 2.4 soonish. Say by April next year, > > >which gives us 6 months to prepare. > > My original vision for 2.4 was to be compliant with the LEMONADE v2 profile. > Sounds like a good plan :) > > At this point is can morph into anything we want. Some of the 2.4 > > features required changes that I felt were too in depth to put into > > a relatively stable 2.3. > > I'm pretty close to having the time to dive back into the 2.4 code. > > The first thing that needs to be done is to merge all of the new 2.3 > > stuff into 2.4. > Sure. I'm happy to put the more unstable stuff (even including the > charset changes) into 2.4. I just want to have some idea that they > won't get stuck waiting for some lemonade scented towlettes forever. Is Lemonade still alive? I haven't heard much about it, and I'm pretty sure I read a couple of articles/BLOGs that the initiative was essentially dead. > Dump and restore your DB and do it this way! In general I'd like to > simplify configuration where possible even at the expense of backwards > compatibility. +1 Fine with me. > We could have a "config_version: 2.4" key which needs > to be changed over major version differences, and keep configs compatible > within the major versions. (2.x that is) -- OpenGroupware developer: awilliam at whitemice.org OpenGroupare & Cyrus IMAPd documenation @ From alexey.melnikov at isode.com Wed Nov 18 09:04:01 2009 From: alexey.melnikov at isode.com (Alexey Melnikov) Date: Wed, 18 Nov 2009 14:04:01 +0000 Subject: ANNOTATEMORE => METADATA and rfc 5464 In-Reply-To: <1258551721.11136.6.camel@linux-m3mt> References: <20091117122849.GA4503@brong.net> <4B02AD1F.9030606@andrew.cmu.edu> <20091117210135.GA5302@brong.net> <4B0312FF.9080302@andrew.cmu.edu> <20091117232746.GA6069@brong.net> <4B0333A9.50901@andrew.cmu.edu> <20091118113444.GA18502@brong.net> <1258551721.11136.6.camel@linux-m3mt> Message-ID: <4B03FED1.8050102@isode.com> Adam Tauno Williams wrote: >>>>Do we have a roadmap for what else people want on the 2.4 branch? >>>>I'd be happy to put a bit more effort into polishing up those features >>>>that are there so we can ship a 2.4 soonish. Say by April next year, >>>>which gives us 6 months to prepare. >>>> >>>> >>>My original vision for 2.4 was to be compliant with the LEMONADE v2 profile. >>> >>> >>Sounds like a good plan :) >> >> >>>At this point is can morph into anything we want. Some of the 2.4 >>>features required changes that I felt were too in depth to put into >>>a relatively stable 2.3. >>>I'm pretty close to having the time to dive back into the 2.4 code. >>>The first thing that needs to be done is to merge all of the new 2.3 >>>stuff into 2.4. >>> >>> >>Sure. I'm happy to put the more unstable stuff (even including the >>charset changes) into 2.4. I just want to have some idea that they >>won't get stuck waiting for some lemonade scented towlettes forever. >> >> >Is Lemonade still alive? I haven't heard much about it, and I'm pretty >sure I read a couple of articles/BLOGs that the initiative was >essentially dead. > IETF Lemonade WG successfully completed all work with publication of Lemonade Profile (RFC 5550) in August 2009. Multiple implementations of various parts of it exist, including opensource implementations such as Dovecot and Zimbra (and Cyrus, of course). From Cyril.Servant at atosorigin.com Thu Nov 19 11:44:06 2009 From: Cyril.Servant at atosorigin.com (Servant Cyril) Date: Thu, 19 Nov 2009 17:44:06 +0100 Subject: Pop optimisation Message-ID: <04DEF5D2955AEB468DCC881BDA2B50CE3EA75E608D@FRSPX100.fr01.awl.atosorigin.net> Hello, Here is a patch for optimizing pop. Let me explain : Here we have lots (millions) of mailboxes. Many people connect to pop every few minutes, doing LIST, and if there are mails, they do RETR and DELE. Most of time, there is no mail (for 138770 pop connections, there was no mail 92498 times => 66.6%). Without the patch, the seen, index, cache and header files are opened. With this patch, we only read statuscache.db (which is already opened) when there is no mail. On the stat image joined, you can see what's happening when we empty statuscache.db (at 10:51) : pop optimization doesn't work the first time a client connects to pop (Lots of reads), and then, as the same clients connect again to pop, reads slowly decrease. Without the patch, reads would stay high. -- Cyril Servant Ce message et les pi?ces jointes sont confidentiels et r?serv?s ? l'usage exclusif de ses destinataires. Il peut ?galement ?tre prot?g? par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir imm?diatement l'exp?diteur et de le d?truire. L'int?grit? du message ne pouvant ?tre assur?e sur Internet, la responsabilit? du groupe Atos Origin ne pourra ?tre recherch?e quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'exp?diteur ne donne aucune garantie ? cet ?gard et sa responsabilit? ne saurait ?tre recherch?e pour tout dommage r?sultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos Origin group liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. -------------- next part -------------- A non-text attachment was scrubbed... Name: optipop.patch Type: text/x-diff Size: 11977 bytes Desc: optipop.patch Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20091119/12bbc620/attachment-0001.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: statNFS.png Type: image/png Size: 24964 bytes Desc: statNFS.png Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20091119/12bbc620/attachment-0001.png