From brong at fastmail.fm Wed Mar 4 23:13:58 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 5 Mar 2009 15:13:58 +1100 Subject: RFC: Charset Conversion Routines In-Reply-To: <20090224031726.GA13743@brong.net> References: <20090224031726.GA13743@brong.net> Message-ID: <20090305041358.GA30612@brong.net> On Tue, Feb 24, 2009 at 02:17:26PM +1100, Bron Gondwana wrote: > I'm in the process of rewriting the lib/mkchartable.c > and lib/charset.c with the eventual goal being a more > flexible charset conversion API that can be used to > make sieve rules match on the decoded values, and > other funky things. OK - significantly more work done. It's now working correctly on my testbed. Behaviour is identical to the old code, using a reasonably large snapshot of email I had lying around. Reconstruct creates an idential cyrus.cache and downloads, etc all work correctly. There are some pathological error cases that might work slightly differently, though I got rid of the worst of those with a change to the couple of charset.t files that had GOSUBs in them. http://github.com/brong/cyrus-imapd/commit/78591d3a5c2f2ed5cd4d1bf935fffa073081198c brong at launde:/extra/src/git/cmu/cyrus-imapd$ git diff origin/master | diffstat b/imap/.cvsignore | 1 b/imap/Makefile.in | 6 b/imap/cyr_charset.c | 160 b/lib/Makefile.in | 15 b/lib/charset.c | 1794 +-- b/lib/charset.h | 22 b/lib/charset/iso-2022-jp.t | 33 b/lib/charset/iso-2022-kr.t | 12 b/lib/charset/unidata5_1.txt |19336 +++++++++++++++++++++++++++++++++++++++++++ b/lib/chartable.h | 27 b/lib/mkchartable.pl | 531 + lib/charset/unidata2.txt | 6629 -------------- lib/mkchartable.c | 974 -- 13 files changed, 20866 insertions(+), 8674 deletions(-) Yikes! So the changes to the .t files just conver the ESC tables into multibyte sequences for all valid escape codes in all mode tables, allowing the "invalid escape code" to drop you back in the current mode again. cyr_charset is just a little tool to allow you to see what the input in a particular charset produces as output. unidata5.1 overrides everything, because that's huge. I've made the code able to support the latest unicode standard including 24 bit codepoints. mkchartable is rewritten in perl rather than C, because it was so very, very much easier. I'd be willing to convert it back if people really, really don't want to depend on Perl, but I'd probably *sigh* a lot. charset.c is pretty much totally rewritten. Git tells me it's 70% changed, and actually logs it as a "rewrite" when committing. Major, major changes to how just about everything works. All "translations" are chainable, so you write code like this: struct convert_rock *translate = qp_init(); struct convert_rock *decode = table_init(charset); struct convert_rock *canon = canon_init(); struct convert_rock *toutf8 = uni_init(); struct convert_rock *tobuffer = buffer_init(0, 0); translate->next = decode; decode->next = canon; canon->next = toutf8; toutf8->next = tobuffer; convert_cat(translate, s); res = buffer_cstring(tobuffer); basic_free(translate); basic_free(decode); basic_free(canon); basic_free(toutf8); buffer_free(tobuffer); And you have a freshly malloced cstring in "res". It's annoying that you have to alloc and free in so many lines, but otherwise the API is simple to use, and easy to mix-and-match as required. Each layer has a "state" object, and gets called with a single character. It changes its state as requried, and possibly calls convert_putc on its "next" pointer with translated characters. Overall, it's a code saving, and it makes doing things like converting to utf8 rather than search form as easy as removing a translation layer. On the downside, it does cost a little more CPU for all the extra function calls, as opposed to the direct A => B translation tables of the old way. Running a full squatter on my mailboxes cost about 10% more CPU this way. I think it's justified for the flexibility and full unicode handling capability and all that jazz. I still need to document how this stuff works, in particular how the almost-stateless search consumer works! I've explained it on paper to Rob and Richard to make sure it makes sense to them... Comments and code review gratefully appreciated! I'll be doing some more testing here, and then possibly pushing it to production on one of our machines for a full smoketest! Bron. From brong at fastmail.fm Tue Mar 10 05:01:13 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 10 Mar 2009 20:01:13 +1100 Subject: Calling all regressions Message-ID: <1236675674.12318.1304575661@webmail.messagingengine.com> I was looking through the bug tracker, and one of the items was "skiplist issues in 2.2" - well, the ticket has morphed into that anyway. The question is - do we backport the skiplist bugfixes to Cyrus 2.2? For that matter, do we try to isolate just the actual bug fixes and backport them, or do we copy the whole cyrusdb_skiplist.c file (pretty self-contained, and the API is unchaged) back? What about other bugfixes in the 2.3 branch? There have been plenty, and not all related to new 2.3 branch features. So - I ask now. Does anybody know of any regressions in 2.3 that aren't present in the earlier branches? Even performance regressions are interesting to some sites. Is there any good reason for a site to continue running 2.2 or even 2.1? If there is - I want to know it, and I want to fix it, so we don't have to think about maintaining the old Cyrus. There aren't enough of us working on Cyrus to split the effort! Now - also of interest is the cost of having to change configurations when upgrading. 2.3.x is NOT a direct dropin for 2.[12].x because some defaults have changed. That's annoying. I don't know any good solution other than refusing to start or complaining very loudly if those keys are NOT present in the config file. I would make the error spit out: a) the backwards compatible value b) the new default value c) a BRIEF description of the benefits of using the new value. Along with a line to copy and paste into the config file for each of the above. Distro maintainers can of course avoid most of this by having clever upgrade scripts, but we need to make sure each site can find and fix the issues caused by an upgrade quickly. Ok, enough manifesto for me for one evening! Let me know about those regressions, or forever hold your peace as I push for deprecation of the old branches... -- Bron Gondwana brong at fastmail.fm From brong at fastmail.fm Thu Mar 12 07:58:52 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 12 Mar 2009 22:58:52 +1100 Subject: Make CACHE_ITEM_NEXT go (mostly) away Message-ID: <20090312115852.GB20450@brong.net> I'm sick of that interface. It blows. Random memory jumps trusting stuff in an mmaped file leads to reading from pretty much anywhere. At least we don't try to write! Still, segfaults galore. This doesn't change the fact that index.c is a copy-paster's enterprisy paradise, but at least it gives named access to all the bits of the cache record! The basic concept is this: struct cacheitem contains a length and a string pointer typedef cacherecord is 10 items (per NUMCACHEITEMS constant) ALL access goes through cache_parserecord() in one way or another (note: this is going to come in real handy if I ever resurrect checksummed cache records!) cache_parserecord fills a 'cacherecord' datastructure, and it returns the length of the full cacherecord (handy, you can just pass a null cacherecord pointer and get the size). If it returns a length of zero then the cache is bogus, so cacherecord is undefined. With that said - anyone feel like glancing through this code and seeing that it makes sense? I've only compile tested it so far, not even run up a test box... (also available as the "cacherewrite" branch on github) Bron. -------------- next part -------------- A non-text attachment was scrubbed... Name: cacherewrite.diff Type: text/x-diff Size: 50853 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20090312/6f5935c8/attachment-0001.bin From murch at andrew.cmu.edu Fri Mar 13 09:12:30 2009 From: murch at andrew.cmu.edu (Ken Murchison) Date: Fri, 13 Mar 2009 09:12:30 -0400 Subject: Cyrus 2.3.13 RC1 Message-ID: <49BA5BBE.5040803@andrew.cmu.edu> I just put together a release candidate for Cyrus 2.3.14. I'd appreciate any independent testing before I release this to the masses. http://www.contrib.andrew.cmu.edu/~murch/cyrus-imapd-2.3.14rc1.tar.gz Check doc/changes.html for a complete list of changes. If there are any outstanding issues that you believe still need to be addressed in 2.3.14, please let me know. -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University From murch at andrew.cmu.edu Fri Mar 13 09:14:11 2009 From: murch at andrew.cmu.edu (Ken Murchison) Date: Fri, 13 Mar 2009 09:14:11 -0400 Subject: Cyrus 2.3.14 RC1 (Was: Cyrus 2.3.13 RC1) In-Reply-To: <49BA5BBE.5040803@andrew.cmu.edu> References: <49BA5BBE.5040803@andrew.cmu.edu> Message-ID: <49BA5C23.1050701@andrew.cmu.edu> Sorry for the typo in the Subject. Ken Murchison wrote: > I just put together a release candidate for Cyrus 2.3.14. I'd > appreciate any independent testing before I release this to the masses. > > http://www.contrib.andrew.cmu.edu/~murch/cyrus-imapd-2.3.14rc1.tar.gz > > > Check doc/changes.html for a complete list of changes. > > If there are any outstanding issues that you believe still need to be > addressed in 2.3.14, please let me know. > -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University From bawood at umich.edu Fri Mar 13 18:04:26 2009 From: bawood at umich.edu (Brian Awood) Date: Fri, 13 Mar 2009 18:04:26 -0400 Subject: Cyrus 2.3.14 RC1 (Was: Cyrus 2.3.13 RC1) In-Reply-To: <49BA5C23.1050701@andrew.cmu.edu> References: <49BA5BBE.5040803@andrew.cmu.edu> <49BA5C23.1050701@andrew.cmu.edu> Message-ID: <200903131804.26280.bawood@umich.edu> Any chance the patches for https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3008 can be commited? It looks like it may have been forgotten, the patches don't apply cleanly anymore but I have diff of both against the current version in cvs if you'd like me to post it in BZ. Brian On Friday 13 March 2009 @ 09:14, Ken Murchison wrote: > Sorry for the typo in the Subject. > > Ken Murchison wrote: > > I just put together a release candidate for Cyrus 2.3.14. I'd > > appreciate any independent testing before I release this to the > > masses. > > > > http://www.contrib.andrew.cmu.edu/~murch/cyrus-imapd-2.3.14rc1.ta > >r.gz > > > > > > Check doc/changes.html for a complete list of changes. > > > > If there are any outstanding issues that you believe still need > > to be addressed in 2.3.14, please let me know. From brong at fastmail.fm Fri Mar 13 22:12:17 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sat, 14 Mar 2009 13:12:17 +1100 Subject: Cyrus 2.3.14 RC1 (Was: Cyrus 2.3.13 RC1) In-Reply-To: <200903131804.26280.bawood@umich.edu> References: <49BA5BBE.5040803@andrew.cmu.edu> <49BA5C23.1050701@andrew.cmu.edu> <200903131804.26280.bawood@umich.edu> Message-ID: <20090314021217.GA4469@brong.net> On Fri, Mar 13, 2009 at 06:04:26PM -0400, Brian Awood wrote: > Any chance the patches for > https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3008 can be commited? > It looks like it may have been forgotten, the patches don't apply > cleanly anymore but I have diff of both against the current version > in cvs if you'd like me to post it in BZ. Man, that's going to hurt against my cache rewrite code, which was written in response to a bug report of a crash in that very code path. Good point though. It's a nice idea to make them more readable! Can we queue this one for 2.3.15? I'll rewrite it on top of the cache stuff. I'd like to abstract out the envelope and structure handling code as well - because there's a lot more magic number copy 'n' paste code in index.c that needs to be cleaned up. Bron. From bawood at umich.edu Sat Mar 14 17:03:22 2009 From: bawood at umich.edu (Brian Awood) Date: Sat, 14 Mar 2009 17:03:22 -0400 Subject: Cyrus 2.3.14 RC1 (Was: Cyrus 2.3.13 RC1) In-Reply-To: <20090314021217.GA4469@brong.net> References: <49BA5BBE.5040803@andrew.cmu.edu> <200903131804.26280.bawood@umich.edu> <20090314021217.GA4469@brong.net> Message-ID: <200903141703.22570.bawood@umich.edu> On Friday 13 March 2009 @ 22:12, Bron Gondwana wrote: > Man, that's going to hurt against my cache rewrite code, which was > written in response to a bug report of a crash in that very code > path. > > Good point though. It's a nice idea to make them more readable! > > Can we queue this one for 2.3.15? I'll rewrite it on top of the > cache stuff. I'd like to abstract out the envelope and structure > handling code as well - because there's a lot more magic number > copy 'n' paste code in index.c that needs to be cleaned up. We're mainly interested in it because we are planing on rolling out a self-serve restore web interface for users, but we can just use the patches until it's included in a released version. Obviously improved cache handling is more important, no reason to commit the patches and then have to completely rewrite it. Brian From brong at fastmail.fm Sat Mar 14 17:45:18 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sun, 15 Mar 2009 08:45:18 +1100 Subject: Cyrus 2.3.14 RC1 (Was: Cyrus 2.3.13 RC1) In-Reply-To: <200903141703.22570.bawood@umich.edu> References: <49BA5BBE.5040803@andrew.cmu.edu> <200903131804.26280.bawood@umich.edu> <20090314021217.GA4469@brong.net> <200903141703.22570.bawood@umich.edu> Message-ID: <20090314214518.GB16668@brong.net> On Sat, Mar 14, 2009 at 05:03:22PM -0400, Brian Awood wrote: > > On Friday 13 March 2009 @ 22:12, Bron Gondwana wrote: > > Man, that's going to hurt against my cache rewrite code, which was > > written in response to a bug report of a crash in that very code > > path. > > > > Good point though. It's a nice idea to make them more readable! > > > > Can we queue this one for 2.3.15? I'll rewrite it on top of the > > cache stuff. I'd like to abstract out the envelope and structure > > handling code as well - because there's a lot more magic number > > copy 'n' paste code in index.c that needs to be cleaned up. > > We're mainly interested in it because we are planing on rolling out a > self-serve restore web interface for users, but we can just use the > patches until it's included in a released version. Obviously > improved cache handling is more important, no reason to commit the > patches and then have to completely rewrite it. > Brian Is there any reason for the "non-human" version? I'd be happy to just change the code to _only_ (or at least preferentially) offer the human friendly version. Bron. From bawood at umich.edu Sat Mar 14 18:07:16 2009 From: bawood at umich.edu (Brian Awood) Date: Sat, 14 Mar 2009 18:07:16 -0400 Subject: Cyrus 2.3.14 RC1 (Was: Cyrus 2.3.13 RC1) In-Reply-To: <20090314214518.GB16668@brong.net> References: <49BA5BBE.5040803@andrew.cmu.edu> <200903141703.22570.bawood@umich.edu> <20090314214518.GB16668@brong.net> Message-ID: <200903141807.16832.bawood@umich.edu> On Saturday 14 March 2009 @ 17:45, Bron Gondwana wrote: > On Sat, Mar 14, 2009 at 05:03:22PM -0400, Brian Awood wrote: > > On Friday 13 March 2009 @ 22:12, Bron Gondwana wrote: > > > Man, that's going to hurt against my cache rewrite code, which > > > was written in response to a bug report of a crash in that very > > > code path. > > > > > > Good point though. It's a nice idea to make them more > > > readable! > > > > > > Can we queue this one for 2.3.15? I'll rewrite it on top of > > > the cache stuff. I'd like to abstract out the envelope and > > > structure handling code as well - because there's a lot more > > > magic number copy 'n' paste code in index.c that needs to be > > > cleaned up. > > > > We're mainly interested in it because we are planing on rolling > > out a self-serve restore web interface for users, but we can just > > use the patches until it's included in a released version. > > Obviously improved cache handling is more important, no reason to > > commit the patches and then have to completely rewrite it. > > Brian > > Is there any reason for the "non-human" version? I'd be happy to > just change the code to _only_ (or at least preferentially) offer > the human friendly version. > I'm not aware of any reason, it seems like "unexpunge -l" is intended for human interaction. Brian From brong at fastmail.fm Thu Mar 19 05:50:59 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 19 Mar 2009 20:50:59 +1100 Subject: Cyrus 2.3.14 RC1 (Was: Cyrus 2.3.13 RC1) In-Reply-To: <49BA5C23.1050701@andrew.cmu.edu> References: <49BA5BBE.5040803@andrew.cmu.edu> <49BA5C23.1050701@andrew.cmu.edu> Message-ID: <20090319095059.GA13869@brong.net> On Fri, Mar 13, 2009 at 09:14:11AM -0400, Ken Murchison wrote: > Sorry for the typo in the Subject. > > > Ken Murchison wrote: > > I just put together a release candidate for Cyrus 2.3.14. I'd > > appreciate any independent testing before I release this to the masses. > > > > http://www.contrib.andrew.cmu.edu/~murch/cyrus-imapd-2.3.14rc1.tar.gz > > > > > > Check doc/changes.html for a complete list of changes. > > > > If there are any outstanding issues that you believe still need to be > > addressed in 2.3.14, please let me know. For what it's worth, FastMail has been running our patches on top of that tag (plus a bunch of other stuff, feel free to poke around the github page - I sync all my changes through there) for days now with no problems (not that I expected any - after all, we've been tracking CVS since I did the git integration) More excitingly, we've been running one store with the charset changes for a week now, and apart from a few minor teething bugs, it's looking pretty slick. UTF8 pattern matching, full decoding of body parts and mime headers in sieve, spaces in search (you need to reconstruct all the mailboxes, annoyingly, to make this happy). It's amazing how easy it is once you simplify the underlying APIs! So - I'll start pushing stuff into CVS again once 2.3.14 goes live. Consider this my vote that it's ready! Bron ( looking forward to a quicker release cycle to 2.3.15 as well... ) From dpc22 at cam.ac.uk Thu Mar 19 07:25:56 2009 From: dpc22 at cam.ac.uk (David Carter) Date: Thu, 19 Mar 2009 11:25:56 +0000 (GMT) Subject: Use of isspace() and friends in Cyrus Message-ID: I spent some time last night working out why the cyrus.cache entry for a specific message differed on a master and replica system. An eight bit character in a To: header (allowed by "reject8bit: no") had been transformed into a space character in the cache on the replica. The root cause turned out to be some code in parseaddr_phrase() which replaces arbitrary sequences of "space" characters with a single space: ./lib/parseaddr.c: static int parseaddr_phrase(inp, phrasep, specials) char **inp; char **phrasep; char *specials; { int c; char *src = *inp; /* XXX signed char on most platforms */ for (;;) { c = *src++; /* XXX signed char promoted to signed int */ . . . else if (isspace(c) || c == '(') { src--; SKIPWHITESPACE(src); *dst++ = ' '; } . . . } } isspace(-10) isn't defined. Worse isspace() is normally implemented as an array lookup and can segfault with a negative index if you are really unlucky. A quick search through the code reveals quite a lot of places where char variables are passed to isspace() without a cast to (unsigned char). The ctype macros often causes grief, and the normal workaround is to define replacement macros of the form: #define Uisspace(c) isspace((int)((unsigned char)(c))) I'm happy to knock up a sample patch if people agree. Curiously reconstruct on the replica fixed the cache entry generated by sync_server, despite the fact both are just using message_parse_file(). I infer that isspace(-10) is normally false but occasionally true, at least on Solaris x86. It appears to always be false on Linux x86. -- David Carter Email: David.Carter at ucs.cam.ac.uk University Computing Service, Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. From brong at fastmail.fm Thu Mar 19 17:35:31 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 20 Mar 2009 08:35:31 +1100 Subject: Use of isspace() and friends in Cyrus In-Reply-To: References: Message-ID: <20090319213530.GA7097@brong.net> On Thu, Mar 19, 2009 at 11:25:56AM +0000, David Carter wrote: > I spent some time last night working out why the cyrus.cache entry for a > specific message differed on a master and replica system. An eight bit > character in a To: header (allowed by "reject8bit: no") had been > transformed into a space character in the cache on the replica. > > The root cause turned out to be some code in parseaddr_phrase() which > replaces arbitrary sequences of "space" characters with a single space: > > ./lib/parseaddr.c: > > static int parseaddr_phrase(inp, phrasep, specials) > char **inp; > char **phrasep; > char *specials; > { > int c; > char *src = *inp; /* XXX signed char on most platforms */ > > for (;;) { > c = *src++; /* XXX signed char promoted to signed int */ Yeah, ouch! > isspace(-10) isn't defined. > > Worse isspace() is normally implemented as an array lookup and can > segfault with a negative index if you are really unlucky. > > A quick search through the code reveals quite a lot of places where char > variables are passed to isspace() without a cast to (unsigned char). The > ctype macros often causes grief, and the normal workaround is to define > replacement macros of the form: > > #define Uisspace(c) isspace((int)((unsigned char)(c))) > > I'm happy to knock up a sample patch if people agree. > > Curiously reconstruct on the replica fixed the cache entry generated by > sync_server, despite the fact both are just using message_parse_file(). > I infer that isspace(-10) is normally false but occasionally true, at > least on Solaris x86. It appears to always be false on Linux x86. So anyway, "isspace" in this case is actually a unicode function presumably. From the charset fiddling I've been doing it would be pretty easy to implement "charset_isspace" in a couple of lines of code :) But of course, that's not the issue. Yeah - there are two choices there, Uisspace, or actually fix all the code to pass unsigned explicitly to isspace. I don't see any problem with Uisspace. There's already cyrus_isdigit that I wrote a while back because isdigit() was taking an awful amount of processor for something that really was /[0-9]/ in meaning. So - where would you put the macro? Ok - next for _my_ little proposal... ;) It will make this look like nothing! Bron. From brong at fastmail.fm Thu Mar 19 17:46:49 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 20 Mar 2009 08:46:49 +1100 Subject: String handling Message-ID: <20090319214649.GB7097@brong.net> So we have a stack of interfaces that take a character pointer, a length - sometimes even a malloced length, and have reallocation possibilities. If you want to pass back the length you realloced to, you need to pass _three_ different variables to the function like so: int f(char *base, int *len, int *alloc, ...) Plus whatever else you actually need f() to do. ... and then you have to figure out who's responsible for freeing it! Now, I don't know that many C string libraries - I'm spoilt by spending a lot of my time in Perl where string management is handled very nicely thank-you- very-much. But - I wound up writing a very simple string library just for the charset stuff's internal 'buffer' structure: struct buffer_state { unsigned char *base; size_t offset; size_t alloced; }; And functions to manipulate that. Obviously a more general string library would have a flags item as well, with at least BUF_ISREADONLY, BUF_CANREALLOC, BUF_NEEDFREE and maybe even BUF_ISUTF8 if we wanted to get fancy. But before I go generalise one of these and start rewriting interfaces to use it, does anybody know of a good lightweight, compatible licence string library? I particularly want to make the final consumer of functions responsible for both allocating and freeing the wrapper datastructure, so you can pass them down with impunity and not have to worry about freeing them in every error case. (I note that timsieved already has a 'mystring' datatype, which is particularly lightweight, but doesn't have all the properties I want - in particular it doesn't support "slop", or any sort of append at all for that matter!) Bron. From brong at fastmail.fm Fri Mar 20 01:18:07 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 20 Mar 2009 16:18:07 +1100 Subject: iso-2022-kr charset Message-ID: <20090320051807.GB4643@brong.net> Hi, Does anybody actually use the ISO-2022-KR charset here? Does anybody know who's responsible for it? I ask because of: http://en.wikipedia.org/wiki/EUC-JP#EUC-KR and https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=2301 Which leads me to believe that using the new KSC-5601 charset is actually wrong. I've also found a bunch of messages in iso-2022-kr out there on the net, and they all seem to be 7bit transfer encoding, which doesn't match what's in the charset file in Cyrus... Anyway, I've rewritten it using KSX-1001 and KSX-1003 switching, along with US-ASCII separately before the initial ESC jump code. Now to implement EUC-KR somehow as an unholy table merger on top of iso-2022-kr! Does anyone see a problem with that? Is anything actually using KSC-5601-1992? If so, what's it called? Bron. From dpc22 at cam.ac.uk Fri Mar 20 05:59:37 2009 From: dpc22 at cam.ac.uk (David Carter) Date: Fri, 20 Mar 2009 09:59:37 +0000 (GMT) Subject: Use of isspace() and friends in Cyrus In-Reply-To: <20090319213530.GA7097@brong.net> References: <20090319213530.GA7097@brong.net> Message-ID: On Fri, 20 Mar 2009, Bron Gondwana wrote: > Yeah - there are two choices there, Uisspace, or actually fix all the > code to pass unsigned explicitly to isspace. I don't see any problem > with Uisspace. There's already cyrus_isdigit that I wrote a while > back because isdigit() was taking an awful amount of processor for > something that really was /[0-9]/ in meaning. > > So - where would you put the macro? lib/util.h? That already has: #define TOUPPER(c) (convert_to_uppercase[(unsigned char)(c)]) #define TOLOWER(c) (convert_to_lowercase[(unsigned char)(c)]) -- David Carter Email: David.Carter at ucs.cam.ac.uk University Computing Service, Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. From brong at fastmail.fm Fri Mar 20 06:36:45 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 20 Mar 2009 21:36:45 +1100 Subject: Use of isspace() and friends in Cyrus In-Reply-To: References: <20090319213530.GA7097@brong.net> Message-ID: <20090320103645.GA15518@brong.net> On Fri, Mar 20, 2009 at 09:59:37AM +0000, David Carter wrote: > On Fri, 20 Mar 2009, Bron Gondwana wrote: > >> Yeah - there are two choices there, Uisspace, or actually fix all the >> code to pass unsigned explicitly to isspace. I don't see any problem >> with Uisspace. There's already cyrus_isdigit that I wrote a while >> back because isdigit() was taking an awful amount of processor for >> something that really was /[0-9]/ in meaning. >> >> So - where would you put the macro? > > lib/util.h? That already has: > > #define TOUPPER(c) (convert_to_uppercase[(unsigned char)(c)]) > #define TOLOWER(c) (convert_to_lowercase[(unsigned char)(c)]) Yeah, that makes sense. I'd say go for it :) Bron. From woods-cyrus at weird.com Fri Mar 20 11:55:33 2009 From: woods-cyrus at weird.com (Greg A. Woods) Date: Fri, 20 Mar 2009 11:55:33 -0400 Subject: Use of isspace() and friends in Cyrus In-Reply-To: References: Message-ID: At Thu, 19 Mar 2009 11:25:56 +0000 (GMT), David Carter wrote: Subject: Use of isspace() and friends in Cyrus > > isspace(-10) isn't defined. > > Worse isspace() is normally implemented as an array lookup and can > segfault with a negative index if you are really unlucky. s/unlucky/lucky/ I.e. it's lucky to get some abort-causing error as soon as possible if you are a developer -- the sooner one learns the error of one's ways, the better! :-) -- Greg A. Woods +1 416 218-0098 VE3TCP RoboHack Planix, Inc. Secrets of the Weird From brong at fastmail.fm Mon Mar 23 09:31:59 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 24 Mar 2009 00:31:59 +1100 Subject: Lazy opening of cyrus.cache file Message-ID: <20090323133159.GA6059@brong.net> Only very lightly tested... but hey. Don't open the cyrus.cache file unless you need to, handy for those people running with small fast meta partitions and slow big partitions with the cyrus.cache on them :) It's probably not perfectly polished either yet - I really want to eliminate the bloody globals in index.c, but that's a bit further than I wanted to go with this patch. This is actually two separate patches on github - the first abstracts away the cache lookups, and is for crash-safety. The second takes advantage of the abstraction to delay opening the cache file until it first gets requested. Once needed, it stays open until the mailbox is closed. Bron. -------------- next part -------------- A non-text attachment was scrubbed... Name: cache.diff Type: text/x-diff Size: 63442 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20090324/2d4fd966/attachment-0001.bin From dpc22 at cam.ac.uk Mon Mar 23 10:31:53 2009 From: dpc22 at cam.ac.uk (David Carter) Date: Mon, 23 Mar 2009 14:31:53 +0000 (GMT) Subject: Lazy opening of cyrus.cache file In-Reply-To: <20090323133159.GA6059@brong.net> References: <20090323133159.GA6059@brong.net> Message-ID: On Tue, 24 Mar 2009, Bron Gondwana wrote: > Only very lightly tested... but hey. > > Don't open the cyrus.cache file unless you need to, > handy for those people running with small fast meta > partitions and slow big partitions with the > cyrus.cache on them :) Curiously I was having similar thoughts after laying my grubby paws on a pair of Intel X25e SSDs last week. What are you doing about the generation numbers in the cyrus.index and cyrus.cache file? There is a fairly elaborate dance in mailbox_open_index() to make sure that the correct pair of files are open in the face of mailbox_expunge() renaming two files in rapid succession in immediate or cleanup mode. -- David Carter Email: David.Carter at ucs.cam.ac.uk University Computing Service, Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. From brong at fastmail.fm Mon Mar 23 16:51:37 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 24 Mar 2009 07:51:37 +1100 Subject: Lazy opening of cyrus.cache file In-Reply-To: References: <20090323133159.GA6059@brong.net> Message-ID: <20090323205137.GA9815@brong.net> On Mon, Mar 23, 2009 at 02:31:53PM +0000, David Carter wrote: > On Tue, 24 Mar 2009, Bron Gondwana wrote: > >> Only very lightly tested... but hey. >> >> Don't open the cyrus.cache file unless you need to, >> handy for those people running with small fast meta >> partitions and slow big partitions with the >> cyrus.cache on them :) > > Curiously I was having similar thoughts after laying my grubby paws on a > pair of Intel X25e SSDs last week. So I heard in "the other place", which is what prompted me to visit this, since I already had the patch which does the heavy lifting! > What are you doing about the generation numbers in the cyrus.index and > cyrus.cache file? Failing to open it. My eventual solution is going to involve rewriting the bloody thing, but that means abstracting out the cache writing bits of reconstruct so that I can call a "fix cache file" command if corruption is found rather than just syslogging it and expecting someone else to come and run the reconstruct. > There is a fairly elaborate dance in mailbox_open_index() to make sure > that the correct pair of files are open in the face of mailbox_expunge() > renaming two files in rapid succession in immediate or cleanup mode. See, mailbox_expunge SHOULD be keeping an exclusive lock on cyrus.header, meaning everybody else waits until it's stable. The only place that needs anything like an elaborate dance should be cyrus.header, because there's nothing that can be locked before it. Bron. From murch at andrew.cmu.edu Tue Mar 24 06:32:10 2009 From: murch at andrew.cmu.edu (Ken Murchison) Date: Tue, 24 Mar 2009 06:32:10 -0400 Subject: Last call for Cyrus 2.3.14 Message-ID: <49C8B6AA.5040003@andrew.cmu.edu> If there are any outstanding issues that you believe still need to be addressed in 2.3.14, please let me know ASAP. -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University From thomas.jarosch at intra2net.com Tue Mar 24 06:47:28 2009 From: thomas.jarosch at intra2net.com (Thomas Jarosch) Date: Tue, 24 Mar 2009 11:47:28 +0100 Subject: Last call for Cyrus 2.3.14 In-Reply-To: <49C8B6AA.5040003@andrew.cmu.edu> References: <49C8B6AA.5040003@andrew.cmu.edu> Message-ID: <200903241147.29417.thomas.jarosch@intra2net.com> Hi Ken, On Tuesday, 24. March 2009 11:32:10 Ken Murchison wrote: > If there are any outstanding issues that you believe still need to be > addressed in 2.3.14, please let me know ASAP. Thanks for calling. I don't like to spoil the party, what about the issue which allows to create empty ACLs? The subject of the discussion was "[RFC PATCH] Prevent setacl for empty identifiers" I seem to have forgotten to open a bug report, what do you think is the best approach? Best regards, Thomas From wes at umich.edu Tue Mar 24 13:21:39 2009 From: wes at umich.edu (Wesley Craig) Date: Tue, 24 Mar 2009 13:21:39 -0400 Subject: Last call for Cyrus 2.3.14 In-Reply-To: <200903241147.29417.thomas.jarosch@intra2net.com> References: <49C8B6AA.5040003@andrew.cmu.edu> <200903241147.29417.thomas.jarosch@intra2net.com> Message-ID: <90723998-BD9F-4E22-BEDF-D3B14C1B5ED2@umich.edu> On 24 Mar 2009, at 06:47, Thomas Jarosch wrote: > Thanks for calling. I don't like to spoil the party, > what about the issue which allows to create empty ACLs? > > I seem to have forgotten to open a bug report, > what do you think is the best approach? 2.3.14 is more or less baked and people have been testing it. This bug has been around since sometime in 2004, and the solution needs some discussion. So, please open a bug report. :wes From Dilyan.Palauzov at aegee.org Wed Mar 25 04:28:07 2009 From: Dilyan.Palauzov at aegee.org (=?UTF-8?B?0JTQuNC70Y/QvSDQn9Cw0LvQsNGD0LfQvtCy?=) Date: Wed, 25 Mar 2009 09:28:07 +0100 Subject: Bug 3113 / Re: Last call for Cyrus 2.3.14 In-Reply-To: <49C8B6AA.5040003@andrew.cmu.edu> References: <49C8B6AA.5040003@andrew.cmu.edu> Message-ID: <49C9EB17.7050209@aegee.org> Hello, Ken Murchison wrote: > If there are any outstanding issues that you believe still need to be > addressed in 2.3.14, please let me know ASAP. > I'm for considering bug 3113 ( http://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3113 ) into the new release. ??? ??????, ????? From brong at fastmail.fm Wed Mar 25 05:39:26 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 25 Mar 2009 20:39:26 +1100 Subject: Bug 3113 / Re: Last call for Cyrus 2.3.14 In-Reply-To: <49C9EB17.7050209@aegee.org> References: <49C8B6AA.5040003@andrew.cmu.edu> <49C9EB17.7050209@aegee.org> Message-ID: <20090325093926.GA16176@brong.net> On Wed, Mar 25, 2009 at 09:28:07AM +0100, ????? ???????? wrote: > Hello, > > Ken Murchison wrote: >> If there are any outstanding issues that you believe still need to be >> addressed in 2.3.14, please let me know ASAP. >> > > I'm for considering bug 3113 ( > http://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3113 ) into the new > release. I've got a patch queued up, but I wanted to leave it until next release so we don't get caught up adding new things that haven't been tested very much. I'm running my patch on one of our production servers now, so I'm pretty happy with it - but there's no need to rush it to production! The downside is a performance hit during sieve script parsing, which is relatively rare anyway... Bron. From murch at andrew.cmu.edu Wed Mar 25 09:04:05 2009 From: murch at andrew.cmu.edu (Ken Murchison) Date: Wed, 25 Mar 2009 09:04:05 -0400 Subject: Cyrus IMAPd 2.3.14 Released Message-ID: <49CA2BC5.10902@andrew.cmu.edu> I am pleased to announce the release of Cyrus IMAPd 2.3.14. This release should be considered production quality. This is mostly a bugfix release. For full details, please see doc/changes.html and doc/install-upgrade.html which are included in the distribution. URLs for this release: ftp://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.3.14.tar.gz or http://ftp.andrew.cmu.edu/pub/cyrus/cyrus-imapd-2.3.14.tar.gz Questions and comments can be directed to info-cyrus at lists.andrew.cmu.edu (public list), or cyrus-bugs at andrew.cmu.edu. -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University From brong at fastmail.fm Thu Mar 26 00:21:22 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 26 Mar 2009 15:21:22 +1100 Subject: Convertion to Git Message-ID: <20090326042122.GA2430@brong.net> Wow - I'm getting pretty tired of dealing with the CVS/Git conversion, I just noticed that git-cvsimport doesn't properly pick up multiple tags on the same revision - and there was also a caching issue that caused it not to pick up the 2.3.14 tag at all. Obviously, that's a pretty half-baked "solution". There's a much more accurate tool cvs2git, but it doesn't support incremental operation - it's a one-shot repository conversion tool. So... We've talked about converting to something other than CVS already, and we have pretty strong support for git on the mailing list. Nothing else even got a positive mention. I've got a testing web interface to my current mirror-of-CVS repositories here: http://cyrus-devel.andrew.cmu.edu/cgit.cgi But that's just running out of a directory on one machine. It's not what I'd call "mature infrastructure". Off the top of my head here's what would be needed for a real conversion to using git: 1) Buy-in from everybody with commit rights. Both to using git, and to some sort of usable workflow. Git is significantly more flexible than CVS, and we all have to agree how to use it. 1a) this includes buy-in from CMU management and sysadmin. Dave, I'll need your input on that! 2) Figure out how to deal with 'sieve' and 'cmulocal' being separate repositories (either merge into one git repository or keep separate with 'git submodule') 3) Convert from CVS to git and validate the conversion (at least HEAD & the tip of any still-active branches) 4) Infrastructure at CMU. A build of git. Somewhere reliable, backed up. A copy of cgit.cgi or similar for web access. A monitored git server daemon. 5) Security setup to access the git repository somehow - either each having an individual repository with an automated merge daemon or another access mechanism. Also, gitorious or similar so we can use keyed SSH (every other access method is a real pain to use) 6) Commit hooks to send emails to the cyrus-cvs mailing list on changes. Maybe even something like: http://github.com/stephenh/git-central/ 7) Release process changes, possibly. ---------- Have I missed anything? If we can agree on the basic roadmap then we can start setting timelines and tasks to get there. My gut feeling is that we're looking at an absolute minimum 3 month timeframe for this if we push really hard, more realistically perhaps 6 months. Especially if I'm doing most of the pushing, and in a rather incompatible timezone from the infrastructure people! But - I do think this is the way of the future[tm], and choosing git is a good basis from which to build. Some parts of the surrounding infrastructure (ticket management, web interfaces, intregrated wikis, etc) aren't complete yet, but they will stabilise eventually - and we don't have that level of integration now anyway, so just converting CVS => git and not touching anything else is still a net win. Regards, Bron. From lists at egidy.de Thu Mar 26 05:38:33 2009 From: lists at egidy.de (Gerd v. Egidy) Date: Thu, 26 Mar 2009 10:38:33 +0100 Subject: Convertion to Git In-Reply-To: <20090326042122.GA2430@brong.net> References: <20090326042122.GA2430@brong.net> Message-ID: <200903261038.34339.lists@egidy.de> Hi Bron, we did a migration from svn to git at our company last fall. Currently we are happy with the switch and especially enjoy the low overhead of creating & merging back branches. > 2) Figure out how to deal with 'sieve' and 'cmulocal' being > separate repositories (either merge into one git repository > or keep separate with 'git submodule') We had the same issue with separate repositories and tried the git submodule approach. Our experience was that submodules really complicated the development workflow and their use is not documented as well as the rest of git. So we decided that they are not worth the trouble and either put the stuff in one repository or kept them completely separate git repositories. Kind regards, Gerd -- Address (better: trap) for people I really don't want to get mail from: jonas at cactusamerica.com From cyril.servant at atosorigin.com Thu Mar 26 06:38:59 2009 From: cyril.servant at atosorigin.com (Cyril Servant) Date: Thu, 26 Mar 2009 11:38:59 +0100 Subject: ipurge evolution Message-ID: <1238063939.6427.18.camel@localhost> Hello, with ipurge, we can delete mails _sent_ n days ago (option -d). I'd like to add an option (-u) in order to delete mails _received_ n days ago (I use OFFSET_LAST_UPDATED). An other thing I'd like to add is the "do nothing" mode (-n option) : in this mode, display in stdout is unchanged, but no mail is deleted (useful for tests). Here is the patch for ipurge.c 2.3.14 : 80a81 > int udays = -1; 102a104 > int donothing = 0; 119c121 < while ((option = getopt(argc, argv, "C:hxd:b:k:m:fsXi")) != EOF) { --- > while ((option = getopt(argc, argv, "C:hxd:b:k:m:fsXiu:n")) != EOF) { 129a132,137 > case 'u': { > if (optarg == 0) { > usage(argv[0]); > } > udays = atoi(optarg) * 86400 /* nominal # of seconds in a 'day' */; > } break; 153a162,164 > case 'n' : { > donothing = 1; > } break; 167,168c178,179 < if ((days == -1 ) && (size == -1)) { < printf("One of these must be specified -d, -b -k, -m\n"); --- > if ((days == -1 ) && (size == -1) && (udays == -1 )) { > printf("One of these must be specified -u, -d, -b, -k, -m\n"); 173a185,187 > if (donothing) > printf("warning : do nothing mode !\n"); > 217,218c231,232 < printf("usage: %s [-f] [-s] [-C ] [-x] [-X] [-i] {-d days | -b bytes|-k Kbytes|-m Mbytes}\n\t[mboxpattern1 ... [mboxpatternN]]\n", name); < printf("\tthere are no defaults and at least one of -d, -b, -k, -m\n \tmust be specified\n"); --- > printf("usage: %s [-n] [-f] [-s] [-C ] [-x] [-X] [-i] {-u days|-d days | -b bytes|-k Kbytes|-m Mbytes}\n\t[mboxpattern1 ... [mboxpatternN]]\n", name); > printf("\tthere are no defaults and at least one of -u, -d, -b, -k, -m\n\tmust be specified\n"); 221a236 > printf("\t -n do not really delete mails\n"); 296a312 > bit32 uptime; 300a317 > uptime = ntohl(*((bit32 *)(buf + OFFSET_LAST_UPDATED))); 316a334,352 > if (donothing) > return 0; > return 1; > } else { > if (!invertmatch) return 0; > deleteit(msgsize, stats); > if (donothing) > return 0; > return 1; > } > } > if (udays >= 0) { > my_time = time(0); > /* printf("comparing %ld :: %ld\n", my_time, uptime); */ > if (((my_time - (time_t) uptime)/86400) == (udays/86400)) { > if (invertmatch) return 0; > deleteit(msgsize, stats); > if (donothing) > return 0; 320a357,358 > if (donothing) > return 0; 328a367,368 > if (donothing) > return 0; 332a373,374 > if (donothing) > return 0; 342a385,386 > if (donothing) > return 0; 346a391,408 > if (donothing) > return 0; > return 1; > } > } > if (udays >= 0) { > my_time = time(0); > /* printf("comparing %ld :: %ld\n", my_time, uptime); */ > if (!invertmatch && ((my_time - (time_t) uptime) > udays)) { > deleteit(msgsize, stats); > if (donothing) > return 0; > return 1; > } > if (invertmatch && ((my_time - (time_t) uptime) < udays)) { > deleteit(msgsize, stats); > if (donothing) > return 0; 353a416,417 > if (donothing) > return 0; 357a422,423 > if (donothing) > return 0; -- Cyril Servant From wes at umich.edu Thu Mar 26 22:09:07 2009 From: wes at umich.edu (Wesley Craig) Date: Thu, 26 Mar 2009 22:09:07 -0400 Subject: ipurge evolution In-Reply-To: <1238063939.6427.18.camel@localhost> References: <1238063939.6427.18.camel@localhost> Message-ID: Could you enter this in Bugzilla as a feature request, please? Also, a unified diff would be preferable. :wes On 26 Mar 2009, at 06:38, Cyril Servant wrote: > Here is the patch for ipurge.c 2.3.14 : From brong at fastmail.fm Thu Mar 26 23:09:42 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 27 Mar 2009 14:09:42 +1100 Subject: ipurge evolution In-Reply-To: References: <1238063939.6427.18.camel@localhost> Message-ID: <20090327030942.GA26113@brong.net> On Thu, Mar 26, 2009 at 10:09:07PM -0400, Wesley Craig wrote: > Could you enter this in Bugzilla as a feature request, please? Also, a > unified diff would be preferable. The feature list certainly looks reasonable - particularly the "don't actually perform the changes, just detail them" idea. Bron. From cyril.servant at atosorigin.com Fri Mar 27 05:49:50 2009 From: cyril.servant at atosorigin.com (Cyril Servant) Date: Fri, 27 Mar 2009 10:49:50 +0100 Subject: ipurge evolution In-Reply-To: References: <1238063939.6427.18.camel@localhost> Message-ID: <1238147390.6373.0.camel@localhost> Hello, Le vendredi 27 mars 2009 ? 03:09 +0100, Wesley Craig a ?crit : > Could you enter this in Bugzilla as a feature request, please? Also, > a unified diff would be preferable. Yes, it's done. > > :wes > > On 26 Mar 2009, at 06:38, Cyril Servant wrote: > > Here is the patch for ipurge.c 2.3.14 : > -- Cyril Servant From dpc22 at cam.ac.uk Fri Mar 27 06:07:29 2009 From: dpc22 at cam.ac.uk (David Carter) Date: Fri, 27 Mar 2009 10:07:29 +0000 (GMT) Subject: Four patches against 2.3.14 Message-ID: http://www-uxsup.csx.cam.ac.uk/~dpc22/cyrus/patches/2.3.14 autoconf.patch ============== Stop aclocal moaning about invalid header names. Was surprised that Linux wasn't moaning about the same thing. SunFreeware provides autoconf 2.63, which is more recent than the 2.59 shipped with SLES10. ctype.patch =========== Replace ctype macros, which all expect int argument in the range 0..255 or EOF (typically -1). I have left a few instances of isspace() and friends where the input comes from fgetc() or prot_getc(), and consequently is in the expected range. Mostly automatic using find and sed, but I did go back and remove casts to (unsigned char) [correct but redundant], and (int) [typically incorrect if the source is signed char, probably put there to stop the compiler moaning]. robust_squatter.patch ===================== Don't give up if index_me() throws an error. Not wildly happy about mboxlist_findall which can run for several days in any case. sync_combine_commit.patch ========================= Update quota correctly in sync_combine_commit(). Only normally triggered when replicating unexpunged messages in normal use. -- David Carter Email: David.Carter at ucs.cam.ac.uk University Computing Service, Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. From brong at fastmail.fm Fri Mar 27 07:34:31 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 27 Mar 2009 22:34:31 +1100 Subject: Four patches against 2.3.14 In-Reply-To: References: Message-ID: <20090327113431.GA25932@brong.net> On Fri, Mar 27, 2009 at 10:07:29AM +0000, David Carter wrote: > ctype.patch > =========== > Mostly automatic using find and sed, but I did go back and remove > casts to (unsigned char) [correct but redundant], and (int) [typically > incorrect if the source is signed char, probably put there to stop > the compiler moaning]. Why did this touch sieve.h? It's generated by bison. Did you not work on a clean tree? I guess it doesn't matter too much, I'll be working on a clean tree anyway :) From dpc22 at cam.ac.uk Fri Mar 27 07:38:15 2009 From: dpc22 at cam.ac.uk (David Carter) Date: Fri, 27 Mar 2009 11:38:15 +0000 (GMT) Subject: Four patches against 2.3.14 In-Reply-To: <20090327113431.GA25932@brong.net> References: <20090327113431.GA25932@brong.net> Message-ID: On Fri, 27 Mar 2009, Bron Gondwana wrote: > Why did this touch sieve.h? It's generated by bison. Did you not > work on a clean tree? 2.3.14 ships with a sieve/sieve.h I did remove a number of the .c files which are generated from Bison or Flex, but missed this one. -- David Carter Email: David.Carter at ucs.cam.ac.uk University Computing Service, Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. From brong at fastmail.fm Fri Mar 27 07:46:58 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 27 Mar 2009 22:46:58 +1100 Subject: Four patches against 2.3.14 In-Reply-To: References: Message-ID: <20090327114658.GA26886@brong.net> On Fri, Mar 27, 2009 at 10:07:29AM +0000, David Carter wrote: > robust_squatter.patch > ===================== > > Don't give up if index_me() throws an error. Not wildly happy about > mboxlist_findall which can run for several days in any case. l->tail = l->tail->next = n; Is that _guaranteed_ to evaluate correctly? It looks like someone being a smartarse to avoid writing: l->tail->next = n; l->tail = l->tail->next; Or even: tail = l->tail; tail->next = n; l->tail = n; Which is ultra-extra clear about what you're doing. It took me a couple of seconds parsing to understand what you were doing with that line, and it scares me. If the compiler re-optimised the order in which it did the assignment and lookup, things would get really sad. (it's annoying re-inventing such a common wheel in the first place really. The bikeshed painter in me wonders why it's not a realloced array, but that's more accounting and not worth it for a one-shot piece of code) Bron. From brong at fastmail.fm Fri Mar 27 08:08:53 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 27 Mar 2009 23:08:53 +1100 Subject: Four patches against 2.3.14 In-Reply-To: References: Message-ID: <20090327120853.GB25932@brong.net> On Fri, Mar 27, 2009 at 10:07:29AM +0000, David Carter wrote: > http://www-uxsup.csx.cam.ac.uk/~dpc22/cyrus/patches/2.3.14 Ok - I've loaded all 4 of these patches on top of my 'master' tree on github and called the branch 'davidcarter', then rebased the fastmaildev branch on top of it. That means that it will be included in my dev builds from now on :) I did change the linked list append into two lines in my copy. Assuming this all runs nicely in my testbed, I'll roll it out to the usual suspects (man, people are going to LOVE being on our store...) and if that's good then I'm happy to commit it to CVS. NOTE: there are three separate trees on github, and they all have to stay in sync :( Bloody CVS separate repostiories. I've used exactly the same commit messages (and falsified the author as David on the git commits!) for each set across the respositories. Bron. From brong at fastmail.fm Fri Mar 27 08:10:42 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 27 Mar 2009 23:10:42 +1100 Subject: Four patches against 2.3.14 In-Reply-To: References: <20090327113431.GA25932@brong.net> Message-ID: <20090327121042.GA30410@brong.net> On Fri, Mar 27, 2009 at 11:38:15AM +0000, David Carter wrote: > On Fri, 27 Mar 2009, Bron Gondwana wrote: > >> Why did this touch sieve.h? It's generated by bison. Did you not >> work on a clean tree? > > 2.3.14 ships with a sieve/sieve.h I did remove a number of the .c files > which are generated from Bison or Flex, but missed this one. Fair enough :) It's not in my git tree, so I won't be updating it anyway on my copy... Bron. From dpc22 at cam.ac.uk Fri Mar 27 08:25:38 2009 From: dpc22 at cam.ac.uk (David Carter) Date: Fri, 27 Mar 2009 12:25:38 +0000 (GMT) Subject: Four patches against 2.3.14 In-Reply-To: <20090327114658.GA26886@brong.net> References: <20090327114658.GA26886@brong.net> Message-ID: On Fri, 27 Mar 2009, Bron Gondwana wrote: > l->tail = l->tail->next = n; > > Is that _guaranteed_ to evaluate correctly? It looks like someone > being a smartarse to avoid writing: > > l->tail->next = n; > l->tail = l->tail->next; It's a fairly common idiom for managing linked lists in C. Or at least I thought that it was a fairly common idiom: the only other obvious incidences I can find in Cyrus are sync_support.c, make_md5.c and make_sha1.c, which are probably all down to me. Smartarse. '=' has right to left associativity (K&R second edition, page 53), so: a = b = c evaluates as: a = (b = c) But if you are happier with two or three lines of code, then go for it. -- David Carter Email: David.Carter at ucs.cam.ac.uk University Computing Service, Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. From brong at fastmail.fm Fri Mar 27 09:30:34 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sat, 28 Mar 2009 00:30:34 +1100 Subject: Four patches against 2.3.14 In-Reply-To: References: <20090327114658.GA26886@brong.net> Message-ID: <20090327133034.GA3118@brong.net> On Fri, Mar 27, 2009 at 12:25:38PM +0000, David Carter wrote: > '=' has right to left associativity (K&R second edition, page 53), so: > > a = b = c > > evaluates as: > > a = (b = c) > > But if you are happier with two or three lines of code, then go for it. Yeah, I think it's clearer that it's two separate operations. Besides, I've followed some of the stuff on LKML about various versions of GCC screwing up optimisations, and debuging that stuff sounds like it would give me ulcers! I tell you, debugging the issue with blocks of zeros appearing in our skiplist files on 64 bit machines sure did... thankfully Linus found the bug quickly, though he did say that reading via mmap and writing with fwrite rather than just using mmap for writes was probably up there in insane territory as well. Surprised it worked so well everywhere, he was. Speaking of "smartarse" - you forgot to include util.h in sieve/test.c. It's not part of the standard build, so easy not to notice - but our install script builds it, so I noticed when I went to build it on my test machine. I've rolled that into my tree and pushed back to github again. Git is nice :) Bron. From dpc22 at cam.ac.uk Fri Mar 27 12:12:41 2009 From: dpc22 at cam.ac.uk (David Carter) Date: Fri, 27 Mar 2009 16:12:41 +0000 (GMT) Subject: Four patches against 2.3.14 In-Reply-To: <20090327133034.GA3118@brong.net> References: <20090327114658.GA26886@brong.net> <20090327133034.GA3118@brong.net> Message-ID: On Sat, 28 Mar 2009, Bron Gondwana wrote: > Speaking of "smartarse" - you forgot to include util.h in sieve/test.c. > It's not part of the standard build, so easy not to notice - but our > install script builds it, so I noticed when I went to build it on my > test machine. Actually that's not the only case. I meant to run a cross check this morning to make sure that all of the files that had been patched #include'd "util.h", but I got diverted. Sorry about that. Here's a second attempt: http://www-uxsup.csx.cam.ac.uk/~dpc22/cyrus/patches/2.3.14/ctype2.patch This adds the #include to a number of extra files. It also doesn't patch ./makedepend, ./syslog or ./doc/text/htmlstrip.c, which are standalones. -- David Carter Email: David.Carter at ucs.cam.ac.uk University Computing Service, Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. From thomas.jarosch at intra2net.com Fri Mar 27 12:27:38 2009 From: thomas.jarosch at intra2net.com (Thomas Jarosch) Date: Fri, 27 Mar 2009 17:27:38 +0100 Subject: Last call for Cyrus 2.3.14 In-Reply-To: <90723998-BD9F-4E22-BEDF-D3B14C1B5ED2@umich.edu> References: <49C8B6AA.5040003@andrew.cmu.edu> <200903241147.29417.thomas.jarosch@intra2net.com> <90723998-BD9F-4E22-BEDF-D3B14C1B5ED2@umich.edu> Message-ID: <200903271727.39149.thomas.jarosch@intra2net.com> On Tuesday, 24. March 2009 18:21:39 Wesley Craig wrote: > On 24 Mar 2009, at 06:47, Thomas Jarosch wrote: > > Thanks for calling. I don't like to spoil the party, > > what about the issue which allows to create empty ACLs? > > > > I seem to have forgotten to open a bug report, > > what do you think is the best approach? > > 2.3.14 is more or less baked and people have been testing it. This > bug has been around since sometime in 2004, and the solution needs > some discussion. So, please open a bug report. Ok, filed as #3147: https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3147 btw: There was no "2.3.x" version number in bugzilla's "create new bug report" dialog. After the bug was created, I was able to change the version. Have a nice weekend, Thomas From brong at fastmail.fm Fri Mar 27 18:19:49 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sat, 28 Mar 2009 09:19:49 +1100 Subject: Four patches against 2.3.14 In-Reply-To: References: <20090327114658.GA26886@brong.net> <20090327133034.GA3118@brong.net> Message-ID: <20090327221949.GA4346@brong.net> On Fri, Mar 27, 2009 at 04:12:41PM +0000, David Carter wrote: > On Sat, 28 Mar 2009, Bron Gondwana wrote: > >> Speaking of "smartarse" - you forgot to include util.h in sieve/test.c. >> It's not part of the standard build, so easy not to notice - but our >> install script builds it, so I noticed when I went to build it on my >> test machine. > > Actually that's not the only case. I meant to run a cross check this > morning to make sure that all of the files that had been patched > #include'd "util.h", but I got diverted. Sorry about that. That's OK. It builds correctly for me now. > Here's a second attempt: > > http://www-uxsup.csx.cam.ac.uk/~dpc22/cyrus/patches/2.3.14/ctype2.patch > > This adds the #include to a number of extra files. It also doesn't patch > ./makedepend, ./syslog or ./doc/text/htmlstrip.c, which are standalones. Cool, I'll merge it in after I've woken up a bit more :) Bron. From brong at fastmail.fm Fri Mar 27 22:09:30 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sat, 28 Mar 2009 13:09:30 +1100 Subject: Four patches against 2.3.14 In-Reply-To: References: <20090327114658.GA26886@brong.net> <20090327133034.GA3118@brong.net> Message-ID: <20090328020930.GA21859@brong.net> On Fri, Mar 27, 2009 at 04:12:41PM +0000, David Carter wrote: > On Sat, 28 Mar 2009, Bron Gondwana wrote: > >> Speaking of "smartarse" - you forgot to include util.h in sieve/test.c. >> It's not part of the standard build, so easy not to notice - but our >> install script builds it, so I noticed when I went to build it on my >> test machine. > > Actually that's not the only case. I meant to run a cross check this > morning to make sure that all of the files that had been patched > #include'd "util.h", but I got diverted. Sorry about that. > > Here's a second attempt: > > http://www-uxsup.csx.cam.ac.uk/~dpc22/cyrus/patches/2.3.14/ctype2.patch > > This adds the #include to a number of extra files. It also doesn't patch > ./makedepend, ./syslog or ./doc/text/htmlstrip.c, which are standalones. Yeah, that's good :) Github is rebuilt on top of that... (basically I created a new branch forking off just before the old ctype patch called ctype2, then applied the patch, then used 'rebase -i' for the davidcarter branch on top of the ctype2 branch, and deleted the line for the old ctype branch. All good :) Bron. From brong at fastmail.fm Sat Mar 28 07:36:24 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sat, 28 Mar 2009 22:36:24 +1100 Subject: Locking Policy Message-ID: <20090328113624.GA21819@brong.net> I have a question about Cyrus locking policy - I know there are comments about violating it: /* Note that we are locking the mailbox here. Technically, this function * can be called with a lock on the mailbox list. This would otherwise * violate the locking semantics, but it is okay since the mailbox list * changes have not been committed, and the mailbox we create here *can't* * be opened by anyone else */ ... /* if it is not a remote mailbox, we need to unlock the mailbox list, * lock the mailbox, and re-lock the mailboxes list */ /* we must do this to obey our locking rules */ ... So from these I deduce that you can't lock a mailbox with the mailboxes.db locked. Fair enough, ordering constraints to avoid deadlocks. What I'm wondering is what other locking constraints are actually codified? I couldn't find anything about them grepping the doc tree for obvious phrases (like... locking). It seems to me that a whole lot more locking should be happening on the header file, which along with strict ordering of locks and the removal of ACLs from the header file (we don't use them anyway) would mean we never get stupid issues like .index and .cache files being out of sync and needing the dodgy "sleep a second and see if the version numbers match" logic. Basically, if you need to rewrite or change ANYTHING in the mailbox, you'd just lock the header and go ahead and change the other files. Obviously there's a tiny crash window (one file renamed, the other not) so you still need to check for cache file version mismatches, but the sane response is to rebuild the cache file from scratch. Bron ( but then I'm crazy enough that I want to write "intent log" files somewhere before doing anything that gets sync logged, so after a crash/shutdown we can cat all the individual pid intent logs into sync_client and make sure all the folders of interest get re-synced ) From dpc22 at cam.ac.uk Sat Mar 28 09:40:03 2009 From: dpc22 at cam.ac.uk (David Carter) Date: Sat, 28 Mar 2009 13:40:03 +0000 (GMT) Subject: Locking Policy In-Reply-To: <20090328113624.GA21819@brong.net> References: <20090328113624.GA21819@brong.net> Message-ID: On Sat, 28 Mar 2009, Bron Gondwana wrote: > What I'm wondering is what other locking constraints are actually > codified? I couldn't find anything about them grepping the doc > tree for obvious phrases (like... locking). The only documentation that I've found is doc/internal/locking, which is rather, um, minimal. > It seems to me that a whole lot more locking should be happening on the > header file, which along with strict ordering of locks and the removal > of ACLs from the header file (we don't use them anyway) would mean we > never get stupid issues like .index and .cache files being out of sync > and needing the dodgy "sleep a second and see if the version numbers > match" logic. At the moment we need exclusive locks on two files as cyrus.header and cyrus.index are both updated using a .NEW temporary file and a rename(), which breaks the lock. Life would be much simpler if Unix provided a primitive to both open and lock a file as a single atomic action. I agree that a dedicated .lock file (which might be the header file if we can move out the variable state) would be more elegant. We do however need somewhere to store the mailbox UIDvalidity, user flag names and ACL, all of which can change. UIDvalidity and flags names are critical, and the user flag names can get quite large given 128 user flags. ACLs are less critical, but I do like having a redundant copy of the ACL somewhere just in case I ever needed to reconstruct mailboxes.db from scratch sometime. The exclusive POP lock on cyrus.cache is an additional complication, but that is off to one side and seems to work reasonably well. (I'm not entirely convinced that a server which implements delayed expunge actually needs the exclusive lock, but it is mandated by RFC 1725, section 4). -- David Carter Email: David.Carter at ucs.cam.ac.uk University Computing Service, Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. From brong at fastmail.fm Sat Mar 28 18:12:34 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sun, 29 Mar 2009 09:12:34 +1100 Subject: Locking Policy In-Reply-To: References: <20090328113624.GA21819@brong.net> Message-ID: <1238278354.32568.1307806757@webmail.messagingengine.com> On Sat, 28 Mar 2009 13:40 +0000, "David Carter" wrote: > On Sat, 28 Mar 2009, Bron Gondwana wrote: > > > What I'm wondering is what other locking constraints are actually > > codified? I couldn't find anything about them grepping the doc > > tree for obvious phrases (like... locking). > > The only documentation that I've found is doc/internal/locking, which is > rather, um, minimal. Maybe I'd better pad that out when I do some work on this. I'm particularly interested in stopping the "header doesn't exist" errors you get when renaming a folder and having anther connection open to it - things like that show that locking and reopening aren't coving 100% of cases properly. > > It seems to me that a whole lot more locking should be happening on the > > header file, which along with strict ordering of locks and the removal > > of ACLs from the header file (we don't use them anyway) would mean we > > never get stupid issues like .index and .cache files being out of sync > > and needing the dodgy "sleep a second and see if the version numbers > > match" logic. > > At the moment we need exclusive locks on two files as cyrus.header and > cyrus.index are both updated using a .NEW temporary file and a rename(), > which breaks the lock. Life would be much simpler if Unix provided a > primitive to both open and lock a file as a single atomic action. > > I agree that a dedicated .lock file (which might be the header file if we > can move out the variable state) would be more elegant. We do however > need > somewhere to store the mailbox UIDvalidity, user flag names and ACL, all > of which can change. UIDvalidity and flags names are critical, and the > user flag names can get quite large given 128 user flags. ACLs are less > critical, but I do like having a redundant copy of the ACL somewhere just > in case I ever needed to reconstruct mailboxes.db from scratch sometime. Fair enough. I guess the .header file is a sensible place. So it sounds like maybe a "cyrus.lock" file is the way to go. And even a cyrus.poplock too. It would be tempting to put the whole lot on a tmpfs, though the data/meta partition split doesn't allow for a third base location easily! > The exclusive POP lock on cyrus.cache is an additional complication, but > that is off to one side and seems to work reasonably well. (I'm not > entirely convinced that a server which implements delayed expunge > actually > needs the exclusive lock, but it is mandated by RFC 1725, section 4). It would need to lock something that prohibited the cyr_expire while the pop connection was active, and besides - we move the record into cyrus.expunge, that would be pretty messy to keep track of. But why cyrus.cache? It feels to me like it was used because it was there, rather than actually having any semantic meaning. Messy. Bron. -- Bron Gondwana brong at fastmail.fm From brong at fastmail.fm Sat Mar 28 18:12:37 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sun, 29 Mar 2009 09:12:37 +1100 Subject: Locking Policy Message-ID: <1238278357.32571.1307806757@webmail.messagingengine.com> On Sat, 28 Mar 2009 13:40 +0000, "David Carter" wrote: > On Sat, 28 Mar 2009, Bron Gondwana wrote: > > > What I'm wondering is what other locking constraints are actually > > codified? I couldn't find anything about them grepping the doc > > tree for obvious phrases (like... locking). > > The only documentation that I've found is doc/internal/locking, which is > rather, um, minimal. Maybe I'd better pad that out when I do some work on this. I'm particularly interested in stopping the "header doesn't exist" errors you get when renaming a folder and having anther connection open to it - things like that show that locking and reopening aren't coving 100% of cases properly. > > It seems to me that a whole lot more locking should be happening on the > > header file, which along with strict ordering of locks and the removal > > of ACLs from the header file (we don't use them anyway) would mean we > > never get stupid issues like .index and .cache files being out of sync > > and needing the dodgy "sleep a second and see if the version numbers > > match" logic. > > At the moment we need exclusive locks on two files as cyrus.header and > cyrus.index are both updated using a .NEW temporary file and a rename(), > which breaks the lock. Life would be much simpler if Unix provided a > primitive to both open and lock a file as a single atomic action. > > I agree that a dedicated .lock file (which might be the header file if we > can move out the variable state) would be more elegant. We do however > need > somewhere to store the mailbox UIDvalidity, user flag names and ACL, all > of which can change. UIDvalidity and flags names are critical, and the > user flag names can get quite large given 128 user flags. ACLs are less > critical, but I do like having a redundant copy of the ACL somewhere just > in case I ever needed to reconstruct mailboxes.db from scratch sometime. Fair enough. I guess the .header file is a sensible place. So it sounds like maybe a "cyrus.lock" file is the way to go. And even a cyrus.poplock too. It would be tempting to put the whole lot on a tmpfs, though the data/meta partition split doesn't allow for a third base location easily! > The exclusive POP lock on cyrus.cache is an additional complication, but > that is off to one side and seems to work reasonably well. (I'm not > entirely convinced that a server which implements delayed expunge > actually > needs the exclusive lock, but it is mandated by RFC 1725, section 4). It would need to lock something that prohibited the cyr_expire while the pop connection was active, and besides - we move the record into cyrus.expunge, that would be pretty messy to keep track of. But why cyrus.cache? It feels to me like it was used because it was there, rather than actually having any semantic meaning. Messy. Bron. -- Bron Gondwana brong at fastmail.fm From brong at fastmail.fm Sun Mar 29 07:57:25 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sun, 29 Mar 2009 22:57:25 +1100 Subject: AMD64 and PIC Message-ID: <20090329115725.GA10073@brong.net> http://www.gentoo.org/proj/en/base/amd64/howtos/index.xml?part=1&chap=3 (just because they seem to have documented it well) I ran in to this with libcyrus.a when I put Ubuntu Jaunty Beta amd64 on my laptop today. Passing CFLAGS with -fPIC to configure does the trick, but it would be nice if our configure could handle it. Bron ( not at all an expert in configure writing! ) From dave64 at andrew.cmu.edu Mon Mar 30 08:52:46 2009 From: dave64 at andrew.cmu.edu (Dave McMurtrie) Date: Mon, 30 Mar 2009 08:52:46 -0400 Subject: Use of "fast" mutexes in mupdate.c Message-ID: <49D0C09E.3080308@andrew.cmu.edu> Good morning, I was reading through the mupdate.c code this morning looking for a solution to a problem we're experiencing. That led to me reading the pthread_mutex_lock() manpage. The mupdate.c code is using "fast" mutexes. pthread_mutex_lock() on Linux has this interesting feature: ----- begin quote ----- On ''error checking'' mutexes, pthread_mutex_unlock actually checks at run-time that the mutex is locked on entrance, and that it was locked by the same thread that is now calling pthread_mutex_unlock. If these conditions are not met, an error code is returned and the mutex remains unchanged. ''Fast'' and ''recursive'' mutexes perform no such checks, thus allowing a locked mutex to be unlocked by a thread other than its owner. This is non-portable behavior and must not be relied upon. ----- end quote ----- Doesn't this mean that even if the code is properly written to lock idle_worker_mutex before modifying the linked list of idle connections, some other thread is free to unlock the mutex out from under us? If that is the case, does anyone see a reason to not change this to use "error-checking" mutexes and then add error checking to the pthread_lock_mutex() and pthread_unlock_mutex() calls? If performance is the only potential impact, I'm willing to test it here and see if we really need to be concerned. As the code is currently written, it doesn't appear to be thread-safe, at least on Linux. Thanks, Dave -- Dave McMurtrie, SPE Email Systems Team Leader Carnegie Mellon University, Computing Services From dave64 at andrew.cmu.edu Mon Mar 30 12:41:45 2009 From: dave64 at andrew.cmu.edu (Dave McMurtrie) Date: Mon, 30 Mar 2009 12:41:45 -0400 Subject: More mupdate questions. Message-ID: <49D0F649.2040100@andrew.cmu.edu> Looking further into mupdate this morning (see my earlier message today), I've discovered something else that looks like an issue. It looks like a good deal of the calling code that knows about mupdate relies on basic logic like: - do a local database lookup - if the mailbox we're looking for doesn't exist, call kick_mupdate() to ensure that our local database is fresh, then do the local database lookup again. It appears that the thought behind kick_mupdate() is that it was written to exploit the fact that mupdate will not answer slave requests (on the local unix socket) while a sync operation is in process. I notice one fundamental problem with this, and one area for improvement. The fundamental problem is that kick_mupdate() returns void. If a mupdate slave is not currently listening on its local AF_UNIX socket (like when it disconnects because it loses contact with the mupdate master) kick_mupdate() will attempt to connect, fail, log an error and return. Most of the calling code I looked at assumes that if kick_mupdate() returns, the local database is current. In this case, that may or may not be true. At the very least, I think kick_mupdate() needs to be updated to be able to return a status so any calling code can make a more informed decision about what to do. The area for improvement is that kick_mupdate() never times out its read() from mupdate's local unix socket. Wouldn't it be a good idea to wrap this read() in an alarm() call and return failure? Before I start working on a patch, can someone who is familiar with this code let me know if I'm way off base or not? Thanks, Dave -- Dave McMurtrie, SPE Email Systems Team Leader Carnegie Mellon University, Computing Services From wes at umich.edu Mon Mar 30 13:43:34 2009 From: wes at umich.edu (Wesley Craig) Date: Mon, 30 Mar 2009 13:43:34 -0400 Subject: Use of "fast" mutexes in mupdate.c In-Reply-To: <49D0C09E.3080308@andrew.cmu.edu> References: <49D0C09E.3080308@andrew.cmu.edu> Message-ID: On 30 Mar 2009, at 08:52, Dave McMurtrie wrote: > I was reading through the mupdate.c code this morning looking for a > solution to a problem we're experiencing. What problem is that? > That led to me reading the > pthread_mutex_lock() manpage. The mupdate.c code is using "fast" > mutexes. pthread_mutex_lock() on Linux has this interesting feature: > > ----- begin quote ----- > On ''error checking'' mutexes, pthread_mutex_unlock actually checks at > run-time that the mutex is locked on entrance, and that it was locked > by the same thread that is now calling pthread_mutex_unlock. If these > conditions are not met, an error code is returned and the mutex > remains unchanged. ''Fast'' and ''recursive'' mutexes perform no such > checks, thus allowing a locked mutex to be unlocked by a thread other > than its owner. This is non-portable behavior and must not be > relied upon. > ----- end quote ----- > > Doesn't this mean that even if the code is properly written to lock > idle_worker_mutex before modifying the linked list of idle > connections, > some other thread is free to unlock the mutex out from under us? In properly written code, all of the threads are working together. Thus, while any thread is free to (incorrectly) unlock a critical mutex that they have no business messing with, in properly written code, they don't. That would be a bug. PTHREAD_MUTEX_NORMAL, _ERRORCHECK, and _RECURSIVE are specified in POSIX, BTW. > If that is the case, does anyone see a reason to not change this to > use > "error-checking" mutexes and then add error checking to the > pthread_lock_mutex() and pthread_unlock_mutex() calls? If performance > is the only potential impact, I'm willing to test it here and see > if we > really need to be concerned. As the code is currently written, it > doesn't appear to be thread-safe, at least on Linux. Which platforms are you thinking of testing it on? I could well imagine Linux doing very well with this error checking enabled, while other platforms might suffer quite a bit. I think the thing to keep in mind is that the error checking code is really there for debugging. Much like you might fill code you are testing with printf's, once you're done debugging, you remove that code. So, in summary, if using PTHREAD_MUTEX_ERRORCHECK is going to help you track down whatever problem you're having, you should definitely use it. :wes From dave64 at andrew.cmu.edu Mon Mar 30 14:01:11 2009 From: dave64 at andrew.cmu.edu (Dave McMurtrie) Date: Mon, 30 Mar 2009 14:01:11 -0400 Subject: Use of "fast" mutexes in mupdate.c In-Reply-To: References: <49D0C09E.3080308@andrew.cmu.edu> Message-ID: <49D108E7.8070307@andrew.cmu.edu> Wesley Craig wrote: > On 30 Mar 2009, at 08:52, Dave McMurtrie wrote: >> I was reading through the mupdate.c code this morning looking for a >> solution to a problem we're experiencing. > > What problem is that? > sigsegv received at the prot_printf below in mupdate.c. C->pout was an invalid address. pthread_mutex_lock(&idle_connlist_mutex); /* LOCK */ for(C=idle_connlist; C; C=C->next_idle) { prot_printf(C->pout, "* BYE \"no longer ready for connections\"\r\n"); > In properly written code, all of the threads are working together. > Thus, while any thread is free to (incorrectly) unlock a critical mutex > that they have no business messing with, in properly written code, they > don't. That would be a bug. You're saying that as long as no other thread ever calls pthread_mutex_unlock() without having already acquired the lock, this could never happen, right? Makes sense. > So, in summary, if using PTHREAD_MUTEX_ERRORCHECK is going to help you > track down whatever problem you're having, you should definitely use it. This isn't directly related to the problem we're having. The second e-mail I sent about mupdate this morning is. In trying to track down that problem, I found a core file that led me to wonder what had stomped on C->pout. Thanks for the feedback. That is helpful. Dave -- Dave McMurtrie, SPE Email Systems Team Leader Carnegie Mellon University, Computing Services From wes at umich.edu Mon Mar 30 14:08:13 2009 From: wes at umich.edu (Wesley Craig) Date: Mon, 30 Mar 2009 14:08:13 -0400 Subject: More mupdate questions. In-Reply-To: <49D0F649.2040100@andrew.cmu.edu> References: <49D0F649.2040100@andrew.cmu.edu> Message-ID: <8D504228-E2B8-4825-9238-C86AC5CA2CD4@umich.edu> On 30 Mar 2009, at 12:41, Dave McMurtrie wrote: > The fundamental problem is that kick_mupdate() returns void. If a > mupdate slave is not currently listening on its local AF_UNIX socket > (like when it disconnects because it loses contact with the mupdate > master) kick_mupdate() will attempt to connect, fail, log an error and > return. Most of the calling code I looked at assumes that if > kick_mupdate() returns, the local database is current. In this case, > that may or may not be true. At the very least, I think > kick_mupdate() > needs to be updated to be able to return a status so any calling code > can make a more informed decision about what to do. I don't have a big problem with more information being provided to the caller, but I'm not all that clear what you'll do with the information in most cases. If the socket was open, you'd block waiting for the MUPDATE NOOP to happen. With the socket closed, what are the superior behaviors? > The area for improvement is that kick_mupdate() never times out its > read() from mupdate's local unix socket. Wouldn't it be a good > idea to > wrap this read() in an alarm() call and return failure? See above. I'd want to see an analysis of what each of the callers would do with the returned kick_mupdate() status. :wes From dave64 at andrew.cmu.edu Mon Mar 30 15:19:12 2009 From: dave64 at andrew.cmu.edu (Dave McMurtrie) Date: Mon, 30 Mar 2009 15:19:12 -0400 Subject: More mupdate questions. In-Reply-To: <8D504228-E2B8-4825-9238-C86AC5CA2CD4@umich.edu> References: <49D0F649.2040100@andrew.cmu.edu> <8D504228-E2B8-4825-9238-C86AC5CA2CD4@umich.edu> Message-ID: <49D11B30.4080000@andrew.cmu.edu> Wesley Craig wrote: > On 30 Mar 2009, at 12:41, Dave McMurtrie wrote: >> The fundamental problem is that kick_mupdate() returns void. If a >> mupdate slave is not currently listening on its local AF_UNIX socket >> (like when it disconnects because it loses contact with the mupdate >> master) kick_mupdate() will attempt to connect, fail, log an error and >> return. Most of the calling code I looked at assumes that if >> kick_mupdate() returns, the local database is current. In this case, >> that may or may not be true. At the very least, I think kick_mupdate() >> needs to be updated to be able to return a status so any calling code >> can make a more informed decision about what to do. > > I don't have a big problem with more information being provided to the > caller, but I'm not all that clear what you'll do with the information > in most cases. If the socket was open, you'd block waiting for the > MUPDATE NOOP to happen. With the socket closed, what are the superior > behaviors? > >> The area for improvement is that kick_mupdate() never times out its >> read() from mupdate's local unix socket. Wouldn't it be a good idea to >> wrap this read() in an alarm() call and return failure? > > See above. I'd want to see an analysis of what each of the callers > would do with the returned kick_mupdate() status. > That depends on what the purpose of kick_mupdate() is supposed to be. The comments I found and the context around where it's called seem to indicate that the caller is trying to ensure that the local mailbox copy is up to date when it calls kick_mupdate(). Looking at the code in kick_mupdate() it looks like the only thing it will ever successfully accomplish is causing a caller to wait until a mailbox sync has completed if the mupdate slave is in the middle of a sync operation. Otherwise, it does nothing. Worse, it fails to report obvious error conditions to the caller. I think the only really bad case where this could break something is in lmtpd.c, inside mlookup(). This bit of code could cause a message to bounce: r = mboxlist_detail(name, &type, NULL, NULL, server, aclp, tid); if (r == IMAP_MAILBOX_NONEXISTENT && config_mupdate_server) { kick_mupdate(); r = mboxlist_detail(name, &type, NULL, NULL, server, aclp, tid); It does a local lookup, calls kick_mupdate(), then does the exact same lookup again. In the case of a network problem like I mentioned previously, mupdate closes its local unix socket while it's attempting to reconnect with the mupdate master. Calls to kick_mupdate() will simply return when the connect fails and mlookup() will return (possibly incorrectly) IMAP_MAILBOX_NONEXISTENT to the caller. Thoughts? Thanks, Dave -- Dave McMurtrie, SPE Email Systems Team Leader Carnegie Mellon University, Computing Services From bernhard at intevation.de Tue Mar 31 03:52:37 2009 From: bernhard at intevation.de (Bernhard Reiter) Date: Tue, 31 Mar 2009 09:52:37 +0200 Subject: "Cyrus IMAP4" to "Cyrus IMAP" accidental change of greeting? Message-ID: <200903310952.41282.bernhard@intevation.de> Just saw that between 2.3.11 and 2.3.12 the greeting changed to not include the "4" of "IMAP4" anymore. https://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/imap/imapd.c.diff?r1=1.545;r2=1.547;f=h The log message indicate no rational of this change, is it indentional? (The second commit fixed a typo that "POP3" was in there, so it does look like a typo indeed.) Ken, both commits are from you. :) I detected this change because we wanted to detect if there is the ability to have shared seen flags in KDE Kontact enterprise and our regex contained "IMAP4", thus we've noticed the change. We'll change the regex, this kind of change should not happen without a reason I guess. (Even better would be to have a capability or so or a better way of detecting the existance of the shared seen flags annotation switch. :) ) Best Regards, Bernhard -- Managing Director - Owner: www.intevation.net (Free Software Company) Germany Coordinator: fsfeurope.org. Coordinator: www.Kolab-Konsortium.com. Intevation GmbH, Osnabr?ck, DE; Amtsgericht Osnabr?ck, HRB 18998 Gesch?ftsf?hrer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20090331/fbbad708/attachment.bin From baconm at email.unc.edu Tue Mar 31 11:23:35 2009 From: baconm at email.unc.edu (Michael Bacon) Date: Tue, 31 Mar 2009 11:23:35 -0400 Subject: Small auth_krb5.c patch for debugging Message-ID: <8E91641EB4BA5E91E58FFE07@ITSBACONM.depts.unc.edu> In a recent system upgrade, I experienced a weekend full of head-pounding hours trying to figure out why saslauthd was failing. It turned out to be a silly problem with non-fully qualified domain names in the /etc/hosts file, but it took forever to figure this out. Based on some google searches, I'm not the only one to have run into this issue. I added a debugging code snippet to auth_krb5.c that provided the critical insight into what was going on, which just logs what saslauthd has decided as the service principal it should use. I think it may be worth incorporating into the trunk. diff -u -r1.1 auth_krb5.c --- /tmp/T0..aWpw Tue Mar 31 11:18:05 2009 +++ auth_krb5.c Sat Mar 14 13:14:51 2009 @@ -266,6 +266,7 @@ krb5_keytab kt = NULL; char thishost[BUFSIZ]; int result = 0; + char *princ_name; memset(&packet, 0, sizeof(packet)); @@ -274,6 +275,11 @@ return 0; } + k5_retcode = krb5_unparse_name(context, server, &princ_name); + syslog(LOG_DEBUG,"k5support_verify_tgt: Principal name set to %s", + princ_name); + free(princ_name); + if (keytabname) { if (krb5_kt_resolve(context, keytabname, &kt)) { goto fini; Michael Bacon ITS Messaging UNC Chapel Hill From dave64 at andrew.cmu.edu Tue Mar 31 12:07:23 2009 From: dave64 at andrew.cmu.edu (Dave McMurtrie) Date: Tue, 31 Mar 2009 12:07:23 -0400 Subject: proposed patch for kick_mupdate Message-ID: <49D23FBB.4020209@andrew.cmu.edu> Good afternoon, Here's a patch I'm proposing for kick_mupdate to make it report error conditions. Wes, you specifically wanted to see how different callers would deal with the returned error status. You'll notice that in some cases I ultimately didn't change the behavior from the calling side. The biggest difference in behavior is in lmtpd.c. So far I've only confirmed that this code will compile. I haven't done any other testing with it yet. Comments are welcome. Thanks, Dave -- Dave McMurtrie, SPE Email Systems Team Leader Carnegie Mellon University, Computing Services -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: kick_mupdate.patch Url: http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20090331/adf881fa/attachment.ksh