From brong at fastmail.fm Mon Jan 5 23:38:01 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Tue, 6 Jan 2009 15:38:01 +1100 Subject: Another day, another cyrus bug :( Message-ID: <20090106043801.GA32108@brong.net> This one is a doozy. mboxlist_lookup returns a live pointer to a malloc'ed copy of the acl. So far so good. Except (I presume to reduce memory management effort for callers of the function) this value is overwritten next time you call mboxlist_lookup again. So - user_renameacl was clever. It got the acl and proceeded to replace the \t values with \0, and pass through the "rights" string to mboxlist_setacl. Which promptly called mboxlist_lookup AGAIN. So basically the user would get all their own ACLs, plus any ACL character that existed in either the usernames or acls of any user after them in the ACL string. v'classy. I figure the easy fix is just to take a copy of the acl with xstrdup. The better fix would be a less insanely dangerous API with action-at-a-distance on existing copies of the string. C gives you enough rope to shoot yourself in the foot (excuse my metaphores), we don't need to help it out! Bron. -------------- next part -------------- A non-text attachment was scrubbed... Name: aclfix.diff Type: text/x-diff Size: 1301 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20090106/737c8738/attachment.bin From wes at umich.edu Tue Jan 6 17:12:13 2009 From: wes at umich.edu (Wesley Craig) Date: Tue, 6 Jan 2009 17:12:13 -0500 Subject: Another day, another cyrus bug :( In-Reply-To: <20090106043801.GA32108@brong.net> References: <20090106043801.GA32108@brong.net> Message-ID: <55E80279-9196-4689-B886-239EAF27B7FA@umich.edu> On 05 Jan 2009, at 23:38, Bron Gondwana wrote: > mboxlist_lookup returns a live pointer to a malloc'ed copy of the > acl. So far so good. > > Except (I presume to reduce memory management effort for callers of > the function) this value is overwritten next time you call > mboxlist_lookup again. > > So - user_renameacl was clever. It got the acl and proceeded to > replace the \t values with \0, and pass through the "rights" string > to mboxlist_setacl. > > Which promptly called mboxlist_lookup AGAIN. This fix looks fine to me. Will you commit it? > The better fix would be a less insanely dangerous API with > action-at-a-distance on existing copies of the string. C gives you > enough rope to shoot yourself in the foot (excuse my metaphores), we > don't need to help it out! I agree. Kind of looks like a case of premature optimization: the author was worried about findall cases where mylookup was going to be called 1M times or so. In the absence of a note about how this kind of memory optimization is necessary for some case, on some platform, or something, I'd be OK seeing the API simplified. :wes From brong at fastmail.fm Tue Jan 6 17:46:43 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 7 Jan 2009 09:46:43 +1100 Subject: Another day, another cyrus bug :( In-Reply-To: <55E80279-9196-4689-B886-239EAF27B7FA@umich.edu> References: <20090106043801.GA32108@brong.net> <55E80279-9196-4689-B886-239EAF27B7FA@umich.edu> Message-ID: <20090106224643.GA13691@brong.net> On Tue, Jan 06, 2009 at 05:12:13PM -0500, Wesley Craig wrote: > This fix looks fine to me. Will you commit it? When I get access ;) I'm just waiting on my password now. I've changed over to using git for all my cyrus work. rebase --interactive and friends are just so nice to work with even compared to quilt. No more patch refreshing to remove fuzz. So my work is in the "fastmail" branch on: git://github.com/brong/cyrus-imapd.git (and git://github.com/brong/cyrus-sieve.git as well, bah) You can view all the patches and changelogs quite nicely on github. I'm moving pretty much all my open source stuff in to there. The best thing with git is that every "clone" of the repository has the full change history with it, so even if github disappeared tomorrow, I wouldn't lose anything of value. I'd just find a new git hosting site (or set up my own) and do something like: git remote newsite git at newsite.com:/brong/$REPOS.git git push --mirror newsite And I'd be back up and running, sharing my work :) >> The better fix would be a less insanely dangerous API with >> action-at-a-distance on existing copies of the string. C gives you >> enough rope to shoot yourself in the foot (excuse my metaphores), we >> don't need to help it out! > > I agree. Kind of looks like a case of premature optimization: the > author was worried about findall cases where mylookup was going to be > called 1M times or so. In the absence of a note about how this kind of > memory optimization is necessary for some case, on some platform, or > something, I'd be OK seeing the API simplified. Cool. I'll have a look around and see where it's used and what changes would need to be made. Building in my Ubuntu Intrepid desktop machine here, I get a pile of warnings as well: master.c:1814: warning: ignoring return value of ?write?, declared with attribute warn_unused_result master.c:1828: warning: ignoring return value of ?write?, declared with attribute warn_unused_result I guess gcc has been strictified up a bit. I'd like to tidy those up too, since I'm sure when Debian Lenny actually gets released we'll upgrade our servers, and it will probably have the warnings too. Bron. From dimma at higis.ru Wed Jan 7 19:11:31 2009 From: dimma at higis.ru (Dmitriy Kirhlarov) Date: Thu, 08 Jan 2009 03:11:31 +0300 Subject: cyrus 2.3.13 ptloader issue Message-ID: Hi, list After upgrade cyrus imapd 2.3.8 -> 2.3.13 ldap authentication (over saslauthd) stop working. We find a problem. ptloader dosn't work without ldap_member_attribute option in imapd.conf I can't get access in cyrus bugzilla, because, password dosn't send me to e-mail Could somebody register this bug? WBR. Dmitriy From Duncan.Gibb at SiriusIT.co.uk Wed Jan 14 12:14:37 2009 From: Duncan.Gibb at SiriusIT.co.uk (Duncan Gibb) Date: Wed, 14 Jan 2009 17:14:37 +0000 Subject: [PATCH] client certificates for TLS In-Reply-To: <491481E8.2010103@SiriusIT.co.uk> References: <491481E8.2010103@SiriusIT.co.uk> Message-ID: <496E1D7D.5020503@SiriusIT.co.uk> On Friday 7 November 2008, Duncan Gibb wrote: DG> The attached patch extends the functionality of backend.c to support DG> client certificates when setting up TLS sessions between one Cyrus DG> machine and another. Perhaps I'm the only person who wants this feature, but I've put in the CMU bugzilla so it won't get completely forgotten, and so it can be referred to conveniently: https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3133 Cheers Duncan -- Duncan Gibb, Technical Director Sirius Corporation plc - The Open Source Experts http://www.siriusit.co.uk/ Tel: +44 870 608 0063 From brong at fastmail.fm Tue Jan 27 23:15:28 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 28 Jan 2009 15:15:28 +1100 Subject: [PATCH 01/13] Correctly truncate cache file on aborted append In-Reply-To: <1233116140-4815-1-git-send-email-brong@fastmail.fm> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> Message-ID: We were truncating to the size of the mmaped area rather than the length of the file, causing blocks of bogus zero bytes in the resulting cache file after an aborted append. =================================================================== --- imap/append.c | 6 ++++-- imap/append.h | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/imap/append.c b/imap/append.c index cb4025b..8f06896 100644 --- a/imap/append.c +++ b/imap/append.c @@ -234,8 +234,10 @@ int append_setup(struct appendstate *as, const char *name, as->userid[0] = '\0'; } + /* store original size to truncate if append is aborted */ + as->orig_cache_size = as->m.cache_size; + /* zero out metadata */ - as->orig_cache_len = as->m.cache_len; as->nummsg = as->numanswered = as->numdeleted = as->numflagged = 0; as->quota_used = 0; @@ -369,7 +371,7 @@ int append_abort(struct appendstate *as) } /* truncate the cache */ - ftruncate(as->m.cache_fd, as->orig_cache_len); + ftruncate(as->m.cache_fd, as->orig_cache_size); /* unlock mailbox */ mailbox_unlock_index(&as->m); diff --git a/imap/append.h b/imap/append.h index 81d1cb6..ec726bb 100644 --- a/imap/append.h +++ b/imap/append.h @@ -76,7 +76,7 @@ struct appendstate { /* current state of append */ /* if we abort, where should we truncate the cache file? */ - unsigned long orig_cache_len; + unsigned long orig_cache_size; int writeheader; /* did we change the mailbox header? */ -- 1.5.6.3 From brong at fastmail.fm Tue Jan 27 23:15:27 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 28 Jan 2009 15:15:27 +1100 Subject: FastMail.FM cyrus patches for review Message-ID: <1233116140-4815-1-git-send-email-brong@fastmail.fm> Here are all the FastMail patches that I think are candidates for upstream. I've love feedback and/or code review on them. They're all in production on our servers now. Unfortunately, the CVS import/export stuff for git is remarkably less nice to use than the equivalent SVN stuff, so I'm actually tracking a slightly older CVS than current. So - CMU people. Any plan to replace your aging CVS install with something modern? Anyway - on to the list. I'm using git's "send email" funtionality, which sends one patch per commit. You can also view this entire tree at: http://github.com/brong/cyrus-imapd/tree/for-upstream (I've separated these out from the other patches that aren't really in upstreamable shape and rebased our "fastmail" branch on top of this list) Bron. From brong at fastmail.fm Tue Jan 27 23:15:29 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 28 Jan 2009 15:15:29 +1100 Subject: [PATCH 02/13] mailbox default options In-Reply-To: References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> Message-ID: <0c1899144fee2faff74e0198a0d5fdf746e40da2.1233115949.git.brong@fastmail.fm> Add a config variable mailbox_default_options which allows you to, for example, enable CONDSTORE on all new mailboxes by default. It's a bitmask, which is on the skanky end of interfaces, but there you go. --- imap/mailbox.c | 8 +++++--- lib/imapoptions | 5 +++++ 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/imap/mailbox.c b/imap/mailbox.c index 58ed79b..c421b56 100644 --- a/imap/mailbox.c +++ b/imap/mailbox.c @@ -959,7 +959,8 @@ int mailbox_read_index_header(struct mailbox *mailbox) ntohl(*((bit32 *)(mailbox->index_base+OFFSET_HIGHESTMODSEQ))); #endif - if (!mailbox->exists) mailbox->options |= OPT_POP3_NEW_UIDL; + if (!mailbox->exists) + mailbox->options |= OPT_POP3_NEW_UIDL; if (!mailbox_doing_reconstruct && (mailbox->minor_version < MAILBOX_MINOR_VERSION)) { @@ -1637,7 +1638,8 @@ static void mailbox_upgrade_index_work(struct mailbox *mailbox, calculate_flagcounts = 1; } if (oldstart_offset < OFFSET_MAILBOX_OPTIONS-quota_offset+sizeof(bit32)) { - unsigned long options = !exists ? OPT_POP3_NEW_UIDL : 0; + unsigned long options = config_getint(IMAPOPT_MAILBOX_DEFAULT_OPTIONS); + if (!exists) options |= OPT_POP3_NEW_UIDL; *((bit32 *)(buf+OFFSET_MAILBOX_OPTIONS)) = htonl(options); } @@ -2730,7 +2732,7 @@ int mailbox_create(const char *name, mailbox.deleted = 0; mailbox.answered = 0; mailbox.flagged = 0; - mailbox.options = OPT_POP3_NEW_UIDL; + mailbox.options = config_getint(IMAPOPT_MAILBOX_DEFAULT_OPTIONS) | OPT_POP3_NEW_UIDL; mailbox.leaked_cache_records = 0; mailbox.highestmodseq = 1; diff --git a/lib/imapoptions b/lib/imapoptions index 0e6f4eb..cb05645 100644 --- a/lib/imapoptions +++ b/lib/imapoptions @@ -534,6 +534,11 @@ are listed with ``''. /* Include notations in the protocol telemetry logs indicating the number of seconds since the last command or response. */ +{ "mailbox_default_options", 0, INT } +/* Default "options" field for the mailbox on create. You'll want to know + what you're doing before setting this, but it can apply some default + annotations like condstore or duplicate supression */ + { "mailnotifier", NULL, STRING } /* Notifyd(8) method to use for "MAIL" notifications. If not set, "MAIL" notifications are disabled. */ -- 1.5.6.3 From brong at fastmail.fm Tue Jan 27 23:15:30 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 28 Jan 2009 15:15:30 +1100 Subject: [PATCH 03/13] setseen overwrite In-Reply-To: <0c1899144fee2faff74e0198a0d5fdf746e40da2.1233115949.git.brong@fastmail.fm> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <0c1899144fee2faff74e0198a0d5fdf746e40da2.1233115949.git.brong@fastmail.fm> Message-ID: <5bfba4579d6adbb3c611cfa94f396043b4d10898.1233115949.git.brong@fastmail.fm> When syncserver gets a setseen_all, it contains an entire skiplist file from the master. What, skiplist you say? Yeah - I'm afraid so. It's sent in binary. That sucks, but here we have it. Fixing that would be a protocol change. But... at least we can just copy the whole thing into place rather than fiddling about updating individual fields. We _know_ it's correct because the master is always right[tm]. Why bother checking each field? =================================================================== --- imap/sync_server.c | 9 ++++++--- 1 files changed, 6 insertions(+), 3 deletions(-) diff --git a/imap/sync_server.c b/imap/sync_server.c index 11a7360..4808c47 100644 --- a/imap/sync_server.c +++ b/imap/sync_server.c @@ -2209,11 +2209,14 @@ static void cmd_setseen_all(char *user, struct buf *data) r = IMAP_IOERROR; } - /* we were operating on the seen state, so merge it and cleanup */ - if (!r) seen_merge(fnamebuf, seen_file); + /* XXX - load fnamebuf as a skiplist file and ensure that + * it's actually valid! */ + + /* overwrite the old seen file */ + if (!r) rename(fnamebuf, seen_file); + else unlink(fnamebuf); free(seen_file); - unlink(fnamebuf); if (filefd != -1) close(filefd); if (r) -- 1.5.6.3 From brong at fastmail.fm Tue Jan 27 23:15:32 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 28 Jan 2009 15:15:32 +1100 Subject: [PATCH 05/13] More logging of DB errors In-Reply-To: <61a00b94d9233f0218a203f1f11b16c9485124f7.1233115949.git.brong@fastmail.fm> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <0c1899144fee2faff74e0198a0d5fdf746e40da2.1233115949.git.brong@fastmail.fm> <5bfba4579d6adbb3c611cfa94f396043b4d10898.1233115949.git.brong@fastmail.fm> <61a00b94d9233f0218a203f1f11b16c9485124f7.1233115949.git.brong@fastmail.fm> Message-ID: We're seeing some DB errors which are probably quota related, but the debugging info isn't very clear. This will make it much more clear exactly what is wrong! --- imap/mboxlist.c | 2 +- imap/quota_db.c | 9 ++++++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/imap/mboxlist.c b/imap/mboxlist.c index f0e9436..5cf0580 100644 --- a/imap/mboxlist.c +++ b/imap/mboxlist.c @@ -239,7 +239,7 @@ static int mboxlist_mylookup(const char *name, int *typep, break; default: - syslog(LOG_ERR, "DBERROR: error fetching %s: %s", + syslog(LOG_ERR, "DBERROR: error fetching mboxlist %s: %s", name, cyrusdb_strerror(r)); return IMAP_IOERROR; break; diff --git a/imap/quota_db.c b/imap/quota_db.c index 96b6720..f63859b 100644 --- a/imap/quota_db.c +++ b/imap/quota_db.c @@ -94,6 +94,13 @@ int quota_read(struct quota *quota, struct txn **tid, int wrlock) if (!data || sscanf(data, UQUOTA_T_FMT " %d", "a->used, "a->limit) != 2) { + char *buf; + buf = (char *) xmalloc(sizeof(char) * datalen + 1); + memcpy(buf, data, datalen); + buf[datalen] = '\0'; + syslog(LOG_ERR, "DBERROR: parsed bogus quota data <%s> for %s", + buf, quota->root); + free(buf); r = CYRUSDB_IOERROR; } break; @@ -108,7 +115,7 @@ int quota_read(struct quota *quota, struct txn **tid, int wrlock) } if (r) { - syslog(LOG_ERR, "DBERROR: error fetching %s: %s", + syslog(LOG_ERR, "DBERROR: error fetching quota %s: %s", quota->root, cyrusdb_strerror(r)); return IMAP_IOERROR; } -- 1.5.6.3 From brong at fastmail.fm Tue Jan 27 23:15:33 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 28 Jan 2009 15:15:33 +1100 Subject: [PATCH 06/13] Fix sync of non-user mailboxes In-Reply-To: References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <0c1899144fee2faff74e0198a0d5fdf746e40da2.1233115949.git.brong@fastmail.fm> <5bfba4579d6adbb3c611cfa94f396043b4d10898.1233115949.git.brong@fastmail.fm> <61a00b94d9233f0218a203f1f11b16c9485124f7.1233115949.git.brong@fastmail.fm> Message-ID: <5e270044bfecb2df177ecc4466ecf70bee15d01b.1233115949.git.brong@fastmail.fm> If a non-user mailbox is being considered for promotion to a USER event in sync_client, it trys to xstrdup the folder name starting at the user name - except that's a NULL string and it segfaults. This patch tests the result of mboxname_isusermailbox in the conditional instead, and then makes the copy only if it's non-NULL. --- imap/sync_client.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/imap/sync_client.c b/imap/sync_client.c index d62c833..8a70974 100644 --- a/imap/sync_client.c +++ b/imap/sync_client.c @@ -3197,12 +3197,12 @@ static int do_sync(const char *filename) if (r) { /* promote failed personal mailboxes to USER */ struct sync_folder *folder; - char *userid, *p; + char *userid, *p, *useridp; for (folder = folder_list->head; folder && folder->mark; folder = folder->next); - if (folder && - (userid = xstrdup(mboxname_isusermailbox(folder->name, 0)))) { + if (folder && (useridp = mboxname_isusermailbox(folder->name, 0))) { + userid = xstrdup(useridp); if ((p = strchr(userid, '.'))) *p = '\0'; folder->mark = 1; if (--folder_list->count == 0) r = 0; -- 1.5.6.3 From brong at fastmail.fm Tue Jan 27 23:15:31 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 28 Jan 2009 15:15:31 +1100 Subject: [PATCH 04/13] Write entire header during upgrade In-Reply-To: <5bfba4579d6adbb3c611cfa94f396043b4d10898.1233115949.git.brong@fastmail.fm> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <0c1899144fee2faff74e0198a0d5fdf746e40da2.1233115949.git.brong@fastmail.fm> <5bfba4579d6adbb3c611cfa94f396043b4d10898.1233115949.git.brong@fastmail.fm> Message-ID: <61a00b94d9233f0218a203f1f11b16c9485124f7.1233115949.git.brong@fastmail.fm> This patch keeps a copy of the index header record written at the start of the index upgrade so it can write it entirely again at the end rather than seeking inside and writing just a part of the record. This is to set up for CRC32 patches later, which will need to have the complete header record on hand to recalculate. Otherwise, it's just neater anyway. --- imap/mailbox.c | 60 ++++++++++++++++++++++++++++---------------------------- 1 files changed, 30 insertions(+), 30 deletions(-) diff --git a/imap/mailbox.c b/imap/mailbox.c index c421b56..696a79a 100644 --- a/imap/mailbox.c +++ b/imap/mailbox.c @@ -1557,7 +1557,9 @@ static void mailbox_upgrade_index_work(struct mailbox *mailbox, unsigned msgno; bit32 oldstart_offset, oldrecord_size; indexbuffer_t ibuf; + indexbuffer_t hbuf; unsigned char *buf = ibuf.buf, *bufp; + unsigned char *headerbuf = hbuf.buf; int quota_offset = 0; int calculate_flagcounts = 0; bit32 numansweredflag = 0; @@ -1566,43 +1568,43 @@ static void mailbox_upgrade_index_work(struct mailbox *mailbox, int old_minor_version = 0; /* Copy existing header so we can upgrade it */ - memcpy(buf, index_base, INDEX_HEADER_SIZE); + memcpy(headerbuf, index_base, INDEX_HEADER_SIZE); - exists = ntohl(*((bit32 *)(buf+OFFSET_EXISTS))); + exists = ntohl(*((bit32 *)(headerbuf+OFFSET_EXISTS))); - old_minor_version = ntohl(*((bit32 *)(buf+OFFSET_MINOR_VERSION))); + old_minor_version = ntohl(*((bit32 *)(headerbuf+OFFSET_MINOR_VERSION))); /* QUOTA_MAILBOX_USED64 added with minor version 6 */ if (old_minor_version < 6) { quota_offset = sizeof(bit32); /* upgrade quota to 64-bits (bump existing fields) */ - memmove(buf+OFFSET_QUOTA_MAILBOX_USED, buf+OFFSET_QUOTA_MAILBOX_USED64, + memmove(headerbuf+OFFSET_QUOTA_MAILBOX_USED, headerbuf+OFFSET_QUOTA_MAILBOX_USED64, INDEX_HEADER_SIZE - OFFSET_QUOTA_MAILBOX_USED64 - quota_offset); /* zero the unused 32-bits */ - *((bit32 *)(buf+OFFSET_QUOTA_MAILBOX_USED64)) = htonl(0); + *((bit32 *)(headerbuf+OFFSET_QUOTA_MAILBOX_USED64)) = htonl(0); } /* HIGHESTMODSEQ[_64] added with minor version 8 */ if (old_minor_version < 8) { /* Set the initial highestmodseq to 1 */ #ifdef HAVE_LONG_LONG_INT - align_htonll(buf+OFFSET_HIGHESTMODSEQ_64, 1); + align_htonll(headerbuf+OFFSET_HIGHESTMODSEQ_64, 1); #else - *((bit32 *)(buf+OFFSET_HIGHESTMODSEQ_64)) = htonl(0); - *((bit32 *)(buf+OFFSET_HIGHESTMODSEQ)) = htonl(1); + *((bit32 *)(headerbuf+OFFSET_HIGHESTMODSEQ_64)) = htonl(0); + *((bit32 *)(headerbuf+OFFSET_HIGHESTMODSEQ)) = htonl(1); #endif } /* change version number */ - *((bit32 *)(buf+OFFSET_MINOR_VERSION)) = htonl(MAILBOX_MINOR_VERSION); + *((bit32 *)(headerbuf+OFFSET_MINOR_VERSION)) = htonl(MAILBOX_MINOR_VERSION); /* save old start_offset; change start_offset */ - oldstart_offset = ntohl(*((bit32 *)(buf+OFFSET_START_OFFSET))); - *((bit32 *)(buf+OFFSET_START_OFFSET)) = htonl(INDEX_HEADER_SIZE); + oldstart_offset = ntohl(*((bit32 *)(headerbuf+OFFSET_START_OFFSET))); + *((bit32 *)(headerbuf+OFFSET_START_OFFSET)) = htonl(INDEX_HEADER_SIZE); /* save old record_size; change record_size */ - oldrecord_size = ntohl(*((bit32 *)(buf+OFFSET_RECORD_SIZE))); - *((bit32 *)(buf+OFFSET_RECORD_SIZE)) = htonl(INDEX_RECORD_SIZE); + oldrecord_size = ntohl(*((bit32 *)(headerbuf+OFFSET_RECORD_SIZE))); + *((bit32 *)(headerbuf+OFFSET_RECORD_SIZE)) = htonl(INDEX_RECORD_SIZE); /* sanity check the record size */ if (oldrecord_size > INDEX_RECORD_SIZE) { @@ -1618,10 +1620,10 @@ static void mailbox_upgrade_index_work(struct mailbox *mailbox, * minor version wasn't updated religiously in the early days, * so we need to use the old offset instead */ if (oldstart_offset < OFFSET_POP3_LAST_LOGIN-quota_offset+sizeof(bit32)) { - *((bit32 *)(buf+OFFSET_POP3_LAST_LOGIN)) = htonl(0); + *((bit32 *)(headerbuf+OFFSET_POP3_LAST_LOGIN)) = htonl(0); } if (oldstart_offset < OFFSET_UIDVALIDITY-quota_offset+sizeof(bit32)) { - *((bit32 *)(buf+OFFSET_UIDVALIDITY)) = htonl(1); + *((bit32 *)(headerbuf+OFFSET_UIDVALIDITY)) = htonl(1); } if (oldstart_offset < OFFSET_FLAGGED-quota_offset+sizeof(bit32)) { struct stat sbuf; @@ -1640,17 +1642,17 @@ static void mailbox_upgrade_index_work(struct mailbox *mailbox, if (oldstart_offset < OFFSET_MAILBOX_OPTIONS-quota_offset+sizeof(bit32)) { unsigned long options = config_getint(IMAPOPT_MAILBOX_DEFAULT_OPTIONS); if (!exists) options |= OPT_POP3_NEW_UIDL; - *((bit32 *)(buf+OFFSET_MAILBOX_OPTIONS)) = htonl(options); + *((bit32 *)(headerbuf+OFFSET_MAILBOX_OPTIONS)) = htonl(options); } - *((bit32 *)(buf+OFFSET_SPARE0)) = htonl(0); /* RESERVED */ - *((bit32 *)(buf+OFFSET_SPARE1)) = htonl(0); /* RESERVED */ - *((bit32 *)(buf+OFFSET_SPARE2)) = htonl(0); /* RESERVED */ - *((bit32 *)(buf+OFFSET_SPARE3)) = htonl(0); /* RESERVED */ - *((bit32 *)(buf+OFFSET_SPARE4)) = htonl(0); /* RESERVED */ + *((bit32 *)(headerbuf+OFFSET_SPARE0)) = htonl(0); /* RESERVED */ + *((bit32 *)(headerbuf+OFFSET_SPARE1)) = htonl(0); /* RESERVED */ + *((bit32 *)(headerbuf+OFFSET_SPARE2)) = htonl(0); /* RESERVED */ + *((bit32 *)(headerbuf+OFFSET_SPARE3)) = htonl(0); /* RESERVED */ + *((bit32 *)(headerbuf+OFFSET_SPARE4)) = htonl(0); /* RESERVED */ /* Write new header */ - fwrite(buf, 1, INDEX_HEADER_SIZE, newindex); + fwrite(headerbuf, 1, INDEX_HEADER_SIZE, newindex); /* Write the rest of new index */ for (msgno = 1; msgno <= exists; msgno++) { @@ -1735,14 +1737,12 @@ static void mailbox_upgrade_index_work(struct mailbox *mailbox, if (calculate_flagcounts) { /* go back and add flag counts to header */ - memset(buf, 0, INDEX_RECORD_SIZE); - *((bit32 *)(buf+OFFSET_DELETED)) = htonl(numdeletedflag); - *((bit32 *)(buf+OFFSET_ANSWERED)) = htonl(numansweredflag); - *((bit32 *)(buf+OFFSET_FLAGGED)) = htonl(numflaggedflag); - - fseek(newindex, OFFSET_DELETED, SEEK_SET); - fwrite(buf+OFFSET_DELETED, - OFFSET_FLAGGED+sizeof(bit32)-OFFSET_DELETED, 1, newindex); + *((bit32 *)(headerbuf+OFFSET_DELETED)) = htonl(numdeletedflag); + *((bit32 *)(headerbuf+OFFSET_ANSWERED)) = htonl(numansweredflag); + *((bit32 *)(headerbuf+OFFSET_FLAGGED)) = htonl(numflaggedflag); + + rewind(newindex); + fwrite(headerbuf, INDEX_HEADER_SIZE, 1, newindex); } } -- 1.5.6.3 From brong at fastmail.fm Tue Jan 27 23:15:38 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 28 Jan 2009 15:15:38 +1100 Subject: [PATCH 11/13] Log "USER" events after renaming a user's "INBOX" to a new username. In-Reply-To: <20851bf77d31d3aa8693c9949b2c38adaa05fb8a.1233115949.git.brong@fastmail.fm> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <0c1899144fee2faff74e0198a0d5fdf746e40da2.1233115949.git.brong@fastmail.fm> <5bfba4579d6adbb3c611cfa94f396043b4d10898.1233115949.git.brong@fastmail.fm> <61a00b94d9233f0218a203f1f11b16c9485124f7.1233115949.git.brong@fastmail.fm> <5e270044bfecb2df177ecc4466ecf70bee15d01b.1233115949.git.brong@fastmail.fm> <0e1c09bb59420fc212bec4cbafcb19fe8fecd085.1233115949.git.brong@fastmail.fm> <20851bf77d31d3aa8693c9949b2c38adaa05fb8a.1233115949.git.brong@fastmail.fm> Message-ID: <4b3de2362d39ac26fafbebb8164465298e591b4a.1233115949.git.brong@fastmail.fm> From: Bron Gondwana --- imap/imapd.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-) diff --git a/imap/imapd.c b/imap/imapd.c index 8501c34..0bd144a 100644 --- a/imap/imapd.c +++ b/imap/imapd.c @@ -5305,6 +5305,13 @@ static int renmbox(char *name, oldextname, newextname); sync_log_mailbox_double(name, text->newmailboxname); + + if (text->rename_user) { + /* allow the replica to get the correct new quotaroot and acls copied across */ + sync_log_user(text->newuser); + /* allow the replica to clean up the old meta files */ + sync_log_user(text->olduser); + } } prot_flush(imapd_out); -- 1.5.6.3 From brong at fastmail.fm Tue Jan 27 23:15:34 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 28 Jan 2009 15:15:34 +1100 Subject: [PATCH 07/13] sync mailbox renames for delayed delete In-Reply-To: <5e270044bfecb2df177ecc4466ecf70bee15d01b.1233115949.git.brong@fastmail.fm> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <0c1899144fee2faff74e0198a0d5fdf746e40da2.1233115949.git.brong@fastmail.fm> <5bfba4579d6adbb3c611cfa94f396043b4d10898.1233115949.git.brong@fastmail.fm> <61a00b94d9233f0218a203f1f11b16c9485124f7.1233115949.git.brong@fastmail.fm> <5e270044bfecb2df177ecc4466ecf70bee15d01b.1233115949.git.brong@fastmail.fm> Message-ID: DelayedDelete and USER events in sync log break Not a candidate for upstream at this point - needs addional debate sync_client "optimises" its sync list by removing MAILBOX events where there's an associated USER event. This is fine until you have delayed delete, because MAILBOX user.foo MAILBOX DELETED.user.foo.4703FDCF USER foo Becomes: MAILBOX DELETED.user.foo.4703FDCF RESET foo And the messages wind up being copied again entirely because the server doesn't know about the mailbox user.foo when the MAILBOX event is running. This patch stops removing the first MAILBOX event. To make CREATE work correctly now, this patch also makes a "MAILBOX user.foo" event be generated along with a "USER foo" event on user create - otherwise sub mailboxes or appends that happen before the first sync will cause bogus deletes on the replica. --- imap/imapd.c | 3 +-- imap/sync_client.c | 4 +++- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/imap/imapd.c b/imap/imapd.c index e872e3e..8501c34 100644 --- a/imap/imapd.c +++ b/imap/imapd.c @@ -5073,8 +5073,7 @@ void cmd_create(char *tag, char *name, char *partition, int localonly) if ( !localonly ) { if (mboxname_isusermailbox(mailboxname, 1)) sync_log_user(mboxname_inbox_touserid(mailboxname)); - else - sync_log_mailbox(mailboxname); + sync_log_mailbox(mailboxname); } } } diff --git a/imap/sync_client.c b/imap/sync_client.c index 8a70974..da6e271 100644 --- a/imap/sync_client.c +++ b/imap/sync_client.c @@ -3021,7 +3021,9 @@ static int do_sync(const char *filename) action for same user */ (sync_namespace.mboxname_tointernal)(&sync_namespace, "INBOX", action->user, inboxname); - remove_folder(inboxname, mailbox_list, 1); + /* breaks DELETED namespace renames + * remove_folder(inboxname, mailbox_list, 1); + */ remove_folder(inboxname, append_list, 1); remove_folder(inboxname, acl_list, 1); remove_folder(inboxname, quota_list, 1); -- 1.5.6.3 From brong at fastmail.fm Tue Jan 27 23:15:39 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 28 Jan 2009 15:15:39 +1100 Subject: [PATCH 12/13] Fix realloc for quotas In-Reply-To: <4b3de2362d39ac26fafbebb8164465298e591b4a.1233115949.git.brong@fastmail.fm> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <0c1899144fee2faff74e0198a0d5fdf746e40da2.1233115949.git.brong@fastmail.fm> <5bfba4579d6adbb3c611cfa94f396043b4d10898.1233115949.git.brong@fastmail.fm> <61a00b94d9233f0218a203f1f11b16c9485124f7.1233115949.git.brong@fastmail.fm> <5e270044bfecb2df177ecc4466ecf70bee15d01b.1233115949.git.brong@fastmail.fm> <0e1c09bb59420fc212bec4cbafcb19fe8fecd085.1233115949.git.brong@fastmail.fm> <20851bf77d31d3aa8693c9949b2c38adaa05fb8a.1233115949.git.brong@fastmail.fm> <4b3de2362d39ac26fafbebb8164465298e591b4a.1233115949.git.brong@fastmail.fm> Message-ID: <65563f87621fab54b551009b3cb35da180fd3fe2.1233115949.git.brong@fastmail.fm> From: Bron Gondwana quotadb_legacyquota was allocing bytes rather than sizeof(char *) and crashing when it got over 100 entries (plus stomping all over the memory after it randomly when over 25 entries) --- lib/cyrusdb_quotalegacy.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/cyrusdb_quotalegacy.c b/lib/cyrusdb_quotalegacy.c index fbe104e..6067c18 100644 --- a/lib/cyrusdb_quotalegacy.c +++ b/lib/cyrusdb_quotalegacy.c @@ -529,7 +529,7 @@ static void scan_qr_dir(char *quota_path, char *prefix, struct qr_path *pathbuf) if (!strncmp(next->d_name, prefix, strlen(prefix))) { if (pathbuf->count == pathbuf->alloc) { pathbuf->alloc += PATH_ALLOC; - pathbuf->path = xrealloc(pathbuf->path, pathbuf->alloc); + pathbuf->path = xrealloc(pathbuf->path, pathbuf->alloc * sizeof(char *)); } pathbuf->path[pathbuf->count] = xmalloc(MAX_QUOTA_PATH+1); sprintf(pathbuf->path[pathbuf->count++], @@ -551,7 +551,7 @@ static void scan_qr_dir(char *quota_path, char *prefix, struct qr_path *pathbuf) if (!stat(quota_path, &buf)) { if (pathbuf->count == pathbuf->alloc) { pathbuf->alloc += PATH_ALLOC; - pathbuf->path = xrealloc(pathbuf->path, pathbuf->alloc); + pathbuf->path = xrealloc(pathbuf->path, pathbuf->alloc * sizeof(char *)); } pathbuf->path[pathbuf->count] = xmalloc(MAX_QUOTA_PATH+1); sprintf(pathbuf->path[pathbuf->count++], -- 1.5.6.3 From brong at fastmail.fm Tue Jan 27 23:15:35 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 28 Jan 2009 15:15:35 +1100 Subject: [PATCH 08/13] Syslog about changes to mailbox stats when reconstructing In-Reply-To: References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <0c1899144fee2faff74e0198a0d5fdf746e40da2.1233115949.git.brong@fastmail.fm> <5bfba4579d6adbb3c611cfa94f396043b4d10898.1233115949.git.brong@fastmail.fm> <61a00b94d9233f0218a203f1f11b16c9485124f7.1233115949.git.brong@fastmail.fm> <5e270044bfecb2df177ecc4466ecf70bee15d01b.1233115949.git.brong@fastmail.fm> Message-ID: <0e1c09bb59420fc212bec4cbafcb19fe8fecd085.1233115949.git.brong@fastmail.fm> From: Bron Gondwana This will help with debugging quota changes, but also finding out anything else odd that's going on in the mailboxes. In _theory_ it should always print nothing if stuff hasn't broken. --- imap/reconstruct.c | 41 +++++++++++++++++++++++++++++++++-------- 1 files changed, 33 insertions(+), 8 deletions(-) diff --git a/imap/reconstruct.c b/imap/reconstruct.c index 83833b4..f6b8aaa 100644 --- a/imap/reconstruct.c +++ b/imap/reconstruct.c @@ -903,7 +903,7 @@ int reconstruct(char *name, struct discovered *found) mailbox.exists = 0; mailbox.last_uid = 0; mailbox.last_appenddate = 0; - mailbox.uidvalidity = time(0); + mailbox.uidvalidity = now; /* If we can't read the index, assume new UIDL so that stupid clients will retrieve all of the messages in the mailbox. */ mailbox.options = OPT_POP3_NEW_UIDL; @@ -1127,8 +1127,8 @@ int reconstruct(char *name, struct discovered *found) message_index.modseq = 1; } - if (message_index.modseq > mailbox.highestmodseq) { - mailbox.highestmodseq = message_index.modseq; + if (message_index.modseq > highestmodseq) { + highestmodseq = message_index.modseq; } /* Force rebuild from message_create_record() */ @@ -1166,12 +1166,37 @@ int reconstruct(char *name, struct discovered *found) } /* Write out new index and expunge file headers */ - if (uid_num && mailbox.last_uid < uid[uid_num-1]) + if (uid_num && mailbox.last_uid < uid[uid_num-1]) { + syslog (LOG_ERR, "Updating last_uid for %s: %lu => %lu", + mailbox.name, mailbox.last_uid, uid[uid_num-1] + 100); mailbox.last_uid = uid[uid_num-1] + 100; - if (mailbox.last_appenddate == 0 || mailbox.last_appenddate > time(0)) - mailbox.last_appenddate = time(0); - if (mailbox.uidvalidity == 0 || mailbox.uidvalidity > (unsigned) time(0)) - mailbox.uidvalidity = time(0); + } + + if (mailbox.last_appenddate == 0 || mailbox.last_appenddate > now) { + syslog (LOG_ERR, "Updating last_appenddate for %s: %lu => %lu", + mailbox.name, mailbox.last_appenddate, now); + mailbox.last_appenddate = now; + } + + if (mailbox.uidvalidity == 0 || mailbox.uidvalidity > (unsigned)now) { + syslog (LOG_ERR, "Updating uidvalidity for %s: %lu => %lu", + mailbox.name, mailbox.uidvalidity, now); + mailbox.uidvalidity = (unsigned)now; + } + + if (mailbox.highestmodseq < highestmodseq) { + syslog (LOG_ERR, "Updating highestmodseq for %s: %lu => %lu", + mailbox.name, (unsigned long)mailbox.highestmodseq, + (unsigned long)highestmodseq); + mailbox.highestmodseq = highestmodseq; + } + + if (mailbox.quota_mailbox_used != index_counts.newquota_used) { + syslog (LOG_ERR, "Updating quota_mailbox_used for %s: %lu => %lu", + mailbox.name, (unsigned long)mailbox.quota_mailbox_used, + (unsigned long)index_counts.newquota_used); + /* updated by the counts_tobuf below, different in each file */ + } rewind(newindex); reconstruct_counts_tobuf(buf, &mailbox, &index_counts); -- 1.5.6.3 From brong at fastmail.fm Tue Jan 27 23:15:36 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 28 Jan 2009 15:15:36 +1100 Subject: [PATCH 09/13] sync all mailbox annotations, not just CONDSTORE In-Reply-To: <0e1c09bb59420fc212bec4cbafcb19fe8fecd085.1233115949.git.brong@fastmail.fm> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <0c1899144fee2faff74e0198a0d5fdf746e40da2.1233115949.git.brong@fastmail.fm> <5bfba4579d6adbb3c611cfa94f396043b4d10898.1233115949.git.brong@fastmail.fm> <61a00b94d9233f0218a203f1f11b16c9485124f7.1233115949.git.brong@fastmail.fm> <5e270044bfecb2df177ecc4466ecf70bee15d01b.1233115949.git.brong@fastmail.fm> <0e1c09bb59420fc212bec4cbafcb19fe8fecd085.1233115949.git.brong@fastmail.fm> Message-ID: From: Bron Gondwana Rewrite the mailboxopt annotations to be data driven, and use the same struct to process ALL annotations on the mailbox options field during a sync run. Fixes duplicatedeliver and sharedseen synchronisation, and is future-proof. --- imap/annotate.c | 47 ++++++++++++++++++++++++++--------------------- imap/annotate.h | 8 ++++++++ imap/mailbox.h | 3 ++- imap/sync_client.c | 13 +++++++------ 4 files changed, 43 insertions(+), 28 deletions(-) diff --git a/imap/annotate.c b/imap/annotate.c index 45db495..1484079 100644 --- a/imap/annotate.c +++ b/imap/annotate.c @@ -422,6 +422,17 @@ struct mailbox_annotation_rock char *server, *partition, *acl, *path, *mpath; }; +const struct annotate_info_t annotate_mailbox_flags[] = +{ + { "/vendor/cmu/cyrus-imapd/duplicatedeliver", + OPT_IMAP_DUPDELIVER }, + { "/vendor/cmu/cyrus-imapd/sharedseen", + OPT_IMAP_SHAREDSEEN }, + { "/vendor/cmu/cyrus-imapd/condstore", + OPT_IMAP_CONDSTORE }, + { NULL, 0 } +}; + /* To free values in the mailbox_annotation_rock as needed */ static void cleanup_mbrock(struct mailbox_annotation_rock *mbrock __attribute__((unused))) { @@ -878,7 +889,7 @@ static void annotation_get_mailboxopt(const char *int_mboxname, void *rock __attribute__((unused))) { struct mailbox mailbox; - int flag, r = 0; + int flag = 0, r = 0, i; char value[40]; struct annotation_data attrib; @@ -891,16 +902,14 @@ static void annotation_get_mailboxopt(const char *int_mboxname, /* Make sure its a local mailbox */ if (mbrock->server) return; - /* Check entry */ - if (!strcmp(entry, "/vendor/cmu/cyrus-imapd/condstore")) { - flag = OPT_IMAP_CONDSTORE; - } else if (!strcmp(entry, "/vendor/cmu/cyrus-imapd/sharedseen")) { - flag = OPT_IMAP_SHAREDSEEN; - } else if (!strcmp(entry, "/vendor/cmu/cyrus-imapd/duplicatedeliver")) { - flag = OPT_IMAP_DUPDELIVER; - } else { - return; + /* check that this is a mailboxopt annotation */ + for (i = 0; annotate_mailbox_flags[i].name; i++) { + if (!strcmp(entry, annotate_mailbox_flags[i].name)) { + flag = annotate_mailbox_flags[i].flag; + break; + } } + if (!flag) return; /* Check ACL */ if(!fdata->isadmin && @@ -1717,20 +1726,16 @@ static int annotation_set_mailboxopt(const char *int_mboxname, void *rock __attribute__((unused))) { struct mailbox mailbox; - int flag, r = 0; + int flag = 0, r = 0, i; /* Check entry */ - if (!strcmp(entry->entry->name, "/vendor/cmu/cyrus-imapd/condstore")) { - flag = OPT_IMAP_CONDSTORE; - } else if (!strcmp(entry->entry->name, - "/vendor/cmu/cyrus-imapd/sharedseen")) { - flag = OPT_IMAP_SHAREDSEEN; - } else if (!strcmp(entry->entry->name, - "/vendor/cmu/cyrus-imapd/duplicatedeliver")) { - flag = OPT_IMAP_DUPDELIVER; - } else { - return IMAP_PERMISSION_DENIED; + for (i = 0; annotate_mailbox_flags[i].name; i++) { + if (!strcmp(entry->entry->name, annotate_mailbox_flags[i].name)) { + flag = annotate_mailbox_flags[i].flag; + break; + } } + if (!flag) return IMAP_PERMISSION_DENIED; /* Check ACL */ if(!sdata->isadmin && diff --git a/imap/annotate.h b/imap/annotate.h index 4482580..5f8d7dc 100644 --- a/imap/annotate.h +++ b/imap/annotate.h @@ -119,6 +119,14 @@ struct annotation_data { const char *contenttype; }; +struct annotate_info_t +{ + const char *name; + int flag; +}; + +extern const struct annotate_info_t annotate_mailbox_flags[]; + /* lookup a single annotation and return result */ int annotatemore_lookup(const char *mboxname, const char *entry, const char *userid, struct annotation_data *attrib); diff --git a/imap/mailbox.h b/imap/mailbox.h index 066a435..0c6c887 100644 --- a/imap/mailbox.h +++ b/imap/mailbox.h @@ -257,11 +257,12 @@ struct index_record { #define FLAG_DRAFT (1<<3) #define OPT_POP3_NEW_UIDL (1<<0) /* added for Outlook stupidity */ +/* these three are annotations, if you add more, update annotate.c + * struct annotate_mailbox_flags */ #define OPT_IMAP_CONDSTORE (1<<1) /* added for CONDSTORE extension */ #define OPT_IMAP_SHAREDSEEN (1<<2) /* added for shared \Seen flag */ #define OPT_IMAP_DUPDELIVER (1<<3) /* added to allow duplicate delivery */ - struct mailbox_header_cache { const char *name; /* Name of header */ bit32 min_cache_version; /* Cache version it appeared in */ diff --git a/imap/sync_client.c b/imap/sync_client.c index da6e271..d08f5c4 100644 --- a/imap/sync_client.c +++ b/imap/sync_client.c @@ -1828,7 +1828,7 @@ int do_folders(struct sync_folder_list *client_list, int doing_user) { struct mailbox m; - int r = 0, mailbox_open = 0; + int r = 0, mailbox_open = 0, i; struct sync_rename_list *rename_list = sync_rename_list_create(); struct sync_folder *folder, *folder2; @@ -1978,13 +1978,14 @@ int do_folders(struct sync_folder_list *client_list, if (!r && (m.uidvalidity != folder2->uidvalidity)) r = folder_setuidvalidity(folder->name, m.uidvalidity); - if (!r && - (folder2->options ^ m.options) & OPT_IMAP_CONDSTORE) { - r = folder_setannotation(m.name, - "/vendor/cmu/cyrus-imapd/condstore", + for (i = 0; !r && annotate_mailbox_flags[i].name; i++) { + if ((folder2->options ^ m.options) & annotate_mailbox_flags[i].flag) { + r = folder_setannotation(m.name, + annotate_mailbox_flags[i].name, "", - (m.options & OPT_IMAP_CONDSTORE) ? + (m.options & annotate_mailbox_flags[i].flag) ? "true" : "false"); + } } if (!r && m.quota.root && !strcmp(m.name, m.quota.root)) -- 1.5.6.3 From brong at fastmail.fm Tue Jan 27 23:15:37 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 28 Jan 2009 15:15:37 +1100 Subject: [PATCH 10/13] Fix ACL copying when renaming users In-Reply-To: References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <0c1899144fee2faff74e0198a0d5fdf746e40da2.1233115949.git.brong@fastmail.fm> <5bfba4579d6adbb3c611cfa94f396043b4d10898.1233115949.git.brong@fastmail.fm> <61a00b94d9233f0218a203f1f11b16c9485124f7.1233115949.git.brong@fastmail.fm> <5e270044bfecb2df177ecc4466ecf70bee15d01b.1233115949.git.brong@fastmail.fm> <0e1c09bb59420fc212bec4cbafcb19fe8fecd085.1233115949.git.brong@fastmail.fm> Message-ID: <20851bf77d31d3aa8693c9949b2c38adaa05fb8a.1233115949.git.brong@fastmail.fm> From: Bron Gondwana Gosh mboxlist_lookup provides a dangerous interface! --- imap/user.c | 17 +++++++++++++++-- 1 files changed, 15 insertions(+), 2 deletions(-) diff --git a/imap/user.c b/imap/user.c index 8d439c0..5ae03d7 100644 --- a/imap/user.c +++ b/imap/user.c @@ -91,8 +91,12 @@ static int user_deleteacl(char *name, int matchlen, int maycreate, void* rock) int r; char *acl; char *rights, *nextid; + char *origacl, *aclalloc; - r = mboxlist_lookup(name, &acl, NULL); + r = mboxlist_lookup(name, &origacl, NULL); + + /* setacl re-calls mboxlist_lookup and will stomp on us */ + aclalloc = acl = xstrdup(origacl); while (!r && acl) { rights = strchr(acl, '\t'); @@ -111,6 +115,9 @@ static int user_deleteacl(char *name, int matchlen, int maycreate, void* rock) acl = nextid; } + + free(aclalloc); + return 0; } #endif @@ -370,8 +377,12 @@ int user_renameacl(char *name, char *olduser, char *newuser) int r = 0; char *acl; char *rights, *nextid; + char *origacl, *aclalloc; - r = mboxlist_lookup(name, &acl, NULL); + r = mboxlist_lookup(name, &origacl, NULL); + + /* setacl re-calls mboxlist_lookup and will stomp on us */ + aclalloc = acl = xstrdup(origacl); while (!r && acl) { rights = strchr(acl, '\t'); @@ -393,6 +404,8 @@ int user_renameacl(char *name, char *olduser, char *newuser) acl = nextid; } + free(aclalloc); + return r; } -- 1.5.6.3 From brong at fastmail.fm Tue Jan 27 23:15:40 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 28 Jan 2009 15:15:40 +1100 Subject: [PATCH 13/13] Check subscription database rather than explicit ADDSUB/DELSUB In-Reply-To: <65563f87621fab54b551009b3cb35da180fd3fe2.1233115949.git.brong@fastmail.fm> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <0c1899144fee2faff74e0198a0d5fdf746e40da2.1233115949.git.brong@fastmail.fm> <5bfba4579d6adbb3c611cfa94f396043b4d10898.1233115949.git.brong@fastmail.fm> <61a00b94d9233f0218a203f1f11b16c9485124f7.1233115949.git.brong@fastmail.fm> <5e270044bfecb2df177ecc4466ecf70bee15d01b.1233115949.git.brong@fastmail.fm> <0e1c09bb59420fc212bec4cbafcb19fe8fecd085.1233115949.git.brong@fastmail.fm> <20851bf77d31d3aa8693c9949b2c38adaa05fb8a.1233115949.git.brong@fastmail.fm> <4b3de2362d39ac26fafbebb8164465298e591b4a.1233115949.git.brong@fastmail.fm> <65563f87621fab54b551009b3cb35da180fd3fe2.1233115949.git.brong@fastmail.fm> Message-ID: We were seeing occasional subscription mismatches at FastMail, and the root cause was traced back to the re-ordering of log files. All the ADDSUB commands would run first, then all the DELSUB commands. If a log file contained a DELSUB followed by an ADDSUB, they would be re-ordered, and the subscription would exists on the master but not the replica. This patch changes the specialness of subscriptions, so that instead of SUB and UNSUB events, it's just SUB events, and the sync_client will check the subscription database to find what the value should be, and tell the backend to make it so. --- imap/cyr_synclog.c | 2 +- imap/imapd.c | 2 +- imap/mboxlist.c | 16 +++++++++++++ imap/mboxlist.h | 3 ++ imap/sync_client.c | 64 ++++++++++++++++++++------------------------------- imap/sync_log.h | 5 +-- 6 files changed, 48 insertions(+), 44 deletions(-) diff --git a/imap/cyr_synclog.c b/imap/cyr_synclog.c index 78489ac..b935c84 100644 --- a/imap/cyr_synclog.c +++ b/imap/cyr_synclog.c @@ -153,7 +153,7 @@ int main(int argc, char *argv[]) sync_log_seen(argv[optind], argv[optind+1]); break; case 'b': /* suBscription */ - sync_log_subscribe(argv[optind], argv[optind+1], argv[optind+2]); + sync_log_subscribe(argv[optind], argv[optind+1]); break; default: /* just as is! */ diff --git a/imap/imapd.c b/imap/imapd.c index 0bd144a..40acf66 100644 --- a/imap/imapd.c +++ b/imap/imapd.c @@ -6071,7 +6071,7 @@ void cmd_changesub(char *tag, char *namespace, char *name, int add) else { prot_printf(imapd_out, "%s OK %s\r\n", tag, error_message(IMAP_OK_COMPLETED)); - sync_log_subscribe(imapd_userid, mailboxname, add); + sync_log_subscribe(imapd_userid, mailboxname); } } diff --git a/imap/mboxlist.c b/imap/mboxlist.c index 5cf0580..166eb4e 100644 --- a/imap/mboxlist.c +++ b/imap/mboxlist.c @@ -3311,6 +3311,22 @@ int mboxlist_findsub_alt(struct namespace *namespace, return r; } +/* returns CYRUSDB_NOTFOUND if the folder doesn't exist, and 0 if it does! */ +int mboxlist_checksub(const char *name, const char *userid) +{ + int r; + struct db *subs; + char *val; + int vallen; + + r = mboxlist_opensubs(userid, &subs); + + if (!r) r = SUBDB->fetch(subs, name, strlen(name), &val, &vallen, NULL); + + mboxlist_closesubs(subs); + return r; +} + /* * Change 'user's subscription status for mailbox 'name'. * Subscribes if 'add' is nonzero, unsubscribes otherwise. diff --git a/imap/mboxlist.h b/imap/mboxlist.h index 59af143..b336a3d 100644 --- a/imap/mboxlist.h +++ b/imap/mboxlist.h @@ -179,6 +179,9 @@ int mboxlist_findsub_alt(struct namespace *namespace, 'stagedir' should be MAX_MAILBOX_PATH. */ int mboxlist_findstage(const char *name, char *stagedir, size_t sd_len); +/* Check 'user's subscription status for mailbox 'name' */ +int mboxlist_checksub(const char *name, const char *userid); + /* Change 'user's subscription status for mailbox 'name'. */ int mboxlist_changesub(const char *name, const char *userid, struct auth_state *auth_state, int add, int force); diff --git a/imap/sync_client.c b/imap/sync_client.c index d08f5c4..45fc8f5 100644 --- a/imap/sync_client.c +++ b/imap/sync_client.c @@ -88,6 +88,8 @@ #include "backend.h" #include "xstrlcat.h" #include "xstrlcpy.h" +#include "signals.h" +#include "cyrusdb.h" /* signal to config.c */ const int config_need_data = 0; /* YYY */ @@ -697,40 +699,41 @@ static int folder_delete(char *name) return(sync_parse_code("DELETE", fromserver, SYNC_PARSE_EAT_OKLINE, NULL)); } -static int user_addsub(char *user, char *name) +static int set_sub(char *user, char *name, int add) { + char *cmd = add ? "ADDSUB" : "DELSUB"; + if (verbose) - printf("ADDSUB %s %s\n", user, name); + printf("%s %s %s\n", cmd, user, name); if (verbose_logging) - syslog(LOG_INFO, "ADDSUB %s %s", user, name); + syslog(LOG_INFO, "%s %s %s", cmd, user, name); - prot_printf(toserver, "ADDSUB "); + sync_printastring(toserver, cmd); + prot_printf(toserver, " "); sync_printastring(toserver, user); prot_printf(toserver, " "); sync_printastring(toserver, name); prot_printf(toserver, "\r\n"); prot_flush(toserver); - return(sync_parse_code("ADDSUB", fromserver, SYNC_PARSE_EAT_OKLINE, NULL)); + return(sync_parse_code(cmd, fromserver, SYNC_PARSE_EAT_OKLINE, NULL)); } -static int user_delsub(char *user, char *name) +static int user_sub(char *user, char *name) { - if (verbose) - printf("DELSUB %s %s\n", user, name); - - if (verbose_logging) - syslog(LOG_INFO, "DELSUB %s %s", user, name); + int r; - prot_printf(toserver, "DELSUB "); - sync_printastring(toserver, user); - prot_printf(toserver, " "); - sync_printastring(toserver, name); - prot_printf(toserver, "\r\n"); - prot_flush(toserver); + r = mboxlist_checksub(name, user); - return(sync_parse_code("DELSUB", fromserver, SYNC_PARSE_EAT_OKLINE, NULL)); + switch (r) { + case CYRUSDB_OK: + return set_sub(user, name, 1); + case CYRUSDB_NOTFOUND: + return set_sub(user, name, 0); + default: + return r; + } } static int folder_setacl(char *name, char *acl) @@ -2354,7 +2357,7 @@ int do_user_sub(char *user, struct sync_folder_list *server_list) do { (sync_namespace.mboxname_tointernal)(&sync_namespace, s->name, user, buf); - if ((r = user_delsub(user, buf))) goto bail; + if ((r = set_sub(user, buf, 0))) goto bail; s = s->next; if (!s) n = -1; /* end of server list, we're done */ else if (!c) n = 1; /* remove all server subscriptions */ @@ -2368,7 +2371,7 @@ int do_user_sub(char *user, struct sync_folder_list *server_list) } else if (c && n < 0) { /* add the current client subscription */ - if ((r = user_addsub(user, c->name))) goto bail; + if ((r = set_sub(user, c->name, 1))) goto bail; } } @@ -2926,7 +2929,6 @@ static int do_sync(const char *filename) struct sync_action_list *annot_list = sync_action_list_create(); struct sync_action_list *seen_list = sync_action_list_create(); struct sync_action_list *sub_list = sync_action_list_create(); - struct sync_action_list *unsub_list = sync_action_list_create(); struct sync_folder_list *folder_list = sync_folder_list_create(); static struct buf type, arg1, arg2; char *arg1s, *arg2s; @@ -3007,7 +3009,7 @@ static int do_sync(const char *filename) else if (!strcmp(type.s, "SUB")) sync_action_list_add(sub_list, arg2s, arg1s); else if (!strcmp(type.s, "UNSUB")) - sync_action_list_add(unsub_list, arg2s, arg1s); + sync_action_list_add(sub_list, arg2s, arg1s); else syslog(LOG_ERR, "Unknown action type: %s", type.s); } @@ -3033,7 +3035,6 @@ static int do_sync(const char *filename) remove_meta(action->user, sieve_list); remove_meta(action->user, seen_list); remove_meta(action->user, sub_list); - remove_meta(action->user, unsub_list); } for (action = meta_list->head ; action ; action = action->next) { @@ -3042,7 +3043,6 @@ static int do_sync(const char *filename) remove_meta(action->user, sieve_list); remove_meta(action->user, seen_list); remove_meta(action->user, sub_list); - remove_meta(action->user, unsub_list); } for (action = mailbox_list->head ; action ; action = action->next) { @@ -3159,7 +3159,7 @@ static int do_sync(const char *filename) } for (action = sub_list->head ; action ; action = action->next) { - if (action->active && user_addsub(action->user, action->name)) { + if (action->active && user_sub(action->user, action->name)) { sync_action_list_add(meta_list, NULL, action->user); if (verbose) { printf(" Promoting: SUB %s %s -> META %s\n", @@ -3172,19 +3172,6 @@ static int do_sync(const char *filename) } } - for (action = unsub_list->head ; action ; action = action->next) { - if (action->active && user_delsub(action->user, action->name)) { - sync_action_list_add(meta_list, NULL, action->user); - if (verbose) { - printf(" Promoting: UNSUB %s %s -> META %s\n", - action->user, action->name, action->user); - } - if (verbose_logging) { - syslog(LOG_INFO, " Promoting: UNSUB %s %s -> META %s", - action->user, action->name, action->name); - } - } - } for (action = mailbox_list->head ; action ; action = action->next) { if (!action->active) continue; @@ -3275,7 +3262,6 @@ static int do_sync(const char *filename) sync_action_list_free(&annot_list); sync_action_list_free(&seen_list); sync_action_list_free(&sub_list); - sync_action_list_free(&unsub_list); sync_folder_list_free(&folder_list); prot_free(input); diff --git a/imap/sync_log.h b/imap/sync_log.h index f1c267d..fb349e6 100644 --- a/imap/sync_log.h +++ b/imap/sync_log.h @@ -81,8 +81,7 @@ void sync_log(char *fmt, ...); #define sync_log_seen(user, name) \ sync_log("SEEN %s %s\n", user, name) -#define sync_log_subscribe(user, name, add) \ - if (add) sync_log("SUB %s %s\n", user, name); \ - else sync_log("UNSUB %s %s\n", user, name) +#define sync_log_subscribe(user, name) \ + sync_log("SUB %s %s\n", user, name) #endif /* INCLUDED_SYNC_LOG_H */ -- 1.5.6.3 From dave64 at andrew.cmu.edu Wed Jan 28 08:41:01 2009 From: dave64 at andrew.cmu.edu (Dave McMurtrie) Date: Wed, 28 Jan 2009 08:41:01 -0500 Subject: FastMail.FM cyrus patches for review In-Reply-To: <1233116140-4815-1-git-send-email-brong@fastmail.fm> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> Message-ID: <4980606D.6090103@andrew.cmu.edu> Bron Gondwana wrote: > Here are all the FastMail patches that I think are candidates > for upstream. > > I've love feedback and/or code review on them. They're all in > production on our servers now. > > Unfortunately, the CVS import/export stuff for git is remarkably > less nice to use than the equivalent SVN stuff, so I'm actually > tracking a slightly older CVS than current. > > So - CMU people. Any plan to replace your aging CVS install with > something modern? Our primary goal is always focused on providing the best possible e-mail service to the CMU community, so if replacing CVS makes Ken's job easier or makes it easier for outside folks to contribute to the Cyrus code base we'd be behind it. I know you've been using git and seem to be pleased with it. Is there anything else you have in mind? Ken ultimately outvotes me in this decision since he's the primary in-house user of the Cyrus CVS repository. If changing it makes him unhappy, that would make me unhappy, too :) Thanks, Dave -- Dave McMurtrie, SPE Email Systems Team Leader Carnegie Mellon University, Computing Services From brong at fastmail.fm Wed Jan 28 15:43:23 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 29 Jan 2009 07:43:23 +1100 Subject: FastMail.FM cyrus patches for review In-Reply-To: <4980606D.6090103@andrew.cmu.edu> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <4980606D.6090103@andrew.cmu.edu> Message-ID: <20090128204323.GA7769@brong.net> On Wed, Jan 28, 2009 at 08:41:01AM -0500, Dave McMurtrie wrote: > Bron Gondwana wrote: > > Here are all the FastMail patches that I think are candidates > > for upstream. > > > > I've love feedback and/or code review on them. They're all in > > production on our servers now. > > > > Unfortunately, the CVS import/export stuff for git is remarkably > > less nice to use than the equivalent SVN stuff, so I'm actually > > tracking a slightly older CVS than current. > > > > So - CMU people. Any plan to replace your aging CVS install with > > something modern? > > Our primary goal is always focused on providing the best possible e-mail > service to the CMU community, so if replacing CVS makes Ken's job easier > or makes it easier for outside folks to contribute to the Cyrus code > base we'd be behind it. Fair enough. > I know you've been using git and seem to be pleased with it. Is there > anything else you have in mind? I like git, but seriously, for a code base the size of Cyrus I'd be happy with any of the modern distributed VCSes, or even just Subversion. Subversion, for all that it's sort of "end of line" has all the features required (atomic commits in particular) for other tools to interface with it. It's become sort of a lowest common denominator. (as an example, I track DJabberd's SVN using git, and FastMail's own internal SVN using git as well!) > Ken ultimately outvotes me in this decision since he's the primary > in-house user of the Cyrus CVS repository. If changing it makes him > unhappy, that would make me unhappy, too :) Yeah, sounds fair. I'm happy with just about anything (except no source control at all... man that's unfun), but CVS is pretty much the worst of what's out there these days. It's not something that really stresses me whatever way it goes. Bron ( so, is anyone going to comment on the actual patches? ;) From wes at umich.edu Wed Jan 28 19:36:29 2009 From: wes at umich.edu (Wesley Craig) Date: Wed, 28 Jan 2009 19:36:29 -0500 Subject: [PATCH 01/13] Correctly truncate cache file on aborted append In-Reply-To: References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> Message-ID: <32596CA7-D430-43D7-8D3E-C65C18612069@umich.edu> Looks fine. Is this an open bugzilla? I recall you reporting the problem. :wes On 27 Jan 2009, at 23:15, Bron Gondwana wrote: > We were truncating to the size of the mmaped area rather than the > length of the file, causing blocks of bogus zero bytes in the > resulting cache file after an aborted append. > > =================================================================== > --- > imap/append.c | 6 ++++-- > imap/append.h | 2 +- > 2 files changed, 5 insertions(+), 3 deletions(-) > > diff --git a/imap/append.c b/imap/append.c > index cb4025b..8f06896 100644 > --- a/imap/append.c > +++ b/imap/append.c > @@ -234,8 +234,10 @@ int append_setup(struct appendstate *as, const > char *name, > as->userid[0] = '\0'; > } > > + /* store original size to truncate if append is aborted */ > + as->orig_cache_size = as->m.cache_size; > + > /* zero out metadata */ > - as->orig_cache_len = as->m.cache_len; > as->nummsg = as->numanswered = > as->numdeleted = as->numflagged = 0; > as->quota_used = 0; > @@ -369,7 +371,7 @@ int append_abort(struct appendstate *as) > } > > /* truncate the cache */ > - ftruncate(as->m.cache_fd, as->orig_cache_len); > + ftruncate(as->m.cache_fd, as->orig_cache_size); > > /* unlock mailbox */ > mailbox_unlock_index(&as->m); > diff --git a/imap/append.h b/imap/append.h > index 81d1cb6..ec726bb 100644 > --- a/imap/append.h > +++ b/imap/append.h > @@ -76,7 +76,7 @@ struct appendstate { > /* current state of append */ > > /* if we abort, where should we truncate the cache file? */ > - unsigned long orig_cache_len; > + unsigned long orig_cache_size; > > int writeheader; /* did we change the mailbox header? */ > > -- > 1.5.6.3 From wes at umich.edu Wed Jan 28 19:38:01 2009 From: wes at umich.edu (Wesley Craig) Date: Wed, 28 Jan 2009 19:38:01 -0500 Subject: [PATCH 06/13] Fix sync of non-user mailboxes In-Reply-To: <5e270044bfecb2df177ecc4466ecf70bee15d01b.1233115949.git.brong@fastmail.fm> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <0c1899144fee2faff74e0198a0d5fdf746e40da2.1233115949.git.brong@fastmail.fm> <5bfba4579d6adbb3c611cfa94f396043b4d10898.1233115949.git.brong@fastmail.fm> <61a00b94d9233f0218a203f1f11b16c9485124f7.1233115949.git.brong@fastmail.fm> <5e270044bfecb2df177ecc4466ecf70bee15d01b.1233115949.git.brong@fastmail.fm> Message-ID: On 27 Jan 2009, at 23:15, Bron Gondwana wrote: > If a non-user mailbox is being considered for promotion to a USER > event in sync_client, it trys to xstrdup the folder name starting > at the user name - except that's a NULL string and it segfaults. > > This patch tests the result of mboxname_isusermailbox in the > conditional instead, and then makes the copy only if it's non-NULL. Looks good. It would be cool if it had a BZ number, since it's pretty obviously a bug. :wes From wes at umich.edu Wed Jan 28 19:39:17 2009 From: wes at umich.edu (Wesley Craig) Date: Wed, 28 Jan 2009 19:39:17 -0500 Subject: [PATCH 05/13] More logging of DB errors In-Reply-To: References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <0c1899144fee2faff74e0198a0d5fdf746e40da2.1233115949.git.brong@fastmail.fm> <5bfba4579d6adbb3c611cfa94f396043b4d10898.1233115949.git.brong@fastmail.fm> <61a00b94d9233f0218a203f1f11b16c9485124f7.1233115949.git.brong@fastmail.fm> Message-ID: Looks good. :wes On 27 Jan 2009, at 23:15, Bron Gondwana wrote: > We're seeing some DB errors which are probably quota related, > but the debugging info isn't very clear. This will make it > much more clear exactly what is wrong! From wes at umich.edu Wed Jan 28 19:45:45 2009 From: wes at umich.edu (Wesley Craig) Date: Wed, 28 Jan 2009 19:45:45 -0500 Subject: [PATCH 10/13] Fix ACL copying when renaming users In-Reply-To: <20851bf77d31d3aa8693c9949b2c38adaa05fb8a.1233115949.git.brong@fastmail.fm> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <0c1899144fee2faff74e0198a0d5fdf746e40da2.1233115949.git.brong@fastmail.fm> <5bfba4579d6adbb3c611cfa94f396043b4d10898.1233115949.git.brong@fastmail.fm> <61a00b94d9233f0218a203f1f11b16c9485124f7.1233115949.git.brong@fastmail.fm> <5e270044bfecb2df177ecc4466ecf70bee15d01b.1233115949.git.brong@fastmail.fm> <0e1c09bb59420fc212bec4cbafcb19fe8fecd085.1233115949.git.brong@fastmail.fm> <20851bf77d31d3aa8693c9949b2c38adaa05fb8a.1233115949.git.brong@fastmail.fm> Message-ID: I've seen this before... :) Looks fine. Is there a BZ open for it? :wes On 27 Jan 2009, at 23:15, Bron Gondwana wrote: > Gosh mboxlist_lookup provides a dangerous interface! From jeaton at cmu.edu Wed Jan 28 20:11:56 2009 From: jeaton at cmu.edu (Jeffrey T Eaton) Date: Wed, 28 Jan 2009 20:11:56 -0500 Subject: FastMail.FM cyrus patches for review In-Reply-To: <1233116140-4815-1-git-send-email-brong@fastmail.fm> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> Message-ID: > So - CMU people. Any plan to replace your aging CVS install with > something modern? It's on Ken's list to look at replacing our CVS and Bugzilla installations with something better. Suggestions are welcome. -jeaton From brong at fastmail.fm Wed Jan 28 22:51:18 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 29 Jan 2009 14:51:18 +1100 Subject: FastMail.FM cyrus patches for review In-Reply-To: References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> Message-ID: <20090129035118.GA12102@brong.net> On Wed, Jan 28, 2009 at 08:11:56PM -0500, Jeffrey T Eaton wrote: > > So - CMU people. Any plan to replace your aging CVS install with > > something modern? > > It's on Ken's list to look at replacing our CVS and Bugzilla > installations with something better. > > Suggestions are welcome. Most people seem to be liking Trac, but then it doesn't integrate so shinily with git. http://trac-hacks.org/wiki/TracGitPlugin though there are happy users out there: http://hiddenreflex.com/main/?p=28 Bron ( will keep looking ) From alexey.melnikov at isode.com Thu Jan 29 04:51:33 2009 From: alexey.melnikov at isode.com (Alexey Melnikov) Date: Thu, 29 Jan 2009 09:51:33 +0000 Subject: FastMail.FM cyrus patches for review In-Reply-To: <20090129035118.GA12102@brong.net> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <20090129035118.GA12102@brong.net> Message-ID: <49817C25.4020408@isode.com> Bron Gondwana wrote: >On Wed, Jan 28, 2009 at 08:11:56PM -0500, Jeffrey T Eaton wrote: > > >>>So - CMU people. Any plan to replace your aging CVS install with >>>something modern? >>> >>> >>It's on Ken's list to look at replacing our CVS and Bugzilla >>installations with something better. >> >>Suggestions are welcome. >> >> >Most people seem to be liking Trac, but then it doesn't integrate >so shinily with git. > > Maybe I am ignorant, but I am not sure what Trac gives that Bugzilla doesn't, apart from allowing for Wiki syntax (which I find quite annoying when one wants to paste some code or configuration examples). So I am "-1" on moving from Bugzilla. From brong at fastmail.fm Thu Jan 29 07:12:56 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Thu, 29 Jan 2009 23:12:56 +1100 Subject: FastMail.FM cyrus patches for review In-Reply-To: <49817C25.4020408@isode.com> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <20090129035118.GA12102@brong.net> <49817C25.4020408@isode.com> Message-ID: <20090129121256.GA15206@brong.net> On Thu, Jan 29, 2009 at 09:51:33AM +0000, Alexey Melnikov wrote: > Bron Gondwana wrote: > >> On Wed, Jan 28, 2009 at 08:11:56PM -0500, Jeffrey T Eaton wrote: >> >> Most people seem to be liking Trac, but then it doesn't integrate >> so shinily with git. >> >> > Maybe I am ignorant, but I am not sure what Trac gives that Bugzilla > doesn't, apart from allowing for Wiki syntax (which I find quite > annoying when one wants to paste some code or configuration examples). > So I am "-1" on moving from Bugzilla. Fair enough. I haven't researched it deeply. Just about anything over CVS is more what I care about. Bron. From ffarid at pragmatic-source.com Thu Jan 29 11:46:18 2009 From: ffarid at pragmatic-source.com (Farzad FARID) Date: Thu, 29 Jan 2009 17:46:18 +0100 Subject: FastMail.FM cyrus patches for review In-Reply-To: <20090129035118.GA12102@brong.net> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <20090129035118.GA12102@brong.net> Message-ID: <4981DD5A.1040001@pragmatic-source.com> Hi, Le 29.01.2009 04:51, Bron Gondwana a ?crit : > On Wed, Jan 28, 2009 at 08:11:56PM -0500, Jeffrey T Eaton wrote: > >> It's on Ken's list to look at replacing our CVS and Bugzilla >> installations with something better. >> >> Suggestions are welcome. >> > > Most people seem to be liking Trac, but then it doesn't integrate > so shinily with git. > In the Trac-like category (i.e. integrated versatile tools but with a less powerful bug tracker than Bugzilla), there is the increasingly popular Redmine (http://www.redmine.org/), which supports both Git and SVN natively. I, too, second the use of a modern VCS for Cyrus Imap whether it's a distributed one or not. But let's not forget that the available web interfaces count a lot for the ease of use and popularity of the VCSs. For example, I've been following Bron's copy of the Cyrus Imap repository on GitHub, http://github.com/brong/cyrus-imapd, and I think it clearly outperforms any other CVS or SVN web viewer available :) In the same category there is BitBucket (http://bitbucket.org/) for the Mercurial DVCS, but it's basically a GitHub clone. Regards, -- Farzad FARID / Architecte Open Source - Associ? Pragmatic Source / http://www.pragmatic-source.com Tel : +33 9 53 19 21 90 / Mob : +33 6 03 70 65 46 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20090129/e51a8381/attachment-0001.html From dimma at higis.ru Thu Jan 29 12:57:55 2009 From: dimma at higis.ru (Dmitriy Kirhlarov) Date: Thu, 29 Jan 2009 20:57:55 +0300 Subject: FastMail.FM cyrus patches for review In-Reply-To: <1233116140-4815-1-git-send-email-brong@fastmail.fm> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> Message-ID: <4981EE23.6050203@higis.ru> Hi. Bron Gondwana wrote: > Here are all the FastMail patches that I think are candidates > for upstream. Some time ago, we are discuss improvement of cyrus imapd logging: http://lists.andrew.cmu.edu/pipermail/info-cyrus/2008-April/028951.html Any plans for implement this functionality? PS. We are talks about functionality like http://johannes.sipsolutions.net/Projects/dovecot-dspam-integration WBR Dmitriy From wes at umich.edu Thu Jan 29 14:49:24 2009 From: wes at umich.edu (Wesley Craig) Date: Thu, 29 Jan 2009 14:49:24 -0500 Subject: version control In-Reply-To: <4981DD5A.1040001@pragmatic-source.com> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <20090129035118.GA12102@brong.net> <4981DD5A.1040001@pragmatic-source.com> Message-ID: <35A5FD31-238E-4A99-9CE8-9860411A5EE1@umich.edu> On 29 Jan 2009, at 11:46, Farzad FARID wrote: > I, too, second the use of a modern VCS for Cyrus Imap whether it's > a distributed one or not. But let's not forget that the available > web interfaces count a lot for the ease of use and popularity of > the VCSs. I see no up side to moving from CVS to SVN. The only thing I don't like about CVS is that I have to contact a central hub to do most of what I use CVS for, e.g., diff. Assuming Cyrus moves away from CVS, it should move to something distributed, e.g., git. I agree that a functional web interface is a requirement. :wes From wes at umich.edu Thu Jan 29 15:02:48 2009 From: wes at umich.edu (Wesley Craig) Date: Thu, 29 Jan 2009 15:02:48 -0500 Subject: DSPAM training integration In-Reply-To: <4981EE23.6050203@higis.ru> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <4981EE23.6050203@higis.ru> Message-ID: <77C55422-A6B9-4CF6-AC78-DCCD986C7FBC@umich.edu> I have no plans to implement it, at least not at this time. If I were designing such a system, I'd problem want a configuration that "watched" a special folder, and if refiling actions too place on that folder, logged (or perhaps took an action on) a header. If you're interested in writing something like that, propose it on the dev list (this one), get comments from people, and get a cyrus committer (perhaps me) to agree to work with you to get it committed. :wes On 29 Jan 2009, at 12:57, Dmitriy Kirhlarov wrote: > Some time ago, we are discuss improvement of cyrus imapd logging: > http://lists.andrew.cmu.edu/pipermail/info-cyrus/2008-April/ > 028951.html > > Any plans for implement this functionality? > > PS. We are talks about functionality like > http://johannes.sipsolutions.net/Projects/dovecot-dspam-integration From ghibo at mandriva.com Thu Jan 29 16:50:27 2009 From: ghibo at mandriva.com (=?ISO-8859-1?Q?Giuseppe_Ghib=F2?=) Date: Thu, 29 Jan 2009 22:50:27 +0100 Subject: DSPAM training integration In-Reply-To: <77C55422-A6B9-4CF6-AC78-DCCD986C7FBC@umich.edu> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <4981EE23.6050203@higis.ru> <77C55422-A6B9-4CF6-AC78-DCCD986C7FBC@umich.edu> Message-ID: <498224A3.30009@mandriva.com> Wesley Craig wrote: > I have no plans to implement it, at least not at this time. If I were > designing such a system, I'd problem want a configuration that > "watched" a special folder, and if refiling actions too place on that > folder, logged (or perhaps took an action on) a header. If you're > interested in writing something like that, propose it on the dev list > (this one), get comments from people, and get a cyrus committer > (perhaps me) to agree to work with you to get it committed. > > :wes in many situations cyrus-imapd is used in configuration with postfix as MTA, and amavis-new (http://www.ijs.si/software/amavisd/) which uses also to "spamassassin", "clamav" and IIRC there should be the possibility to add support for dspam (although amavisd-new should not handle it as good as SA). Bye Giuseppe. > > On 29 Jan 2009, at 12:57, Dmitriy Kirhlarov wrote: >> Some time ago, we are discuss improvement of cyrus imapd logging: >> http://lists.andrew.cmu.edu/pipermail/info-cyrus/2008-April/028951.html >> >> Any plans for implement this functionality? >> >> PS. We are talks about functionality like >> http://johannes.sipsolutions.net/Projects/dovecot-dspam-integration > From wes at umich.edu Thu Jan 29 17:10:20 2009 From: wes at umich.edu (Wesley Craig) Date: Thu, 29 Jan 2009 17:10:20 -0500 Subject: DSPAM training integration In-Reply-To: <498224A3.30009@mandriva.com> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <4981EE23.6050203@higis.ru> <77C55422-A6B9-4CF6-AC78-DCCD986C7FBC@umich.edu> <498224A3.30009@mandriva.com> Message-ID: <203E3B2C-8F49-4346-863A-AFB7A06DBBD3@umich.edu> On 29 Jan 2009, at 16:50, Giuseppe Ghib? wrote: > in many situations cyrus-imapd is used in configuration with > postfix as > MTA, and amavis-new > (http://www.ijs.si/software/amavisd/) which uses also to > "spamassassin", > "clamav" > and IIRC there should be the possibility to add support for dspam > (although amavisd-new should not > handle it as good as SA). The point of a patch like Dmitriy proposes is for training DSPAM through user actions in the MUA. For instance, logging (or otherwise handling) the contents of the X-DSPAM-Signature when the user refiles messages to or from their Spam folder. :wes From robm at fastmail.fm Thu Jan 29 18:50:36 2009 From: robm at fastmail.fm (Rob Mueller) Date: Fri, 30 Jan 2009 10:50:36 +1100 Subject: cyrus parsing email addresses problem? Message-ID: I've just come across a user reporting a bug with an email. They have a From header that looks like: From: =?iso-8859-1?Q?SOMETHING:other_text_here?= This gets parsed into the envelope: ((NIL NIL "=?iso-8859-1?Q?SOMETHING" NIL)("other_text_here?=" NIL "email" "address.com")(NIL NIL NIL NIL)) What's happening is it's treating the SOMETHING: as a email "group" (see section 3.4 of RFC 2822 - http://www.faqs.org/rfcs/rfc2822.html) Looking at RFC 1522 section 5 (3), it replaces the definition of phrase. Lets put the relevant parts of RFC2822 + RFC1522 together: address = mailbox / group mailbox = name-addr / addr-spec name-addr = [display-name] angle-addr group = display-name ":" [mailbox-list / CFWS] ";" [CFWS] display-name = phrase phrase = 1*(encoded-word / word) encoded-word = "=?" charset "?" encoding "?" encoded-text "?=" encoded-text = 1* word = atom / quoted-string atom = [CFWS] 1*atext [CFWS] atext = ALPHA / DIGIT / !#$%&'*+-/=?^_`{|}~ Applying all that to: =?iso-8859-1?Q?SOMETHING:other_text_here?= Is "almost" ambiguous. The one thing that means this should be interpreted as a "mailbox" rather than a "group" is that there's no ending ";" in there, so the "group" pattern shouldn't match. Wow, this is a pain to parse, because basically you have to potentially look way forward (to if there is an ending ;) to decide on how to interpret this :( A "hack" that would probably handle this better would be to alter parseaddr_phrase(inp, phrasep, specials) to be aware of RFC1522 type encoding, so that when you're in a =?....?= sequence, we ignore the "specials" parameter character list. Anyone have any thoughts? Rob From brong at fastmail.fm Thu Jan 29 21:29:32 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 30 Jan 2009 13:29:32 +1100 Subject: version control In-Reply-To: <35A5FD31-238E-4A99-9CE8-9860411A5EE1@umich.edu> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <20090129035118.GA12102@brong.net> <4981DD5A.1040001@pragmatic-source.com> <35A5FD31-238E-4A99-9CE8-9860411A5EE1@umich.edu> Message-ID: <20090130022932.GA11552@brong.net> On Thu, Jan 29, 2009 at 02:49:24PM -0500, Wesley Craig wrote: > On 29 Jan 2009, at 11:46, Farzad FARID wrote: >> I, too, second the use of a modern VCS for Cyrus Imap whether it's a >> distributed one or not. But let's not forget that the available web >> interfaces count a lot for the ease of use and popularity of the VCSs. > > I see no up side to moving from CVS to SVN. The only thing I don't like > about CVS is that I have to contact a central hub to do most of what I > use CVS for, e.g., diff. Assuming Cyrus moves away from CVS, it should > move to something distributed, e.g., git. I agree that a functional web > interface is a requirement. The only saving grace of svn is that git-svn is so good :) That and it's already on andrew machines. Not a big point in its favour. Yes, github rocks. I suspect CMU probably wants to host the canonical repository themselves though! I'll have a look at redmine. I've heard of it, but not looked closely. Needs ruby: -bash-3.00$ ruby --version ruby 1.8.7 (2008-06-20 patchlevel 22) [i686-linux] Well, Andrew has a not too insanely old ruby :) It's something else for the sysadmins to understand and maintain of course. Bron ( though the nice thing with git is that every repository is equal, so your work is automatically backed up in the preferred format everywhere ) From sebastian.maus at bluemars.net Fri Jan 30 04:24:45 2009 From: sebastian.maus at bluemars.net (Sebastian Maus) Date: Fri, 30 Jan 2009 10:24:45 +0100 Subject: DSPAM training integration In-Reply-To: <203E3B2C-8F49-4346-863A-AFB7A06DBBD3@umich.edu> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <4981EE23.6050203@higis.ru> <77C55422-A6B9-4CF6-AC78-DCCD986C7FBC@umich.edu> <498224A3.30009@mandriva.com> <203E3B2C-8F49-4346-863A-AFB7A06DBBD3@umich.edu> Message-ID: Hi, --On January 29, 2009 17:10:20 -0500 Wesley Craig wrote: > On 29 Jan 2009, at 16:50, Giuseppe Ghib? wrote: [...] > The point of a patch like Dmitriy proposes is for training DSPAM through > user actions in the MUA. For instance, logging (or otherwise handling) > the contents of the X-DSPAM-Signature when the user refiles messages to > or from their Spam folder. I don't know DSPAM, so this could be irrelevant information, but we did something like that for training personal SpamAssassin Bayes databases. Each customer has three SPAM folders (Filtered, LearnSPAM, LearnHAM). If mail gets dropped into one of the "Learn*" folders, a cron job (which is in fact a self written, customized IMAP client) finds them, feeds them to a sa-learn process that uses the customer's Bayes database and finally deletes and expunges the folder's content. The inbound MTA's SpamAssassin process uses exactly that database. The only thing that is potentially problematic about this setup is that "LearnHAM" mail is removed from the folder and thus has to be *copied* in the first place to avoid data loss. The big plus of this is that the personal taste of what is SPAM and what not can be learnt by the email system. The described setup is operational here since mid 2006. HTH, -S- -- BlueMars - Gesellschaft f?r digitale Kommunikation mbH Sebastian Maus mailto:sebastian.maus at bluemars.net System Administrator http://www.bluemars.net/ Ebersheimstra?e 5 T +49.69.469973-0 60320 Frankfurt Main F +49.69.469973-99 Gesch?ftsf?hrer: Tobias Kirchhofer, Frank Auth Registergericht: Amtsgericht Frankfurt/Main, HRB 44165 From Duncan.Gibb at SiriusIT.co.uk Fri Jan 30 05:11:31 2009 From: Duncan.Gibb at SiriusIT.co.uk (Duncan Gibb) Date: Fri, 30 Jan 2009 10:11:31 +0000 Subject: DSPAM training integration In-Reply-To: References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <4981EE23.6050203@higis.ru> <77C55422-A6B9-4CF6-AC78-DCCD986C7FBC@umich.edu> <498224A3.30009@mandriva.com> <203E3B2C-8F49-4346-863A-AFB7A06DBBD3@umich.edu> Message-ID: <4982D253.30100@SiriusIT.co.uk> Sebastian Maus wrote: WC> The point of a patch like Dmitriy proposes is for training DSPAM through WC> user actions in the MUA. For instance, logging (or otherwise handling) WC> the contents of the X-DSPAM-Signature when the user refiles messages to WC> or from their Spam folder. SM> we did something like that for training personal SpamAssassin Bayes SM> databases. Each customer has three SPAM folders (Filtered, SM> LearnSPAM, LearnHAM). If mail gets dropped into one of the "Learn*" SM> folders, a cron job (which is in fact a self written, customized SM> IMAP client) finds them, feeds them to a sa-learn process that SM> uses the customer's Bayes database and finally deletes and expunges SM> the folder's content. Our typical solution is similar, but using a long-running perl script to look up IMAP servers and the users expected on each from LDAP, then connect with Mail::IMAPClient as the user (authz as an admin agent). It reads user-filed false negatives from a learn-spam folder, runs them through a learning command (typically SA's "spamc -L"), and drops them in a spam folder (where sieve would have put them if the MTA had got it right the first time). False positives are read from a different folder and moved back into the user's INBOX once they've been learnt. The folder structure and learning command are configurable but currently have to the the same across a particular installation. I think what Giuseppe is asking for is a mechanism whereby the user's action triggers the learning command directly, rather than relying on an external agent (eg your cron job or our daemon) to come round and look what the user has done. That would save a fair amount of CPU resource in large installations - and be more responsive from the user's point of view. An SME deployment would typically have a daemon cycle time around all mailboxes of several minutes - longer if users have filed lots of messages to learn. In a large deployment agent-based learning would be slower (or need multiple instances running over different servers or groups of users). Cheers Duncan -- Duncan Gibb, Technical Director Sirius Corporation plc - The Open Source Experts http://www.siriusit.co.uk/ Tel: +44 870 608 0063 From Rudy.Gevaert at UGent.be Fri Jan 30 05:00:00 2009 From: Rudy.Gevaert at UGent.be (Rudy Gevaert) Date: Fri, 30 Jan 2009 11:00:00 +0100 Subject: (offtopic) Re: DSPAM training integration In-Reply-To: References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <4981EE23.6050203@higis.ru> <77C55422-A6B9-4CF6-AC78-DCCD986C7FBC@umich.edu> <498224A3.30009@mandriva.com> <203E3B2C-8F49-4346-863A-AFB7A06DBBD3@umich.edu> Message-ID: <4982CFA0.8020509@UGent.be> Sebastian Maus wrote: > Each customer has three SPAM folders (Filtered, LearnSPAM, LearnHAM). If > mail gets dropped into one of the "Learn*" folders, a cron job (which is in My problem with such solutions is that you are dependant on the user. How many users take the effort to use this system? I prefer a system where as much spam as possible is stopped on the first line, without interaction of me, nor the user. Rudy PS I'm still looking for such a system :) -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Rudy Gevaert Rudy.Gevaert at UGent.be tel:+32 9 264 4734 Directie ICT, afd. Infrastructuur ICT Department, Infrastructure office Groep Systemen Systems group Universiteit Gent Ghent University Krijgslaan 281, gebouw S9, 9000 Gent, Belgie www.UGent.be -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- From sebastian.maus at bluemars.net Fri Jan 30 06:01:55 2009 From: sebastian.maus at bluemars.net (Sebastian Maus) Date: Fri, 30 Jan 2009 12:01:55 +0100 Subject: (offtopic) Re: DSPAM training integration In-Reply-To: <4982CFA0.8020509@UGent.be> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <4981EE23.6050203@higis.ru> <77C55422-A6B9-4CF6-AC78-DCCD986C7FBC@umich.edu> <498224A3.30009@mandriva.com> <203E3B2C-8F49-4346-863A-AFB7A06DBBD3@umich.edu> <4982CFA0.8020509@UGent.be> Message-ID: --On January 30, 2009 11:00:00 +0100 Rudy Gevaert wrote: > Sebastian Maus wrote: >> Each customer has three SPAM folders (Filtered, LearnSPAM, LearnHAM). If >> mail gets dropped into one of the "Learn*" folders, a cron job (which is >> in > My problem with such solutions is that you are dependant on the user. SpamAssassin is quite effective without any training, so user interaction is not neccessary. It just helps a bit to train the thing. > How many users take the effort to use this system? Everyone feeling pain sooner or later notices those folders. And of course they're mentioned in a leaflet they receive when they first use their mail accounts. > I prefer a system where as much spam as possible is stopped on the first > line, without interaction of me, nor the user. Stopping SPAM (i.e.: mail!) may be an illegal action, depending on where you live and what kind of system you run (corporate vs. private). > PS I'm still looking for such a system :) Tell me if you find one :-) Sebastian -- From wes at umich.edu Fri Jan 30 15:20:35 2009 From: wes at umich.edu (Wesley Craig) Date: Fri, 30 Jan 2009 15:20:35 -0500 Subject: DSPAM training integration In-Reply-To: <4982D253.30100@SiriusIT.co.uk> References: <1233116140-4815-1-git-send-email-brong@fastmail.fm> <4981EE23.6050203@higis.ru> <77C55422-A6B9-4CF6-AC78-DCCD986C7FBC@umich.edu> <498224A3.30009@mandriva.com> <203E3B2C-8F49-4346-863A-AFB7A06DBBD3@umich.edu> <4982D253.30100@SiriusIT.co.uk> Message-ID: <4045ACEA-A5D3-4C08-AF13-B7C084FDC131@umich.edu> On 30 Jan 2009, at 05:11, Duncan Gibb wrote: > I think what Giuseppe is asking for is a mechanism whereby the user's > action triggers the learning command directly, rather than relying > on an > external agent (eg your cron job or our daemon) to come round and look > what the user has done. One of the interesting aspects of DSPAM is that only the signature of the message is required for training. So the user interaction can be very simple, e.g., only one folder. User action to copy mail into & out of the folder logs the signature (or optionally acts on the signature). There's no needs for a "LearnSPAM" or "LearnHAM" folder. :wes From brong at fastmail.fm Sat Jan 31 09:47:37 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sun, 1 Feb 2009 01:47:37 +1100 Subject: Man, you've got to love edge cases Message-ID: <20090131144737.GA17785@brong.net> (it's 1:38 am, yikes) So I've just been trying for hours to recreate this really rare crash we get. It only occurs with messages that are delivered to users with a sieve script that has a regex match on the body, so we get one every few months. We got one today. It's been causing a segfault every hour or so as postfix tries to deliver it again, and I've been trying to recreate it. Got my test machine almost exactly identical, no luck. Finally I realised I had a brand new slot on the same machine, which had no users in it yet. I could test in there. Tried, still nothing. Diffed the file on disk after delivery against the failed files in the staging directory of the other slot - the _ONLY_ difference was that the connections came from a different machine. I copied my LMTP session code that ran the delivery over there and tried. Still delivered fine. Diff. The only difference was that the "LHLO" name hadn't been changed. I changed that. Segfault! Wow. So I got suspicious. Did some numbers. [brong at imap11 ~]$ wc 5138-1233372374-0 2033 28211 221184 5138-1233372374-0 [brong at imap11 ~]$ bc 221184/4096 54 Well, what do you know. A multiple of 4096. #ifdef ENABLE_REGEX static int octet_regex(const char *text, size_t tlen, const char *pat, void *rock __attribute__((unused))) { if (!text[tlen]) { /* NUL-terminated string */ return (!regexec((regex_t *) pat, text, 0, NULL, 0)); } else { /* regexec() requires a NUL-terminated string */ char *buf = (char *) xstrndup(text, tlen); int r = !regexec((regex_t *) pat, buf, 0, NULL, 0); free(buf); return r; } } #endif I'm annoyed I didn't notice this when I searched the code the first time... grr. A cute idea, checking if the string just happens to be null terminated, but it turns out that you have no guarantee that "const char *text" can be indexed past [tlen-1] without a segfault. In particular, if you're looking at the body of an mmaped file that happens to be an exact multiple of blocks in size, and hence has no slop! This is why I've failed to find this bloody bug before - because all my test cases just succeeded, I couldn't recreate it. One in 4096 emails, and only to a person with a body regex match. Actually, it's probably fewer than that many, since I bet that statistically most emails are in a smallish set of sizes that don't cluster near a multiple of 4096. --- Anyway, I see no way to guarantee that we can check if the string is NULL terminated, so I'm just going to force the else case. Bron. From brong at fastmail.fm Sat Jan 31 19:44:26 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sun, 1 Feb 2009 11:44:26 +1100 Subject: Man, you've got to love edge cases In-Reply-To: <20090131144737.GA17785@brong.net> References: <20090131144737.GA17785@brong.net> Message-ID: <20090201004426.GA26151@brong.net> On Sun, Feb 01, 2009 at 01:47:37AM +1100, Bron Gondwana wrote: > /* regexec() requires a NUL-terminated string */ > char *buf = (char *) xstrndup(text, tlen); > int r = !regexec((regex_t *) pat, buf, 0, NULL, 0); > free(buf); > return r; It occurs to me that this can be a pretty large lump of memory use if the message being delivered is big. What a pain. int pcre_exec(const pcre *code, const pcre_extra *extra, const char *subject, int length, int startoffset, int options, int *ovector, int ovecsize); Now - at FastMail we use PCRE, but we use the PCRE posix compatibility layer, which doesn't provide a length-based interface. int regexec(const regex_t *restrict preg, const char *restrict string, size_t nmatch, regmatch_t pmatch[restrict], int eflags); I think probably the best bet is to use conditional compilation here, so people using PCRE don't pay the price of duplicating the entire message body. Does anybody know if the "rx" library has a length based interface? AC_CHECK_HEADERS(pcreposix.h rxposix.h) Bron ( looks like rx is using a compatibility layer too ) From brong at fastmail.fm Sat Jan 31 19:53:48 2009 From: brong at fastmail.fm (Bron Gondwana) Date: Sun, 1 Feb 2009 11:53:48 +1100 Subject: Man, you've got to love edge cases In-Reply-To: <20090201004426.GA26151@brong.net> References: <20090131144737.GA17785@brong.net> <20090201004426.GA26151@brong.net> Message-ID: <20090201005348.GA27203@brong.net> On Sun, Feb 01, 2009 at 11:44:26AM +1100, Bron Gondwana wrote: > Does anybody know if the "rx" library has a length based interface? > > AC_CHECK_HEADERS(pcreposix.h rxposix.h) I see that rx has regncomp. Annoyingly: http://sources.redhat.com/ml/libc-alpha/2000-01/msg00160.html It's probably not worth agitating for, since it would be years before we could rely on it anyway. Better to just use PCRE like the rest of the world seems to :) Bron.