From dembek at alcatel-lucent.com Wed Dec 3 08:00:49 2008 From: dembek at alcatel-lucent.com (DEMBEK, Adam (Adam)) Date: Wed, 3 Dec 2008 14:00:49 +0100 Subject: Cyrus replication performance improvement Message-ID: <79F10B7259440E468DB6875CC2AF6A7701EC2252@DEEXC1U02.de.lucent.com> Welcome We are working on improving performance of our Cyrus server replication. We are using Cyrus 2.3.7. When installed our software on server with slower 10 RPM disks we notice that one sync_client is not able to synchronize all data in real time. On servers with 15 RPM disk performance was better. We prepared few modifications that allow to run 3 sync_client that will replicate different mailboxes at the same time. We would like to get opinion of other Cyrus developers about these modifications to identify if they can introduce any additional risks (Details below). Our tests show 30 - 50 % improvement of Cyrus replication speed on server with 10 RPM disks. We tested that no data is lost during synchronization for CREATE, APPEND, STORE, RENAME and SETANNOTATION commands. Our changes: sync_log.c Write log to one of 3 files depending on mailbox name. The same mailbox is always sent to the same log file. sync_client.c Read from different sync/log file. Path to correct log file is configured in imapd.conf Each sync_client is run with different imapd.conf. cyrus.conf Start 3 syncservers on different ports syncserver cmd="/apsw/ms/cyrus/imapd/bin/sync_server -C /etc/imapd_s.conf" listen="50055" maxchild=1 syncserver1 cmd="/apsw/ms/cyrus/imapd/bin/sync_server -C /etc/imapd_s1conf" listen="50056" maxchild=1 syncserver2 cmd="/apsw/ms/cyrus/imapd/bin/sync_server -C /etc/imapd_s2.conf" listen="50057" maxchild=1 Code changes in sync_log.c void sync_log_mailbox(char *name) { int sync_file_id = getFileId(name + 5); sync_log(sync_file_id, "MAILBOX %s\n", name); } int getFileId(const char *name) { if (multi_sync_log > 1) { unsigned int result = 0; int pos = 0; for (; name[pos] != 0; ++pos) { result += name[pos]; } return (result % multi_sync_log); } return 0; } static void sync_log_base(int sync_file_id, const char *string, int len) { int fd, rc; struct stat sbuffile, sbuffd; int retries = 0; char *file_name; if (!sync_log_enabled) return; if (sync_file_id == 0) { file_name = sync_log_file; } else if (sync_file_id == 1) { file_name = sync_log_file1; } else if (sync_file_id == 2) { file_name = sync_log_file2; } while (retries++ < SYNC_LOG_RETRIES) { fd = open(file_name, O_WRONLY|O_APPEND|O_CREAT, 0640); if (fd < 0 && errno == ENOENT) { if (!cyrus_mkdir(file_name, 0755)) { fd = open(file_name, O_WRONLY|O_APPEND|O_CREAT, 0640); } } if (fd < 0) { syslog(LOG_ERR, "sync_log(): Unable to write to log file %s: %s", file_name, strerror(errno)); return; } if (lock_blocking(fd) == -1) { syslog(LOG_ERR, "sync_log(): Failed to lock %s for %s: %m", file_name, string); close(fd); return; } ... } Regards, Adam Dembek Messaging Applications Poland phone: +48 52 349 1908 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20081203/834f15f5/attachment.html From zablocki at alcatel-lucent.com Thu Dec 4 05:57:02 2008 From: zablocki at alcatel-lucent.com (ZABLOCKI, T (T)) Date: Thu, 4 Dec 2008 11:57:02 +0100 Subject: Cyrus replication leak Message-ID: <4EF0F84A5B4D564386DBD8A3DFA1C54F018AD80D@DEEXC1U01.de.lucent.com> > Hello, > > I found leak of file descriptors in cyrus code in case of > communication errors. > I the log file is very big and you kill sync_server you can turn into > case where there are no more file descriptors for sync_client. > Fix: > sync_client.c > > 2574 r = do_user_parse(user, server_list, server_sub_list, > server_sieve_list); > 2575 > 2576 if (r) { > 2577 sync_folder_list_free(&server_list); > 2578 sync_folder_list_free(&server_sub_list); > <<-- 1 LINES ADDED -->> > 2579 if (mailbox_open) mailbox_close(&m); > 2580 return(r); > 2581 } > > Regards, > Tomasz Zablocki > Alcatel-Lucent > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20081204/46f6c6b5/attachment.html From brong at fastmail.fm Thu Dec 4 22:14:27 2008 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 05 Dec 2008 14:14:27 +1100 Subject: Cache truncation bug on aborted appends Message-ID: <1228446867.20912.1288375755@webmail.messagingengine.com> Hi all, Cyrus stores the end of the cache file before starting an append operation so that it can truncate back to that point if the append is aborted. Unfortunately, it actually stores cache_len rather than cache_size. That sort of sucks, because cache_len is rounded up by quite a bit to allow "slop". As the cache file gets bigger, the slop gets bigger too, and you wind up with a whole pile of zero blocks in your cache file, making it (even if sparse) somewhat massive. Oh, and the bogus record(s) that you wrote are going to still be (possibly only partially) in the file, because the truncate will either be past them, or in the middle of them. This is exacerbated by the fact that duplicate suppression seems to need to write to the cache file _before_ it decides not to accept the message! The attached patch fixes the issue, adds a comment, and renames the temporary variable to reflect the value it's actually storing. Wes/Ken, please apply to CVS for the next stable release. Everyone else, I recommend you apply this patch. We have had to reconstruct the occasional mailbox as their cache file size spirals out of control and the process hits memory limits trying to map it. Bron. -- Bron Gondwana brong at fastmail.fm -------------- next part -------------- A non-text attachment was scrubbed... Name: cyrus-truncate-length-2.3.13.diff Type: text/x-patch Size: 1353 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20081205/c17666ab/attachment.bin From brong at fastmail.fm Fri Dec 5 04:44:22 2008 From: brong at fastmail.fm (Bron Gondwana) Date: Fri, 05 Dec 2008 20:44:22 +1100 Subject: Cache truncation bug on aborted appends In-Reply-To: <54a13dbc47935c472fe50efb9678ed7f.squirrel@webmail.bi.corp.invoca.ch> References: <1228446867.20912.1288375755@webmail.messagingengine.com> <54a13dbc47935c472fe50efb9678ed7f.squirrel@webmail.bi.corp.invoca.ch> Message-ID: <1228470262.3091.1288413187@webmail.messagingengine.com> On Fri, 5 Dec 2008 08:25:38 +0100 (CET), "Simon Matter" wrote: > > Hi all, > > > > Cyrus stores the end of the cache file before starting an > > append operation so that it can truncate back to that point > > if the append is aborted. > > > > Unfortunately, it actually stores cache_len rather than > > cache_size. That sort of sucks, because cache_len is > > rounded up by quite a bit to allow "slop". As the cache > > file gets bigger, the slop gets bigger too, and you wind > > up with a whole pile of zero blocks in your cache file, > > making it (even if sparse) somewhat massive. > > > > Oh, and the bogus record(s) that you wrote are going to > > still be (possibly only partially) in the file, because > > the truncate will either be past them, or in the middle > > of them. > > > > This is exacerbated by the fact that duplicate suppression > > seems to need to write to the cache file _before_ it decides > > not to accept the message! > > > > The attached patch fixes the issue, adds a comment, and > > Hi Bron, > > is there something missing in the patch, because I can't see the "adds a > comment" part? Hmm... could be. Damn. I'll just go back and have a look. I did the comment after the rest of it - it's pretty meaningless anyway. The code in the patch is certainly fine... Ok - here's the copy with the comment! Bron. -- Bron Gondwana brong at fastmail.fm -------------- next part -------------- A non-text attachment was scrubbed... Name: cyrus-truncate-length-2.3.13.diff Type: text/x-patch Size: 1444 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20081205/fe3d7a16/attachment.bin From brong at fastmail.fm Tue Dec 9 18:36:56 2008 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 10 Dec 2008 10:36:56 +1100 Subject: Another cache bug! Message-ID: <1228865816.22867.1289197119@webmail.messagingengine.com> Wow, this is the thanks I get for doing sanity checks on files, find more bugs! This one is due to delayed expunge, plain and simple. Cyrus decides what cache records to copy during an IMAP COPY command by reading the cache offsets for msgno and msgno+1 (or the end of the cache file if it's the last msgno). Obviously if some intervening messages have already been deleted from the cyrus.index then it will be copying all those cache records as well to the destination folder. Oops. The attached patch reworks mailbox_cache_size into two functions, the second being mailbox_cache_size_detail that takes memory locations for the cache mmap rather than a mailbox object (because imapd doesn't update the locations in the object correctly according to my testing, *sigh*. Gotta love global variables) It then uses mailbox_cache_size_detail to calculate the ACTUAL record length for this single cache item rather than blindly copying everything up to the next index record's pointer. Also note: in the event of cache corruption, mailbox_cache_size_detail returns zero bytes, which correctly makes append_copy re-parse the message file. It's all shiny :) Wes/Ken - please apply to CVS :) Thanks, Bron. -- Bron Gondwana brong at fastmail.fm -------------- next part -------------- A non-text attachment was scrubbed... Name: cyrus-copy-cachelen-2.3.13.diff Type: text/x-patch Size: 3530 bytes Desc: not available Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20081210/7e67bc3c/attachment.bin From murch at andrew.cmu.edu Sun Dec 14 12:30:51 2008 From: murch at andrew.cmu.edu (Ken Murchison) Date: Sun, 14 Dec 2008 12:30:51 -0500 Subject: ManageSieve problems Message-ID: <494542CB.5030800@andrew.cmu.edu> Can I get some feedback on the patch that I attached to bug #3098? -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University From ghibo at mandriva.com Tue Dec 16 11:11:10 2008 From: ghibo at mandriva.com (=?ISO-8859-15?Q?Giuseppe_Ghib=F2?=) Date: Tue, 16 Dec 2008 17:11:10 +0100 Subject: NUL character Message-ID: <4947D31E.50808@mandriva.com> Hi, I've noticed recently when I was setting up a new cyrus server (in particular 2.3.12) bundled with postfix that cyrus sometimes returns a 554 error saying host blabla[cyrus_socket/lmtp] said: 554 5.6.0 Message contains NUL characters (in reply to end of DATA command) I remember that in the past releases there was something similar (in the past the NUL character was generated by cyrus when large lines [> 8191 chars] were occurring, but IIRC that was fixed). In this case I found the NUL character was in a end part of a mail contaning a base64 section (and not lines longer than 8191 characters). As result of this error, postfix send a bounce (but usually the mail is a SPAM, so fake address). To avoid this is it possible for cyrus to avoid to generate this error, or ignoring or sanitizing the \0 or adding a config option for doing so? Thanks Bye Giuseppe. From hdembkowski at alcatel-lucent.com Tue Dec 16 12:11:41 2008 From: hdembkowski at alcatel-lucent.com (DEMBKOWSKI, Henryk (Henryk)) Date: Tue, 16 Dec 2008 18:11:41 +0100 Subject: Problem with sync_client Message-ID: <9E30FE5A3FF4DA4AB0EC1F135A6F763B328378E776@FRMRSSXCHMBSB2.dc-m.alcatel-lucent.com> Hi, I am trying to get working replication between two message servers, both with new Cyrus 2.3.13. >From logs it seems that sync_client "hangs up". I have the following functions flows for sync_client: - main() calls replica_connect() to open connection to server - replica_connect() calls backend_connect() - backend_connect() calls ask_capability() since "banner is capability response" from protocol definition for "csync" static struct protocol_t csync_protocol = { "csync", "csync", { 1, "* OK" }, <========================== { NULL, NULL, "* OK", NULL, { { "* SASL ", CAPA_AUTH }, { "* STARTTLS", CAPA_STARTTLS }, { NULL, 0 } } }, { "STARTTLS", "OK", "NO", 0 }, { "AUTHENTICATE", INT_MAX, 0, "OK", "NO", "+ ", "*", NULL, 0 }, { "NOOP", NULL, "OK" }, { "EXIT", NULL, "OK" } }; Below code from backend_connect() - ask_capability() is called with automatic==AUTO_BANNER if (prot->banner.is_capa) { /* try to get the capabilities from the banner */ mechlist = ask_capability(ret->out, ret->in, prot, &ret->capability, AUTO_BANNER); if (mechlist || ret->capability) { /* found capabilities in banner -> don't ask */ ask = 0; } } - sync_server returns the following banner "OK ms04 Cyrus sync server v2.3.13" - ask_capability() tries to find capabilities (from protocol definition for "csync") in banner. So it tries to find "* SASL " or "* STARTTLS" in "* OK ms04 Cyrus sync server v2.3.13" And it fails. - therefore function backend_connect() calls once again ask_capability() - this time with automatic==AUTO_NO if (ask) { /* get the capabilities */ mechlist = ask_capability(ret->out, ret->in, prot, &ret->capability, AUTO_NO); } - however this time it seems that function ask_capability() "hangs up". It never ends. It tries to read stream but probably sync_server doesn't send anything. do { if (prot_fgets(str, sizeof(str), pin) == NULL) break; <================================ /* look for capabilities in the string */ for (c = prot->capa_cmd.capa; c->str; c++) { if ((tmp = strstr(str, c->str)) != NULL) { *capa = *capa | c->flag; if (c->flag == CAPA_AUTH) { if (prot->capa_cmd.parse_mechlist) ret = prot->capa_cmd.parse_mechlist(str, prot); else ret = xstrdup(tmp+strlen(c->str)); } } } if (!resp) { /* multiline response with no distinct end (IMAP banner) */ prot_NONBLOCK(pin); } /* look for the end of the capabilities */ } while (!resp || strncasecmp(str, resp, strlen(resp))); Do you know what can be wrong? Kind Regards, Henryk From lists at egidy.de Tue Dec 16 12:46:33 2008 From: lists at egidy.de (Gerd v. Egidy) Date: Tue, 16 Dec 2008 18:46:33 +0100 Subject: NUL character In-Reply-To: <4947D31E.50808@mandriva.com> References: <4947D31E.50808@mandriva.com> Message-ID: <200812161846.33493.lists@egidy.de> Hi, > As result of this error, postfix send a bounce (but usually the mail is > a SPAM, so fake address). > To avoid this is it possible for cyrus to avoid to generate this error, > or ignoring or sanitizing the \0 > or adding a config option for doing so? you can let postfix do this with either message_strip_characters = \0 or message_reject_characters = \0 so there is no need to change anything in cyrus. Kind regards, Gerd -- Address (better: trap) for people I really don't want to get mail from: jonas at cactusamerica.com From zablocki at alcatel-lucent.com Thu Dec 18 09:45:51 2008 From: zablocki at alcatel-lucent.com (ZABLOCKI, T (T)) Date: Thu, 18 Dec 2008 15:45:51 +0100 Subject: Cyrus.index problem References: <9E30FE5A3FF4DA4AB0EC1F135A6F763B328378EA05@FRMRSSXCHMBSB2.dc-m.alcatel-lucent.com> Message-ID: <729C6B00596C624FB7C156D7A6A1582D328387CF7D@FRMRSSXCHMBSB2.dc-m.alcatel-lucent.com> Guys, What I found in index.c: static int index_storeflag(struct mailbox *mailbox, unsigned msgno, void *rock) That check is made on particular mailbox: /* Check the modseq against unchangedsince */ if (MODSEQ(msgno) > storeargs->unchangedsince) return 0; It blocks update of flags. But why is that check here unconditional, I may do not have mod sequences enabled on my mailbox. Shouldn't it be: if (mailbox->options & OPT_IMAP_CONDSTORE) { if (MODSEQ(msgno) > storeargs->unchangedsince) return 0; } ?? * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID AUTH=ANONYMOUS SASL-IR] ms14 Cyrus IMAP4 v2.3.7 server ready 1 login 17431341000 1234 1 OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID LOGINDISABLED ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE CATENATE CONDSTORE IDLE URLAUTH] User logged in 2 select inbox * FLAGS (\Answered \Flagged \Draft \Deleted \Seen New QMboxSize QMsgCount SenderOffComplex Voice TUISkipped SenderOnComplex Accessed Expired AnyPathSeen) * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted New QMboxSize QMsgCount SenderOffComplex Voice TUISkipped SenderOnComplex Accessed Expired AnyPathSeen \*)] * 4 EXISTS * 4 RECENT * OK [UNSEEN 1] * OK [UIDVALIDITY 1185655835] * OK [UIDNEXT 664] * OK [NOMODSEQ] Sorry, modsequences have not been enabled on this mailbox I found some mailboxes which doesn't have modsequences but it has MODSEQ>unchangedsince and that check block update of any flag in that mailbox. Do you have an idea why I am having such sitiuation? BTW: I use cyrus 2.3.7 but this check is unconditional also in 2.3.13. Regards, Tomasz From brong at fastmail.fm Wed Dec 24 07:11:03 2008 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 24 Dec 2008 23:11:03 +1100 Subject: Git clone of Cyrus CVS Message-ID: <20081224121103.GA31379@brong.net> I've run up a clone of the Cyrus CVS on github for my own experimentation. I'll be fiddling around in a branch called brondev, and applying our patch series as a branch called fastmail. You can look around online at: http://github.com/brong/cyrus-imapd/ Or clone the whole repository and have a play yourself: git clone git://github.com/brong/cyrus-imapd.git Enjoy, Bron. From brong at fastmail.fm Wed Dec 24 07:38:21 2008 From: brong at fastmail.fm (Bron Gondwana) Date: Wed, 24 Dec 2008 23:38:21 +1100 Subject: Git clone of Cyrus CVS In-Reply-To: <20081224121103.GA31379@brong.net> References: <20081224121103.GA31379@brong.net> Message-ID: <20081224123821.GA4847@brong.net> On Wed, Dec 24, 2008 at 11:11:03PM +1100, Bron Gondwana wrote: > I've run up a clone of the Cyrus CVS on github for my own > experimentation. I'll be fiddling around in a branch called > brondev, and applying our patch series as a branch called > fastmail. You can look around online at: > > http://github.com/brong/cyrus-imapd/ > > Or clone the whole repository and have a play yourself: > > git clone git://github.com/brong/cyrus-imapd.git It appears we have patches that hit sieve as well, and for historical reasons that's in its own repository - cloned as cyrus-sieve, replace in URLs above at your convenience. Bron.