switch to cyrus murder (aggregator) feedback

Mon Sep 22 09:56:52 EDT 2014

Quoting Bron Gondwana <brong at fastmail.fm>:

>> Since the migration we discovered some small issues and some bugs.
>>
>> 1. usually Cyrus is not CPU bound. One exception is the mupdate master
>>     keeping encrypted connection to all frontends and establishing
>>     new encrypted connections from the backend for every mailbox creation,
>>     rename and remove, was too much for the 4 cores so we added 4 additional
>>     cores to the VMs.
>
> This sounds like another thing for the STARTTLS in case number 2...

the mupdate connection use STARTTLS as far as i know.

>
>> 2. Our frontend instances use IMAPs and POP3s and don't allow STARTTLS.
>>     But we hat to use IMAP and POP3 with STARTTLS on our backends, as
>>     the frontends will always use STARTTLS over IMAP and POP3 to proxy
>>     the connection.
>
> That kind of sucks.  It should be a configuration option to control whether
> frontends use STARTTLS.  I wonder if they're smart enough to not try it if
> the server doesn't advertise it?  You could use suppress_capabilities in
> that case.

As far as i have seen in the code the proxy only try the standard
imap and pop port so if i suspress starttls and use "allowplaintext: 0"
I guess it will fail to connect.

It was more a problem because we didn't test the setup with pop3.
We uses the imap on the backends for cyradm, so imap was working,
and that pop3 was not working was only dicovered after the migration.

I don't think STARTTLS instead of IMAPs/POPs will make a huge difference
in performance. The main argument against starttls for IMAP/POP3 on  
the frontend is, that there are some "stupid" clients which don't  
check the
capabilities and will send the plain password even if no auth mechs are
available.

>
>> 3. We see more IOERRORs in our cyrus logs. In the standalone
>>     cyrus imap IOERROR indicated a corruption in one of the cyrus files
>>     but that is not the case for the new errors we have found:
>>
>>     a) "reading message: unexpected end of file" as far as i can tell,
>>        this is triggert by the imap append command. I suspect when the
>>        connection between frontend and backend is lost or the frontend
>>        dies during upload of the message.
>
> Yeah, I'm not a fan of this.  It happens in non-murder too.
>
>>     b) "opening index %s: Invalid mailbox name" the mailbox name seem to
>>        be fine in most cases. I haven only figured out why the mailbox
>>        name was considered invalid in one case (the Sting "Posteingang"
>>        was translated by the client and the name "INBOX" ins reserved.
>
> If you give me some other names, I might be able to see why...

I will send the to you and not to the list, as some names may contain
non public informations.

>
>>     It would help if the String IOERROR would not be used in these cases,
>>     and if the mailbox name would always be logged consistent to the
>>     unixhierarchysep option.
>
> The mailbox names will not be changed.  The format you see is the  
> internal format, and it also includes domain!user. if you are domain  
> split.  If anything, I would change more of the tools to use that  
> format, because it's exact.  The algorithm to convert between them  
> is easy enough if you need to for display purposes, but for low  
> level debugging, exactness matters more.
>

I know the difference and how to convert, but it annoying to change "." to
"/" if I copy/past a mailbox name from the log to an command.
The Unix hierarchy seperator has the adwantage that it can be used
on the filesystem as well.

The other point is that it is used incosistend:
e.g. "squatter: indexing mailbox" and "unexpunge" "restored X expunged  
messages in mailbox" use both unix hierarchy sep.

>> 4. Deleting an mailbox with delete_mode: delayed can create a corrupt
>>     mailbox in the DELETED tree. In the logs we found the following:
>>
>>     be/beimap[62020]: Rename: user.LoginID.Mail.drafts ->
>> DELETED.user.LoginID.Mail.drafts.5416CD11
>>
>>     be/beimap[62020]: MUPDATE: can't commit mailbox entry for
>> 'DELETED.user.LoginID.Mail.drafts.5416CD11'
>>     be/beimap[62020]: Deleted mailbox  
>> DELETED.user.LoginID.Mail.drafts.5416CD11
>
> OOh - so that's a problem with murder specifically.  I wonder why it  
> can't be committed.
>
>>     and on the next cyr_expire run
>>
>>     be/cyr_expire[144388]: IOERROR: opening index
>> DELETED.user.LoginID.Mail.drafts.5416CD11: System I/O error
>
> Yeah, makes sense.  Maybe it's being created with the wrong type fields.
>
>>     in the filesystem DELETED/user/LoginID/Mail/drafts was an empty  
>> directory.
>>     I couldn't find any hints why the mupdate master couldn't commit the
>>     mailbox entry, but as "5416CD11" is the timestamp of the action, I am
>>     certain that the mailbox did not exist in the mailboxdb before. And as
>>     this only happens in some rare cases I suspect a race condition.
>
> Smells like it from your description.
>
>> 5. Some frontend imapd processes receive a SIGSEGV.
>>     As this seams to happen in the libopenssl I asked on their mailinglist,
>>     but didn't receive an answer jet. At the end you will fine an BT of the
>>     core dump.
>
> Ok...
>
>> I would be glad if changes regarding the logging of IOERRORs
>> and mailbox names would be included in Cyrus 2.5
>
> The IOERROR for append could certainly be changed.  It's tricky because it's
> deep in a library layer, but since it's the ONLY place I've ever  
> seen that error
> message, it could at least be changed to be more descriptive about probable
> cause.

>> Regarding 4. and 5. are these known bugs? I could not find any matching
>> entries in the bug tracker. If they are not know I would add them to
>> the bug tracker.
>
> I don't know about them.  Go ahead and add them, worst case we find  
> a duplicate
> and merge it.
>

for 4. https://bugzilla.cyrusimap.org/show_bug.cgi?id=3862
for 5. https://bugzilla.cyrusimap.org/show_bug.cgi?id=3863

--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universität Tübingen                   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung          mail:  
michael.menge at zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5425 bytes
Desc: S/MIME Signatur
Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20140922/d0cba767/attachment.bin 