From egoitz at sarenet.es Tue Feb 5 05:57:59 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Tue, 05 Feb 2019 11:57:59 +0100 Subject: Conversations db in 3.0 Message-ID: <9c2c0256697b5650fb2bbdc19a483cc5@sarenet.es> Hi! In an upgrade proccesses to upper version you don't have a way revert back the update without loosing content. I was very interested in knowing if I could have a 2.4 master server (without Xapian, conversations and so obviously) replicating to a 3.0 server and if I could perform the : - reconstruct -V max - ctl_conversationsdb -z USER - ctl_conversationsdb -b USER but while the 3.0 is slave and while is replicating from 2.4. Is this possible? Thanks a lot mates, -- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From dilyan.palauzov at aegee.org Tue Feb 5 08:18:21 2019 From: dilyan.palauzov at aegee.org (=?UTF-8?Q?=D0=94=D0=B8=D0=BB=D1=8F=D0=BD_?= =?UTF-8?Q?=D0=9F=D0=B0=D0=BB=D0=B0=D1=83=D0=B7=D0=BE=D0=B2?=) Date: Tue, 05 Feb 2019 13:18:21 +0000 Subject: Conversations db in 3.0 In-Reply-To: <9c2c0256697b5650fb2bbdc19a483cc5@sarenet.es> References: <9c2c0256697b5650fb2bbdc19a483cc5@sarenet.es> Message-ID: <16b1f1a4f308bb80c50946777ff131b7779fa65d.camel@aegee.org> Hello Egoitz, you have been asked on 7th January by Nic not to post anymore the same message to both cyrus-devel at lists.andrew.cmu.edu and info-cyrus at lists.andrew.cmu.edu . Your current email was sent to both mailing lists. You do not do, what people ask you for, but you expect that people do, what you ask them for. After a new version of Cyrus is released people migrate to it within a year or two and then the knowledge about the concrete migration and its chaveats is lost. You will likely get more useful feedback, if you first migrate to 2.5 and then to 3.0 and ask question about this workflow. The software is supposed to do also more reliably when this workflow is followed. Doing anything else is a risk, I would not take. This workflow should also be documentated, as many people have done the version jumps in exactly this way. Regards ????? On Tue, 2019-02-05 at 11:57 +0100, Egoitz Aurrekoetxea wrote: > Hi! > > In an upgrade proccesses to upper version you don't have a way revert back the update without loosing content. I was very interested in knowing if I could have a 2.4 master server (without Xapian, conversations and so obviously) replicating to a 3.0 server and if I could perform the : > > - reconstruct -V max > - ctl_conversationsdb -z USER > - ctl_conversationsdb -b USER > > but while the 3.0 is slave and while is replicating from 2.4. > > Is this possible? > Thanks a lot mates, > -- > > Egoitz Aurrekoetxea > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es > > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. From egoitz at sarenet.es Fri Feb 8 08:15:44 2019 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Fri, 08 Feb 2019 14:15:44 +0100 Subject: Conversations db in 3.0 In-Reply-To: <16b1f1a4f308bb80c50946777ff131b7779fa65d.camel@aegee.org> References: <9c2c0256697b5650fb2bbdc19a483cc5@sarenet.es> <16b1f1a4f308bb80c50946777ff131b7779fa65d.camel@aegee.org> Message-ID: <91893e8d6c3da10b5e272a94b805c52e@sarenet.es> Thank you so much mate. Sorry for sending to both lists... was not my proposal (has been totally a mistake...). Apologize again, --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 05-02-2019 14:18, ????? ???????? escribi?: > Hello Egoitz, > > you have been asked on 7th January by Nic not to post anymore the same message to both cyrus-devel at lists.andrew.cmu.edu > and info-cyrus at lists.andrew.cmu.edu . Your current email was sent to both mailing lists. You do not do, what people > ask you for, but you expect that people do, what you ask them for. > > After a new version of Cyrus is released people migrate to it within a year or two and then the knowledge about the > concrete migration and its chaveats is lost. > > You will likely get more useful feedback, if you first migrate to 2.5 and then to 3.0 and ask question about this > workflow. The software is supposed to do also more reliably when this workflow is followed. Doing anything else is a > risk, I would not take. This workflow should also be documentated, as many people have done the version jumps in > exactly this way. > > Regards > ????? > > On Tue, 2019-02-05 at 11:57 +0100, Egoitz Aurrekoetxea wrote: > >> Hi! >> >> In an upgrade proccesses to upper version you don't have a way revert back the update without loosing content. I was very interested in knowing if I could have a 2.4 master server (without Xapian, conversations and so obviously) replicating to a 3.0 server and if I could perform the : >> >> - reconstruct -V max >> - ctl_conversationsdb -z USER >> - ctl_conversationsdb -b USER >> >> but while the 3.0 is slave and while is replicating from 2.4. >> >> Is this possible? >> Thanks a lot mates, >> -- >> >> Egoitz Aurrekoetxea >> Departamento de sistemas >> 944 209 470 >> Parque Tecnol?gico. Edificio 103 >> 48170 Zamudio (Bizkaia) >> egoitz at sarenet.es >> www.sarenet.es [1] >> >> Antes de imprimir este correo electr?nico piense si es necesario hacerlo. Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From max at tcen.ru Fri Feb 8 09:23:13 2019 From: max at tcen.ru (Max Kosmach) Date: Fri, 8 Feb 2019 17:23:13 +0300 Subject: Failed CUnit test in 2.5.12 and 3.0.8 Message-ID: <20190208142313.bscy7yxhrf4bmdqv@tcen.ru> Hi, cyrus imap devs! I'm trying to build 2.5.12 and 3.0.8 cyrus imapd and in both versions I have same problems with backend suite: Suite: backend Test: badhost ...passed Test: badservice ...passed Test: sasl_plain ...Server failed to find requested SASL mechanism "PLAIN" FAILED 1. ./cunit/backend.testc:198 - CU_ASSERT_PTR_NOT_NULL_FATAL(be) Test: sasl_digestmd5 ...Server failed to find requested SASL mechanism "DIGEST-MD5" FAILED 1. ./cunit/backend.testc:265 - CU_ASSERT_PTR_NOT_NULL_FATAL(be) Test: multiline_caps ...Server failed to find requested SASL mechanism "PLAIN" FAILED 1. ./cunit/backend.testc:314 - CU_ASSERT_PTR_NOT_NULL_FATAL(be) Test: oneline_caps ...Server failed to find requested SASL mechanism "PLAIN" FAILED 1. ./cunit/backend.testc:314 - CU_ASSERT_PTR_NOT_NULL_FATAL(be) Test: starttls ...FAILED 1. ./cunit/backend.testc:408 - CU_ASSERT_PTR_NOT_NULL_FATAL(be) Is this correct behavior and I can ignore this? PS. strace show thak this test search nonexistent testserver.conf file - [pid 21654] openat(AT_FDCWD, "/etc/sasl2/testserver.conf", O_RDONLY [pid 21654] openat(AT_FDCWD, "/etc/sasl/testserver.conf", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 21654] openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/sasl2/testserver.conf", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 21654] openat(AT_FDCWD, "/usr/lib/sasl2/testserver.conf", O_RDONLY) = -1 ENOENT (No such file or directory) From dilyan.palauzov at aegee.org Mon Feb 11 11:51:48 2019 From: dilyan.palauzov at aegee.org (=?UTF-8?Q?=D0=94=D0=B8=D0=BB=D1=8F=D0=BD_?= =?UTF-8?Q?=D0=9F=D0=B0=D0=BB=D0=B0=D1=83=D0=B7=D0=BE=D0=B2?=) Date: Mon, 11 Feb 2019 16:51:48 +0000 Subject: Failed CUnit test in 2.5.12 and 3.0.8 In-Reply-To: <20190208142313.bscy7yxhrf4bmdqv@tcen.ru> References: <20190208142313.bscy7yxhrf4bmdqv@tcen.ru> Message-ID: Hello Max, a possible explanation is the lack of libplain.so or libdigestmd5.so in $libdir/sasl2 , coming from cyrus-sasl . Until recently, the place where cyrus-sasl/configure && make install put those files, when compiled, was different from the place, where the files were searched at run time. Regards ????? On Fri, 2019-02-08 at 17:23 +0300, Max Kosmach wrote: > Hi, cyrus imap devs! > > I'm trying to build 2.5.12 and 3.0.8 cyrus imapd and in both versions I have same problems with backend suite: > > Suite: backend > Test: badhost ...passed > Test: badservice ...passed > Test: sasl_plain ...Server failed to find requested SASL mechanism "PLAIN" > FAILED > 1. ./cunit/backend.testc:198 - CU_ASSERT_PTR_NOT_NULL_FATAL(be) > Test: sasl_digestmd5 ...Server failed to find requested SASL mechanism "DIGEST-MD5" > FAILED > 1. ./cunit/backend.testc:265 - CU_ASSERT_PTR_NOT_NULL_FATAL(be) > Test: multiline_caps ...Server failed to find requested SASL mechanism "PLAIN" > FAILED > 1. ./cunit/backend.testc:314 - CU_ASSERT_PTR_NOT_NULL_FATAL(be) > Test: oneline_caps ...Server failed to find requested SASL mechanism "PLAIN" > FAILED > 1. ./cunit/backend.testc:314 - CU_ASSERT_PTR_NOT_NULL_FATAL(be) > Test: starttls ...FAILED > 1. ./cunit/backend.testc:408 - CU_ASSERT_PTR_NOT_NULL_FATAL(be) > > Is this correct behavior and I can ignore this? > > PS. strace show thak this test search nonexistent testserver.conf file - > [pid 21654] openat(AT_FDCWD, "/etc/sasl2/testserver.conf", O_RDONLY > [pid 21654] openat(AT_FDCWD, "/etc/sasl/testserver.conf", O_RDONLY) = -1 ENOENT (No such file or directory) > [pid 21654] openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/sasl2/testserver.conf", O_RDONLY) = -1 ENOENT (No such file or directory) > [pid 21654] openat(AT_FDCWD, "/usr/lib/sasl2/testserver.conf", O_RDONLY) = -1 ENOENT (No such file or directory) > From brong at fastmailteam.com Mon Feb 11 17:41:19 2019 From: brong at fastmailteam.com (Bron Gondwana) Date: Mon, 11 Feb 2019 17:41:19 -0500 Subject: Notes Feb 12 Message-ID: Present: Bron, Ken, Robert, ellie Ken: * last week at CalConnect * fixed issue with Windows TZ names - used existing code in ICU * fixed bug with Outlook truncated timezones - made our code not truncate existing RRULE by sliding start-date forward * did some reorg towards implementing shareWith for /set * new: - maybe 40-50% done with shareWith work - then back to mailbox path by ID on-the-fly upgrade work - confident that what's there now will work on a new system! - but first: drafts for calext! (at least managed attachments before IETF) Bron: * fixed issue with G keys not being created on replicas if messages added without CID, breaking search. * went to Fosdem - key notes are: - spoke about JMAP. Plenty of interest, including some people from Dovecot who are keen to implement JMAP but busy, so can't promise exactly when - went to a session on fsync in Postgresql. We should make our code just die on failed fsync as well, that way we'll clean up properly. - dinner with Perl people - went to Dovecot's Chat Over IMAP presentation - it's actually pretty cool. Just a couple of email headers and then layers on top to make the chat experience good. I like it. Suggested the should bring it to IETF EXTRA. * Then CalConnect: - I hacked on a new module, Text::JSCalendar, which is just the conversion parts of JSCalendar. I'll change Net::CalDAVTalk to use it. - new member Alex does an Outlook connector for CalDAV and is based in Vienna, will catch up next week. - timezones - ongoing problems with naming. ISO interested in being clearinghouse for country-owned names. IANA already do central DB. Need to find a home for a tzdist service now it's standard. - IETF work - I did a presentation about why to keep bringing work to IETF. - enumerated existing half-done drafts and proposed to bring most of them through - I suggested just submitting them all at IETF then we can choose to progress them as interest applies, but at least the process is started. - JSCalendar - a few points were raised, Robert is working on finishing them into the doc. - I'm keen to find another person or two from non-FastMail who can join me at IETF and maybe take the mantle at some point in the future. Succession planning! - next CalConnect in Bedford in June, then FM hosting in Philly in October. Hopefully Apple in the bay area next Jan/Feb. Robert: * fixed a couple of bugs in the JMAP ical layer last week * updated parts of JSCalendar doc and will discuss the rest during the week * this week, continue with Xapian languages update - would like to get it to the point where we can experiment with it * bug Michael reported with recurrence overrides - going to put a test together! - we'll try to recreate it ellie: * the chat stuff sounded glastly at first, but seeing more, it seems pretty neat actually! - how will we implement it in JMAP? - could use EmailSubmission delivery status to show that it's been delivered and maybe even that it's been seen if permissions allow! * been lots of pull requests recently, good quality, been merging them! - nice to see the community chipping in! Planning to hold the meeting again next week. Bron and Robert stayed on the call and confirmed the bug that Michael found. We'll work on it tomorrow. -- Bron Gondwana, CEO, FastMail Pty Ltd brong at fastmailteam.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From max at tcen.ru Tue Feb 12 00:34:40 2019 From: max at tcen.ru (Max Kosmach) Date: Tue, 12 Feb 2019 08:34:40 +0300 Subject: Failed CUnit test in 2.5.12 and 3.0.8 In-Reply-To: References: <20190208142313.bscy7yxhrf4bmdqv@tcen.ru> Message-ID: Thank You very much, ?????! After i installed missing package libsasl2-modules and replace expired cert.pem and cacert.pem all cunit tests are passed. 11 ??????? 2019 ?. 19:51:48 GMT+03:00, "????? ????????" ?????: >Hello Max, > >a possible explanation is the lack of libplain.so or libdigestmd5.so in >$libdir/sasl2 , coming from cyrus-sasl . > >Until recently, the place where cyrus-sasl/configure && make install >put those files, when compiled, was different from >the place, where the files were searched at run time. > >Regards > ????? > >On Fri, 2019-02-08 at 17:23 +0300, Max Kosmach wrote: >> Hi, cyrus imap devs! >> >> I'm trying to build 2.5.12 and 3.0.8 cyrus imapd and in both versions >I have same problems with backend suite: >> >> Suite: backend >> Test: badhost ...passed >> Test: badservice ...passed >> Test: sasl_plain ...Server failed to find requested SASL mechanism >"PLAIN" >> FAILED >> 1. ./cunit/backend.testc:198 - CU_ASSERT_PTR_NOT_NULL_FATAL(be) >> Test: sasl_digestmd5 ...Server failed to find requested SASL >mechanism "DIGEST-MD5" >> FAILED >> 1. ./cunit/backend.testc:265 - CU_ASSERT_PTR_NOT_NULL_FATAL(be) >> Test: multiline_caps ...Server failed to find requested SASL >mechanism "PLAIN" >> FAILED >> 1. ./cunit/backend.testc:314 - CU_ASSERT_PTR_NOT_NULL_FATAL(be) >> Test: oneline_caps ...Server failed to find requested SASL >mechanism "PLAIN" >> FAILED >> 1. ./cunit/backend.testc:314 - CU_ASSERT_PTR_NOT_NULL_FATAL(be) >> Test: starttls ...FAILED >> 1. ./cunit/backend.testc:408 - CU_ASSERT_PTR_NOT_NULL_FATAL(be) >> >> Is this correct behavior and I can ignore this? >> >> PS. strace show thak this test search nonexistent testserver.conf >file - >> [pid 21654] openat(AT_FDCWD, "/etc/sasl2/testserver.conf", O_RDONLY > >> [pid 21654] openat(AT_FDCWD, "/etc/sasl/testserver.conf", O_RDONLY) = >-1 ENOENT (No such file or directory) >> [pid 21654] openat(AT_FDCWD, >"/usr/lib/x86_64-linux-gnu/sasl2/testserver.conf", O_RDONLY) = -1 >ENOENT (No such file or directory) >> [pid 21654] openat(AT_FDCWD, "/usr/lib/sasl2/testserver.conf", >O_RDONLY) = -1 ENOENT (No such file or directory) >> -- Max Kosmach -- Max Kosmach -- Max Kosmach -------------- next part -------------- An HTML attachment was scrubbed... URL: From dilyan.palauzov at aegee.org Sun Feb 17 10:35:43 2019 From: dilyan.palauzov at aegee.org (=?UTF-8?Q?=D0=94=D0=B8=D0=BB=D1=8F=D0=BD_?= =?UTF-8?Q?=D0=9F=D0=B0=D0=BB=D0=B0=D1=83=D0=B7=D0=BE=D0=B2?=) Date: Sun, 17 Feb 2019 15:35:43 +0000 Subject: conversations_expire_days and Xapian search Message-ID: <341a4f950eaefb531eaebeb3578f117212afad9c.camel@aegee.org> Hello, I have an understanding question about searches, conversations, and Xapian in 3.0.x: To my knowledge data (headers and body) is filled in a Xapian database. The search with the Xapian database works however only, if these messages are mapped in conversations_db. There is a conversations_expire_days property with default 90: How long the conversations database keeps the message tracking information needed for receiving new messages in existing conversations, in days. Does this mean, that by default after 90 days messages are deleted from the conversations db and cannot be found anymore using the Xapian database? How to setup the days, so that all messages in Xapian can be mapped to convesations_db and therefore found? Regards ????? From egoitz at sarenet.es Sun Feb 17 13:13:35 2019 From: egoitz at sarenet.es (egoitz at sarenet.es) Date: Sun, 17 Feb 2019 19:13:35 +0100 Subject: conversations_expire_days and Xapian search In-Reply-To: <341a4f950eaefb531eaebeb3578f117212afad9c.camel@aegee.org> References: <341a4f950eaefb531eaebeb3578f117212afad9c.camel@aegee.org> Message-ID: Hi! Headers are only mapped if you set search_index_headers to yes. Although you can search by from, to, cc, subject without having it and pretty fastly. Perhaps then, Cyrus even when setting it, does a normal lookup, a non xapian lookup... although I'm not sure about it... I can say you, that you can search by from, to, cc and subject even when forcing to use Xapian (search_fuzzy_always) and having configured to not index headers (search_index_headers). You should not confuse conversations with indexes of messages. I'll clarify all this, because I have needed too to have clear this concepts. - Conversations database : It has some information from the messages and Cyrus uses it for keeping the messages sha signature of the message content. (I think headers and body...). - Xapian database : database where content becomes indexed of all your indexed mailboxes (wanted to mean too... folders inside the same user). Cyrus can return you with XCONVERSATIONS IMAP extension, mail threaded in conversations. It can only return you the mails related to a conversation from the past three months... that's not tunable in the config file... so no parameter for modifying it. That does not mean, you can only search mail from three months. This just means, you can only receive conversations messages with XCONV that have been received in the past three months. If a conversation lasts 1 year (which is extremely extrange) you would only receive the present month, previous one and the previous to the previous one (if exist them) in the XCONV output. I should say too... XCONV I think is not almost supported in several IMAP mua... I think it's more designed for JMAP. IMAP muas often group by themselves mail in conversations. Apart from that, and without any relationship with conversations, you can search mail from any time back... the three months are just in terms of conversations with XCONV, which is what returns conversations from your mailbox. I think this answers to your questions... please If I'm wrong with something (which I don't think...) corrections are always very welcome :) Cheers! El 2019-02-17 16:35, ????? ???????? escribi?: > Hello, > > I have an understanding question about searches, conversations, and Xapian in 3.0.x: > > To my knowledge data (headers and body) is filled in a Xapian database. > > The search with the Xapian database works > however only, if these messages are mapped in conversations_db. There is a conversations_expire_days property with > default 90: > > How long the conversations database keeps the message tracking information needed for receiving new messages in > existing conversations, in days. > > Does this mean, that by default after 90 days messages are deleted from the conversations db and cannot be found anymore > using the Xapian database? How to setup the days, so that all messages in Xapian can be mapped to convesations_db and > therefore found? > > Regards > ????? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsto at fastmailteam.com Mon Feb 18 05:13:14 2019 From: rsto at fastmailteam.com (Robert Stepanek) Date: Mon, 18 Feb 2019 05:13:14 -0500 Subject: conversations_expire_days and Xapian search In-Reply-To: <341a4f950eaefb531eaebeb3578f117212afad9c.camel@aegee.org> References: <341a4f950eaefb531eaebeb3578f117212afad9c.camel@aegee.org> Message-ID: <86b24ece-d2de-4058-9dce-46dbc5c46cce@beta.fastmail.com> Hi, On Sun, Feb 17, 2019, at 4:36 PM, ????? ???????? wrote: > Does this mean, that by default after 90 days messages are deleted from the conversations db and cannot be found anymore > using the Xapian database? How to setup the days, so that all messages in Xapian can be mapped to convesations_db and > therefore found? The conversations expiry date does not affect search. Conversations.db is kind of a misnomer nowadays: initially, it just stored the mapping from thread identifiers to the messages in that thread. Nowadays, it also maps the globally unique id of a message to all its instances in mailboxes, which is called the G record in conversations.db. E.g. if you two copies of the same message in mailboxes A and B, then the G records in conversations.db will be Gc1ce7414bf85c0ae35f417288198b1f095c8077c:1:5 ... Gc1ce7414bf85c0ae35f417288198b1f095c8077c:2:12 ... where c1ce7414bf85c0ae35f417288198b1f095c8077c is the SHA1 of the raw MIME message, the following "1" and "2" values are an index into the db-internal folders list (mailboxes "A" and "B" in this case), and the last integer value is the UID of the message in the respective folder. These records will never expire as long as the message exists, and Xapian will store the SHA1 has as identifier of the indexed document. Cheers, Rboert -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsto at fastmailteam.com Mon Feb 18 16:35:08 2019 From: rsto at fastmailteam.com (Robert Stepanek) Date: Mon, 18 Feb 2019 16:35:08 -0500 Subject: Cyrus meeting notes - Feb 18, 2019 Message-ID: <412acf51-6758-4f8d-8094-d63849ac5eff@www.fastmail.com> Participants: Bron, Ellie, Rik, RobS - Bron and RobS worked together in Vienna last week. Met Davx5 and Outlook Synchronizer developers. - All changes from last week including JMAP fixes deployed. - Fix pushed to base64-encode all email parts generated from blobIds. Will use QP-encoding for text parts for readibility. - There's an issue with conversations splitting, e.g. more than 100 messages in a thread. - JSCalendar updates done both on RFC and implementation. - JSContact RFC will get sent out this week. - Xapian languages patch is ready for testing. Need to set up environment for realistic load testing. - Ellie worked mainly on pull requests, landed a nice one yesterday. Cheers, Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: