From chose at ajetaci.cz Sat Dec 1 02:20:06 2018 From: chose at ajetaci.cz (chose) Date: Sat, 01 Dec 2018 08:20:06 +0100 Subject: cyrus 2.4 and DBERROR Message-ID: <26795ac29a5bdcc5696f190107c978ed@ajetaci.cz> Good morning, what caused this and how to recover. All files in db/ directory were deleted and cyrus restarted, still no progress. Dec 1 07:57:44 email11 imap[4498]: DBERROR db4: PANIC: fatal region error detected; run recovery Dec 1 07:57:44 email11 imap[4498]: DBERROR: critical database situation Dec 1 07:57:44 email11 idled[4422]: DBERROR db4: PANIC: fatal region error detected; run recovery Dec 1 07:57:44 email11 idled[4422]: DBERROR: critical database situation Dec 1 07:57:44 email11 imap[4428]: DBERROR db4: PANIC: fatal region error detected; run recovery Dec 1 07:57:44 email11 imap[4428]: DBERROR: critical database situation Dec 1 07:57:44 email11 imap[4482]: DBERROR db4: PANIC: fatal region error detected; run recovery Cyrus imap is working, but it si slow. Thanks and best regards J.Karliak -- Ma domena pouziva zabezpeceni a kontrolu SPF (www.openspf.org) a DomainKeys/DKIM (s ADSP) a implementaci DMARC. Pokud mate problemy s dorucenim emailu, zacnete pouzivat metody overeni puvody emailu zminene vyse. Dekuji. My domain use SPF (www.openspf.org) and DomainKeys/DKIM (with ADSP) policy and implementation of the DMARC. If you've problem with sending emails to me, start using email origin methods mentioned above. Thank you. From brong at fastmailteam.com Sat Dec 1 03:12:26 2018 From: brong at fastmailteam.com (Bron Gondwana) Date: Sat, 01 Dec 2018 03:12:26 -0500 Subject: Thunderbird's color labels In-Reply-To: References: Message-ID: <0bac1485-2963-43d0-8749-dc6c291d070e@sloti7d1t02> Thunderbird is using the $label flags for storing colours. I don't believe the colour itself is stored on the server. On Sat, 1 Dec 2018, at 07:07, Joseph Brennan wrote: > > I am trying to find where cyrus imap stores the colors Thunderbird uses in the message index display. I thought they were probably custom flags and I could use mbexamine to find them. > > I have a mailbox that has this in the first stanza from mbexamine, which looked promising-- > User Flags: $label1 $label2 $label3 $label4 $label5 $Forwarded Redirected > > Each message has a line with "> USERFLAGS" but those are mostly four groups of zeros, and the values do not correspond to the Tbird colors. > > What else can I try? > > > -- > Joseph Brennan > Lead, Email and Systems Applications > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd brong at fastmailteam.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From chose at ajetaci.cz Sat Dec 1 06:22:39 2018 From: chose at ajetaci.cz (chose) Date: Sat, 01 Dec 2018 12:22:39 +0100 Subject: Cyrus 2.4 and debug Message-ID: Good afternoon, I've setup debug: 1 in the imapd.conf, created log/ directory with owner cyrus, but I do not see any log. What did I missed ? Cyrus imap is slow, reading emails takes a seconds, server is not loaded. After some verifications I thing that cyrus imap (also whole server is restarted) has some problems. Or not, but how to be sure he has no internal problems that could be found with debug. Thanks and best regards J.Karliak -- Ma domena pouziva zabezpeceni a kontrolu SPF (www.openspf.org) a DomainKeys/DKIM (s ADSP) a implementaci DMARC. Pokud mate problemy s dorucenim emailu, zacnete pouzivat metody overeni puvody emailu zminene vyse. Dekuji. My domain use SPF (www.openspf.org) and DomainKeys/DKIM (with ADSP) policy and implementation of the DMARC. If you've problem with sending emails to me, start using email origin methods mentioned above. Thank you. From lists at kwiatek.eu.org Sat Dec 1 08:57:57 2018 From: lists at kwiatek.eu.org (Andrzej Kwiatkowski) Date: Sat, 1 Dec 2018 14:57:57 +0100 Subject: Cyrus 2.4 and debug In-Reply-To: References: Message-ID: <0d2ec127-637d-a347-6ca8-4451755043f9@kwiatek.eu.org> W dniu 01.12.2018 o?12:22, chose pisze: > Good afternoon, > I've setup debug: 1 in the imapd.conf, created log/ directory > with owner cyrus, but I do not see any log. What did I missed ? > Cyrus imap is slow, reading emails takes a seconds, server is not > loaded. After some verifications I thing that cyrus imap (also whole > server is restarted) has some problems. Or not, but how to be sure he > has no internal problems that could be found with debug. > Thanks and best regards > J.Karliak > Check your random number source, maybe you need to run rngd or something similar. Regards AK From charles.bradshaw at ntlworld.com Sun Dec 2 09:19:23 2018 From: charles.bradshaw at ntlworld.com (Charles Bradshaw) Date: Sun, 2 Dec 2018 14:19:23 +0000 Subject: suddenly 'User unknown'? In-Reply-To: References: <10a3d609-11b2-e714-fc6b-e4893119d998@ntlworld.com> <20181128161203.GB20152@dan.olp.net> <28dd4fb1-74e6-41da-ae0d-24a076aa2a2e@ntlworld.com> <20181129143913.GE20152@dan.olp.net> <33c227e8-6f77-6669-eca1-3f3c7516d1cd@ntlworld.com> <43b420d0b1727118d29460b4082686a1.squirrel@webmail.bi.invoca.ch> <9fbd7a4d-3569-94c1-2ed8-fce5bca12c4b@jangulo.net> <3c52506c-a01b-b05f-db97-eb0c4228cc23@ntlworld.com> <15566cca-2c0c-263e-c806-a61eeff8c816@sendmaid.org> <70ab7724-c73a-3462-b282-7cbfb80d542a@ntlworld.com> Message-ID: <0635aac4-47df-3f62-c12b-df5e612cf133@ntlworld.com> Edda, I think there might be some clues, but I'm struggling to understand the below results. On 30/11/2018 17:36, Edda wrote: > Am 30.11.18 um 17:34 schrieb Charles Bradshaw: >> Edda, >> >> On 30/11/2018 15:48, Edda wrote: >>> Not a cyrus issue. Apparently sendmail strips the domain as you see in >>> lines like "RCPT To:" >>> >>> Your cyrusv2 Mailer in sendmail.mc seems correct to me. >>> >>> What do you get from (you can skip all the line for user root) >>> >>> sendmail -d21.1 -bv brad at bradcan.homelinux.com >> as brad: >> >> [brad at dell2600-1 ~]$ sendmail -d21.1 -bv brad at bradcan.homelinux.com >> Notice: -bv may give misleading output for non-privileged user >> can not chdir(/var/spool/mqueue/): Permission denied >> Program mode requires special privileges, e.g., root or TrustedUser. >> >> How do I setup TrustUser? > > The message is a bit missleading. You can't just add a TrustedUser to > sendmail and run this test. You would have to change all > privileges.... It's absolutely ok to check the daemon as root. > >> >> but as root: >> >> [root at dell2600-1 brad]# sendmail -d21.1 -bv brad at bradcan.homelinux.com >> >> [...] >> . com . > >> rewrite: ruleset Parse1???????????? input: brad < @ bradcan . homelinux >> . com . > >> rewrite: ruleset Parse1?????????? returns: $# cyrusv2 $: brad >> rewrite: ruleset parse??????????? returns: $# cyrusv2 $: brad >> rewrite: ruleset 2????????????????? input: brad >> rewrite: ruleset 2??????????????? returns: brad >> rewrite: ruleset EnvToSMT?????????? input: brad >> rewrite: ruleset EnvToSMT???????? returns: brad >> rewrite: ruleset final????????????? input: brad >> rewrite: ruleset final??????????? returns: brad >> brad at bradcan.homelinux.com... deliverable: mailer cyrusv2, user brad > > The Parse1 ruleset considers brad at bradcan.homeliniux.com as a local > machine's user (Class $=w in sendmail). Therefore it strips the domain. > > You can check $=w like this: > > sendmail -C sendmail.cf -bt > > $=w > localhost > [127.0.0.1] > dell2600-1.bradcan.homelinux.com > [...] > >/quit > > Look for bradcan.homelinux.com > > Do you have an entry for bradcan.homelinux.com in /etc/hosts? Then you > can simply delete it. No bradcan.homelinux.com is not in /etc/hosts or /etc/resolv.conf - But this: [brad at dell2600-1 ~]$ sendmail -C /etc/mail/sendmail.cf -bt ADDRESS TEST MODE (ruleset 3 NOT automatically invoked) Enter
> $=w dell2600-1.bradcan.homelinux.com [213.106.111.18] localhost.localdomain localhost bradcan.co.uk bradcan.homelinux.com > /quit And this: [root at dell2600-1 mail]# hostname --fqdn bradcan.homelinux.com Whereas other hosts on the network show: host.bradcan.homelinux.com ?~o~ And after removing some comments and ignoring binary file matches # grep bradcan.homelinyx.com /etc/mail/* produces: access:bradcan.homelinux.com??? ??? ??? RELAY mailertable:bradcan.homelinux.com??? ??? cyrusv2:/var/lib/imap/socket/lmtp sendmail.cf:C{M}bradcan.homelinux.com sendmail.mc:MASQUERADE_DOMAIN(bradcan.homelinux.com) sendmail.mc~:MASQUERADE_DOMAIN(bradcan.homelinux.com) virtusertable:@bradcan.co.uk??? %1 at bradcan.homelinux.com ??? are any of the above now somehow incorrect ??? ??? Should I have the following line in virtusertable, The comments seem to suggest that I should. ???: @bradcan.homelinux.com %1%3 > Edda > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From charles.bradshaw at ntlworld.com Sun Dec 2 10:45:29 2018 From: charles.bradshaw at ntlworld.com (Charles Bradshaw) Date: Sun, 2 Dec 2018 15:45:29 +0000 Subject: suddenly 'User unknown'? In-Reply-To: <0635aac4-47df-3f62-c12b-df5e612cf133@ntlworld.com> References: <10a3d609-11b2-e714-fc6b-e4893119d998@ntlworld.com> <20181128161203.GB20152@dan.olp.net> <28dd4fb1-74e6-41da-ae0d-24a076aa2a2e@ntlworld.com> <20181129143913.GE20152@dan.olp.net> <33c227e8-6f77-6669-eca1-3f3c7516d1cd@ntlworld.com> <43b420d0b1727118d29460b4082686a1.squirrel@webmail.bi.invoca.ch> <9fbd7a4d-3569-94c1-2ed8-fce5bca12c4b@jangulo.net> <3c52506c-a01b-b05f-db97-eb0c4228cc23@ntlworld.com> <15566cca-2c0c-263e-c806-a61eeff8c816@sendmaid.org> <70ab7724-c73a-3462-b282-7cbfb80d542a@ntlworld.com> <0635aac4-47df-3f62-c12b-df5e612cf133@ntlworld.com> Message-ID: <8409acf0-1d19-90ac-64d8-c1c900573dee@ntlworld.com> Ha.. SOLVED IT :-)) /etc/host REQUIRES the entry: 192.168.0.3 ??? dell2600-1.bradcan.homelinux.com dell2600-1 Then: [root at dell2600-1 brad]# sendmail -C /etc/mail/sendmail.cf -bt ADDRESS TEST MODE (ruleset 3 NOT automatically invoked) Enter
> $=w dell2600-1.bradcan.homelinux.com dell2600-1 localhost.localdomain localhost bradcan.co.uk [192.168.0.3] > /quit Don't ask me what changed... Thanks for the 'help' On 02/12/2018 14:19, Charles Bradshaw via Info-cyrus wrote: > Edda, > > I think there might be some clues, but I'm struggling to understand the > below results. > > On 30/11/2018 17:36, Edda wrote: >> Am 30.11.18 um 17:34 schrieb Charles Bradshaw: >>> Edda, >>> >>> On 30/11/2018 15:48, Edda wrote: >>>> Not a cyrus issue. Apparently sendmail strips the domain as you see in >>>> lines like "RCPT To:" >>>> >>>> Your cyrusv2 Mailer in sendmail.mc seems correct to me. >>>> >>>> What do you get from (you can skip all the line for user root) >>>> >>>> sendmail -d21.1 -bv brad at bradcan.homelinux.com >>> as brad: >>> >>> [brad at dell2600-1 ~]$ sendmail -d21.1 -bv brad at bradcan.homelinux.com >>> Notice: -bv may give misleading output for non-privileged user >>> can not chdir(/var/spool/mqueue/): Permission denied >>> Program mode requires special privileges, e.g., root or TrustedUser. >>> >>> How do I setup TrustUser? >> The message is a bit missleading. You can't just add a TrustedUser to >> sendmail and run this test. You would have to change all >> privileges.... It's absolutely ok to check the daemon as root. >> >>> but as root: >>> >>> [root at dell2600-1 brad]# sendmail -d21.1 -bv brad at bradcan.homelinux.com >>> >>> [...] >>> . com . > >>> rewrite: ruleset Parse1???????????? input: brad < @ bradcan . homelinux >>> . com . > >>> rewrite: ruleset Parse1?????????? returns: $# cyrusv2 $: brad >>> rewrite: ruleset parse??????????? returns: $# cyrusv2 $: brad >>> rewrite: ruleset 2????????????????? input: brad >>> rewrite: ruleset 2??????????????? returns: brad >>> rewrite: ruleset EnvToSMT?????????? input: brad >>> rewrite: ruleset EnvToSMT???????? returns: brad >>> rewrite: ruleset final????????????? input: brad >>> rewrite: ruleset final??????????? returns: brad >>> brad at bradcan.homelinux.com... deliverable: mailer cyrusv2, user brad >> The Parse1 ruleset considers brad at bradcan.homeliniux.com as a local >> machine's user (Class $=w in sendmail). Therefore it strips the domain. >> >> You can check $=w like this: >> >> sendmail -C sendmail.cf -bt >>> $=w >> localhost >> [127.0.0.1] >> dell2600-1.bradcan.homelinux.com >> [...] >>> /quit >> Look for bradcan.homelinux.com >> >> Do you have an entry for bradcan.homelinux.com in /etc/hosts? Then you >> can simply delete it. > No bradcan.homelinux.com is not in /etc/hosts or /etc/resolv.conf - But > this: > > [brad at dell2600-1 ~]$ sendmail -C /etc/mail/sendmail.cf -bt > ADDRESS TEST MODE (ruleset 3 NOT automatically invoked) > Enter
>> $=w > dell2600-1.bradcan.homelinux.com > [213.106.111.18] > localhost.localdomain > localhost > bradcan.co.uk > bradcan.homelinux.com >> /quit > And this: > > [root at dell2600-1 mail]# hostname --fqdn > bradcan.homelinux.com > > Whereas other hosts on the network show: host.bradcan.homelinux.com > > ?~o~ > > And after removing some comments and ignoring binary file matches # grep > bradcan.homelinyx.com /etc/mail/* produces: > > access:bradcan.homelinux.com??? ??? ??? RELAY > > mailertable:bradcan.homelinux.com??? ??? cyrusv2:/var/lib/imap/socket/lmtp > > sendmail.cf:C{M}bradcan.homelinux.com > > sendmail.mc:MASQUERADE_DOMAIN(bradcan.homelinux.com) > sendmail.mc~:MASQUERADE_DOMAIN(bradcan.homelinux.com) > > virtusertable:@bradcan.co.uk??? %1 at bradcan.homelinux.com > > ??? are any of the above now somehow incorrect ??? > > ??? Should I have the following line in virtusertable, The comments seem > to suggest that I should. ???: > > @bradcan.homelinux.com %1%3 > >> Edda >> >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From anthony.prades at chezouam.net Mon Dec 3 06:40:00 2018 From: anthony.prades at chezouam.net (Anthony Prades) Date: Mon, 3 Dec 2018 12:40:00 +0100 Subject: cyrus 2.4 and DBERROR In-Reply-To: <26795ac29a5bdcc5696f190107c978ed@ajetaci.cz> References: <26795ac29a5bdcc5696f190107c978ed@ajetaci.cz> Message-ID: <916c37cf-d54f-90c6-0086-0609a5e6e6d6@chezouam.net> Hi, Try db_recover tool in /var/lib/cyrus, somithing like (depends of your install): -- stop cyrus -- cd /var/lib/cyrus && db_recover -v -h /var/lib/cyrus/db -- start cyrus -- or herder version: -- stop cyrus -- cd /var/lib/cyrus && db_recover -c -v -h /var/lib/cyrus/db -- start cyrus -- If problem persists, you may convert your db file to flat, then flat to db using cvt_cyrusdb tool - something like: -- stop cyrus -- cvt_cyrusdb /var/lib/cyrus/mailboxes.db skiplist /tmp/mailboxes flat mv /var/lib/cyrus/mailboxes.db /var/lib/cyrus/mailboxes.db.orig cvt_cyrusdb /tmp/mailboxes flat /var/lib/cyrus/mailboxes.db skiplist chown cyrus:mail /var/lib/cyrus/mailboxes.db -- start cyrus -- Note: do backup before... Anthony On 12/1/18 8:20 AM, chose wrote: > ?Good morning, > ?what caused this and how to recover. All files in db/ directory were > deleted and cyrus restarted, still no progress. > > Dec? 1 07:57:44 email11 imap[4498]: DBERROR db4: PANIC: fatal region > error detected; run recovery > Dec? 1 07:57:44 email11 imap[4498]: DBERROR: critical database situation > Dec? 1 07:57:44 email11 idled[4422]: DBERROR db4: PANIC: fatal region > error detected; run recovery > Dec? 1 07:57:44 email11 idled[4422]: DBERROR: critical database situation > Dec? 1 07:57:44 email11 imap[4428]: DBERROR db4: PANIC: fatal region > error detected; run recovery > Dec? 1 07:57:44 email11 imap[4428]: DBERROR: critical database situation > Dec? 1 07:57:44 email11 imap[4482]: DBERROR db4: PANIC: fatal region > error detected; run recovery > > ? Cyrus imap is working, but it si slow. > ? Thanks and best regards > ? J.Karliak > From egoitz at sarenet.es Tue Dec 4 11:35:48 2018 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Tue, 04 Dec 2018 17:35:48 +0100 Subject: Operating system performance for Cyrus and one more question Message-ID: Good afternoon, Is there any noticiable operating systems whose syscalls, filesystems... etc... could make Cyrus run faster?. We usually run it on FreeBSD and are pretty happy. Perhaps newer versions run better in any Linux like os?. About Xapian, XCONV, Jmap... I have tried doing a search with my mailbox in a new testing env, where I have connected a tesing mailbox imap partition with messages and mailboxes, coming from an older version of Cyrus. Even having Xapian, Conversations and so enabled, it does not find messages newer than 3 month located in the Sent folder. Perhaps Xapian, only works for new incoming messages?. Or should the mailboxes be reconstructed in order to a search to find and show all messages from a conversation existing in all mailboxes?. Or perhaps it's my mua's fault and is not searching properly in the IMAP server, because is not issuing some new commands to be used in order to make this kind of search to be done by Cyrus?. Best regards, -- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Tue Dec 4 11:38:58 2018 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Tue, 04 Dec 2018 17:38:58 +0100 Subject: From 2.3.16 to 3.0.8 - lost expunged messages In-Reply-To: <7932368d-e596-dd33-2d8f-4d7ae4d18f86@cc.upv.es> References: <7932368d-e596-dd33-2d8f-4d7ae4d18f86@cc.upv.es> Message-ID: Hi Carlos, Did you finally lost these expunged messages?. Best regards, --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 01-10-2018 14:26, Carlos Larra?aga escribi?: > Hello, > > We are now testing a per user based migration from cyrus-imapd 2.3.16 to 3.0.8 (both with delayed expunge) and noticed that we lose user's expunged messages when executing "reconstruct -r -G -V max user.foo". Is there any way to not lose the expunged messages? > > Regards and thanks in advance. > > Carlos Larra?aga > Analista de Sistemas > ASIC - UPV- SPAIN > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Tue Dec 4 11:43:24 2018 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Tue, 04 Dec 2018 17:43:24 +0100 Subject: Missing Email & Folders In-Reply-To: References: <0de401d4760c$370ed0f0$a52c72d0$@rolet.com> <20181106205144.GB8433@dan.olp.net> <0f1001d47626$73768910$5a639b30$@rolet.com> <20181107091612.Horde.tYa6sMTRD21LC9ZdCUzeFUV@webmail.uni-tuebingen.de> Message-ID: <4abaf67d28fb11fd64dfbd3387264644@sarenet.es> Can anyone provide some more details about this bug?. Have seen something similar to this... although have not been able to deep on that issue because have not had reproduced it... Cheers! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 10-11-2018 22:22, Marcus Schopen escribi?: > Am Mittwoch, den 07.11.2018, 09:16 +0100 schrieb Michael Menge: Hi, > > Quoting Robert Covell : > > If you suspect this is due to a client related problem, you could > enable telemetry logging to find out who/what is causeing the > emails to go > missing. > https://www.cyrusimap.org/imap/reference/faqs/o-telemetry.html > Good idea will turn this on. > > If the purpose is to (mostly) copy emails into the folder and > rarely > delete, you could restrict delete access to a specific account > via ACL. https://www.cyrusimap.org/imap/reference/admin/access-control/rights > - > reference.html > Believe it is setup like this, been awhile since I did it. Will > confirm. > > Also forgot to say the one account is ~90GB if that makes any > difference... > > -Bob > > (sent twice) We have seen the same kind of problems with Outlook with much smaller mailboxes. I didn't have the time to debug it. So fare I suspected that some Data in the Outlook profile got somehow corrupted. Isn't this the known bug that Outlook 2013 doesn't really work well if it is only used via IMAP and not with Exchange? There have been similar cases with IMAP accounts that have been completely compromised. On the IMAP there were whole folders missing, but they still existed locally. ---- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From heiler.bemerguy at cinbesa.com.br Tue Dec 4 11:53:37 2018 From: heiler.bemerguy at cinbesa.com.br (Heiler Bemerguy) Date: Tue, 4 Dec 2018 13:53:37 -0300 Subject: HELP: Can't fix quota, it seems renamed-back mailbox doesn't exists at all Message-ID: Hail, this is a Cyrus 2.5.10-3 on Debian 9 A mailbox was accidentaly deleted, as the log shows: Dec? 4 11:56:42 localhost cyrus/imap[27354]: Rename: user.planejamento^funbosque -> DELETED.user.planejamento^funbosque.5C0695AA Dec? 4 11:56:42 localhost cyrus/imap[27354]: Deleted mailbox user.planejamento^funbosque Dec? 4 11:56:42 localhost cyrus/imap[27354]: Rename: user.planejamento^funbosque.Drafts -> DELETED.user.planejamento^funbosque.Drafts.5C0695AA Dec? 4 11:56:42 localhost cyrus/imap[27354]: Deleted mailbox user.planejamento^funbosque.Drafts Dec? 4 11:56:42 localhost cyrus/imap[27354]: Rename: user.planejamento^funbosque.Junk -> DELETED.user.planejamento^funbosque.Junk.5C0695AA Dec? 4 11:56:42 localhost cyrus/imap[27354]: Deleted mailbox user.planejamento^funbosque.Junk Dec? 4 11:56:42 localhost cyrus/imap[27354]: Rename: user.planejamento^funbosque.Sent -> DELETED.user.planejamento^funbosque.Sent.5C0695AA Dec? 4 11:56:42 localhost cyrus/imap[27354]: Deleted mailbox user.planejamento^funbosque.Sent Dec? 4 11:56:42 localhost cyrus/imap[27354]: Rename: user.planejamento^funbosque.Spam -> DELETED.user.planejamento^funbosque.Spam.5C0695AA Dec? 4 11:56:42 localhost cyrus/imap[27354]: Deleted mailbox user.planejamento^funbosque.Spam Dec? 4 11:56:42 localhost cyrus/imap[27354]: Rename: user.planejamento^funbosque.Trash -> DELETED.user.planejamento^funbosque.Trash.5C0695AA Dec? 4 11:56:42 localhost cyrus/imap[27354]: Deleted mailbox user.planejamento^funbosque.Trash I renamed it back to try to recover it, manually with cyradm: Dec? 4 12:59:38 localhost cyrus/imap[39601]: Rename: DELETED.user.planejamento^funbosque.5C0695AA -> user.planejamento^funbosque Dec? 4 12:59:38 localhost cyrus/imap[39601]: Deleted mailbox DELETED.user.planejamento^funbosque.5C0695AA Dec? 4 13:02:03 localhost cyrus/imap[39601]: Rename: DELETED.user.planejamento^funbosque.Drafts.5C0695AA -> user.planejamento^funbosque.Drafts Dec? 4 13:02:03 localhost cyrus/imap[39601]: Deleted mailbox DELETED.user.planejamento^funbosque.Drafts.5C0695AA Dec? 4 13:02:27 localhost cyrus/imap[39601]: Rename: DELETED.user.planejamento^funbosque.Junk.5C0695AA -> user.planejamento^funbosque.Junk Dec? 4 13:02:27 localhost cyrus/imap[39601]: Deleted mailbox DELETED.user.planejamento^funbosque.Junk.5C0695AA Dec? 4 13:02:54 localhost cyrus/imap[39601]: Rename: DELETED.user.planejamento^funbosque.Sent.5C0695AA -> user.planejamento^funbosque.Sent Dec? 4 13:02:54 localhost cyrus/imap[39601]: Deleted mailbox DELETED.user.planejamento^funbosque.Sent.5C0695AA Dec? 4 13:03:20 localhost cyrus/imap[39601]: Rename: DELETED.user.planejamento^funbosque.Spam.5C0695AA -> user.planejamento^funbosque.Spam Dec? 4 13:03:20 localhost cyrus/imap[39601]: Deleted mailbox DELETED.user.planejamento^funbosque.Spam.5C0695AA Dec? 4 13:04:18 localhost cyrus/imap[39601]: Rename: DELETED.user.planejamento^funbosque.Trash.5C0695AA -> user.planejamento^funbosque.Trash Dec? 4 13:04:18 localhost cyrus/imap[39601]: Deleted mailbox DELETED.user.planejamento^funbosque.Trash.5C0695AA All the files are there on the filesystem, but the quota usage isn't right. I've already tried ? cyrus reconstruct -r -f user/planejamento^funbosque ? cyrus quota -f user/planejamento^funbosque ? cyrus quota -f user/planejamento.funbosque And a full "cyrus check" with no luck.. The command LQ simply shows nothing. Like this: 127.0.0.1> lq user/planejamento.funbosque Any ideas?! root at mailer:/var/spool/cyrus/mail/p/user/planejamento^funbosque# cat cyrus.header Cyrus mailbox header "The best thing about this system was that it had lots of goals." ??????? --Jim Morris on Andrew ??????? 3e5650f459303706 $MDNSent $Forwarded planejamento.funbosque? lrswipkxtecdan root at mailer:/var/spool/cyrus/mail/p/user/planejamento^funbosque# du -hs 570M??? . -- Atenciosamente, Heiler Bensimon Bemerguy - CINBESA Analista de Redes, Wi-Fi, Virtualiza??o e Servi?os Internet (55) 91 98151-4894 From egoitz at sarenet.es Wed Dec 5 02:51:11 2018 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Wed, 05 Dec 2018 08:51:11 +0100 Subject: Cyrus 3.0.8 longlocks and unexpunge Message-ID: <3b4bef5d55628c0af9027ed5420c6680@sarenet.es> Good morning, I'm testing 3.0.8 Cyrus IMAP. I have seen that for instance, when you remove a couple of messages (with a vanila Roundcube webmail so IMAP, for instance) you cause a copy to the trash and a expunge in the INBOX. This operation takes very long comparing with older versions of Cyrus. I mean : Dec 5 08:38:42 testenv imap[88161]: login: [172.22.0.1] egoitz at sarenet.es PLAIN User logged in SESSIONID= Dec 5 08:38:42 testenv squatter[20782]: indexing mailbox user/egoitz/Papelera at sarenet.es... Dec 5 08:38:42 testenv imap[88160]: Expunged 2 messages from sarenet.es!user.egoitz Dec 5 08:38:42 testenv imap[88160]: Repacking mailbox sarenet.es!user.egoitz version 10 Dec 5 08:38:42 testenv squatter[20782]: Xapian committed 2 updates in 0.049130 sec Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown charset RCMAIL_CHARSET for text/PLAIN Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown charset RCMAIL_CHARSET for text/CALENDAR Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown charset RCMAIL_CHARSET for text/PLAIN Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown charset RCMAIL_CHARSET for text/CALENDAR Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown charset RCMAIL_CHARSET for text/PLAIN Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown charset RCMAIL_CHARSET for text/CALENDAR Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown charset RCMAIL_CHARSET for text/PLAIN Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown charset RCMAIL_CHARSET for text/CALENDAR DEC 5 08:39:02 TESTENV IMAP[88160]: MAILBOX: LONGLOCK SARENET.ES!USER.EGOITZ FOR 19.9 SECONDS Dec 5 08:39:02 testenv imap[88160]: USAGE egoitz at sarenet.es user: 8.469829 sys: 0.584660 Is this a normal or expected behavior?. By the way, I have configured : expunge_mode: delayed delete_mode: delayed deletedprefix: DELETED But now, if I do : % /usr/local/cyrus/sbin/unexpunge -lv user/egoitz at sarenet.es % So, no results... unexpunge gives an error for instance if the mailbox does not exist.... Why this last problem can happen?. Why it could take so long to perform the removal of the last commented messages?. Best regards, -- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From javier at jangulo.net Wed Dec 5 04:45:58 2018 From: javier at jangulo.net (Javier Angulo) Date: Wed, 5 Dec 2018 10:45:58 +0100 Subject: Cyrus 3.0.8 longlocks and unexpunge In-Reply-To: <3b4bef5d55628c0af9027ed5420c6680@sarenet.es> References: <3b4bef5d55628c0af9027ed5420c6680@sarenet.es> Message-ID: On 12/5/18 8:51 AM, Egoitz Aurrekoetxea wrote: > Good morning, > > > I'm testing 3.0.8 Cyrus IMAP. I have seen that for instance, when you > remove a couple of messages (with a vanila Roundcube webmail so IMAP, > for instance) you cause a copy to the trash and a expunge in the INBOX. > This operation takes very long comparing with older versions of Cyrus. I > mean : > > Dec? 5 08:38:42 testenv imap[88161]: login: [172.22.0.1] > egoitz at sarenet.es PLAIN User logged in > SESSIONID= > Dec? 5 08:38:42 testenv squatter[20782]: indexing mailbox > user/egoitz/Papelera at sarenet.es... > Dec? 5 08:38:42 testenv imap[88160]: Expunged 2 messages from > sarenet.es!user.egoitz > Dec? 5 08:38:42 testenv imap[88160]: Repacking mailbox > sarenet.es!user.egoitz version 10 > Dec? 5 08:38:42 testenv squatter[20782]: Xapian committed 2 updates in > 0.049130 sec > Dec? 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown > charset RCMAIL_CHARSET for text/PLAIN > Dec? 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown > charset RCMAIL_CHARSET for text/CALENDAR > Dec? 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown > charset RCMAIL_CHARSET for text/PLAIN > Dec? 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown > charset RCMAIL_CHARSET for text/CALENDAR > Dec? 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown > charset RCMAIL_CHARSET for text/PLAIN > Dec? 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown > charset RCMAIL_CHARSET for text/CALENDAR > Dec? 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown > charset RCMAIL_CHARSET for text/PLAIN > Dec? 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown > charset RCMAIL_CHARSET for text/CALENDAR > *Dec? 5 08:39:02 testenv imap[88160]: mailbox: longlock > sarenet.es!user.egoitz for 19.9 seconds* > Dec? 5 08:39:02 testenv imap[88160]: USAGE egoitz at sarenet.es user: > 8.469829 sys: 0.584660 > > Is this a normal or expected behavior?. > > By the way, I have configured : > > expunge_mode: delayed > delete_mode: delayed > deletedprefix: DELETED > > But now, if I do : > > % /usr/local/cyrus/sbin/unexpunge -lv user/egoitz at sarenet.es > % > > So, no results... unexpunge gives an error for instance if the mailbox > does not exist.... > > > Why this last problem can happen?. Why it could take so long to perform > the removal of the last commented messages?. > Hi Egoitz, from this line: > Dec 5 08:38:42 testenv imap[88160]: Repacking mailbox > sarenet.es!user.egoitz version 10 it seems that you have not run a reconstruct on the mailbox (should be version 13 on cyrus 3.0) try: reconstruct -V max user/egoitz at sarenet.es As you are probably coming from cyrus version 2.3 reconstruct is going to take a long time ... I guess the unexpunge problem is also related and you need to reconstruct the mailbox. Cheers, Javier From javier at jangulo.net Wed Dec 5 04:53:14 2018 From: javier at jangulo.net (Javier Angulo) Date: Wed, 5 Dec 2018 10:53:14 +0100 Subject: HELP: Can't fix quota, it seems renamed-back mailbox doesn't exists at all In-Reply-To: References: Message-ID: <7be2d9e5-960b-e06c-0dd2-3bcd49d4b5f7@jangulo.net> On 12/4/18 5:53 PM, Heiler Bemerguy via Info-cyrus wrote: > Hail, this is a Cyrus 2.5.10-3 on Debian 9 > > A mailbox was accidentaly deleted, as the log shows: > > Dec? 4 11:56:42 localhost cyrus/imap[27354]: Rename: > user.planejamento^funbosque -> DELETED.user.planejamento^funbosque.5C0695AA > Dec? 4 11:56:42 localhost cyrus/imap[27354]: Deleted mailbox > user.planejamento^funbosque > Dec? 4 11:56:42 localhost cyrus/imap[27354]: Rename: > user.planejamento^funbosque.Drafts -> > DELETED.user.planejamento^funbosque.Drafts.5C0695AA > Dec? 4 11:56:42 localhost cyrus/imap[27354]: Deleted mailbox > user.planejamento^funbosque.Drafts > Dec? 4 11:56:42 localhost cyrus/imap[27354]: Rename: > user.planejamento^funbosque.Junk -> > DELETED.user.planejamento^funbosque.Junk.5C0695AA > Dec? 4 11:56:42 localhost cyrus/imap[27354]: Deleted mailbox > user.planejamento^funbosque.Junk > Dec? 4 11:56:42 localhost cyrus/imap[27354]: Rename: > user.planejamento^funbosque.Sent -> > DELETED.user.planejamento^funbosque.Sent.5C0695AA > Dec? 4 11:56:42 localhost cyrus/imap[27354]: Deleted mailbox > user.planejamento^funbosque.Sent > Dec? 4 11:56:42 localhost cyrus/imap[27354]: Rename: > user.planejamento^funbosque.Spam -> > DELETED.user.planejamento^funbosque.Spam.5C0695AA > Dec? 4 11:56:42 localhost cyrus/imap[27354]: Deleted mailbox > user.planejamento^funbosque.Spam > Dec? 4 11:56:42 localhost cyrus/imap[27354]: Rename: > user.planejamento^funbosque.Trash -> > DELETED.user.planejamento^funbosque.Trash.5C0695AA > Dec? 4 11:56:42 localhost cyrus/imap[27354]: Deleted mailbox > user.planejamento^funbosque.Trash > > I renamed it back to try to recover it, manually with cyradm: > > Dec? 4 12:59:38 localhost cyrus/imap[39601]: Rename: > DELETED.user.planejamento^funbosque.5C0695AA -> user.planejamento^funbosque > Dec? 4 12:59:38 localhost cyrus/imap[39601]: Deleted mailbox > DELETED.user.planejamento^funbosque.5C0695AA > Dec? 4 13:02:03 localhost cyrus/imap[39601]: Rename: > DELETED.user.planejamento^funbosque.Drafts.5C0695AA -> > user.planejamento^funbosque.Drafts > Dec? 4 13:02:03 localhost cyrus/imap[39601]: Deleted mailbox > DELETED.user.planejamento^funbosque.Drafts.5C0695AA > Dec? 4 13:02:27 localhost cyrus/imap[39601]: Rename: > DELETED.user.planejamento^funbosque.Junk.5C0695AA -> > user.planejamento^funbosque.Junk > Dec? 4 13:02:27 localhost cyrus/imap[39601]: Deleted mailbox > DELETED.user.planejamento^funbosque.Junk.5C0695AA > Dec? 4 13:02:54 localhost cyrus/imap[39601]: Rename: > DELETED.user.planejamento^funbosque.Sent.5C0695AA -> > user.planejamento^funbosque.Sent > Dec? 4 13:02:54 localhost cyrus/imap[39601]: Deleted mailbox > DELETED.user.planejamento^funbosque.Sent.5C0695AA > Dec? 4 13:03:20 localhost cyrus/imap[39601]: Rename: > DELETED.user.planejamento^funbosque.Spam.5C0695AA -> > user.planejamento^funbosque.Spam > Dec? 4 13:03:20 localhost cyrus/imap[39601]: Deleted mailbox > DELETED.user.planejamento^funbosque.Spam.5C0695AA > Dec? 4 13:04:18 localhost cyrus/imap[39601]: Rename: > DELETED.user.planejamento^funbosque.Trash.5C0695AA -> > user.planejamento^funbosque.Trash > Dec? 4 13:04:18 localhost cyrus/imap[39601]: Deleted mailbox > DELETED.user.planejamento^funbosque.Trash.5C0695AA > > All the files are there on the filesystem, but the quota usage isn't > right. I've already tried > > ? cyrus reconstruct -r -f user/planejamento^funbosque > ? cyrus quota -f user/planejamento^funbosque > ? cyrus quota -f user/planejamento.funbosque > > And a full "cyrus check" with no luck.. The command LQ simply shows > nothing. Like this: > > 127.0.0.1> lq user/planejamento.funbosque > > Any ideas?! > > root at mailer:/var/spool/cyrus/mail/p/user/planejamento^funbosque# cat > cyrus.header > Cyrus mailbox header > "The best thing about this system was that it had lots of goals." > ??????? --Jim Morris on Andrew > ??????? 3e5650f459303706 > $MDNSent $Forwarded > planejamento.funbosque? lrswipkxtecdan > root at mailer:/var/spool/cyrus/mail/p/user/planejamento^funbosque# du -hs > 570M??? . > > >From my experience (recently tested this on cyrus 3.0), quota info is removed on deletion and you have to set it again after renaming back from DELETED. Cheers, Javier. From ismael.tanguy at univ-brest.fr Wed Dec 5 06:13:43 2018 From: ismael.tanguy at univ-brest.fr (=?UTF-8?Q?Isma=c3=abl_Tanguy?=) Date: Wed, 5 Dec 2018 12:13:43 +0100 Subject: Help on reconstruct - Cyrus2.3.11 Message-ID: <6a284bf5-807f-db0d-2abe-4cee22e87acc@univ-brest.fr> Hello, this is a Cyrus 2.3.11 on Centos 5. About 5000 users for 10 To. Mail storage has been moved from NetApp NFS to FluidsFS (aka Dell Compellent NFS). Since an update on FluidFS, Imap spool undergoes daily NFS timeouts which leads to corrupt mailboxes. Typically, this begins with lines like this in /var/log/messages: Dec? 5 09:54:43 mailhost kernel: lockd: server 192.xxx.xx.xx not responding, timed out Which is followed by IOERROR for accessed mailboxes during NFS timeout: Dec? 5 09:54:47 mailhost lmtpunix[14542]: IOERROR: locking index for user.xxxx: Input/output error Dec? 5 09:54:47 mailhost imaps[21999]: IOERROR: locking header for user.xxxx.Sent: Input/output error Dec? 5 09:54:47 mailhost imaps[26935]: IOERROR: locking index for user.xxxx: Input/output error Dec? 5 09:54:47 mailhost imaps[24013]: IOERROR: locking index for user.xxxx: Input/output error Dec? 5 09:54:47 mailhost imaps[15672]: IOERROR: locking index for user.xxxx: Input/output error Dec? 5 09:54:47 mailhost imaps[3999]: IOERROR: locking index for user.xxxx: Input/output error Dec? 5 09:54:47 mailhost imaps[30671]: IOERROR: locking index for user.xxxx: Input/output error ................... Around 15 maiboxes are corrupted at each timeouts. Manually, we can repair this mailbox: * first, we have to delete all cyrus files in mailbox, if not the following reconstruct can be blocked * then, we reconstruct the mailbox (reconstruct -s user.. The downside of this method is that all messages in the reconstructed folder are marked 'Not seen'. To automate this, a Python script has been written, but sometimes not all cyrus files (cyrus.index) are recreated: Dec? 5 01:03:53 mailhost lmtpunix[497]: IOERROR: opening /var/spool/imap/x/user/xxxxxx/cyrus.index: No such file or directory Timeouts happen about 3 times per day, and cyrus deliver process is blocked when delivering to a corrupted mailbox. So my first question is : how can we reconstruct a mailbox without marking mails as not seen? And my second question is : why cyrus files are not recreated everytime? Is this due to the -s parameter with reconstruct? Any help will be appreciated. Thanks ------------------ Ismael TANGUY -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From kameryildiz at windowslive.com Wed Dec 5 11:55:32 2018 From: kameryildiz at windowslive.com (=?utf-8?B?S2FtZXIgWcSxbGTEsXo=?=) Date: Wed, 5 Dec 2018 16:55:32 +0000 Subject: Too Many imapd process Message-ID: Hi, i upgraded cyrus from 2.x to 3.0.8. cyrus working, but too many imap proccess "status: close_wait? see netstat -punta please help.. From egoitz at sarenet.es Thu Dec 6 02:06:32 2018 From: egoitz at sarenet.es (egoitz at sarenet.es) Date: Thu, 6 Dec 2018 08:06:32 +0100 Subject: Cyrus 3.0.8 longlocks and unexpunge In-Reply-To: References: <3b4bef5d55628c0af9027ed5420c6680@sarenet.es> Message-ID: <24C38791-448A-4F31-B325-48535AA57816@sarenet.es> Hi Javier!! Thanks a Lot !! I?ll try it. I was wondering if reconstruct could be running the same time as the other services... you know, if you have a medium-Big sized server... could take a long time and for avoiding service disruptions... Cheers, Egoitz, > El 5 dic 2018, a las 10:45, Javier Angulo escribi?: > > > >> On 12/5/18 8:51 AM, Egoitz Aurrekoetxea wrote: >> Good morning, >> >> >> I'm testing 3.0.8 Cyrus IMAP. I have seen that for instance, when you >> remove a couple of messages (with a vanila Roundcube webmail so IMAP, >> for instance) you cause a copy to the trash and a expunge in the INBOX. >> This operation takes very long comparing with older versions of Cyrus. I >> mean : >> >> Dec 5 08:38:42 testenv imap[88161]: login: [172.22.0.1] >> egoitz at sarenet.es PLAIN User logged in >> SESSIONID= >> Dec 5 08:38:42 testenv squatter[20782]: indexing mailbox >> user/egoitz/Papelera at sarenet.es... >> Dec 5 08:38:42 testenv imap[88160]: Expunged 2 messages from >> sarenet.es!user.egoitz >> Dec 5 08:38:42 testenv imap[88160]: Repacking mailbox >> sarenet.es!user.egoitz version 10 >> Dec 5 08:38:42 testenv squatter[20782]: Xapian committed 2 updates in >> 0.049130 sec >> Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown >> charset RCMAIL_CHARSET for text/PLAIN >> Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown >> charset RCMAIL_CHARSET for text/CALENDAR >> Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown >> charset RCMAIL_CHARSET for text/PLAIN >> Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown >> charset RCMAIL_CHARSET for text/CALENDAR >> Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown >> charset RCMAIL_CHARSET for text/PLAIN >> Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown >> charset RCMAIL_CHARSET for text/CALENDAR >> Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown >> charset RCMAIL_CHARSET for text/PLAIN >> Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown >> charset RCMAIL_CHARSET for text/CALENDAR >> *Dec 5 08:39:02 testenv imap[88160]: mailbox: longlock >> sarenet.es!user.egoitz for 19.9 seconds* >> Dec 5 08:39:02 testenv imap[88160]: USAGE egoitz at sarenet.es user: >> 8.469829 sys: 0.584660 >> >> Is this a normal or expected behavior?. >> >> By the way, I have configured : >> >> expunge_mode: delayed >> delete_mode: delayed >> deletedprefix: DELETED >> >> But now, if I do : >> >> % /usr/local/cyrus/sbin/unexpunge -lv user/egoitz at sarenet.es >> % >> >> So, no results... unexpunge gives an error for instance if the mailbox >> does not exist.... >> >> >> Why this last problem can happen?. Why it could take so long to perform >> the removal of the last commented messages?. >> > > > Hi Egoitz, > > from this line: >> Dec 5 08:38:42 testenv imap[88160]: Repacking mailbox >> sarenet.es!user.egoitz version 10 > > it seems that you have not run a reconstruct on the mailbox (should be > version 13 on cyrus 3.0) > > try: > reconstruct -V max user/egoitz at sarenet.es > > As you are probably coming from cyrus version 2.3 reconstruct is going > to take a long time ... > > I guess the unexpunge problem is also related and you need to > reconstruct the mailbox. > > Cheers, > Javier > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From egoitz at sarenet.es Thu Dec 6 02:21:04 2018 From: egoitz at sarenet.es (egoitz at sarenet.es) Date: Thu, 6 Dec 2018 08:21:04 +0100 Subject: Help on reconstruct - Cyrus2.3.11 In-Reply-To: <6a284bf5-807f-db0d-2abe-4cee22e87acc@univ-brest.fr> References: <6a284bf5-807f-db0d-2abe-4cee22e87acc@univ-brest.fr> Message-ID: <07C7ED81-6704-48FB-B842-ED8727D7559C@sarenet.es> Hi! Mate nfs, is no tan appropiate storage for Cyrus. I?d recommend you using machine local storage. Using that kind of config won?t success. Cheers, Egoitz, > El 5 dic 2018, a las 12:13, Isma?l Tanguy escribi?: > > Hello, this is a Cyrus 2.3.11 on Centos 5. > About 5000 users for 10 To. > > Mail storage has been moved from NetApp NFS to FluidsFS (aka Dell Compellent NFS). > Since an update on FluidFS, Imap spool undergoes daily NFS timeouts which leads to corrupt mailboxes. > Typically, this begins with lines like this in /var/log/messages: > > Dec 5 09:54:43 mailhost kernel: lockd: server 192.xxx.xx.xx not responding, timed out > > Which is followed by IOERROR for accessed mailboxes during NFS timeout: > > Dec 5 09:54:47 mailhost lmtpunix[14542]: IOERROR: locking index for user.xxxx: Input/output error > Dec 5 09:54:47 mailhost imaps[21999]: IOERROR: locking header for user.xxxx.Sent: Input/output error > Dec 5 09:54:47 mailhost imaps[26935]: IOERROR: locking index for user.xxxx: Input/output error > Dec 5 09:54:47 mailhost imaps[24013]: IOERROR: locking index for user.xxxx: Input/output error > Dec 5 09:54:47 mailhost imaps[15672]: IOERROR: locking index for user.xxxx: Input/output error > Dec 5 09:54:47 mailhost imaps[3999]: IOERROR: locking index for user.xxxx: Input/output error > Dec 5 09:54:47 mailhost imaps[30671]: IOERROR: locking index for user.xxxx: Input/output error > ................... > Around 15 maiboxes are corrupted at each timeouts. > Manually, we can repair this mailbox: > > first, we have to delete all cyrus files in mailbox, if not the following reconstruct can be blocked > then, we reconstruct the mailbox (reconstruct -s user.. > The downside of this method is that all messages in the reconstructed folder are marked 'Not seen'. > To automate this, a Python script has been written, but sometimes not all cyrus files (cyrus.index) are recreated: > > Dec 5 01:03:53 mailhost lmtpunix[497]: IOERROR: opening /var/spool/imap/x/user/xxxxxx/cyrus.index: No such file or directory > Timeouts happen about 3 times per day, and cyrus deliver process is blocked when delivering to a corrupted mailbox. > So my first question is : how can we reconstruct a mailbox without marking mails as not seen? > And my second question is : why cyrus files are not recreated everytime? Is this due to the -s parameter with reconstruct? > > Any help will be appreciated. > > Thanks > > ------------------ > > Ismael TANGUY > > -- > > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------- next part -------------- An HTML attachment was scrubbed... URL: From ismael.tanguy at univ-brest.fr Thu Dec 6 05:17:25 2018 From: ismael.tanguy at univ-brest.fr (=?UTF-8?Q?Isma=c3=abl_Tanguy?=) Date: Thu, 6 Dec 2018 11:17:25 +0100 Subject: Help on reconstruct - Cyrus2.3.11 In-Reply-To: <07C7ED81-6704-48FB-B842-ED8727D7559C@sarenet.es> References: <6a284bf5-807f-db0d-2abe-4cee22e87acc@univ-brest.fr> <07C7ED81-6704-48FB-B842-ED8727D7559C@sarenet.es> Message-ID: Hello, thanks for your answer. We have been using for more than 10 years Cyrus with NFS because of the snapshot. Snapshot give a way to restore mail or mailbox. It has worked like a charm until the migration. Now we're stuck on daily mailbox corruption due to this storage. We're looking, first, to a way to automate safely the reconstruct of mailbox, ideally keeping the Seen State of mails. In parrellel, we're studying the migration to Cyrus 2.4.17 without the use of NFS. Cheers, Ismael Le 06/12/2018 ? 08:21, egoitz at sarenet.es a ?crit?: > Hi! > > Mate nfs, is no tan appropiate storage for Cyrus. I?d recommend you > using machine local storage. Using that kind of config won?t success. > > Cheers, > > Egoitz, > > El 5 dic 2018, a las 12:13, Isma?l Tanguy > escribi?: > >> Hello, this is a Cyrus 2.3.11 on Centos 5. >> About 5000 users for 10 To. >> >> Mail storage has been moved from NetApp NFS to FluidsFS (aka Dell >> Compellent NFS). >> Since an update on FluidFS, Imap spool undergoes daily NFS timeouts >> which leads to corrupt mailboxes. >> Typically, this begins with lines like this in /var/log/messages: >> >> Dec? 5 09:54:43 mailhost kernel: lockd: server 192.xxx.xx.xx not responding, timed out >> >> Which is followed by IOERROR for accessed mailboxes during NFS timeout: >> >> Dec? 5 09:54:47 mailhost lmtpunix[14542]: IOERROR: locking index for user.xxxx: Input/output error >> Dec? 5 09:54:47 mailhost imaps[21999]: IOERROR: locking header for user.xxxx.Sent: Input/output error >> Dec? 5 09:54:47 mailhost imaps[26935]: IOERROR: locking index for user.xxxx: Input/output error >> Dec? 5 09:54:47 mailhost imaps[24013]: IOERROR: locking index for user.xxxx: Input/output error >> Dec? 5 09:54:47 mailhost imaps[15672]: IOERROR: locking index for user.xxxx: Input/output error >> Dec? 5 09:54:47 mailhost imaps[3999]: IOERROR: locking index for user.xxxx: Input/output error >> Dec? 5 09:54:47 mailhost imaps[30671]: IOERROR: locking index for user.xxxx: Input/output error >> >> ................... >> Around 15 maiboxes are corrupted at each timeouts. >> Manually, we can repair this mailbox: >> >> * first, we have to delete all cyrus files in mailbox, if not the >> following reconstruct can be blocked >> * then, we reconstruct the mailbox (reconstruct -s user.. >> >> The downside of this method is that all messages in the reconstructed >> folder are marked 'Not seen'. >> To automate this, a Python script has been written, but sometimes not >> all cyrus files (cyrus.index) are recreated: >> >> Dec? 5 01:03:53 mailhost lmtpunix[497]: IOERROR: opening /var/spool/imap/x/user/xxxxxx/cyrus.index: No such file or directory >> >> Timeouts happen about 3 times per day, and cyrus deliver process is >> blocked when delivering to a corrupted mailbox. >> So my first question is : how can we reconstruct a mailbox without >> marking mails as not seen? >> And my second question is : why cyrus files are not recreated >> everytime? Is this due to the -s parameter with reconstruct? >> >> Any help will be appreciated. >> >> Thanks >> >> ------------------ >> >> Ismael TANGUY >> >> -- >> >> >> >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------- next part -------------- An HTML attachment was scrubbed... URL: From Eric.Luyten at vub.ac.be Thu Dec 6 05:55:09 2018 From: Eric.Luyten at vub.ac.be (Eric Luyten) Date: Thu, 6 Dec 2018 11:55:09 +0100 Subject: Help on reconstruct - Cyrus2.3.11 In-Reply-To: References: <6a284bf5-807f-db0d-2abe-4cee22e87acc@univ-brest.fr> <07C7ED81-6704-48FB-B842-ED8727D7559C@sarenet.es> Message-ID: <4f335d2d-80ff-7c74-647b-27efd4541e6a@vub.ac.be> On 06/12/2018 11:17, Isma?l Tanguy wrote: > > Hello, > > thanks for your answer. > We have been using for more than 10 years Cyrus with NFS because of > the snapshot. > Snapshot give a way to restore mail or mailbox. > It has worked like a charm until the migration. > Now we're stuck on daily mailbox corruption due to this storage. > Isma?l, In the course of the past years there have been quite a few remarks made on this list regarding succesful use of NFS as a Cyrus mail spool. You stated yourself that the problems started when switching from one type of NFS server to another. You may want to have a close(r) look at the way you mount the NFS share(s) on the Cyrus server(s). I am looking at moving to NFS storage for (approx. 10 TB) Cyrus spool in the course of 2019 but intend to keep the metadata (header/index/cache) on an SSD pool local to the (virtual) Cyrus server. Haven't decided yet whether we'll move from 2.3 to 2.5 or 3.0, certainly not 2.4 Eric Luyten. > We're looking, first, to a way to automate safely the reconstruct of > mailbox, ideally keeping the Seen State of mails. > In parrellel, we're studying the migration to Cyrus 2.4.17 without the > use of NFS. > > Cheers, > Ismael > > > > Le 06/12/2018 ? 08:21, egoitz at sarenet.es a ?crit?: >> Hi! >> >> Mate nfs, is no tan appropiate storage for Cyrus. I?d recommend you >> using machine local storage. Using that kind of config won?t success. >> >> Cheers, >> >> Egoitz, >> >> El 5 dic 2018, a las 12:13, Isma?l Tanguy >> > >> escribi?: >> >>> Hello, this is a Cyrus 2.3.11 on Centos 5. >>> About 5000 users for 10 To. >>> >>> Mail storage has been moved from NetApp NFS to FluidsFS (aka Dell >>> Compellent NFS). >>> Since an update on FluidFS, Imap spool undergoes daily NFS timeouts >>> which leads to corrupt mailboxes. >>> Typically, this begins with lines like this in /var/log/messages: >>> >>> Dec? 5 09:54:43 mailhost kernel: lockd: server 192.xxx.xx.xx not responding, timed out >>> >>> Which is followed by IOERROR for accessed mailboxes during NFS timeout: >>> >>> Dec? 5 09:54:47 mailhost lmtpunix[14542]: IOERROR: locking index for user.xxxx: Input/output error >>> Dec? 5 09:54:47 mailhost imaps[21999]: IOERROR: locking header for user.xxxx.Sent: Input/output error >>> Dec? 5 09:54:47 mailhost imaps[26935]: IOERROR: locking index for user.xxxx: Input/output error >>> Dec? 5 09:54:47 mailhost imaps[24013]: IOERROR: locking index for user.xxxx: Input/output error >>> Dec? 5 09:54:47 mailhost imaps[15672]: IOERROR: locking index for user.xxxx: Input/output error >>> Dec? 5 09:54:47 mailhost imaps[3999]: IOERROR: locking index for user.xxxx: Input/output error >>> Dec? 5 09:54:47 mailhost imaps[30671]: IOERROR: locking index for user.xxxx: Input/output error >>> >>> ................... >>> Around 15 maiboxes are corrupted at each timeouts. >>> Manually, we can repair this mailbox: >>> >>> * first, we have to delete all cyrus files in mailbox, if not the >>> following reconstruct can be blocked >>> * then, we reconstruct the mailbox (reconstruct -s >>> user.. >>> >>> The downside of this method is that all messages in the >>> reconstructed folder are marked 'Not seen'. >>> To automate this, a Python script has been written, but sometimes >>> not all cyrus files (cyrus.index) are recreated: >>> >>> Dec? 5 01:03:53 mailhost lmtpunix[497]: IOERROR: opening /var/spool/imap/x/user/xxxxxx/cyrus.index: No such file or directory >>> >>> Timeouts happen about 3 times per day, and cyrus deliver process is >>> blocked when delivering to a corrupted mailbox. >>> So my first question is : how can we reconstruct a mailbox without >>> marking mails as not seen? >>> And my second question is : why cyrus files are not recreated >>> everytime? Is this due to the -s parameter with reconstruct? >>> >>> Any help will be appreciated. >>> >>> Thanks >>> >>> ------------------ >>> >>> Ismael TANGUY >>> >>> -- >>> >>> >>> >>> ---- >>> Cyrus Home Page: http://www.cyrusimap.org/ >>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >>> To Unsubscribe: >>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Thu Dec 6 06:40:15 2018 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Thu, 06 Dec 2018 12:40:15 +0100 Subject: Help on reconstruct - Cyrus2.3.11 In-Reply-To: <83e713ee-d751-ea3c-7b10-c34cfffe1a97@univ-brest.fr> References: <6a284bf5-807f-db0d-2abe-4cee22e87acc@univ-brest.fr> <07C7ED81-6704-48FB-B842-ED8727D7559C@sarenet.es> <83e713ee-d751-ea3c-7b10-c34cfffe1a97@univ-brest.fr> Message-ID: Hi!, I'm an experienced user too... more than 10 too :) It's just an advice... I'd recommend using Cyrus replication as an HA mech. We use an own made mech for restoring mailboxes... the snapshot causes often mailboxes to need a reconstruction. Obviously if it has worked for you perhaps something has now changed... but as a general advise... I'll tell you to use replication as ha and deletion delay for instance as a recovery method. Cheers! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 06-12-2018 10:32, Isma?l Tanguy escribi?: > Hello, > > thanks for your answer. > We have been using for more than 10 years Cyrus with NFS because of the snapshot. > Snapshot give a way to restore mail or mailbox. > It has worked like a charm until the migration. > Now we're stuck on daily mailbox corruption due to this storage. > > We're looking, first, to a way to automate safely the reconstruct of mailbox, ideally keeping the Seen State of mails. > In parrellel, we're studying the migration to Cyrus 2.4.17 without the use of NFS. > > Cheers, > Ismael > > Le 06/12/2018 ? 08:21, egoitz at sarenet.es a ?crit : Hi! > > Mate nfs, is no tan appropiate storage for Cyrus. I'd recommend you using machine local storage. Using that kind of config won't success. > > Cheers, > > Egoitz, > > El 5 dic 2018, a las 12:13, Isma?l Tanguy escribi?: > > Hello, this is a Cyrus 2.3.11 on Centos 5. > About 5000 users for 10 To. > > Mail storage has been moved from NetApp NFS to FluidsFS (aka Dell Compellent NFS). > Since an update on FluidFS, Imap spool undergoes daily NFS timeouts which leads to corrupt mailboxes. > Typically, this begins with lines like this in /var/log/messages: > > Dec 5 09:54:43 mailhost kernel: lockd: server 192.xxx.xx.xx not responding, timed out > > Which is followed by IOERROR for accessed mailboxes during NFS timeout: > > Dec 5 09:54:47 mailhost lmtpunix[14542]: IOERROR: locking index for user.xxxx: Input/output error > Dec 5 09:54:47 mailhost imaps[21999]: IOERROR: locking header for user.xxxx.Sent: Input/output error > Dec 5 09:54:47 mailhost imaps[26935]: IOERROR: locking index for user.xxxx: Input/output error > Dec 5 09:54:47 mailhost imaps[24013]: IOERROR: locking index for user.xxxx: Input/output error > Dec 5 09:54:47 mailhost imaps[15672]: IOERROR: locking index for user.xxxx: Input/output error > Dec 5 09:54:47 mailhost imaps[3999]: IOERROR: locking index for user.xxxx: Input/output error > Dec 5 09:54:47 mailhost imaps[30671]: IOERROR: locking index for user.xxxx: Input/output error > > ................... > Around 15 maiboxes are corrupted at each timeouts. > Manually, we can repair this mailbox: > > * first, we have to delete all cyrus files in mailbox, if not the following reconstruct can be blocked > * then, we reconstruct the mailbox (reconstruct -s user.. > > The downside of this method is that all messages in the reconstructed folder are marked 'Not seen'. > To automate this, a Python script has been written, but sometimes not all cyrus files (cyrus.index) are recreated: > > Dec 5 01:03:53 mailhost lmtpunix[497]: IOERROR: opening /var/spool/imap/x/user/xxxxxx/cyrus.index: No such file or directory > > Timeouts happen about 3 times per day, and cyrus deliver process is blocked when delivering to a corrupted mailbox. > So my first question is : how can we reconstruct a mailbox without marking mails as not seen? > And my second question is : why cyrus files are not recreated everytime? Is this due to the -s parameter with reconstruct? > > Any help will be appreciated. > > Thanks > > ------------------ > > Ismael TANGUY > > -- > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From ismael.tanguy at univ-brest.fr Thu Dec 6 09:50:49 2018 From: ismael.tanguy at univ-brest.fr (=?UTF-8?Q?Isma=c3=abl_Tanguy?=) Date: Thu, 6 Dec 2018 15:50:49 +0100 Subject: Help on reconstruct - Cyrus2.3.11 In-Reply-To: <4f335d2d-80ff-7c74-647b-27efd4541e6a@vub.ac.be> References: <6a284bf5-807f-db0d-2abe-4cee22e87acc@univ-brest.fr> <07C7ED81-6704-48FB-B842-ED8727D7559C@sarenet.es> <4f335d2d-80ff-7c74-647b-27efd4541e6a@vub.ac.be> Message-ID: Eric, Egoitz thank you very much, all your advices are very interesting. Egoitz, replication and delayed expunged are previewed for the next migration. I'm not sure that cyrus2.3 implements delayed expunged. Eric, here is a piece of fstab for mounting nfs: 192.168.xxx.xxx:/mailperso-fghijkl/perso /var/spool/imap-fghijkl???????? nfs rsize=262144,wsize=262144,hard,intr,proto=tcp 192.168.xxx.xxx:/mailperso-abcde/perso /var/spool/imap-abcde?????????? nfs rsize=262144,wsize=262144,hard,intr,proto=tcp 192.168.xxx.xxx:/mailperso-mnopq/perso /var/spool/imap-mnopq?????????? nfs rsize=262144,wsize=262144,hard,intr,proto=tcp 192.168.xxx.xxx/mailperso-rstuvwxyz/perso /var/spool/imap-rstuvwxyz?????? nfs rsize=262144,wsize=262144,hard,intr,proto=tcp Is your one quite the same? Why don't you want to move to Cyrus2.4, too old? keeping metadata is really a good idea, but is it possible in Cyrus 2.3? I have to verify. Thanks again Isma?l Le 06/12/2018 ? 11:55, Eric Luyten a ?crit?: > > > On 06/12/2018 11:17, Isma?l Tanguy wrote: >> >> Hello, >> >> thanks for your answer. >> We have been using for more than 10 years Cyrus with NFS because of >> the snapshot. >> Snapshot give a way to restore mail or mailbox. >> It has worked like a charm until the migration. >> Now we're stuck on daily mailbox corruption due to this storage. >> > > > Isma?l, > > > > In the course of the past years there have been quite a few remarks > made on this list regarding succesful use of NFS as a Cyrus mail spool. > > You stated yourself that the problems started when switching from one > type of NFS server to another. > You may want to have a close(r) look at the way you mount the NFS > share(s) on the Cyrus server(s). > > > I am looking at moving to NFS storage for (approx. 10 TB) Cyrus spool > in the course of 2019 but intend to keep the metadata > (header/index/cache) on an SSD pool local to the (virtual) Cyrus > server. Haven't decided yet whether we'll move from 2.3 to 2.5 or 3.0, > certainly not 2.4 > > > > Eric Luyten. > > > > >> We're looking, first, to a way to automate safely the reconstruct of >> mailbox, ideally keeping the Seen State of mails. >> In parrellel, we're studying the migration to Cyrus 2.4.17 without >> the use of NFS. >> >> Cheers, >> Ismael >> >> >> >> Le 06/12/2018 ? 08:21, egoitz at sarenet.es a ?crit?: >>> Hi! >>> >>> Mate nfs, is no tan appropiate storage for Cyrus. I?d recommend you >>> using machine local storage. Using that kind of config won?t success. >>> >>> Cheers, >>> >>> Egoitz, >>> >>> El 5 dic 2018, a las 12:13, Isma?l Tanguy >>> > >>> escribi?: >>> >>>> Hello, this is a Cyrus 2.3.11 on Centos 5. >>>> About 5000 users for 10 To. >>>> >>>> Mail storage has been moved from NetApp NFS to FluidsFS (aka Dell >>>> Compellent NFS). >>>> Since an update on FluidFS, Imap spool undergoes daily NFS timeouts >>>> which leads to corrupt mailboxes. >>>> Typically, this begins with lines like this in /var/log/messages: >>>> >>>> Dec? 5 09:54:43 mailhost kernel: lockd: server 192.xxx.xx.xx not responding, timed out >>>> >>>> Which is followed by IOERROR for accessed mailboxes during NFS timeout: >>>> >>>> Dec? 5 09:54:47 mailhost lmtpunix[14542]: IOERROR: locking index for user.xxxx: Input/output error >>>> Dec? 5 09:54:47 mailhost imaps[21999]: IOERROR: locking header for user.xxxx.Sent: Input/output error >>>> Dec? 5 09:54:47 mailhost imaps[26935]: IOERROR: locking index for user.xxxx: Input/output error >>>> Dec? 5 09:54:47 mailhost imaps[24013]: IOERROR: locking index for user.xxxx: Input/output error >>>> Dec? 5 09:54:47 mailhost imaps[15672]: IOERROR: locking index for user.xxxx: Input/output error >>>> Dec? 5 09:54:47 mailhost imaps[3999]: IOERROR: locking index for user.xxxx: Input/output error >>>> Dec? 5 09:54:47 mailhost imaps[30671]: IOERROR: locking index for user.xxxx: Input/output error >>>> >>>> ................... >>>> Around 15 maiboxes are corrupted at each timeouts. >>>> Manually, we can repair this mailbox: >>>> >>>> * first, we have to delete all cyrus files in mailbox, if not the >>>> following reconstruct can be blocked >>>> * then, we reconstruct the mailbox (reconstruct -s >>>> user.. >>>> >>>> The downside of this method is that all messages in the >>>> reconstructed folder are marked 'Not seen'. >>>> To automate this, a Python script has been written, but sometimes >>>> not all cyrus files (cyrus.index) are recreated: >>>> >>>> Dec? 5 01:03:53 mailhost lmtpunix[497]: IOERROR: opening /var/spool/imap/x/user/xxxxxx/cyrus.index: No such file or directory >>>> >>>> Timeouts happen about 3 times per day, and cyrus deliver process is >>>> blocked when delivering to a corrupted mailbox. >>>> So my first question is : how can we reconstruct a mailbox without >>>> marking mails as not seen? >>>> And my second question is : why cyrus files are not recreated >>>> everytime? Is this due to the -s parameter with reconstruct? >>>> >>>> Any help will be appreciated. >>>> >>>> Thanks >>>> >>>> ------------------ >>>> >>>> Ismael TANGUY >>>> >>>> -- >>>> >>>> >>>> >>>> ---- >>>> Cyrus Home Page: http://www.cyrusimap.org/ >>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >>>> To Unsubscribe: >>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >> >> >> ---- >> Cyrus Home Page:http://www.cyrusimap.org/ >> List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Tue Dec 11 07:58:20 2018 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Tue, 11 Dec 2018 13:58:20 +0100 Subject: Cyrus 3.0.8 longlocks and unexpunge In-Reply-To: References: <3b4bef5d55628c0af9027ed5420c6680@sarenet.es> Message-ID: Hi mate! Thank you so much!. The reconstruct solved the unexpunge issue. Could anyone know something about long locks?. Best regards, --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 05-12-2018 10:45, Javier Angulo escribi?: > On 12/5/18 8:51 AM, Egoitz Aurrekoetxea wrote: > >> Good morning, >> >> I'm testing 3.0.8 Cyrus IMAP. I have seen that for instance, when you >> remove a couple of messages (with a vanila Roundcube webmail so IMAP, >> for instance) you cause a copy to the trash and a expunge in the INBOX. >> This operation takes very long comparing with older versions of Cyrus. I >> mean : >> >> Dec 5 08:38:42 testenv imap[88161]: login: [172.22.0.1] >> egoitz at sarenet.es PLAIN User logged in >> SESSIONID= >> Dec 5 08:38:42 testenv squatter[20782]: indexing mailbox >> user/egoitz/Papelera at sarenet.es... >> Dec 5 08:38:42 testenv imap[88160]: Expunged 2 messages from >> sarenet.es!user.egoitz >> Dec 5 08:38:42 testenv imap[88160]: Repacking mailbox >> sarenet.es!user.egoitz version 10 >> Dec 5 08:38:42 testenv squatter[20782]: Xapian committed 2 updates in >> 0.049130 sec >> Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown >> charset RCMAIL_CHARSET for text/PLAIN >> Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown >> charset RCMAIL_CHARSET for text/CALENDAR >> Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown >> charset RCMAIL_CHARSET for text/PLAIN >> Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown >> charset RCMAIL_CHARSET for text/CALENDAR >> Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown >> charset RCMAIL_CHARSET for text/PLAIN >> Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown >> charset RCMAIL_CHARSET for text/CALENDAR >> Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown >> charset RCMAIL_CHARSET for text/PLAIN >> Dec 5 08:38:43 testenv imap[88160]: message_parse_charset: unknown >> charset RCMAIL_CHARSET for text/CALENDAR >> *Dec 5 08:39:02 testenv imap[88160]: mailbox: longlock >> sarenet.es!user.egoitz for 19.9 seconds* >> Dec 5 08:39:02 testenv imap[88160]: USAGE egoitz at sarenet.es user: >> 8.469829 sys: 0.584660 >> >> Is this a normal or expected behavior?. >> >> By the way, I have configured : >> >> expunge_mode: delayed >> delete_mode: delayed >> deletedprefix: DELETED >> >> But now, if I do : >> >> % /usr/local/cyrus/sbin/unexpunge -lv user/egoitz at sarenet.es >> % >> >> So, no results... unexpunge gives an error for instance if the mailbox >> does not exist.... >> >> Why this last problem can happen?. Why it could take so long to perform >> the removal of the last commented messages?. > > Hi Egoitz, > > from this line: > >> Dec 5 08:38:42 testenv imap[88160]: Repacking mailbox >> sarenet.es!user.egoitz version 10 > > it seems that you have not run a reconstruct on the mailbox (should be > version 13 on cyrus 3.0) > > try: > reconstruct -V max user/egoitz at sarenet.es > > As you are probably coming from cyrus version 2.3 reconstruct is going > to take a long time ... > > I guess the unexpunge problem is also related and you need to > reconstruct the mailbox. > > Cheers, > Javier > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From kia at solvo.ru Thu Dec 13 06:21:04 2018 From: kia at solvo.ru (Ivan Kuznetsov) Date: Thu, 13 Dec 2018 14:21:04 +0300 Subject: Cyrus-imapd 2.4.17: processes stick on mailbox locking resulting in total mailsystem failure Message-ID: <50036990-66af-bfea-900d-ae9d64428c1c@solvo.ru> Hello We had a company mail server under Oracle Linux 6 (a clone of RHEL6) with cyrus-imapd 2.3.16 working for years w/o problems. There are >9M messages in ~9500 mailboxes in two partitions. Daily mail traffic is 20-50K messages. Some weeks ago we migrated the server to a new one under Oracle Linux 7 with cyrus-imapd 2.4.17 (cyrus-imapd-2.4.17-13.el7.x86_64 rpm) and now have problem. Some times a week one of imapd processes locks an "INBOX" mailbox with corresponding /var/lib/imap/lock/user/.lock file and does not unlock it anymore. Other imapd processes trying to access this mailbox sticks waiting to obtain the lock. The bigger problem is that lmtpd processes trying to deliver new mail to this mailbox hangs too. The number of lmtpd processes is limited (maxchild=32) to limit the server load, so free lmtpd pool become empty after a time, and mail delivery stopsto all the mailboxes. MTA (postfix) mail queue blows up quickly Example lsof output: lmtpd 8182 cyrus 18uR REG 249,0 0 202944402 /var/lib/imap/lock/user/smilovsky.lock lmtpd 8183 cyrus 18uR REG 249,0 0 202944402 /var/lib/imap/lock/user/smilovsky.lock lmtpd 8187 cyrus 18uR REG 249,0 0 202944402 /var/lib/imap/lock/user/smilovsky.lock lmtpd 8771 cyrus 18uR REG 249,0 0 202944402 /var/lib/imap/lock/user/smilovsky.lock lmtpd 8834 cyrus 18uR REG 249,0 0 202944402 /var/lib/imap/lock/user/smilovsky.lock lmtpd 9123 cyrus 18uR REG 249,0 0 202944402 /var/lib/imap/lock/user/smilovsky.lock lmtpd 9631 cyrus 18uR REG 249,0 0 202944402 /var/lib/imap/lock/user/smilovsky.lock lmtpd 10343 cyrus 18uR REG 249,0 0 202944402 /var/lib/imap/lock/user/smilovsky.lock lmtpd 10437 cyrus 18uR REG 249,0 0 202944402 /var/lib/imap/lock/user/smilovsky.lock lmtpd 11411 cyrus 18uR REG 249,0 0 202944402 /var/lib/imap/lock/user/smilovsky.lock lmtpd 11413 cyrus 18uR REG 249,0 0 202944402 /var/lib/imap/lock/user/smilovsky.lock lmtpd 11506 cyrus 18uR REG 249,0 0 202944402 /var/lib/imap/lock/user/smilovsky.lock [root at hippo4 bin]# lsof /var/lib/imap/lock/user/smilovsky.lock | grep '^imapd' imapd 9132 cyrus 19uR REG 249,0 0 202944402 /var/lib/imap/lock/user/smilovsky.lock imapd 9154 cyrus 19uR REG 249,0 0 202944402 /var/lib/imap/lock/user/smilovsky.lock imapd 10311 cyrus 19uR REG 249,0 0 202944402 /var/lib/imap/lock/user/smilovsky.lock imapd 10746 cyrus 19uR REG 249,0 0 202944402 /var/lib/imap/lock/user/smilovsky.lock imapd 11843 cyrus 19uR REG 249,0 0 202944402 /var/lib/imap/lock/user/smilovsky.lock imapd 12059 cyrus 19uR REG 249,0 0 202944402 /var/lib/imap/lock/user/smilovsky.lock imapd 21885 cyrus 19uR REG 249,0 0 202944402 /var/lib/imap/lock/user/smilovsky.lock In this case the root cause imapd process is 21885. strace shows that the process waits on a futex forever: [root at hippo4 bin]# strace -tt -f -p 21885 strace: Process 21885 attached 13:07:28.264713 futex(0x7f6dcc5ac760, FUTEX_WAIT_PRIVATE, 2, NULL^Cstrace: Process 21885 detached Killing the process frees the lock, all the other stuck processes runs and the problem disappears in a few seconds. Investigation shows that this process has a long-term connection from a user MUA (Thunderbird) and deletes mail messages time-to-time. This may be a Thunderbird 'filter' thread [root at hippo4 bin]# grep 'imap\[21885\]' /var/log/maillog [...] Dec 13 09:59:11 hippo4 imap[21885]: starttls: TLSv1.2 with cipher DHE-RSA-AES256-SHA (256/256 bits new) no authentication Dec 13 09:59:11 hippo4 imap[21885]: login: thunder.solvo.ru [172.16.85.69] smilovsky CRAM-MD5+TLS User logged in SESSIONID= Dec 13 09:59:11 hippo4 imap[21885]: client id: "name" "Thunderbird" "version" "52.5.2" Dec 13 09:59:11 hippo4 imap[21885]: Expunged 1 messages from user.smilovsky Dec 13 09:59:12 hippo4 imap[21885]: Expunged 1 messages from user.smilovsky Dec 13 09:59:12 hippo4 imap[21885]: Expunged 1 messages from user.smilovsky Dec 13 09:59:12 hippo4 imap[21885]: Expunged 1 messages from user.smilovsky Dec 13 10:01:45 hippo4 imap[21885]: Expunged 1 messages from user.smilovsky Dec 13 10:01:53 hippo4 imap[21885]: Expunged 1 messages from user.smilovsky Dec 13 10:03:17 hippo4 imap[21885]: Expunged 1 messages from user.smilovsky Dec 13 10:09:06 hippo4 imap[21885]: Expunged 1 messages from user.smilovsky Dec 13 10:09:19 hippo4 imap[21885]: Expunged 1 messages from user.smilovsky Problem is sporadic, I didn't see any correlation with server load, time of day, user name and so on Removing "maxchild=" parameter for lmtp just delay the final as the server system resources are limited. Now I wrote a patrol script counting locks and killing imapd if locks number is growing up. The trouble is I can't reliably determine the root cause imapd and kill them one-by-one Any ideas more? Ivan Kuznetsov, SOLVO ltd From pkoch at bgc-jena.mpg.de Thu Dec 13 07:04:48 2018 From: pkoch at bgc-jena.mpg.de (Dr. Peer-Joachim Koch) Date: Thu, 13 Dec 2018 13:04:48 +0100 Subject: Missing sieve folder Message-ID: Hi, we moved from an old cyrus system to a newer one (2.14.19) ~18 months ago. The system runs very smooth. However I noticed a couple of problems: 1) We are creating a user using a script. The script is executing on the mail system some commands to create the mailboxes and the aliases. I found one case where 2 mailboxes where created "user.username" and "user.Username". The folders for "Username" where missing, so it gave some IOERROR. Any idea how this can happen (username is converted to lower case BEFORE the script on the mail system is executed) ? 2)For a large number of people the sieve folders are either empty or not existing! Dec 11 17:18:53 mail lmtpunix[687]: IOERROR: fstating sieve script /var/lib/sieve/m/mxxx/defaultbc: No such file or directory Dec 11 17:18:54 mail lmtpunix[1692]: IOERROR: fstating sieve script /var/lib/sieve/m/myyyy/defaultbc: No such file or directory Dec 11 17:18:54 mail lmtpunix[1680]: IOERROR: fstating sieve script /var/lib/sieve/m/mzzzz/defaultbc: No such file or directory ll /var/lib/sieve/m/mxxx/defaultbc ls: cannot access '/var/lib/sieve/m/mxxx/defaultbc': No such file or directory mail:/tmp # ll /var/lib/sieve/m/mxxx/ ls: cannot access '/var/lib/sieve/m/mxxx/': No such file or directory How can this happen ? What is the pest way to fix it ? -- Bye, Peer ________________________________________________________ Max-Planck-Institut f?r Biogeochemie Dr. Peer-Joachim Koch Hans-Kn?ll Str.10 Telefon: ++49 3641 57-6705 D-07745 Jena Telefax: ++49 3641 57-7705 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5224 bytes Desc: S/MIME Cryptographic Signature URL: From jwade at oakton.edu Thu Dec 13 09:34:01 2018 From: jwade at oakton.edu (John Wade) Date: Thu, 13 Dec 2018 08:34:01 -0600 Subject: Cyrus-imapd 2.4.17: processes stick on mailbox locking resulting in total mailsystem failure In-Reply-To: <50036990-66af-bfea-900d-ae9d64428c1c@solvo.ru> References: <50036990-66af-bfea-900d-ae9d64428c1c@solvo.ru> Message-ID: <87554ee5-c872-eea4-7ef9-cd31f984c7bf@oakton.edu> Without running gdb on the process, I have no idea, but your problem sounds similar to something we hit a very long time ago: See https://www.oakton.edu/user/3/jwade/cyrus/Readme.html In our cases, the problem was the imapd process that was holding the lock was trying to obtain a second lock on the same file.?? What does a stack trace look like on the imapd process that is holding the lock????? It would appear the lock process has changed since I last looked at this, so this may not be a help at all. Good luck, John On 12/13/2018 5:21 AM, Ivan Kuznetsov wrote: > Hello > > We had a company mail server under Oracle Linux 6 (a clone of RHEL6) > with cyrus-imapd 2.3.16 working for years w/o problems. There are >9M > messages in ~9500 mailboxes in two partitions. Daily mail traffic is > 20-50K messages. > > Some weeks ago we migrated the server to a new one under Oracle Linux > 7 with cyrus-imapd 2.4.17 (cyrus-imapd-2.4.17-13.el7.x86_64 rpm) and > now have problem. Some times a week one of imapd processes locks an > "INBOX" mailbox with corresponding /var/lib/imap/lock/user/.lock > file and does not unlock it anymore. Other imapd processes trying to > access this mailbox sticks waiting to obtain the lock. The bigger > problem is that lmtpd processes trying to deliver new mail to this > mailbox hangs too. The number of lmtpd processes is limited > (maxchild=32) to limit the server load, so free lmtpd pool become > empty after a time, and mail delivery stopsto all the mailboxes. MTA > (postfix) mail queue blows up quickly > > Example lsof output: > > lmtpd?? 8182 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 > /var/lib/imap/lock/user/smilovsky.lock > lmtpd?? 8183 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 > /var/lib/imap/lock/user/smilovsky.lock > lmtpd?? 8187 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 > /var/lib/imap/lock/user/smilovsky.lock > lmtpd?? 8771 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 > /var/lib/imap/lock/user/smilovsky.lock > lmtpd?? 8834 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 > /var/lib/imap/lock/user/smilovsky.lock > lmtpd?? 9123 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 > /var/lib/imap/lock/user/smilovsky.lock > lmtpd?? 9631 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 > /var/lib/imap/lock/user/smilovsky.lock > lmtpd?? 10343 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 > /var/lib/imap/lock/user/smilovsky.lock > lmtpd?? 10437 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 > /var/lib/imap/lock/user/smilovsky.lock > lmtpd?? 11411 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 > /var/lib/imap/lock/user/smilovsky.lock > lmtpd?? 11413 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 > /var/lib/imap/lock/user/smilovsky.lock > lmtpd?? 11506 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 > /var/lib/imap/lock/user/smilovsky.lock > > [root at hippo4 bin]# lsof /var/lib/imap/lock/user/smilovsky.lock | grep > '^imapd' > imapd??? 9132 cyrus?? 19uR? REG? 249,0??????? 0 202944402 > /var/lib/imap/lock/user/smilovsky.lock > imapd??? 9154 cyrus?? 19uR? REG? 249,0??????? 0 202944402 > /var/lib/imap/lock/user/smilovsky.lock > imapd?? 10311 cyrus?? 19uR? REG? 249,0??????? 0 202944402 > /var/lib/imap/lock/user/smilovsky.lock > imapd?? 10746 cyrus?? 19uR? REG? 249,0??????? 0 202944402 > /var/lib/imap/lock/user/smilovsky.lock > imapd?? 11843 cyrus?? 19uR? REG? 249,0??????? 0 202944402 > /var/lib/imap/lock/user/smilovsky.lock > imapd?? 12059 cyrus?? 19uR? REG? 249,0??????? 0 202944402 > /var/lib/imap/lock/user/smilovsky.lock > imapd?? 21885 cyrus?? 19uR? REG? 249,0??????? 0 202944402 > /var/lib/imap/lock/user/smilovsky.lock > > In this case the root cause imapd process is 21885. strace shows that > the process waits on a futex forever: > > [root at hippo4 bin]# strace -tt -f -p 21885 > strace: Process 21885 attached > 13:07:28.264713 futex(0x7f6dcc5ac760, FUTEX_WAIT_PRIVATE, 2, > NULL^Cstrace: Process 21885 detached > ? > > Killing the process frees the lock, all the other stuck processes runs > and the problem disappears in a few seconds. > > Investigation shows that this process has a long-term connection from > a user MUA (Thunderbird) and deletes mail messages time-to-time. This > may be a Thunderbird 'filter' thread > > [root at hippo4 bin]# grep 'imap\[21885\]' /var/log/maillog > [...] > Dec 13 09:59:11 hippo4 imap[21885]: starttls: TLSv1.2 with cipher > DHE-RSA-AES256-SHA (256/256 bits new) no authentication > Dec 13 09:59:11 hippo4 imap[21885]: login: thunder.solvo.ru > [172.16.85.69] smilovsky CRAM-MD5+TLS User logged in > SESSIONID= > Dec 13 09:59:11 hippo4 imap[21885]: client id: "name" "Thunderbird" > "version" "52.5.2" > Dec 13 09:59:11 hippo4 imap[21885]: Expunged 1 messages from > user.smilovsky > Dec 13 09:59:12 hippo4 imap[21885]: Expunged 1 messages from > user.smilovsky > Dec 13 09:59:12 hippo4 imap[21885]: Expunged 1 messages from > user.smilovsky > Dec 13 09:59:12 hippo4 imap[21885]: Expunged 1 messages from > user.smilovsky > Dec 13 10:01:45 hippo4 imap[21885]: Expunged 1 messages from > user.smilovsky > Dec 13 10:01:53 hippo4 imap[21885]: Expunged 1 messages from > user.smilovsky > Dec 13 10:03:17 hippo4 imap[21885]: Expunged 1 messages from > user.smilovsky > Dec 13 10:09:06 hippo4 imap[21885]: Expunged 1 messages from > user.smilovsky > Dec 13 10:09:19 hippo4 imap[21885]: Expunged 1 messages from > user.smilovsky > > Problem is sporadic, I didn't see any correlation with server load, > time of day, user name and so on > > Removing "maxchild=" parameter for lmtp just delay the final as the > server system resources are limited. Now I wrote a patrol script > counting locks and killing imapd if locks number is growing up. The > trouble is I can't reliably determine the root cause imapd and kill > them one-by-one > > Any ideas more? > > Ivan Kuznetsov, SOLVO ltd > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From kia at solvo.ru Thu Dec 13 09:50:02 2018 From: kia at solvo.ru (Ivan Kuznetsov) Date: Thu, 13 Dec 2018 17:50:02 +0300 Subject: Cyrus-imapd 2.4.17: processes stick on mailbox locking resulting in total mailsystem failure In-Reply-To: <87554ee5-c872-eea4-7ef9-cd31f984c7bf@oakton.edu> References: <50036990-66af-bfea-900d-ae9d64428c1c@solvo.ru> <87554ee5-c872-eea4-7ef9-cd31f984c7bf@oakton.edu> Message-ID: <69869d48-3eb4-ec65-99bc-6de60505c9df@solvo.ru> Jonk, thank you for the idea. Somewhat looks strange as old mail server worked w/o this problem 5+ years. But the system environment changed dramatically, may be some filesystem quircks are significant for this locks... I will try gdb'ing the process when problem occurs once more 13.12.2018 17:34, John Wade ?????: > Without running gdb on the process, I have no idea, but your problem > sounds similar to something we hit a very long time ago: > > See https://www.oakton.edu/user/3/jwade/cyrus/Readme.html > > In our cases, the problem was the imapd process that was holding the > lock was trying to obtain a second lock on the same file.?? What does a > stack trace look like on the imapd process that is holding the lock? It > would appear the lock process has changed since I last looked at this, > so this may not be a help at all. > > Good luck, > John > > > > On 12/13/2018 5:21 AM, Ivan Kuznetsov wrote: >> Hello >> >> We had a company mail server under Oracle Linux 6 (a clone of RHEL6) >> with cyrus-imapd 2.3.16 working for years w/o problems. There are >9M >> messages in ~9500 mailboxes in two partitions. Daily mail traffic is >> 20-50K messages. >> >> Some weeks ago we migrated the server to a new one under Oracle Linux >> 7 with cyrus-imapd 2.4.17 (cyrus-imapd-2.4.17-13.el7.x86_64 rpm) and >> now have problem. Some times a week one of imapd processes locks an >> "INBOX" mailbox with corresponding /var/lib/imap/lock/user/.lock >> file and does not unlock it anymore. Other imapd processes trying to >> access this mailbox sticks waiting to obtain the lock. The bigger >> problem is that lmtpd processes trying to deliver new mail to this >> mailbox hangs too. The number of lmtpd processes is limited >> (maxchild=32) to limit the server load, so free lmtpd pool become >> empty after a time, and mail delivery stopsto all the mailboxes. MTA >> (postfix) mail queue blows up quickly >> >> Example lsof output: >> >> lmtpd?? 8182 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 >> /var/lib/imap/lock/user/smilovsky.lock >> lmtpd?? 8183 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 >> /var/lib/imap/lock/user/smilovsky.lock >> lmtpd?? 8187 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 >> /var/lib/imap/lock/user/smilovsky.lock >> lmtpd?? 8771 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 >> /var/lib/imap/lock/user/smilovsky.lock >> lmtpd?? 8834 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 >> /var/lib/imap/lock/user/smilovsky.lock >> lmtpd?? 9123 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 >> /var/lib/imap/lock/user/smilovsky.lock >> lmtpd?? 9631 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 >> /var/lib/imap/lock/user/smilovsky.lock >> lmtpd?? 10343 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 >> /var/lib/imap/lock/user/smilovsky.lock >> lmtpd?? 10437 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 >> /var/lib/imap/lock/user/smilovsky.lock >> lmtpd?? 11411 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 >> /var/lib/imap/lock/user/smilovsky.lock >> lmtpd?? 11413 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 >> /var/lib/imap/lock/user/smilovsky.lock >> lmtpd?? 11506 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 >> /var/lib/imap/lock/user/smilovsky.lock >> >> [root at hippo4 bin]# lsof /var/lib/imap/lock/user/smilovsky.lock | grep >> '^imapd' >> imapd??? 9132 cyrus?? 19uR? REG? 249,0??????? 0 202944402 >> /var/lib/imap/lock/user/smilovsky.lock >> imapd??? 9154 cyrus?? 19uR? REG? 249,0??????? 0 202944402 >> /var/lib/imap/lock/user/smilovsky.lock >> imapd?? 10311 cyrus?? 19uR? REG? 249,0??????? 0 202944402 >> /var/lib/imap/lock/user/smilovsky.lock >> imapd?? 10746 cyrus?? 19uR? REG? 249,0??????? 0 202944402 >> /var/lib/imap/lock/user/smilovsky.lock >> imapd?? 11843 cyrus?? 19uR? REG? 249,0??????? 0 202944402 >> /var/lib/imap/lock/user/smilovsky.lock >> imapd?? 12059 cyrus?? 19uR? REG? 249,0??????? 0 202944402 >> /var/lib/imap/lock/user/smilovsky.lock >> imapd?? 21885 cyrus?? 19uR? REG? 249,0??????? 0 202944402 >> /var/lib/imap/lock/user/smilovsky.lock >> >> In this case the root cause imapd process is 21885. strace shows that >> the process waits on a futex forever: >> >> [root at hippo4 bin]# strace -tt -f -p 21885 >> strace: Process 21885 attached >> 13:07:28.264713 futex(0x7f6dcc5ac760, FUTEX_WAIT_PRIVATE, 2, >> NULL^Cstrace: Process 21885 detached >> ? >> >> Killing the process frees the lock, all the other stuck processes runs >> and the problem disappears in a few seconds. >> >> Investigation shows that this process has a long-term connection from >> a user MUA (Thunderbird) and deletes mail messages time-to-time. This >> may be a Thunderbird 'filter' thread >> >> [root at hippo4 bin]# grep 'imap\[21885\]' /var/log/maillog >> [...] >> Dec 13 09:59:11 hippo4 imap[21885]: starttls: TLSv1.2 with cipher >> DHE-RSA-AES256-SHA (256/256 bits new) no authentication >> Dec 13 09:59:11 hippo4 imap[21885]: login: thunder.solvo.ru >> [172.16.85.69] smilovsky CRAM-MD5+TLS User logged in >> SESSIONID= >> Dec 13 09:59:11 hippo4 imap[21885]: client id: "name" "Thunderbird" >> "version" "52.5.2" >> Dec 13 09:59:11 hippo4 imap[21885]: Expunged 1 messages from >> user.smilovsky >> Dec 13 09:59:12 hippo4 imap[21885]: Expunged 1 messages from >> user.smilovsky >> Dec 13 09:59:12 hippo4 imap[21885]: Expunged 1 messages from >> user.smilovsky >> Dec 13 09:59:12 hippo4 imap[21885]: Expunged 1 messages from >> user.smilovsky >> Dec 13 10:01:45 hippo4 imap[21885]: Expunged 1 messages from >> user.smilovsky >> Dec 13 10:01:53 hippo4 imap[21885]: Expunged 1 messages from >> user.smilovsky >> Dec 13 10:03:17 hippo4 imap[21885]: Expunged 1 messages from >> user.smilovsky >> Dec 13 10:09:06 hippo4 imap[21885]: Expunged 1 messages from >> user.smilovsky >> Dec 13 10:09:19 hippo4 imap[21885]: Expunged 1 messages from >> user.smilovsky >> >> Problem is sporadic, I didn't see any correlation with server load, >> time of day, user name and so on >> >> Removing "maxchild=" parameter for lmtp just delay the final as the >> server system resources are limited. Now I wrote a patrol script >> counting locks and killing imapd if locks number is growing up. The >> trouble is I can't reliably determine the root cause imapd and kill >> them one-by-one >> >> Any ideas more? >> >> Ivan Kuznetsov, SOLVO ltd >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > -- ? ?????????, ???? ???????? ???????????? ???????????? ?????? ???????? "?????" +7(812)60-60-555 +7(495)66-83-003 +7(921)740-72-61 http://www.solvo.ru From egoitz at sarenet.es Thu Dec 13 10:52:28 2018 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Thu, 13 Dec 2018 16:52:28 +0100 Subject: Question for upgrading Message-ID: <9e71e82ddcba5500a175c0e447131413@sarenet.es> Good afternoon, I was trying to upgrade part of our Cyrus imap installation, concretely that one consisting in still 2.3. I was planning to set up Cyrus 3.0. I have seen all works properly except for the unexpunge command because as someone stated here, a reconstruct -V max was needed.The problem is that this reconstruct command, takes ages and I'm not able to keep the service offline so many time. So I have been thinking in the following scenario : - Cyrus 2.3 master -> Cyrus 2.4 slave Get this 2.4 slave ready and set it as master. But here comes my first doubt. Does the 2.4 replication work with the 2.3 replication?. Can in this pair, both (the 2.3 and the 2.4) be both master and slave?. I mean to switch roles in the pair. Make one become master and the other slave and vice versa?. Let's think now Cyrus 2.4 is ready and working. - Now, I would set up a new 3.0 slave. I know 2.4 could replicate with 3.0. So I would get the 3.0 ready and then set 3.0 as master. Can in this pair both the 2.4 and 3.0 be master and slave?. Meaning again to the same role switching commented before... to make one to be master and the other slave or vice versa.... I'l will end up with 2 3.0 master and slave... but I need to trace the path... Does anyone see any other way?. Best regards, - -- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Thu Dec 13 13:25:40 2018 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Thu, 13 Dec 2018 19:25:40 +0100 Subject: Question for upgrading In-Reply-To: <9e71e82ddcba5500a175c0e447131413@sarenet.es> References: <9e71e82ddcba5500a175c0e447131413@sarenet.es> Message-ID: <5ed0a3707ca4b486ede31779d1cbf4da@sarenet.es> Hi again! Else as a simplication way can you replicate any manner a 2.3 with some newer version?. At least in manual mode (not rolling)?. Cheers. --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 13-12-2018 16:52, Egoitz Aurrekoetxea escribi?: > Good afternoon, > > I was trying to upgrade part of our Cyrus imap installation, concretely that one consisting in still 2.3. I was planning to set up Cyrus 3.0. I have seen all works properly except for the unexpunge command because as someone stated here, a reconstruct -V max was needed.The problem is that this reconstruct command, takes ages and I'm not able to keep the service offline so many time. So I have been thinking in the following scenario : > > - Cyrus 2.3 master -> Cyrus 2.4 slave > > Get this 2.4 slave ready and set it as master. But here comes my first doubt. Does the 2.4 replication work with the 2.3 replication?. Can in this pair, both (the 2.3 and the 2.4) be both master and slave?. I mean to switch roles in the pair. Make one become master and the other slave and vice versa?. > > Let's think now Cyrus 2.4 is ready and working. > > - Now, I would set up a new 3.0 slave. I know 2.4 could replicate with 3.0. So I would get the 3.0 ready and then set 3.0 as master. Can in this pair both the 2.4 and 3.0 be master and slave?. Meaning again to the same role switching commented before... to make one to be master and the other slave or vice versa.... > > I'l will end up with 2 3.0 master and slave... but I need to trace the path... > > Does anyone see any other way?. > > Best regards, > > - > > -- > > EGOITZ AURREKOETXEA > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es [1] > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From egoitz at sarenet.es Fri Dec 14 10:11:51 2018 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Fri, 14 Dec 2018 16:11:51 +0100 Subject: Question for upgrading In-Reply-To: <5ed0a3707ca4b486ede31779d1cbf4da@sarenet.es> References: <9e71e82ddcba5500a175c0e447131413@sarenet.es> <5ed0a3707ca4b486ede31779d1cbf4da@sarenet.es> Message-ID: <2715a40ab7c773b132308983d1aed5a0@sarenet.es> Common Bron, Ellie, mates :) any advise? Cheers :) :) --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 13-12-2018 19:25, Egoitz Aurrekoetxea escribi?: > Hi again! > > Else as a simplication way can you replicate any manner a 2.3 with some newer version?. At least in manual mode (not rolling)?. > > Cheers. > > --- > > EGOITZ AURREKOETXEA > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es [1] > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. > > El 13-12-2018 16:52, Egoitz Aurrekoetxea escribi?: > >> Good afternoon, >> >> I was trying to upgrade part of our Cyrus imap installation, concretely that one consisting in still 2.3. I was planning to set up Cyrus 3.0. I have seen all works properly except for the unexpunge command because as someone stated here, a reconstruct -V max was needed.The problem is that this reconstruct command, takes ages and I'm not able to keep the service offline so many time. So I have been thinking in the following scenario : >> >> - Cyrus 2.3 master -> Cyrus 2.4 slave >> >> Get this 2.4 slave ready and set it as master. But here comes my first doubt. Does the 2.4 replication work with the 2.3 replication?. Can in this pair, both (the 2.3 and the 2.4) be both master and slave?. I mean to switch roles in the pair. Make one become master and the other slave and vice versa?. >> >> Let's think now Cyrus 2.4 is ready and working. >> >> - Now, I would set up a new 3.0 slave. I know 2.4 could replicate with 3.0. So I would get the 3.0 ready and then set 3.0 as master. Can in this pair both the 2.4 and 3.0 be master and slave?. Meaning again to the same role switching commented before... to make one to be master and the other slave or vice versa.... >> >> I'l will end up with 2 3.0 master and slave... but I need to trace the path... >> >> Does anyone see any other way?. >> >> Best regards, >> >> - >> >> -- >> >> EGOITZ AURREKOETXEA >> Departamento de sistemas >> 944 209 470 >> Parque Tecnol?gico. Edificio 103 >> 48170 Zamudio (Bizkaia) >> egoitz at sarenet.es >> www.sarenet.es [1] >> Antes de imprimir este correo electr?nico piense si es necesario hacerlo. >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From nic at onlight.com Sat Dec 15 11:05:29 2018 From: nic at onlight.com (Nic Bernstein) Date: Sat, 15 Dec 2018 10:05:29 -0600 Subject: Question for upgrading In-Reply-To: <9e71e82ddcba5500a175c0e447131413@sarenet.es> References: <9e71e82ddcba5500a175c0e447131413@sarenet.es> Message-ID: <07899a02-1bc4-8f47-8f47-7cc0d4f3f7ee@onlight.com> On 12/13/18 9:52 AM, Egoitz Aurrekoetxea wrote: > > I was trying to upgrade part of our Cyrus imap installation, > concretely that one consisting in still 2.3. I was planning to set up > Cyrus 3.0. I have seen all works properly except for the unexpunge > command because as someone stated here, a reconstruct -V max was > needed.The problem is that this reconstruct command, takes ages and > I'm not able to keep the service offline so many time. So I have been > thinking in the following scenario : > Egoitz, As long as you've followed all of the various steps needed to account for version changes, as outlined in the Release Notes for /all/ intermediary releases, then the last step should be the updating of the indexes.? Rather than running "reconstruct -V max" on all mailboxes at once, simply run it on subsets.? We've done this on large installations without ill effect.? You can have the service on line during the reconstruct, and, as you note, have most all functionality present. We build a list of users (mboxlist.txt), and then run a cron job, during off-peak hours, using the 'parallel' command like so (from 'crontab -e -u cyrus'): ### ## Start a batch of recursive reconstruct jobs at 7PM and discontinue ## at 6:53AM. 0 19 * * * parallel -j 5 --resume --joblog /var/lib/imap/reconstruct.log cyrus reconstruct -G -V max -r -u {} < /var/lib/imap/mboxlist.txt 53 06 * * * kill -TERM `ps ax | grep [p]arallel | awk '{print $1}'` Cheers, ??? -nic -- Nic Bernstein nic at onlight.com Onlight Inc. www.onlight.com 6525 W Bluemound Rd., Ste 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 f. 414.290.0335 -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at binarus.de Sun Dec 16 04:04:58 2018 From: lists at binarus.de (Binarus) Date: Sun, 16 Dec 2018 10:04:58 +0100 Subject: Running a script with cyradm throwing ReadLine errors Message-ID: Dear all, I was just trying to explore cyradm a little bit further and hence was experimenting with its scripting capabilities. Having cyradm run a script should be pretty easy. man cyradm tells us: perl -MCyrus::IMAP::Shell -e 'run("myscript")' So I created the simplest possible script (that means an empty one) and tried to run it: touch 000 chmod a+x 000 (just in case ...) perl -MCyrus::IMAP::Shell -e 'run("000")' The only thing I got was: Use of uninitialized value within @layers in string eq at /usr/local/lib/x86_64-linux-gnu/perl/5.24.1/Term/ReadLine/Gnu.pm line 280. Bad filehandle: __DATA__ at /usr/local/lib/x86_64-linux-gnu/perl/5.24.1/Term/ReadLine/Gnu.pm line 769. Putting something meaningful into the script did not change the situation. I have googled and read documentation (mainly on cyrusimapd.org) for several hours, but could not find the reason for the problem. I even have put allowplaintext=yes into imapd.conf and restarted imapd (knowing that this probably wasn't very smart, but the term "layers" in the error message made me mistrustful because there are authentication "layers", and I don't have any problems with Term::ReadLine::Gnu in general). As expected, this didn't change the situation either. This happened with 2.4.16 as well as with 2.5.10. Could anybody please tell me what I might do wrong here? Thank you very much in advance, Binarus From lists at binarus.de Sun Dec 16 04:38:53 2018 From: lists at binarus.de (Binarus) Date: Sun, 16 Dec 2018 10:38:53 +0100 Subject: Feature request: Add wildcard capabilities to cyradm's xfer command or make it handle public namespaces reasonably Message-ID: <492a94fe-b24f-b9e8-f301-83441c127116@binarus.de> Dear all, yesterday, I have retired a server running 2.4.16 and have transferred the folder structure and all messages to a new server running 2.5.10. In the past, I have used imapsync to do that sort of transfers, which worked reliably. But this time, I decided to prefer an "original vendor tool"; hence, I used cyradm and its xfer command. After having fiddled around with the configuration files on the old and the new server, I could move the user mailboxes without any problem (all mailboxes whose name begins with "user."). This process went smoothly and worked reasonably fast, and (the most important thing for me) all subfolders in each mailbox have been moved recursively. The problem: I don't have too many users / mailboxes, but a huge and well-crafted (in terms of folder structure and permissions) public folder (namespace) which is an important collaboration tool. It did not seem possible to relocate the public folder in a reasonable manner using cyradm's xfer command. Obviously, it is able to relocate a single folder from one public namespace to another, but it does not relocate this folder's subfolders. Unfortunately, in contrast to some other of cyradm's commands, xfer does not seem to support wildcards either. This was the moment when I thought about stopping using cyradm for this task and using imapsync again. But I am a bit stubborn sometimes, so I went on and ended up with manually moving several thousands of folders (actually, it was semi-automatic: from within cyradm, I issued an lm command, copied a part of the listing from the terminal window into a text editor, generated the respective commands there (using macros within the editor), pasted the commands back into the terminal window, waited until it had finished, and restarted that process). This is ugly and silly. Hence the question: Could we extend cyradm's xfer command so that we could give wildcards to it, like, for example, cyradm's lm command? Or (more elegant and desirable) so that it recursively includes all subfolders when it relocates a folder, even when this folder is in the public namespace? Thank you very much for any thoughts, Binarus P.S. Please see my next messages for reasons why I didn't write a Perl script using Cyrus::IMAP:Admin for this task. From lists at binarus.de Sun Dec 16 05:06:16 2018 From: lists at binarus.de (Binarus) Date: Sun, 16 Dec 2018 11:06:16 +0100 Subject: Why do we need to run reconstruct after having moved mailboxes using cyradm's xfer command? Message-ID: Dear all, yesterday, I have moved a bunch of user mailboxes and public folders from a server running 2.4.16 to another server running 2.5.10, using cyradm's xfer command (see my previous messages). After having finished the migration, I noticed a weird behavior in Thunderbird (which is our standard email client): When trying to move a message from the Inbox to the Junk or Trash folder, the message disappeared from the Inbox for a short time, then reappeared. The log files on the server were showing the dreaded entries: Dec 16 01:12:59 morn cyrus/imaps[14914]: Fatal error: Internal error: assertion failed: imap/mailbox.c: 2850: !message_guid_isnull(&record->guid) "Fortunately", a lot of other users are affected by such log entries and weird email client behavior as well, so finding the solution on the 'net was not too difficult: Running reconstruct did not lead to anywhere, but running reconstruct -V max was the solution. This lets me scratch my head: In the past, I have upgraded Cyrus imapd at least three times, each time using imapsync (instead of cyradm's xfer command) to move the mailboxes and the public namespace (including all messages and subfolders) from the old server to the new one. In none of these cases, it was necessary to reconstruct anything afterwards. This would have been illogical anyway: Each time, the new server had been setup from scratch, and no mailboxes / messages / folders / subfolders had been moved directly via file transfer from the old to the new server. Now that I have used cyradm's xfer command to relocate the mailboxes / messages / folders / subfolders, I surprisingly had to run the "heaviest" form of reconstruct (-V max). Could anybody please shortly explain why? What exactly are the techniques and mechanisms cyradm's xfer uses to do its thing? I had been quite sure that it just uses the IMAP protocol, but there seems to be more to it ... Thank you very much for any insight, Binarus From lists at binarus.de Sun Dec 16 05:55:27 2018 From: lists at binarus.de (Binarus) Date: Sun, 16 Dec 2018 11:55:27 +0100 Subject: Documentation for Cyrus::IMAP::Admin and friends Message-ID: <4ee8cab3-5348-ae25-a9b1-a7854601b7e4@binarus.de> Dear all, as described in my previous messages, I recently had a hard time relocating (i.e. moving) mailboxes and the public namespace from 2.4.16 to a new server running 2.5.10. I would have happily written a Perl script which surely had solved much of my problems, but I couldn't do that due to the lack of documentation for the respective Perl modules. This is best explained by example: I have found a Perl script on the 'net which looks trustworthy at the first sight and which contains (among others) the following lines of code: use Cyrus::IMAP::Admin; # Connect to Cyrus $imap = Cyrus::IMAP::Admin->new("my_server") || die "Unable to connect to my_server"; $imap->authenticate(-user => "foo", -mechanism => "LOGIN", -password => "password", ); I wanted to understand exactly what the script does and thus tried to find Cyrus::IMAP::Admin's documentation. So I issued man Cyrus::IMAP::Admin and found that some methods are documented, but not the authenticate() method which is needed first and one of the most important ones. However, the man page states (in the section about the "new()" method): "Instantiates a cyradm object. This is in fact an Cyrus::IMAP object with a few additional methods, so all Cyrus::IMAP methods are available if needed. (In particular, you will always want to use the "authenticate" method.)" This tricked me into believing that the "authenticate" method would be part of Cyrus::IMAP and would be explained in that module's documentation, so I issued man Cyrus::IMAP Well, this man page "documents" the method by exactly one line (identically in two places): $client->authenticate; It does not explain anything about it; notably, it does not mention any parameters. But from the example script mentioned above, I knew that there was more to it. So I read the rest of that man page and found: "The Cyrus::IMAP module provides an interface to the Cyrus imclient library." So I installed the development files for Cyrus imapd and issued man imclient Now, finally, this is a man page a C programmer probably can live with. But when looking more thoroughly into it, I saw that there is no "password" parameter to the authenticate() function although the Perl module's authenticate() method has one: int imclient_authenticate (struct imclient *imclient, struct sasl_client **availmech, const char *service, const char *user, int protallowed); Plus, I could not find any hints regarding the relationship between the parameters of the C functions and those of the Perl module methods. I then headed over to cyrusimap.org and tried to find documentation for the Perl modules. Surprisingly, I couldn't. Not a single word. They don't seem to exist. The tools and helper programs are documented, but the Perl modules are not even mentioned. The same applies to CPAN: The modules cannot be found there. To make a long story short: Where can I find reasonable (in the sense: may be short and bad, but must be *complete*) documentation for Cyrus::IMAP::Admin and Cyrus::IMAP so that I can write my own helper scripts in the future? Do I really have to unpack Cyrus imapd's sources and read the module source code to get some ideas? Thank you very much in advance, Binarus From lists at binarus.de Sun Dec 16 11:48:53 2018 From: lists at binarus.de (Binarus) Date: Sun, 16 Dec 2018 17:48:53 +0100 Subject: Shared folders (AKA public) namespace showing twice Message-ID: Dear all, hopefully, this is my last post for today. I just have put a new server running 2.5.10 live and have noticed that the Shared Folders are appearing twice. Please note that I have renamed the "Shared Folders" prefix to "public" (see below). When a user who has access to the Shared Folders logs in (e.g. via Thunderbird), that user's mailbox folder hierarchy shows up as follows: Inbox Drafts Sent Junk Trash public <--- expected ("Shared folders" namespace) Folder1 Folder2 public <--- This is my problem Folder3 ... As you can see, the "virtual" public folder contains all shared (public) subfolders (nothing is missing, the permissions and all IMAP operations are working as expected), but there is an additional subfolder "public" which never has been actively created and which was empty every time we examined it. I originally thought that this was an artifact which had been created when moving all data from the old server to the new one, and deleted it. Upon deletion, Thunderbird did not show any error and moved that folder into "Trash". From then on, it was not visible any more at its original place. But guess what: It was there again as soon as Thunderbird had been restarted. To make sure that this was not due to a Thunderbird bug, I connected to the server in question as administrator (cyrus in this case) via cyradm and listed all mailboxes and subfolders (lm command). Indeed, there was *no* mailbox or folder which had "public" as its name. I investigated the issue further with the help of telnet, which revealed something interesting (I logged in as a normal non-admin user who has access to the shared (public) folders; I have left out the prologue and the LOGIN dialog for brevity and clarity): root at morn:/etc# telnet localhost imap [...] A1 NAMESPACE * NAMESPACE (("" ".")) (("other." ".")) (("public." ".")) A1 OK Completed A1 LIST "" "%" * LIST (\Noinferiors \HasNoChildren) "." INBOX * LIST (\HasNoChildren) "." Drafts * LIST (\HasNoChildren) "." Junk * LIST (\HasNoChildren) "." Sent * LIST (\HasChildren) "." Trash * LIST (\Noselect \HasChildren) "." public.public A1 OK Completed (0.003 secs 7 calls) The NAMESPACE command's return value is as expected, but the LIST command returns "public.public" at the place where it (IMHO) just should return "public". I believe that this is the reason for our problem, and that this can't be correct. This might or might not be related to the following bug report / discussion from 2005: https://github.com/cyrusimap/cyrus-imapd/issues/744 In every case, listing "public.public" is wrong, isn't it? I can further tell that we did not have that problem with our old server running 2.4.16 with the *literally same* configuration files (imapd.conf and cyrus.conf). These are the snippets from imapd.conf which are probably relevant: altnamespace: yes userprefix: other sharedprefix: public unixhierarchysep: no autocreate_quota: 0 Could somebody please tell me if the behavior I have observed is a bug or if I am doing something wrong? If it's a bug, is there something I can do about it (besides compiling the newest version of Cyrus imapd myself or changing the Linux distro)? Thank you very much in advance, Binarus From jc at irbs.com Sun Dec 16 16:24:35 2018 From: jc at irbs.com (John Capo) Date: Sun, 16 Dec 2018 16:24:35 -0500 (EST) Subject: Question for upgrading In-Reply-To: <5ed0a3707ca4b486ede31779d1cbf4da@sarenet.es> References: <9e71e82ddcba5500a175c0e447131413@sarenet.es> <5ed0a3707ca4b486ede31779d1cbf4da@sarenet.es> Message-ID: <2063.205.237.194.2.1544995475.squirrel@beta.mxes.net> On Thu, December 13, 2018 13:25, Egoitz Aurrekoetxea wrote: > Hi again! > > > Else as a simplication way can you replicate any manner a 2.3 with some > newer version?. At least in manual mode (not rolling)?. The replication protocol in 2.4 is not compatible wit 2.3. There is no easy way. I'm rsync'ing one account at a time between 2.3 and 2.4 with IMAP/POP and LMTP access disabled, followed by a reconstruct, and then enabling IMAP/POP and LMTP access. Many terrabytes to go. You could also look at imapsync but its slow. > > Cheers. > > > --- > > > EGOITZ AURREKOETXEA > Departamento de sistemas > 944 209 470 > Parque Tecnol??gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr??nico piense si es > necesario hacerlo. > > El 13-12-2018 16:52, Egoitz Aurrekoetxea escribi??: > > >> Good afternoon, >> >> >> I was trying to upgrade part of our Cyrus imap installation, concretely that one consisting in >> still 2.3. I was planning to set up Cyrus 3.0. I have seen all works properly except for the >> unexpunge command because as someone stated here, a reconstruct -V max was needed.The problem >> is that this reconstruct command, takes ages and I'm not able to keep the service offline so >> many time. So I have been thinking in the following scenario : >> >> - Cyrus 2.3 master -> Cyrus 2.4 slave >> >> >> Get this 2.4 slave ready and set it as master. But here comes my first doubt. Does the 2.4 >> replication work with the 2.3 replication?. Can in this pair, both (the 2.3 and the 2.4) be >> both master and slave?. I mean to switch roles in the pair. Make one become master and the >> other slave and vice versa?. >> >> Let's think now Cyrus 2.4 is ready and working. >> >> >> - Now, I would set up a new 3.0 slave. I know 2.4 could replicate with 3.0. So I would get the >> 3.0 ready and then set 3.0 as master. Can in this pair both the 2.4 and 3.0 be master and >> slave?. Meaning again to the same role switching commented before... to make one to be master >> and the other slave or vice versa.... >> >> I'l will end up with 2 3.0 master and slave... but I need to trace the path... >> >> >> Does anyone see any other way?. >> >> >> Best regards, >> >> >> - >> >> >> -- >> >> >> EGOITZ AURREKOETXEA >> Departamento de sistemas >> 944 209 470 >> Parque Tecnol??gico. Edificio 103 >> 48170 Zamudio (Bizkaia) >> egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr??nico piense si es >> necesario hacerlo. ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >> > > > Links: > ------ > [1] http://www.sarenet.es From egoitz at sarenet.es Mon Dec 17 04:40:36 2018 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Mon, 17 Dec 2018 10:40:36 +0100 Subject: Question for upgrading In-Reply-To: <2063.205.237.194.2.1544995475.squirrel@beta.mxes.net> References: <9e71e82ddcba5500a175c0e447131413@sarenet.es> <5ed0a3707ca4b486ede31779d1cbf4da@sarenet.es> <2063.205.237.194.2.1544995475.squirrel@beta.mxes.net> Message-ID: <38a2eed166ef29ab1f38cd21660465a0@sarenet.es> Hi mate, I think (and say think :) )I finally found a method. Although I'm testing it deeply... it seems (say seems too :) ) 2.4 is compatible with a mail spool in 2.3 (at least with my config). So I'll try to upgrade first to 2.4 and later to 3.0 setting up a replication from 2.4 to 3.0. Would be fine if Bron, Ellie or someone at Fastmail could tell something about it to us :) :) Cheers! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 16-12-2018 22:24, John Capo escribi?: > On Thu, December 13, 2018 13:25, Egoitz Aurrekoetxea wrote: > >> Hi again! >> >> Else as a simplication way can you replicate any manner a 2.3 with some >> newer version?. At least in manual mode (not rolling)?. > > The replication protocol in 2.4 is not compatible wit 2.3. > > There is no easy way. I'm rsync'ing one account at a time between 2.3 and 2.4 with IMAP/POP and LMTP access disabled, followed by a reconstruct, and then enabling IMAP/POP and LMTP access. > > Many terrabytes to go. > > You could also look at imapsync but its slow. > > Cheers. > > --- > > EGOITZ AURREKOETXEA > Departamento de sistemas > 944 209 470 > Parque Tecnol??gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es www.sarenet.es [1] [1 [1]] Antes de imprimir este correo electr??nico piense si es > necesario hacerlo. > > El 13-12-2018 16:52, Egoitz Aurrekoetxea escribi??: > > Good afternoon, > > I was trying to upgrade part of our Cyrus imap installation, concretely that one consisting in > still 2.3. I was planning to set up Cyrus 3.0. I have seen all works properly except for the > unexpunge command because as someone stated here, a reconstruct -V max was needed.The problem > is that this reconstruct command, takes ages and I'm not able to keep the service offline so > many time. So I have been thinking in the following scenario : > > - Cyrus 2.3 master -> Cyrus 2.4 slave > > Get this 2.4 slave ready and set it as master. But here comes my first doubt. Does the 2.4 > replication work with the 2.3 replication?. Can in this pair, both (the 2.3 and the 2.4) be > both master and slave?. I mean to switch roles in the pair. Make one become master and the > other slave and vice versa?. > > Let's think now Cyrus 2.4 is ready and working. > > - Now, I would set up a new 3.0 slave. I know 2.4 could replicate with 3.0. So I would get the > 3.0 ready and then set 3.0 as master. Can in this pair both the 2.4 and 3.0 be master and > slave?. Meaning again to the same role switching commented before... to make one to be master > and the other slave or vice versa.... > > I'l will end up with 2 3.0 master and slave... but I need to trace the path... > > Does anyone see any other way?. > > Best regards, > > - > > -- > > EGOITZ AURREKOETXEA > Departamento de sistemas > 944 209 470 > Parque Tecnol??gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es www.sarenet.es [1] [1 [1]] Antes de imprimir este correo electr??nico piense si es > necesario hacerlo. ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > Links: > ------ > [1] http://www.sarenet.es Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From awilliam at whitemice.org Mon Dec 17 06:38:36 2018 From: awilliam at whitemice.org (Adam Tauno Williams) Date: Mon, 17 Dec 2018 06:38:36 -0500 Subject: Question for upgrading In-Reply-To: <38a2eed166ef29ab1f38cd21660465a0@sarenet.es> References: <9e71e82ddcba5500a175c0e447131413@sarenet.es> <5ed0a3707ca4b486ede31779d1cbf4da@sarenet.es> <2063.205.237.194.2.1544995475.squirrel@beta.mxes.net> <38a2eed166ef29ab1f38cd21660465a0@sarenet.es> Message-ID: <1545046716.9041.3.camel@whitemice.org> On Mon, 2018-12-17 at 10:40 +0100, Egoitz Aurrekoetxea wrote: > I think (and say think :) )I finally found a method. Although I'm > testing it deeply... it seems (say seems too :) ) 2.4 is compatible > with a mail spool in 2.3 (at least with my config). So I'll try to > upgrade first to 2.4 and later to 3.0 setting up a replication from > 2.4 to 3.0. Would be fine if Bron, Ellie or someone at Fastmail could > tell something about it to us :) :) Yes, I have performed an in-place upgrade of 2.3 to 2.4. Other than some delay due to index reconstruction it went very smoothly. The only issues I recall is that some users got some deleted messages back after the reconstruct, and there were some mailboxes which lost Seen status; I never figured out why [they were related to that handful of users which somehow always have problems with everything]. -- Executive Committee Vice-Chair Michigan Association of Railroad Passengers 537 Shirley St NE Grand Rapids, MI 49503-1754 Phone: 616.581.8010 E-mail: awilliam at whitemice.org GPG#D95ED383 From egoitz at sarenet.es Mon Dec 17 12:36:33 2018 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Mon, 17 Dec 2018 18:36:33 +0100 Subject: Question for upgrading In-Reply-To: <1545046716.9041.3.camel@whitemice.org> References: <9e71e82ddcba5500a175c0e447131413@sarenet.es> <5ed0a3707ca4b486ede31779d1cbf4da@sarenet.es> <2063.205.237.194.2.1544995475.squirrel@beta.mxes.net> <38a2eed166ef29ab1f38cd21660465a0@sarenet.es> <1545046716.9041.3.camel@whitemice.org> Message-ID: Hi! Yes, I assume there shouldn't be problems... in fact reconstruct in 2.4 does not have the option for upgrading the mailbox databases version with a reconstruct -V max command... So... I assume it's compatible.... There I could then setup a slave replication... I hope... I'll tell you :) Cheers! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 17-12-2018 12:38, Adam Tauno Williams escribi?: > On Mon, 2018-12-17 at 10:40 +0100, Egoitz Aurrekoetxea wrote: > >> I think (and say think :) )I finally found a method. Although I'm >> testing it deeply... it seems (say seems too :) ) 2.4 is compatible >> with a mail spool in 2.3 (at least with my config). So I'll try to >> upgrade first to 2.4 and later to 3.0 setting up a replication from >> 2.4 to 3.0. Would be fine if Bron, Ellie or someone at Fastmail could >> tell something about it to us :) :) > > Yes, I have performed an in-place upgrade of 2.3 to 2.4. Other than > some delay due to index reconstruction it went very smoothly. > > The only issues I recall is that some users got some deleted messages > back after the reconstruct, and there were some mailboxes which lost > Seen status; I never figured out why [they were related to that > handful of users which somehow always have problems with everything]. Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From jschaeffer0922 at gmail.com Mon Dec 17 13:09:26 2018 From: jschaeffer0922 at gmail.com (Joshua Schaeffer) Date: Mon, 17 Dec 2018 18:09:26 +0000 Subject: sieve runtime error Message-ID: I'm trying to setup sieve and getting the following error in my logs: Dec 17 10:36:07 bllmail01 cyrus/lmtp[14530]: sieve runtime error for jschaeffer at harmonywave.net id : Reject: Sendmail process terminated normally, exit status 255 I'm following the documentation here: https://www.cyrusimap.org/imap/reference/admin/sieve.html?highlight=sieve#testing-the-sieve-server I'm trying to get sieve working on my IMAP server. Using Ubuntu 16.04 with the cyrus-imapd 2.4.18-3 package. I'm using the test sieve script shown in the documentation to reject everything from my personal email: require ["reject","fileinto"]; if address :is :all "From" "jschaeffer0922 at gmail.com" { reject "testing"; } I then connect using sieveshell, upload the file, and activate it: root at bllmail01:~# sieveshell -u jschaeffer at harmonywave.net -a jschaeffer at harmonywave.net mail.harmonywave.cloud connecting to mail.harmonywave.cloud Please enter your password: > put /tmp/testing.sieve testing > activate testing > list testing <- active script > quit However when I send a test email from my personal account to the email I have setup on the IMAP server it always comes through and I get this in mail.log: Dec 17 10:36:07 bllmail01 cyrus/lmtp[14530]: sieve runtime error for jschaeffer at harmonywave.net id : Reject: Sendmail process terminated normally, exit status 255 I've tried searching for causes of this error but couldn't find much. Any help would be appreciated. Thanks, Joshua Schaeffer -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellie at fastmail.com Mon Dec 17 16:46:06 2018 From: ellie at fastmail.com (ellie timoney) Date: Tue, 18 Dec 2018 08:46:06 +1100 Subject: Question for upgrading In-Reply-To: <38a2eed166ef29ab1f38cd21660465a0@sarenet.es> References: <9e71e82ddcba5500a175c0e447131413@sarenet.es> <5ed0a3707ca4b486ede31779d1cbf4da@sarenet.es> <2063.205.237.194.2.1544995475.squirrel@beta.mxes.net> <38a2eed166ef29ab1f38cd21660465a0@sarenet.es> Message-ID: <1545083166.1800663.1611834480.2001673C@webmail.messagingengine.com> > Would be fine if Bron, Ellie or someone at Fastmail could tell > something about it to us :) :) Cheeky! Thing is, at FM our production servers more or less track the master branch, so we were running "3.0" from the moment 2.5 forked off, and so our upgrade process only ever really needs to deal with iterative changes. Our accumulated experience is what informs the "upgrade notes" included with the 2.5.0 and 3.0.0 releases (not sure about older versions, I wasn't around then). This mailing list is the right place to find people with real experience doing big-bang upgrades. :) Cheers, ellie On Mon, Dec 17, 2018, at 8:40 PM, Egoitz Aurrekoetxea wrote: > Hi mate, > > I think (and say think :) )I finally found a method. Although I'm > testing it deeply... it seems (say seems too :) ) 2.4 is compatible > with a mail spool in 2.3 (at least with my config). So I'll try to > upgrade first to 2.4 and later to 3.0 setting up a replication from > 2.4 to 3.0. Would be fine if Bron, Ellie or someone at Fastmail could > tell something about it to us :) :)> > Cheers! > --- > > sarenet > *Egoitz Aurrekoetxea* > Departamento de sistemas > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > www.sarenet.es > > Antes de imprimir este correo electr?nico piense si es necesario > hacerlo.> > El 16-12-2018 22:24, John Capo escribi?: >> On Thu, December 13, 2018 13:25, Egoitz Aurrekoetxea wrote: >>> Hi again! >>> >>> >>> Else as a simplication way can you replicate any manner a 2.3 >>> with some>>> newer version?. At least in manual mode (not rolling)?. >> >> The replication protocol in 2.4 is not compatible wit 2.3. >> >> There is no easy way. I'm rsync'ing one account at a time between >> 2.3 and 2.4 with IMAP/POP and LMTP access disabled, followed by a >> reconstruct, and then enabling IMAP/POP and LMTP access.>> >> Many terrabytes to go. >> >> You could also look at imapsync but its slow. >> >> >>> >>> Cheers. >>> >>> >>> --- >>> >>> >>> EGOITZ AURREKOETXEA >>> Departamento de sistemas >>> 944 209 470 >>> Parque Tecnol??gico. Edificio 103 >>> 48170 Zamudio (Bizkaia) >>> egoitz at sarenet.es www.sarenet.es [1[1]] Antes de imprimir este >>> correo electr??nico piense si es>>> necesario hacerlo. >>> >>> El 13-12-2018 16:52, Egoitz Aurrekoetxea escribi??: >>> >>> >>> >>>> Good afternoon, >>>> >>>> >>>> I was trying to upgrade part of our Cyrus imap installation, >>>> concretely that one consisting in>>>> still 2.3. I was planning to set up Cyrus 3.0. I have seen all >>>> works properly except for the>>>> unexpunge command because as someone stated here, a reconstruct -V >>>> max was needed.The problem>>>> is that this reconstruct command, takes ages and I'm not able to >>>> keep the service offline so>>>> many time. So I have been thinking in the following scenario : >>>> >>>> - Cyrus 2.3 master -> Cyrus 2.4 slave >>>> >>>> >>>> Get this 2.4 slave ready and set it as master. But here comes my >>>> first doubt. Does the 2.4>>>> replication work with the 2.3 replication?. Can in this pair, both >>>> (the 2.3 and the 2.4) be>>>> both master and slave?. I mean to switch roles in the pair. Make >>>> one become master and the>>>> other slave and vice versa?. >>>> >>>> Let's think now Cyrus 2.4 is ready and working. >>>> >>>> >>>> - Now, I would set up a new 3.0 slave. I know 2.4 could replicate >>>> with 3.0. So I would get the>>>> 3. ready and then set 3.0 as master. Can in this pair both the 2.4 >>>> and 3.0 be master and>>>> slave?. Meaning again to the same role switching commented >>>> before... to make one to be master>>>> and the other slave or vice versa.... >>>> >>>> I'l will end up with 2 3.0 master and slave... but I need to trace >>>> the path...>>>> >>>> >>>> Does anyone see any other way?. >>>> >>>> >>>> Best regards, >>>> >>>> >>>> - >>>> >>>> >>>> -- >>>> >>>> >>>> EGOITZ AURREKOETXEA >>>> Departamento de sistemas >>>> 944 209 470 >>>> Parque Tecnol??gico. Edificio 103 >>>> 48170 Zamudio (Bizkaia) >>>> egoitz at sarenet.es www.sarenet.es [1[2]] Antes de imprimir este >>>> correo electr??nico piense si es>>>> necesario hacerlo. ---- >>>> Cyrus Home Page: http://www.cyrusimap.org/ >>>> List Archives/Info: >>>> http://lists.andrew.cmu.edu/pipermail/info-cyrus/>>>> To Unsubscribe: >>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >>> >>> >>> Links: >>> ------ >>> [1] http://www.sarenet.es Links: 1. http://www.sarenet.es 2. http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellie at fastmail.com Mon Dec 17 17:38:02 2018 From: ellie at fastmail.com (ellie timoney) Date: Tue, 18 Dec 2018 09:38:02 +1100 Subject: Why do we need to run reconstruct after having moved mailboxes using cyradm's xfer command? In-Reply-To: References: Message-ID: <1545086282.1815013.1611857728.363A16D4@webmail.messagingengine.com> Hi Binarus, > Could anybody please shortly explain why? What exactly are the > techniques and mechanisms cyradm's xfer uses to do its thing? I had been > quite sure that it just uses the IMAP protocol, but there seems to be > more to it ... You're right, but missing a detail. The XFER command is an IMAP extension, for use in cyrus clusters ("murders"), where mailboxes need to be identical between servers. I don't know much about the details, but it makes sense to me that it would copy the mailboxes across at a "deeper" level, to make sure internal identifiers and such match. cyradm uses the IMAP protocol ? think of it as a command-driven IMAP client, except that it knows about cyrus extensions (like XFER) that generic IMAP clients (like imapsync) probably don't Something like imapsync would probably be using the CREATE and APPEND commands instead, which let the destination server choose its own internal identifiers, and which will probably be different from the source ones. But they would be stored in the exact method the destination server prefers (such as using the newest version of the mailbox index format...) > Now that I have used cyradm's xfer command to relocate the mailboxes / messages / folders / subfolders, I surprisingly had to run the "heaviest" form of reconstruct (-V max). That's not the "heaviest" form of reconstruct, it's just updating the mailbox indexes to the "max"imum (-V)ersion supported by the server. This ends up adding a bunch of extra information to the indexes that the newer server version can make use of. Generally if the index version is too old for what it's trying to do, it'll fall back to calculating missing values the long way, or provide less info, or just refuse the requested operation. The assertion failure you saw is clearly a bug, but this smells familiar, so maybe it's already been fixed. The latest version of the 2.5 release is 2.5.12, and is nearly 2 years newer than 2.5.10, for what it's worth. Hope this helps! ellie On Sun, Dec 16, 2018, at 9:06 PM, Binarus wrote: > Dear all, > > yesterday, I have moved a bunch of user mailboxes and public folders > from a server running 2.4.16 to another server running 2.5.10, using > cyradm's xfer command (see my previous messages). > > After having finished the migration, I noticed a weird behavior in > Thunderbird (which is our standard email client): When trying to move a > message from the Inbox to the Junk or Trash folder, the message > disappeared from the Inbox for a short time, then reappeared. The log > files on the server were showing the dreaded entries: > > Dec 16 01:12:59 morn cyrus/imaps[14914]: Fatal error: Internal error: > assertion failed: imap/mailbox.c: 2850: !message_guid_isnull(&record- > >guid) > > "Fortunately", a lot of other users are affected by such log entries and > weird email client behavior as well, so finding the solution on the 'net > was not too difficult: Running reconstruct did not lead to anywhere, but > running reconstruct -V max was the solution. > > This lets me scratch my head: > > In the past, I have upgraded Cyrus imapd at least three times, each time > using imapsync (instead of cyradm's xfer command) to move the mailboxes > and the public namespace (including all messages and subfolders) from > the old server to the new one. In none of these cases, it was necessary > to reconstruct anything afterwards. This would have been illogical > anyway: Each time, the new server had been setup from scratch, and no > mailboxes / messages / folders / subfolders had been moved directly via > file transfer from the old to the new server. > > Now that I have used cyradm's xfer command to relocate the mailboxes / > messages / folders / subfolders, I surprisingly had to run the > "heaviest" form of reconstruct (-V max). > > Could anybody please shortly explain why? What exactly are the > techniques and mechanisms cyradm's xfer uses to do its thing? I had been > quite sure that it just uses the IMAP protocol, but there seems to be > more to it ... > > Thank you very much for any insight, > > Binarus > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From ellie at fastmail.com Mon Dec 17 17:57:59 2018 From: ellie at fastmail.com (ellie timoney) Date: Tue, 18 Dec 2018 09:57:59 +1100 Subject: Running a script with cyradm throwing ReadLine errors In-Reply-To: References: Message-ID: <1545087479.1822015.1611895992.1214BC95@webmail.messagingengine.com> Hi Binarus, > Could anybody please tell me what I might do wrong here? This kind of smells like maybe your system has two versions of perl installed (or two versions of Term::ReadLine, or maybe even two versions of Cyrus::IMAP::Shell), and they're getting in each other's way? I'm having a quick glance at the (2.5.10) source of Cyrus::IMAP::Shell and this caught my eye: > # ugh. ugh. suck. aieee. > my $use_rl = 'Cyrus::IMAP::DummyReadline'; > { > if (eval { require Term::ReadLine; }) { > $use_rl = 'Term::ReadLine'; > } > } Which... fills me with confidence. Looks like a workaround for missing (broken?) Term::Readline but that comment isn't super enlightening. I wonder if it will Just Work if you uninstall Term::Readline? I haven't really used cyradm at all myself, so take all this with a grain of salt. Hopefully someone who has can chime in! Cheers, ellie On Sun, Dec 16, 2018, at 8:04 PM, Binarus wrote: > Dear all, > > I was just trying to explore cyradm a little bit further and hence was > experimenting with its scripting capabilities. Having cyradm run a > script should be pretty easy. man cyradm tells us: > > perl -MCyrus::IMAP::Shell -e 'run("myscript")' > > So I created the simplest possible script (that means an empty one) and > tried to run it: > > touch 000 > chmod a+x 000 (just in case ...) > perl -MCyrus::IMAP::Shell -e 'run("000")' > > The only thing I got was: > > Use of uninitialized value within @layers in string eq at /usr/local/ > lib/x86_64-linux-gnu/perl/5.24.1/Term/ReadLine/Gnu.pm line 280. > Bad filehandle: __DATA__ at /usr/local/lib/x86_64-linux-gnu/perl/ > 5.24.1/Term/ReadLine/Gnu.pm line 769. > > Putting something meaningful into the script did not change the situation. > > I have googled and read documentation (mainly on cyrusimapd.org) for > several hours, but could not find the reason for the problem. > > I even have put allowplaintext=yes into imapd.conf and restarted imapd > (knowing that this probably wasn't very smart, but the term "layers" in > the error message made me mistrustful because there are authentication > "layers", and I don't have any problems with Term::ReadLine::Gnu in > general). As expected, this didn't change the situation either. > > This happened with 2.4.16 as well as with 2.5.10. > > Could anybody please tell me what I might do wrong here? > > Thank you very much in advance, > > Binarus > > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From ellie at fastmail.com Mon Dec 17 18:39:13 2018 From: ellie at fastmail.com (ellie timoney) Date: Tue, 18 Dec 2018 10:39:13 +1100 Subject: Documentation for Cyrus::IMAP::Admin and friends In-Reply-To: <4ee8cab3-5348-ae25-a9b1-a7854601b7e4@binarus.de> References: <4ee8cab3-5348-ae25-a9b1-a7854601b7e4@binarus.de> Message-ID: <1545089953.1835428.1611917400.541BFF83@webmail.messagingengine.com> Hi Binarus, Looks like the documentation for the authenticate() function is pretty incomplete... > Well, this man page "documents" the method by exactly one line > (identically in two places): > > $client->authenticate; >From my read of the documentation, this is enough. It will figure out on its own which SASL options are appropriate, and then prompt you (on the controlling terminal) for the password if/as needed. This nicely avoids implicitly instructing users to hardcode their credentials in a script, so in this sense it's probably good that it's underdocumented (maybe deliberately so). Looking in the source of Cyrus::IMAP, the accepted arguments are: > -mechanism, -service, -authz, -user, -minssf, -maxssf, -password, -tlskey, -notls, -cafile, -capath If unspecified, "-service" defaults to "imap", "-maxssf" defaults to 10000, "user" defaults to the current unix user, and the rest default to zero or blank. Note that "user" is the user to authenticate as (i.e. whose credentials are to be checked), whereas "authz" is who to authoriZe as (i.e. which identity to assume, once successfully authenticated). authz only works for SASL mechanisms that allow proxy authentication... the imtest(1) man page says "e.g. PLAIN, DIGEST-MD5", I'm not sure if there are others. > The same applies to CPAN: The modules cannot be found there. The Cyrus::* perl modules are tightly coupled at build time to the installed cyrus version, so it doesn't make any sense to distribute them independently via CPAN. > Where can I find reasonable (in the sense: may be short and bad, but > must be *complete*) documentation for Cyrus::IMAP::Admin and Cyrus::IMAP > so that I can write my own helper scripts in the future? Do I really > have to unpack Cyrus imapd's sources and read the module source code to > get some ideas? Sorta, but you don't need the Cyrus imapd sources -- if you have the perl module, you have the perl module's source. Look for the "Cyrus/IMAP.pm" file somewhere on your system (something like "sudo find / -name IMAP.pm" should do the trick) and there it is. Cheers, ellie On Sun, Dec 16, 2018, at 9:55 PM, Binarus wrote: > Dear all, > > as described in my previous messages, I recently had a hard time > relocating (i.e. moving) mailboxes and the public namespace from 2.4.16 > to a new server running 2.5.10. I would have happily written a Perl > script which surely had solved much of my problems, but I couldn't do > that due to the lack of documentation for the respective Perl modules. > > This is best explained by example: > > I have found a Perl script on the 'net which looks trustworthy at the > first sight and which contains (among others) the following lines of > code: > > use Cyrus::IMAP::Admin; > > # Connect to Cyrus > $imap = Cyrus::IMAP::Admin->new("my_server") || die "Unable to connect > to my_server"; > > $imap->authenticate(-user => "foo", > -mechanism => "LOGIN", > -password => "password", > ); > > I wanted to understand exactly what the script does and thus tried to > find Cyrus::IMAP::Admin's documentation. So I issued > > man Cyrus::IMAP::Admin > > and found that some methods are documented, but not the authenticate() > method which is needed first and one of the most important ones. > However, the man page states (in the section about the "new()" method): > > "Instantiates a cyradm object. This is in fact an Cyrus::IMAP object > with a few additional methods, so all Cyrus::IMAP methods are available > if needed. (In particular, you will always want to use the > "authenticate" method.)" > > This tricked me into believing that the "authenticate" method would be > part of Cyrus::IMAP and would be explained in that module's > documentation, so I issued > > man Cyrus::IMAP > > Well, this man page "documents" the method by exactly one line > (identically in two places): > > $client->authenticate; > > It does not explain anything about it; notably, it does not mention any > parameters. But from the example script mentioned above, I knew that > there was more to it. So I read the rest of that man page and found: > > "The Cyrus::IMAP module provides an interface to the Cyrus imclient library." > > So I installed the development files for Cyrus imapd and issued > > man imclient > > Now, finally, this is a man page a C programmer probably can live with. > But when looking more thoroughly into it, I saw that there is no > "password" parameter to the authenticate() function although the Perl > module's authenticate() method has one: > > int imclient_authenticate (struct imclient *imclient, struct sasl_client > **availmech, const char *service, const char *user, int protallowed); > > Plus, I could not find any hints regarding the relationship between the > parameters of the C functions and those of the Perl module methods. > > I then headed over to cyrusimap.org and tried to find documentation for > the Perl modules. Surprisingly, I couldn't. Not a single word. They > don't seem to exist. The tools and helper programs are documented, but > the Perl modules are not even mentioned. > > The same applies to CPAN: The modules cannot be found there. > > To make a long story short: > > Where can I find reasonable (in the sense: may be short and bad, but > must be *complete*) documentation for Cyrus::IMAP::Admin and Cyrus::IMAP > so that I can write my own helper scripts in the future? Do I really > have to unpack Cyrus imapd's sources and read the module source code to > get some ideas? > > Thank you very much in advance, > > Binarus > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From simon.matter at invoca.ch Mon Dec 17 23:32:15 2018 From: simon.matter at invoca.ch (Simon Matter) Date: Tue, 18 Dec 2018 05:32:15 +0100 Subject: sieve runtime error In-Reply-To: References: Message-ID: > I'm trying to setup sieve and getting the following error in my logs: > > Dec 17 10:36:07 bllmail01 cyrus/lmtp[14530]: sieve runtime error for > jschaeffer at harmonywave.net id > : > Reject: Sendmail process terminated normally, exit status 255 > > > I'm following the documentation here: > https://www.cyrusimap.org/imap/reference/admin/sieve.html?highlight=sieve#testing-the-sieve-server > > I'm trying to get sieve working on my IMAP server. Using Ubuntu 16.04 with > the cyrus-imapd 2.4.18-3 package. I'm using the test sieve script shown in > the documentation to reject everything from my personal email: > > require ["reject","fileinto"]; > if address :is :all "From" "jschaeffer0922 at gmail.com" > { > reject "testing"; > } > > I then connect using sieveshell, upload the file, and activate it: > > root at bllmail01:~# sieveshell -u jschaeffer at harmonywave.net -a > jschaeffer at harmonywave.net mail.harmonywave.cloud > connecting to mail.harmonywave.cloud > Please enter your password: >> put /tmp/testing.sieve testing >> activate testing >> list > testing <- active script >> quit > > However when I send a test email from my personal account to the email > I have setup on the IMAP server it always comes through and I get this > in mail.log: > > Dec 17 10:36:07 bllmail01 cyrus/lmtp[14530]: sieve runtime error for > jschaeffer at harmonywave.net id > : > Reject: Sendmail process terminated normally, exit status 255 I think sieve tries to send mail using the configured sendmail binary and that doesn't work for some reason. You may check the sendmail config in your imapd.conf and also consult the mail logs to learn more. Regards, Simon From egoitz at sarenet.es Tue Dec 18 01:26:24 2018 From: egoitz at sarenet.es (Egoitz Aurrekoetxea) Date: Tue, 18 Dec 2018 07:26:24 +0100 Subject: Question for upgrading In-Reply-To: <1545083166.1800663.1611834480.2001673C@webmail.messagingengine.com> References: <9e71e82ddcba5500a175c0e447131413@sarenet.es> <5ed0a3707ca4b486ede31779d1cbf4da@sarenet.es> <2063.205.237.194.2.1544995475.squirrel@beta.mxes.net> <38a2eed166ef29ab1f38cd21660465a0@sarenet.es> <1545083166.1800663.1611834480.2001673C@webmail.messagingengine.com> Message-ID: <0f434203c44493a197ec5aa94d5267ac@sarenet.es> Thank you so much Ellie!! I would test, read code when needed and finally attempt the upgrade... I think this coud work.... should say I'm in testing phase still, I'll keep you informed :) Cheers! --- EGOITZ AURREKOETXEA Departamento de sistemas 944 209 470 Parque Tecnol?gico. Edificio 103 48170 Zamudio (Bizkaia) egoitz at sarenet.es www.sarenet.es [1] Antes de imprimir este correo electr?nico piense si es necesario hacerlo. El 17-12-2018 22:46, ellie timoney escribi?: >> Would be fine if Bron, Ellie or someone at Fastmail could tell something about it to us :) :) > > Cheeky! Thing is, at FM our production servers more or less track the master branch, so we were running "3.0" from the moment 2.5 forked off, and so our upgrade process only ever really needs to deal with iterative changes. Our accumulated experience is what informs the "upgrade notes" included with the 2.5.0 and 3.0.0 releases (not sure about older versions, I wasn't around then). > > This mailing list is the right place to find people with real experience doing big-bang upgrades. :) > > Cheers, > > ellie > > On Mon, Dec 17, 2018, at 8:40 PM, Egoitz Aurrekoetxea wrote: > > Hi mate, > > I think (and say think :) )I finally found a method. Although I'm testing it deeply... it seems (say seems too :) ) 2.4 is compatible with a mail spool in 2.3 (at least with my config). So I'll try to upgrade first to 2.4 and later to 3.0 setting up a replication from 2.4 to 3.0. Would be fine if Bron, Ellie or someone at Fastmail could tell something about it to us :) :) > > Cheers! > > --- > > EGOITZ AURREKOETXEA > Departamento de sistemas > > 944 209 470 > Parque Tecnol?gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es > > www.sarenet.es [1] > > Antes de imprimir este correo electr?nico piense si es necesario hacerlo. > > El 16-12-2018 22:24, John Capo escribi?: > > On Thu, December 13, 2018 13:25, Egoitz Aurrekoetxea wrote: > Hi again! > > Else as a simplication way can you replicate any manner a 2.3 with some > newer version?. At least in manual mode (not rolling)?. > > The replication protocol in 2.4 is not compatible wit 2.3. > > There is no easy way. I'm rsync'ing one account at a time between 2.3 and 2.4 with IMAP/POP and LMTP access disabled, followed by a reconstruct, and then enabling IMAP/POP and LMTP access. > > Many terrabytes to go. > > You could also look at imapsync but its slow. > > Cheers. > > --- > > EGOITZ AURREKOETXEA > Departamento de sistemas > 944 209 470 > Parque Tecnol??gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es www.sarenet.es [1] [1 [1]] Antes de imprimir este correo electr??nico piense si es > necesario hacerlo. > > El 13-12-2018 16:52, Egoitz Aurrekoetxea escribi??: > > Good afternoon, > > I was trying to upgrade part of our Cyrus imap installation, concretely that one consisting in > still 2.3. I was planning to set up Cyrus 3.0. I have seen all works properly except for the > unexpunge command because as someone stated here, a reconstruct -V max was needed.The problem > is that this reconstruct command, takes ages and I'm not able to keep the service offline so > many time. So I have been thinking in the following scenario : > > - Cyrus 2.3 master -> Cyrus 2.4 slave > > Get this 2.4 slave ready and set it as master. But here comes my first doubt. Does the 2.4 > replication work with the 2.3 replication?. Can in this pair, both (the 2.3 and the 2.4) be > both master and slave?. I mean to switch roles in the pair. Make one become master and the > other slave and vice versa?. > > Let's think now Cyrus 2.4 is ready and working. > > - Now, I would set up a new 3.0 slave. I know 2.4 could replicate with 3.0. So I would get the > 3.0 ready and then set 3.0 as master. Can in this pair both the 2.4 and 3.0 be master and > slave?. Meaning again to the same role switching commented before... to make one to be master > and the other slave or vice versa.... > > I'l will end up with 2 3.0 master and slave... but I need to trace the path... > > Does anyone see any other way?. > > Best regards, > > - > > -- > > EGOITZ AURREKOETXEA > Departamento de sistemas > 944 209 470 > Parque Tecnol??gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz at sarenet.es www.sarenet.es [1] [1 [1]] Antes de imprimir este correo electr??nico piense si es > necesario hacerlo. ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > Links: > ------ > [1] http://www.sarenet.es Links: ------ [1] http://www.sarenet.es -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at binarus.de Tue Dec 18 03:07:40 2018 From: lists at binarus.de (Binarus) Date: Tue, 18 Dec 2018 09:07:40 +0100 Subject: Running a script with cyradm throwing ReadLine errors In-Reply-To: <1545087479.1822015.1611895992.1214BC95@webmail.messagingengine.com> References: <1545087479.1822015.1611895992.1214BC95@webmail.messagingengine.com> Message-ID: Dear ellie, thank you very much for your help! On 17.12.2018 23:57, ellie timoney wrote: > Hi Binarus, > >> Could anybody please tell me what I might do wrong here? > > This kind of smells like maybe your system has two versions of perl installed (or two versions of Term::ReadLine, or maybe even two versions of Cyrus::IMAP::Shell), and they're getting in each other's way? Since this is a fresh installation of Debian stretch, and since I didn't compile or install anything by hand yet, and since the Debian package management is usually very reliable, I am quite sure that this is not the problem. > Which... fills me with confidence. Looks like a workaround for missing (broken?) Term::Readline but that comment isn't super enlightening. I wonder if it will Just Work if you uninstall Term::Readline? This idea is very interesting, and you are absolutely right! While I didn't want to remove Term::ReadLine itself (because it is a core module and the usual module uninstall tools have difficulties with uninstalling it), I removed Term::ReadLine:Gnu (which I had additionally installed) instead. This made the error go away, and it seems that I can execute scripts now. So you have provided the solution and solved the problem. However, there is a downside. I am using cyradm quite often, mainly for setting permissions in a large shared folder (i.e. public) hierarchy. For this reason, I really need the nice feature which bash and many sorts of other shells provide: Hit the "Cursor-Up" key and have the shell repeat the previous command; the ability to edit the command line is often associated with this. Obviously, we can't have this feature in cyradm when only Term::ReadLine is installed. When this is the case, I even can't use "Cursor-Left" or "Cursor-Right" keys because they only produce weird character sequences instead of moving the cursor. This was the reason why I installed Term::ReadLine::Gnu in addition to Term::ReadLine. When Term::ReadLine:Gnu is installed, the command history feature in cyradm works as expected, and I can edit the command line (including using cursor keys) in a reasonable manner. Now it looks that I can either run scripts with cyradm _or_ can have its command line history and editing, but not both features at the same time. I think I could live with that, but of course I would be grateful if somebody would share a method to enable both features. Perhaps there is another module which I could use as a replacement for Term::ReadLine::Gnu and which does not break scripting? Thank you very much again, Binarus From lists at binarus.de Tue Dec 18 04:46:52 2018 From: lists at binarus.de (Binarus) Date: Tue, 18 Dec 2018 10:46:52 +0100 Subject: Why do we need to run reconstruct after having moved mailboxes using cyradm's xfer command? In-Reply-To: <1545086282.1815013.1611857728.363A16D4@webmail.messagingengine.com> References: <1545086282.1815013.1611857728.363A16D4@webmail.messagingengine.com> Message-ID: Dear ellie, thanks again for answering my questions! On 17.12.2018 23:38, ellie timoney wrote: > Hi Binarus, > >> Could anybody please shortly explain why? What exactly are the >> techniques and mechanisms cyradm's xfer uses to do its thing? I had been >> quite sure that it just uses the IMAP protocol, but there seems to be >> more to it ... > > You're right, but missing a detail. The XFER command is an IMAP extension, for use in cyrus clusters ("murders"), where mailboxes need to be identical between servers. I don't know much about the details, but it makes sense to me that it would copy the mailboxes across at a "deeper" level, to make sure internal identifiers and such match. OK, got it. I am glad that I at least didn't misunderstand cyradm completely. I originally thought that xfer was just a command which cyradm internally would translate into standard IMAP commands, which is wrong as I have understood now. > Something like imapsync would probably be using the CREATE and APPEND commands instead, which let the destination server choose its own internal identifiers, and which will probably be different from the source ones. But they would be stored in the exact method the destination server prefers (such as using the newest version of the mailbox index format...) Then it actually was the right idea to try cyradm xfer instead of imapsync. After all, we want our mailboxes, folders and messages transferred to the new server without anything being altered as far as possible, don't we? Now that I can use scripts with cyradm (thanks to your answer to one of my other posts), this will be a pleasure next time. >> Now that I have used cyradm's xfer command to relocate the mailboxes / messages / folders / subfolders, I surprisingly had to run the "heaviest" form of reconstruct (-V max). > > That's not the "heaviest" form of reconstruct, it's just updating the mailbox indexes to the "max"imum (-V)ersion supported by the server. This ends up adding a bunch of extra information to the indexes that the newer server version can make use of. Generally if the index version is too old for what it's trying to do, it'll fall back to calculating missing values the long way, or provide less info, or just refuse the requested operation. The assertion failure you saw is clearly a bug, but this smells familiar, so maybe it's already been fixed. The latest version of the 2.5 release is 2.5.12, and is nearly 2 years newer than 2.5.10, for what it's worth. OK, understood. In my case, it decided to refuse the operation ... somehow. Obviously, it didn't tell the client that it refused, but probably believed that the operation would be successful, sent an OK to the client, began the operation (the message indeed disappeared from Thunderbird's message list), noticed that it couldn't perform it and then reverted it (the message re-appeared in Thunderbird's message list about one second after it had disappeared). This is no real issue to me even if the log entries are due to a bug, as long as we can solve that problem by running the reconstruct. I just wanted to know why we have to reconstruct; thanks again for the insight! Regarding the version: In general, I don't have any problem with compiling software myself. Unfortunately, I then am responsible for the security updates as well, which I don't want, because I am not a full time administrator (I have to care for apache2, ntp, cyrus, sendmail, mysql and subversion servers, having only a few hours per week besides my normal job for doing this). Therefore, I'd like the Linux distribution / package manager to care for the security fixes, which means that I have to use the distribution's stock packages. However, this is an exceptional case because I will eventually have to upgrade anyway due to the problem with the shared folder namespace showing twice (described in one of my other posts). This is a thing which our users (including myself) probably can't accept, so if upgrading is the only remedy, I'll eventually do that. But I'll first wait for some answers to that other post. > Hope this helps! It really did :-) Thank you very much again! Binarus From lists at binarus.de Tue Dec 18 05:33:04 2018 From: lists at binarus.de (Binarus) Date: Tue, 18 Dec 2018 11:33:04 +0100 Subject: Documentation for Cyrus::IMAP::Admin and friends In-Reply-To: <1545089953.1835428.1611917400.541BFF83@webmail.messagingengine.com> References: <4ee8cab3-5348-ae25-a9b1-a7854601b7e4@binarus.de> <1545089953.1835428.1611917400.541BFF83@webmail.messagingengine.com> Message-ID: Dear ellie, you are really a champion! On 18.12.2018 00:39, ellie timoney wrote: > Hi Binarus, > > Looks like the documentation for the authenticate() function is pretty incomplete... > >> Well, this man page "documents" the method by exactly one line >> (identically in two places): >> >> $client->authenticate; > > From my read of the documentation, this is enough. It will figure out on its own which SASL options are appropriate, and then prompt you (on the controlling terminal) for the password if/as needed. This nicely avoids implicitly instructing users to hardcode their credentials in a script, so in this sense it's probably good that it's underdocumented (maybe deliberately so). This is great, and now I am embarrassed that I did not just try it. Querying the password at runtime is by far better than hardcoding the password and the mechanism in the script. Obviously, sometimes I am trying to read too much or am thinking too complicated instead of just trying ... > Looking in the source of Cyrus::IMAP, the accepted arguments are: > >> -mechanism, -service, -authz, -user, -minssf, -maxssf, -password, -tlskey, -notls, -cafile, -capath > > If unspecified, "-service" defaults to "imap", "-maxssf" defaults to 10000, "user" defaults to the current unix user, and the rest default to zero or blank. Thank you very much for this insight. This is important and really interesting ... > Note that "user" is the user to authenticate as (i.e. whose credentials are to be checked), whereas "authz" is who to authoriZe as (i.e. which identity to assume, once successfully authenticated). authz only works for SASL mechanisms that allow proxy authentication... the imtest(1) man page says "e.g. PLAIN, DIGEST-MD5", I'm not sure if there are others. I think I saw something like that during my research. For example, imapsync provides --proxyauth1 and --proxyauth2 command line parameters. >> The same applies to CPAN: The modules cannot be found there. > > The Cyrus::* perl modules are tightly coupled at build time to the installed cyrus version, so it doesn't make any sense to distribute them independently via CPAN. I see. This makes sense. >> Where can I find reasonable (in the sense: may be short and bad, but >> must be *complete*) documentation for Cyrus::IMAP::Admin and Cyrus::IMAP >> so that I can write my own helper scripts in the future? Do I really >> have to unpack Cyrus imapd's sources and read the module source code to >> get some ideas? > > Sorta, but you don't need the Cyrus imapd sources -- if you have the perl module, you have the perl module's source. Look for the "Cyrus/IMAP.pm" file somewhere on your system (something like "sudo find / -name IMAP.pm" should do the trick) and there it is. Again, thank you very much. You are right, getting it from the Perl installation is by far easier. Before actually opening it, I'll copy it to another place and work with the copy (in case I accidentally alter it while looking at it). > Cheers, I can't tell how grateful I am for your support (and for providing Cyrus imapd in general). IMHO, this is still the best IMAP server for small and large installations! Thank you very much again, Binarus From lists at binarus.de Tue Dec 18 06:03:58 2018 From: lists at binarus.de (Binarus) Date: Tue, 18 Dec 2018 12:03:58 +0100 Subject: Question for upgrading In-Reply-To: <9e71e82ddcba5500a175c0e447131413@sarenet.es> References: <9e71e82ddcba5500a175c0e447131413@sarenet.es> Message-ID: On 13.12.2018 16:52, Egoitz Aurrekoetxea wrote: > I was trying to upgrade part of our Cyrus imap installation, concretely > that one consisting in still 2.3. I was planning to set up Cyrus 3.0. I > have seen all works properly except for the unexpunge command because as > someone stated here, a reconstruct -V max was needed.The problem is that > this reconstruct command, takes ages and I'm not able to keep the > service offline so many time. Eventually you are meaning me by "someone", so I kick in here: The reconstruct (for me) was only needed because I had transferred the mailboxes using cyradm and the xfer command. As ellie has explained me in detail, xfer is an IMAP extension which transfers the data on a lower level than the standard IMAP commands. Therefore, there is a trade-off: If you transfer the data at "low-level" (e.g. xfer, file system), the transfer will be fast, and (at least in case of xfer) everything (flags, permissions etc.) will be transferred precisely, but you probably will have to reconstruct afterwards. If you transfer the data at "higher" level, i.e. using standard imap commands, for example using imapsync, the transfer will be slower, and you eventually lose flags or permissions (note that I personally never had any problem with imapsync, but can't say anything about other software). But you don't need to reconstruct afterwards. Could you eventually give us some figures? In my case, I transferred between 10 and 20 GB of data from 2.4.16 to 2.5.10 (this is not that much data, but it contained a complicated shared folder (public) namespace with thousands of deeply nested folders). However, reconstruct -V max took less than 10 minutes afterwards. Given that the source and the destination server were both running in VMs on the *same* physical hardware which is over 7 years old and even runs from normal HDD storage (it's a dual Xeon 5620 system, no SSDs), this is very fast. So I am curious what the size of your data is. You could do the transfer and let the reconstruct run over the weekend when your users are not at work. Based on my figures, you should be able to transfer and reconstruct several TB of messages during one weekend ... Regards, Binarus From lists at binarus.de Tue Dec 18 06:15:58 2018 From: lists at binarus.de (Binarus) Date: Tue, 18 Dec 2018 12:15:58 +0100 Subject: Question for upgrading In-Reply-To: <07899a02-1bc4-8f47-8f47-7cc0d4f3f7ee@onlight.com> References: <9e71e82ddcba5500a175c0e447131413@sarenet.es> <07899a02-1bc4-8f47-8f47-7cc0d4f3f7ee@onlight.com> Message-ID: <21086e50-2c4b-e46a-a87b-4f35cb3189e8@binarus.de> On 15.12.2018 17:05, Nic Bernstein wrote: > On 12/13/18 9:52 AM, Egoitz Aurrekoetxea wrote: >> >> I was trying to upgrade part of our Cyrus imap installation, >> concretely that one consisting in still 2.3. I was planning to set up >> Cyrus 3.0. I have seen all works properly except for the unexpunge >> command because as someone stated here, a reconstruct -V max was >> needed.The problem is that this reconstruct command, takes ages and >> I'm not able to keep the service offline so many time. So I have been >> thinking in the following scenario : >> > > Egoitz, > As long as you've followed all of the various steps needed to account > for version changes, as outlined in the Release Notes for /all/ > intermediary releases, then the last step should be the updating of the > indexes.? Rather than running "reconstruct -V max" on all mailboxes at > once, simply run it on subsets.? We've done this on large installations > without ill effect.? You can have the service on line during the > reconstruct, and, as you note, have most all functionality present. In my case, before the reconstruct had finished, I had several problems which might be not acceptable for large organizations. For example, users could not move messages between folders in their mailbox. I would consider this quite basic functionality, because deleting a message (with most clients) also means moving it (to trash). Functionality was back not before the reconstruct had finished completely. If interested, please see the respective thread from yesterday / today for details (I don't want to clutter the list by repeating all that stuff here). Regards, Binarus From javier at jangulo.net Tue Dec 18 07:50:49 2018 From: javier at jangulo.net (Javier Angulo) Date: Tue, 18 Dec 2018 13:50:49 +0100 Subject: Question for upgrading In-Reply-To: <21086e50-2c4b-e46a-a87b-4f35cb3189e8@binarus.de> References: <9e71e82ddcba5500a175c0e447131413@sarenet.es> <07899a02-1bc4-8f47-8f47-7cc0d4f3f7ee@onlight.com> <21086e50-2c4b-e46a-a87b-4f35cb3189e8@binarus.de> Message-ID: <292a2603-3ea3-6a5f-f5a0-0028cd9f36a8@jangulo.net> On 12/18/18 12:15 PM, Binarus wrote: > On 15.12.2018 17:05, Nic Bernstein wrote: >> On 12/13/18 9:52 AM, Egoitz Aurrekoetxea wrote: >>> I was trying to upgrade part of our Cyrus imap installation, >>> concretely that one consisting in still 2.3. I was planning to set up >>> Cyrus 3.0. I have seen all works properly except for the unexpunge >>> command because as someone stated here, a reconstruct -V max was >>> needed.The problem is that this reconstruct command, takes ages and >>> I'm not able to keep the service offline so many time. So I have been >>> thinking in the following scenario : >>> >> Egoitz, >> As long as you've followed all of the various steps needed to account >> for version changes, as outlined in the Release Notes for /all/ >> intermediary releases, then the last step should be the updating of the >> indexes.? Rather than running "reconstruct -V max" on all mailboxes at >> once, simply run it on subsets.? We've done this on large installations >> without ill effect.? You can have the service on line during the >> reconstruct, and, as you note, have most all functionality present. > In my case, before the reconstruct had finished, I had several problems > which might be not acceptable for large organizations. > > For example, users could not move messages between folders in their > mailbox. I would consider this quite basic functionality, because > deleting a message (with most clients) also means moving it (to trash). > Functionality was back not before the reconstruct had finished completely. Apart from those we also had some weird problems with the message sort order (using roundcube) before reconstruct was run. We split the mailboxes reconstruction into 8 parallel jobs (IO on the mailbox spool is the limit if you go from 2.3 -> 2.4/3.0). If you ran the reconstruction online, once finished I would suggest to check again all indexes version (some reconstruction jobs fail). Cheers From jschaeffer0922 at gmail.com Tue Dec 18 12:52:50 2018 From: jschaeffer0922 at gmail.com (Joshua Schaeffer) Date: Tue, 18 Dec 2018 17:52:50 +0000 Subject: sieve runtime error In-Reply-To: References: Message-ID: Thanks, this got me looking into sendmail a little closer. I've never used the program and didn't realize a dummy sendmail binary was installed on my system. Actually installed sendmail and it works now, messages are being filtered. On Tue, Dec 18, 2018 at 4:32 AM Simon Matter wrote: > > I'm trying to setup sieve and getting the following error in my logs: > > > > Dec 17 10:36:07 bllmail01 cyrus/lmtp[14530]: sieve runtime error for > > jschaeffer at harmonywave.net id > > : > > Reject: Sendmail process terminated normally, exit status 255 > > > > > > I'm following the documentation here: > > > https://www.cyrusimap.org/imap/reference/admin/sieve.html?highlight=sieve#testing-the-sieve-server > > > > I'm trying to get sieve working on my IMAP server. Using Ubuntu 16.04 > with > > the cyrus-imapd 2.4.18-3 package. I'm using the test sieve script shown > in > > the documentation to reject everything from my personal email: > > > > require ["reject","fileinto"]; > > if address :is :all "From" "jschaeffer0922 at gmail.com" > > { > > reject "testing"; > > } > > > > I then connect using sieveshell, upload the file, and activate it: > > > > root at bllmail01:~# sieveshell -u jschaeffer at harmonywave.net -a > > jschaeffer at harmonywave.net mail.harmonywave.cloud > > connecting to mail.harmonywave.cloud > > Please enter your password: > >> put /tmp/testing.sieve testing > >> activate testing > >> list > > testing <- active script > >> quit > > > > However when I send a test email from my personal account to the email > > I have setup on the IMAP server it always comes through and I get this > > in mail.log: > > > > Dec 17 10:36:07 bllmail01 cyrus/lmtp[14530]: sieve runtime error for > > jschaeffer at harmonywave.net id > > : > > Reject: Sendmail process terminated normally, exit status 255 > > I think sieve tries to send mail using the configured sendmail binary and > that doesn't work for some reason. You may check the sendmail config in > your imapd.conf and also consult the mail logs to learn more. > > Regards, > Simon > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at binarus.de Tue Dec 18 13:00:46 2018 From: lists at binarus.de (Binarus) Date: Tue, 18 Dec 2018 19:00:46 +0100 Subject: Running a script with cyradm throwing ReadLine errors In-Reply-To: <1545087479.1822015.1611895992.1214BC95@webmail.messagingengine.com> References: <1545087479.1822015.1611895992.1214BC95@webmail.messagingengine.com> Message-ID: <8f4f4fd3-70d2-de82-f43f-feab30e6f204@binarus.de> Dear ellie, On 17.12.2018 23:57, ellie timoney wrote: > Hi Binarus, > >> Could anybody please tell me what I might do wrong here? > > This kind of smells like maybe your system has two versions of perl installed (or two versions of Term::ReadLine, or maybe even two versions of Cyrus::IMAP::Shell), and they're getting in each other's way? > > I'm having a quick glance at the (2.5.10) source of Cyrus::IMAP::Shell and this caught my eye: > >> # ugh. ugh. suck. aieee. >> my $use_rl = 'Cyrus::IMAP::DummyReadline'; >> { >> if (eval { require Term::ReadLine; }) { >> $use_rl = 'Term::ReadLine'; >> } >> } I have done some further investigations (very roughly because I don't have the time at the moment). It seems that the code which parses the command line and the run parameters in Cyrus::IMAP::Shell is buggy (or at least not prepared to handle Term::ReadLine::Gnu). As a proof, I have reinstalled Term::ReadLine:Gnu and verified that the problem was showing again. Then I have replaced the following code in Cyrus::IMAP::Shell # trivial; wrapper for _run with correct setup sub run { my $cyradm; _run(\$cyradm, [*STDIN, *STDOUT, *STDERR], *__DATA__); } by the following code: # trivial; wrapper for _run with correct setup sub run { my $cyradm; open(*__DATA__, "./000"); _run(\$cyradm, [*STDIN, *STDOUT, *STDERR], *__DATA__); } In other words, I just have made sure that this mysterious *__DATA__ variable is reasonably defined in every case before _run is called. Now the command perl -MCyrus::IMAP::Shell -e 'run("000")' executed without any error message. To verify that the script worked as intended, I added a few lines to it: connect -noauthenticate localhost auth cyrus lm When run as shown above, it did exactly what it was supposed to. It asked for the password and then listed all mailboxes and their subfolders. So now I have at least a system where I can have Term::ReadLine::Gnu installed (and thus can have a history and command editing capabilities in cyradm) _and_ can execute a script, although the script's filename is hardcoded. Probably it would be absolutely trivial for the authors of Cyrus::IMAP::Shell to fix this issue. It would be very nice if somebody could care about it. Perhaps it's already fixed in the newer versions? I am still on 2.5.10. I have no idea why the "buggy" command line / argument parsing does not strike when Term::ReadLine::Gnu is uninstalled; I haven't grasped yet how *__DATA__ is supposed to be assigned a reasonable value to during the normal course of execution. I currently can only speculate that Term::ReadLine:: does this for us, while Term::ReadLine::Gnu doesn't. Regards, Binarus From ellie at fastmail.com Tue Dec 18 19:38:23 2018 From: ellie at fastmail.com (ellie timoney) Date: Wed, 19 Dec 2018 11:38:23 +1100 Subject: Running a script with cyradm throwing ReadLine errors Message-ID: <1545179903.1577854.1613038144.13B0043D@webmail.messagingengine.com> Hi Binarus, > Then I have replaced the following code in Cyrus::IMAP::Shell That's very interesting. Does the same modified code continue to work if you uninstall Term::Readline::Gnu again? That is to say, does the non-gnu version break with that addition, or continue to work? > In other words, I just have made sure that this mysterious *__DATA__ > variable is reasonably defined in every case before _run is called. I had a look in Shell.pm and found this comment near the top: > # run(*FH|'FH') > # read commands from the filehandle and pass to exec(); defaults to > # __DATA__ So maybe that explains where the expectation for __DATA__ is coming from... so: > # trivial; wrapper for _run with correct setup I wonder if the "correct setup" is not correct enough! > I have no idea why the "buggy" command line / argument parsing does not > strike when Term::ReadLine::Gnu is uninstalled; I haven't grasped yet > how *__DATA__ is supposed to be assigned a reasonable value to during > the normal course of execution. I currently can only speculate that > Term::ReadLine:: does this for us, while > Term::ReadLine::Gnu doesn't. I did a bit of reading, and apparently Term::ReadLine is a stub module that just loads "an implementation", which in your case wants to be Term::ReadLine::Gnu. My guess is that, when you uninstall Term::ReadLine::Gnu, Term::ReadLine no longer successfully compiles because it's missing an implementation, and consequently the fallback code I pointed out previously is used instead. So, from this I'm concluding that the "correct setup" from above is adequate for the Cyrus::IMAP::DummyReadline interface, but is not sufficient for a real ReadLine implementation. Sounds like we've found our bug! I'll have a bit of a play with it and see if I can find/fix the discrepancy between the interfaces :) Cheers, ellie On Wed, Dec 19, 2018, at 5:00 AM, Binarus wrote: > Dear ellie, > > On 17.12.2018 23:57, ellie timoney wrote: > > Hi Binarus, > > > >> Could anybody please tell me what I might do wrong here? > > > > This kind of smells like maybe your system has two versions of perl installed (or two versions of Term::ReadLine, or maybe even two versions of Cyrus::IMAP::Shell), and they're getting in each other's way? > > > > I'm having a quick glance at the (2.5.10) source of Cyrus::IMAP::Shell and this caught my eye: > > > >> # ugh. ugh. suck. aieee. > >> my $use_rl = 'Cyrus::IMAP::DummyReadline'; > >> { > >> if (eval { require Term::ReadLine; }) { > >> $use_rl = 'Term::ReadLine'; > >> } > >> } > > I have done some further investigations (very roughly because I don't > have the time at the moment). It seems that the code which parses the > command line and the run parameters in Cyrus::IMAP::Shell is buggy (or > at least not prepared to handle Term::ReadLine::Gnu). > > As a proof, I have reinstalled Term::ReadLine:Gnu and verified that the > problem was showing again. > > Then I have replaced the following code in Cyrus::IMAP::Shell > > # trivial; wrapper for _run with correct setup > sub run { > my $cyradm; > _run(\$cyradm, [*STDIN, *STDOUT, *STDERR], *__DATA__); > } > > by the following code: > > # trivial; wrapper for _run with correct setup > sub run { > my $cyradm; > open(*__DATA__, "./000"); > _run(\$cyradm, [*STDIN, *STDOUT, *STDERR], *__DATA__); > } > > In other words, I just have made sure that this mysterious *__DATA__ > variable is reasonably defined in every case before _run is called. > > Now the command > > perl -MCyrus::IMAP::Shell -e 'run("000")' > > executed without any error message. > > To verify that the script worked as intended, I added a few lines to it: > > connect -noauthenticate localhost > auth cyrus > lm > > When run as shown above, it did exactly what it was supposed to. It > asked for the password and then listed all mailboxes and their subfolders. > > So now I have at least a system where I can have Term::ReadLine::Gnu > installed (and thus can have a history and command editing capabilities > in cyradm) _and_ can execute a script, although the script's filename is > hardcoded. > > Probably it would be absolutely trivial for the authors of > Cyrus::IMAP::Shell to fix this issue. It would be very nice if somebody > could care about it. Perhaps it's already fixed in the newer versions? I > am still on 2.5.10. > > I have no idea why the "buggy" command line / argument parsing does not > strike when Term::ReadLine::Gnu is uninstalled; I haven't grasped yet > how *__DATA__ is supposed to be assigned a reasonable value to during > the normal course of execution. I currently can only speculate that > Term::ReadLine:: does this for us, while > Term::ReadLine::Gnu doesn't. > > Regards, > > Binarus > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From simon.matter at invoca.ch Tue Dec 18 23:51:11 2018 From: simon.matter at invoca.ch (Simon Matter) Date: Wed, 19 Dec 2018 05:51:11 +0100 Subject: sieve runtime error In-Reply-To: References: Message-ID: <60017efdc4b588a65d206be49f842be1.squirrel@webmail.bi.invoca.ch> > Thanks, this got me looking into sendmail a little closer. I've never used > the program and didn't realize a dummy sendmail binary was installed on my > system. Actually installed sendmail and it works now, messages are being > filtered. You don't have to install sendmail, also postfix has a compatible sendmail binary. Only the sendmail config in imapd.conf may not point to it. Regards, Simon > > On Tue, Dec 18, 2018 at 4:32 AM Simon Matter > wrote: > >> > I'm trying to setup sieve and getting the following error in my logs: >> > >> > Dec 17 10:36:07 bllmail01 cyrus/lmtp[14530]: sieve runtime error for >> > jschaeffer at harmonywave.net id >> > : >> > Reject: Sendmail process terminated normally, exit status 255 >> > >> > >> > I'm following the documentation here: >> > >> https://www.cyrusimap.org/imap/reference/admin/sieve.html?highlight=sieve#testing-the-sieve-server >> > >> > I'm trying to get sieve working on my IMAP server. Using Ubuntu 16.04 >> with >> > the cyrus-imapd 2.4.18-3 package. I'm using the test sieve script >> shown >> in >> > the documentation to reject everything from my personal email: >> > >> > require ["reject","fileinto"]; >> > if address :is :all "From" "jschaeffer0922 at gmail.com" >> > { >> > reject "testing"; >> > } >> > >> > I then connect using sieveshell, upload the file, and activate it: >> > >> > root at bllmail01:~# sieveshell -u jschaeffer at harmonywave.net -a >> > jschaeffer at harmonywave.net mail.harmonywave.cloud >> > connecting to mail.harmonywave.cloud >> > Please enter your password: >> >> put /tmp/testing.sieve testing >> >> activate testing >> >> list >> > testing <- active script >> >> quit >> > >> > However when I send a test email from my personal account to the email >> > I have setup on the IMAP server it always comes through and I get this >> > in mail.log: >> > >> > Dec 17 10:36:07 bllmail01 cyrus/lmtp[14530]: sieve runtime error for >> > jschaeffer at harmonywave.net id >> > : >> > Reject: Sendmail process terminated normally, exit status 255 >> >> I think sieve tries to send mail using the configured sendmail binary >> and >> that doesn't work for some reason. You may check the sendmail config in >> your imapd.conf and also consult the mail logs to learn more. >> >> Regards, >> Simon >> >> > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus From lists at binarus.de Wed Dec 19 04:29:15 2018 From: lists at binarus.de (Binarus) Date: Wed, 19 Dec 2018 10:29:15 +0100 Subject: Running a script with cyradm throwing ReadLine errors In-Reply-To: <1545179903.1577854.1613038144.13B0043D@webmail.messagingengine.com> References: <1545179903.1577854.1613038144.13B0043D@webmail.messagingengine.com> Message-ID: Dear ellie, On 19.12.2018 01:38, ellie timoney wrote: >> Then I have replaced the following code in Cyrus::IMAP::Shell > > That's very interesting. Does the same modified code continue to work if you uninstall Term::Readline::Gnu again? That is to say, does the non-gnu version break with that addition, or continue to work? I have just done that test: Yes, the same modified code continues to work even if Term::ReadLine::Gnu is uninstalled, i.e. my "patch" does not break the non-gnu version. >> In other words, I just have made sure that this mysterious *__DATA__ >> variable is reasonably defined in every case before _run is called. > > I had a look in Shell.pm and found this comment near the top: > >> # run(*FH|'FH') >> # read commands from the filehandle and pass to exec(); defaults to >> # __DATA__ I also had seen this comment, but couldn't make any sense from it. > So maybe that explains where the expectation for __DATA__ is coming from... so: > >> # trivial; wrapper for _run with correct setup > > I wonder if the "correct setup" is not correct enough! There are many aspects I didn't understand yet. To me, it seems that _run is called with a bunch of uninitialized parameters. For example, where are $cyradm and *__DATA__ initialized? I am currently lacking the time to do my homework (i.e. to completely understand how this is supposed to work under normal circumstances), so I don't want to let other persons waste their time for explaining it to me ... However, despite the fact that I haven't grasped the overall concept yet, there is obviously a bug with parsing the command line. >> I have no idea why the "buggy" command line / argument parsing does not >> strike when Term::ReadLine::Gnu is uninstalled; I haven't grasped yet >> how *__DATA__ is supposed to be assigned a reasonable value to during >> the normal course of execution. I currently can only speculate that >> Term::ReadLine:: does this for us, while >> Term::ReadLine::Gnu doesn't. > > I did a bit of reading, and apparently Term::ReadLine is a stub module that just loads "an implementation", which in your case wants to be Term::ReadLine::Gnu. My guess is that, when you uninstall Term::ReadLine::Gnu, Term::ReadLine no longer successfully compiles because it's missing an implementation, and consequently the fallback code I pointed out previously is used instead. So, from this I'm concluding that the "correct setup" from above is adequate for the Cyrus::IMAP::DummyReadline interface, but is not sufficient for a real ReadLine implementation. Sounds like we've found our bug! I have come to a similar conclusion, and "not sufficient" in this case probably means that *__DATA__ is not initialized (or assigned to) correctly. I still have no idea which part of the program is responsible to assign it the desired file descriptor under normal circumstances. Possibly Cyrus::IMAP::DummyReadLine does initialize *__DATA__ correctly (because that module knows who it belongs to :-) and what is needed later), while Term::ReadLine::Gnu can't know about *__DATA__'s existence at all. But this is just a completely uneducated guess. > I'll have a bit of a play with it and see if I can find/fix the discrepancy between the interfaces :) I'll try to free some time and eventually have a look into Cyrus::IMAP::DummyReadLine. I think we'll have to find out where *__DATA__ is normally initialized, and move that initialization to another place so that it happens regardless of the actual ReadLine "plugin". > Cheers, Again, thank you very much for all your help and your support! Binarus > ellie > > On Wed, Dec 19, 2018, at 5:00 AM, Binarus wrote: >> Dear ellie, >> >> On 17.12.2018 23:57, ellie timoney wrote: >>> Hi Binarus, >>> >>>> Could anybody please tell me what I might do wrong here? >>> >>> This kind of smells like maybe your system has two versions of perl installed (or two versions of Term::ReadLine, or maybe even two versions of Cyrus::IMAP::Shell), and they're getting in each other's way? >>> >>> I'm having a quick glance at the (2.5.10) source of Cyrus::IMAP::Shell and this caught my eye: >>> >>>> # ugh. ugh. suck. aieee. >>>> my $use_rl = 'Cyrus::IMAP::DummyReadline'; >>>> { >>>> if (eval { require Term::ReadLine; }) { >>>> $use_rl = 'Term::ReadLine'; >>>> } >>>> } >> >> I have done some further investigations (very roughly because I don't >> have the time at the moment). It seems that the code which parses the >> command line and the run parameters in Cyrus::IMAP::Shell is buggy (or >> at least not prepared to handle Term::ReadLine::Gnu). >> >> As a proof, I have reinstalled Term::ReadLine:Gnu and verified that the >> problem was showing again. >> >> Then I have replaced the following code in Cyrus::IMAP::Shell >> >> # trivial; wrapper for _run with correct setup >> sub run { >> my $cyradm; >> _run(\$cyradm, [*STDIN, *STDOUT, *STDERR], *__DATA__); >> } >> >> by the following code: >> >> # trivial; wrapper for _run with correct setup >> sub run { >> my $cyradm; >> open(*__DATA__, "./000"); >> _run(\$cyradm, [*STDIN, *STDOUT, *STDERR], *__DATA__); >> } >> >> In other words, I just have made sure that this mysterious *__DATA__ >> variable is reasonably defined in every case before _run is called. >> >> Now the command >> >> perl -MCyrus::IMAP::Shell -e 'run("000")' >> >> executed without any error message. >> >> To verify that the script worked as intended, I added a few lines to it: >> >> connect -noauthenticate localhost >> auth cyrus >> lm >> >> When run as shown above, it did exactly what it was supposed to. It >> asked for the password and then listed all mailboxes and their subfolders. >> >> So now I have at least a system where I can have Term::ReadLine::Gnu >> installed (and thus can have a history and command editing capabilities >> in cyradm) _and_ can execute a script, although the script's filename is >> hardcoded. >> >> Probably it would be absolutely trivial for the authors of >> Cyrus::IMAP::Shell to fix this issue. It would be very nice if somebody >> could care about it. Perhaps it's already fixed in the newer versions? I >> am still on 2.5.10. >> >> I have no idea why the "buggy" command line / argument parsing does not >> strike when Term::ReadLine::Gnu is uninstalled; I haven't grasped yet >> how *__DATA__ is supposed to be assigned a reasonable value to during >> the normal course of execution. I currently can only speculate that >> Term::ReadLine:: does this for us, while >> Term::ReadLine::Gnu doesn't. >> >> Regards, >> >> Binarus >> ---- >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > ---- > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > From lists at binarus.de Wed Dec 19 08:48:53 2018 From: lists at binarus.de (Binarus) Date: Wed, 19 Dec 2018 14:48:53 +0100 Subject: Running a script with cyradm throwing ReadLine errors In-Reply-To: <1545179903.1577854.1613038144.13B0043D@webmail.messagingengine.com> References: <1545179903.1577854.1613038144.13B0043D@webmail.messagingengine.com> Message-ID: <04013d36-713d-da2c-4ffc-c3db5be88d9a@binarus.de> Dear ellie, On 19.12.2018 01:38, ellie timoney wrote: > I did a bit of reading, and apparently Term::ReadLine is a stub module that just loads "an implementation", which in your case wants to be Term::ReadLine::Gnu. My guess is that, when you uninstall Term::ReadLine::Gnu, Term::ReadLine no longer successfully compiles because it's missing an implementation, and consequently the fallback code I pointed out previously is used instead. So, from this I'm concluding that the "correct setup" from above is adequate for the Cyrus::IMAP::DummyReadline interface, but is not sufficient for a real ReadLine implementation. Sounds like we've found our bug! Some additional findings: 1) Cyrus::IMAP::DummyReadLine ----------------------------- Looking again at that code # ugh. ugh. suck. aieee. my $use_rl = 'Cyrus::IMAP::DummyReadline'; { if (eval { require Term::ReadLine; }) { $use_rl = 'Term::ReadLine'; } } I believe that $use_rl *always* equals 'Term::ReadLine' after having executed it. This is for the following reason: In newer Perl versions, Term::ReadLine is a core module. Everybody has it installed. This means that the require Term::ReadLine will always be successful. I did a test to prove that. I uninstalled Term::ReadLine::Gnu again and changed the code above to the following (note the last line): # ugh. ugh. suck. aieee. my $use_rl = 'Cyrus::IMAP::DummyReadline'; { if (eval { require Term::ReadLine; }) { $use_rl = 'Term::ReadLine'; } } print $use_rl."\n"; As expected, perl -MCyrus::IMAP::Shell -e 'run("./000")' now prints Term::ReadLine as first line on the terminal. This was still the case (as expected again) after reinstalling Term::ReadLine::Gnu. *That means:* Cyrus::IMAP::DummyReadLine is not related to the problem or its solution in any way. It never gets pulled in, at least with recent Perl distributions which have Term::ReadLine included [as a core module]. 2) *__DATA__ variable / file handle ----------------------------------- After having read the Perl docs about that mysterious __DATA__ variable (see below), grep'ing the whole Perl module trees for the string __DATA__, and analyzing the results, I came to the conclusion that the *__DATA__ variable *never* is assigned any value during normal program execution, meaning that _run() always is called with undef as its last parameter. As a proof, I have replaced the following code # trivial; wrapper for _run with correct setup sub run { my $cyradm; _run(\$cyradm, [*STDIN, *STDOUT, *STDERR], *__DATA__); } by # trivial; wrapper for _run with correct setup sub run { my $cyradm; print Dumper(${*Cyrus::IMAP::Shell::__DATA__})."\n"; _run(\$cyradm, [*STDIN, *STDOUT, *STDERR], *__DATA__); } and have added use Data::Dumper at the beginning of the file. Now, when executing perl -MCyrus::IMAP::Shell -e 'run("./000")', it printed $VAR1 = undef; as the first line on the terminal. This was the case whether Term::ReadLine::Gnu was installed or not. To further back that finding, I reverted my changes and then changed the code again as follows (note the last parameter to _run()): # trivial; wrapper for _run with correct setup sub run { my $cyradm; _run(\$cyradm, [*STDIN, *STDOUT, *STDERR], undef); } This did not change the module's behavior compared to the original code. While it now threw the errors described in my first post again (as expected) when Term::ReadLine::Gnu was installed, it threw no errors when it was not installed. *That means:* *__DATA__ (the third parameter to _run) is always undef, and this does not lead to errors being thrown or the compilation / execution being aborted as long as Term::ReadLine::Gnu is not installed, but makes Term::ReadLine::Gnu (if it is installed) throw errors and abort the compilation / execution of the script. (Too) short explanation of the __DATA__ variable: This is a predefined filehandle in Perl which could be used as follows. Suppose you have a script: package ... [code here] __DATA__ data value 1 data value 2 ... Then you can access the data values (i.e. all values which come behind the __DATA__ statement) using the special filehandle [PACKAGE NAME]::DATA (or __DATA__ as well?) from within the package code. For details, see https://perldoc.perl.org/perldata.html#Special-Literals Since there is no __DATA__ statement in any of Cyrus' Perl modules or in modules they use, it is clear that the *__DATA__ filehandle is always undef. To be honest, I can't understand why it is used. I originally thought that it would be initialized by some other module (directly or indirectly) which is used by Cyrus::IMAP::Shell, but my analysis showed that it isn't (unless I have missed something, which might well be the case). 3) No script execution at all ----------------------------- I have to apologize that I didn't mention this in my first post; the reason was that I did my first tests with an *empty* script. However, now that my script is meaningful, I noticed that it did not get executed at all even if Term::ReadLine::Gnu was not installed. In other words, when I uninstalled Term::ReadLine::Gnu again and ran perl -MCyrus::IMAP::Shell -e 'run("./000")' the script "000" was *not* executed when the original version of Cyrus::IMAP::Shell was in place. I didn't notice this from the beginning on because I did my first tests with an empty script, and no errors were thrown. When I changed the module's code and assigned *__DATA__ a handle to the file desired (as shown in my previous post), the script was executed. To put it all together: ----------------------- - Cyrus::IMAP::DummyReadLine is never used if the Perl distribution is recent (i.e. already includes Term::ReadLine). This applies whether Term::ReadLine::Gnu is installed or not. - _run() is always called with undef as third parameter. If Term::ReadLine::Gnu is installed, errors are thrown if we execute perl -MCyrus::IMAP::Shell -e 'run("./000")' in a terminal. No errors are thrown if Term::ReadLine::Gnu is not installed. The errors Term::ReadLine::Gnu throws are clearly due to the fact that the third argument to _run() is undef (the error thrown reads "Bad filehandle: __DATA__ at ..."). - With the original version of Cyrus::IMAP::Shell, even when Term::ReadLine::Gnu is not installed, the script is not executed when we issue perl -MCyrus::IMAP::Shell -e 'run("")' in a terminal. - If we assign *__DATA__ the filehandle to the desired script in the run() function before calling _run(), the desired script does get executed when we issue perl -MCyrus::IMAP::Shell -e 'run("./000")'. This works whether Term::ReadLine::Gnu is installed or not. Hence, the problem is clearly related to Cyrus::IMAP::Shell. It does either just not execute the script given (if Term::ReadLine::Gnu is not installed), or it makes Term::ReadLine::Gnu throw errors (if it is installed) due to the uninitialized file handle. It would be very nice if somebody could fix that bug or correct Cyrus::IMAP::Shell's man page accordingly. Fortunately, there is a workaround. We could make cyradm execute a script directly with no problem: cat 000 | cyradm Shame on me that I didn't see this earlier - it would have solved my problem and would have saved me a whole day or two. I guess Cyrus::IMAP::Shell's man page had distracted me too much... However, I'd still be strongly interested in a bug fix for the module. In fact, I already have a clean solution in mind, but I'd like to test it before posting it here. I'll report back in a few hours. Thank you very much again! Regards, Binarus From lists at binarus.de Wed Dec 19 12:11:26 2018 From: lists at binarus.de (Binarus) Date: Wed, 19 Dec 2018 18:11:26 +0100 Subject: Running a script with cyradm throwing ReadLine errors In-Reply-To: <1545179903.1577854.1613038144.13B0043D@webmail.messagingengine.com> References: <1545179903.1577854.1613038144.13B0043D@webmail.messagingengine.com> Message-ID: <9e16f66f-ba49-2c4d-ebf6-fd85655bcf0d@binarus.de> Dear ellie, > I did a bit of reading, and apparently Term::ReadLine is a stub module that just loads "an implementation", which in your case wants to be Term::ReadLine::Gnu. My guess is that, when you uninstall Term::ReadLine::Gnu, Term::ReadLine no longer successfully compiles because it's missing an implementation, and consequently the fallback code I pointed out previously is used instead. So, from this I'm concluding that the "correct setup" from above is adequate for the Cyrus::IMAP::DummyReadline interface, but is not sufficient for a real ReadLine implementation. Sounds like we've found our bug! the more I thought about it, the clearer it got. I do not think any more that the *real* issue is which stub Term::ReadLine uses. Different stubs might react differently when fed with undefined file handles, but this is only a distracting secondary issue. The real culprit is how the run function is implemented. Let's consider the original code for that function again: # trivial; wrapper for _run with correct setup sub run { my $cyradm; _run(\$cyradm, [*STDIN, *STDOUT, *STDERR], *__DATA__); } How should *__DATA__ have become a handle to the desired file (which should be executed) in any way? There is absolutely no parameter parsing, and after having researched what special meaning __DATA__ has, it became also clear that *__DATA__ isn't mysteriously assigned a reasonable value before run() is called. So I made some very trivial changes. The function now reads: # trivial; wrapper for _run with correct setup sub run { my ($cyradm, $fh); my $file = shift; defined $file || die "No filename given, aborting.\n"; open($fh, $file) || die "Could not open file '$file', aborting.\n"; _run(\$cyradm, [*STDIN, *STDOUT, *STDERR], $fh); } Now the whole thing works as expected, regardless of what stub modules are installed for Term::ReadLine. We could improve that code further; for example, it lacks a check if there is the right number of parameters (additional parameters are currently just ignored). Personally, I wouldn't need detailed checks; I just want it to execute that script file, avoiding ugly error messages from Perl itself relating to undefined values and so on. At a first glance, I couldn't see how the new code could be incompatible to the existing version. At least, there are no other calls to run() in that module (only to _run() which I didn't alter). I am quite sure that you have a bunch regression tests for all your modules, so let's see what they reveal. I am looking forward to your comments ... Thank you very much again! Regards, Binarus