From ml at netfence.it  Tue May  7 07:35:32 2019
From: ml at netfence.it (Andrea Venturoli)
Date: Tue, 7 May 2019 13:35:32 +0200
Subject: Troubles with CardBook
Message-ID: <2ed58c5f-e108-23cb-e833-1a5a647834e9@netfence.it>

Hello.

I'm maintaining several setups of Cyrus IMAPD 2.5.12 on FreeBSD 11.2/amd64.
I'm also using Thunderbird+CardBook on some of them without too many 
troubles.



Today, however I tried this on one server where I hadn't used CardDAV 
before (but had used CalDav succesfully) and run into some troubles.

First, "Default" addressbook for user myuser at mydomain.it was not 
automatically created when I configured CardBook; after creating it 
manually through cyradmin, CardBook would connect fine.

Later, I moved some hundred contacts into CardBook and let it 
synchronize: I watched it progress and noticed any contact who was 
transferred to the server disappeared from CardBook.
No use synchronizing again, those contacts would not reappear.
So I tried configuring the same addressbook on another machine: it would 
connect fine, but show no contact.
If I add further contacts, they are just "eaten" when I synchronize.
The contacts are however present on the server if I go and look at the 
filesystem level!



Now, this servers differs from the others where I succesfully used 
carddav for two reasons: it uses "virtdomains: userid" and 
"unixhierarchysep: yes".
(I also have "altnamespace: yes", but that's not unique).


I believe "unixhierarchysep: yes" is the culprit here, because this is 
what happened:
_ I pointed CardBook at 
https://mail.mydomain.it/dav/addressbooks/user/myuser at mydomain.it/Default/
_ at the file system level I saw the following appear:
  /var/spool/imap/domain/mydomain.it/user/myuser/#addressbooks
  /var/spool/imap/domain/mydomain.it/user/myuser/#addressbooks^Default
_ still CardBook would say no such addressbook was available;
_ I lunched cyradm and issued "cm user 
user/myuser/#addressbooks/Default at mydomain.it";
_ /var/spool/imap/domain/mydomain.it/user/myuser/#addressbooks/Default 
appeared and CardBook was happy and created the addressbook.

After I added those hundred contacts, I correctly see them in 
/var/spool/imap/domain/mydomain.it/user/myuser/#addressbooks/Default, 
but still CardBook does not find them.




Is this a known bug?
Any hint on how to solve or at least get some more debugging info?

  bye & Thanks
	av.

From karagian at gmail.com  Tue May  7 08:07:17 2019
From: karagian at gmail.com (Savvas Karagiannidis)
Date: Tue, 7 May 2019 15:07:17 +0300
Subject: Troubles with CardBook
In-Reply-To: <2ed58c5f-e108-23cb-e833-1a5a647834e9@netfence.it>
References: <2ed58c5f-e108-23cb-e833-1a5a647834e9@netfence.it>
Message-ID: <CADfvMFN5wY5ZV0O6U8LsG=6W0LJN2gurcDAyPMTih+xtzbNrcA@mail.gmail.com>

hi Andrea,
I believe you hit the bug I reported here:
https://github.com/cyrusimap/cyrus-imapd/issues/2001
version 2.5.11 introduced this bug.
If you can recompile cyrus, you could try reverting these changes to
imap/mboxname.c:
https://github.com/cyrusimap/cyrus-imapd/pull/67/files#diff-3f7e9582b7051be83c0edd299be8f5a5
Otherwise you should either downgrade to 2.5.10 or upgrade to 3.0.x


Regards,
Savvas Karagiannidis

On Tue, May 7, 2019 at 2:36 PM Andrea Venturoli <ml at netfence.it> wrote:

> Hello.
>
> I'm maintaining several setups of Cyrus IMAPD 2.5.12 on FreeBSD 11.2/amd64.
> I'm also using Thunderbird+CardBook on some of them without too many
> troubles.
>
>
>
> Today, however I tried this on one server where I hadn't used CardDAV
> before (but had used CalDav succesfully) and run into some troubles.
>
> First, "Default" addressbook for user myuser at mydomain.it was not
> automatically created when I configured CardBook; after creating it
> manually through cyradmin, CardBook would connect fine.
>
> Later, I moved some hundred contacts into CardBook and let it
> synchronize: I watched it progress and noticed any contact who was
> transferred to the server disappeared from CardBook.
> No use synchronizing again, those contacts would not reappear.
> So I tried configuring the same addressbook on another machine: it would
> connect fine, but show no contact.
> If I add further contacts, they are just "eaten" when I synchronize.
> The contacts are however present on the server if I go and look at the
> filesystem level!
>
>
>
> Now, this servers differs from the others where I succesfully used
> carddav for two reasons: it uses "virtdomains: userid" and
> "unixhierarchysep: yes".
> (I also have "altnamespace: yes", but that's not unique).
>
>
> I believe "unixhierarchysep: yes" is the culprit here, because this is
> what happened:
> _ I pointed CardBook at
> https://mail.mydomain.it/dav/addressbooks/user/myuser at mydomain.it/Default/
> _ at the file system level I saw the following appear:
>   /var/spool/imap/domain/mydomain.it/user/myuser/#addressbooks
>   /var/spool/imap/domain/mydomain.it/user/myuser/#addressbooks^Default
> _ still CardBook would say no such addressbook was available;
> _ I lunched cyradm and issued "cm user
> user/myuser/#addressbooks/Default at mydomain.it";
> _ /var/spool/imap/domain/mydomain.it/user/myuser/#addressbooks/Default
> appeared and CardBook was happy and created the addressbook.
>
> After I added those hundred contacts, I correctly see them in
> /var/spool/imap/domain/mydomain.it/user/myuser/#addressbooks/Default,
> but still CardBook does not find them.
>
>
>
>
> Is this a known bug?
> Any hint on how to solve or at least get some more debugging info?
>
>   bye & Thanks
>         av.
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190507/f7f38080/attachment.html>

From ml at netfence.it  Wed May  8 05:33:09 2019
From: ml at netfence.it (Andrea Venturoli)
Date: Wed, 8 May 2019 11:33:09 +0200
Subject: Troubles with CardBook
In-Reply-To: <CADfvMFN5wY5ZV0O6U8LsG=6W0LJN2gurcDAyPMTih+xtzbNrcA@mail.gmail.com>
References: <2ed58c5f-e108-23cb-e833-1a5a647834e9@netfence.it>
 <CADfvMFN5wY5ZV0O6U8LsG=6W0LJN2gurcDAyPMTih+xtzbNrcA@mail.gmail.com>
Message-ID: <06660bb0-75b1-5298-a78a-d6b93562841e@netfence.it>

On 5/7/19 2:07 PM, Savvas Karagiannidis wrote:
> hi Andrea,
> I believe you hit the bug I reported here:
> https://github.com/cyrusimap/cyrus-imapd/issues/2001
> version 2.5.11 introduced this bug.
> If you can recompile cyrus, you could try reverting these changes to 
> imap/mboxname.c:
> https://github.com/cyrusimap/cyrus-imapd/pull/67/files#diff-3f7e9582b7051be83c0edd299be8f5a5
> Otherwise you should either downgrade to 2.5.10 or upgrade to 3.0.x

Hello.

First of all, thanks for your answer.

I tried looking at reverting the changes you linked me to: that's not an 
easy task, as there are 2.5.10->2.5.11, but this code seems to have 
changed again in 2.5.12.
Is anyone able to pinpoint what exactly needs changing?

I'm evaluating going back to 2.5.10 for now (although I see the fixed 
bug list is long since then :( ); is there anything weird to be 
excpected? Like database incompatilities, reconstructions needed, etc...

I'm not daring the migration to 3.0 right now...
BTW is it known how long 2.5 will still be supported?

  bye & Thanks
	av.

From clarra at cc.upv.es  Fri May 10 07:30:36 2019
From: clarra at cc.upv.es (=?UTF-8?Q?Carlos_Larra=c3=b1aga?=)
Date: Fri, 10 May 2019 13:30:36 +0200
Subject: squatter not used after upgrade to cyrus 3.0.8
In-Reply-To: <20181025150940.Horde.ZSBGZc-bdo4Ww0nHKDroxYH@webmail.uni-tuebingen.de>
References: <20180917140152.Horde.W8VK46keZMD6i3ABlUUhwzX@webmail.uni-tuebingen.de>
 <20180917123123.GA24774@io.chezmoi.fr>
 <20180917171814.Horde.d0cTBJKuZdVAj2dx5sWRBef@webmail.uni-tuebingen.de>
 <20181025150940.Horde.ZSBGZc-bdo4Ww0nHKDroxYH@webmail.uni-tuebingen.de>
Message-ID: <c40a88fb-87e9-537a-f9a3-7f38fc5bb8c5@cc.upv.es>

Hi Michael,

Do you resolved this problem?
I'm having the same issue with cyrus 3.0.9 not accessing cyrus.squat files.

I've put in my impad.conf:
 ? search_engine: squat
 ? search_fuzzy_always: 1

Any recomendation will be appreciated.

Best regards,
Carlos.


El 25/10/2018 a las 15:09, Michael Menge escribi?:
> Hi,
>
> Quoting Michael Menge <michael.menge at zdv.uni-tuebingen.de>:
>
>> Hi,
>>
>> Quoting Albert Shih <Albert.Shih at obspm.fr>:
>>
>>> Le 17/09/2018 ? 14:01:52+0200, Michael Menge a ?crit
>>>> Hi,
>>>>
>>>> we recently upgrade from Cyrus-Imapd 2.4.x to 3.0.8. After some initial
>>>> problems
>>>> which we could fix cyrus imapd 3.0.8 is running stable. The one remaining
>>>> problem
>>>> we receive reports about is, that the search is not working/too slow.
>>>>
>>>> First we discovered that one of the options for Squatter are not supported
>>>> anymore, "-s Skip mailboxes whose index file is older than their current
>>>> squat file (within a small time delta)." and that squatter does not like
>>>> "-r" in combination with the source "user"
>>>>
>>>> ? > squatter -C /etc/imapd_be.conf -r? user
>>>> ? fatal error: Internal error: assertion failed: lib/cyrusdb_twoskip.c:
>>>> 2339: keylen
>>>>
>>>>
>>>> But after reindexing all mailboxes the search is still slow. I tried to
>>>> debug this and
>>>> found with strace that cyrus didn't try to open the cyrus.squat file for the
>>>> mailbox.
>>>>
>>>> I suspect that I am missed some configuration change. So here is our
>>>> imapd.conf for our backends
>>>
>>> If I'm correct you need :
>>>
>>> ?search_fuzzy_always: on
>>>
>>> in your config.
>>>
>>> If you can tell me if it's work...I would really appreciate. Because I
>>> activated that but I'm not able to see if it's work really.
>>>
>> Thanks for your help.
>>
>> I did tried it on my test server and it seems to be a bit faster,
>> but that could be due to caching. Strace still didn't show any access
>> to the cyrus.squat file.
>>
>> For information: We only use squatter. No Xapia. And we had much faster
>> search with Cyrus-Imapd 2.3.x and 2.4.x. I don't have the timings form
>> the old system but our users are complaining and they receive timeouts
>> in our horde/imp webmailer, which they did't receive before.
>>
>> Any other ideas are appreciated.
>
> I still have the problem that search in cyrus imap 3.0.8 with search engine
> squatter is slow compared to 2.4.20. I try to figure out if the squatter
> search engine is working in cyurs imapd 3.0 and I messed up my configuration,
> or if my configuration should work but squatter is broken.
>
> I did set up a test environment to compare the old and new versions.
> To verify that the search is indeed slower with 3.0.8
>
> I used two different searches 'B SEARCH TEXT "Test"' and 'B SEARCH HEADER 
> X-comment Unirundmail'
>
> === 2.4.20 === SEARCH TEXT
>
> A SELECT INBOX
> * 4321 EXISTS
> * 4321 RECENT
> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
> * OK [UNSEEN 1] Ok
> * OK [UIDVALIDITY 1540372444] Ok
> * OK [UIDNEXT 93369] Ok
> * OK [HIGHESTMODSEQ 2] Ok
> * OK [URLMECH INTERNAL] Ok
> A OK [READ-WRITE] Completed
> B SEARCH TEXT "Test"
> * SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 
> 27 28 29 30 31 33 34 35 36 37 38 39
> ....
> 4292 4294 4295 4296 4298 4299 4300 4301 4303 4306 4307 4309 4310 4315 4316 
> 4317 4318 4321
> B OK Completed (1996 msgs in 0.690 secs)
>
>
> === 3.0.8 === SEARCH TEXT
>
> a SELECT INBOX
> * 4321 EXISTS
> * 0 RECENT
> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk $Forwarded $mdnsent 
> $label1 $label2 $label3 hordetest testflag)
> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk 
> $Forwarded $mdnsent $label1 $label2 $label3 hordetest testflag \*)] Ok
> * OK [UNSEEN 3737] Ok
> * OK [UIDVALIDITY 1238498084] Ok
> * OK [UIDNEXT 93373] Ok
> * OK [HIGHESTMODSEQ 98491] Ok
> * OK [URLMECH INTERNAL] Ok
> * OK [ANNOTATIONS 65536] Ok
> a OK [READ-WRITE] Completed
> B SEARCH TEXT "Test"
> * SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 
> 27 28 29 30 31 33 34 35 36 37 38 39
> ....
> 4274 4275 4276 4277 4278 4279 4285 4286 4287 4288 4292 4294 4295 4296 4298 
> 4299 4300 4301 4303 4306 4307 4309 4310 4315 4316 4317 4318 4321
> B OK Completed (1935 msgs in 2.580 secs)
>
> ==== 2.4.20 === SEARCH HEADER
>
> a SELECT INBOX
> * 4321 EXISTS
> * 0 RECENT
> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
> * OK [UNSEEN 1] Ok
> * OK [UIDVALIDITY 1540372444] Ok
> * OK [UIDNEXT 93369] Ok
> * OK [HIGHESTMODSEQ 2] Ok
> * OK [URLMECH INTERNAL] Ok
> a OK [READ-WRITE] Completed
> b SEARCH HEADER X-comment Unirundmail
> * SEARCH 4283 4291 4313 4319 4320
> b OK Completed (5 msgs in 0.020 secs)
>
> ==== 3.0.8 === SEARCH HEADER
>
> a SELECT INBOX
> * 4321 EXISTS
> * 0 RECENT
> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk $Forwarded $mdnsent 
> $label1 $label2 $label3 hordetest testflag)
> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk 
> $Forwarded $mdnsent $label1 $label2 $label3 hordetest testflag \*)] Ok
> * OK [UNSEEN 3737] Ok
> * OK [UIDVALIDITY 1238498084] Ok
> * OK [UIDNEXT 93373] Ok
> * OK [HIGHESTMODSEQ 98491] Ok
> * OK [URLMECH INTERNAL] Ok
> * OK [ANNOTATIONS 65536] Ok
> a OK [READ-WRITE] Completed
> b SEARCH HEADER X-comment Unirundmail
> * SEARCH 4283 4291 4313 4319 4320
> b OK Completed (5 msgs in 0.370 secs)
>
> ===
>
> There is also a big discrepancy between time indicated in the "OK Completed" 
> and the time from
> sending the search command till the return of the result, which is 0.890 sec 
> compared to ~30 sec
> on the production system.
>
> I used strace on the imapd processes while searching to verify that the squat 
> file was used
> in 2.4 but not in 3.0.
> I could see open events for the squat file and the messages that where found 
> for 2.4.20
> and no open event (not even a failed one) to the squat file but instead open 
> events for
> all message files in that folder for 3.0.8
>
> I read the documentation and source code and tried to figure out if i am 
> missing some
> config options, or if i could pinpoint a function where the search was turning 
> the
> wrong way. I used "perf -g" the see the call graphs and to figure out where the
> call graphs change
>
> I can see that the same functions are called up to "index_search", and that 
> the called functions
> change at that point. I know that the search code got restructured to support 
> different search
> engines and that some functions got renamed. I have attached the perf report 
> output, so that
> someone with a better understanding of the code can see whats going on. I can 
> provide the
> perf.data files if it helps.
>
> Can someone confirm or refute that the squatter search engine is working with 
> cyrus imapd 3.0.x?
>
> Is "search_engine: squat" in imapd.conf and a "squatter" event in cyrus.conf 
> is sufficient to
> use the squatter search index in 3.0 or are there other options steps required.
>
> Regards
>
> ?? Michael Menge
>
> PS. link to my original post with my imapd.conf
> https://lists.andrew.cmu.edu/pipermail/info-cyrus/2018-September/040395.html
>
>
>
>
>
>
>
> --------------------------------------------------------------------------------
> M.Menge??????????????????????????????? Tel.: (49) 7071/29-70316
> Universit?t T?bingen?????????????????? Fax.: (49) 7071/29-5912
> Zentrum f?r Datenverarbeitung????????? mail: michael.menge at zdv.uni-tuebingen.de
> W?chterstra?e 76
> 72074 T?bingen
>
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190510/ec8eecd6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3627 bytes
Desc: Firma criptogr??fica S/MIME
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190510/ec8eecd6/attachment-0001.p7s>

From michael.menge at zdv.uni-tuebingen.de  Fri May 10 08:18:12 2019
From: michael.menge at zdv.uni-tuebingen.de (Michael Menge)
Date: Fri, 10 May 2019 14:18:12 +0200
Subject: squatter not used after upgrade to cyrus 3.0.8
In-Reply-To: <c40a88fb-87e9-537a-f9a3-7f38fc5bb8c5@cc.upv.es>
References: <20180917140152.Horde.W8VK46keZMD6i3ABlUUhwzX@webmail.uni-tuebingen.de>
 <20180917123123.GA24774@io.chezmoi.fr>
 <20180917171814.Horde.d0cTBJKuZdVAj2dx5sWRBef@webmail.uni-tuebingen.de>
 <20181025150940.Horde.ZSBGZc-bdo4Ww0nHKDroxYH@webmail.uni-tuebingen.de>
 <c40a88fb-87e9-537a-f9a3-7f38fc5bb8c5@cc.upv.es>
Message-ID: <20190510141812.Horde.Esvmh3nCs6jL7BoAnCE6bLN@webmail.uni-tuebingen.de>

Hi Carlos,

Quoting Carlos Larra?aga <clarra at cc.upv.es>:

> Hi Michael,
>
> Do you resolved this problem?
> I'm having the same issue with cyrus 3.0.9 not accessing cyrus.squat files.
>

no the problem is not completly resolved jet.
See https://github.com/cyrusimap/cyrus-imapd/issues/2598 for details.

> I've put in my impad.conf:
> ? search_engine: squat
> ? search_fuzzy_always: 1

AFAIK the squat search engine does not support fuzzy search.
I am sure in my testing this setting didn't resolve the slow
search, but i don't remember if this setting did nothing,
or  failed to find anything at all or what else happened
with this configuration.

>
> Any recomendation will be appreciated.
>
> Best regards,
> Carlos.
>
>
> El 25/10/2018 a las 15:09, Michael Menge escribi?:
>> Hi,
>>
>> Quoting Michael Menge <michael.menge at zdv.uni-tuebingen.de>:
>>
>>> Hi,
>>>
>>> Quoting Albert Shih <Albert.Shih at obspm.fr>:
>>>
>>>> Le 17/09/2018 ? 14:01:52+0200, Michael Menge a ?crit
>>>>> Hi,
>>>>>
>>>>> we recently upgrade from Cyrus-Imapd 2.4.x to 3.0.8. After some initial
>>>>> problems
>>>>> which we could fix cyrus imapd 3.0.8 is running stable. The one remaining
>>>>> problem
>>>>> we receive reports about is, that the search is not working/too slow.
>>>>>
>>>>> First we discovered that one of the options for Squatter are not
>>>>> supported
>>>>> anymore, "-s Skip mailboxes whose index file is older than their current
>>>>> squat file (within a small time delta)." and that squatter does not like
>>>>> "-r" in combination with the source "user"
>>>>>
>>>>> ? > squatter -C /etc/imapd_be.conf -r? user
>>>>> ? fatal error: Internal error: assertion failed: lib/cyrusdb_twoskip.c:
>>>>> 2339: keylen
>>>>>
>>>>>
>>>>> But after reindexing all mailboxes the search is still slow. I tried to
>>>>> debug this and
>>>>> found with strace that cyrus didn't try to open the cyrus.squat
>>>>> file for the
>>>>> mailbox.
>>>>>
>>>>> I suspect that I am missed some configuration change. So here is our
>>>>> imapd.conf for our backends
>>>>
>>>> If I'm correct you need :
>>>>
>>>> ?search_fuzzy_always: on
>>>>
>>>> in your config.
>>>>
>>>> If you can tell me if it's work...I would really appreciate. Because I
>>>> activated that but I'm not able to see if it's work really.
>>>>
>>> Thanks for your help.
>>>
>>> I did tried it on my test server and it seems to be a bit faster,
>>> but that could be due to caching. Strace still didn't show any access
>>> to the cyrus.squat file.
>>>
>>> For information: We only use squatter. No Xapia. And we had much faster
>>> search with Cyrus-Imapd 2.3.x and 2.4.x. I don't have the timings form
>>> the old system but our users are complaining and they receive timeouts
>>> in our horde/imp webmailer, which they did't receive before.
>>>
>>> Any other ideas are appreciated.
>>
>> I still have the problem that search in cyrus imap 3.0.8 with search engine
>> squatter is slow compared to 2.4.20. I try to figure out if the squatter
>> search engine is working in cyurs imapd 3.0 and I messed up my
>> configuration,
>> or if my configuration should work but squatter is broken.
>>
>> I did set up a test environment to compare the old and new versions.
>> To verify that the search is indeed slower with 3.0.8
>>
>> I used two different searches 'B SEARCH TEXT "Test"' and 'B SEARCH
>> HEADER X-comment Unirundmail'
>>
>> === 2.4.20 === SEARCH TEXT
>>
>> A SELECT INBOX
>> * 4321 EXISTS
>> * 4321 RECENT
>> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
>> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
>> * OK [UNSEEN 1] Ok
>> * OK [UIDVALIDITY 1540372444] Ok
>> * OK [UIDNEXT 93369] Ok
>> * OK [HIGHESTMODSEQ 2] Ok
>> * OK [URLMECH INTERNAL] Ok
>> A OK [READ-WRITE] Completed
>> B SEARCH TEXT "Test"
>> * SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
>> 23 24 25 26 27 28 29 30 31 33 34 35 36 37 38 39
>> ....
>> 4292 4294 4295 4296 4298 4299 4300 4301 4303 4306 4307 4309 4310
>> 4315 4316 4317 4318 4321
>> B OK Completed (1996 msgs in 0.690 secs)
>>
>>
>> === 3.0.8 === SEARCH TEXT
>>
>> a SELECT INBOX
>> * 4321 EXISTS
>> * 0 RECENT
>> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk
>> $Forwarded $mdnsent $label1 $label2 $label3 hordetest testflag)
>> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen
>> NonJunk $Forwarded $mdnsent $label1 $label2 $label3 hordetest
>> testflag \*)] Ok
>> * OK [UNSEEN 3737] Ok
>> * OK [UIDVALIDITY 1238498084] Ok
>> * OK [UIDNEXT 93373] Ok
>> * OK [HIGHESTMODSEQ 98491] Ok
>> * OK [URLMECH INTERNAL] Ok
>> * OK [ANNOTATIONS 65536] Ok
>> a OK [READ-WRITE] Completed
>> B SEARCH TEXT "Test"
>> * SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
>> 23 24 25 26 27 28 29 30 31 33 34 35 36 37 38 39
>> ....
>> 4274 4275 4276 4277 4278 4279 4285 4286 4287 4288 4292 4294 4295
>> 4296 4298 4299 4300 4301 4303 4306 4307 4309 4310 4315 4316 4317
>> 4318 4321
>> B OK Completed (1935 msgs in 2.580 secs)
>>
>> ==== 2.4.20 === SEARCH HEADER
>>
>> a SELECT INBOX
>> * 4321 EXISTS
>> * 0 RECENT
>> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
>> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
>> * OK [UNSEEN 1] Ok
>> * OK [UIDVALIDITY 1540372444] Ok
>> * OK [UIDNEXT 93369] Ok
>> * OK [HIGHESTMODSEQ 2] Ok
>> * OK [URLMECH INTERNAL] Ok
>> a OK [READ-WRITE] Completed
>> b SEARCH HEADER X-comment Unirundmail
>> * SEARCH 4283 4291 4313 4319 4320
>> b OK Completed (5 msgs in 0.020 secs)
>>
>> ==== 3.0.8 === SEARCH HEADER
>>
>> a SELECT INBOX
>> * 4321 EXISTS
>> * 0 RECENT
>> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen NonJunk
>> $Forwarded $mdnsent $label1 $label2 $label3 hordetest testflag)
>> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen
>> NonJunk $Forwarded $mdnsent $label1 $label2 $label3 hordetest
>> testflag \*)] Ok
>> * OK [UNSEEN 3737] Ok
>> * OK [UIDVALIDITY 1238498084] Ok
>> * OK [UIDNEXT 93373] Ok
>> * OK [HIGHESTMODSEQ 98491] Ok
>> * OK [URLMECH INTERNAL] Ok
>> * OK [ANNOTATIONS 65536] Ok
>> a OK [READ-WRITE] Completed
>> b SEARCH HEADER X-comment Unirundmail
>> * SEARCH 4283 4291 4313 4319 4320
>> b OK Completed (5 msgs in 0.370 secs)
>>
>> ===
>>
>> There is also a big discrepancy between time indicated in the "OK
>> Completed" and the time from
>> sending the search command till the return of the result, which is
>> 0.890 sec compared to ~30 sec
>> on the production system.
>>
>> I used strace on the imapd processes while searching to verify that
>> the squat file was used
>> in 2.4 but not in 3.0.
>> I could see open events for the squat file and the messages that
>> where found for 2.4.20
>> and no open event (not even a failed one) to the squat file but
>> instead open events for
>> all message files in that folder for 3.0.8
>>
>> I read the documentation and source code and tried to figure out if
>> i am missing some
>> config options, or if i could pinpoint a function where the search
>> was turning the
>> wrong way. I used "perf -g" the see the call graphs and to figure
>> out where the
>> call graphs change
>>
>> I can see that the same functions are called up to "index_search",
>> and that the called functions
>> change at that point. I know that the search code got restructured
>> to support different search
>> engines and that some functions got renamed. I have attached the
>> perf report output, so that
>> someone with a better understanding of the code can see whats going
>> on. I can provide the
>> perf.data files if it helps.
>>
>> Can someone confirm or refute that the squatter search engine is
>> working with cyrus imapd 3.0.x?
>>
>> Is "search_engine: squat" in imapd.conf and a "squatter" event in
>> cyrus.conf is sufficient to
>> use the squatter search index in 3.0 or are there other options
>> steps required.
>>
>> Regards
>>
>> ?? Michael Menge
>>
>> PS. link to my original post with my imapd.conf
>> https://lists.andrew.cmu.edu/pipermail/info-cyrus/2018-September/040395.html
>>
>>


--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universit?t T?bingen                   Fax.: (49) 7071/29-5912
Zentrum f?r Datenverarbeitung          mail:  
michael.menge at zdv.uni-tuebingen.de
W?chterstra?e 76
72074 T?bingen


From rsto at fastmailteam.com  Fri May 10 09:11:59 2019
From: rsto at fastmailteam.com (Robert Stepanek)
Date: Fri, 10 May 2019 09:11:59 -0400
Subject: squatter not used after upgrade to cyrus 3.0.8
In-Reply-To: <20190510141812.Horde.Esvmh3nCs6jL7BoAnCE6bLN@webmail.uni-tuebingen.de>
References: <20180917140152.Horde.W8VK46keZMD6i3ABlUUhwzX@webmail.uni-tuebingen.de>
 <20180917123123.GA24774@io.chezmoi.fr>
 <20180917171814.Horde.d0cTBJKuZdVAj2dx5sWRBef@webmail.uni-tuebingen.de>
 <20181025150940.Horde.ZSBGZc-bdo4Ww0nHKDroxYH@webmail.uni-tuebingen.de>
 <c40a88fb-87e9-537a-f9a3-7f38fc5bb8c5@cc.upv.es>
 <20190510141812.Horde.Esvmh3nCs6jL7BoAnCE6bLN@webmail.uni-tuebingen.de>
Message-ID: <90d5edef-1175-4805-9604-896bbc6670e6@www.fastmail.com>

On Fri, May 10, 2019, at 2:18 PM, Michael Menge wrote:
> no the problem is not completly resolved jet.
> See https://github.com/cyrusimap/cyrus-imapd/issues/2598 for details.

I tested this right now on the 3.0 branch, and I can confirm there's an issue of matches not being found. I don't have an answer why this is so, yet. I'm working on it.

> > I've put in my impad.conf:
> > search_engine: squat
> > search_fuzzy_always: 1
> 
> AFAIK the squat search engine does not support fuzzy search.
> I am sure in my testing this setting didn't resolve the slow
> search, but i don't remember if this setting did nothing,
> or failed to find anything at all or what else happened
> with this configuration.

If I recall correctly, Cyrus uses the configured search engine only for fuzzy search. Non-fuzzy search is matched using the Cyrus-builtin routines, which will be slow: for body search it has to examine every message for that mailbox.

If your client is sending something like:

 C: 6 search body "body"

then it won't use the squat/xapian index, unless you have search_fuzzy_always set.

Cheers,
Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190510/1ec10a03/attachment.html>

From michael.menge at zdv.uni-tuebingen.de  Fri May 10 09:56:34 2019
From: michael.menge at zdv.uni-tuebingen.de (Michael Menge)
Date: Fri, 10 May 2019 15:56:34 +0200
Subject: squatter not used after upgrade to cyrus 3.0.8
In-Reply-To: <90d5edef-1175-4805-9604-896bbc6670e6@www.fastmail.com>
References: <20180917140152.Horde.W8VK46keZMD6i3ABlUUhwzX@webmail.uni-tuebingen.de>
 <20180917123123.GA24774@io.chezmoi.fr>
 <20180917171814.Horde.d0cTBJKuZdVAj2dx5sWRBef@webmail.uni-tuebingen.de>
 <20181025150940.Horde.ZSBGZc-bdo4Ww0nHKDroxYH@webmail.uni-tuebingen.de>
 <c40a88fb-87e9-537a-f9a3-7f38fc5bb8c5@cc.upv.es>
 <20190510141812.Horde.Esvmh3nCs6jL7BoAnCE6bLN@webmail.uni-tuebingen.de>
 <90d5edef-1175-4805-9604-896bbc6670e6@www.fastmail.com>
Message-ID: <20190510155634.Horde.2Uc6cpd6Z1kTiXt7nRR7lC6@webmail.uni-tuebingen.de>


Quoting Robert Stepanek <rsto at fastmailteam.com>:

> On Fri, May 10, 2019, at 2:18 PM, Michael Menge wrote:
>> no the problem is not completly resolved jet.
>> See for details.
>
> I tested this right now on the 3.0 branch, and I can confirm there's  
> an issue of matches not being found. I don't have an answer why this  
> is so, yet. I'm working on it.
>
>> > I've put in my impad.conf:
>> > search_engine: squat
>> > search_fuzzy_always: 1
>>
>> AFAIK the squat search engine does not support fuzzy search.
>> I am sure in my testing this setting didn't resolve the slow
>> search, but i don't remember if this setting did nothing,
>> or failed to find anything at all or what else happened
>> with this configuration.
>
> If I recall correctly, Cyrus uses the configured search engine only  
> for fuzzy search. Non-fuzzy search is matched using the  
> Cyrus-builtin routines, which will be slow: for body search it has  
> to examine every message for that mailbox.
>
> If your client is sending something like:
>
>  C: 6 search body "body"
>
> then it won't use the squat/xapian index, unless you have  
> search_fuzzy_always set.
>

Thanks Robert for looking into it.

I think there is some confusion here. I didn't test "search body",
not sure if this was supported by cyrus 2.4 and if yes if it made
use of the index.
What I did test was "search text" and "search header".
Both used the index in 2.4 and didn't use it in 3.0
in my initial configuration.

I was able to make cyrus use the squat index file again
for header search by enabling conversation db, though I had to
disable conversation db again on our production servers because
of other problems.

I can test again if search_fuzzy_always has an influence on
the usage of the usage of the "search text".


--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universit?t T?bingen                   Fax.: (49) 7071/29-5912
Zentrum f?r Datenverarbeitung          mail:  
michael.menge at zdv.uni-tuebingen.de
W?chterstra?e 76
72074 T?bingen


From rsto at fastmailteam.com  Fri May 10 10:08:35 2019
From: rsto at fastmailteam.com (Robert Stepanek)
Date: Fri, 10 May 2019 10:08:35 -0400
Subject: squatter not used after upgrade to cyrus 3.0.8
In-Reply-To: <20190510155634.Horde.2Uc6cpd6Z1kTiXt7nRR7lC6@webmail.uni-tuebingen.de>
References: <20180917140152.Horde.W8VK46keZMD6i3ABlUUhwzX@webmail.uni-tuebingen.de>
 <20180917123123.GA24774@io.chezmoi.fr>
 <20180917171814.Horde.d0cTBJKuZdVAj2dx5sWRBef@webmail.uni-tuebingen.de>
 <20181025150940.Horde.ZSBGZc-bdo4Ww0nHKDroxYH@webmail.uni-tuebingen.de>
 <c40a88fb-87e9-537a-f9a3-7f38fc5bb8c5@cc.upv.es>
 <20190510141812.Horde.Esvmh3nCs6jL7BoAnCE6bLN@webmail.uni-tuebingen.de>
 <90d5edef-1175-4805-9604-896bbc6670e6@www.fastmail.com>
 <20190510155634.Horde.2Uc6cpd6Z1kTiXt7nRR7lC6@webmail.uni-tuebingen.de>
Message-ID: <0286e021-1219-4183-82e6-ed54634e46d6@www.fastmail.com>

On Fri, May 10, 2019, at 3:58 PM, Michael Menge wrote:
> I can test again if search_fuzzy_always has an influence on
> the usage of the usage of the "search text".

Yes, please do.

Cheers,
Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190510/4e83f92f/attachment-0001.html>

From byrnejb at harte-lyne.ca  Wed May 15 13:11:26 2019
From: byrnejb at harte-lyne.ca (James B. Byrne)
Date: Wed, 15 May 2019 13:11:26 -0400
Subject: Purging old email files from Cyrus-IMAPD v.3.0.9 on FreeBSD-12.0
Message-ID: <d65d3b9cfb5cb5f74af35db698a2dee9.squirrel@webmail.harte-lyne.ca>

We converted from v.2.0.? to v.3.0.? last year.  We accepted the
default settings for delete/expunge.  Now we have suspicions that
deleted and expunged email is not actually being deleted from the file
system and the space recovered.

Looking at the man page for cyr_expire I see this:

       The expiration of messages is controlled by the
       /vendor/cmu/cyrus-imapd/expire mailbox annotation
       which specifies the age (in days) of messages in
       the given mailbox that should be deleted. A value
       of 0 means that no expiration is to be performed
       on that mailbox.

       The value of the /vendor/cmu/cyrus-imapd/expire
       annotation is inherited by all children of the mailbox
       on which it is set, so an entire mailbox tree can be
       configured by setting a single annotation on the root of
       that tree.  If a mailbox does not have a
       /vendor/cmu/cyrus-imapd/expire annotation set on it
       (or does not inherit one), then no messages are expired
       from the mailbox.

       The annotation can be examined using the info command
       of cyradm(8), and modified using the mboxconfig and setinfo
       commands of cyradm(8).


The following is a representative user mailbox on our server:


 info user/usrname
{user/usrname}:
  private:
    check: NIL
    checkperiod: NIL
    comment: NIL
    sort: NIL
    specialuse: NIL
    thread: NIL
    expire: NIL
    news2mail: NIL
    sieve: NIL
    squat: NIL
  shared:
    check: NIL
    checkperiod: NIL
    comment: NIL
    sort: NIL
    specialuse: NIL
    thread: NIL
    annotsize: 0
    duplicatedeliver: false
    expire: NIL
    lastpop: NIL
    lastupdate: 15-May-2019 04:16:08 -0400
    news2mail: NIL
    partition: default
    pop3newuidl: true
    pop3showafter: NIL
    sharedseen: false
    sieve: NIL
    size: 2105288
    squat: NIL
    synccrcs: 1325519995 0
    uniqueid: 6bcd81f457596d11


Our cyrus.conf file has this:

  delprune      cmd="cyr_expire -D 180d -E 3d -X 180d" at=0400

My question is:  When are message files actually removed from our
server's the file system and the space recovered?  Is there a setting
or utility option required to perform this action / accomplish this
result?  How does one verify the removal?

-- 
***          e-Mail is NOT a SECURE channel          ***
        Do NOT transmit sensitive data via e-Mail
 Do NOT open attachments nor follow links sent by e-Mail

James B. Byrne                mailto:ByrneJB at Harte-Lyne.ca
Harte & Lyne Limited          http://www.harte-lyne.ca
9 Brockley Drive              vox: +1 905 561 1241
Hamilton, Ontario             fax: +1 905 561 0757
Canada  L8E 3C3


From sylvain at a7e.app  Thu May 16 05:49:55 2019
From: sylvain at a7e.app (Sylvain)
Date: Thu, 16 May 2019 11:49:55 +0200
Subject: Vacation sieve scripts not working
Message-ID: <CAOOBcionzphsytsowgLTC_nFTRxW76Q3FEsnZoZ3X=qvAogARA@mail.gmail.com>

Hi list,

I'm testing Cyrus 3 on the future debian 10 that will be soon released.

Vacation sieve scripts seem not to run. Other scripts do.

For example, this one will never send vacation messages to sender :
require ["vacation"];
vacation :days 1 :subject "OUTOFTHEOFFICE" "I AM OUT OF THE OFFICE";

But this one will reject the mail :
require ["reject","fileinto"];
if address :is :all "From" "foo at example.org"
{
    reject "testing";
}

Of course I double checked that the vacation script is activated, and I
change the from address at each test. I use sieveshell to put them.

Anyone can reproduce this ?
Any idea to debug this situation?

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190516/6703a308/attachment.html>

From karagian at gmail.com  Thu May 16 11:00:53 2019
From: karagian at gmail.com (Savvas Karagiannidis)
Date: Thu, 16 May 2019 18:00:53 +0300
Subject: Purging old email files from Cyrus-IMAPD v.3.0.9 on FreeBSD-12.0
In-Reply-To: <d65d3b9cfb5cb5f74af35db698a2dee9.squirrel@webmail.harte-lyne.ca>
References: <d65d3b9cfb5cb5f74af35db698a2dee9.squirrel@webmail.harte-lyne.ca>
Message-ID: <CADfvMFNmkqJ2jGj85cNLRBS_g7UnOCjjR994HZU132Dk6eamyg@mail.gmail.com>

Hi James,
the command that performs the actual removal of the files from the file
system is cyr_expire
<https://www.cyrusimap.org/imap/reference/manpages/systemcommands/cyr_expire.html>
According to your cyrus.conf and the manual of cyr_expire, the operation is
performed daily at 04:00. The command is executed by the main cyrus
process, so you don't have to do anything else manually...
The parameters -D 180d and -X 180d specify that only mailboxes and messages
that are at least 180 days old will be deleted.

When cyr_expire is executed you should see a line in your log file like
these:

May 11 04:03:18 mail cyr_expire[2573]: Expired 0 and expunged 7033 out of
3098366 messages from 5035 mailboxes
May 12 04:03:14 mail cyr_expire[19325]: Expired 0 and expunged 44241 out of
3092823 messages from 5035 mailboxes
May 13 04:04:23 mail cyr_expire[27698]: Expired 0 and expunged 77735 out of
3050025 messages from 5035 mailboxes

This states how many messages and mailboxes were permanently deleted from
the filesystem.

If you only performed the upgrade recently and delayed delete was not
active until then, you will only notice cyr_expire actually deleting
messages and mailboxes 180 days after the upgrade, since there will be no
messages that old until then...
You should adjust "180d" to your needs and requirements. -D refers to
deleted mailboxes and -X to deleted messages.

Regards,
Savvas Karagiannidis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190516/48e8d483/attachment.html>

From Willem at Offermans.Rompen.nl  Thu May 16 10:29:12 2019
From: Willem at Offermans.Rompen.nl (Willem Offermans)
Date: Thu, 16 May 2019 16:29:12 +0200
Subject: Vacation sieve scripts not working
In-Reply-To: <CAOOBcionzphsytsowgLTC_nFTRxW76Q3FEsnZoZ3X=qvAogARA@mail.gmail.com>
References: <CAOOBcionzphsytsowgLTC_nFTRxW76Q3FEsnZoZ3X=qvAogARA@mail.gmail.com>
Message-ID: <C8D8BA5D-301A-4A64-AB5E-C74F8B436F54@Offermans.Rompen.nl>

Dear Cyrus-imap friends and Sylvain,

Without further info, I cannot tell you what is wrong or why your script is not working.

Is there a way to debug sieve?

I can only confirm that the following is working in my case:

vacation :days 5 :addresses [?myname at example.com", ?othername at example2.com"] :subject ?I'm out of office"

Maybe you need to specify addresses?, though it is not logical.


Wiel Offermans
Willem at Offermans.Rompen.nl




> On 16 May 2019, at 11:49, Sylvain <sylvain at a7e.app> wrote:
> 
> Hi list,
> 
> I'm testing Cyrus 3 on the future debian 10 that will be soon released.
> 
> Vacation sieve scripts seem not to run. Other scripts do.
> 
> For example, this one will never send vacation messages to sender :
> require ["vacation"];
> vacation :days 1 :subject "OUTOFTHEOFFICE" "I AM OUT OF THE OFFICE";
> 
> But this one will reject the mail :
> require ["reject","fileinto"];
> if address :is :all "From" "foo at example.org <mailto:foo at example.org>"
> {
>     reject "testing";
> }
> 
> Of course I double checked that the vacation script is activated, and I change the from address at each test. I use sieveshell to put them.
> 
> Anyone can reproduce this ?
> Any idea to debug this situation?
> 
> Thanks
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190516/89b4ead0/attachment-0001.html>

From nd at syndicat.com  Thu May 16 11:30:49 2019
From: nd at syndicat.com (Niels Dettenbach)
Date: Thu, 16 May 2019 17:30:49 +0200
Subject: Vacation sieve scripts not working
In-Reply-To: <CAOOBcionzphsytsowgLTC_nFTRxW76Q3FEsnZoZ3X=qvAogARA@mail.gmail.com>
References: <CAOOBcionzphsytsowgLTC_nFTRxW76Q3FEsnZoZ3X=qvAogARA@mail.gmail.com>
Message-ID: <31818508.eIsakQmRjS@gongo>

Am Donnerstag, 16. Mai 2019, 11:49:55 CEST schrieb Sylvain:
> For example, this one will never send vacation messages to sender :
> require ["vacation"];
> vacation :days 1 :subject "OUTOFTHEOFFICE" "I AM OUT OF THE OFFICE";

...if i remember correctly, Cyrus expects (at least in some setups) recipient 
addresses within SIEVE vacation scripts to "react". Could you try to set this 
for test?


best regards,


niels.


-- 
 ---
 Niels Dettenbach
 Syndicat IT & Internet
 http://www.syndicat.com
 PGP: https://syndicat.com/pub_key.asc
 ---
 







From sylvain at a7e.app  Thu May 16 12:05:39 2019
From: sylvain at a7e.app (Sylvain)
Date: Thu, 16 May 2019 18:05:39 +0200
Subject: Vacation sieve scripts not working
In-Reply-To: <31818508.eIsakQmRjS@gongo>
References: <CAOOBcionzphsytsowgLTC_nFTRxW76Q3FEsnZoZ3X=qvAogARA@mail.gmail.com>
 <31818508.eIsakQmRjS@gongo>
Message-ID: <CAOOBcirvS_PFpfM1S=wx-jZgSubNWC5SkFp=VvXp1K-CM+LFzg@mail.gmail.com>

Le jeu. 16 mai 2019 ? 17:36, Niels Dettenbach via Info-cyrus <
info-cyrus at lists.andrew.cmu.edu> a ?crit :

> ...if i remember correctly, Cyrus expects (at least in some setups)
> recipient
> addresses within SIEVE vacation scripts to "react". Could you try to set
> this
> for test?
>

You are right : the addresses parameter is mandatory.

Thanks !
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190516/fee70124/attachment.html>

From byrnejb at harte-lyne.ca  Thu May 16 16:08:32 2019
From: byrnejb at harte-lyne.ca (James B. Byrne)
Date: Thu, 16 May 2019 16:08:32 -0400
Subject: Purging old email files from Cyrus-IMAPD v.3.0.9 on FreeBSD-12.0
In-Reply-To: <CADfvMFNmkqJ2jGj85cNLRBS_g7UnOCjjR994HZU132Dk6eamyg@mail.gmail.com>
References: <d65d3b9cfb5cb5f74af35db698a2dee9.squirrel@webmail.harte-lyne.ca>
 <CADfvMFNmkqJ2jGj85cNLRBS_g7UnOCjjR994HZU132Dk6eamyg@mail.gmail.com>
Message-ID: <186304de1ee0ec339af6bc2345b68f75.squirrel@webmail.harte-lyne.ca>



On Thu, May 16, 2019 11:00, Savvas Karagiannidis wrote:
> Hi James,
> the command that performs the actual removal of the files from the
> file system is cyr_expire
> <https://www.cyrusimap.org/imap/reference/manpages/systemcommands/cyr_expire.html>
> According to your cyrus.conf and the manual of cyr_expire, the
> operation is performed daily at 04:00. The command is executed
> by the main cyrus process, so you don't have to do anything
> else manually...
> The parameters -D 180d and -X 180d specify that only mailboxes and
> messages that are at least 180 days old will be deleted.
>
> When cyr_expire is executed you should see a line in your log file
> like these:
>

Thanks.  I have a question though. If expunge == purge then why does
the documentation distinguish between them?

When is What ... Deleted, Expired, Expunged or Purged?
https://www.cyrusimap.org/imap/reference/faqs/o-deleted-expired-expunged-purged.html

Expunged

    The message (which has been flagged as \Deleted) is also expunged,
meaning that the user can in no way retrieve the message
autonomously.

Purged

    The message?s index record may still exist (until they are
expired), but the message file is removed from the filesystem, or
in the context of folders, the mail folder is removed from the
filesystem.


This is what has me confused.

Regards,

-- 
***          e-Mail is NOT a SECURE channel          ***
        Do NOT transmit sensitive data via e-Mail
 Do NOT open attachments nor follow links sent by e-Mail

James B. Byrne                mailto:ByrneJB at Harte-Lyne.ca
Harte & Lyne Limited          http://www.harte-lyne.ca
9 Brockley Drive              vox: +1 905 561 1241
Hamilton, Ontario             fax: +1 905 561 0757
Canada  L8E 3C3


From karagian at gmail.com  Thu May 16 17:14:31 2019
From: karagian at gmail.com (Savvas Karagiannidis)
Date: Fri, 17 May 2019 00:14:31 +0300
Subject: Purging old email files from Cyrus-IMAPD v.3.0.9 on FreeBSD-12.0
In-Reply-To: <186304de1ee0ec339af6bc2345b68f75.squirrel@webmail.harte-lyne.ca>
References: <d65d3b9cfb5cb5f74af35db698a2dee9.squirrel@webmail.harte-lyne.ca>
 <CADfvMFNmkqJ2jGj85cNLRBS_g7UnOCjjR994HZU132Dk6eamyg@mail.gmail.com>
 <186304de1ee0ec339af6bc2345b68f75.squirrel@webmail.harte-lyne.ca>
Message-ID: <CADfvMFMUM_=A3doNbG2LuqFWtPMrAVOkU--GbvcvM4be43jw_Q@mail.gmail.com>

Hi James,
I think deleted and expunged are explained well in the documentation.

Deleted: In IMAP when a user deletes an email, the email is actually only
flagged as deleted. The user can still see the message and even unset the
flag recovering the email.

Expunged: When the user issues the expunge command the flagged as deleted
messages above become inaccessible by the user. Without delayed delete this
is also where the message file gets deleted from the filesystem (purged).

The confusion is probably between purged and expired. For every email
message cyrus keeps a file on the mailbox folder and also an entry in the
cyrus index file  for that message. For a clean delete, both the file and
the index entry must be deleted. For some reasons they may not be both
removed at the same time... I think for performance reasons, replication
etc. So:

Purged: when the message file is deleted, it is considered purged. The
actual email can in no way be recovered.

Expired: The index entry for the message is removed. This usually happens
after it has been purged. Cyrus has no longer any knowledge of the message
ever existed.

Hope this clarifies things. For simplicity just consider cyr_expire purging
and expiring the messages.

Regards,
Savvas Karagiannidis



On Thu, May 16, 2019, 23:08 James B. Byrne <byrnejb at harte-lyne.ca> wrote:

>
>
> On Thu, May 16, 2019 11:00, Savvas Karagiannidis wrote:
> > Hi James,
> > the command that performs the actual removal of the files from the
> > file system is cyr_expire
> > <
> https://www.cyrusimap.org/imap/reference/manpages/systemcommands/cyr_expire.html
> >
> > According to your cyrus.conf and the manual of cyr_expire, the
> > operation is performed daily at 04:00. The command is executed
> > by the main cyrus process, so you don't have to do anything
> > else manually...
> > The parameters -D 180d and -X 180d specify that only mailboxes and
> > messages that are at least 180 days old will be deleted.
> >
> > When cyr_expire is executed you should see a line in your log file
> > like these:
> >
>
> Thanks.  I have a question though. If expunge == purge then why does
> the documentation distinguish between them?
>
> When is What ... Deleted, Expired, Expunged or Purged?
>
> https://www.cyrusimap.org/imap/reference/faqs/o-deleted-expired-expunged-purged.html
>
> Expunged
>
>     The message (which has been flagged as \Deleted) is also expunged,
> meaning that the user can in no way retrieve the message
> autonomously.
>
> Purged
>
>     The message?s index record may still exist (until they are
> expired), but the message file is removed from the filesystem, or
> in the context of folders, the mail folder is removed from the
> filesystem.
>
>
> This is what has me confused.
>
> Regards,
>
> --
> ***          e-Mail is NOT a SECURE channel          ***
>         Do NOT transmit sensitive data via e-Mail
>  Do NOT open attachments nor follow links sent by e-Mail
>
> James B. Byrne                mailto:ByrneJB at Harte-Lyne.ca
> Harte & Lyne Limited          http://www.harte-lyne.ca
> 9 Brockley Drive              vox: +1 905 561 1241
> Hamilton, Ontario             fax: +1 905 561 0757
> Canada  L8E 3C3
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190517/22f8be71/attachment.html>

From egoitz at sarenet.es  Thu May 23 13:18:40 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Thu, 23 May 2019 19:18:40 +0200
Subject: Questions about user mode sync_client (move mailboxes from one server
 to another) and mailbox moving from partition (moving from partition
 in same server, renaming) lock
Message-ID: <088b0fa7e32bf44abfa0796defac3910@sarenet.es>

Good afternoon, 

Our Cyrus machines (Cyrus 3.0.8), usually have 3 mailbox partitions.
Sometimes, one of them becomes highly filled so we usually perform a
mailbox rename to another partition of the same server. For that
purpose, we normally lock at our proxy barrier any access to the mailbox
(we do play with Nginx authentication, Postfix hold and so....). Is it
really needed to lock that way the mailbox, at some "external to Cyrus
level," in order to avoid mailbox corruption?. Or does Cyrus handle that
properly?. Does Cyrus exclusively lock and after done, unlock again?.
Have been taking a look at mboxlist_renamemailbox() and seemed so. Have
noticed too, that it seems that partition rename operation from and to
the same server but different parition at least, is not being inserted
in the rolling mode lock.. is this a new security measure for avoiding
accidents with the rename?. Always I have done a mailbox rename
previously (Cyrus 2.3.X), have stopped the master/slave replication,
done the rename in the master and later if all ended fine... launched in
the slave a dm of the "in the master renamed mailbox" and a sync_client
-u from the master for the mailbox to be copied to the appropiate
partition in the slave. 

My other question is.. with the new replication method (imap based and
so...), can I do a user mode sync_client from a mailbox, to another
server acting as a master?. I mean, in the following scenario :  

Server A (master) => Server B (slave) 

Server C (master) => Server D (slave) 

The aaa at bbb.net mailbox is in A server. I want to move the mailbox from
A=>B couple of master/slave server to C=>D couple of mater/slave. I
launch a "sync_client -v -u aaa at bbb.net -S C -p partition3" in server A.
Server C, has sync_log_chain enabled. Would that mailbox be replicated
in C=>D couple (to both from A to C and from C to D) and been able to be
accesible in C?. If so, does any kind of drawback exist in having always
sync_log_chain enabled?... else for this kind of movement seems to be
useful.. 

But thinking about it... if C is master... is it really needed that
sync_log_chain config statement in that case or it would just be
necessary (as I think), for replicating in the following scenario only?.


Server 1 (master) -> Server 2 (slave) -> Server 3 (slave) 

So, not needed when (there's a master in the middle) : 

Server 1 (master) -> Server 2 (master) -> Server 3 (slave)       perhaps
as in
https://www.cyrusimap.org/imap/reference/admin/sop/replication.html can
be read?. 

Thank you so much for your time, 

Best regards,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190523/8215ff0c/attachment.html>

From Hagedorn at uni-koeln.de  Thu May 23 13:28:20 2019
From: Hagedorn at uni-koeln.de (Sebastian Hagedorn)
Date: Thu, 23 May 2019 19:28:20 +0200
Subject: Questions about user mode sync_client (move mailboxes from one
 server to another) and mailbox moving from partition (moving from partition
 in same server, renaming) lock
In-Reply-To: <088b0fa7e32bf44abfa0796defac3910@sarenet.es>
References: <088b0fa7e32bf44abfa0796defac3910@sarenet.es>
Message-ID: <7F8ED72F3ED505266C7EFAE3@iMac-2019.local>

Hi,

> Our Cyrus machines (Cyrus 3.0.8), usually have 3 mailbox partitions.
> Sometimes, one of them becomes highly filled so we usually perform a
> mailbox rename to another partition of the same server. For that
> purpose, we normally lock at our proxy barrier any access to the mailbox
> (we do play with Nginx authentication, Postfix hold and so....). Is it
> really needed to lock that way the mailbox, at some "external to Cyrus
> level," in order to avoid mailbox corruption?. Or does Cyrus handle that
> properly?. Does Cyrus exclusively lock and after done, unlock again?.

I can only answer that part of the question. We have been doing it like 
that (without blocking access from the outside) for years, but we're still 
on Cyrus 2.4. We only make sure there are no active processes by the user 
before starting the RENAME, and we do it at night. There haven't been any 
problems with that approach.
--
Sebastian Hagedorn - Weyertal 121, Zimmer 2.02
Regionales Rechenzentrum (RRZK)
Universit?t zu K?ln / Cologne University - Tel. +49-221-470-89578

From egoitz at sarenet.es  Fri May 24 03:18:56 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Fri, 24 May 2019 09:18:56 +0200
Subject: Questions about user mode sync_client (move mailboxes from one
 server to another) and mailbox moving from partition (moving from partition in
 same server, renaming) lock
In-Reply-To: <7F8ED72F3ED505266C7EFAE3@iMac-2019.local>
References: <088b0fa7e32bf44abfa0796defac3910@sarenet.es>
 <7F8ED72F3ED505266C7EFAE3@iMac-2019.local>
Message-ID: <eae24a63d79d400c47bea59109110077@sarenet.es>

Hi Sebastian, 

Thanks a lot for your comments. But that, anyway, won't assure no
mailbox access would exist in the middle of the rename... 

I think there should not be problems due that the function
mboxlist_renamemailbox() does a mailbox_open_iwl() which finally checks
if the mailbox is locked and then locks it. Anyway, if none of the gurus
of Cyrus sais it... I would read more deeply the code (for ensuring) and
will do some checks in testing env.... 

Thanks a lot Sebastian :) 

Cheers 

El 2019-05-23 19:28, Sebastian Hagedorn escribi?:

> Hi,
> 
>> Our Cyrus machines (Cyrus 3.0.8), usually have 3 mailbox partitions.
>> Sometimes, one of them becomes highly filled so we usually perform a
>> mailbox rename to another partition of the same server. For that
>> purpose, we normally lock at our proxy barrier any access to the mailbox
>> (we do play with Nginx authentication, Postfix hold and so....). Is it
>> really needed to lock that way the mailbox, at some "external to Cyrus
>> level," in order to avoid mailbox corruption?. Or does Cyrus handle that
>> properly?. Does Cyrus exclusively lock and after done, unlock again?.
> 
> I can only answer that part of the question. We have been doing it like that (without blocking access from the outside) for years, but we're still on Cyrus 2.4. We only make sure there are no active processes by the user before starting the RENAME, and we do it at night. There haven't been any problems with that approach.
> --
> Sebastian Hagedorn - Weyertal 121, Zimmer 2.02
> Regionales Rechenzentrum (RRZK)
> Universit?t zu K?ln / Cologne University - Tel. +49-221-470-89578
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190524/f0100015/attachment.html>

From ellie at fastmail.com  Sun May 26 22:26:59 2019
From: ellie at fastmail.com (ellie timoney)
Date: Mon, 27 May 2019 12:26:59 +1000
Subject: Cyrus IMAP 2.5.13  released
Message-ID: <2fe3f69a-d85a-4e0c-bd07-f8e47d60452d@www.fastmail.com>

The Cyrus team is proud to announce the immediate availability of a new version of Cyrus IMAP: 2.5.13

This release contains a fix for CVE-2019-11356, a buffer overrun vulnerability in the httpd daemon.  If you compile cyrus with HTTP support enabled and your cyrus.conf contains SERVICES entries that run the httpd daemon, you will need this upgrade.

I'm trialling hosting the release files using Github's releases feature.  Please use the Github download links if possible, and advise if you have any problems!  (It may even download faster due to Github's content delivery network.)

Download URLs:

    https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.13/cyrus-imapd-2.5.13.tar.gz
    https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.13/cyrus-imapd-2.5.13.tar.gz.sig

    https://www.cyrusimap.org/releases/cyrus-imapd-2.5.13.tar.gz
    https://www.cyrusimap.org/releases/cyrus-imapd-2.5.13.tar.gz.sig

Please consult the release notes before upgrading to 2.5.13:

    https://www.cyrusimap.org/imap/download/release-notes/2.5/x/2.5.13.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report issues, join in the deliberations of new features for the next Cyrus IMAP release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

From ellie at fastmail.com  Sun May 26 22:27:55 2019
From: ellie at fastmail.com (ellie timoney)
Date: Mon, 27 May 2019 12:27:55 +1000
Subject: Cyrus IMAP 3.0.10  released
Message-ID: <6a10e204-bddf-4d09-911c-dc7d19f2e9dc@www.fastmail.com>

The Cyrus team is proud to announce the immediate availability of a new version of Cyrus IMAP: 3.0.10

This release contains a fix for CVE-2019-11356, a buffer overrun vulnerability in the httpd daemon.  If you compile cyrus with HTTP support enabled and your cyrus.conf contains SERVICES entries that run the httpd daemon, you will need this upgrade.

I'm trialling hosting the release files using Github's releases feature.  Please use the Github download links if possible, and advise if you have any problems!  (It may even download faster due to Github's content delivery network.)

Download URLs:

    https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.10/cyrus-imapd-3.0.10.tar.gz
    https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.10/cyrus-imapd-3.0.10.tar.gz.sig

    https://www.cyrusimap.org/releases/cyrus-imapd-3.0.10.tar.gz
    https://www.cyrusimap.org/releases/cyrus-imapd-3.0.10.tar.gz.sig

Please consult the release notes and upgrade documentation before upgrading to 3.0.10:

    https://www.cyrusimap.org/imap/download/release-notes/3.0/x/3.0.10.html
    https://www.cyrusimap.org/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report issues, join in the deliberations of new features for the next Cyrus IMAP release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

From karagian at gmail.com  Mon May 27 09:10:21 2019
From: karagian at gmail.com (Savvas Karagiannidis)
Date: Mon, 27 May 2019 16:10:21 +0300
Subject: Cyrus IMAP 3.0.10 released
In-Reply-To: <6a10e204-bddf-4d09-911c-dc7d19f2e9dc@www.fastmail.com>
References: <6a10e204-bddf-4d09-911c-dc7d19f2e9dc@www.fastmail.com>
Message-ID: <CADfvMFNg0=QsX2t9D_yeCwfXSiayCGY8sRdGK8D6=N+AV3y=nw@mail.gmail.com>

Hi Ellie,
is there any link for details about CVE-2019-11356?
Tried googling it but couldn't find anything related or in cyrus mailing
lists or in reported issues, other than this:
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-11356

Regards,
Savvas Karagiannidis

On Mon, May 27, 2019 at 5:28 AM ellie timoney <ellie at fastmail.com> wrote:

> The Cyrus team is proud to announce the immediate availability of a new
> version of Cyrus IMAP: 3.0.10
>
> This release contains a fix for CVE-2019-11356, a buffer overrun
> vulnerability in the httpd daemon.  If you compile cyrus with HTTP support
> enabled and your cyrus.conf contains SERVICES entries that run the httpd
> daemon, you will need this upgrade.
>
> I'm trialling hosting the release files using Github's releases feature.
> Please use the Github download links if possible, and advise if you have
> any problems!  (It may even download faster due to Github's content
> delivery network.)
>
> Download URLs:
>
>
> https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.10/cyrus-imapd-3.0.10.tar.gz
>
> https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.10/cyrus-imapd-3.0.10.tar.gz.sig
>
>     https://www.cyrusimap.org/releases/cyrus-imapd-3.0.10.tar.gz
>     https://www.cyrusimap.org/releases/cyrus-imapd-3.0.10.tar.gz.sig
>
> Please consult the release notes and upgrade documentation before
> upgrading to 3.0.10:
>
>
> https://www.cyrusimap.org/imap/download/release-notes/3.0/x/3.0.10.html
>     https://www.cyrusimap.org/imap/download/upgrade.html
>
> And join us on Github at https://github.com/cyrusimap/cyrus-imapd to
> report issues, join in the deliberations of new features for the next Cyrus
> IMAP release, and to contribute to the documentation.
>
> On behalf of the Cyrus team,
>
> Kind regards,
>
> ellie timoney
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190527/b68a19f8/attachment.html>

From ellie at fastmail.com  Tue May 28 20:12:08 2019
From: ellie at fastmail.com (ellie timoney)
Date: Wed, 29 May 2019 10:12:08 +1000
Subject: Cyrus IMAP 3.0.10 released
In-Reply-To: <CADfvMFNg0=QsX2t9D_yeCwfXSiayCGY8sRdGK8D6=N+AV3y=nw@mail.gmail.com>
References: <6a10e204-bddf-4d09-911c-dc7d19f2e9dc@www.fastmail.com>
 <CADfvMFNg0=QsX2t9D_yeCwfXSiayCGY8sRdGK8D6=N+AV3y=nw@mail.gmail.com>
Message-ID: <1f78ca45-48bb-47fd-bcaf-87e14d94f36b@www.fastmail.com>

Yep, that's the correct link, we're waiting on the CVE folks to update the listing out of reserved status

On Mon, May 27, 2019, at 11:10 PM, Savvas Karagiannidis wrote:
> Hi Ellie,
> is there any link for details about CVE-2019-11356?
> Tried googling it but couldn't find anything related or in cyrus mailing lists or in reported issues, other than this:
> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-11356
> 
> Regards,
> Savvas Karagiannidis
> 
> On Mon, May 27, 2019 at 5:28 AM ellie timoney <ellie at fastmail.com> wrote:
>> The Cyrus team is proud to announce the immediate availability of a new version of Cyrus IMAP: 3.0.10
>> 
>>  This release contains a fix for CVE-2019-11356, a buffer overrun vulnerability in the httpd daemon. If you compile cyrus with HTTP support enabled and your cyrus.conf contains SERVICES entries that run the httpd daemon, you will need this upgrade.
>> 
>>  I'm trialling hosting the release files using Github's releases feature. Please use the Github download links if possible, and advise if you have any problems! (It may even download faster due to Github's content delivery network.)
>> 
>>  Download URLs:
>> 
>> https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.10/cyrus-imapd-3.0.10.tar.gz
>> https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.10/cyrus-imapd-3.0.10.tar.gz.sig
>> 
>> https://www.cyrusimap.org/releases/cyrus-imapd-3.0.10.tar.gz
>> https://www.cyrusimap.org/releases/cyrus-imapd-3.0.10.tar.gz.sig
>> 
>>  Please consult the release notes and upgrade documentation before upgrading to 3.0.10:
>> 
>> https://www.cyrusimap.org/imap/download/release-notes/3.0/x/3.0.10.html
>> https://www.cyrusimap.org/imap/download/upgrade.html
>> 
>>  And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report issues, join in the deliberations of new features for the next Cyrus IMAP release, and to contribute to the documentation.
>> 
>>  On behalf of the Cyrus team,
>> 
>>  Kind regards,
>> 
>>  ellie timoney
>>  ----
>>  Cyrus Home Page: http://www.cyrusimap.org/
>>  List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>  To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190529/dd6ed943/attachment.html>