From ml at netfence.it  Fri Feb  1 11:59:21 2019
From: ml at netfence.it (Andrea Venturoli)
Date: Fri, 1 Feb 2019 17:59:21 +0100
Subject: Sieve script not working
In-Reply-To: <c270e06b-2ccc-9134-8b91-7701dfa26b54@andreaventuroli.it>
References: <f212e3ad-bf37-6da5-ca21-3f20d2fcf9c4@andreaventuroli.it>
 <3980dfd9-d397-7f3e-5781-e58185c2746f@onlight.com>
 <c270e06b-2ccc-9134-8b91-7701dfa26b54@andreaventuroli.it>
Message-ID: <9f0dfe24-1d79-2ce8-7847-f78a80b369ea@netfence.it>

On 1/20/19 3:21 PM, Andrea Venturoli wrote:
> On 1/20/19 12:13 AM, Nic Bernstein wrote:
>> Andrea,
>> It would help to know which version of Cyrus you're having difficulty 
>> with, and what setting you're using for 'unixhierarchysep'?
> 
> Sorry, I obviously should have said it from the start: I'm using version 
> 2.5.12 on FreeBSD 11.2/amd64.
> I'm not using 'unixhierarchysep', so the "default" separator shoud be a 
> dot.
> I'm also not using "altnamespace".

So, is there no way of debugging a sieve script or log its execution?

  Thanks
	av.

From brong at fastmailteam.com  Sat Feb  2 08:34:54 2019
From: brong at fastmailteam.com (Bron Gondwana)
Date: Sat, 02 Feb 2019 08:34:54 -0500
Subject: another problem with conversations db
In-Reply-To: <20190131093559.Horde.Vu9F7oPMtLNy30qgYXzWgKL@webmail.uni-tuebingen.de>
References: <20190124180232.Horde.Tya11Hrse_nHFrp7Zj39dP4@webmail.uni-tuebingen.de>
 <363a2149-1b36-4dc0-aca5-b121c8ae3266@beta.fastmail.com>
 <20190125095538.Horde.BZLLa6BdReb8ZHa6b9Vxg1r@webmail.uni-tuebingen.de>
 <5d089935-a59f-45d7-a50c-cec40e23082c@beta.fastmail.com>
 <20190131093559.Horde.Vu9F7oPMtLNy30qgYXzWgKL@webmail.uni-tuebingen.de>
Message-ID: <c02770dd-4238-4aab-a034-bd447b58cd06@www.fastmail.com>

Hi Michael,

Sorry about the delay in looking at this - I was mad crazy busy getting ready to go overseas. At Fosdem now, about to give a talk about JMAP!

OK, let's start with the things that give me a little bit of hives...

configdirectory: /srv/cyrus-be
partition-default: /srv/cyrus-be
partition-ssd: /srv/cyrus-be/ssd-part

Ouch. There's a couple of things I wouldn't do there - having the partition be the same as the config directory, and having a separate partition be a subdirectory of a partition. They're both asking for trouble. I would probably lay my system out like:

configdirectory: /srv/cyrus-be/conf
partition-default: /srv/cyrus-be/default-part
partition-ssd: /srv/cyrus-be/ssd-part

And then each tree would only have one type of thing in it.

Anyway, I don't think that would break anything.

metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part
metapartition_files: header index cache expunge squat annotations lock dav archivecache

Ooh, I haven't tested having cache and archivecache on the same location. That's really interesting. Again, I'd be in favour of separation here, give them different paths. That might be tricky with ssd though, the way this is laid out. I assume you have some kind of symlink farm going on?

Otherwise it all looks OK. Are you getting other IOERRORs in your log files which could show things aborting? It really looks like your conversations DB is getting out of sync due to other failures.

Cheers,

Bron.



On Thu, Jan 31, 2019, at 19:36, Michael Menge wrote:
> 
> Quoting Bron Gondwana <brong at fastmail.fm>:
> 
> > On Fri, Jan 25, 2019, at 20:00, Michael Menge wrote:
> >> Hi Bron
> >>
> >>
> >> Quoting Bron Gondwana <brong at fastmail.fm>:
> >>
> >> > Sorry I haven't been following along with this earlier. Can you post
> >> > your imapd.conf and cyrus.conf as well as let her know if you run
> >> > anything which directly messes with files on disk.
> >>
> >> Thanks for looking into it. I have attached the backend and replica
> >> configs. There should be nothing messing with the files on disk
> >> directly. Some times we need to restore mails from file based
> >> backup (bacula) followed by a reconstruct but this was not the
> >> case this time.
> >
> > Do you always rebuild the conversations db after doing the 
> > reconstruct? That will be necessary now. We switched to doing all 
> > our restores using IMAP append a while back so we're never fiddling 
> > the file system under Cyrus.
> >
> >> >
> >> > Also, what operating system is this on and what Cyrus version?
> >> >
> >>
> >> We are running a cyrus 3.0.8 compiled with the following Options
> >> (./configure --enable-murder --enable-http --enable-calalarmd
> >> --enable-replication --enable-backup --enable-idled
> >> --enable-autocreate CFLAGS="-fPIC -g")
> >> in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs.
> >
> > They should be fine. I'll have a read of the config when I'm at a 
> > real computer.
> >
> 
> did you find anything in the config that would explain the problems?
> 
> >>
> >> > Bron.
> >> >
> >> > On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote:
> >> >> Hi,
> >> >>
> >> >> I have discovered an other problem with the conversations db:
> >> >>
> >> >> Thousends of lines with "IOERROR: conversations_audit on load:" and
> >> >> "IOERROR: conversations_audit on store:"
> >> >> A look at the source code shows that these errors are logged after
> >> >> "_sanity_check_counts" is called.
> >> >> The log level LOG_ERR and the prefix IOERROR indicate that I have a
> >> >> serious problem. Do I?
> >> >>
> >> >> This problem occurred for accounts where the rebuild of the
> >> >> conversations db was successful.
> >> >>
> >> >> I don't want to rebuild the conversations db every few days.
> >> >>
> >> >> Any help is appreciated.
> >> >>
> >> >> Kind regards
> >> >>
> >> >>
> >> >> Michael Menge
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> 
> >> --------------------------------------------------------------------------------
> >> >> M.Menge Tel.: (49) 7071/29-70316
> >> >> Universit?t T?bingen Fax.: (49) 7071/29-5912
> >> >> Zentrum f?r Datenverarbeitung mail:
> >> >> michael.menge at zdv.uni-tuebingen.de
> >> >> W?chterstra?e 76
> >> >> 72074 T?bingen
> >> >>
> >> >> ----
> >> >> Cyrus Home Page: http://www.cyrusimap.org/
> >> >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> >> >> To Unsubscribe:
> >> >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> >> >
> >> > --
> >> > Bron Gondwana
> >> > brong at fastmail.fm
> >>
> >>
> >>
> >> --------------------------------------------------------------------------------
> >> M.Menge Tel.: (49) 7071/29-70316
> >> Universit?t T?bingen Fax.: (49) 7071/29-5912
> >> Zentrum f?r Datenverarbeitung mail:
> >> michael.menge at zdv.uni-tuebingen.de
> >> W?chterstra?e 76
> >> 72074 T?bingen
> >>
> >> ----
> >> Cyrus Home Page: http://www.cyrusimap.org/
> >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> >> To Unsubscribe:
> >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> >>
> >> *Attachments:*
> >> * imapd_be.conf
> >> * cyrus_be.conf
> >> * cyrus_re.conf
> >> * imapd_re.conf
> >
> > --
> > Bron Gondwana
> > brong at fastmail.fm
> 
> 
> 
> --------------------------------------------------------------------------------
> M.Menge Tel.: (49) 7071/29-70316
> Universit?t T?bingen Fax.: (49) 7071/29-5912
> Zentrum f?r Datenverarbeitung mail: 
> michael.menge at zdv.uni-tuebingen.de
> W?chterstra?e 76
> 72074 T?bingen
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

--
 Bron Gondwana, CEO, FastMail Pty Ltd
 brong at fastmailteam.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190202/3bc55856/attachment.html>

From brong at fastmailteam.com  Sat Feb  2 14:25:59 2019
From: brong at fastmailteam.com (Bron Gondwana)
Date: Sat, 02 Feb 2019 14:25:59 -0500
Subject: Sieve script not working
In-Reply-To: <9f0dfe24-1d79-2ce8-7847-f78a80b369ea@netfence.it>
References: <f212e3ad-bf37-6da5-ca21-3f20d2fcf9c4@andreaventuroli.it>
 <3980dfd9-d397-7f3e-5781-e58185c2746f@onlight.com>
 <c270e06b-2ccc-9134-8b91-7701dfa26b54@andreaventuroli.it>
 <9f0dfe24-1d79-2ce8-7847-f78a80b369ea@netfence.it>
Message-ID: <67349212-1882-4ba5-a8d6-e06d569f37a0@beta.fastmail.com>

What is being written to syslog by your lmtpd? Sieve failures should be logged to syslog.

There's also a sieve-test binary that you can build in the Cyrus source code. I don't think it's built by default on most platforms though, we build it specially at FM. It takes a script and a raw email and spits out a list of actions.

Cheers,

Bron.

On Fri, Feb 1, 2019, at 17:59, Andrea Venturoli wrote:
> On 1/20/19 3:21 PM, Andrea Venturoli wrote:
> > On 1/20/19 12:13 AM, Nic Bernstein wrote:
> >> Andrea,
> >> It would help to know which version of Cyrus you're having difficulty 
> >> with, and what setting you're using for 'unixhierarchysep'?
> > 
> > Sorry, I obviously should have said it from the start: I'm using version 
> > 2.5.12 on FreeBSD 11.2/amd64.
> > I'm not using 'unixhierarchysep', so the "default" separator shoud be a 
> > dot.
> > I'm also not using "altnamespace".
> 
> So, is there no way of debugging a sieve script or log its execution?
> 
>  Thanks
> av.
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 

--
 Bron Gondwana, CEO, FastMail Pty Ltd
 brong at fastmailteam.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190202/86968c33/attachment.html>

From daniel-listas at gmx.net  Sat Feb  2 20:05:46 2019
From: daniel-listas at gmx.net (Daniel Bareiro)
Date: Sat, 2 Feb 2019 22:05:46 -0300
Subject: Fatal error: Exists wrong 513 512 483 17057
Message-ID: <df524faf-eefc-82c6-4605-34793bfe07a4@gmx.net>

Hi!

Several days ago I am trying to solve this issue but without success.
When opening a folder inside a mailbox through Thunderbird, I get this
message: "Server [servername] has disconnected. The server may have gone
down or there may be a network problem".

In that access attempt the log shows an entry like this:

----------
Feb  2 21:55:38 mail cyrus/imaps[6510]: Fatal error: Exists wrong 513
512 483 17057
----------

Maybe the index is corrupt and has some duplicate message?

Investigating on the web I found this reference [1]:

----------
/* make sure we don't overflow the memory we mapped */
	if (msgno >= state->mapsize) {
	    char buf[2048];
	    sprintf(buf, "Exists wrong %u %u %u %u", msgno,
		    state->mapsize, mailbox->i.exists, mailbox->i.num_records);
	    fatal(buf, EC_IOERR);
	}
----------

I would like to know what you think. Is it possible that this is a
problem with the index for this mailbox? If so, do you recommend some
way to regenerate it?


Thanks in advance for your reply.

Kind regards,
Daniel

[1]
https://forge.bluemind.net/stash/projects/BM/repos/bm-cyrus-imapd/browse/imap/index.c?at=c196236d937608270283c681ae27f1bb00265b62&raw

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190202/7da1ed97/attachment.sig>

From brong at fastmail.fm  Sun Feb  3 06:33:59 2019
From: brong at fastmail.fm (Bron Gondwana)
Date: Sun, 03 Feb 2019 06:33:59 -0500
Subject: Fatal error: Exists wrong 513 512 483 17057
In-Reply-To: <df524faf-eefc-82c6-4605-34793bfe07a4@gmx.net>
References: <df524faf-eefc-82c6-4605-34793bfe07a4@gmx.net>
Message-ID: <c106bf37-0c79-4793-a812-0fe3e717d819@beta.fastmail.com>

Wow. A reconstruct should fix that. What version are you running?

On Sun, Feb 3, 2019, at 02:11, Daniel Bareiro wrote:
> Hi!
> 
> Several days ago I am trying to solve this issue but without success.
> When opening a folder inside a mailbox through Thunderbird, I get this
> message: "Server [servername] has disconnected. The server may have gone
> down or there may be a network problem".
> 
> In that access attempt the log shows an entry like this:
> 
> ----------
> Feb 2 21:55:38 mail cyrus/imaps[6510]: Fatal error: Exists wrong 513
> 512 483 17057
> ----------
> 
> Maybe the index is corrupt and has some duplicate message?
> 
> Investigating on the web I found this reference [1]:
> 
> ----------
> /* make sure we don't overflow the memory we mapped */
> if (msgno >= state->mapsize) {
>  char buf[2048];
>  sprintf(buf, "Exists wrong %u %u %u %u", msgno,
>  state->mapsize, mailbox->i.exists, mailbox->i.num_records);
>  fatal(buf, EC_IOERR);
> }
> ----------
> 
> I would like to know what you think. Is it possible that this is a
> problem with the index for this mailbox? If so, do you recommend some
> way to regenerate it?
> 
> 
> Thanks in advance for your reply.
> 
> Kind regards,
> Daniel
> 
> [1]
> https://forge.bluemind.net/stash/projects/BM/repos/bm-cyrus-imapd/browse/imap/index.c?at=c196236d937608270283c681ae27f1bb00265b62&raw
> 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> *Attachments:*
>  * signature.asc

-- 
 Bron Gondwana
 brong at fastmail.fm
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190203/b703c57d/attachment.html>

From daniel-listas at gmx.net  Sun Feb  3 16:43:34 2019
From: daniel-listas at gmx.net (Daniel Bareiro)
Date: Sun, 3 Feb 2019 18:43:34 -0300
Subject: Fatal error: Exists wrong 513 512 483 17057
In-Reply-To: <c106bf37-0c79-4793-a812-0fe3e717d819@beta.fastmail.com>
References: <df524faf-eefc-82c6-4605-34793bfe07a4@gmx.net>
 <c106bf37-0c79-4793-a812-0fe3e717d819@beta.fastmail.com>
Message-ID: <7ed47a3a-ff4d-e91d-01c2-1141bd85e432@gmx.net>

Hi, Bron. Thanks for your reply.

On 3/2/19 08:33, Bron Gondwana wrote:

> Wow. A reconstruct should fix that. What version are you running?

I tried using 'reconstruct' as you suggested and this was the result:

----------
# /usr/lib/cyrus/bin/reconstruct -r user.admin
user.admin
user.admin.Backups
user.admin.Drafts
user.admin.Sent
user.admin.TareasCron: record corrupted 13 (maybe uid 9836)
user.admin.TareasCron: Mailbox format corruption detected Failed to
reconstruct mailbox
user.admin.Trash
----------

Just 'TasksCron' is the folder that I can not open. What do you suggest
to do in this case?

I'm using Cyrus 2.5.10-3 on Debian Stretch. A few days ago I upgraded
the entire operating system with Debian Jessie and Cyrus 2.4.17.

Kind regards,
Daniel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190203/bb7a0241/attachment.sig>

From brong at fastmail.fm  Mon Feb  4 03:45:43 2019
From: brong at fastmail.fm (Bron Gondwana)
Date: Mon, 04 Feb 2019 03:45:43 -0500
Subject: Fatal error: Exists wrong 513 512 483 17057
In-Reply-To: <7ed47a3a-ff4d-e91d-01c2-1141bd85e432@gmx.net>
References: <df524faf-eefc-82c6-4605-34793bfe07a4@gmx.net>
 <c106bf37-0c79-4793-a812-0fe3e717d819@beta.fastmail.com>
 <7ed47a3a-ff4d-e91d-01c2-1141bd85e432@gmx.net>
Message-ID: <d39918e4-b350-49dd-8c89-8d81422e9d7a@www.fastmail.com>

Oh how frustrating...

On Mon, Feb 4, 2019, at 08:48, Daniel Bareiro wrote:
> Hi, Bron. Thanks for your reply.
> 
> On 3/2/19 08:33, Bron Gondwana wrote:
> 
> > Wow. A reconstruct should fix that. What version are you running?
> 
> I tried using 'reconstruct' as you suggested and this was the result:
> 
> ----------
> # /usr/lib/cyrus/bin/reconstruct -r user.admin
> user.admin
> user.admin.Backups
> user.admin.Drafts
> user.admin.Sent
> user.admin.TareasCron: record corrupted 13 (maybe uid 9836)
> user.admin.TareasCron: Mailbox format corruption detected Failed to
> reconstruct mailbox

This clearly isn't syslogging enough! Can you check the permissions on the underlying filesystem! Also, try running reconstruct with -G, though it shouldn't make a difference.

> user.admin.Trash
> ----------
> 
> Just 'TasksCron' is the folder that I can not open. What do you suggest
> to do in this case?

I'd take a backup, and then delete the cyrus.index file in that folder and run reconstruct again... That's a pretty extreme approach, but it should work.

> I'm using Cyrus 2.5.10-3 on Debian Stretch. A few days ago I upgraded
> the entire operating system with Debian Jessie and Cyrus 2.4.17.

That upgrade should be fine, and 2.5.10 should be fine as well. I just poked around in the code there to see how the reconstruct might be failing, and there's not enough information to know what's happening.

Bron.

-- 
 Bron Gondwana
 brong at fastmail.fm

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190204/c783499f/attachment.html>

From michael.menge at zdv.uni-tuebingen.de  Mon Feb  4 04:17:12 2019
From: michael.menge at zdv.uni-tuebingen.de (Michael Menge)
Date: Mon, 04 Feb 2019 10:17:12 +0100
Subject: another problem with conversations db
In-Reply-To: <c02770dd-4238-4aab-a034-bd447b58cd06@www.fastmail.com>
References: <20190124180232.Horde.Tya11Hrse_nHFrp7Zj39dP4@webmail.uni-tuebingen.de>
 <363a2149-1b36-4dc0-aca5-b121c8ae3266@beta.fastmail.com>
 <20190125095538.Horde.BZLLa6BdReb8ZHa6b9Vxg1r@webmail.uni-tuebingen.de>
 <5d089935-a59f-45d7-a50c-cec40e23082c@beta.fastmail.com>
 <20190131093559.Horde.Vu9F7oPMtLNy30qgYXzWgKL@webmail.uni-tuebingen.de>
 <c02770dd-4238-4aab-a034-bd447b58cd06@www.fastmail.com>
Message-ID: <20190204101712.Horde.33ManGeIHA8D5TVzSiQ7825@webmail.uni-tuebingen.de>

Hi,

Quoting Bron Gondwana <brong at fastmailteam.com>:

> Hi Michael,
>
> Sorry about the delay in looking at this - I was mad crazy busy  
> getting ready to go overseas. At Fosdem now, about to give a talk  
> about JMAP!
>
> OK, let's start with the things that give me a little bit of hives...
>
> configdirectory: /srv/cyrus-be
> partition-default: /srv/cyrus-be
> partition-ssd: /srv/cyrus-be/ssd-part
>
> Ouch. There's a couple of things I wouldn't do there - having the  
> partition be the same as the config directory, and having a separate  
> partition be a subdirectory of a partition. They're both asking for  
> trouble. I would probably lay my system out like:
>
> configdirectory: /srv/cyrus-be/conf
> partition-default: /srv/cyrus-be/default-part
> partition-ssd: /srv/cyrus-be/ssd-part
>

partition-default isn't used any more. To use the metapartition we moved
all accounts form the default partition to the ssd partition which is the
the new defaultpartition ("defaultpartition: ssd")

> And then each tree would only have one type of thing in it.
>
> Anyway, I don't think that would break anything.
>
> metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part
> metapartition_files: header index cache expunge squat annotations  
> lock dav archivecache
>
> Ooh, I haven't tested having cache and archivecache on the same  
> location. That's really interesting. Again, I'd be in favour of  
> separation here, give them different paths. That might be tricky  
> with ssd though, the way this is laid out. I assume you have some  
> kind of symlink farm going on?
>

I didn't know that there could be a problem with cache and archivecache.
At the time we decided on the configuration for cyrus 3.0 I looked at the
imapd.conf man page and for metapartition_files decided that I want all
meta files on the ssd storage. There was no indication in the man page
that there could be a problem.

How do I separate location of archivecache from the other metapartition path?
And fix the cache and archivecache files?

No there is no sysmlink farm.  We have mounted different iSCSI volumes to
/srv/cyrus-ssd-be, /srv/cyrus-hdd-be and /srv/cyrus-be


> Otherwise it all looks OK. Are you getting other IOERRORs in your  
> log files which could show things aborting? It really looks like  
> your conversations DB is getting out of sync due to other failures.

I found a few instances of 3 related errors.

Feb  4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening  
/srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory
Feb  4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening  
/srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory
Feb  4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR archive  
user.XXXX 2185 failed to copyfile  
(/srv/cyrus-be/ssd-part/L/user/XXXX/2185. =>  
/srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.): Unknown code  
____ 255

The file was already at /srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.

stat 2185.
   File: '2185.'
   Size: 424542    	Blocks: 832        IO Block: 4096   regular file
Device: 860h/2144d	Inode: 4677438462  Links: 1
Access: (0600/-rw-------)  Uid: (   96/   cyrus)   Gid: (   12/    mail)
Context: system_u:object_r:unlabeled_t:s0
Access: 2018-11-27 01:12:46.585864239 +0100
Modify: 2018-11-27 01:12:46.585864239 +0100
Change: 2018-11-27 01:12:46.585864239 +0100
  Birth: -


>
> Cheers,
>
> Bron.
>
>
>
> On Thu, Jan 31, 2019, at 19:36, Michael Menge wrote:
>>
>> Quoting Bron Gondwana <brong at fastmail.fm>:
>>
>> > On Fri, Jan 25, 2019, at 20:00, Michael Menge wrote:
>> >> Hi Bron
>> >>
>> >>
>> >> Quoting Bron Gondwana <brong at fastmail.fm>:
>> >>
>> >> > Sorry I haven't been following along with this earlier. Can you post
>> >> > your imapd.conf and cyrus.conf as well as let her know if you run
>> >> > anything which directly messes with files on disk.
>> >>
>> >> Thanks for looking into it. I have attached the backend and replica
>> >> configs. There should be nothing messing with the files on disk
>> >> directly. Some times we need to restore mails from file based
>> >> backup (bacula) followed by a reconstruct but this was not the
>> >> case this time.
>> >
>> > Do you always rebuild the conversations db after doing the
>> > reconstruct? That will be necessary now. We switched to doing all
>> > our restores using IMAP append a while back so we're never fiddling
>> > the file system under Cyrus.
>> >
>> >> >
>> >> > Also, what operating system is this on and what Cyrus version?
>> >> >
>> >>
>> >> We are running a cyrus 3.0.8 compiled with the following Options
>> >> (./configure --enable-murder --enable-http --enable-calalarmd
>> >> --enable-replication --enable-backup --enable-idled
>> >> --enable-autocreate CFLAGS="-fPIC -g")
>> >> in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs.
>> >
>> > They should be fine. I'll have a read of the config when I'm at a
>> > real computer.
>> >
>>
>> did you find anything in the config that would explain the problems?
>>
>> >>
>> >> > Bron.
>> >> >
>> >> > On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote:
>> >> >> Hi,
>> >> >>
>> >> >> I have discovered an other problem with the conversations db:
>> >> >>
>> >> >> Thousends of lines with "IOERROR: conversations_audit on load:" and
>> >> >> "IOERROR: conversations_audit on store:"
>> >> >> A look at the source code shows that these errors are logged after
>> >> >> "_sanity_check_counts" is called.
>> >> >> The log level LOG_ERR and the prefix IOERROR indicate that I have a
>> >> >> serious problem. Do I?
>> >> >>
>> >> >> This problem occurred for accounts where the rebuild of the
>> >> >> conversations db was successful.
>> >> >>
>> >> >> I don't want to rebuild the conversations db every few days.
>> >> >>
>> >> >> Any help is appreciated.
>> >> >>
>> >> >> Kind regards
>> >> >>
>> >> >>
>> >> >> Michael Menge
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >>  
>> --------------------------------------------------------------------------------
>> >> >> M.Menge Tel.: (49) 7071/29-70316
>> >> >> Universit?t T?bingen Fax.: (49) 7071/29-5912
>> >> >> Zentrum f?r Datenverarbeitung mail:
>> >> >> michael.menge at zdv.uni-tuebingen.de
>> >> >> W?chterstra?e 76
>> >> >> 72074 T?bingen
>> >> >>
>> >> >> ----
>> >> >> Cyrus Home Page: http://www.cyrusimap.org/
>> >> >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> >> >> To Unsubscribe:
>> >> >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>> >> >
>> >> > --
>> >> > Bron Gondwana
>> >> > brong at fastmail.fm
>> >>
>> >>
>> >>
>> >>  
>> --------------------------------------------------------------------------------
>> >> M.Menge Tel.: (49) 7071/29-70316
>> >> Universit?t T?bingen Fax.: (49) 7071/29-5912
>> >> Zentrum f?r Datenverarbeitung mail:
>> >> michael.menge at zdv.uni-tuebingen.de
>> >> W?chterstra?e 76
>> >> 72074 T?bingen
>> >>
>> >> ----
>> >> Cyrus Home Page: http://www.cyrusimap.org/
>> >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> >> To Unsubscribe:
>> >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>> >>
>> >> *Attachments:*
>> >> * imapd_be.conf
>> >> * cyrus_be.conf
>> >> * cyrus_re.conf
>> >> * imapd_re.conf
>> >
>> > --
>> > Bron Gondwana
>> > brong at fastmail.fm
>>
>>
>>
>> --------------------------------------------------------------------------------
>> M.Menge Tel.: (49) 7071/29-70316
>> Universit?t T?bingen Fax.: (49) 7071/29-5912
>> Zentrum f?r Datenverarbeitung mail:
>> michael.menge at zdv.uni-tuebingen.de
>> W?chterstra?e 76
>> 72074 T?bingen
>>
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>
> --
>  Bron Gondwana, CEO, FastMail Pty Ltd
>  brong at fastmailteam.com



--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universit?t T?bingen                   Fax.: (49) 7071/29-5912
Zentrum f?r Datenverarbeitung          mail:  
michael.menge at zdv.uni-tuebingen.de
W?chterstra?e 76
72074 T?bingen


From brong at fastmail.fm  Mon Feb  4 05:12:46 2019
From: brong at fastmail.fm (Bron Gondwana)
Date: Mon, 04 Feb 2019 05:12:46 -0500
Subject: another problem with conversations db
In-Reply-To: <20190204101712.Horde.33ManGeIHA8D5TVzSiQ7825@webmail.uni-tuebingen.de>
References: <20190124180232.Horde.Tya11Hrse_nHFrp7Zj39dP4@webmail.uni-tuebingen.de>
 <363a2149-1b36-4dc0-aca5-b121c8ae3266@beta.fastmail.com>
 <20190125095538.Horde.BZLLa6BdReb8ZHa6b9Vxg1r@webmail.uni-tuebingen.de>
 <5d089935-a59f-45d7-a50c-cec40e23082c@beta.fastmail.com>
 <20190131093559.Horde.Vu9F7oPMtLNy30qgYXzWgKL@webmail.uni-tuebingen.de>
 <c02770dd-4238-4aab-a034-bd447b58cd06@www.fastmail.com>
 <20190204101712.Horde.33ManGeIHA8D5TVzSiQ7825@webmail.uni-tuebingen.de>
Message-ID: <02bb719c-27ce-4e62-84e8-f11a32adb3dc@www.fastmail.com>

On Mon, Feb 4, 2019, at 20:21, Michael Menge wrote:
> Hi,
> 
> Quoting Bron Gondwana <brong at fastmailteam.com>:
> 
> > Hi Michael,
> >
> > Sorry about the delay in looking at this - I was mad crazy busy 
> > getting ready to go overseas. At Fosdem now, about to give a talk 
> > about JMAP!
> >
> > OK, let's start with the things that give me a little bit of hives...
> >
> > configdirectory: /srv/cyrus-be
> > partition-default: /srv/cyrus-be
> > partition-ssd: /srv/cyrus-be/ssd-part
> >
> > Ouch. There's a couple of things I wouldn't do there - having the 
> > partition be the same as the config directory, and having a separate 
> > partition be a subdirectory of a partition. They're both asking for 
> > trouble. I would probably lay my system out like:
> >
> > configdirectory: /srv/cyrus-be/conf
> > partition-default: /srv/cyrus-be/default-part
> > partition-ssd: /srv/cyrus-be/ssd-part
> >
> 
> partition-default isn't used any more. To use the metapartition we moved
> all accounts form the default partition to the ssd partition which is the
> the new defaultpartition ("defaultpartition: ssd")

Right - that makes sense.

> > And then each tree would only have one type of thing in it.
> >
> > Anyway, I don't think that would break anything.
> >
> > metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part
> > metapartition_files: header index cache expunge squat annotations 
> > lock dav archivecache
> >
> > Ooh, I haven't tested having cache and archivecache on the same 
> > location. That's really interesting. Again, I'd be in favour of 
> > separation here, give them different paths. That might be tricky 
> > with ssd though, the way this is laid out. I assume you have some 
> > kind of symlink farm going on?
> >
> 
> I didn't know that there could be a problem with cache and archivecache.
> At the time we decided on the configuration for cyrus 3.0 I looked at the
> imapd.conf man page and for metapartition_files decided that I want all
> meta files on the ssd storage. There was no indication in the man page
> that there could be a problem.

Fair. I'd have to test that to see if it works correctly. I would hope so, but I haven't tested that configuration. This is the downside with having lots of different ways to do things!

> How do I separate location of archivecache from the other metapartition path?
> And fix the cache and archivecache files?

This I don't know a good answer for. I will test if having the same path for cache and archivecache could fail. I THINK that I made the code safe for it, but I'm not sure that it's been tested.

> No there is no sysmlink farm. We have mounted different iSCSI volumes to
> /srv/cyrus-ssd-be, /srv/cyrus-hdd-be and /srv/cyrus-be

Right. That makes sense.

> > Otherwise it all looks OK. Are you getting other IOERRORs in your 
> > log files which could show things aborting? It really looks like 
> > your conversations DB is getting out of sync due to other failures.
> 
> I found a few instances of 3 related errors.
> 
> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening 
> /srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory
> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening 
> /srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory
> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR archive 
> user.XXXX 2185 failed to copyfile 
> (/srv/cyrus-be/ssd-part/L/user/XXXX/2185. => 
> /srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.): Unknown code 
> ____ 255


Ouch. Yeah, that could have been caused by a bug in delivery, and would definitely cause conversations DB corruption if the index file was updated but the conversations DB wasn't or vice versa.

> The file was already at /srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.

Right! I do wonder if there are some bugs in 3.0.x which are fixed on master around delivery to archive partition. We definitely had bugs on master, but I thought they were newly introduced on master as well, which is why the fixes weren't backported. But if you're having files be in the wrong location, maybe there are bugs on 3.0.x as well.

Do you have the syslog lines at the time that email was delivered?

Bron.
-- 
 Bron Gondwana
 brong at fastmail.fm

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190204/e70bba44/attachment-0001.html>

From michael.menge at zdv.uni-tuebingen.de  Mon Feb  4 06:00:13 2019
From: michael.menge at zdv.uni-tuebingen.de (Michael Menge)
Date: Mon, 04 Feb 2019 12:00:13 +0100
Subject: another problem with conversations db
In-Reply-To: <02bb719c-27ce-4e62-84e8-f11a32adb3dc@www.fastmail.com>
References: <20190124180232.Horde.Tya11Hrse_nHFrp7Zj39dP4@webmail.uni-tuebingen.de>
 <363a2149-1b36-4dc0-aca5-b121c8ae3266@beta.fastmail.com>
 <20190125095538.Horde.BZLLa6BdReb8ZHa6b9Vxg1r@webmail.uni-tuebingen.de>
 <5d089935-a59f-45d7-a50c-cec40e23082c@beta.fastmail.com>
 <20190131093559.Horde.Vu9F7oPMtLNy30qgYXzWgKL@webmail.uni-tuebingen.de>
 <c02770dd-4238-4aab-a034-bd447b58cd06@www.fastmail.com>
 <20190204101712.Horde.33ManGeIHA8D5TVzSiQ7825@webmail.uni-tuebingen.de>
 <02bb719c-27ce-4e62-84e8-f11a32adb3dc@www.fastmail.com>
Message-ID: <20190204120013.Horde.6MsNbwrJRdIBADBt9I1z3QP@webmail.uni-tuebingen.de>


Quoting Bron Gondwana <brong at fastmail.fm>:

> On Mon, Feb 4, 2019, at 20:21, Michael Menge wrote:
>> Hi,
>>
>> Quoting Bron Gondwana <brong at fastmailteam.com>:
>>
>> > Hi Michael,
>> >
>> > Sorry about the delay in looking at this - I was mad crazy busy
>> > getting ready to go overseas. At Fosdem now, about to give a talk
>> > about JMAP!
>> >
>> > OK, let's start with the things that give me a little bit of hives...
>> >
>> > configdirectory: /srv/cyrus-be
>> > partition-default: /srv/cyrus-be
>> > partition-ssd: /srv/cyrus-be/ssd-part
>> >
>> > Ouch. There's a couple of things I wouldn't do there - having the
>> > partition be the same as the config directory, and having a separate
>> > partition be a subdirectory of a partition. They're both asking for
>> > trouble. I would probably lay my system out like:
>> >
>> > configdirectory: /srv/cyrus-be/conf
>> > partition-default: /srv/cyrus-be/default-part
>> > partition-ssd: /srv/cyrus-be/ssd-part
>> >
>>
>> partition-default isn't used any more. To use the metapartition we moved
>> all accounts form the default partition to the ssd partition which is the
>> the new defaultpartition ("defaultpartition: ssd")
>
> Right - that makes sense.
>
>> > And then each tree would only have one type of thing in it.
>> >
>> > Anyway, I don't think that would break anything.
>> >
>> > metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part
>> > metapartition_files: header index cache expunge squat annotations
>> > lock dav archivecache
>> >
>> > Ooh, I haven't tested having cache and archivecache on the same
>> > location. That's really interesting. Again, I'd be in favour of
>> > separation here, give them different paths. That might be tricky
>> > with ssd though, the way this is laid out. I assume you have some
>> > kind of symlink farm going on?
>> >
>>
>> I didn't know that there could be a problem with cache and archivecache.
>> At the time we decided on the configuration for cyrus 3.0 I looked at the
>> imapd.conf man page and for metapartition_files decided that I want all
>> meta files on the ssd storage. There was no indication in the man page
>> that there could be a problem.
>
> Fair. I'd have to test that to see if it works correctly. I would  
> hope so, but I haven't tested that configuration. This is the  
> downside with having lots of different ways to do things!
>
>> How do I separate location of archivecache from the other  
>> metapartition path?
>> And fix the cache and archivecache files?
>
> This I don't know a good answer for. I will test if having the same  
> path for cache and archivecache could fail. I THINK that I made the  
> code safe for it, but I'm not sure that it's been tested.
>
>> No there is no sysmlink farm. We have mounted different iSCSI volumes to
>> /srv/cyrus-ssd-be, /srv/cyrus-hdd-be and /srv/cyrus-be
>
> Right. That makes sense.
>
>> > Otherwise it all looks OK. Are you getting other IOERRORs in your
>> > log files which could show things aborting? It really looks like
>> > your conversations DB is getting out of sync due to other failures.
>>
>> I found a few instances of 3 related errors.
>>
>> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening
>> /srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory
>> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening
>> /srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory
>> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR archive
>> user.XXXX 2185 failed to copyfile
>> (/srv/cyrus-be/ssd-part/L/user/XXXX/2185. =>
>> /srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.): Unknown code
>> ____ 255
>
>
> Ouch. Yeah, that could have been caused by a bug in delivery, and  
> would definitely cause conversations DB corruption if the index file  
> was updated but the conversations DB wasn't or vice versa.
>
>> The file was already at /srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.
>
> Right! I do wonder if there are some bugs in 3.0.x which are fixed  
> on master around delivery to archive partition. We definitely had  
> bugs on master, but I thought they were newly introduced on master  
> as well, which is why the fixes weren't backported. But if you're  
> having files be in the wrong location, maybe there are bugs on 3.0.x  
> as well.
>
> Do you have the syslog lines at the time that email was delivered?

I dont' have the log, for that message, but I will search for a
more recent example.


 From the mail headers i know that it was not dilivered to the archive  
partition
but moved by cyr_expire. The conversation db was not used at that time.

PS.: the timesamp of the file is not the internal date but the time
the mail was moved to the archive partition. I was wondering about the reason.

Michael


--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universit?t T?bingen                   Fax.: (49) 7071/29-5912
Zentrum f?r Datenverarbeitung          mail:  
michael.menge at zdv.uni-tuebingen.de
W?chterstra?e 76
72074 T?bingen


From brong at fastmailteam.com  Mon Feb  4 06:09:44 2019
From: brong at fastmailteam.com (Bron Gondwana)
Date: Mon, 04 Feb 2019 06:09:44 -0500
Subject: another problem with conversations db
In-Reply-To: <20190204120013.Horde.6MsNbwrJRdIBADBt9I1z3QP@webmail.uni-tuebingen.de>
References: <20190124180232.Horde.Tya11Hrse_nHFrp7Zj39dP4@webmail.uni-tuebingen.de>
 <363a2149-1b36-4dc0-aca5-b121c8ae3266@beta.fastmail.com>
 <20190125095538.Horde.BZLLa6BdReb8ZHa6b9Vxg1r@webmail.uni-tuebingen.de>
 <5d089935-a59f-45d7-a50c-cec40e23082c@beta.fastmail.com>
 <20190131093559.Horde.Vu9F7oPMtLNy30qgYXzWgKL@webmail.uni-tuebingen.de>
 <c02770dd-4238-4aab-a034-bd447b58cd06@www.fastmail.com>
 <20190204101712.Horde.33ManGeIHA8D5TVzSiQ7825@webmail.uni-tuebingen.de>
 <02bb719c-27ce-4e62-84e8-f11a32adb3dc@www.fastmail.com>
 <20190204120013.Horde.6MsNbwrJRdIBADBt9I1z3QP@webmail.uni-tuebingen.de>
Message-ID: <bfac9abf-ad3b-42ad-a668-0e57961d5a2b@www.fastmail.com>

On Mon, Feb 4, 2019, at 22:00, Michael Menge wrote:
> 
> Quoting Bron Gondwana <brong at fastmail.fm>:
> 
> > On Mon, Feb 4, 2019, at 20:21, Michael Menge wrote:
> >> Hi,
> >>
> >> Quoting Bron Gondwana <brong at fastmailteam.com>:
> >>
> >> > Hi Michael,
> >> >
> >> > Sorry about the delay in looking at this - I was mad crazy busy
> >> > getting ready to go overseas. At Fosdem now, about to give a talk
> >> > about JMAP!
> >> >
> >> > OK, let's start with the things that give me a little bit of hives...
> >> >
> >> > configdirectory: /srv/cyrus-be
> >> > partition-default: /srv/cyrus-be
> >> > partition-ssd: /srv/cyrus-be/ssd-part
> >> >
> >> > Ouch. There's a couple of things I wouldn't do there - having the
> >> > partition be the same as the config directory, and having a separate
> >> > partition be a subdirectory of a partition. They're both asking for
> >> > trouble. I would probably lay my system out like:
> >> >
> >> > configdirectory: /srv/cyrus-be/conf
> >> > partition-default: /srv/cyrus-be/default-part
> >> > partition-ssd: /srv/cyrus-be/ssd-part
> >> >
> >>
> >> partition-default isn't used any more. To use the metapartition we moved
> >> all accounts form the default partition to the ssd partition which is the
> >> the new defaultpartition ("defaultpartition: ssd")
> >
> > Right - that makes sense.
> >
> >> > And then each tree would only have one type of thing in it.
> >> >
> >> > Anyway, I don't think that would break anything.
> >> >
> >> > metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part
> >> > metapartition_files: header index cache expunge squat annotations
> >> > lock dav archivecache
> >> >
> >> > Ooh, I haven't tested having cache and archivecache on the same
> >> > location. That's really interesting. Again, I'd be in favour of
> >> > separation here, give them different paths. That might be tricky
> >> > with ssd though, the way this is laid out. I assume you have some
> >> > kind of symlink farm going on?
> >> >
> >>
> >> I didn't know that there could be a problem with cache and archivecache.
> >> At the time we decided on the configuration for cyrus 3.0 I looked at the
> >> imapd.conf man page and for metapartition_files decided that I want all
> >> meta files on the ssd storage. There was no indication in the man page
> >> that there could be a problem.
> >
> > Fair. I'd have to test that to see if it works correctly. I would 
> > hope so, but I haven't tested that configuration. This is the 
> > downside with having lots of different ways to do things!
> >
> >> How do I separate location of archivecache from the other 
> >> metapartition path?
> >> And fix the cache and archivecache files?
> >
> > This I don't know a good answer for. I will test if having the same 
> > path for cache and archivecache could fail. I THINK that I made the 
> > code safe for it, but I'm not sure that it's been tested.
> >
> >> No there is no sysmlink farm. We have mounted different iSCSI volumes to
> >> /srv/cyrus-ssd-be, /srv/cyrus-hdd-be and /srv/cyrus-be
> >
> > Right. That makes sense.
> >
> >> > Otherwise it all looks OK. Are you getting other IOERRORs in your
> >> > log files which could show things aborting? It really looks like
> >> > your conversations DB is getting out of sync due to other failures.
> >>
> >> I found a few instances of 3 related errors.
> >>
> >> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening
> >> /srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory
> >> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening
> >> /srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory
> >> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR archive
> >> user.XXXX 2185 failed to copyfile
> >> (/srv/cyrus-be/ssd-part/L/user/XXXX/2185. =>
> >> /srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.): Unknown code
> >> ____ 255
> >
> >
> > Ouch. Yeah, that could have been caused by a bug in delivery, and 
> > would definitely cause conversations DB corruption if the index file 
> > was updated but the conversations DB wasn't or vice versa.
> >
> >> The file was already at /srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.
> >
> > Right! I do wonder if there are some bugs in 3.0.x which are fixed 
> > on master around delivery to archive partition. We definitely had 
> > bugs on master, but I thought they were newly introduced on master 
> > as well, which is why the fixes weren't backported. But if you're 
> > having files be in the wrong location, maybe there are bugs on 3.0.x 
> > as well.
> >
> > Do you have the syslog lines at the time that email was delivered?
> 
> I dont' have the log, for that message, but I will search for a
> more recent example.

Great.

> 
> From the mail headers i know that it was not dilivered to the archive 
> partition
> but moved by cyr_expire. The conversation db was not used at that time.

OK - that shouldn't matter then - because the conversations rebuild should have found it.

> PS.: the timesamp of the file is not the internal date but the time
> the mail was moved to the archive partition. I was wondering about the reason.

Hmm, yeah:

r = cyrus_copyfile(srcname, destname, COPYFILE_MKDIR);

That's how the file is moved. It only does a hardlink if it's the same filesystem. Interestingly, it does NOT set the timestamp correctly. This is clearly a bug.

https://github.com/cyrusimap/cyrus-imapd/issues/2641

Bron.


--
 Bron Gondwana, CEO, FastMail Pty Ltd
 brong at fastmailteam.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190204/c3bb8d6c/attachment-0001.html>

From daniel-listas at gmx.net  Mon Feb  4 11:25:19 2019
From: daniel-listas at gmx.net (Daniel Bareiro)
Date: Mon, 4 Feb 2019 13:25:19 -0300
Subject: Fatal error: Exists wrong 513 512 483 17057
In-Reply-To: <d39918e4-b350-49dd-8c89-8d81422e9d7a@www.fastmail.com>
References: <df524faf-eefc-82c6-4605-34793bfe07a4@gmx.net>
 <c106bf37-0c79-4793-a812-0fe3e717d819@beta.fastmail.com>
 <7ed47a3a-ff4d-e91d-01c2-1141bd85e432@gmx.net>
 <d39918e4-b350-49dd-8c89-8d81422e9d7a@www.fastmail.com>
Message-ID: <21346e2a-d8c6-d76a-71a8-251ae04a380b@gmx.net>

Hi, Bron. Thanks for your reply.

On 4/2/19 05:45, Bron Gondwana wrote:

> Oh how frustrating...

>> I tried using 'reconstruct' as you suggested and this was the result:
>>
>> ----------
>> # /usr/lib/cyrus/bin/reconstruct -r user.admin
>> user.admin
>> user.admin.Backups
>> user.admin.Drafts
>> user.admin.Sent
>> user.admin.TareasCron: record corrupted 13 (maybe uid 9836)
>> user.admin.TareasCron: Mailbox format corruption detected Failed to
>> reconstruct mailbox
>> user.admin.Trash
>> ----------

> This clearly isn't syslogging enough!? Can you check the permissions on
> the underlying filesystem!? Also, try running reconstruct with -G,
> though it shouldn't make a difference.

The previous thing was the output that I was obtaining in the console
when executing 'reconstruct'. This is what I get in the syslog when
executing the command with the '-G' flag:

----------
Feb  4 11:48:29 mail cyrus/reconstruct[22727]: reconstructing user.admin
Feb  4 11:48:47 mail cyrus/reconstruct[22727]: mailbox: longlock
user.admin for 17.8 seconds
Feb  4 11:48:47 mail cyrus/reconstruct[22727]: Repacking mailbox
user.admin version 13
Feb  4 11:48:48 mail cyrus/reconstruct[22727]: reconstructing
user.admin.Backups
Feb  4 11:48:50 mail cyrus/reconstruct[22727]: mailbox: longlock
user.admin.Backups for 2.5 seconds
Feb  4 11:48:50 mail cyrus/reconstruct[22727]: Repacking mailbox
user.admin.Backups version 13
Feb  4 11:48:50 mail cyrus/reconstruct[22727]: reconstructing
user.admin.Drafts
Feb  4 11:48:51 mail cyrus/reconstruct[22727]: reconstructing
user.admin.Sent
Feb  4 11:48:51 mail cyrus/reconstruct[22727]: reconstructing
user.admin.TareasCron
----------

But in essence the console shows the same output as without adding '-G':

----------
# /usr/lib/cyrus/bin/reconstruct -G -r user.admin
user.admin
user.admin.Backups
user.admin.Drafts
user.admin.Sent
user.admin.TareasCron: record corrupted 13 (maybe uid 9836)
user.admin.TareasCron: Mailbox format corruption detected Failed to
reconstruct mailbox
user.admin.Trash
----------

>> Just 'TasksCron' is the folder that I can not open. What do you suggest
>> to do in this case?

> I'd take a backup, and then delete the cyrus.index file in that folder
> and run reconstruct again...? That's a pretty extreme approach, but it
> should work.

Let's try the pretty extreme approach... :-D

----------
Feb  4 12:04:43 mail cyrus/reconstruct[22990]: reconstructing user.admin
Feb  4 12:04:44 mail cyrus/reconstruct[22990]: reconstructing
user.admin.Backups
Feb  4 12:04:44 mail cyrus/reconstruct[22990]: reconstructing
user.admin.Drafts
Feb  4 12:04:44 mail cyrus/reconstruct[22990]: reconstructing
user.admin.Sent
Feb  4 12:04:44 mail cyrus/reconstruct[22990]: reconstructing
user.admin.TareasCron
Feb  4 12:04:44 mail cyrus/reconstruct[22990]: IOERROR: opening index
user.admin.TareasCron: System I/O error
Feb  4 12:04:44 mail cyrus/reconstruct[22990]: create new mailbox
user.admin.TareasCron
Feb  4 12:04:44 mail cyrus/reconstruct[22990]: failed to read index
header for user.admin.TareasCron
Feb  4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
9837 found - adding
Feb  4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
9838 found - adding
Feb  4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
9839 found - adding
Feb  4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
9840 found - adding
Feb  4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
9841 found - adding
Feb  4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
9842 found - adding
Feb  4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
9843 found - adding
Feb  4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
9844 found - adding
Feb  4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
9845 found - adding
[...]
Feb  4 12:07:21 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
27841 found - adding
Feb  4 12:07:21 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
27842 found - adding
Feb  4 12:07:21 mail cyrus/reconstruct[22990]: mailbox: longlock
user.admin.TareasCron for 157.3 seconds
Feb  4 12:07:21 mail cyrus/reconstruct[22990]: reconstructing
user.admin.Trash
----------

I guess the 'IOERROR' is because the cyrus.index file can not be found.
When afterwards it says 'create new mailbox user.admin.TareasCron' what
it is creating is the index? Because, if so, I don't understand why
later it says 'failed to read index header for user.admin.TareasCron'.

In any case, I am now able to access the 'TareasCron' folder without
problems. Thank you so much!

>> I'm using Cyrus 2.5.10-3 on Debian Stretch. A few days ago I upgraded
>> the entire operating system with Debian Jessie and Cyrus 2.4.17.

> That upgrade should be fine, and 2.5.10 should be fine as well.? I just
> poked around in the code there to see how the reconstruct might be
> failing, and there's not enough information to know what's happening.

I think the only problem was with this folder. The access to the other
mailboxes worked without problems after the upgrade.


Thank you very much for your time.

Kind regards,
Daniel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190204/c55b1488/attachment.sig>

From brong at fastmailteam.com  Mon Feb  4 12:02:50 2019
From: brong at fastmailteam.com (Bron Gondwana)
Date: Mon, 04 Feb 2019 12:02:50 -0500
Subject: Fatal error: Exists wrong 513 512 483 17057
In-Reply-To: <21346e2a-d8c6-d76a-71a8-251ae04a380b@gmx.net>
References: <df524faf-eefc-82c6-4605-34793bfe07a4@gmx.net>
 <c106bf37-0c79-4793-a812-0fe3e717d819@beta.fastmail.com>
 <7ed47a3a-ff4d-e91d-01c2-1141bd85e432@gmx.net>
 <d39918e4-b350-49dd-8c89-8d81422e9d7a@www.fastmail.com>
 <21346e2a-d8c6-d76a-71a8-251ae04a380b@gmx.net>
Message-ID: <3e848eb7-a772-4b2a-88d7-a99e37f4c00d@www.fastmail.com>

Awesome - yes, the IOERROR messages are because files it expected to find weren't there. It's frustrating that reconstruct isn't robust enough to bring a slightly bogus cyrus.index back from the dead, but the end result is a working mailbox with a fully correct cyrus.index file :)

The only thing that may have happened is that some emails which were previously deleted in the TareasCron folder came back from the dead - and of course all the flags on those messages will have gone away too. But the mailbox will work correctly now.

Cheers,

Bron.

On Tue, Feb 5, 2019, at 03:26, Daniel Bareiro wrote:
> Hi, Bron. Thanks for your reply.
> 
> On 4/2/19 05:45, Bron Gondwana wrote:
> 
> > Oh how frustrating...
> 
> >> I tried using 'reconstruct' as you suggested and this was the result:
> >>
> >> ----------
> >> # /usr/lib/cyrus/bin/reconstruct -r user.admin
> >> user.admin
> >> user.admin.Backups
> >> user.admin.Drafts
> >> user.admin.Sent
> >> user.admin.TareasCron: record corrupted 13 (maybe uid 9836)
> >> user.admin.TareasCron: Mailbox format corruption detected Failed to
> >> reconstruct mailbox
> >> user.admin.Trash
> >> ----------
> 
> > This clearly isn't syslogging enough! Can you check the permissions on
> > the underlying filesystem! Also, try running reconstruct with -G,
> > though it shouldn't make a difference.
> 
> The previous thing was the output that I was obtaining in the console
> when executing 'reconstruct'. This is what I get in the syslog when
> executing the command with the '-G' flag:
> 
> ----------
> Feb 4 11:48:29 mail cyrus/reconstruct[22727]: reconstructing user.admin
> Feb 4 11:48:47 mail cyrus/reconstruct[22727]: mailbox: longlock
> user.admin for 17.8 seconds
> Feb 4 11:48:47 mail cyrus/reconstruct[22727]: Repacking mailbox
> user.admin version 13
> Feb 4 11:48:48 mail cyrus/reconstruct[22727]: reconstructing
> user.admin.Backups
> Feb 4 11:48:50 mail cyrus/reconstruct[22727]: mailbox: longlock
> user.admin.Backups for 2.5 seconds
> Feb 4 11:48:50 mail cyrus/reconstruct[22727]: Repacking mailbox
> user.admin.Backups version 13
> Feb 4 11:48:50 mail cyrus/reconstruct[22727]: reconstructing
> user.admin.Drafts
> Feb 4 11:48:51 mail cyrus/reconstruct[22727]: reconstructing
> user.admin.Sent
> Feb 4 11:48:51 mail cyrus/reconstruct[22727]: reconstructing
> user.admin.TareasCron
> ----------
> 
> But in essence the console shows the same output as without adding '-G':
> 
> ----------
> # /usr/lib/cyrus/bin/reconstruct -G -r user.admin
> user.admin
> user.admin.Backups
> user.admin.Drafts
> user.admin.Sent
> user.admin.TareasCron: record corrupted 13 (maybe uid 9836)
> user.admin.TareasCron: Mailbox format corruption detected Failed to
> reconstruct mailbox
> user.admin.Trash
> ----------
> 
> >> Just 'TasksCron' is the folder that I can not open. What do you suggest
> >> to do in this case?
> 
> > I'd take a backup, and then delete the cyrus.index file in that folder
> > and run reconstruct again... That's a pretty extreme approach, but it
> > should work.
> 
> Let's try the pretty extreme approach... :-D
> 
> ----------
> Feb 4 12:04:43 mail cyrus/reconstruct[22990]: reconstructing user.admin
> Feb 4 12:04:44 mail cyrus/reconstruct[22990]: reconstructing
> user.admin.Backups
> Feb 4 12:04:44 mail cyrus/reconstruct[22990]: reconstructing
> user.admin.Drafts
> Feb 4 12:04:44 mail cyrus/reconstruct[22990]: reconstructing
> user.admin.Sent
> Feb 4 12:04:44 mail cyrus/reconstruct[22990]: reconstructing
> user.admin.TareasCron
> Feb 4 12:04:44 mail cyrus/reconstruct[22990]: IOERROR: opening index
> user.admin.TareasCron: System I/O error
> Feb 4 12:04:44 mail cyrus/reconstruct[22990]: create new mailbox
> user.admin.TareasCron
> Feb 4 12:04:44 mail cyrus/reconstruct[22990]: failed to read index
> header for user.admin.TareasCron
> Feb 4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
> 9837 found - adding
> Feb 4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
> 9838 found - adding
> Feb 4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
> 9839 found - adding
> Feb 4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
> 9840 found - adding
> Feb 4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
> 9841 found - adding
> Feb 4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
> 9842 found - adding
> Feb 4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
> 9843 found - adding
> Feb 4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
> 9844 found - adding
> Feb 4 12:04:44 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
> 9845 found - adding
> [...]
> Feb 4 12:07:21 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
> 27841 found - adding
> Feb 4 12:07:21 mail cyrus/reconstruct[22990]: user.admin.TareasCron uid
> 27842 found - adding
> Feb 4 12:07:21 mail cyrus/reconstruct[22990]: mailbox: longlock
> user.admin.TareasCron for 157.3 seconds
> Feb 4 12:07:21 mail cyrus/reconstruct[22990]: reconstructing
> user.admin.Trash
> ----------
> 
> I guess the 'IOERROR' is because the cyrus.index file can not be found.
> When afterwards it says 'create new mailbox user.admin.TareasCron' what
> it is creating is the index? Because, if so, I don't understand why
> later it says 'failed to read index header for user.admin.TareasCron'.
> 
> In any case, I am now able to access the 'TareasCron' folder without
> problems. Thank you so much!
> 
> >> I'm using Cyrus 2.5.10-3 on Debian Stretch. A few days ago I upgraded
> >> the entire operating system with Debian Jessie and Cyrus 2.4.17.
> 
> > That upgrade should be fine, and 2.5.10 should be fine as well. I just
> > poked around in the code there to see how the reconstruct might be
> > failing, and there's not enough information to know what's happening.
> 
> I think the only problem was with this folder. The access to the other
> mailboxes worked without problems after the upgrade.
> 
> 
> Thank you very much for your time.
> 
> Kind regards,
> Daniel
> 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> *Attachments:*
>  * signature.asc

--
 Bron Gondwana, CEO, FastMail Pty Ltd
 brong at fastmailteam.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190204/03043303/attachment.html>

From ml at netfence.it  Mon Feb  4 14:12:59 2019
From: ml at netfence.it (Andrea Venturoli)
Date: Mon, 4 Feb 2019 20:12:59 +0100
Subject: Sieve script not working
In-Reply-To: <67349212-1882-4ba5-a8d6-e06d569f37a0@beta.fastmail.com>
References: <f212e3ad-bf37-6da5-ca21-3f20d2fcf9c4@andreaventuroli.it>
 <3980dfd9-d397-7f3e-5781-e58185c2746f@onlight.com>
 <c270e06b-2ccc-9134-8b91-7701dfa26b54@andreaventuroli.it>
 <9f0dfe24-1d79-2ce8-7847-f78a80b369ea@netfence.it>
 <67349212-1882-4ba5-a8d6-e06d569f37a0@beta.fastmail.com>
Message-ID: <b372dc51-8120-6ded-f018-d70965dc9859@netfence.it>

On 2/2/19 8:25 PM, Bron Gondwana wrote:
> What is being written to syslog by your lmtpd?

Absolutely nothing (apart from transient errors on the rare occasions 
when I reboot the server and sendmail comes up before imapd).

Is this log to be enabled somehow?




> There's also a sieve-test binary that you can build in the Cyrus source 
> code. I don't think it's built by default on most platforms though, we 
> build it specially at FM. It takes a script and a raw email and spits 
> out a list of actions.

I don't have this (FreeBSD 11.2).
AFAICT the port doesn't disable this (in configure) or leave it behind 
when installing.
I tried searching for this binary between "make" and "make install" 
phases (*), but I didn't find it (or any source named anything like it).
What's the procedure to compile it?

(*)
> # cd work/cyrus-imapd-2.5.12/
> # find .|grep -i "sieve-test"
> # find . -type f -exec grep -i "sieve-test" "{}" ";"





  bye & Thanks
	av.

From egoitz at sarenet.es  Tue Feb  5 05:57:59 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Tue, 05 Feb 2019 11:57:59 +0100
Subject: Conversations db in 3.0
Message-ID: <9c2c0256697b5650fb2bbdc19a483cc5@sarenet.es>

Hi! 

In an upgrade proccesses to upper version you don't have a way revert
back the update without loosing content. I was very interested in
knowing if I could have a 2.4 master server (without Xapian,
conversations and so obviously) replicating to a 3.0 server and if I
could perform the : 

- reconstruct -V max 

- ctl_conversationsdb -z USER 

- ctl_conversationsdb -b USER 

but while the 3.0 is slave and while is replicating from 2.4. 

Is this possible? 

Thanks a lot mates,

-- 

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia) 
egoitz at sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electr?nico piense si es necesario
hacerlo. 

Links:
------
[1] http://www.sarenet.es
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190205/d505b11c/attachment-0001.html>

From lists at localguru.de  Tue Feb  5 12:08:43 2019
From: lists at localguru.de (Marcus Schopen)
Date: Tue, 05 Feb 2019 18:08:43 +0100
Subject: folder left after dm user.mailboxname, remove by rm ?
Message-ID: <a548cf1c968a67bfcb894cee9fcad56a326786f9.camel@localguru.de>

Hi,

I've deleted an unused mailbox "dm user.emil" and cleaned up DELETED
with 'su - cyrus -c "/usr/sbin/cyrus expire -E 1 -X 0 -D 0 -v -p
DELETED.user.emil'.

/var/spool/cyrus/mail/u/DELETED/user is empty now and 
"su - cyrus -c "/usr/sbin/ctl_mboxlist -d" | grep -i emil" doesn't show
any user.emil mailboxes anymore.

Nevertheless there is one single folder left on the master in 

 /var/spool/cyrus/mail/e/user/emil/team/

which contains some messages and cyrus.cache, cyrus.header,
cyrus.header files. So this must have been a normal folder once.

If I understood that correctly, Cyrus doesn't know about the folder
anymore because the dump with "ctl_mboxlist -d" shows that the user has
been removed completely, no user.email folders are left in the dump
output.

Can I remove that folder securely just by

  rm -rf /var/spool/cyrus/mail/e/user/emil/

The following question would be, why the folder was not available in
the cyrus database. But since the user is deleted anyway, this would
not be my primary problem, especially since the user had worked with
Outlook and had hamstered about 11 GB and 1200 folders in his mailbox.

Ciao!
Marcus


From Eric.Luyten at vub.ac.be  Tue Feb  5 12:19:45 2019
From: Eric.Luyten at vub.ac.be (Eric Luyten)
Date: Tue, 5 Feb 2019 18:19:45 +0100
Subject: folder left after dm user.mailboxname, remove by rm ?
In-Reply-To: <a548cf1c968a67bfcb894cee9fcad56a326786f9.camel@localguru.de>
References: <a548cf1c968a67bfcb894cee9fcad56a326786f9.camel@localguru.de>
Message-ID: <886dde05-71eb-6584-6444-c9b31c383e17@vub.ac.be>


On 05/02/2019 18:08, Marcus Schopen wrote:
> Hi,
>
> I've deleted an unused mailbox "dm user.emil" and cleaned up DELETED
> with 'su - cyrus -c "/usr/sbin/cyrus expire -E 1 -X 0 -D 0 -v -p
> DELETED.user.emil'.
>
> /var/spool/cyrus/mail/u/DELETED/user is empty now and
> "su - cyrus -c "/usr/sbin/ctl_mboxlist -d" | grep -i emil" doesn't show
> any user.emil mailboxes anymore.
>
> Nevertheless there is one single folder left on the master in
>
>   /var/spool/cyrus/mail/e/user/emil/team/
>
> which contains some messages and cyrus.cache, cyrus.header,
> cyrus.header files. So this must have been a normal folder once.
>
> If I understood that correctly, Cyrus doesn't know about the folder
> anymore because the dump with "ctl_mboxlist -d" shows that the user has
> been removed completely, no user.email folders are left in the dump
> output.
>
> Can I remove that folder securely just by
>
>    rm -rf /var/spool/cyrus/mail/e/user/emil/
>
> The following question would be, why the folder was not available in
> the cyrus database. But since the user is deleted anyway, this would
> not be my primary problem, especially since the user had worked with
> Outlook and had hamstered about 11 GB and 1200 folders in his mailbox.
>

Hello Marcus,



I do not know what Cyrus version you are running but very occasionally, 
on a 2.3 system, I witness the same phenomenon.

To give you an idea : this happens on average once every thousand (or 
so) account removals.

If Cyrus doesn't know about the directory through mailboxes.db it is 
never going to remove it by itself.



Cheers,

Eric Luyten.



From daniel-listas at gmx.net  Tue Feb  5 12:40:33 2019
From: daniel-listas at gmx.net (Daniel Bareiro)
Date: Tue, 5 Feb 2019 14:40:33 -0300
Subject: Fatal error: Exists wrong 513 512 483 17057
In-Reply-To: <3e848eb7-a772-4b2a-88d7-a99e37f4c00d@www.fastmail.com>
References: <df524faf-eefc-82c6-4605-34793bfe07a4@gmx.net>
 <c106bf37-0c79-4793-a812-0fe3e717d819@beta.fastmail.com>
 <7ed47a3a-ff4d-e91d-01c2-1141bd85e432@gmx.net>
 <d39918e4-b350-49dd-8c89-8d81422e9d7a@www.fastmail.com>
 <21346e2a-d8c6-d76a-71a8-251ae04a380b@gmx.net>
 <3e848eb7-a772-4b2a-88d7-a99e37f4c00d@www.fastmail.com>
Message-ID: <2bf6e152-52bb-eb0c-daf3-210109487a4e@gmx.net>

Hi, Bron.

On 4/2/19 14:02, Bron Gondwana wrote:

> Awesome - yes, the IOERROR messages are because files it expected to
> find weren't there.? It's frustrating that reconstruct isn't robust
> enough to bring a slightly bogus cyrus.index back from the dead, but the
> end result is a working mailbox with a fully correct cyrus.index file :)
> 
> The only thing that may have happened is that some emails which were
> previously deleted in the TareasCron folder came back from the dead -
> and of course all the flags on those messages will have gone away too.?
> But the mailbox will work correctly now.

Maybe that explains why I came across about 16,000 messages in the
folder :) But everything works correctly now.

I deleted the very old messages and only left the last ones, although in
the 'TasksCron' directory I am seeing messages on filesystem
(.<SomeNumber> files) of 2016 still. Could it be that for some reason
Cyrus keeps these files? In fact, in the 'Trash' directory I'm also
seeing files from 2016, even though I deleted everything I had there
using Thunderbird.

Thanks for your reply.

Kind regards,
Daniel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190205/20ffd4af/attachment.sig>

From lists at localguru.de  Tue Feb  5 14:35:57 2019
From: lists at localguru.de (Marcus Schopen)
Date: Tue, 05 Feb 2019 20:35:57 +0100
Subject: folder left after dm user.mailboxname, remove by rm ?
In-Reply-To: <886dde05-71eb-6584-6444-c9b31c383e17@vub.ac.be>
References: <a548cf1c968a67bfcb894cee9fcad56a326786f9.camel@localguru.de>
 <886dde05-71eb-6584-6444-c9b31c383e17@vub.ac.be>
Message-ID: <1967d200cb2ad56332fe987c398560f7f167a93d.camel@localguru.de>

Hi Eric,

thanks for your time!

Am Dienstag, den 05.02.2019, 18:19 +0100 schrieb Eric Luyten:
> 
> I do not know what Cyrus version you are running but very
> occasionally, 
> on a 2.3 system, I witness the same phenomenon.

It's a 2.4 version.

> To give you an idea : this happens on average once every thousand
> (or 
> so) account removals.
> 
> If Cyrus doesn't know about the directory through mailboxes.db it is 
> never going to remove it by itself.

I understand. Is it safe then to remove that "forgotten" folder by hand
(rm -rf ...).

Ciao!



From brong at fastmail.fm  Tue Feb  5 15:21:36 2019
From: brong at fastmail.fm (Bron Gondwana)
Date: Tue, 05 Feb 2019 15:21:36 -0500
Subject: Fatal error: Exists wrong 513 512 483 17057
In-Reply-To: <2bf6e152-52bb-eb0c-daf3-210109487a4e@gmx.net>
References: <df524faf-eefc-82c6-4605-34793bfe07a4@gmx.net>
 <c106bf37-0c79-4793-a812-0fe3e717d819@beta.fastmail.com>
 <7ed47a3a-ff4d-e91d-01c2-1141bd85e432@gmx.net>
 <d39918e4-b350-49dd-8c89-8d81422e9d7a@www.fastmail.com>
 <21346e2a-d8c6-d76a-71a8-251ae04a380b@gmx.net>
 <3e848eb7-a772-4b2a-88d7-a99e37f4c00d@www.fastmail.com>
 <2bf6e152-52bb-eb0c-daf3-210109487a4e@gmx.net>
Message-ID: <f48a0f7b-c7ad-4ab9-af3a-3113225ab034@www.fastmail.com>

You need to run cyr_expire occasionally to clean up deleted messages!

Bron.

On Wed, Feb 6, 2019, at 04:45, Daniel Bareiro wrote:
> Hi, Bron.
> 
> On 4/2/19 14:02, Bron Gondwana wrote:
> 
> > Awesome - yes, the IOERROR messages are because files it expected to
> > find weren't there. It's frustrating that reconstruct isn't robust
> > enough to bring a slightly bogus cyrus.index back from the dead, but the
> > end result is a working mailbox with a fully correct cyrus.index file :)
> > 
> > The only thing that may have happened is that some emails which were
> > previously deleted in the TareasCron folder came back from the dead -
> > and of course all the flags on those messages will have gone away too. 
> > But the mailbox will work correctly now.
> 
> Maybe that explains why I came across about 16,000 messages in the
> folder :) But everything works correctly now.
> 
> I deleted the very old messages and only left the last ones, although in
> the 'TasksCron' directory I am seeing messages on filesystem
> (.<SomeNumber> files) of 2016 still. Could it be that for some reason
> Cyrus keeps these files? In fact, in the 'Trash' directory I'm also
> seeing files from 2016, even though I deleted everything I had there
> using Thunderbird.
> 
> Thanks for your reply.
> 
> Kind regards,
> Daniel
> 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> *Attachments:*
>  * signature.asc

-- 
 Bron Gondwana
 brong at fastmail.fm

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190205/0c1b9902/attachment-0001.html>

From boutilpj at ednet.ns.ca  Tue Feb  5 15:31:00 2019
From: boutilpj at ednet.ns.ca (Patrick Boutilier)
Date: Tue, 5 Feb 2019 16:31:00 -0400
Subject: folder left after dm user.mailboxname, remove by rm ?
In-Reply-To: <1967d200cb2ad56332fe987c398560f7f167a93d.camel@localguru.de>
References: <a548cf1c968a67bfcb894cee9fcad56a326786f9.camel@localguru.de>
 <886dde05-71eb-6584-6444-c9b31c383e17@vub.ac.be>
 <1967d200cb2ad56332fe987c398560f7f167a93d.camel@localguru.de>
Message-ID: <94411b9f-ab65-106d-79c8-777f7387715a@ednet.ns.ca>

On 2/5/19 3:35 PM, Marcus Schopen wrote:
> Hi Eric,
> 
> thanks for your time!
> 
> Am Dienstag, den 05.02.2019, 18:19 +0100 schrieb Eric Luyten:
>>
>> I do not know what Cyrus version you are running but very
>> occasionally,
>> on a 2.3 system, I witness the same phenomenon.
> 
> It's a 2.4 version.
> 
>> To give you an idea : this happens on average once every thousand
>> (or
>> so) account removals.
>>
>> If Cyrus doesn't know about the directory through mailboxes.db it is
>> never going to remove it by itself.
> 
> I understand. Is it safe then to remove that "forgotten" folder by hand
> (rm -rf ...).

I don't see why not. I have done it before. You can always just move it 
somewhere out of the way and move it back if it causes any issues.


Also, just wanted to point out another way of deleting users from 
"DELETED" without having to use expire.


localhost> lm DELETED.user.soandso.*
DELETED.user.soandso.5C3F3CFC (\HasNoChildren)
DELETED.user.soandso.sent-mail.5C3F3CFC (\HasNoChildren)


localhost> dm DELETED.user.soandso.*
Deleting mailbox DELETED.user.soandso.5C3F3CFC...OK.
Deleting mailbox DELETED.user.soandso.sent-mail.5C3F3CFC...OK.

> 
> Ciao!
> 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: boutilpj.vcf
Type: text/x-vcard
Size: 286 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190205/e9f7593b/attachment.vcf>

From daniel-listas at gmx.net  Tue Feb  5 17:08:28 2019
From: daniel-listas at gmx.net (Daniel Bareiro)
Date: Tue, 5 Feb 2019 19:08:28 -0300
Subject: Fatal error: Exists wrong 513 512 483 17057
In-Reply-To: <f48a0f7b-c7ad-4ab9-af3a-3113225ab034@www.fastmail.com>
References: <df524faf-eefc-82c6-4605-34793bfe07a4@gmx.net>
 <c106bf37-0c79-4793-a812-0fe3e717d819@beta.fastmail.com>
 <7ed47a3a-ff4d-e91d-01c2-1141bd85e432@gmx.net>
 <d39918e4-b350-49dd-8c89-8d81422e9d7a@www.fastmail.com>
 <21346e2a-d8c6-d76a-71a8-251ae04a380b@gmx.net>
 <3e848eb7-a772-4b2a-88d7-a99e37f4c00d@www.fastmail.com>
 <2bf6e152-52bb-eb0c-daf3-210109487a4e@gmx.net>
 <f48a0f7b-c7ad-4ab9-af3a-3113225ab034@www.fastmail.com>
Message-ID: <f1c7bbe7-cd2e-b7b1-d60f-0f84273e2b02@gmx.net>

Hi, Bron. Thanks for your reply.

On 5/2/19 17:21, Bron Gondwana wrote:

> You need to run cyr_expire occasionally to clean up deleted messages!

Checking the syslog, apparently it ran automatically this morning:

----------
# grep cyr_expire /var/log/syslog /var/log/syslog.1
/var/log/syslog.1:Feb  5 04:01:03 mail cyrus/cyr_expire[6111]: Expired 0
and expunged 0 out of 53555 messages from 80 mailboxes
/var/log/syslog.1:Feb  5 04:01:03 mail cyrus/cyr_expire[6111]:
duplicate_prune: pruning back 3.00 days
/var/log/syslog.1:Feb  5 04:03:16 mail cyrus/cyr_expire[6111]:
duplicate_prune: purged 562 out of 1756 entries
/var/log/syslog.1:Feb  5 04:30:00 mail cyrus/cyr_expire[6585]: Expired 0
and expunged 0 out of 53558 messages from 80 mailboxes
/var/log/syslog.1:Feb  5 04:30:00 mail cyrus/cyr_expire[6585]: Removed 0
deleted mailboxes
/var/log/syslog.1:Feb  5 04:30:00 mail cyrus/cyr_expire[6585]:
duplicate_prune: pruning back 4.00 days
/var/log/syslog.1:Feb  5 04:30:00 mail cyrus/cyr_expire[6585]:
duplicate_prune: purged 0 out of 1200 entries
/var/log/syslog.1:Feb  5 04:45:00 mail cyrus/cyr_expire[6849]: Expired 0
and expunged 0 out of 53558 messages from 80 mailboxes
/var/log/syslog.1:Feb  5 04:45:00 mail cyrus/cyr_expire[6849]:
duplicate_prune: pruning back 4.00 days
/var/log/syslog.1:Feb  5 04:45:00 mail cyrus/cyr_expire[6849]:
duplicate_prune: purged 0 out of 1200 entries
----------

But still I see those files from 2016 to the present.

Checking /etc/cyrus.conf I found this configuration as default in Stretch:

----------
EVENTS {
(...)

        # this is only necessary if using duplicate delivery suppression
        delprune        cmd="/usr/sbin/cyrus expire -E 3" at=0401

(...)

        # Expire data older than 28 days.
        deleteprune     cmd="/usr/sbin/cyrus expire -E 4 -D 28" at=0430
        expungeprune    cmd="/usr/sbin/cyrus expire -E 4 -X 28" at=0445
(...)
----------

Am I missing something?


Thanks for your time.

Kind regards,
Daniel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190205/8f3ecf43/attachment.sig>

From brong at fastmailteam.com  Wed Feb  6 12:43:21 2019
From: brong at fastmailteam.com (Bron Gondwana)
Date: Wed, 06 Feb 2019 12:43:21 -0500
Subject: Sieve script not working
In-Reply-To: <b372dc51-8120-6ded-f018-d70965dc9859@netfence.it>
References: <f212e3ad-bf37-6da5-ca21-3f20d2fcf9c4@andreaventuroli.it>
 <3980dfd9-d397-7f3e-5781-e58185c2746f@onlight.com>
 <c270e06b-2ccc-9134-8b91-7701dfa26b54@andreaventuroli.it>
 <9f0dfe24-1d79-2ce8-7847-f78a80b369ea@netfence.it>
 <67349212-1882-4ba5-a8d6-e06d569f37a0@beta.fastmail.com>
 <b372dc51-8120-6ded-f018-d70965dc9859@netfence.it>
Message-ID: <33634f01-9c07-4435-b2e4-9a5e4b9a558e@www.fastmail.com>

On Tue, Feb 5, 2019, at 06:20, Andrea Venturoli wrote:
> On 2/2/19 8:25 PM, Bron Gondwana wrote:
> > What is being written to syslog by your lmtpd?
> 
> Absolutely nothing (apart from transient errors on the rare occasions 
> when I reboot the server and sendmail comes up before imapd).
> 
> Is this log to be enabled somehow?
> 

No, there should be something in the syslog if sieve errors:

2019-02-06T12:31:26-05:00 imap38 sloti38d1t08/lmtp[2916700]: sieve runtime error for foo at fastmail.com id <foo-123214321342 at example.com>: Fileinto (Testing): Mailbox does not exist

> > There's also a sieve-test binary that you can build in the Cyrus source 
> > code. I don't think it's built by default on most platforms though, we 
> > build it specially at FM. It takes a script and a raw email and spits 
> > out a list of actions.
> 
> I don't have this (FreeBSD 11.2).
> AFAICT the port doesn't disable this (in configure) or leave it behind 
> when installing.
> I tried searching for this binary between "make" and "make install" 
> phases (*), but I didn't find it (or any source named anything like it).
> What's the procedure to compile it?
> 
> (*)
> > # cd work/cyrus-imapd-2.5.12/
> > # find .|grep -i "sieve-test"
> > # find . -type f -exec grep -i "sieve-test" "{}" ";"
> 

Yeah, it's just called test.c in the sieve directory. Here's our buildscript code:

make install DESTDIR=\$(CURDIR)/debian/$basename
make install-binsymlinks DESTDIR=\$(CURDIR)/debian/$basename
/bin/bash ./libtool --mode=install install -o root -m 755 sieve/test \$(PWD)/debian/$basename/$basedir/bin/sieve-test
install -o root -m 755 tools/rehash debian/$basename/$basedir/bin/rehash
install -o root -m 755 tools/mkimap debian/$basename/$basedir/bin/mkimap

So the binary will probably be sieve/test from a standard build

Oh yeah, you might be interested in the config options we build with too :)

./configure --without-krb --with-perl=/usr/bin/perl --enable-silent-rules --enable-http --enable-calalarmd --enable-idled --with-extraident=$tag --prefix=/$basedir --with-lmdb --with-zlib --without-snmp --enable-replication --enable-xapian --enable-jmap --enable-backup XAPIAN_CONFIG=/usr/local/$CYRUSLIBS/bin/xapian-config-1.5
$LEXFIX
make -j 8 all CFLAGS="-g -fPIC -W -Wall -Werror -fstack-protector-all"


Cheers,

Bron.

--
 Bron Gondwana, CEO, FastMail Pty Ltd
 brong at fastmailteam.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190206/1ddc15e8/attachment.html>

From paul at vandervlis.nl  Fri Feb  8 05:48:30 2019
From: paul at vandervlis.nl (Paul van der Vlis)
Date: Fri, 8 Feb 2019 11:48:30 +0100
Subject: Cyrus IMAP in the next Debian
Message-ID: <0a71caa2-8f72-d439-b162-1e6586c58817@vandervlis.nl>

Hello,

The freeze for the next version of Debian is in the next week.

Somebody has packaged Cyrus version 3.08, but there are problems with
some of the Cassandane tests.

When my information is up-to-date, Debian 10 will ship with Cyrus IMAP
2.5.11, with security support for 5 years. Many other distro's like
Ubuntu are distributing this package too.

As a sysadmin, I don't like running programs from other sources then
from my distro. I like the integration and the extra checks.

I think it would be good if there would be more contact between the
Cyrus Debian developers and the Cyrus IMAP community. If somebody here
is interested, I would like to make contact.

In Debian you have also "backports", it would be possible to make a
backport with a newer version. But this never happened in the past for
Cyrus IMAP.

With regards,
Paul van der Vlis



-- 
Paul van der Vlis Linux systeembeheer Groningen
https://www.vandervlis.nl/


From egoitz at sarenet.es  Fri Feb  8 08:46:37 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Fri, 08 Feb 2019 14:46:37 +0100
Subject: Question about conversationsdb
Message-ID: <28b14465bd3906564f4a957db80be753@sarenet.es>

Hi!, 

One little question. If I wanted to upgrade a Cyrus server older then
3.0 but greater than 2.3 (so replication is compatible) the
conversations database would be created automatically in the slave if I
used an empty slave 3.0 and I replicated the whole content there from
the master running 2.4 for instance (and then no Xapian searches or
conversations for instance)?. 

Thanks a lot for your time :) 

Cheers!

-- 

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia) 
egoitz at sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electr?nico piense si es necesario
hacerlo. 

Links:
------
[1] http://www.sarenet.es
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190208/1a363724/attachment.html>

From egoitz at sarenet.es  Fri Feb  8 08:58:17 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Fri, 08 Feb 2019 14:58:17 +0100
Subject: Question about conversationsdb
In-Reply-To: <28b14465bd3906564f4a957db80be753@sarenet.es>
References: <28b14465bd3906564f4a957db80be753@sarenet.es>
Message-ID: <963797661b8ae9120b1b73e117333301@sarenet.es>

In the replica I assume sync_log should be off. But If I enable there
sync_log_unsuppressable_channels: squatter then I assume all Xapian
searches needing elements would be generated when the slave received
data from the master?. Even when the slave starts empty?. 

Thanks mates :)

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia) 
egoitz at sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electr?nico piense si es necesario
hacerlo. 

El 08-02-2019 14:46, Egoitz Aurrekoetxea escribi?:

> Hi!, 
> 
> One little question. If I wanted to upgrade a Cyrus server older then 3.0 but greater than 2.3 (so replication is compatible) the conversations database would be created automatically in the slave if I used an empty slave 3.0 and I replicated the whole content there from the master running 2.4 for instance (and then no Xapian searches or conversations for instance)?. 
> 
> Thanks a lot for your time :) 
> 
> Cheers!
> 
> -- 
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 944 209 470
> Parque Tecnol?gico. Edificio 103
> 48170 Zamudio (Bizkaia) 
> egoitz at sarenet.es 
> www.sarenet.es [1] 
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo. 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 

Links:
------
[1] http://www.sarenet.es
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190208/eff4eafb/attachment.html>

From paul at vandervlis.nl  Fri Feb  8 12:39:44 2019
From: paul at vandervlis.nl (Paul van der Vlis)
Date: Fri, 8 Feb 2019 18:39:44 +0100
Subject: Cyrus IMAP in the next Debian
In-Reply-To: <0a71caa2-8f72-d439-b162-1e6586c58817@vandervlis.nl>
References: <0a71caa2-8f72-d439-b162-1e6586c58817@vandervlis.nl>
Message-ID: <q3ket0$4vmo$1@blaine.gmane.org>

Op 08-02-19 om 11:48 schreef Paul van der Vlis:
> Hello,
> 
> The freeze for the next version of Debian is in the next week.
> 
> Somebody has packaged Cyrus version 3.08, but there are problems with
> some of the Cassandane tests.
> 
> When my information is up-to-date, Debian 10 will ship with Cyrus IMAP
> 2.5.11, with security support for 5 years. Many other distro's like
> Ubuntu are distributing this package too.

I've heard Ond?ej Sur? has corrected and uploaded Cyrus Imap 3.0.8 to
Debian unstable now, great! The packaging was done by Anthony Prade.

It's in the Debian autobuilder system now:
https://buildd.debian.org/status/package.php?p=cyrus-imapd&suite=sid

In some time it will be in Debian unstable and a few days later in
testing. https://packages.debian.org/search?keywords=cyrus-imapd

Ond?ej is still asking for testers. It is still possible to make changes
after the freeze next week. The stable release will be in a few months.

The git is here: https://salsa.debian.org/debian/cyrus-imapd

With regards,
Paul van der Vlis





-- 
Paul van der Vlis Linux systeembeheer Groningen
https://www.vandervlis.nl/


From paul at vandervlis.nl  Fri Feb  8 14:46:00 2019
From: paul at vandervlis.nl (Paul van der Vlis)
Date: Fri, 8 Feb 2019 20:46:00 +0100
Subject: cyrus-imapd not starting after upgrade
In-Reply-To: <2608b63d-2149-0b33-a261-0739baef0992@gmx.net>
References: <2608b63d-2149-0b33-a261-0739baef0992@gmx.net>
Message-ID: <1400e6b1-7fa6-5064-0aa3-19e41846ff5a@vandervlis.nl>

Op 15-01-19 om 17:33 schreef Daniel Bareiro:
> Hi all!
> 
> After quite some time, today I decided to update the mail server from
> Debian Jessie (cyrus-imapd 2.4.17) to Debian Stretch (cyrus-imapd 2.5.10-3).
Hello Daniel,

I use cyrus-imapd 2.5.10-3 from Debian stable on serveral machines and I
can tell you that it gives much TLS problems.

What I do is this using a cronjob every night, and after rebooting:
----
service cyrus-imapd stop
mv /var/lib/cyrus/tls_sessions.db /var/lib/cyrus/tls_sessions.db-weg
touch /var/lib/cyrus/tls_sessions.db
chown cyrus:mail /var/lib/cyrus/tls_sessions.db
service cyrus-imapd start
----

But it takes sometimes 30 minutes before I see "imapd" in "ps aux" again.

It seems to be better to use the patches from upstream, to backport the
version in testing, or to use 2.5.12 from the Debian salsa git repo.

In a few days Cyrus 3.0.8 will be in unstable/testing it's autobuilding
at the moment.

Bye,
Paul van der Vlis


-- 
Paul van der Vlis Linux systeembeheer Groningen
https://www.vandervlis.nl/


From tibbs at math.uh.edu  Fri Feb  8 16:49:20 2019
From: tibbs at math.uh.edu (Jason L Tibbitts III)
Date: Fri, 08 Feb 2019 15:49:20 -0600
Subject: Cyrus IMAP in the next Debian
In-Reply-To: <0a71caa2-8f72-d439-b162-1e6586c58817@vandervlis.nl> (Paul van
 der Vlis's message of "Fri, 8 Feb 2019 11:48:30 +0100")
References: <0a71caa2-8f72-d439-b162-1e6586c58817@vandervlis.nl>
Message-ID: <128880_1549662564_x18LnNgU043583_ufawomahrxb.fsf@epithumia.math.uh.edu>

>>>>> "PV" == Paul van der Vlis <paul at vandervlis.nl> writes:

PV> Somebody has packaged Cyrus version 3.08, but there are problems
PV> with some of the Cassandane tests.

It may be useful to see how Fedora handles Cassandane as part of its
build process.  I did a lot of work to get things functioning and get
patches pushed back upstream to make it easier to run Cassandane as part
of our package build process.  A fair bit of work to get things working
better on our less-common architectures is also in there.

That said, we do still have to disable a few Cassandane tests for
various reasons.  The Fedora specfile
(https://src.fedoraproject.org/rpms/cyrus-imapd/raw/master/f/cyrus-imapd.spec)
has explanations and information about each disabled test.  Search for
"Run the Cassandane test suite".

PV> I think it would be good if there would be more contact between the
PV> Cyrus Debian developers and the Cyrus IMAP community.

I have always find the Cyrus developers to be helpful.  Nobody had to
put me in contact with them; I just filed tickets and asked questions
here and on IRC.

 - J<

From egoitz at sarenet.es  Sat Feb  9 07:03:11 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Sat, 09 Feb 2019 13:03:11 +0100
Subject: Removing email from Xapian tier databases
Message-ID: <3c3cc51f8e04986eeda4ee62f889374a@sarenet.es>

Good morning, 

As far as I understood, for Xapian you first create it's conversation
database in order to work. Later you create database(s) for each mailbox
where Xapian can search in. You can move data between them, new mails
become indexed for instance Squatter in rolling mode... that's ok... and
understood I  think. I was wondering, what happens when mail indexed in
the archive database in removed and then does not exist any more in the
database... does Squatter rolling log manage that too?. 

By the way. I was wondering if mail gets indexed in the tier databases
(for instance in Fastmail in temp, meta, data, archine...) what's the
role or function of conversations databases you create with
ctl_conversationsdb -b -r ?. 

Cheers!

-- 

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia) 
egoitz at sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electr?nico piense si es necesario
hacerlo. 

Links:
------
[1] http://www.sarenet.es
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190209/ce4ce76e/attachment.html>

From paul at vandervlis.nl  Sat Feb  9 15:19:38 2019
From: paul at vandervlis.nl (Paul van der Vlis)
Date: Sat, 9 Feb 2019 21:19:38 +0100
Subject: Cyrus IMAP in the next Debian
In-Reply-To: <128880_1549662564_x18LnNgU043583_ufawomahrxb.fsf@epithumia.math.uh.edu>
References: <0a71caa2-8f72-d439-b162-1e6586c58817@vandervlis.nl>
 <128880_1549662564_x18LnNgU043583_ufawomahrxb.fsf@epithumia.math.uh.edu>
Message-ID: <ecaf43d1-7de4-7fe3-6f1a-1b22e62256fe@vandervlis.nl>

Op 08-02-19 om 22:49 schreef Jason L Tibbitts III:
>>>>>> "PV" == Paul van der Vlis <paul at vandervlis.nl> writes:
> 
> PV> Somebody has packaged Cyrus version 3.08, but there are problems
> PV> with some of the Cassandane tests.
> 
> It may be useful to see how Fedora handles Cassandane as part of its
> build process.  I did a lot of work to get things functioning and get
> patches pushed back upstream to make it easier to run Cassandane as part
> of our package build process.  A fair bit of work to get things working
> better on our less-common architectures is also in there.
> 
> That said, we do still have to disable a few Cassandane tests for
> various reasons.  The Fedora specfile
> (https://src.fedoraproject.org/rpms/cyrus-imapd/raw/master/f/cyrus-imapd.spec)
> has explanations and information about each disabled test.  Search for
> "Run the Cassandane test suite".

Thanks for your help. I have forwarded the information.

> PV> I think it would be good if there would be more contact between the
> PV> Cyrus Debian developers and the Cyrus IMAP community.
> 
> I have always find the Cyrus developers to be helpful.  Nobody had to
> put me in contact with them; I just filed tickets and asked questions
> here and on IRC.

I meaned the other side around. If people from the list would like to
help the Debian developpers to get a good working Cyrus in Debian.

But it lookes-like there will come an 3.0.8 in Debian, it is in
"unstable" now.

Bye,
Paul




-- 
Paul van der Vlis Linux systeembeheer Groningen
https://www.vandervlis.nl/


From brong at fastmailteam.com  Mon Feb 11 04:23:16 2019
From: brong at fastmailteam.com (Bron Gondwana)
Date: Mon, 11 Feb 2019 04:23:16 -0500
Subject: Removing email from Xapian tier databases
In-Reply-To: <3c3cc51f8e04986eeda4ee62f889374a@sarenet.es>
References: <3c3cc51f8e04986eeda4ee62f889374a@sarenet.es>
Message-ID: <4069ba6e-4dc3-47d2-8789-4f5edfafafa1@www.fastmail.com>

Conversations.db is an index over lots of interesting bits of the message, but the key part that's used by Xapian is the mapping from G key (aka: GUID, aka: sha1 of the message RFC822 data) to individual email. It's used for deduplication and for mapping from results to messages.

The data in conversations.db is added and removed in real time as messages are appended and updated in the cyrus.index.

The data in the xapian databases on the other hand is append only - so you can wind up with hits that no longer map to existing emails. The way to solve that is with a xapian repack that filters messages - which can be done using the -F flag to squatter.

Cheers,

Bron.

On Sat, Feb 9, 2019, at 23:04, Egoitz Aurrekoetxea wrote:
> Good morning,


> 


> As far as I understood, for Xapian you first create it's conversation database in order to work. Later you create database(s) for each mailbox where Xapian can search in. You can move data between them, new mails become indexed for instance Squatter in rolling mode... that's ok... and understood I think. I was wondering, what happens when mail indexed in the archive database in removed and then does not exist any more in the database... does Squatter rolling log manage that too?.


> 


> By the way. I was wondering if mail gets indexed in the tier databases (for instance in Fastmail in temp, meta, data, archine...) what's the role or function of conversations databases you create with ctl_conversationsdb -b -r ?.


> 


> Cheers!


> -- 
>  
> sarenet
> *Egoitz Aurrekoetxea*
> Departamento de sistemas
> 944 209 470
> Parque Tecnol?gico. Edificio 103
> 48170 Zamudio (Bizkaia)
> egoitz at sarenet.es
> www.sarenet.es
> 
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

--
 Bron Gondwana, CEO, FastMail Pty Ltd
 brong at fastmailteam.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190211/59d6e775/attachment.html>

From l.schimmer at cgv.tugraz.at  Mon Feb 11 04:45:26 2019
From: l.schimmer at cgv.tugraz.at (Lars Schimmer)
Date: Mon, 11 Feb 2019 10:45:26 +0100
Subject: Cyrus IMAP in the next Debian
In-Reply-To: <q3ket0$4vmo$1@blaine.gmane.org>
References: <0a71caa2-8f72-d439-b162-1e6586c58817@vandervlis.nl>
 <q3ket0$4vmo$1@blaine.gmane.org>
Message-ID: <d6178ec3-6c2d-208f-0aec-5ea6ff9f3cca@cgv.tugraz.at>

On 2/8/19 6:39 PM, Paul van der Vlis wrote:
> Op 08-02-19 om 11:48 schreef Paul van der Vlis:
>> Hello,
>>
>> The freeze for the next version of Debian is in the next week.
>>
>> Somebody has packaged Cyrus version 3.08, but there are problems with
>> some of the Cassandane tests.
>>
>> When my information is up-to-date, Debian 10 will ship with Cyrus IMAP
>> 2.5.11, with security support for 5 years. Many other distro's like
>> Ubuntu are distributing this package too.
> 
> I've heard Ond?ej Sur? has corrected and uploaded Cyrus Imap 3.0.8 to
> Debian unstable now, great! The packaging was done by Anthony Prade.
> 
> It's in the Debian autobuilder system now:
> https://buildd.debian.org/status/package.php?p=cyrus-imapd&suite=sid
> 
> In some time it will be in Debian unstable and a few days later in
> testing. https://packages.debian.org/search?keywords=cyrus-imapd

Hip Hip Hooray!
Thanks to all involved.
Will test as soon as I got time.

> Ond?ej is still asking for testers. It is still possible to make changes
> after the freeze next week. The stable release will be in a few months.
> 
> The git is here: https://salsa.debian.org/debian/cyrus-imapd
> 
> With regards,
> Paul van der Vlis
> 
> 
> 
> 
> 


MfG,
Lars Schimmer
-- 
-------------------------------------------------------------
TU Graz, Institut f?r ComputerGraphik & WissensVisualisierung
Tel: +43 316 873-5405       E-Mail: l.schimmer at cgv.tugraz.at
Fax: +43 316 873-5402       PGP-Key-ID: 0x4A9B1723



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190211/f92e588a/attachment.sig>

From Hagedorn at uni-koeln.de  Mon Feb 11 05:07:36 2019
From: Hagedorn at uni-koeln.de (Sebastian Hagedorn)
Date: Mon, 11 Feb 2019 11:07:36 +0100
Subject: Removing email from Xapian tier databases
In-Reply-To: <4069ba6e-4dc3-47d2-8789-4f5edfafafa1@www.fastmail.com>
References: <3c3cc51f8e04986eeda4ee62f889374a@sarenet.es>
 <4069ba6e-4dc3-47d2-8789-4f5edfafafa1@www.fastmail.com>
Message-ID: <396C766107CDB9C1B4785EE2@tyrion.rrz.uni-koeln.de>

Hi Bron,

--On 11. Februar 2019 um 04:23:16 -0500 Bron Gondwana 
<brong at fastmailteam.com> wrote:

> The data in conversations.db is added and removed in real time as
> messages are appended and updated in the cyrus.index.

do you know why that does not seem to happen when using the "old" sync 
protocol for replication?

<https://github.com/cyrusimap/cyrus-imapd/issues/2376>

Cheers,
Sebastian
-- 
    .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:.
                 .:.Regionales Rechenzentrum (RRZK).:.
   .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:.

From brong at fastmailteam.com  Mon Feb 11 05:16:47 2019
From: brong at fastmailteam.com (Bron Gondwana)
Date: Mon, 11 Feb 2019 05:16:47 -0500
Subject: Removing email from Xapian tier databases
In-Reply-To: <396C766107CDB9C1B4785EE2@tyrion.rrz.uni-koeln.de>
References: <3c3cc51f8e04986eeda4ee62f889374a@sarenet.es>
 <4069ba6e-4dc3-47d2-8789-4f5edfafafa1@www.fastmail.com>
 <396C766107CDB9C1B4785EE2@tyrion.rrz.uni-koeln.de>
Message-ID: <30c664c6-bf54-4550-8d68-c2b52c9c4170@www.fastmail.com>

That sounds like the source messages have no thread id, and hence they aren't being stored.

This is an interesting question actually, should we still store G keys for messages without thread identifier (CID)?

Bron.

On Mon, Feb 11, 2019, at 21:11, Sebastian Hagedorn wrote:
> Hi Bron,
> 
> --On 11. Februar 2019 um 04:23:16 -0500 Bron Gondwana 
> <brong at fastmailteam.com> wrote:
> 
> > The data in conversations.db is added and removed in real time as
> > messages are appended and updated in the cyrus.index.
> 
> do you know why that does not seem to happen when using the "old" sync 
> protocol for replication?
> 
> <https://github.com/cyrusimap/cyrus-imapd/issues/2376>
> 
> Cheers,
> Sebastian
> -- 
>  .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:.
>  .:.Regionales Rechenzentrum (RRZK).:.
>  .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:.
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

--
 Bron Gondwana, CEO, FastMail Pty Ltd
 brong at fastmailteam.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190211/b4d6e72e/attachment-0001.html>

From egoitz at sarenet.es  Mon Feb 11 05:22:09 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Mon, 11 Feb 2019 11:22:09 +0100
Subject: Removing email from Xapian tier databases
In-Reply-To: <4069ba6e-4dc3-47d2-8789-4f5edfafafa1@www.fastmail.com>
References: <3c3cc51f8e04986eeda4ee62f889374a@sarenet.es>
 <4069ba6e-4dc3-47d2-8789-4f5edfafafa1@www.fastmail.com>
Message-ID: <4b891639b7d16ae7e22d3778a36857de@sarenet.es>

Hi Bron, 

So, it would be interesting to run once a day... for instance in
cyrus.conf in events section : 

repack_xapian      cmd="squatter -F" at=0200 

Is it needed top stop the other rolling Squatter we run, in same
cyrus.conf as : 

START {
  # do not delete this entry!
  recover       cmd="ctl_cyrusdb -r"

  squatter cmd="squatter -R"
} 

Thank you so much for all the clarifications mate :) really :) 

Cheers!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia) 
egoitz at sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electr?nico piense si es necesario
hacerlo. 

El 11-02-2019 10:23, Bron Gondwana escribi?:

> Conversations.db is an index over lots of interesting bits of the message, but the key part that's used by Xapian is the mapping from G key (aka: GUID, aka: sha1 of the message RFC822 data) to individual email.  It's used for deduplication and for mapping from results to messages. 
> 
> The data in conversations.db is added and removed in real time as messages are appended and updated in the cyrus.index. 
> 
> The data in the xapian databases on the other hand is append only - so you can wind up with hits that no longer map to existing emails.  The way to solve that is with a xapian repack that filters messages - which can be done using the -F flag to squatter. 
> 
> Cheers, 
> 
> Bron. 
> 
> On Sat, Feb 9, 2019, at 23:04, Egoitz Aurrekoetxea wrote: 
> 
>> Good morning, 
>> 
>> As far as I understood, for Xapian you first create it's conversation database in order to work. Later you create database(s) for each mailbox where Xapian can search in. You can move data between them, new mails become indexed for instance Squatter in rolling mode... that's ok... and understood I  think. I was wondering, what happens when mail indexed in the archive database in removed and then does not exist any more in the database... does Squatter rolling log manage that too?. 
>> 
>> By the way. I was wondering if mail gets indexed in the tier databases (for instance in Fastmail in temp, meta, data, archine...) what's the role or function of conversations databases you create with ctl_conversationsdb -b -r ?. 
>> 
>> Cheers!
>> 
>> -- 
>> 
>> EGOITZ AURREKOETXEA 
>> Departamento de sistemas 
>> 
>> 944 209 470 
>> Parque Tecnol?gico. Edificio 103 
>> 48170 Zamudio (Bizkaia) 
>> egoitz at sarenet.es
>> 
>> www.sarenet.es [1] 
>> 
>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo. 
>> ---- 
>> Cyrus Home Page: http://www.cyrusimap.org/ 
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 
>> To Unsubscribe: 
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> -- 
> Bron Gondwana, CEO, FastMail Pty Ltd 
> brong at fastmailteam.com 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 

Links:
------
[1] http://www.sarenet.es
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190211/d69a8ecf/attachment.html>

From Hagedorn at uni-koeln.de  Mon Feb 11 05:26:16 2019
From: Hagedorn at uni-koeln.de (Sebastian Hagedorn)
Date: Mon, 11 Feb 2019 11:26:16 +0100
Subject: Removing email from Xapian tier databases
In-Reply-To: <30c664c6-bf54-4550-8d68-c2b52c9c4170@www.fastmail.com>
References: <3c3cc51f8e04986eeda4ee62f889374a@sarenet.es>
 <4069ba6e-4dc3-47d2-8789-4f5edfafafa1@www.fastmail.com>
 <396C766107CDB9C1B4785EE2@tyrion.rrz.uni-koeln.de>
 <30c664c6-bf54-4550-8d68-c2b52c9c4170@www.fastmail.com>
Message-ID: <E0B70DAD3D8002855EE72A6E@tyrion.rrz.uni-koeln.de>

So running ctl_conversationsdb -z followed by -b would assign thread ids to 
those messages? Because it works when I do that. Clearly this is an edge 
case, but IMO it should be handled somehow other than silently failing ;-)

--On 11. Februar 2019 um 05:16:47 -0500 Bron Gondwana 
<brong at fastmailteam.com> wrote:

> That sounds like the source messages have no thread id, and hence they
> aren't being stored.
>
> This is an interesting question actually, should we still store G keys
> for messages without thread identifier (CID)?
>
> Bron.
>
> On Mon, Feb 11, 2019, at 21:11, Sebastian Hagedorn wrote:
>> Hi Bron,
>>
>> --On 11. Februar 2019 um 04:23:16 -0500 Bron Gondwana
>> <brong at fastmailteam.com> wrote:
>>
>> > The data in conversations.db is added and removed in real time as
>> > messages are appended and updated in the cyrus.index.
>>
>> do you know why that does not seem to happen when using the "old" sync
>> protocol for replication?
>>
>> <https://github.com/cyrusimap/cyrus-imapd/issues/2376>
-- 
    .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:.
                 .:.Regionales Rechenzentrum (RRZK).:.
   .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:.

From egoitz at sarenet.es  Mon Feb 11 05:52:03 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Mon, 11 Feb 2019 11:52:03 +0100
Subject: Removing email from Xapian tier databases
In-Reply-To: <4b891639b7d16ae7e22d3778a36857de@sarenet.es>
References: <3c3cc51f8e04986eeda4ee62f889374a@sarenet.es>
 <4069ba6e-4dc3-47d2-8789-4f5edfafafa1@www.fastmail.com>
 <4b891639b7d16ae7e22d3778a36857de@sarenet.es>
Message-ID: <39fad7dfc1bec6070c61d757e5ceff2c@sarenet.es>

Now I'm noticing for instance, for moving data between Xapian
databases.. you need to launch something like : 

sudo -u cyrus /usr/cyrus/bin/squatter -C /usr/local/etc/imapd.conf -v -z
archive -t temp,meta,data,archive -u user/egoitz at sarenet.es

perhaps would be better to do :

sudo -u cyrus /usr/cyrus/bin/squatter -C /usr/local/etc/imapd.conf -F -v
-z archive -t temp,meta,data,archive -u user/egoitz at sarenet.es

But then, having two Squatter processes running at same time, one for
rolling mode and one for moving/repacking data, should not be an issue?.

Thanks mates!!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia) 
egoitz at sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electr?nico piense si es necesario
hacerlo. 

El 11-02-2019 11:22, Egoitz Aurrekoetxea escribi?:

> Hi Bron, 
> 
> So, it would be interesting to run once a day... for instance in cyrus.conf in events section : 
> 
> repack_xapian      cmd="squatter -F" at=0200 
> 
> Is it needed top stop the other rolling Squatter we run, in same cyrus.conf as : 
> 
> START {
> # do not delete this entry!
> recover       cmd="ctl_cyrusdb -r"
> 
> squatter cmd="squatter -R"
> } 
> 
> Thank you so much for all the clarifications mate :) really :) 
> 
> Cheers!
> 
> ---
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 944 209 470
> Parque Tecnol?gico. Edificio 103
> 48170 Zamudio (Bizkaia) 
> egoitz at sarenet.es 
> www.sarenet.es [1] 
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo. 
> 
> El 11-02-2019 10:23, Bron Gondwana escribi?: 
> Conversations.db is an index over lots of interesting bits of the message, but the key part that's used by Xapian is the mapping from G key (aka: GUID, aka: sha1 of the message RFC822 data) to individual email.  It's used for deduplication and for mapping from results to messages. 
> 
> The data in conversations.db is added and removed in real time as messages are appended and updated in the cyrus.index. 
> 
> The data in the xapian databases on the other hand is append only - so you can wind up with hits that no longer map to existing emails.  The way to solve that is with a xapian repack that filters messages - which can be done using the -F flag to squatter. 
> 
> Cheers, 
> 
> Bron. 
> 
> On Sat, Feb 9, 2019, at 23:04, Egoitz Aurrekoetxea wrote: 
> 
> Good morning, 
> 
> As far as I understood, for Xapian you first create it's conversation database in order to work. Later you create database(s) for each mailbox where Xapian can search in. You can move data between them, new mails become indexed for instance Squatter in rolling mode... that's ok... and understood I  think. I was wondering, what happens when mail indexed in the archive database in removed and then does not exist any more in the database... does Squatter rolling log manage that too?. 
> 
> By the way. I was wondering if mail gets indexed in the tier databases (for instance in Fastmail in temp, meta, data, archine...) what's the role or function of conversations databases you create with ctl_conversationsdb -b -r ?. 
> 
> Cheers!
> 
> -- 
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 
> 944 209 470 
> Parque Tecnol?gico. Edificio 103 
> 48170 Zamudio (Bizkaia) 
> egoitz at sarenet.es
> 
> www.sarenet.es [1] 
> 
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo. 
> ---- 
> Cyrus Home Page: http://www.cyrusimap.org/ 
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 
> To Unsubscribe: 
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 
> 
> -- 
> Bron Gondwana, CEO, FastMail Pty Ltd 
> brong at fastmailteam.com 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 

Links:
------
[1] http://www.sarenet.es
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190211/61e22080/attachment-0001.html>

From brong at fastmailteam.com  Mon Feb 11 09:12:14 2019
From: brong at fastmailteam.com (Bron Gondwana)
Date: Mon, 11 Feb 2019 09:12:14 -0500
Subject: Removing email from Xapian tier databases
In-Reply-To: <E0B70DAD3D8002855EE72A6E@tyrion.rrz.uni-koeln.de>
References: <3c3cc51f8e04986eeda4ee62f889374a@sarenet.es>
 <4069ba6e-4dc3-47d2-8789-4f5edfafafa1@www.fastmail.com>
 <396C766107CDB9C1B4785EE2@tyrion.rrz.uni-koeln.de>
 <30c664c6-bf54-4550-8d68-c2b52c9c4170@www.fastmail.com>
 <E0B70DAD3D8002855EE72A6E@tyrion.rrz.uni-koeln.de>
Message-ID: <5620116c-8ef3-4106-a6bb-bafda2acf801@www.fastmail.com>

Yep, it's fixed in git now, so the next release will automatically create G keys for messages, even if they don't have a threadid!

Bron.

On Mon, Feb 11, 2019, at 21:30, Sebastian Hagedorn wrote:
> So running ctl_conversationsdb -z followed by -b would assign thread ids to 
> those messages? Because it works when I do that. Clearly this is an edge 
> case, but IMO it should be handled somehow other than silently failing ;-)
> 
> --On 11. Februar 2019 um 05:16:47 -0500 Bron Gondwana 
> <brong at fastmailteam.com> wrote:
> 
> > That sounds like the source messages have no thread id, and hence they
> > aren't being stored.
> >
> > This is an interesting question actually, should we still store G keys
> > for messages without thread identifier (CID)?
> >
> > Bron.
> >
> > On Mon, Feb 11, 2019, at 21:11, Sebastian Hagedorn wrote:
> >> Hi Bron,
> >>
> >> --On 11. Februar 2019 um 04:23:16 -0500 Bron Gondwana
> >> <brong at fastmailteam.com> wrote:
> >>
> >> > The data in conversations.db is added and removed in real time as
> >> > messages are appended and updated in the cyrus.index.
> >>
> >> do you know why that does not seem to happen when using the "old" sync
> >> protocol for replication?
> >>
> >> <https://github.com/cyrusimap/cyrus-imapd/issues/2376>
> -- 
>  .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:.
>  .:.Regionales Rechenzentrum (RRZK).:.
>  .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:.
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

--
 Bron Gondwana, CEO, FastMail Pty Ltd
 brong at fastmailteam.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190211/788ad675/attachment.html>

From brong at fastmailteam.com  Mon Feb 11 09:19:51 2019
From: brong at fastmailteam.com (Bron Gondwana)
Date: Mon, 11 Feb 2019 09:19:51 -0500
Subject: Removing email from Xapian tier databases
In-Reply-To: <39fad7dfc1bec6070c61d757e5ceff2c@sarenet.es>
References: <3c3cc51f8e04986eeda4ee62f889374a@sarenet.es>
 <4069ba6e-4dc3-47d2-8789-4f5edfafafa1@www.fastmail.com>
 <4b891639b7d16ae7e22d3778a36857de@sarenet.es>
 <39fad7dfc1bec6070c61d757e5ceff2c@sarenet.es>
Message-ID: <2afdf00b-c031-4ca8-bb59-2665f0183b7c@www.fastmail.com>

It's definitely safe to have one rolling mode writing and one repacking. I wouldn't run multiple repacks in parallel, as they can wind up doing duplicate work (though the end result should always be correct and safe).

Here's what we run:

# Any time the disk gets over 50%, compress -o single down to data
13 * * * * [% INCLUDE cronjob c='/home/mod_perl/hm/scripts/xapian_compact.pl -a -o -d 50 temp data' %]
# Copy the temporary search databases down to data during the week
43 1 * * 1,2,3,4,5,6 [% INCLUDE cronjob c='/home/mod_perl/hm/scripts/xapian_compact.pl -a temp,meta data' %]
# Sundays repack the entire data directory
43 1 * * 0 [% INCLUDE cronjob c='/home/mod_perl/hm/scripts/xapian_compact.pl -a temp,meta,data data' %]
# Late on Sundays, pack any oversized data directories down to archive
0 15 * * 0 [% INCLUDE cronjob c='/home/mod_perl/hm/scripts/xapian_archive.pl -a' %]

And here's the interesting logic. In xapian_compact.pl:

 if ($Opts{d}) {
 my $Path = $Slot->SearchPath();
 my $Usage = df($Path);
 my $RunUsage = df("/run/cyrus");
 return Process::Status->new(0) if ($Usage->{per} < $Opts{d} and $RunUsage->{per} < $Opts{d});
 }

 my @args = (-z => $dest, -t => $src);
 push @args, '-v' if $Opts{v};
 push @args, '-o' if $Opts{o};
 push @args, '-F' if $Opts{F};
 push @args, '-X' if $Opts{X};
 push @args, ('-T' => $Opts{T}) if $Opts{T};
 push @args, ('-u' => $Opts{u}) if $Opts{u};
 my %RunOpts = (
 PrintOutput => 1,
 );
 $RunOpts{Nice} = 1 unless $Opts{N};
 $RunOpts{Daemon} = 1 if $Opts{D};

 $0 = "xapian_compact: $SN";
 $Slot->RunCommand(\%RunOpts, 'squatter', @args);

And in xapian_archive.pl:

my $Percent = $Opts{P} || 20;
[...]

 foreach my $user (sort keys %$DataUsage) {
 my $au = $ArchiveUsage->{$user} || 1;
 my $du = $DataUsage->{$user} || 1;
 if ($du < 5000) {
 print "Too small $user ($du)\n";
 next;
 }
 my $This = int($du * 100 / $au);
 if ($This < $Percent) {
 print "Not enough dirty $user: ($du, $au)\n";
 next;
 }
 print "Recompacting $user: ($du, $au)\n";
 my @args = (-z => 'archive', -t => 'data,archive');
[...]
 
In summary, repack data down to archive if data is more than 1/5 size of existing archive. So each of these scripts is a wrapper around squatter to help it run automatically.

Bron.


On Mon, Feb 11, 2019, at 21:55, Egoitz Aurrekoetxea wrote:
> Now I'm noticing for instance, for moving data between Xapian databases.. you need to launch something like :


> 
> sudo -u cyrus /usr/cyrus/bin/squatter -C /usr/local/etc/imapd.conf -v -z archive -t temp,meta,data,archive -u user/egoitz at sarenet.es
> 
> 
> perhaps would be better to do :
> sudo -u cyrus /usr/cyrus/bin/squatter -C /usr/local/etc/imapd.conf _*-F*_ -v -z archive -t temp,meta,data,archive -u user/egoitz at sarenet.es
> But then, having two Squatter processes running at same time, one for rolling mode and one for moving/repacking data, should not be an issue?.
> 
> 
> Thanks mates!!
> 
> ---
>  
> sarenet
> *Egoitz Aurrekoetxea*
> Departamento de sistemas
> 944 209 470
> Parque Tecnol?gico. Edificio 103
> 48170 Zamudio (Bizkaia)
> egoitz at sarenet.es
> www.sarenet.es
> 
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
> 


> El 11-02-2019 11:22, Egoitz Aurrekoetxea escribi?:


>> Hi Bron,


>> 


>> So, it would be interesting to run once a day... for instance in cyrus.conf in events section :


>> repack_xapian cmd="squatter -F" at=0200


>> Is it needed top stop the other rolling Squatter we run, in same cyrus.conf as :




>> START {
>>  # do not delete this entry!
>>  recover cmd="ctl_cyrusdb -r"
>> 
>>  squatter cmd="squatter -R"
>> }


>> 


>> Thank you so much for all the clarifications mate :) really :)


>> 


>> Cheers!


>> ---
>>  
>> sarenet
>> *Egoitz Aurrekoetxea*
>> Departamento de sistemas
>> 944 209 470
>> Parque Tecnol?gico. Edificio 103
>> 48170 Zamudio (Bizkaia)
>> egoitz at sarenet.es
>> www.sarenet.es
>> 
>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>> 


>> El 11-02-2019 10:23, Bron Gondwana escribi?:


>>> Conversations.db is an index over lots of interesting bits of the message, but the key part that's used by Xapian is the mapping from G key (aka: GUID, aka: sha1 of the message RFC822 data) to individual email. It's used for deduplication and for mapping from results to messages.
>>>  
>>> The data in conversations.db is added and removed in real time as messages are appended and updated in the cyrus.index.
>>>  
>>> The data in the xapian databases on the other hand is append only - so you can wind up with hits that no longer map to existing emails. The way to solve that is with a xapian repack that filters messages - which can be done using the -F flag to squatter.
>>>  
>>> Cheers,
>>>  
>>> Bron.
>>>  
>>> On Sat, Feb 9, 2019, at 23:04, Egoitz Aurrekoetxea wrote:
>>>> Good morning,


>>>> 


>>>> As far as I understood, for Xapian you first create it's conversation database in order to work. Later you create database(s) for each mailbox where Xapian can search in. You can move data between them, new mails become indexed for instance Squatter in rolling mode... that's ok... and understood I think. I was wondering, what happens when mail indexed in the archive database in removed and then does not exist any more in the database... does Squatter rolling log manage that too?.


>>>> 


>>>> By the way. I was wondering if mail gets indexed in the tier databases (for instance in Fastmail in temp, meta, data, archine...) what's the role or function of conversations databases you create with ctl_conversationsdb -b -r ?.


>>>> 


>>>> Cheers!


>>>> --
>>>>  
>>>> sarenet
>>>> *Egoitz Aurrekoetxea*
>>>> Departamento de sistemas
>>>> 944 209 470
>>>> Parque Tecnol?gico. Edificio 103
>>>> 48170 Zamudio (Bizkaia)
>>>> egoitz at sarenet.es
>>>> www.sarenet.es
>>>>  
>>>> Antes de imprimir este correo electr?nico piense si es necesario hacerlo.
>>>> ----
>>>> Cyrus Home Page: http://www.cyrusimap.org/
>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>>> To Unsubscribe:
>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>>  
>>> --
>>>  Bron Gondwana, CEO, FastMail Pty Ltd
>>>  brong at fastmailteam.com
>>>  
>>>  
>>> 
>>> ----
>>>  Cyrus Home Page: http://www.cyrusimap.org/
>>>  List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>>  To Unsubscribe:
>>>  https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

--
 Bron Gondwana, CEO, FastMail Pty Ltd
 brong at fastmailteam.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190211/cbef85e5/attachment-0001.html>

From egoitz at sarenet.es  Mon Feb 11 09:25:22 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Mon, 11 Feb 2019 15:25:22 +0100
Subject: Removing email from Xapian tier databases
In-Reply-To: <5620116c-8ef3-4106-a6bb-bafda2acf801@www.fastmail.com>
References: <3c3cc51f8e04986eeda4ee62f889374a@sarenet.es>
 <4069ba6e-4dc3-47d2-8789-4f5edfafafa1@www.fastmail.com>
 <396C766107CDB9C1B4785EE2@tyrion.rrz.uni-koeln.de>
 <30c664c6-bf54-4550-8d68-c2b52c9c4170@www.fastmail.com>
 <E0B70DAD3D8002855EE72A6E@tyrion.rrz.uni-koeln.de>
 <5620116c-8ef3-4106-a6bb-bafda2acf801@www.fastmail.com>
Message-ID: <281f16b654c84236ebd3b6106ee164e6@sarenet.es>

Hi mates! 

Just for finishing this thread... Two squatter proccesses then... one in
rolling mode and another one for info movement between Xapian databases
and repacking databases as Brong said... can be running without known
issues?. I say for avoid damaging something... thanks a lot mates! 

Cheers!!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia) 
egoitz at sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electr?nico piense si es necesario
hacerlo. 

El 11-02-2019 15:12, Bron Gondwana escribi?:

> Yep, it's fixed in git now, so the next release will automatically create G keys for messages, even if they don't have a threadid! 
> 
> Bron. 
> 
> On Mon, Feb 11, 2019, at 21:30, Sebastian Hagedorn wrote: 
> 
>> So running ctl_conversationsdb -z followed by -b would assign thread ids to  
>> those messages? Because it works when I do that. Clearly this is an edge  
>> case, but IMO it should be handled somehow other than silently failing ;-) 
>> 
>> --On 11. Februar 2019 um 05:16:47 -0500 Bron Gondwana  
>> <brong at fastmailteam.com> wrote: 
>> 
>>> That sounds like the source messages have no thread id, and hence they 
>>> aren't being stored. 
>>> 
>>> This is an interesting question actually, should we still store G keys 
>>> for messages without thread identifier (CID)? 
>>> 
>>> Bron. 
>>> 
>>> On Mon, Feb 11, 2019, at 21:11, Sebastian Hagedorn wrote: 
>>>> Hi Bron, 
>>>> 
>>>> --On 11. Februar 2019 um 04:23:16 -0500 Bron Gondwana 
>>>> <brong at fastmailteam.com> wrote: 
>>>> 
>>>>> The data in conversations.db is added and removed in real time as 
>>>>> messages are appended and updated in the cyrus.index. 
>>>> 
>>>> do you know why that does not seem to happen when using the "old" sync 
>>>> protocol for replication? 
>>>> 
>>>> <https://github.com/cyrusimap/cyrus-imapd/issues/2376> 
>> --  
>> .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:. 
>> .:.Regionales Rechenzentrum (RRZK).:. 
>> .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:. 
>> ---- 
>> Cyrus Home Page: http://www.cyrusimap.org/ 
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 
>> To Unsubscribe: 
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> -- 
> Bron Gondwana, CEO, FastMail Pty Ltd 
> brong at fastmailteam.com 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 

Links:
------
[1] http://www.sarenet.es
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190211/85be5cb7/attachment.html>

From egoitz at sarenet.es  Mon Feb 11 09:26:08 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Mon, 11 Feb 2019 15:26:08 +0100
Subject: Removing email from Xapian tier databases
In-Reply-To: <2afdf00b-c031-4ca8-bb59-2665f0183b7c@www.fastmail.com>
References: <3c3cc51f8e04986eeda4ee62f889374a@sarenet.es>
 <4069ba6e-4dc3-47d2-8789-4f5edfafafa1@www.fastmail.com>
 <4b891639b7d16ae7e22d3778a36857de@sarenet.es>
 <39fad7dfc1bec6070c61d757e5ceff2c@sarenet.es>
 <2afdf00b-c031-4ca8-bb59-2665f0183b7c@www.fastmail.com>
Message-ID: <93b5a6ac41a6ce6cf6882b463065afa1@sarenet.es>

Many many many many thanks a lot Brong!!!!!! :) :) :) :) :) :) :)

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia) 
egoitz at sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electr?nico piense si es necesario
hacerlo. 

El 11-02-2019 15:19, Bron Gondwana escribi?:

> It's definitely safe to have one rolling mode writing and one repacking.  I wouldn't run multiple repacks in parallel, as they can wind up doing duplicate work (though the end result should always be correct and safe). 
> 
> Here's what we run: 
> 
> # Any time the disk gets over 50%, compress -o single down to data 
> 13 *  * * * [% INCLUDE cronjob c='/home/mod_perl/hm/scripts/xapian_compact.pl -a -o -d 50 temp data' %] 
> # Copy the temporary search databases down to data during the week 
> 43 1  * * 1,2,3,4,5,6 [% INCLUDE cronjob c='/home/mod_perl/hm/scripts/xapian_compact.pl -a temp,meta data' %] 
> # Sundays repack the entire data directory 
> 43 1  * * 0 [% INCLUDE cronjob c='/home/mod_perl/hm/scripts/xapian_compact.pl -a temp,meta,data data' %] 
> # Late on Sundays, pack any oversized data directories down to archive 
> 0 15 * * 0 [% INCLUDE cronjob c='/home/mod_perl/hm/scripts/xapian_archive.pl -a' %] 
> 
> And here's the interesting logic.  In xapian_compact.pl: 
> 
> if ($Opts{d}) {
> 
> my $Path = $Slot->SearchPath();
> 
> my $Usage = df($Path);
> 
> my $RunUsage = df("/run/cyrus");
> 
> return Process::Status->new(0) if ($Usage->{per} < $Opts{d} and $RunUsage->{per} < $Opts{d});
> 
> }
> 
> my @args = (-z => $dest, -t => $src);
> 
> push @args, '-v' if $Opts{v};
> 
> push @args, '-o' if $Opts{o};
> 
> push @args, '-F' if $Opts{F};
> 
> push @args, '-X' if $Opts{X};
> 
> push @args, ('-T' => $Opts{T}) if $Opts{T};
> 
> push @args, ('-u' => $Opts{u}) if $Opts{u};
> 
> my %RunOpts = (
> 
> PrintOutput => 1,
> 
> );
> 
> $RunOpts{Nice} = 1 unless $Opts{N};
> 
> $RunOpts{Daemon} = 1 if $Opts{D};
> 
> $0 = "xapian_compact: $SN";
> 
> $Slot->RunCommand(\%RunOpts, 'squatter', @args);
> 
> And in xapian_archive.pl: 
> 
> my $Percent = $Opts{P} || 20; 
> [...]
> 
> foreach my $user (sort keys %$DataUsage) {
> 
> my $au = $ArchiveUsage->{$user} || 1;
> 
> my $du = $DataUsage->{$user} || 1;
> 
> if ($du < 5000) {
> 
> print "Too small $user ($du)\n";
> 
> next;
> 
> }
> 
> my $This = int($du * 100 / $au);
> 
> if ($This < $Percent) {
> 
> print "Not enough dirty $user: ($du, $au)\n";
> 
> next;
> 
> }
> 
> print "Recompacting $user: ($du, $au)\n";
> 
> my @args = (-z => 'archive', -t => 'data,archive'); 
> [...]
> 
> In summary, repack data down to archive if data is more than 1/5 size of existing archive.  So each of these scripts is a wrapper around squatter to help it run automatically. 
> 
> Bron. 
> 
> On Mon, Feb 11, 2019, at 21:55, Egoitz Aurrekoetxea wrote: 
> 
> Now I'm noticing for instance, for moving data between Xapian databases.. you need to launch something like :
> 
> sudo -u cyrus /usr/cyrus/bin/squatter -C /usr/local/etc/imapd.conf -v -z archive -t temp,meta,data,archive -u user/egoitz at sarenet.es 
> 
> perhaps would be better to do : 
> sudo -u cyrus /usr/cyrus/bin/squatter -C /usr/local/etc/imapd.conf -F -v -z archive -t temp,meta,data,archive -u user/egoitz at sarenet.es 
> But then, having two Squatter processes running at same time, one for rolling mode and one for moving/repacking data, should not be an issue?. 
> 
> Thanks mates!! 
> 
> --- 
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 
> 944 209 470 
> Parque Tecnol?gico. Edificio 103 
> 48170 Zamudio (Bizkaia) 
> egoitz at sarenet.es
> 
> www.sarenet.es [1] 
> 
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo. 
> 
> El 11-02-2019 11:22, Egoitz Aurrekoetxea escribi?: 
> 
> Hi Bron, 
> 
> So, it would be interesting to run once a day... for instance in cyrus.conf in events section : 
> 
> repack_xapian      cmd="squatter -F" at=0200 
> 
> Is it needed top stop the other rolling Squatter we run, in same cyrus.conf as : 
> 
> START { 
> # do not delete this entry! 
> recover       cmd="ctl_cyrusdb -r" 
> 
> squatter cmd="squatter -R" 
> } 
> 
> Thank you so much for all the clarifications mate :) really :) 
> 
> Cheers!
> 
> --- 
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 
> 944 209 470 
> Parque Tecnol?gico. Edificio 103 
> 48170 Zamudio (Bizkaia) 
> egoitz at sarenet.es
> 
> www.sarenet.es [1] 
> 
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo. 
> 
> El 11-02-2019 10:23, Bron Gondwana escribi?: 
> Conversations.db is an index over lots of interesting bits of the message, but the key part that's used by Xapian is the mapping from G key (aka: GUID, aka: sha1 of the message RFC822 data) to individual email.  It's used for deduplication and for mapping from results to messages. 
> 
> The data in conversations.db is added and removed in real time as messages are appended and updated in the cyrus.index. 
> 
> The data in the xapian databases on the other hand is append only - so you can wind up with hits that no longer map to existing emails.  The way to solve that is with a xapian repack that filters messages - which can be done using the -F flag to squatter. 
> 
> Cheers, 
> 
> Bron. 
> 
> On Sat, Feb 9, 2019, at 23:04, Egoitz Aurrekoetxea wrote: 
> 
> Good morning, 
> 
> As far as I understood, for Xapian you first create it's conversation database in order to work. Later you create database(s) for each mailbox where Xapian can search in. You can move data between them, new mails become indexed for instance Squatter in rolling mode... that's ok... and understood I  think. I was wondering, what happens when mail indexed in the archive database in removed and then does not exist any more in the database... does Squatter rolling log manage that too?. 
> 
> By the way. I was wondering if mail gets indexed in the tier databases (for instance in Fastmail in temp, meta, data, archine...) what's the role or function of conversations databases you create with ctl_conversationsdb -b -r ?. 
> 
> Cheers!
> 
> -- 
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 
> 944 209 470 
> Parque Tecnol?gico. Edificio 103 
> 48170 Zamudio (Bizkaia) 
> egoitz at sarenet.es
> 
> www.sarenet.es [1] 
> 
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo. 
> ---- 
> Cyrus Home Page: http://www.cyrusimap.org/ 
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 
> To Unsubscribe: 
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 
> 
> -- 
> Bron Gondwana, CEO, FastMail Pty Ltd 
> brong at fastmailteam.com 
> 
> ---- 
> Cyrus Home Page: http://www.cyrusimap.org/ 
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 
> To Unsubscribe: 
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

---- 
Cyrus Home Page: http://www.cyrusimap.org/ 
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 
To Unsubscribe: 
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 

-- 
  Bron Gondwana, CEO, FastMail Pty Ltd 
  brong at fastmailteam.com 

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 

Links:
------
[1] http://www.sarenet.es
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190211/041f3193/attachment-0001.html>

From Hagedorn at uni-koeln.de  Mon Feb 11 10:17:51 2019
From: Hagedorn at uni-koeln.de (Sebastian Hagedorn)
Date: Mon, 11 Feb 2019 16:17:51 +0100
Subject: Removing email from Xapian tier databases
In-Reply-To: <5620116c-8ef3-4106-a6bb-bafda2acf801@www.fastmail.com>
References: <3c3cc51f8e04986eeda4ee62f889374a@sarenet.es>
 <4069ba6e-4dc3-47d2-8789-4f5edfafafa1@www.fastmail.com>
 <396C766107CDB9C1B4785EE2@tyrion.rrz.uni-koeln.de>
 <30c664c6-bf54-4550-8d68-c2b52c9c4170@www.fastmail.com>
 <E0B70DAD3D8002855EE72A6E@tyrion.rrz.uni-koeln.de>
 <5620116c-8ef3-4106-a6bb-bafda2acf801@www.fastmail.com>
Message-ID: <3725AB5C60D0C6DD83D8A84C@tyrion.rrz.uni-koeln.de>

Thanks! I rolled my own RPM with that patch, and I can confirm that it 
works.

--On 11. Februar 2019 um 09:12:14 -0500 Bron Gondwana 
<brong at fastmailteam.com> wrote:

> Yep, it's fixed in git now, so the next release will automatically create
> G keys for messages, even if they don't have a threadid!
>
> Bron.
>
> On Mon, Feb 11, 2019, at 21:30, Sebastian Hagedorn wrote:
>> So running ctl_conversationsdb -z followed by -b would assign thread ids
>> to  those messages? Because it works when I do that. Clearly this is an
>> edge  case, but IMO it should be handled somehow other than silently
>> failing ;-)
>>
>> --On 11. Februar 2019 um 05:16:47 -0500 Bron Gondwana
>> <brong at fastmailteam.com> wrote:
>>
>> > That sounds like the source messages have no thread id, and hence they
>> > aren't being stored.
>> >
>> > This is an interesting question actually, should we still store G keys
>> > for messages without thread identifier (CID)?
>> >
>> > Bron.
>> >
>> > On Mon, Feb 11, 2019, at 21:11, Sebastian Hagedorn wrote:
>> >> Hi Bron,
>> >>
>> >> --On 11. Februar 2019 um 04:23:16 -0500 Bron Gondwana
>> >> <brong at fastmailteam.com> wrote:
>> >>
>> >> > The data in conversations.db is added and removed in real time as
>> >> > messages are appended and updated in the cyrus.index.
>> >>
>> >> do you know why that does not seem to happen when using the "old" sync
>> >> protocol for replication?
>> >>
>> >> <https://github.com/cyrusimap/cyrus-imapd/issues/2376>
>> --
-- 
    .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:.
                 .:.Regionales Rechenzentrum (RRZK).:.
   .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:.

From paul at vandervlis.nl  Mon Feb 11 11:18:13 2019
From: paul at vandervlis.nl (Paul van der Vlis)
Date: Mon, 11 Feb 2019 17:18:13 +0100
Subject: Cyrus IMAP in the next Debian
In-Reply-To: <d6178ec3-6c2d-208f-0aec-5ea6ff9f3cca@cgv.tugraz.at>
References: <0a71caa2-8f72-d439-b162-1e6586c58817@vandervlis.nl>
 <q3ket0$4vmo$1@blaine.gmane.org>
 <d6178ec3-6c2d-208f-0aec-5ea6ff9f3cca@cgv.tugraz.at>
Message-ID: <22aa7dbd-99ab-1ef6-18c2-546a7744269d@vandervlis.nl>

Op 11-02-19 om 10:45 schreef Lars Schimmer:
> On 2/8/19 6:39 PM, Paul van der Vlis wrote:

>> In some time it will be in Debian unstable and a few days later in
>> testing. https://packages.debian.org/search?keywords=cyrus-imapd
> 
> Hip Hip Hooray!
> Thanks to all involved.
> Will test as soon as I got time.

I have installed it, and my first impression is that everything works fine!

With regards,
Paul van der Vlis




-- 
Paul van der Vlis Linux systeembeheer Groningen
https://www.vandervlis.nl/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190211/1c1611a2/attachment.sig>

From brong at fastmailteam.com  Mon Feb 11 11:40:46 2019
From: brong at fastmailteam.com (Bron Gondwana)
Date: Mon, 11 Feb 2019 11:40:46 -0500
Subject: Removing email from Xapian tier databases
In-Reply-To: <3725AB5C60D0C6DD83D8A84C@tyrion.rrz.uni-koeln.de>
References: <3c3cc51f8e04986eeda4ee62f889374a@sarenet.es>
 <4069ba6e-4dc3-47d2-8789-4f5edfafafa1@www.fastmail.com>
 <396C766107CDB9C1B4785EE2@tyrion.rrz.uni-koeln.de>
 <30c664c6-bf54-4550-8d68-c2b52c9c4170@www.fastmail.com>
 <E0B70DAD3D8002855EE72A6E@tyrion.rrz.uni-koeln.de>
 <5620116c-8ef3-4106-a6bb-bafda2acf801@www.fastmail.com>
 <3725AB5C60D0C6DD83D8A84C@tyrion.rrz.uni-koeln.de>
Message-ID: <3851565d-08b4-4c78-ab7a-5be8151beaa6@www.fastmail.com>

Excellent - I hope you grabbed both that commit and the one afterwards where I fixed the order of CID parsing.

Actually, it might not be as big a deal on 3.0, but not calculating the CID first did break one JMAP case on future.

Cheers,

Bron.

On Tue, Feb 12, 2019, at 02:22, Sebastian Hagedorn wrote:
> Thanks! I rolled my own RPM with that patch, and I can confirm that it 
> works.
> 
> --On 11. Februar 2019 um 09:12:14 -0500 Bron Gondwana 
> <brong at fastmailteam.com> wrote:
> 
> > Yep, it's fixed in git now, so the next release will automatically create
> > G keys for messages, even if they don't have a threadid!
> >
> > Bron.
> >
> > On Mon, Feb 11, 2019, at 21:30, Sebastian Hagedorn wrote:
> >> So running ctl_conversationsdb -z followed by -b would assign thread ids
> >> to those messages? Because it works when I do that. Clearly this is an
> >> edge case, but IMO it should be handled somehow other than silently
> >> failing ;-)
> >>
> >> --On 11. Februar 2019 um 05:16:47 -0500 Bron Gondwana
> >> <brong at fastmailteam.com> wrote:
> >>
> >> > That sounds like the source messages have no thread id, and hence they
> >> > aren't being stored.
> >> >
> >> > This is an interesting question actually, should we still store G keys
> >> > for messages without thread identifier (CID)?
> >> >
> >> > Bron.
> >> >
> >> > On Mon, Feb 11, 2019, at 21:11, Sebastian Hagedorn wrote:
> >> >> Hi Bron,
> >> >>
> >> >> --On 11. Februar 2019 um 04:23:16 -0500 Bron Gondwana
> >> >> <brong at fastmailteam.com> wrote:
> >> >>
> >> >> > The data in conversations.db is added and removed in real time as
> >> >> > messages are appended and updated in the cyrus.index.
> >> >>
> >> >> do you know why that does not seem to happen when using the "old" sync
> >> >> protocol for replication?
> >> >>
> >> >> <https://github.com/cyrusimap/cyrus-imapd/issues/2376>
> >> --
> -- 
>  .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:.
>  .:.Regionales Rechenzentrum (RRZK).:.
>  .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:.
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

--
 Bron Gondwana, CEO, FastMail Pty Ltd
 brong at fastmailteam.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190211/94608837/attachment-0001.html>

From max at tcen.ru  Tue Feb 12 00:35:32 2019
From: max at tcen.ru (Max Kosmach)
Date: Tue, 12 Feb 2019 08:35:32 +0300
Subject: cyrus-imapd not starting after upgrade
In-Reply-To: <1400e6b1-7fa6-5064-0aa3-19e41846ff5a@vandervlis.nl>
References: <2608b63d-2149-0b33-a261-0739baef0992@gmx.net>
 <1400e6b1-7fa6-5064-0aa3-19e41846ff5a@vandervlis.nl>
Message-ID: <8DE6D2C3-5987-4E1D-993F-E3F76E533E73@tcen.ru>

Hi!

As far as I remember You should rebuild 2.5.20-3 package with at least those patches
https://github.com/cyrusimap/cyrus-imapd/commit/1e7cf05a6bcba8c639fe8222c28851b1a9802e63
https://github.com/cyrusimap/cyrus-imapd/commit/575b6b4503a5566115a6259f98c812b2b465f8b6


8 ??????? 2019 ?. 22:46:00 GMT+03:00, Paul van der Vlis <paul at vandervlis.nl> ?????:
>Op 15-01-19 om 17:33 schreef Daniel Bareiro:
>> Hi all!
>> 
>> After quite some time, today I decided to update the mail server from
>> Debian Jessie (cyrus-imapd 2.4.17) to Debian Stretch (cyrus-imapd
>2.5.10-3).
>Hello Daniel,
>
>I use cyrus-imapd 2.5.10-3 from Debian stable on serveral machines and
>I
>can tell you that it gives much TLS problems.
>
>What I do is this using a cronjob every night, and after rebooting:
>----
>service cyrus-imapd stop
>mv /var/lib/cyrus/tls_sessions.db /var/lib/cyrus/tls_sessions.db-weg
>touch /var/lib/cyrus/tls_sessions.db
>chown cyrus:mail /var/lib/cyrus/tls_sessions.db
>service cyrus-imapd start
>----
>
>But it takes sometimes 30 minutes before I see "imapd" in "ps aux"
>again.
>
>It seems to be better to use the patches from upstream, to backport the
>version in testing, or to use 2.5.12 from the Debian salsa git repo.
>
>In a few days Cyrus 3.0.8 will be in unstable/testing it's autobuilding
>at the moment.
>
>Bye,
>Paul van der Vlis
>
>
>-- 
>Paul van der Vlis Linux systeembeheer Groningen
>https://www.vandervlis.nl/
>
>----
>Cyrus Home Page: http://www.cyrusimap.org/
>List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>To Unsubscribe:
>https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
Max Kosmach
-- 
Max Kosmach
-- 
Max Kosmach
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190212/51d94648/attachment.html>

From Hagedorn at uni-koeln.de  Tue Feb 12 05:47:50 2019
From: Hagedorn at uni-koeln.de (Sebastian Hagedorn)
Date: Tue, 12 Feb 2019 11:47:50 +0100
Subject: Removing email from Xapian tier databases
In-Reply-To: <3851565d-08b4-4c78-ab7a-5be8151beaa6@www.fastmail.com>
References: <3c3cc51f8e04986eeda4ee62f889374a@sarenet.es>
 <4069ba6e-4dc3-47d2-8789-4f5edfafafa1@www.fastmail.com>
 <396C766107CDB9C1B4785EE2@tyrion.rrz.uni-koeln.de>
 <30c664c6-bf54-4550-8d68-c2b52c9c4170@www.fastmail.com>
 <E0B70DAD3D8002855EE72A6E@tyrion.rrz.uni-koeln.de>
 <5620116c-8ef3-4106-a6bb-bafda2acf801@www.fastmail.com>
 <3725AB5C60D0C6DD83D8A84C@tyrion.rrz.uni-koeln.de>
 <3851565d-08b4-4c78-ab7a-5be8151beaa6@www.fastmail.com>
Message-ID: <FA6F4AD7627725608FE7702F@tyrion.rrz.uni-koeln.de>

No, I hadn't, but I have now ? just in case.

Thanks, Sebastian

--On 11. Februar 2019 um 11:40:46 -0500 Bron Gondwana 
<brong at fastmailteam.com> wrote:

> Excellent - I hope you grabbed both that commit and the one afterwards
> where I fixed the order of CID parsing.
>
> Actually, it might not be as big a deal on 3.0, but not calculating the
> CID first did break one JMAP case on future.
>
> On Tue, Feb 12, 2019, at 02:22, Sebastian Hagedorn wrote:
>> Thanks! I rolled my own RPM with that patch, and I can confirm that it
>> works.
>>
>> --On 11. Februar 2019 um 09:12:14 -0500 Bron Gondwana
>> <brong at fastmailteam.com> wrote:
>>
>> > Yep, it's fixed in git now, so the next release will automatically
>> > create G keys for messages, even if they don't have a threadid!
-- 
    .:.Sebastian Hagedorn - Weyertal 121 (Geb?ude 133), Zimmer 2.02.:.
                 .:.Regionales Rechenzentrum (RRZK).:.
   .:.Universit?t zu K?ln / Cologne University - ? +49-221-470-89578.:.

From daniel-listas at gmx.net  Tue Feb 12 08:29:21 2019
From: daniel-listas at gmx.net (Daniel Bareiro)
Date: Tue, 12 Feb 2019 10:29:21 -0300
Subject: cyrus-imapd not starting after upgrade
In-Reply-To: <1400e6b1-7fa6-5064-0aa3-19e41846ff5a@vandervlis.nl>
References: <2608b63d-2149-0b33-a261-0739baef0992@gmx.net>
 <1400e6b1-7fa6-5064-0aa3-19e41846ff5a@vandervlis.nl>
Message-ID: <d1e264ec-2daa-4c4c-aca2-29ce3caa2693@gmx.net>


On 8/2/19 16:46, Paul van der Vlis wrote:

>> After quite some time, today I decided to update the mail server from
>> Debian Jessie (cyrus-imapd 2.4.17) to Debian Stretch (cyrus-imapd 2.5.10-3).

> Hello Daniel,

Hello, Paul.

> I use cyrus-imapd 2.5.10-3 from Debian stable on serveral machines and I
> can tell you that it gives much TLS problems.
> 
> What I do is this using a cronjob every night, and after rebooting:
> ----
> service cyrus-imapd stop
> mv /var/lib/cyrus/tls_sessions.db /var/lib/cyrus/tls_sessions.db-weg
> touch /var/lib/cyrus/tls_sessions.db
> chown cyrus:mail /var/lib/cyrus/tls_sessions.db
> service cyrus-imapd start
> ----
> 
> But it takes sometimes 30 minutes before I see "imapd" in "ps aux" again.
> 
> It seems to be better to use the patches from upstream, to backport the
> version in testing, or to use 2.5.12 from the Debian salsa git repo.
> 
> In a few days Cyrus 3.0.8 will be in unstable/testing it's autobuilding
> at the moment.

Thank you for sharing this. I don't remember seeing errors with TLS in
the syslog. Could you share some syslog entries to check? From what I
see, for some reason you create a new tls_sessions.db file.

Have you opened a bug in Debian BTS about this?

The problem that I am observing after the update is the following at the
time of trying to deliver each mail:

----------
[/var/run/cyrus/socket/lmtp]: Permission denied
----------

The delivery is normalized after executing this command:

----------
# dpkg-statoverride --force --update --add cyrus lmtp 750
/var/run/cyrus/socket
----------

But I have noticed that after doing a reboot I have this problem again.
Any idea what could be a definitive solution?

Kind regards,
Daniel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190212/a8d1c9d1/attachment.sig>

From michael.menge at zdv.uni-tuebingen.de  Tue Feb 12 08:39:10 2019
From: michael.menge at zdv.uni-tuebingen.de (Michael Menge)
Date: Tue, 12 Feb 2019 14:39:10 +0100
Subject: another problem with conversations db
In-Reply-To: <bfac9abf-ad3b-42ad-a668-0e57961d5a2b@www.fastmail.com>
References: <20190124180232.Horde.Tya11Hrse_nHFrp7Zj39dP4@webmail.uni-tuebingen.de>
 <363a2149-1b36-4dc0-aca5-b121c8ae3266@beta.fastmail.com>
 <20190125095538.Horde.BZLLa6BdReb8ZHa6b9Vxg1r@webmail.uni-tuebingen.de>
 <5d089935-a59f-45d7-a50c-cec40e23082c@beta.fastmail.com>
 <20190131093559.Horde.Vu9F7oPMtLNy30qgYXzWgKL@webmail.uni-tuebingen.de>
 <c02770dd-4238-4aab-a034-bd447b58cd06@www.fastmail.com>
 <20190204101712.Horde.33ManGeIHA8D5TVzSiQ7825@webmail.uni-tuebingen.de>
 <02bb719c-27ce-4e62-84e8-f11a32adb3dc@www.fastmail.com>
 <20190204120013.Horde.6MsNbwrJRdIBADBt9I1z3QP@webmail.uni-tuebingen.de>
 <bfac9abf-ad3b-42ad-a668-0e57961d5a2b@www.fastmail.com>
Message-ID: <20190212143910.Horde.xEe2bT5UT3887Oel1PIjtTT@webmail.uni-tuebingen.de>

Hi Bron,

sorry, i had to rearrange some quotes to put them my answers in a more  
meaningful order.


Quoting Bron Gondwana <brong at fastmailteam.com>:

> On Mon, Feb 4, 2019, at 22:00, Michael Menge wrote:
>>
>> Quoting Bron Gondwana <brong at fastmail.fm>:
>>
>> > On Mon, Feb 4, 2019, at 20:21, Michael Menge wrote:
>> >>

>> >> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening
>> >> /srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory
>> >> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening
>> >> /srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory
>> >> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR archive
>> >> user.XXXX 2185 failed to copyfile
>> >> (/srv/cyrus-be/ssd-part/L/user/XXXX/2185. =>
>> >> /srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.): Unknown code
>> >> ____ 255
>> >
>> >
>> > Ouch. Yeah, that could have been caused by a bug in delivery, and
>> > would definitely cause conversations DB corruption if the index file
>> > was updated but the conversations DB wasn't or vice versa.
>> >
>> >> The file was already at  
>> /srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.

I was able to fix these problems with reconstruct, and the didn't  
reappear till now.
Also there where other accounts which had IOERRORS regarding the  
conversation db,
with no cyr_expire archive errors, so i believe that these problems  
are not related.

I tried rebuilding the conversation db for the accounts with errors,  
but some other
accounts will show up with errors some time later. I counldn't find a  
some thing in
common jet.


>> >> > Anyway, I don't think that would break anything.
>> >> >
>> >> > metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part
>> >> > metapartition_files: header index cache expunge squat annotations
>> >> > lock dav archivecache
>> >> >
>> >> > Ooh, I haven't tested having cache and archivecache on the same
>> >> > location. That's really interesting. Again, I'd be in favour of
>> >> > separation here, give them different paths. That might be tricky
>> >> > with ssd though, the way this is laid out. I assume you have some
>> >> > kind of symlink farm going on?
>> >> >
>> >>
>> >> I didn't know that there could be a problem with cache and archivecache.
>> >> At the time we decided on the configuration for cyrus 3.0 I looked at the
>> >> imapd.conf man page and for metapartition_files decided that I want all
>> >> meta files on the ssd storage. There was no indication in the man page
>> >> that there could be a problem.
>> >
>> > Fair. I'd have to test that to see if it works correctly. I would
>> > hope so, but I haven't tested that configuration. This is the
>> > downside with having lots of different ways to do things!
>> >
>> >> How do I separate location of archivecache from the other
>> >> metapartition path?
>> >> And fix the cache and archivecache files?
>> >
>> > This I don't know a good answer for. I will test if having the same
>> > path for cache and archivecache could fail. I THINK that I made the
>> > code safe for it, but I'm not sure that it's been tested.
>> >
>> >> No there is no sysmlink farm. We have mounted different iSCSI volumes to
>> >> /srv/cyrus-ssd-be, /srv/cyrus-hdd-be and /srv/cyrus-be
>> >
>> > Right. That makes sense.

Did you have time to look into the cache/archivecache situation jet?


>> > Right! I do wonder if there are some bugs in 3.0.x which are fixed
>> > on master around delivery to archive partition. We definitely had
>> > bugs on master, but I thought they were newly introduced on master
>> > as well, which is why the fixes weren't backported. But if you're
>> > having files be in the wrong location, maybe there are bugs on 3.0.x
>> > as well.

Are all fixes from master backported to 3.0?

Is the new Commit "I will try your new commits regarding CID" related to the
"IOERROR: conversations_audit on load:" and "IOERROR:  
conversations_audit on store"?

I will try your new commits in the next days on my test servers to sea  
if the fix
the endless loop in ctl_conversationsdb I have seen for some accounts.

Quoting myself (Re: prblems rebuilding conversations db) Jan 24, 2019

> The program loops in build_cid_cb (imap/ctl_conversationsdb.c:189)
>
> For the problematic mailbox that I tested, for every message
> record->cid was NULLCONVERSATION, so mailbox_cacherecord,
> message_update_conversations and mailbox_rewrite_index_record
> where called, each returned 0.
>
> After iterating trough all messages in the mailbox count was > 0, and r==0,
> so the while condition (!r && count) was true for the next run.
> At this point record->cid was still NULLCONVERSATION for every message,
> which I guess should not be the case.

Michael

--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universit?t T?bingen                   Fax.: (49) 7071/29-5912
Zentrum f?r Datenverarbeitung          mail:  
michael.menge at zdv.uni-tuebingen.de
W?chterstra?e 76
72074 T?bingen


From lists at localguru.de  Wed Feb 13 03:01:06 2019
From: lists at localguru.de (Marcus Schopen)
Date: Wed, 13 Feb 2019 09:01:06 +0100
Subject: disk space used by a mailbox without expunged
Message-ID: <ce4afb3e785a8e35d6dfd692d25713b3509b8f42.camel@localguru.de>

Hi,

is there a way to count the disk space used by a mailbox without
expunged messages?

Ciao
M.



From michael.menge at zdv.uni-tuebingen.de  Wed Feb 13 03:17:26 2019
From: michael.menge at zdv.uni-tuebingen.de (Michael Menge)
Date: Wed, 13 Feb 2019 09:17:26 +0100
Subject: disk space used by a mailbox without expunged
In-Reply-To: <ce4afb3e785a8e35d6dfd692d25713b3509b8f42.camel@localguru.de>
Message-ID: <20190213091726.Horde.w-lvGgE-YxwYOHG-QoBax6M@webmail.uni-tuebingen.de>

Hi Marcus,

Quoting Marcus Schopen <lists at localguru.de>:

> Hi,
>
> is there a way to count the disk space used by a mailbox without
> expunged messages?
>

mbexamine user/LoginID | grep Size

on older cyrus versions (2.3 and 2.4) mbexamine did examine all  
subfolders as well
in 3.0 only the info for the given folder is shown



--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universit?t T?bingen                   Fax.: (49) 7071/29-5912
Zentrum f?r Datenverarbeitung          mail:  
michael.menge at zdv.uni-tuebingen.de
W?chterstra?e 76
72074 T?bingen


From brong at fastmailteam.com  Wed Feb 13 04:37:22 2019
From: brong at fastmailteam.com (Bron Gondwana)
Date: Wed, 13 Feb 2019 04:37:22 -0500
Subject: another problem with conversations db
In-Reply-To: <20190212143910.Horde.xEe2bT5UT3887Oel1PIjtTT@webmail.uni-tuebingen.de>
References: <20190124180232.Horde.Tya11Hrse_nHFrp7Zj39dP4@webmail.uni-tuebingen.de>
 <363a2149-1b36-4dc0-aca5-b121c8ae3266@beta.fastmail.com>
 <20190125095538.Horde.BZLLa6BdReb8ZHa6b9Vxg1r@webmail.uni-tuebingen.de>
 <5d089935-a59f-45d7-a50c-cec40e23082c@beta.fastmail.com>
 <20190131093559.Horde.Vu9F7oPMtLNy30qgYXzWgKL@webmail.uni-tuebingen.de>
 <c02770dd-4238-4aab-a034-bd447b58cd06@www.fastmail.com>
 <20190204101712.Horde.33ManGeIHA8D5TVzSiQ7825@webmail.uni-tuebingen.de>
 <02bb719c-27ce-4e62-84e8-f11a32adb3dc@www.fastmail.com>
 <20190204120013.Horde.6MsNbwrJRdIBADBt9I1z3QP@webmail.uni-tuebingen.de>
 <bfac9abf-ad3b-42ad-a668-0e57961d5a2b@www.fastmail.com>
 <20190212143910.Horde.xEe2bT5UT3887Oel1PIjtTT@webmail.uni-tuebingen.de>
Message-ID: <c41ab3d3-f6cf-43d4-8127-98253338f701@www.fastmail.com>

On Wed, Feb 13, 2019, at 00:39, Michael Menge wrote:
> Hi Bron,
> 
> sorry, i had to rearrange some quotes to put them my answers in a more 
> meaningful order.
> 
> 
> Quoting Bron Gondwana <brong at fastmailteam.com>:
> 
> >> >> The file was already at 
> >> /srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.
> 
> I was able to fix these problems with reconstruct, and the didn't 
> reappear till now.
> Also there where other accounts which had IOERRORS regarding the 
> conversation db,
> with no cyr_expire archive errors, so i believe that these problems 
> are not related.
> 
> I tried rebuilding the conversation db for the accounts with errors, 
> but some other
> accounts will show up with errors some time later. I counldn't find a 
> some thing in
> common jet.

OK. That's hard to disagnore from remotely :(

> 
> >> >> > Anyway, I don't think that would break anything.
> >> >> >
> >> >> > metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part
> >> >> > metapartition_files: header index cache expunge squat annotations
> >> >> > lock dav archivecache
> >> >> >
> >> >> > Ooh, I haven't tested having cache and archivecache on the same
> >> >> > location. That's really interesting. Again, I'd be in favour of
> >> >> > separation here, give them different paths. That might be tricky
> >> >> > with ssd though, the way this is laid out. I assume you have some
> >> >> > kind of symlink farm going on?
> >> >> >
> >> >>
> >> >> I didn't know that there could be a problem with cache and archivecache.
> >> >> At the time we decided on the configuration for cyrus 3.0 I looked at the
> >> >> imapd.conf man page and for metapartition_files decided that I want all
> >> >> meta files on the ssd storage. There was no indication in the man page
> >> >> that there could be a problem.
> >> >
> >> > Fair. I'd have to test that to see if it works correctly. I would
> >> > hope so, but I haven't tested that configuration. This is the
> >> > downside with having lots of different ways to do things!
> >> >
> >> >> How do I separate location of archivecache from the other
> >> >> metapartition path?
> >> >> And fix the cache and archivecache files?
> >> >
> >> > This I don't know a good answer for. I will test if having the same
> >> > path for cache and archivecache could fail. I THINK that I made the
> >> > code safe for it, but I'm not sure that it's been tested.
> >> >
> >> >> No there is no sysmlink farm. We have mounted different iSCSI volumes to
> >> >> /srv/cyrus-ssd-be, /srv/cyrus-hdd-be and /srv/cyrus-be
> >> >
> >> > Right. That makes sense.
> 
> Did you have time to look into the cache/archivecache situation jet?

Yes, I've looked at the code!

in mailbox_archive():

 /* got a new cache record to write */
 if (differentcache)
 {
 dirtycache = 1;
 copyrecord.cache_offset = 0;
 if (mailbox_append_cache(mailbox, &copyrecord))
 continue;
 }

And the code for differentcache is:

 char *spoolcache = xstrdup(mailbox_meta_fname(mailbox, META_CACHE));
 char *archivecache = xstrdup(mailbox_meta_fname(mailbox, META_ARCHIVECACHE));
 int differentcache = strcmp(spoolcache, archivecache);

So it looks like the answer is cache/archivecache is fine. It is not your problem.

> 
> >> > Right! I do wonder if there are some bugs in 3.0.x which are fixed
> >> > on master around delivery to archive partition. We definitely had
> >> > bugs on master, but I thought they were newly introduced on master
> >> > as well, which is why the fixes weren't backported. But if you're
> >> > having files be in the wrong location, maybe there are bugs on 3.0.x
> >> > as well.
> 
> Are all fixes from master backported to 3.0?

Unfortunately it's hard to tell, because many of the fixes on master are fixes to bugs that were only introduced on master, and some bugs on 3.0 we just say "the fix is so invasive that it's basically just backporting 90% of master, which is pointless for a stable release".

> Is the new Commit "I will try your new commits regarding CID" related to the
> "IOERROR: conversations_audit on load:" and "IOERROR: 
> conversations_audit on store"?

Shouldn't be. It just means we store the G keys regardless of whether the record has a CID.

> I will try your new commits in the next days on my test servers to sea 
> if the fix
> the endless loop in ctl_conversationsdb I have seen for some accounts.

I guess one more question - are you running the most recent index version? (reconstruct -V max)

> Quoting myself (Re: prblems rebuilding conversations db) Jan 24, 2019
> 
> > The program loops in build_cid_cb (imap/ctl_conversationsdb.c:189)
> >
> > For the problematic mailbox that I tested, for every message
> > record->cid was NULLCONVERSATION, so mailbox_cacherecord,
> > message_update_conversations and mailbox_rewrite_index_record
> > where called, each returned 0.
> >
> > After iterating trough all messages in the mailbox count was > 0, and r==0,
> > so the while condition (!r && count) was true for the next run.
> > At this point record->cid was still NULLCONVERSATION for every message,
> > which I guess should not be the case.

ctl_conversationsdb -b should update the cid. BUT - if you're running old mailboxes which have a format which doesn't support saving the CID, that would for sure break things!

Bron.

--
 Bron Gondwana, CEO, FastMail Pty Ltd
 brong at fastmailteam.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190213/bdffdb55/attachment-0001.html>

From Eric.Luyten at vub.ac.be  Wed Feb 13 06:01:13 2019
From: Eric.Luyten at vub.ac.be (Eric Luyten)
Date: Wed, 13 Feb 2019 12:01:13 +0100
Subject: disk space used by a mailbox without expunged
In-Reply-To: <20190213091726.Horde.w-lvGgE-YxwYOHG-QoBax6M@webmail.uni-tuebingen.de>
References: <20190213091726.Horde.w-lvGgE-YxwYOHG-QoBax6M@webmail.uni-tuebingen.de>
Message-ID: <ecd1d0be-2e5c-f689-d5b4-afa383eac2d4@vub.ac.be>


On 13/02/2019 09:17, Michael Menge wrote:
> Hi Marcus,
>
> Quoting Marcus Schopen <lists at localguru.de>:
>
>> Hi,
>>
>> is there a way to count the disk space used by a mailbox without
>> expunged messages?
>>
>
> mbexamine user/LoginID | grep Size
>
> on older cyrus versions (2.3 and 2.4) mbexamine did examine all 
> subfolders as well
> in 3.0 only the info for the given folder is shown



O dear.

We (2.3 server, upgrading this year) use the subfolder size information 
extensively in our management procedures.



Eric Luyten.





From boutilpj at ednet.ns.ca  Wed Feb 13 07:08:30 2019
From: boutilpj at ednet.ns.ca (Patrick Boutilier)
Date: Wed, 13 Feb 2019 08:08:30 -0400
Subject: disk space used by a mailbox without expunged
In-Reply-To: <ecd1d0be-2e5c-f689-d5b4-afa383eac2d4@vub.ac.be>
References: <20190213091726.Horde.w-lvGgE-YxwYOHG-QoBax6M@webmail.uni-tuebingen.de>
 <ecd1d0be-2e5c-f689-d5b4-afa383eac2d4@vub.ac.be>
Message-ID: <c0b8a737-54c6-4301-1cbf-4d17ee58b226@ednet.ns.ca>

On 2/13/19 7:01 AM, Eric Luyten wrote:
> 
> On 13/02/2019 09:17, Michael Menge wrote:
>> Hi Marcus,
>>
>> Quoting Marcus Schopen <lists at localguru.de>:
>>
>>> Hi,
>>>
>>> is there a way to count the disk space used by a mailbox without
>>> expunged messages?
>>>
>>
>> mbexamine user/LoginID | grep Size
>>
>> on older cyrus versions (2.3 and 2.4) mbexamine did examine all 
>> subfolders as well
>> in 3.0 only the info for the given folder is shown
> 
> 
> 
> O dear.
> 
> We (2.3 server, upgrading this year) use the subfolder size information 
> extensively in our management procedures.
> 
> 

Use /* to get the subfolders. Such as:


/usr/local/cyrus/sbin/mbexamine user/testuser|grep 'Mailbox Size'

   Number of Messages: 21  Mailbox Size: 332856 bytes  Annotations Size: 
0 bytes




/usr/local/cyrus/sbin/mbexamine user/testuser/*|grep 'Mailbox Size'

   Number of Messages: 61  Mailbox Size: 1578340 bytes  Annotations 
Size: 0 bytes
   Number of Messages: 1  Mailbox Size: 22909 bytes  Annotations Size: 0 
bytes
   Number of Messages: 0  Mailbox Size: 0 bytes  Annotations Size: 0 bytes
   Number of Messages: 1573  Mailbox Size: 50237802 bytes  Annotations 
Size: 0 bytes
   Number of Messages: 41  Mailbox Size: 639825 bytes  Annotations Size: 
0 bytes
   Number of Messages: 0  Mailbox Size: 0 bytes  Annotations Size: 0 bytes

> 
> Eric Luyten.
> 
> 
> 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-------------- next part --------------
A non-text attachment was scrubbed...
Name: boutilpj.vcf
Type: text/x-vcard
Size: 286 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190213/a947e181/attachment.vcf>

From egoitz at sarenet.es  Wed Feb 13 07:10:59 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Wed, 13 Feb 2019 13:10:59 +0100
Subject: Cyrus 3.0.8 running :) and feedback of the upgrade
Message-ID: <3d58ba2b56ebbe5857096b0b618fcc4b@sarenet.es>

Hi mates! 

This has been the great day. We have upgraded a server pair
(master/slave). Almost all has gone fine, apart from the fact that we
have two extrange issues : 

- Some users had found their folders unsubscribed and seems the same
too, lost them Sieve filters. Perhaps due to autocreate or xlist?? which
we weren't previously using?? 

- I'm not really sure, if Xapian is "actively" working. 

We have upgraded from 2.4 Cyrus to 3.0.8 with the replication method.
First has been finally fixed with a couple of scripts. The second one...
I paste the master/slave config here : https://pastebin.com/CtCEedty I
think it's all ok... and by the way log is saying : 

Feb 13 13:07:19 masterserver squatter[13446]: Xapian committed 1 updates
in 0.146300 sec
Feb 13 13:07:19 masterserver squatter[13446]: Xapian committed 3 updates
in 0.075679 sec
Feb 13 13:07:21 masterserver squatter[13446]: Xapian committed 36
updates in 1.219806 sec
Feb 13 13:07:22 masterserver squatter[13446]: Xapian committed 1 updates
in 0.256899 sec
Feb 13 13:07:22 masterserver squatter[13446]: Xapian committed 5 updates
in 0.165767 sec
Feb 13 13:07:22 masterserver squatter[13446]: Xapian committed 1 updates
in 0.252658 sec
Feb 13 13:07:22 masterserver squatter[13446]: Xapian committed 4 updates
in 0.083638 sec
Feb 13 13:07:24 masterserver squatter[13446]: Xapian committed 4 updates
in 0.952737 sec 

Xapian version is xapian-core-1.4.9_1 (FreeBSD port). Could I see by
some debbuging log Xapian is working in the searchs in order to know,
that can't be faster and that all is ok?. At the moment I have tree
tiers. But I'm just with the default one in almost all mailboxes. 

Thank you so much mates :) :) (as always) 

Cheers! 

-- 

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia) 
egoitz at sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electr?nico piense si es necesario
hacerlo. 

Links:
------
[1] http://www.sarenet.es
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190213/4e0bd5d7/attachment.html>

From rsto at fastmailteam.com  Wed Feb 13 07:21:39 2019
From: rsto at fastmailteam.com (Robert Stepanek)
Date: Wed, 13 Feb 2019 13:21:39 +0100
Subject: Cyrus 3.0.8 running :) and feedback of the upgrade
In-Reply-To: <3d58ba2b56ebbe5857096b0b618fcc4b@sarenet.es>
References: <3d58ba2b56ebbe5857096b0b618fcc4b@sarenet.es>
Message-ID: <1550060499.1975585.1657120072.50771DA3@webmail.messagingengine.com>

Hi Egoitz,

On Wed, Feb 13, 2019, at 1:10 PM, Egoitz Aurrekoetxea wrote:
> Xapian version is xapian-core-1.4.9_1 (FreeBSD port). Could I see by
> some debbuging log Xapian is working in the searchs in order to know,
> that can't be faster and that all is ok?. At the moment I have tree
> tiers. But I'm just with the default one in almost all mailboxes
If you use FUZZY search in your search expression then it must go
through Xapian.
Cheers,
Robert

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190213/bb0c5a1e/attachment.html>

From Eric.Luyten at vub.ac.be  Wed Feb 13 07:22:14 2019
From: Eric.Luyten at vub.ac.be (Eric Luyten)
Date: Wed, 13 Feb 2019 13:22:14 +0100
Subject: disk space used by a mailbox without expunged
In-Reply-To: <c0b8a737-54c6-4301-1cbf-4d17ee58b226@ednet.ns.ca>
References: <20190213091726.Horde.w-lvGgE-YxwYOHG-QoBax6M@webmail.uni-tuebingen.de>
 <ecd1d0be-2e5c-f689-d5b4-afa383eac2d4@vub.ac.be>
 <c0b8a737-54c6-4301-1cbf-4d17ee58b226@ednet.ns.ca>
Message-ID: <c2277527-56ca-880e-c492-103df2bc5c0e@vub.ac.be>


On 13/02/2019 13:08, Patrick Boutilier wrote:
> On 2/13/19 7:01 AM, Eric Luyten wrote:
>>
>> On 13/02/2019 09:17, Michael Menge wrote:
>>> Hi Marcus,
>>>
>>> Quoting Marcus Schopen <lists at localguru.de>:
>>>
>>>> Hi,
>>>>
>>>> is there a way to count the disk space used by a mailbox without
>>>> expunged messages?
>>>>
>>>
>>> mbexamine user/LoginID | grep Size
>>>
>>> on older cyrus versions (2.3 and 2.4) mbexamine did examine all 
>>> subfolders as well
>>> in 3.0 only the info for the given folder is shown
>>
>>
>>
>> O dear.
>>
>> We (2.3 server, upgrading this year) use the subfolder size 
>> information extensively in our management procedures.
>>
>>
>
> Use /* to get the subfolders.


Patrick,


So there is only a command syntax and no functionality change.

We can handle that.


Eric.



From egoitz at sarenet.es  Wed Feb 13 08:24:31 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Wed, 13 Feb 2019 14:24:31 +0100
Subject: Cyrus 3.0.8 running :) and feedback of the upgrade
In-Reply-To: <1550060499.1975585.1657120072.50771DA3@webmail.messagingengine.com>
References: <3d58ba2b56ebbe5857096b0b618fcc4b@sarenet.es>
 <1550060499.1975585.1657120072.50771DA3@webmail.messagingengine.com>
Message-ID: <7f504df395c7331ab203f5663e4ce426@sarenet.es>

Or.. if using in imapd.conf "search_fuzzy_always: 1" isn't it?. 

If a search returns results... I assume is going through it... isn't
it?. else.. I assume nothing would be found... I really attached too the
configs (in url format of pastebin), https://pastebin.com/CtCEedty  just
for... the cause you could see something missing  there... which by the
way... I don't think something could be missed but I'm new to Xapian :)
:) although have been running Cyrus more than ten years ago :) :) ... 

Cheers!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia) 
egoitz at sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electr?nico piense si es necesario
hacerlo. 

El 13-02-2019 13:21, Robert Stepanek escribi?:

> Hi Egoitz, 
> 
> On Wed, Feb 13, 2019, at 1:10 PM, Egoitz Aurrekoetxea wrote: 
> 
>> Xapian version is xapian-core-1.4.9_1 (FreeBSD port). Could I see by some debbuging log Xapian is working in the searchs in order to know, that can't be faster and that all is ok?. At the moment I have tree tiers. But I'm just with the default one in almost all mailboxes
> 
> If you use FUZZY search in your search expression then it must go through Xapian. 
> 
> Cheers, 
> Robert 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 

Links:
------
[1] http://www.sarenet.es
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190213/ad0fd8c1/attachment-0001.html>

From rsto at fastmailteam.com  Wed Feb 13 11:58:43 2019
From: rsto at fastmailteam.com (Robert Stepanek)
Date: Wed, 13 Feb 2019 17:58:43 +0100
Subject: Cyrus 3.0.8 running :) and feedback of the upgrade
In-Reply-To: <7f504df395c7331ab203f5663e4ce426@sarenet.es>
References: <3d58ba2b56ebbe5857096b0b618fcc4b@sarenet.es>
 <1550060499.1975585.1657120072.50771DA3@webmail.messagingengine.com>
 <7f504df395c7331ab203f5663e4ce426@sarenet.es>
Message-ID: <1550077123.2255016.1657312216.15D52C59@webmail.messagingengine.com>

On Wed, Feb 13, 2019, at 2:24 PM, Egoitz Aurrekoetxea wrote:
> Or.. if using in imapd.conf "search_fuzzy_always: 1" isn't it?.



Yes, that will instruct imapd to always use FUZZY search for IMAP SEARCH
commands [1]. If you use JMAP, it always use fuzzy search (and hence the
Xapian backend).
[1]https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/imapd.c#L6059
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190213/521c7390/attachment.html>

From egoitz at sarenet.es  Wed Feb 13 12:19:40 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Wed, 13 Feb 2019 18:19:40 +0100
Subject: Cyrus 3.0.8 running :) and feedback of the upgrade
In-Reply-To: <1550077123.2255016.1657312216.15D52C59@webmail.messagingengine.com>
References: <3d58ba2b56ebbe5857096b0b618fcc4b@sarenet.es>
 <1550060499.1975585.1657120072.50771DA3@webmail.messagingengine.com>
 <7f504df395c7331ab203f5663e4ce426@sarenet.es>
 <1550077123.2255016.1657312216.15D52C59@webmail.messagingengine.com>
Message-ID: <b7ee844d19d08e5f1af11a4318f50b7b@sarenet.es>

thanks a lot mate! 

I'm doing checks... for comparing the previous testing env and live
production :) 

Cheers!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia) 
egoitz at sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electr?nico piense si es necesario
hacerlo. 

El 13-02-2019 17:58, Robert Stepanek escribi?:

> On Wed, Feb 13, 2019, at 2:24 PM, Egoitz Aurrekoetxea wrote: 
> 
>> Or.. if using in imapd.conf "search_fuzzy_always: 1" isn't it?.
> 
> Yes, that will instruct imapd to always use FUZZY search for IMAP SEARCH commands [1]. If you use JMAP, it always use fuzzy search (and hence the Xapian backend). 
> 
> [1] https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/imapd.c#L6059
 

Links:
------
[1] http://www.sarenet.es
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190213/33905f00/attachment.html>

From egoitz at sarenet.es  Thu Feb 14 02:37:47 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Thu, 14 Feb 2019 08:37:47 +0100
Subject: Cyrus 3.0.8 running :) and feedback of the upgrade
In-Reply-To: <b7ee844d19d08e5f1af11a4318f50b7b@sarenet.es>
References: <3d58ba2b56ebbe5857096b0b618fcc4b@sarenet.es>
 <1550060499.1975585.1657120072.50771DA3@webmail.messagingengine.com>
 <7f504df395c7331ab203f5663e4ce426@sarenet.es>
 <1550077123.2255016.1657312216.15D52C59@webmail.messagingengine.com>
 <b7ee844d19d08e5f1af11a4318f50b7b@sarenet.es>
Message-ID: <f8604248dd5cc9577fb8d9023e55150d@sarenet.es>

Hi mates! 

I don't really know where the difference is, but in the dev environment,
searches are hugely faster. I know, in dev is not the same as production
in terms of muas traffic and so... but... we are talking in differences
like from 5 seconds to perhaps 40-50 seconds and obviously when saying
this... production is a non loaded env... I mean we have there 4000
accounts, but io is fine, cpu and mem are nice... os limits too....
there's no any bottleneck... 

The real difference between my dev env and prod one... is basically the
way they were built. 

FOR DEV I created a copy of one production machine. A copy... for being
able to be the most close as we could, from the prod env in this testing
env. That prod env, was at 2.3. I then, connected to it 2.4-Cyrus root
fs (with Cyrus 2.4). Tested this version and converted to 12 version
some databases (just the ones for testing, the same ones as where I have
seen this speed effect). Later I disconnected the 2.4-Cyrus root disk
and connected a 3.0-Cyrus root disk. Did a reconstruct -r -V max,
ctl_conversationsdb -z on-testing-used-mailboxes and ctl_conversationsdb
-b on-testing-used-mailboxes. 

FOR PROD we connected to 2.3 server the 2.4-Cyrus root disk. Later we
setup an empty slave with 3.0 and started a user by user, user mode
replication there. Later a rolling one (with the file generated between
last "all users, user by user sync" and the moment we started the
rolling one) when almost all in the slave was very near to the present
state of the master in that moment. This replication was done with Csync
not IMAP. 

These too are the basic differences...  could this give you some kind of
clue?... I'll try to debug code too in order to see something.... 

Cheers!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia) 
egoitz at sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electr?nico piense si es necesario
hacerlo. 

El 13-02-2019 18:19, Egoitz Aurrekoetxea escribi?:

> thanks a lot mate! 
> 
> I'm doing checks... for comparing the previous testing env and live production :) 
> 
> Cheers!
> 
> ---
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 944 209 470
> Parque Tecnol?gico. Edificio 103
> 48170 Zamudio (Bizkaia) 
> egoitz at sarenet.es 
> www.sarenet.es [1] 
> Antes de imprimir este correo electr?nico piense si es necesario hacerlo. 
> 
> El 13-02-2019 17:58, Robert Stepanek escribi?: 
> On Wed, Feb 13, 2019, at 2:24 PM, Egoitz Aurrekoetxea wrote: 
> 
> Or.. if using in imapd.conf "search_fuzzy_always: 1" isn't it?. 
> 
> Yes, that will instruct imapd to always use FUZZY search for IMAP SEARCH commands [1]. If you use JMAP, it always use fuzzy search (and hence the Xapian backend). 
> 
> [1] https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/imapd.c#L6059

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 

Links:
------
[1] http://www.sarenet.es
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190214/2af7c40f/attachment.html>

From raphael.halimi at gmail.com  Thu Feb 14 07:27:17 2019
From: raphael.halimi at gmail.com (=?UTF-8?Q?Rapha=c3=abl_Halimi?=)
Date: Thu, 14 Feb 2019 13:27:17 +0100
Subject: Various questions about databases (upgrade and migration)
Message-ID: <q43mr6$1jq7$1@blaine.gmane.org>

Hi,

I have a small (3 users) Cyrus IMAP server that started as a very old
Debian (Woody or Sarge, I don't remember exactly) and that I upgraded
through the years, and yesterday I upgraded from Jessie to Stretch
(Cyrus 2.4.17 without caldav, to 2.5.10 with caldav).

As I had some trouble in the past with databases migrations between
BerkeleyDB versions during some upgrades, and although Cyrus started
without any serious problem this time (after minor tuning of some
options in the configuration files), I still wanted to check the state
of the databases.

So, the contents of /usr/lib/cyrus/cyrus-db-types.txt is:

ANNOTATION twoskip
DBENGINE BerkeleyDB5.3
DUPLICATE twoskip
MBOX twoskip
PTS twoskip
QUOTA quotalegacy
SEEN twoskip
STATUSCACHE twoskip
SUBS flat
TLS DEFAULT
TLS twoskip
USERDENY flat
ZONEINFO twoskip

(note that in Debian, no format is forced in /etc/imapd.conf)

So my first questions is: I thought that Cyrus had abandoned BerkeleyDB
a long time ago for skiplist, and then twoskip. Why is it still listed
in this file ? Is there a part of Cyrus that still uses it ?

Now for the databases themselves. In /var/lib/cyrus the global databases
were converted on-the-fly:

# file /var/lib/cyrus/*.db
/var/lib/cyrus/annotations.db:  Cyrus twoskip DB
/var/lib/cyrus/deliver.db:      Cyrus twoskip DB
/var/lib/cyrus/mailboxes.db:    Cyrus twoskip DB
/var/lib/cyrus/statuscache.db:  Cyrus twoskip DB
/var/lib/cyrus/tls_sessions.db: Cyrus twoskip DB
/var/lib/cyrus/user_deny.db:    empty

However, the user databases were not converted:

# file /var/lib/cyrus/user/*/*
/var/lib/cyrus/user/u/user1.seen: Cyrus skiplist DB
/var/lib/cyrus/user/u/user1.sub:  ASCII text
/var/lib/cyrus/user/u/user2.seen:   Cyrus skiplist DB
/var/lib/cyrus/user/u/user2.sub:    ASCII text
/var/lib/cyrus/user/u/user3.seen:   Cyrus skiplist DB
/var/lib/cyrus/user/u/user3.sub:    ASCII text

So my next questions are: why are the databases still in skiplist
format, whereas according to /usr/lib/cyrus/cyrus-db-types.txt, they
should be twoskip ? Why didn't Cyrus convert them on-the-fly like the
global databases ? Do I have to manually do it myself ? And if I do
convert them, will it change anything (performance, reliability, etc etc) ?

Also, what about the various databases in the mail directories
(cyrus.cache, cyrus.header, cyrus.index) ? For most of them, the "file"
command only reports "data". What format are they actually in ? Do I
have to convert them too ?

My last question is about a planned migration of this home server to a
hosted private server. This old server and the new one are now both
Debian Stretch and thus, have the same Cyrus version, but they're not
the same architecture: the old one is i386, and the new one is amd64.

When I mill migrate, will I have to convert the databases through the
flat format and back, or can I blindly copy the whole contents of
/var/spool/cyrus and /var/lib/cyrus to the new server and expect it to
work out of the box ?

Thanks a lot in advance for your answers.

Regards,

-- 
Rapha?l Halimi

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: OpenPGP digital signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190214/025c01d1/attachment-0001.sig>

From egoitz at sarenet.es  Thu Feb 14 08:46:13 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Thu, 14 Feb 2019 14:46:13 +0100
Subject: Various questions about databases (upgrade and migration)
In-Reply-To: <q43mr6$1jq7$1@blaine.gmane.org>
References: <q43mr6$1jq7$1@blaine.gmane.org>
Message-ID: <2e96ab38210745b4ba4e9165820e7c91@sarenet.es>

Hi Rapha?l, 

Answering below in blue for instance......

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia) 
egoitz at sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electr?nico piense si es necesario
hacerlo. 

El 14-02-2019 13:27, Rapha?l Halimi escribi?:

> Hi,
> 
> I have a small (3 users) Cyrus IMAP server that started as a very old
> Debian (Woody or Sarge, I don't remember exactly) and that I upgraded
> through the years, and yesterday I upgraded from Jessie to Stretch
> (Cyrus 2.4.17 without caldav, to 2.5.10 with caldav).
> 
> As I had some trouble in the past with databases migrations between
> BerkeleyDB versions during some upgrades, and although Cyrus started
> without any serious problem this time (after minor tuning of some
> options in the configuration files), I still wanted to check the state
> of the databases.
> 
> So, the contents of /usr/lib/cyrus/cyrus-db-types.txt is:
> 
> ANNOTATION twoskip
> DBENGINE BerkeleyDB5.3
> DUPLICATE twoskip
> MBOX twoskip
> PTS twoskip
> QUOTA quotalegacy
> SEEN twoskip
> STATUSCACHE twoskip
> SUBS flat
> TLS DEFAULT
> TLS twoskip
> USERDENY flat
> ZONEINFO twoskip
> 
> (note that in Debian, no format is forced in /etc/imapd.conf)
> 
> So my first questions is: I thought that Cyrus had abandoned BerkeleyDB
> a long time ago for skiplist, and then twoskip. Why is it still listed
> in this file ? Is there a part of Cyrus that still uses it ? 
> 
> I SUPPOSE IT COULD SOME DEBIAN STANDARD CONFIG?. WE RUN FREEBSD AND I HAVE NEVER SEE THAT FILE. 
> 
> Now for the databases themselves. In /var/lib/cyrus the global databases
> were converted on-the-fly:
> 
> # file /var/lib/cyrus/*.db
> /var/lib/cyrus/annotations.db:  Cyrus twoskip DB
> /var/lib/cyrus/deliver.db:      Cyrus twoskip DB
> /var/lib/cyrus/mailboxes.db:    Cyrus twoskip DB
> /var/lib/cyrus/statuscache.db:  Cyrus twoskip DB
> /var/lib/cyrus/tls_sessions.db: Cyrus twoskip DB
> /var/lib/cyrus/user_deny.db:    empty
> 
> However, the user databases were not converted:
> 
> # file /var/lib/cyrus/user/*/*
> /var/lib/cyrus/user/u/user1.seen: Cyrus skiplist DB
> /var/lib/cyrus/user/u/user1.sub:  ASCII text
> /var/lib/cyrus/user/u/user2.seen:   Cyrus skiplist DB
> /var/lib/cyrus/user/u/user2.sub:    ASCII text
> /var/lib/cyrus/user/u/user3.seen:   Cyrus skiplist DB
> /var/lib/cyrus/user/u/user3.sub:    ASCII text 
> 
> CYRUS 2.4 CONVERTED DATABASES ON THE FLY. CYRUS 2.5 AND NEWER DON'T. YOU SHOULD LAUNCH A "RECONSTRUCT -R -V MAX" FOR THAT PURPOSE.
> 
> So my next questions are: why are the databases still in skiplist
> format, whereas according to /usr/lib/cyrus/cyrus-db-types.txt, they
> should be twoskip ? Why didn't Cyrus convert them on-the-fly like the
> global databases ? Do I have to manually do it myself ? And if I do
> convert them, will it change anything (performance, reliability, etc etc) ? 
> 
> AS SAID PERHAPS IS A DEBIAN DERIVED CONFIG FOR THE PACKAGE. YES YOU SHOULD WITH THE COMMAND ABOVE.
> 
> Also, what about the various databases in the mail directories
> (cyrus.cache, cyrus.header, cyrus.index) ? For most of them, the "file"
> command only reports "data". What format are they actually in ? Do I
> have to convert them too ? 
> 
> SURE... JUST LAUNCH A RECONSTRUCT -R -V MAX...
> 
> My last question is about a planned migration of this home server to a
> hosted private server. This old server and the new one are now both
> Debian Stretch and thus, have the same Cyrus version, but they're not
> the same architecture: the old one is i386, and the new one is amd64.
> 
> When I mill migrate, will I have to convert the databases through the
> flat format and back, or can I blindly copy the whole contents of
> /var/spool/cyrus and /var/lib/cyrus to the new server and expect it to
> work out of the box ? 
> 
> WHEN DOING SUCH A MIGRATION, IT WOULD BE BETTER TO SETUP A REPLICATION BETWEEN THE 2.4 AND THE NEW 2.5 IN THE HOSTING. YOU SHOULD ENCRYPT THAT COMMUNICATION. YOU COULD USE THE OWN CYRUS ENCRIPTION FOR REPLICATION OR SOMETHING LIKE OPENVPN. ALTHOUGH IT SHOULD WORK, I WOULDN'T COPY DIRECTLY (WITH AN RSYNC OR SCP) THE FILES.
> 
> Thanks a lot in advance for your answers.
> 
> Regards, 
> 
> CHEERS!
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 

Links:
------
[1] http://www.sarenet.es
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190214/39d23eb7/attachment.html>

From raphael.halimi at gmail.com  Thu Feb 14 09:46:03 2019
From: raphael.halimi at gmail.com (=?UTF-8?Q?Rapha=c3=abl_Halimi?=)
Date: Thu, 14 Feb 2019 15:46:03 +0100
Subject: Various questions about databases (upgrade and migration)
In-Reply-To: <2e96ab38210745b4ba4e9165820e7c91@sarenet.es>
References: <q43mr6$1jq7$1@blaine.gmane.org>
 <2e96ab38210745b4ba4e9165820e7c91@sarenet.es>
Message-ID: <1bb618c3-73cb-ef2f-e1cf-06c23b1a57c9@gmail.com>

Hi Egoitz,

Thank you for your quick answer.

Le 14/02/2019 ? 14:46, Egoitz Aurrekoetxea a ?crit?:
> Now for the databases themselves. In /var/lib/cyrus the global databases
> were converted on-the-fly:
>
> # file /var/lib/cyrus/*.db
> /var/lib/cyrus/annotations.db: ?Cyrus twoskip DB
> /var/lib/cyrus/deliver.db: ?????Cyrus twoskip DB
> /var/lib/cyrus/mailboxes.db: ???Cyrus twoskip DB
> /var/lib/cyrus/statuscache.db: ?Cyrus twoskip DB
> /var/lib/cyrus/tls_sessions.db: Cyrus twoskip DB
> /var/lib/cyrus/user_deny.db: ???empty
>
> However, the user databases were not converted:
>
> # file /var/lib/cyrus/user/*/*
> /var/lib/cyrus/user/u/user1.seen: Cyrus skiplist DB
> /var/lib/cyrus/user/u/user1.sub: ?ASCII text
> /var/lib/cyrus/user/u/user2.seen: ??Cyrus skiplist DB
> /var/lib/cyrus/user/u/user2.sub: ???ASCII text
> /var/lib/cyrus/user/u/user3.seen: ??Cyrus skiplist DB
> /var/lib/cyrus/user/u/user3.sub: ???ASCII text
> ?
>> *Cyrus 2.4 converted databases on the fly. Cyrus 2.5 and newer don't.
>> You should launch a "reconstruct -r -V max" for that purpose.*
>
> So my next questions are: why are the databases still in skiplist
> format, whereas according to /usr/lib/cyrus/cyrus-db-types.txt, they
> should be twoskip ? Why didn't Cyrus convert them on-the-fly like the
> global databases ? Do I have to manually do it myself ? And if I do
> convert them, will it change anything (performance, reliability, etc
> etc) ?
> ?
> *As said perhaps is a Debian derived config for the package. Yes you
> should with the command above.*
>>
>> Also, what about the various databases in the mail directories
>> (cyrus.cache, cyrus.header, cyrus.index) ? For most of them, the "file"
>> command only reports "data". What format are they actually in ? Do I
>> have to convert them too ?
>> ?
>> *Sure... Just launch a reconstruct -r -V max...*

I just ran this command as cyrus user on my server (after reading the
manual page). Unfortunately, the "seen" databases in /var/lib/cyrus/user
are still reported by "file" as skiplist, and the "cyrus.cache",
"cyrus.header" and "cyrus.index" in the various (sub)mailboxes, are
still reported as simply "data".

It did create "cyrus.annotations" databases in each subfolder, though
(in twoskip format).

Also,I'm a bit worried. I did see in the logs lines that said:

repacking mailbox user.<user>.<folder>.<subfolder>

and

reconstructing user.<user>.<folder>.<subfolder>

...but also some more worrying lines that said:

uniqueid clash with user.<user>.<folder>.<subfolder> for <some hex
value> - changing user.<user>.<folder>.<subfolder>

Is it something I should worry about ?

Regarding the fact that they're still not in the twoskip format, should
I use cvt_cyrusdb instead ? That would be unfortunate, since I'll have
to create a script fed to the "find" command to mass-convert all
databases; plus, I still don't know what the input format (the "data"
that file talks about) is.

>> When I mill migrate, will I have to convert the databases through the
>> flat format and back, or can I blindly copy the whole contents of
>> /var/spool/cyrus and /var/lib/cyrus to the new server and expect it to
>> work out of the box ?
>> ?
>> *When doing such a migration, it would be better to setup a
>> replication between the 2.4 and the new 2.5 in the hosting. You should
>> encrypt that communication. You could use the own cyrus encription for
>> replication or something like OpenVPN. Although it should work, I
>> wouldn't copy directly (with an rsync or scp) the files.*

Yes, both servers communicate through a VPN, but since both will have
the same Cyrus version, I thought I could just copy the files. Why is it
a bad idea ?

Regards,

-- 
Rapha?l Halimi

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: OpenPGP digital signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190214/6e6a91f0/attachment-0001.sig>

From egoitz at sarenet.es  Thu Feb 14 12:47:07 2019
From: egoitz at sarenet.es (Egoitz Aurrekoetxea)
Date: Thu, 14 Feb 2019 18:47:07 +0100
Subject: Various questions about databases (upgrade and migration)
In-Reply-To: <1bb618c3-73cb-ef2f-e1cf-06c23b1a57c9@gmail.com>
References: <q43mr6$1jq7$1@blaine.gmane.org>
 <2e96ab38210745b4ba4e9165820e7c91@sarenet.es>
 <1bb618c3-73cb-ef2f-e1cf-06c23b1a57c9@gmail.com>
Message-ID: <6467098e0b33edab364baf18cc381583@sarenet.es>

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnol?gico. Edificio 103
48170 Zamudio (Bizkaia) 
egoitz at sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electr?nico piense si es necesario
hacerlo. 

El 14-02-2019 15:46, Rapha?l Halimi escribi?:

> Hi Egoitz,
> 
> Thank you for your quick answer.
> 
> Le 14/02/2019 ? 14:46, Egoitz Aurrekoetxea a ?crit : Now for the databases themselves. In /var/lib/cyrus the global databases
> were converted on-the-fly:
> 
> # file /var/lib/cyrus/*.db
> /var/lib/cyrus/annotations.db:  Cyrus twoskip DB
> /var/lib/cyrus/deliver.db:      Cyrus twoskip DB
> /var/lib/cyrus/mailboxes.db:    Cyrus twoskip DB
> /var/lib/cyrus/statuscache.db:  Cyrus twoskip DB
> /var/lib/cyrus/tls_sessions.db: Cyrus twoskip DB
> /var/lib/cyrus/user_deny.db:    empty
> 
> However, the user databases were not converted:
> 
> # file /var/lib/cyrus/user/*/*
> /var/lib/cyrus/user/u/user1.seen: Cyrus skiplist DB
> /var/lib/cyrus/user/u/user1.sub:  ASCII text
> /var/lib/cyrus/user/u/user2.seen:   Cyrus skiplist DB
> /var/lib/cyrus/user/u/user2.sub:    ASCII text
> /var/lib/cyrus/user/u/user3.seen:   Cyrus skiplist DB
> /var/lib/cyrus/user/u/user3.sub:    ASCII text
> *Cyrus 2.4 converted databases on the fly. Cyrus 2.5 and newer don't.
> You should launch a "reconstruct -r -V max" for that purpose.* 
> So my next questions are: why are the databases still in skiplist
> format, whereas according to /usr/lib/cyrus/cyrus-db-types.txt, they
> should be twoskip ? Why didn't Cyrus convert them on-the-fly like the
> global databases ? Do I have to manually do it myself ? And if I do
> convert them, will it change anything (performance, reliability, etc
> etc) ?
> 
> *As said perhaps is a Debian derived config for the package. Yes you
> should with the command above.* 
> Also, what about the various databases in the mail directories
> (cyrus.cache, cyrus.header, cyrus.index) ? For most of them, the "file"
> command only reports "data". What format are they actually in ? Do I
> have to convert them too ?
> 
> *Sure... Just launch a reconstruct -r -V max...*

I just ran this command as cyrus user on my server (after reading the
manual page). Unfortunately, the "seen" databases in /var/lib/cyrus/user
are still reported by "file" as skiplist, and the "cyrus.cache",
"cyrus.header" and "cyrus.index" in the various (sub)mailboxes, are
still reported as simply "data".

It did create "cyrus.annotations" databases in each subfolder, though
(in twoskip format).

Also,I'm a bit worried. I did see in the logs lines that said:

repacking mailbox user.<user>.<folder>.<subfolder>

and

reconstructing user.<user>.<folder>.<subfolder> 

THOSE TWO LINES ARE PRETTY NORMAL... 

...but also some more worrying lines that said:

uniqueid clash with user.<user>.<folder>.<subfolder> for <some hex
value> - changing user.<user>.<folder>.<subfolder> 

I HAVEN'T EVER SEEN IT.... CAN'T SAY... I SHOULD TAKE A LOT AT THE CODE
FOR ANSWERING.....

Is it something I should worry about ?

Regarding the fact that they're still not in the twoskip format, should
I use cvt_cyrusdb instead ? That would be unfortunate, since I'll have
to create a script fed to the "find" command to mass-convert all
databases; plus, I still don't know what the input format (the "data"
that file talks about) is. 

RECONSTRUCT -R -V MAX SHOULD HANDLE ALL CONVERSIONS...

>> When I mill migrate, will I have to convert the databases through the
>> flat format and back, or can I blindly copy the whole contents of
>> /var/spool/cyrus and /var/lib/cyrus to the new server and expect it to
>> work out of the box ?
>> 
>> *When doing such a migration, it would be better to setup a
>> replication between the 2.4 and the new 2.5 in the hosting. You should
>> encrypt that communication. You could use the own cyrus encription for
>> replication or something like OpenVPN. Although it should work, I
>> wouldn't copy directly (with an rsync or scp) the files.*

Yes, both servers communicate through a VPN, but since both will have
the same Cyrus version, I thought I could just copy the files. Why is it
a bad idea ? 

IT'S NOT A HORRIBLE IDEA... BUT WITH THE REPLICATION YOU CREATE ALL
INDEXES AND SO FROM 0, WITH THE CORRECT VERSION, WITHOUT NEEDING
CONVERSIONS AND WITH THE PROPER VERSION AND, GIVES YOU CLEANER
DATABASES... FOR INSTANCE... AND YOU COULD EVEN COPY CONSTANTLY CONTENT
TO THE DATACENTER FOR JUST FAILING-OVER TO THE DATACENTER WHEN YOU ARE
READY... MOVING TO THE NEW SERVER IS CLEANER AND EASIER... THAT'S WHAT I
WOULD DO :) 

Regards, 

CHEERS! 

Links:
------
[1] http://www.sarenet.es
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190214/075c7ef4/attachment.html>

From jpilfold-bagwell at bordengrammar.kent.sch.uk  Fri Feb 15 06:32:39 2019
From: jpilfold-bagwell at bordengrammar.kent.sch.uk (J Pilfold-Bagwell)
Date: Fri, 15 Feb 2019 11:32:39 +0000
Subject: Sieve not working
Message-ID: <eb2fa1ec-3e14-6315-1467-438349b0bb9a@bordengrammar.kent.sch.uk>

Hi All,

I have a Centos 7 box running with the latest default cyrus install from 
the Centos 7 repo, i.e. cyrus-imapd-2.4.17-13.el7.x86_64 .

The problem I have is that sieve doesn't seem to pay any attention to 
the scripts.? I have sieve running, I can successfully log in to it 
using sieveshell, create, upload and activate scripts, but they don't 
seem to be applied to the incoming mail.? First I was trying the 
vacation and reject scripts so checked that the correct sendmail is in 
use but it fails on fileinto as well.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

imapd.conf looks like this:

[root at mail admin]# cat /etc/imapd.conf
configdirectory: /var/lib/imap
partition-default: /var/spool/imap
admins: cyradmin
sieve_admins: cyradmin
sievedir: /var/lib/imap/sieve
sendmail: /usr/sbin/sendmail
hashimapspool: true
sasl_pwcheck_method: auxprop
sasl_auxprop_plugin: sasldb
sasl_mech_list: PLAIN LOGIN CRAM-MD5 DIGEST-MD5
allowplaintext: yes
allowusermoves: yes
defaultdomain: mail
lmtp_downcase_rcpt: yes

tls_cert_file: /etc/ssl/certs/cyrus-imapd/newcert.pem
tls_key_file: /etc/ssl/certs/cyrus-imapd/newkey.pem
tls_ca_file: /etc/ssl/certs/cyrus-imapd/cacert.pem
tls_ca_path: /etc/ssl/certscyrus-imapd

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

cyrus.conf:

# standard standalone server implementation

START {
 ? # do not delete this entry!
 ? recover??? cmd="ctl_cyrusdb -r"

 ? # this is only necessary if using idled for IMAP IDLE
 ? idled??? ??? cmd="idled"
}

# UNIX sockets start with a slash and are put into /var/lib/imap/sockets
SERVICES {
 ? # add or remove based on preferences
 ? imap??? ??? cmd="imapd" listen="imap" prefork=5
 ? imaps??? ??? cmd="imapd -s" listen="imaps" prefork=1
#? pop3??? ??? cmd="pop3d" listen="pop3" prefork=3
#? pop3s??? ??? cmd="pop3d -s" listen="pop3s" prefork=1
 ? sieve??? ??? cmd="timsieved" listen="0.0.0.0:2000" prefork=0
 ? sieve???????? cmd="timsieved" listen="0.0.0.0:4190" prefork=0
#? managesieve?? cmd="timsieved" listen="localhost:4190" prefork=0

 ? # these are only necessary if receiving/exporting usenet via NNTP
#? nntp??? ??? cmd="nntpd" listen="nntp" prefork=3
#? nntps??? ??? cmd="nntpd -s" listen="nntps" prefork=1

 ? # at least one LMTP is required for delivery
#? lmtp??? ??? cmd="lmtpd" listen="lmtp" prefork=0
 ? lmtpunix??? cmd="lmtpd" listen="/var/lib/imap/socket/lmtp" prefork=1

 ? # this is only necessary if using notifications
#? notify??? cmd="notifyd" listen="/var/lib/imap/socket/notify" 
proto="udp" prefork=1
}

EVENTS {
 ? # this is required
 ? checkpoint??? cmd="ctl_cyrusdb -c" period=30

 ? # this is only necessary if using duplicate delivery suppression,
 ? # Sieve or NNTP
 ? delprune??? cmd="cyr_expire -E 3" at=0400

 ? # this is only necessary if caching TLS sessions
 ? tlsprune??? cmd="tls_prune" at=0400

 ? # reindex changed mailboxes (fulltext) approximately every three hours
 ? squatter1?????? cmd="/usr/bin/ionice -c idle 
/usr/lib/cyrus/bin/squatter -s" period=180

 ? # reindex all mailboxes (fulltext) daily
 ? squattera?????? cmd="/usr/lib/cyrus/bin/squatter" at=0117
}

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

sieveshell logs in fine:

[root at mail admin]# sieveshell --authname=cyradmin --user=testuser1 localhost
connecting to localhost
Please enter your password:
 > list
mail
sieve-test? <- active script

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~telnet

Telnet login provides:

[root at mail admin]# telnet 192.168.0.6 4190
Trying 192.168.0.6...
Connected to 192.168.0.6.
Escape character is '^]'.
"IMPLEMENTATION" "Cyrus timsieved v2.4.17-Fedora-RPM-2.4.17-13.el7"
"SASL" "PLAIN LOGIN CRAM-MD5 DIGEST-MD5"
"SIEVE" "comparator-i;ascii-numeric fileinto reject vacation imapflags 
notify envelope relational regex subaddress copy"
"STARTTLS"
"UNAUTHENTICATE"
OK

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

And this works for both port 2000 and 4190 on all interfaces. LMTP is in 
use but somewhere, they aren't talking.

Does anyone have any troubleshooting tips they can feed me or, can 
anyone see a glaringly obvious error I've made because it's all gone a 
bit wood for the trees here.

The logs are huge but if you'd like to see the contents, let me know 
what you'd like it grep'd for and I'll provide.

Thanks,

Julian

-- 
This email is from Borden Grammar School Trust.

This email, together with any files transmitted with it, is confidential, and is intended solely for the use of the individual or entity to whom it is addressed. Any unauthorised dissemination or copying of this email or its attachments, and any use or disclosure of any information contained in them, is strictly prohibited, and may also be illegal. If you are not the intended recipient you may not use, disclose, distribute, copy, print or relay this email.

Please note that any views expressed by an individual within this email, do not necessarily reflect the views of the Borden Grammar School Trust.

Borden Grammar School Trust has taken reasonable precautions to ensure no viruses are present in this email.  The Academy cannot accept responsibility for any loss or damage arising from the use of this email and/or files attached.

Registered Office: Borden Grammar School, Avenue of Remembrance, Sittingbourne, Kent ME10 4DB

Registered in England: 07827591


From egoitz at sarenet.es  Sun Feb 17 13:22:51 2019
From: egoitz at sarenet.es (egoitz at sarenet.es)
Date: Sun, 17 Feb 2019 19:22:51 +0100
Subject: Question about what is replicated with Cyrus replication and what
 isn't
Message-ID: <caa91e620989215a9b94774bb1609df6@sarenet.es>

Hi! 

Previously (in 2.3 and older versions), cyr_expire and ipurge actions
for instance where not replicated to the slave. So, you needed to launch
them in both, the master and the slave. My question is, are now
replicated as mailbox replication commands?. What about commands like
Squatter -F for removing in Cyrus 3 from the database non existing
mails?. Or when you move data from search tiers in Xapian?. Are those
actions replicated or should be launched in the master and the replica?.


Cheers!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190217/cba4e861/attachment.html>

From egoitz at sarenet.es  Mon Feb 18 03:34:20 2019
From: egoitz at sarenet.es (egoitz at sarenet.es)
Date: Mon, 18 Feb 2019 09:34:20 +0100
Subject: Sieve not working
In-Reply-To: <eb2fa1ec-3e14-6315-1467-438349b0bb9a@bordengrammar.kent.sch.uk>
References: <eb2fa1ec-3e14-6315-1467-438349b0bb9a@bordengrammar.kent.sch.uk>
Message-ID: <b6eeff5b17822c1ce8f080e0a3a3c8e6@sarenet.es>

Hi! 

Could you try enabling local6.debug channel in syslog, so that you could
see additional Sieve debugging information?. Can you then post that log?


Cheers!

El 2019-02-15 12:32, J Pilfold-Bagwell escribi?:

> Hi All,
> 
> I have a Centos 7 box running with the latest default cyrus install from the Centos 7 repo, i.e. cyrus-imapd-2.4.17-13.el7.x86_64 .
> 
> The problem I have is that sieve doesn't seem to pay any attention to the scripts.  I have sieve running, I can successfully log in to it using sieveshell, create, upload and activate scripts, but they don't seem to be applied to the incoming mail.  First I was trying the vacation and reject scripts so checked that the correct sendmail is in use but it fails on fileinto as well.
> 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> imapd.conf looks like this:
> 
> [root at mail admin]# cat /etc/imapd.conf
> configdirectory: /var/lib/imap
> partition-default: /var/spool/imap
> admins: cyradmin
> sieve_admins: cyradmin
> sievedir: /var/lib/imap/sieve
> sendmail: /usr/sbin/sendmail
> hashimapspool: true
> sasl_pwcheck_method: auxprop
> sasl_auxprop_plugin: sasldb
> sasl_mech_list: PLAIN LOGIN CRAM-MD5 DIGEST-MD5
> allowplaintext: yes
> allowusermoves: yes
> defaultdomain: mail
> lmtp_downcase_rcpt: yes
> 
> tls_cert_file: /etc/ssl/certs/cyrus-imapd/newcert.pem
> tls_key_file: /etc/ssl/certs/cyrus-imapd/newkey.pem
> tls_ca_file: /etc/ssl/certs/cyrus-imapd/cacert.pem
> tls_ca_path: /etc/ssl/certscyrus-imapd
> 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> cyrus.conf:
> 
> # standard standalone server implementation
> 
> START {
> # do not delete this entry!
> recover    cmd="ctl_cyrusdb -r"
> 
> # this is only necessary if using idled for IMAP IDLE
> idled        cmd="idled"
> }
> 
> # UNIX sockets start with a slash and are put into /var/lib/imap/sockets
> SERVICES {
> # add or remove based on preferences
> imap        cmd="imapd" listen="imap" prefork=5
> imaps        cmd="imapd -s" listen="imaps" prefork=1
> #  pop3        cmd="pop3d" listen="pop3" prefork=3
> #  pop3s        cmd="pop3d -s" listen="pop3s" prefork=1
> sieve        cmd="timsieved" listen="0.0.0.0:2000" prefork=0
> sieve         cmd="timsieved" listen="0.0.0.0:4190" prefork=0
> #  managesieve   cmd="timsieved" listen="localhost:4190" prefork=0
> 
> # these are only necessary if receiving/exporting usenet via NNTP
> #  nntp        cmd="nntpd" listen="nntp" prefork=3
> #  nntps        cmd="nntpd -s" listen="nntps" prefork=1
> 
> # at least one LMTP is required for delivery
> #  lmtp        cmd="lmtpd" listen="lmtp" prefork=0
> lmtpunix    cmd="lmtpd" listen="/var/lib/imap/socket/lmtp" prefork=1
> 
> # this is only necessary if using notifications
> #  notify    cmd="notifyd" listen="/var/lib/imap/socket/notify" proto="udp" prefork=1
> }
> 
> EVENTS {
> # this is required
> checkpoint    cmd="ctl_cyrusdb -c" period=30
> 
> # this is only necessary if using duplicate delivery suppression,
> # Sieve or NNTP
> delprune    cmd="cyr_expire -E 3" at=0400
> 
> # this is only necessary if caching TLS sessions
> tlsprune    cmd="tls_prune" at=0400
> 
> # reindex changed mailboxes (fulltext) approximately every three hours
> squatter1       cmd="/usr/bin/ionice -c idle /usr/lib/cyrus/bin/squatter -s" period=180
> 
> # reindex all mailboxes (fulltext) daily
> squattera       cmd="/usr/lib/cyrus/bin/squatter" at=0117
> }
> 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> sieveshell logs in fine:
> 
> [root at mail admin]# sieveshell --authname=cyradmin --user=testuser1 localhost
> connecting to localhost
> Please enter your password: 
> 
>> list
> mail
> sieve-test  <- active script
> 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~telnet
> 
> Telnet login provides:
> 
> [root at mail admin]# telnet 192.168.0.6 4190
> Trying 192.168.0.6...
> Connected to 192.168.0.6.
> Escape character is '^]'.
> "IMPLEMENTATION" "Cyrus timsieved v2.4.17-Fedora-RPM-2.4.17-13.el7"
> "SASL" "PLAIN LOGIN CRAM-MD5 DIGEST-MD5"
> "SIEVE" "comparator-i;ascii-numeric fileinto reject vacation imapflags notify envelope relational regex subaddress copy"
> "STARTTLS"
> "UNAUTHENTICATE"
> OK
> 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> And this works for both port 2000 and 4190 on all interfaces. LMTP is in use but somewhere, they aren't talking.
> 
> Does anyone have any troubleshooting tips they can feed me or, can anyone see a glaringly obvious error I've made because it's all gone a bit wood for the trees here.
> 
> The logs are huge but if you'd like to see the contents, let me know what you'd like it grep'd for and I'll provide.
> 
> Thanks,
> 
> Julian
> 
> -- 
> This email is from Borden Grammar School Trust.
> 
> This email, together with any files transmitted with it, is confidential, and is intended solely for the use of the individual or entity to whom it is addressed. Any unauthorised dissemination or copying of this email or its attachments, and any use or disclosure of any information contained in them, is strictly prohibited, and may also be illegal. If you are not the intended recipient you may not use, disclose, distribute, copy, print or relay this email.
> 
> Please note that any views expressed by an individual within this email, do not necessarily reflect the views of the Borden Grammar School Trust.
> 
> Borden Grammar School Trust has taken reasonable precautions to ensure no viruses are present in this email.  The Academy cannot accept responsibility for any loss or damage arising from the use of this email and/or files attached.
> 
> Registered Office: Borden Grammar School, Avenue of Remembrance, Sittingbourne, Kent ME10 4DB
> 
> Registered in England: 07827591
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190218/a03ca772/attachment.html>

From felix.schumacher at internetallee.de  Mon Feb 18 03:56:15 2019
From: felix.schumacher at internetallee.de (Felix Schumacher)
Date: Mon, 18 Feb 2019 09:56:15 +0100
Subject: Speed-differences in listing mailboxes between 2.4 and 3.0
Message-ID: <4b1d1175071750a26d4c189a47176ef7@internetallee.de>

Hi all,

we are trying to migrate our mailservers from cyrus imapd 2.4 to 3.0 and 
are currently looking at speed differences in listing mailboxes. Both 
the imap command 'list "" ""' for a user and a ctl_mboxlist -d seem to 
be quite a bit slower.

At the moment the 3.0 environment has around around 170k mail folders 
listed by ctl_mboxlist -d and takes about 0.8s to list all those entries 
(piping the output to /dev/null). This system has no real load except 
for the list command).

In the 2.4 environment we have 244k mail folders listed and it takes 
about 0.5s to list those entries (again piped to /dev/null). This system 
is our productive mail server and had some load to handle while listing 
was done.

mailboxes.db is in skiplist format on the 2.4 environment. We tried 
twoskip and skiplist on the 3.0 environment, but it didn't change that 
much.

Using perf record -F 99 -g ctl_mboxlist -d > /dev/null and then perf 
report I get for 3.0

+   41.67%     0.00%  ctl_mboxlist  libcyrus.so.0.0.0       [.] 
cyrusdb_foreach
+   41.67%     0.00%  ctl_mboxlist  libcyrus_imap.so.0.0.0  [.] 
mboxlist_allmbox
+   41.67%     0.00%  ctl_mboxlist  ctl_mboxlist            [.] do_dump
+   34.52%     0.00%  ctl_mboxlist  [kernel.kallsyms]       [k] 
entry_SYSCALL_64_fastpath
+   22.62%     5.95%  ctl_mboxlist  libc-2.22.so            [.] do_fcntl
+   22.62%     2.38%  ctl_mboxlist  libc-2.22.so            [.] 
__xstat64
...

and for 2.4 I get

+     11.63%  ctl_mboxlist  [kernel.kallsyms]  [k] kmem_cache_alloc
+      9.30%  ctl_mboxlist  [kernel.kallsyms]  [k] 
system_call_after_swapgs
+      6.98%  ctl_mboxlist  [kernel.kallsyms]  [k] kmem_cache_free
+      6.98%  ctl_mboxlist  [kernel.kallsyms]  [k] fcntl_setlk
+      4.65%  ctl_mboxlist  [kernel.kallsyms]  [k] fget_raw
+      4.65%  ctl_mboxlist  [kernel.kallsyms]  [k] locks_copy_lock
+      4.65%  ctl_mboxlist  [kernel.kallsyms]  [k] _raw_spin_lock
+      2.33%  ctl_mboxlist  libc-2.11.3.so     [.] 
__GI_____strtoll_l_internal
+      2.33%  ctl_mboxlist  libc-2.11.3.so     [.] __free
...

Has anyone an idea, why cyrusdb_foreach and mboxlist_allmbox is so 
promiment in the 3.0 recording or rather why 3.0 appears to be so much 
slower compared to 2.4?

Felix


From Willem at Offermans.Rompen.nl  Mon Feb 18 04:33:22 2019
From: Willem at Offermans.Rompen.nl (Willem Offermans)
Date: Mon, 18 Feb 2019 10:33:22 +0100
Subject: Sieve not working
In-Reply-To: <b6eeff5b17822c1ce8f080e0a3a3c8e6@sarenet.es>
References: <eb2fa1ec-3e14-6315-1467-438349b0bb9a@bordengrammar.kent.sch.uk>
 <b6eeff5b17822c1ce8f080e0a3a3c8e6@sarenet.es>
Message-ID: <C6BDF808-73E3-4474-8F49-3531FC0E0D1D@Offermans.Rompen.nl>

Dear Egoitz and Cyrus friends,

Check if sieve is actually listening at port 2000 as well.


Wiel Offermans
Willem at Offermans.Rompen.nl




> On 18 Feb 2019, at 09:34, egoitz at sarenet.es wrote:
> 
> Hi!
> 
> 
> 
> Could you try enabling local6.debug channel in syslog, so that you could see additional Sieve debugging information?. Can you then post that log?
> 
> 
> 
> Cheers!
> 
>  
> 
> 
> El 2019-02-15 12:32, J Pilfold-Bagwell escribi?:
> 
>> Hi All,
>> 
>> I have a Centos 7 box running with the latest default cyrus install from the Centos 7 repo, i.e. cyrus-imapd-2.4.17-13.el7.x86_64 .
>> 
>> The problem I have is that sieve doesn't seem to pay any attention to the scripts.  I have sieve running, I can successfully log in to it using sieveshell, create, upload and activate scripts, but they don't seem to be applied to the incoming mail.  First I was trying the vacation and reject scripts so checked that the correct sendmail is in use but it fails on fileinto as well.
>> 
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> 
>> imapd.conf looks like this:
>> 
>> [root at mail admin]# cat /etc/imapd.conf
>> configdirectory: /var/lib/imap
>> partition-default: /var/spool/imap
>> admins: cyradmin
>> sieve_admins: cyradmin
>> sievedir: /var/lib/imap/sieve
>> sendmail: /usr/sbin/sendmail
>> hashimapspool: true
>> sasl_pwcheck_method: auxprop
>> sasl_auxprop_plugin: sasldb
>> sasl_mech_list: PLAIN LOGIN CRAM-MD5 DIGEST-MD5
>> allowplaintext: yes
>> allowusermoves: yes
>> defaultdomain: mail
>> lmtp_downcase_rcpt: yes
>> 
>> tls_cert_file: /etc/ssl/certs/cyrus-imapd/newcert.pem
>> tls_key_file: /etc/ssl/certs/cyrus-imapd/newkey.pem
>> tls_ca_file: /etc/ssl/certs/cyrus-imapd/cacert.pem
>> tls_ca_path: /etc/ssl/certscyrus-imapd
>> 
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> 
>> cyrus.conf:
>> 
>> # standard standalone server implementation
>> 
>> START {
>>   # do not delete this entry!
>>   recover    cmd="ctl_cyrusdb -r"
>> 
>>   # this is only necessary if using idled for IMAP IDLE
>>   idled        cmd="idled"
>> }
>> 
>> # UNIX sockets start with a slash and are put into /var/lib/imap/sockets
>> SERVICES {
>>   # add or remove based on preferences
>>   imap        cmd="imapd" listen="imap" prefork=5
>>   imaps        cmd="imapd -s" listen="imaps" prefork=1
>> #  pop3        cmd="pop3d" listen="pop3" prefork=3
>> #  pop3s        cmd="pop3d -s" listen="pop3s" prefork=1
>>   sieve        cmd="timsieved" listen="0.0.0.0:2000" prefork=0
>>   sieve         cmd="timsieved" listen="0.0.0.0:4190" prefork=0
>> #  managesieve   cmd="timsieved" listen="localhost:4190" prefork=0
>> 
>>   # these are only necessary if receiving/exporting usenet via NNTP
>> #  nntp        cmd="nntpd" listen="nntp" prefork=3
>> #  nntps        cmd="nntpd -s" listen="nntps" prefork=1
>> 
>>   # at least one LMTP is required for delivery
>> #  lmtp        cmd="lmtpd" listen="lmtp" prefork=0
>>   lmtpunix    cmd="lmtpd" listen="/var/lib/imap/socket/lmtp" prefork=1
>> 
>>   # this is only necessary if using notifications
>> #  notify    cmd="notifyd" listen="/var/lib/imap/socket/notify" proto="udp" prefork=1
>> }
>> 
>> EVENTS {
>>   # this is required
>>   checkpoint    cmd="ctl_cyrusdb -c" period=30
>> 
>>   # this is only necessary if using duplicate delivery suppression,
>>   # Sieve or NNTP
>>   delprune    cmd="cyr_expire -E 3" at=0400
>> 
>>   # this is only necessary if caching TLS sessions
>>   tlsprune    cmd="tls_prune" at=0400
>> 
>>   # reindex changed mailboxes (fulltext) approximately every three hours
>>   squatter1       cmd="/usr/bin/ionice -c idle /usr/lib/cyrus/bin/squatter -s" period=180
>> 
>>   # reindex all mailboxes (fulltext) daily
>>   squattera       cmd="/usr/lib/cyrus/bin/squatter" at=0117
>> }
>> 
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> 
>> sieveshell logs in fine:
>> 
>> [root at mail admin]# sieveshell --authname=cyradmin --user=testuser1 localhost
>> connecting to localhost
>> Please enter your password:
>>> 
>>> list
>> mail
>> sieve-test  <- active script
>> 
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~telnet
>> 
>> Telnet login provides:
>> 
>> [root at mail admin]# telnet 192.168.0.6 4190
>> Trying 192.168.0.6...
>> Connected to 192.168.0.6.
>> Escape character is '^]'.
>> "IMPLEMENTATION" "Cyrus timsieved v2.4.17-Fedora-RPM-2.4.17-13.el7"
>> "SASL" "PLAIN LOGIN CRAM-MD5 DIGEST-MD5"
>> "SIEVE" "comparator-i;ascii-numeric fileinto reject vacation imapflags notify envelope relational regex subaddress copy"
>> "STARTTLS"
>> "UNAUTHENTICATE"
>> OK
>> 
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> 
>> And this works for both port 2000 and 4190 on all interfaces. LMTP is in use but somewhere, they aren't talking.
>> 
>> Does anyone have any troubleshooting tips they can feed me or, can anyone see a glaringly obvious error I've made because it's all gone a bit wood for the trees here.
>> 
>> The logs are huge but if you'd like to see the contents, let me know what you'd like it grep'd for and I'll provide.
>> 
>> Thanks,
>> 
>> Julian
>> 
>> -- 
>> This email is from Borden Grammar School Trust.
>> 
>> This email, together with any files transmitted with it, is confidential, and is intended solely for the use of the individual or entity to whom it is addressed. Any unauthorised dissemination or copying of this email or its attachments, and any use or disclosure of any information contained in them, is strictly prohibited, and may also be illegal. If you are not the intended recipient you may not use, disclose, distribute, copy, print or relay this email.
>> 
>> Please note that any views expressed by an individual within this email, do not necessarily reflect the views of the Borden Grammar School Trust.
>> 
>> Borden Grammar School Trust has taken reasonable precautions to ensure no viruses are present in this email.  The Academy cannot accept responsibility for any loss or damage arising from the use of this email and/or files attached.
>> 
>> Registered Office: Borden Grammar School, Avenue of Remembrance, Sittingbourne, Kent ME10 4DB
>> 
>> Registered in England: 07827591
>> 
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190218/e9349569/attachment-0001.html>

From michael.menge at zdv.uni-tuebingen.de  Mon Feb 18 07:58:47 2019
From: michael.menge at zdv.uni-tuebingen.de (Michael Menge)
Date: Mon, 18 Feb 2019 13:58:47 +0100
Subject: another problem with conversations db
In-Reply-To: <c41ab3d3-f6cf-43d4-8127-98253338f701@www.fastmail.com>
References: <20190124180232.Horde.Tya11Hrse_nHFrp7Zj39dP4@webmail.uni-tuebingen.de>
 <363a2149-1b36-4dc0-aca5-b121c8ae3266@beta.fastmail.com>
 <20190125095538.Horde.BZLLa6BdReb8ZHa6b9Vxg1r@webmail.uni-tuebingen.de>
 <5d089935-a59f-45d7-a50c-cec40e23082c@beta.fastmail.com>
 <20190131093559.Horde.Vu9F7oPMtLNy30qgYXzWgKL@webmail.uni-tuebingen.de>
 <c02770dd-4238-4aab-a034-bd447b58cd06@www.fastmail.com>
 <20190204101712.Horde.33ManGeIHA8D5TVzSiQ7825@webmail.uni-tuebingen.de>
 <02bb719c-27ce-4e62-84e8-f11a32adb3dc@www.fastmail.com>
 <20190204120013.Horde.6MsNbwrJRdIBADBt9I1z3QP@webmail.uni-tuebingen.de>
 <bfac9abf-ad3b-42ad-a668-0e57961d5a2b@www.fastmail.com>
 <20190212143910.Horde.xEe2bT5UT3887Oel1PIjtTT@webmail.uni-tuebingen.de>
 <c41ab3d3-f6cf-43d4-8127-98253338f701@www.fastmail.com>
Message-ID: <20190218135847.Horde.8xFTYM8yl2O1DQk_wXpNart@webmail.uni-tuebingen.de>


Quoting Bron Gondwana <brong at fastmailteam.com>:

>
> ctl_conversationsdb -b should update the cid. BUT - if you're  
> running old mailboxes which have a format which doesn't support  
> saving the CID, that would for sure break things!
>

I did run "reconstruct -V max" during the upgrade. So all mailboxes  
should have been updated but
checking with

mbexamine user/* | grep "  Minor Version: 12"

I found some mailboxes that where not updated correctly. I didn't see  
any errors during the
initial upgrade, but I don't have the logs anymore so i don't know  
what happened.
Any idea what could lead to "reconstruct -V max" failing for some mailboxes?

I did rerun "reconstruct -r -V max user/LoginID" for those users, but  
I had two accounts where
the reconstruct command didn't update the index for the INBOX of that  
account. Rerunning it
again did update it.

The rebuild of the conversation_db worked fine after the Index was updated..
Do this did resolve the endless loop in ctl_conversationsdb. I would suggest
to add a check for the index version in ctl_conversationsdb

I will monitor my log files if the IOERRORS will reappear, or if the  
wrong index version
was the root of that problem too.

@Bron Thanks for the help



--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universit?t T?bingen                   Fax.: (49) 7071/29-5912
Zentrum f?r Datenverarbeitung          mail:  
michael.menge at zdv.uni-tuebingen.de
W?chterstra?e 76
72074 T?bingen


From egoitz at sarenet.es  Mon Feb 18 09:31:48 2019
From: egoitz at sarenet.es (egoitz at sarenet.es)
Date: Mon, 18 Feb 2019 15:31:48 +0100
Subject: Xapian experience and Roundcube
Message-ID: <2fcfb2ef1e9202cdd4563183144c448d@sarenet.es>

Hi mates, 

After our upgrade of a little part of our mail service to 3.0.8 we tried
to see if Xapian was so nice as we saw in our sandbox. Should say it
certainly is :) . I though it was not working as I expected but yes, it
was totally. All the problem was in the imap client side. I was using
Roundcube as an Imap client for my testing (should say too I use
Roundcube as my only email client). If you use Roundcube with
conversation view (thread view) the search delay is far more bigger
because it tries to do something for grouping the email and takes far
more and it seems to do far more searches. If you see the list view
instead of conversation view you see all behaviour is as expected. I
have reproduced this situation... so please keep in mind this if you use
Roundcube for testing all this... 

Cheers!!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190218/46e8bb4c/attachment.html>

From egoitz at sarenet.es  Mon Feb 18 11:59:40 2019
From: egoitz at sarenet.es (egoitz at sarenet.es)
Date: Mon, 18 Feb 2019 17:59:40 +0100
Subject: Little bug in 3.0.8 when calling sync_client from cyrus.conf
Message-ID: <ce93e2961cbabbd2eefa8419bd8a960e@sarenet.es>

Hi mates, 

It seems cyrus.conf file config parser, does not parse properly
sync_client line when you specify a channel. It seems it ignores it.
When you specify a line like in start section : 

syncclient cmd="/usr/local/cyrus/sbin/sync_client -r -l optarg-n amx8c" 

It ignores the part "-n amx8c" 

I have taken a fast look to split_args() and proccess_section() but have
not figured where the issue could be (have not debbuged what could be
there...)... it seems the issue should be there because if you launch
sync_client from the shell it works as expected. But if you launch from
cyrus.conf it just runs : /usr/local/cyrus/sbin/sync_client -r -l 

I have workarounded from the shell but... it seemed to be something
there... 

Cheers!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190218/2d3a221a/attachment.html>

From egoitz at sarenet.es  Mon Feb 18 12:00:50 2019
From: egoitz at sarenet.es (egoitz at sarenet.es)
Date: Mon, 18 Feb 2019 18:00:50 +0100
Subject: Little bug in 3.0.8 when calling sync_client from cyrus.conf
In-Reply-To: <ce93e2961cbabbd2eefa8419bd8a960e@sarenet.es>
References: <ce93e2961cbabbd2eefa8419bd8a960e@sarenet.es>
Message-ID: <6e864b0c91e7992e755c63d65b140e43@sarenet.es>

Sorry I meant 

syncclient cmd="/usr/local/cyrus/sbin/sync_client -r -l -n amx8c" 

as line for launching it in START section. 

Cheers!

El 2019-02-18 17:59, egoitz at sarenet.es escribi?:

> Hi mates, 
> 
> It seems cyrus.conf file config parser, does not parse properly sync_client line when you specify a channel. It seems it ignores it. When you specify a line like in start section : 
> 
> syncclient cmd="/usr/local/cyrus/sbin/sync_client -r -l optarg-n amx8c" 
> 
> It ignores the part "-n amx8c" 
> 
> I have taken a fast look to split_args() and proccess_section() but have not figured where the issue could be (have not debbuged what could be there...)... it seems the issue should be there because if you launch sync_client from the shell it works as expected. But if you launch from cyrus.conf it just runs : /usr/local/cyrus/sbin/sync_client -r -l 
> 
> I have workarounded from the shell but... it seemed to be something there... 
> 
> Cheers!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190218/699aa8d6/attachment.html>

From egoitz at sarenet.es  Mon Feb 18 12:14:33 2019
From: egoitz at sarenet.es (egoitz at sarenet.es)
Date: Mon, 18 Feb 2019 18:14:33 +0100
Subject: Little bug in 3.0.8 when calling sync_client from cyrus.conf
In-Reply-To: <6e864b0c91e7992e755c63d65b140e43@sarenet.es>
References: <ce93e2961cbabbd2eefa8419bd8a960e@sarenet.es>
 <6e864b0c91e7992e755c63d65b140e43@sarenet.es>
Message-ID: <920f8c185d27eaa7f40a0b318cdf4148@sarenet.es>

I have to do some more checks.... 

I think created the proccess properly as it has done in my testing env
but, it didn't run the amx8c channel log... I remember.... I'll try it
slightly more... and tell the real issue.... 

Sorry for the noise mates.. 

El 2019-02-18 18:00, egoitz at sarenet.es escribi?:

> Sorry I meant 
> 
> syncclient cmd="/usr/local/cyrus/sbin/sync_client -r -l -n amx8c" 
> 
> as line for launching it in START section. 
> 
> Cheers!
> 
> El 2019-02-18 17:59, egoitz at sarenet.es escribi?:
> 
>> Hi mates, 
>> 
>> It seems cyrus.conf file config parser, does not parse properly sync_client line when you specify a channel. It seems it ignores it. When you specify a line like in start section : 
>> 
>> syncclient cmd="/usr/local/cyrus/sbin/sync_client -r -l optarg-n amx8c" 
>> 
>> It ignores the part "-n amx8c" 
>> 
>> I have taken a fast look to split_args() and proccess_section() but have not figured where the issue could be (have not debbuged what could be there...)... it seems the issue should be there because if you launch sync_client from the shell it works as expected. But if you launch from cyrus.conf it just runs : /usr/local/cyrus/sbin/sync_client -r -l 
>> 
>> I have workarounded from the shell but... it seemed to be something there... 
>> 
>> Cheers!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190218/3e52b4cb/attachment.html>

From egoitz at sarenet.es  Mon Feb 18 17:52:10 2019
From: egoitz at sarenet.es (egoitz at sarenet.es)
Date: Mon, 18 Feb 2019 23:52:10 +0100
Subject: Little bug in 3.0.8 when calling sync_client from cyrus.conf
In-Reply-To: <920f8c185d27eaa7f40a0b318cdf4148@sarenet.es>
References: <ce93e2961cbabbd2eefa8419bd8a960e@sarenet.es>
 <6e864b0c91e7992e755c63d65b140e43@sarenet.es>
 <920f8c185d27eaa7f40a0b318cdf4148@sarenet.es>
Message-ID: <628a3a2416ff98adba4b2e08ed0b029e@sarenet.es>

I can confirm this thread was just noise.... it works just fine, please
ignore it... 

I had to have done something wrong the past day... you know too much
work :) in a migration proccess... nights of work etc etc... :) 

sorry again mates,

El 2019-02-18 18:14, egoitz at sarenet.es escribi?:

> I have to do some more checks.... 
> 
> I think created the proccess properly as it has done in my testing env but, it didn't run the amx8c channel log... I remember.... I'll try it slightly more... and tell the real issue.... 
> 
> Sorry for the noise mates.. 
> 
> El 2019-02-18 18:00, egoitz at sarenet.es escribi?: 
> 
> Sorry I meant 
> 
> syncclient cmd="/usr/local/cyrus/sbin/sync_client -r -l -n amx8c" 
> 
> as line for launching it in START section. 
> 
> Cheers!
> 
> El 2019-02-18 17:59, egoitz at sarenet.es escribi?: 
> 
> Hi mates, 
> 
> It seems cyrus.conf file config parser, does not parse properly sync_client line when you specify a channel. It seems it ignores it. When you specify a line like in start section : 
> 
> syncclient cmd="/usr/local/cyrus/sbin/sync_client -r -l optarg-n amx8c" 
> 
> It ignores the part "-n amx8c" 
> 
> I have taken a fast look to split_args() and proccess_section() but have not figured where the issue could be (have not debbuged what could be there...)... it seems the issue should be there because if you launch sync_client from the shell it works as expected. But if you launch from cyrus.conf it just runs : /usr/local/cyrus/sbin/sync_client -r -l 
> 
> I have workarounded from the shell but... it seemed to be something there... 
> 
> Cheers!

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190218/bc4b6906/attachment.html>

From michael.menge at zdv.uni-tuebingen.de  Tue Feb 19 04:28:56 2019
From: michael.menge at zdv.uni-tuebingen.de (Michael Menge)
Date: Tue, 19 Feb 2019 10:28:56 +0100
Subject: another problem with conversations db
In-Reply-To: <20190218135847.Horde.8xFTYM8yl2O1DQk_wXpNart@webmail.uni-tuebingen.de>
References: <20190124180232.Horde.Tya11Hrse_nHFrp7Zj39dP4@webmail.uni-tuebingen.de>
 <363a2149-1b36-4dc0-aca5-b121c8ae3266@beta.fastmail.com>
 <20190125095538.Horde.BZLLa6BdReb8ZHa6b9Vxg1r@webmail.uni-tuebingen.de>
 <5d089935-a59f-45d7-a50c-cec40e23082c@beta.fastmail.com>
 <20190131093559.Horde.Vu9F7oPMtLNy30qgYXzWgKL@webmail.uni-tuebingen.de>
 <c02770dd-4238-4aab-a034-bd447b58cd06@www.fastmail.com>
 <20190204101712.Horde.33ManGeIHA8D5TVzSiQ7825@webmail.uni-tuebingen.de>
 <02bb719c-27ce-4e62-84e8-f11a32adb3dc@www.fastmail.com>
 <20190204120013.Horde.6MsNbwrJRdIBADBt9I1z3QP@webmail.uni-tuebingen.de>
 <bfac9abf-ad3b-42ad-a668-0e57961d5a2b@www.fastmail.com>
 <20190212143910.Horde.xEe2bT5UT3887Oel1PIjtTT@webmail.uni-tuebingen.de>
 <c41ab3d3-f6cf-43d4-8127-98253338f701@www.fastmail.com>
 <20190218135847.Horde.8xFTYM8yl2O1DQk_wXpNart@webmail.uni-tuebingen.de>
Message-ID: <20190219102856.Horde.BhTMG-B9QNdPnTY1IxHsFwg@webmail.uni-tuebingen.de>

Replying to myself,

I still see

IOERROR: conversations_audit on store

and

IOERROR: conversations_audit on load



Quoting Michael Menge <michael.menge at zdv.uni-tuebingen.de>:

> Quoting Bron Gondwana <brong at fastmailteam.com>:
>
>>
>> ctl_conversationsdb -b should update the cid. BUT - if you're  
>> running old mailboxes which have a format which doesn't support  
>> saving the CID, that would for sure break things!
>>
>
> I did run "reconstruct -V max" during the upgrade. So all mailboxes  
> should have been updated but
> checking with
>
> mbexamine user/* | grep "  Minor Version: 12"
>
> I found some mailboxes that where not updated correctly. I didn't  
> see any errors during the
> initial upgrade, but I don't have the logs anymore so i don't know  
> what happened.
> Any idea what could lead to "reconstruct -V max" failing for some mailboxes?
>
> I did rerun "reconstruct -r -V max user/LoginID" for those users,  
> but I had two accounts where
> the reconstruct command didn't update the index for the INBOX of  
> that account. Rerunning it
> again did update it.
>
> The rebuild of the conversation_db worked fine after the Index was updated..
> Do this did resolve the endless loop in ctl_conversationsdb. I would suggest
> to add a check for the index version in ctl_conversationsdb
>
> I will monitor my log files if the IOERRORS will reappear, or if the  
> wrong index version
> was the root of that problem too.
>
> @Bron Thanks for the help
>


--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universit?t T?bingen                   Fax.: (49) 7071/29-5912
Zentrum f?r Datenverarbeitung          mail:  
michael.menge at zdv.uni-tuebingen.de
W?chterstra?e 76
72074 T?bingen


From egoitz at sarenet.es  Tue Feb 19 06:12:21 2019
From: egoitz at sarenet.es (egoitz at sarenet.es)
Date: Tue, 19 Feb 2019 12:12:21 +0100
Subject: Question about what is replicated with Cyrus replication and what
 isn't
In-Reply-To: <caa91e620989215a9b94774bb1609df6@sarenet.es>
References: <caa91e620989215a9b94774bb1609df6@sarenet.es>
Message-ID: <d797a5747548be540f55b9ccfb031081@sarenet.es>

Hi mates, 

I have been taking a lot at the code and trying to debug this... 

I have seen there seems not to be replication aparently for neither
squatter nor cyr_expire. For instance, ipurge seems to be replicated
properly but not this way the two commented commands. I could launch
them in the slave, there's no problem with that but then, you see in the
logs the following lines : 

Feb 19 09:44:21 testserv imap[49589]: higher xconvmodseq on replica
___________________________________ - 1489 < 1496
Feb 19 11:45:14 testserv imap[99424]: higher xconvmodseq on replica
___________________________________ - 179 < 184
Feb 19 11:45:39 testserv imap[99424]: higher xconvmodseq on replica
___________________________________ - 537 < 614
Feb 19 11:45:40 testserv imap[99424]: higher modseq on replica
________________________________________ - 619 < 621
Feb 19 11:45:40 testserv imap[99424]: higher modseq on replica
________________________________________ - 625 < 627 

By what I have been reading previously, this is basically saying you
have made changes in the slave, which you shouldn't... because is that
exactly a slave.... but then... how do you handle delete_delayed and
expunge_delayed in a master/slave env?. Perhaps you should have always
expunge and delete mode to immediate in a slave always??. An later
before setting it as master, activate them again?. 

That could answer to cyr_expire question... but what about squatter and
xapian databases tier nightly movement of data and so?. 

How should all this be handled in a master/slave env?. It seems not to
be replicated with the replication protocol... so then... if it's not
but later replication throws that messages?... what's the proper way to
handle Squatter, Cyr_expire, perhaps ipurge although I think it
shouldn't in this case because ipurrge gets properly replicated due to
having a mailbox_close() call which then goes to a
mailbox_unlock_index() which does a sync_log_mailbox(). Shouldn't
Squatter and cyr_expire launch sync commands to sync log?. I think this
is not a bug because else at Fastmail you probably should have noted
that... but then I don't really know how to handle this situations
(cyr_expire, squatter) in a master/slave server.... 

Bron, Robert, Ellie mates :) could you please tell us something about
how to do it properly?.. 

Thanks a lot mates!!!

El 2019-02-17 19:22, egoitz at sarenet.es escribi?:

> Hi! 
> 
> Previously (in 2.3 and older versions), cyr_expire and ipurge actions for instance where not replicated to the slave. So, you needed to launch them in both, the master and the slave. My question is, are now replicated as mailbox replication commands?. What about commands like Squatter -F for removing in Cyrus 3 from the database non existing mails?. Or when you move data from search tiers in Xapian?. Are those actions replicated or should be launched in the master and the replica?. 
> 
> Cheers! 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190219/ba52699c/attachment-0001.html>

From egoitz at sarenet.es  Tue Feb 19 06:31:14 2019
From: egoitz at sarenet.es (egoitz at sarenet.es)
Date: Tue, 19 Feb 2019 12:31:14 +0100
Subject: Question about what is replicated with Cyrus replication and what
 isn't
In-Reply-To: <d797a5747548be540f55b9ccfb031081@sarenet.es>
References: <caa91e620989215a9b94774bb1609df6@sarenet.es>
 <d797a5747548be540f55b9ccfb031081@sarenet.es>
Message-ID: <f24e6af73b801532841617caca673163@sarenet.es>

An important note hereis that the messages described below are logged at
the slave... so perhaps it's just all fine?.... 

But in the master I see sometimes : 

Feb 19 11:45:40 testmasterserver sync_client[10654]: SYNCNOTICE:
highestmodseq higher on replica __________________, updating 623 => 630
Feb 19 11:45:40 testmasterserver sync_client[10654]: SYNCNOTICE:
highestmodseq higher on replica __________________, updating 633 => 636
Feb 19 11:45:40 testmasterserver sync_client[10654]: SYNCNOTICE:
highestmodseq higher on replica __________________, updating 631 => 638 

And some days ago I saw : 

Feb 14 15:11:20 mx7c sync_client[62663]: SYNCNOTICE: highestmodseq
higher on replica __________________________, updating 8760 => 8776
Feb 14 15:11:20 mx7c sync_client[62663]: MAILBOX received NO response:
IMAP_SYNC_CHECKSUM Checksum Failure
Feb 14 15:11:20 mx7c sync_client[62663]: CRC failure on sync for
________________________, trying full update
Feb 14 15:11:20 mx7c sync_client[62663]: SYNCNOTICE: highestmodseq
higher on replica ____________________, updating 8758 => 8778 

Perhaps all this situations are properly handled by the own replication,
and I should just launch Squatter and Cyr_expire in both master and
slave?. By the code too, is it correct too that for archiving should be
launched in both, the master and the slave?. 

Thanks mates!! 

El 2019-02-19 12:12, egoitz at sarenet.es escribi?:

> Hi mates, 
> 
> I have been taking a lot at the code and trying to debug this... 
> 
> I have seen there seems not to be replication aparently for neither squatter nor cyr_expire. For instance, ipurge seems to be replicated properly but not this way the two commented commands. I could launch them in the slave, there's no problem with that but then, you see in the logs the following lines : 
> 
> Feb 19 09:44:21 testserv imap[49589]: higher xconvmodseq on replica ___________________________________ - 1489 < 1496
> Feb 19 11:45:14 testserv imap[99424]: higher xconvmodseq on replica ___________________________________ - 179 < 184
> Feb 19 11:45:39 testserv imap[99424]: higher xconvmodseq on replica ___________________________________ - 537 < 614
> Feb 19 11:45:40 testserv imap[99424]: higher modseq on replica ________________________________________ - 619 < 621
> Feb 19 11:45:40 testserv imap[99424]: higher modseq on replica ________________________________________ - 625 < 627 
> 
> By what I have been reading previously, this is basically saying you have made changes in the slave, which you shouldn't... because is that exactly a slave.... but then... how do you handle delete_delayed and expunge_delayed in a master/slave env?. Perhaps you should have always expunge and delete mode to immediate in a slave always??. An later before setting it as master, activate them again?. 
> 
> That could answer to cyr_expire question... but what about squatter and xapian databases tier nightly movement of data and so?. 
> 
> How should all this be handled in a master/slave env?. It seems not to be replicated with the replication protocol... so then... if it's not but later replication throws that messages?... what's the proper way to handle Squatter, Cyr_expire, perhaps ipurge although I think it shouldn't in this case because ipurrge gets properly replicated due to having a mailbox_close() call which then goes to a mailbox_unlock_index() which does a sync_log_mailbox(). Shouldn't Squatter and cyr_expire launch sync commands to sync log?. I think this is not a bug because else at Fastmail you probably should have noted that... but then I don't really know how to handle this situations (cyr_expire, squatter) in a master/slave server.... 
> 
> Bron, Robert, Ellie mates :) could you please tell us something about how to do it properly?.. 
> 
> Thanks a lot mates!!!
> 
> El 2019-02-17 19:22, egoitz at sarenet.es escribi?:
> 
>> Hi! 
>> 
>> Previously (in 2.3 and older versions), cyr_expire and ipurge actions for instance where not replicated to the slave. So, you needed to launch them in both, the master and the slave. My question is, are now replicated as mailbox replication commands?. What about commands like Squatter -F for removing in Cyrus 3 from the database non existing mails?. Or when you move data from search tiers in Xapian?. Are those actions replicated or should be launched in the master and the replica?. 
>> 
>> Cheers! 
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190219/49a0c0db/attachment.html>

From kia at solvo.ru  Tue Feb 19 09:10:53 2019
From: kia at solvo.ru (Ivan Kuznetsov)
Date: Tue, 19 Feb 2019 17:10:53 +0300
Subject: Cyrus-imapd 2.4.17: processes stick on mailbox locking resulting
 in total mailsystem failure
In-Reply-To: <69869d48-3eb4-ec65-99bc-6de60505c9df@solvo.ru>
References: <50036990-66af-bfea-900d-ae9d64428c1c@solvo.ru>
 <87554ee5-c872-eea4-7ef9-cd31f984c7bf@oakton.edu>
 <69869d48-3eb4-ec65-99bc-6de60505c9df@solvo.ru>
Message-ID: <b3899975-e6da-f493-4249-8fd596183962@solvo.ru>

Hello

I have gdb'ed the locked process, backtrace is below. It seems that 
problem occurs when imapd process catch a signal when is inside malloc() 
call. The signal handler has a malloc() call too, so finally there is 
interlock between two mallocs

I have only a few time to investigate because locked processes list 
grows up dramatically. So I didn't found what the signal was. But it 
seems that there is a bug in imapd code...

(gdb) thread apply all bt

Thread 1 (Thread 0x7ff6ead5f840 (LWP 22980)):
#0  0x00007ff6e83e282c in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x00007ff6e835e7e5 in _L_lock_16773 () from /lib64/libc.so.6
#2  0x00007ff6e835b833 in malloc () from /lib64/libc.so.6
#3  0x000056193edc9910 in xzmalloc ()
#4  0x000056193edb2664 in seqset_init ()
#5  0x000056193ed821b0 in index_tellchanges ()
#6  0x000056193ed8232b in index_check ()
#7  0x000056193ed6382e in idle_update ()
#8  <signal handler called>
#9  0x00007ff6e83580f0 in _int_malloc () from /lib64/libc.so.6
#10 0x00007ff6e835b7dc in malloc () from /lib64/libc.so.6
#11 0x00007ff6e956f4b8 in CRYPTO_malloc () from /lib64/libcrypto.so.10
#12 0x00007ff6e96397ec in EVP_PKEY_CTX_dup () from /lib64/libcrypto.so.10
#13 0x00007ff6e962b360 in EVP_MD_CTX_copy_ex () from /lib64/libcrypto.so.10
#14 0x00007ff6e999ab3d in tls1_mac () from /lib64/libssl.so.10
#15 0x00007ff6e998db02 in ssl3_read_bytes () from /lib64/libssl.so.10
#16 0x00007ff6e998a644 in ssl3_read_internal () from /lib64/libssl.so.10
#17 0x000056193edb9745 in prot_fill ()
#18 0x000056193eda9ad7 in getword ()
#19 0x000056193ed670e7 in cmd_idle ()
#20 0x000056193ed7848d in cmdloop ()
#21 0x000056193ed79769 in service_main ()
#22 0x000056193ed62875 in main ()


13.12.2018 17:50, Ivan Kuznetsov ?????:
> Jonk, thank you for the idea. Somewhat looks strange as old mail server 
> worked w/o this problem 5+ years. But the system environment changed 
> dramatically, may be some filesystem quircks are significant for this 
> locks...
> 
> I will try gdb'ing the process when problem occurs once more
> 
> 13.12.2018 17:34, John Wade ?????:
>> Without running gdb on the process, I have no idea, but your problem 
>> sounds similar to something we hit a very long time ago:
>>
>> See https://www.oakton.edu/user/3/jwade/cyrus/Readme.html
>>
>> In our cases, the problem was the imapd process that was holding the 
>> lock was trying to obtain a second lock on the same file.?? What does 
>> a stack trace look like on the imapd process that is holding the lock? 
>> It would appear the lock process has changed since I last looked at 
>> this, so this may not be a help at all.
>>
>> Good luck,
>> John
>>
>>
>>
>> On 12/13/2018 5:21 AM, Ivan Kuznetsov wrote:
>>> Hello
>>>
>>> We had a company mail server under Oracle Linux 6 (a clone of RHEL6) 
>>> with cyrus-imapd 2.3.16 working for years w/o problems. There are >9M 
>>> messages in ~9500 mailboxes in two partitions. Daily mail traffic is 
>>> 20-50K messages.
>>>
>>> Some weeks ago we migrated the server to a new one under Oracle Linux 
>>> 7 with cyrus-imapd 2.4.17 (cyrus-imapd-2.4.17-13.el7.x86_64 rpm) and 
>>> now have problem. Some times a week one of imapd processes locks an 
>>> "INBOX" mailbox with corresponding 
>>> /var/lib/imap/lock/user/<user>.lock file and does not unlock it 
>>> anymore. Other imapd processes trying to access this mailbox sticks 
>>> waiting to obtain the lock. The bigger problem is that lmtpd 
>>> processes trying to deliver new mail to this mailbox hangs too. The 
>>> number of lmtpd processes is limited (maxchild=32) to limit the 
>>> server load, so free lmtpd pool become empty after a time, and mail 
>>> delivery stopsto all the mailboxes. MTA (postfix) mail queue blows up 
>>> quickly
>>>
>>> Example lsof output:
>>>
>>> lmtpd?? 8182 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 
>>> /var/lib/imap/lock/user/smilovsky.lock
>>> lmtpd?? 8183 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 
>>> /var/lib/imap/lock/user/smilovsky.lock
>>> lmtpd?? 8187 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 
>>> /var/lib/imap/lock/user/smilovsky.lock
>>> lmtpd?? 8771 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 
>>> /var/lib/imap/lock/user/smilovsky.lock
>>> lmtpd?? 8834 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 
>>> /var/lib/imap/lock/user/smilovsky.lock
>>> lmtpd?? 9123 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 
>>> /var/lib/imap/lock/user/smilovsky.lock
>>> lmtpd?? 9631 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 
>>> /var/lib/imap/lock/user/smilovsky.lock
>>> lmtpd?? 10343 cyrus?? 18uR? REG????????????? 249,0???????? 0 
>>> 202944402 /var/lib/imap/lock/user/smilovsky.lock
>>> lmtpd?? 10437 cyrus?? 18uR? REG????????????? 249,0???????? 0 
>>> 202944402 /var/lib/imap/lock/user/smilovsky.lock
>>> lmtpd?? 11411 cyrus?? 18uR? REG????????????? 249,0???????? 0 
>>> 202944402 /var/lib/imap/lock/user/smilovsky.lock
>>> lmtpd?? 11413 cyrus?? 18uR? REG????????????? 249,0???????? 0 
>>> 202944402 /var/lib/imap/lock/user/smilovsky.lock
>>> lmtpd?? 11506 cyrus?? 18uR? REG????????????? 249,0???????? 0 
>>> 202944402 /var/lib/imap/lock/user/smilovsky.lock
>>>
>>> [root at hippo4 bin]# lsof /var/lib/imap/lock/user/smilovsky.lock | grep 
>>> '^imapd'
>>> imapd??? 9132 cyrus?? 19uR? REG? 249,0??????? 0 202944402 
>>> /var/lib/imap/lock/user/smilovsky.lock
>>> imapd??? 9154 cyrus?? 19uR? REG? 249,0??????? 0 202944402 
>>> /var/lib/imap/lock/user/smilovsky.lock
>>> imapd?? 10311 cyrus?? 19uR? REG? 249,0??????? 0 202944402 
>>> /var/lib/imap/lock/user/smilovsky.lock
>>> imapd?? 10746 cyrus?? 19uR? REG? 249,0??????? 0 202944402 
>>> /var/lib/imap/lock/user/smilovsky.lock
>>> imapd?? 11843 cyrus?? 19uR? REG? 249,0??????? 0 202944402 
>>> /var/lib/imap/lock/user/smilovsky.lock
>>> imapd?? 12059 cyrus?? 19uR? REG? 249,0??????? 0 202944402 
>>> /var/lib/imap/lock/user/smilovsky.lock
>>> imapd?? 21885 cyrus?? 19uR? REG? 249,0??????? 0 202944402 
>>> /var/lib/imap/lock/user/smilovsky.lock
>>>
>>> In this case the root cause imapd process is 21885. strace shows that 
>>> the process waits on a futex forever:
>>>
>>> [root at hippo4 bin]# strace -tt -f -p 21885
>>> strace: Process 21885 attached
>>> 13:07:28.264713 futex(0x7f6dcc5ac760, FUTEX_WAIT_PRIVATE, 2, 
>>> NULL^Cstrace: Process 21885 detached
>>> ?<detached ...>
>>>
>>> Killing the process frees the lock, all the other stuck processes 
>>> runs and the problem disappears in a few seconds.
>>>
>>> Investigation shows that this process has a long-term connection from 
>>> a user MUA (Thunderbird) and deletes mail messages time-to-time. This 
>>> may be a Thunderbird 'filter' thread
>>>
>>> [root at hippo4 bin]# grep 'imap\[21885\]' /var/log/maillog
>>> [...]
>>> Dec 13 09:59:11 hippo4 imap[21885]: starttls: TLSv1.2 with cipher 
>>> DHE-RSA-AES256-SHA (256/256 bits new) no authentication
>>> Dec 13 09:59:11 hippo4 imap[21885]: login: thunder.solvo.ru 
>>> [172.16.85.69] smilovsky CRAM-MD5+TLS User logged in 
>>> SESSIONID=<hippo.solvo.ru-21885-1544684350-1>
>>> Dec 13 09:59:11 hippo4 imap[21885]: client id: "name" "Thunderbird" 
>>> "version" "52.5.2"
>>> Dec 13 09:59:11 hippo4 imap[21885]: Expunged 1 messages from 
>>> user.smilovsky
>>> Dec 13 09:59:12 hippo4 imap[21885]: Expunged 1 messages from 
>>> user.smilovsky
>>> Dec 13 09:59:12 hippo4 imap[21885]: Expunged 1 messages from 
>>> user.smilovsky
>>> Dec 13 09:59:12 hippo4 imap[21885]: Expunged 1 messages from 
>>> user.smilovsky
>>> Dec 13 10:01:45 hippo4 imap[21885]: Expunged 1 messages from 
>>> user.smilovsky
>>> Dec 13 10:01:53 hippo4 imap[21885]: Expunged 1 messages from 
>>> user.smilovsky
>>> Dec 13 10:03:17 hippo4 imap[21885]: Expunged 1 messages from 
>>> user.smilovsky
>>> Dec 13 10:09:06 hippo4 imap[21885]: Expunged 1 messages from 
>>> user.smilovsky
>>> Dec 13 10:09:19 hippo4 imap[21885]: Expunged 1 messages from 
>>> user.smilovsky
>>>
>>> Problem is sporadic, I didn't see any correlation with server load, 
>>> time of day, user name and so on
>>>
>>> Removing "maxchild=" parameter for lmtp just delay the final as the 
>>> server system resources are limited. Now I wrote a patrol script 
>>> counting locks and killing imapd if locks number is growing up. The 
>>> trouble is I can't reliably determine the root cause imapd and kill 
>>> them one-by-one
>>>
>>> Any ideas more?
>>>
>>> Ivan Kuznetsov, SOLVO ltd
>>> ----
>>> Cyrus Home Page: http://www.cyrusimap.org/
>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>> To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>
> 

-- 
? ?????????, ???? ????????
???????????? ???????????? ??????

???????? "?????"
+7(812)60-60-555
+7(495)66-83-003
+7(921)740-72-61
http://www.solvo.ru

From ellie at fastmail.com  Thu Feb 21 20:20:29 2019
From: ellie at fastmail.com (ellie timoney)
Date: Thu, 21 Feb 2019 20:20:29 -0500
Subject: =?UTF-8?Q?Re:_Cyrus-imapd_2.4.17:_processes_stick_on_mailbox_locking_res?=
 =?UTF-8?Q?ulting_in_total_mailsystem_failure?=
In-Reply-To: <b3899975-e6da-f493-4249-8fd596183962@solvo.ru>
References: <50036990-66af-bfea-900d-ae9d64428c1c@solvo.ru>
 <87554ee5-c872-eea4-7ef9-cd31f984c7bf@oakton.edu>
 <69869d48-3eb4-ec65-99bc-6de60505c9df@solvo.ru>
 <b3899975-e6da-f493-4249-8fd596183962@solvo.ru>
Message-ID: <8d6ba9fb-fb2e-4b56-96bf-47032a5980fc@www.fastmail.com>

Hi Ivan,

> #7  0x000056193ed6382e in idle_update ()

> with cyrus-imapd 2.4.17 (cyrus-imapd-2.4.17-13.el7.x86_64 rpm)

It looks like this version is old enough to still be using signal handlers in its IMAP IDLE implementation.  This is known to be a problem, and IDLE was rewritten to not use signals for 2.5 and later.

Thomas Jarosch kindly backported the rewrite for 2.4, and it has been included in releases since 2.4.19 (the current 2.4 release is 2.4.20)

Cheers,

ellie

On Wed, Feb 20, 2019, at 1:15 AM, Ivan Kuznetsov wrote:
> Hello
> 
> I have gdb'ed the locked process, backtrace is below. It seems that 
> problem occurs when imapd process catch a signal when is inside malloc() 
> call. The signal handler has a malloc() call too, so finally there is 
> interlock between two mallocs
> 
> I have only a few time to investigate because locked processes list 
> grows up dramatically. So I didn't found what the signal was. But it 
> seems that there is a bug in imapd code...
> 
> (gdb) thread apply all bt
> 
> Thread 1 (Thread 0x7ff6ead5f840 (LWP 22980)):
> #0  0x00007ff6e83e282c in __lll_lock_wait_private () from /lib64/libc.so.6
> #1  0x00007ff6e835e7e5 in _L_lock_16773 () from /lib64/libc.so.6
> #2  0x00007ff6e835b833 in malloc () from /lib64/libc.so.6
> #3  0x000056193edc9910 in xzmalloc ()
> #4  0x000056193edb2664 in seqset_init ()
> #5  0x000056193ed821b0 in index_tellchanges ()
> #6  0x000056193ed8232b in index_check ()
> #7  0x000056193ed6382e in idle_update ()
> #8  <signal handler called>
> #9  0x00007ff6e83580f0 in _int_malloc () from /lib64/libc.so.6
> #10 0x00007ff6e835b7dc in malloc () from /lib64/libc.so.6
> #11 0x00007ff6e956f4b8 in CRYPTO_malloc () from /lib64/libcrypto.so.10
> #12 0x00007ff6e96397ec in EVP_PKEY_CTX_dup () from /lib64/libcrypto.so.10
> #13 0x00007ff6e962b360 in EVP_MD_CTX_copy_ex () from /lib64/libcrypto.so.10
> #14 0x00007ff6e999ab3d in tls1_mac () from /lib64/libssl.so.10
> #15 0x00007ff6e998db02 in ssl3_read_bytes () from /lib64/libssl.so.10
> #16 0x00007ff6e998a644 in ssl3_read_internal () from /lib64/libssl.so.10
> #17 0x000056193edb9745 in prot_fill ()
> #18 0x000056193eda9ad7 in getword ()
> #19 0x000056193ed670e7 in cmd_idle ()
> #20 0x000056193ed7848d in cmdloop ()
> #21 0x000056193ed79769 in service_main ()
> #22 0x000056193ed62875 in main ()
> 
> 
> 13.12.2018 17:50, Ivan Kuznetsov ?????:
> > Jonk, thank you for the idea. Somewhat looks strange as old mail server 
> > worked w/o this problem 5+ years. But the system environment changed 
> > dramatically, may be some filesystem quircks are significant for this 
> > locks...
> > 
> > I will try gdb'ing the process when problem occurs once more
> > 
> > 13.12.2018 17:34, John Wade ?????:
> >> Without running gdb on the process, I have no idea, but your problem 
> >> sounds similar to something we hit a very long time ago:
> >>
> >> See https://www.oakton.edu/user/3/jwade/cyrus/Readme.html
> >>
> >> In our cases, the problem was the imapd process that was holding the 
> >> lock was trying to obtain a second lock on the same file.?? What does 
> >> a stack trace look like on the imapd process that is holding the lock? 
> >> It would appear the lock process has changed since I last looked at 
> >> this, so this may not be a help at all.
> >>
> >> Good luck,
> >> John
> >>
> >>
> >>
> >> On 12/13/2018 5:21 AM, Ivan Kuznetsov wrote:
> >>> Hello
> >>>
> >>> We had a company mail server under Oracle Linux 6 (a clone of RHEL6) 
> >>> with cyrus-imapd 2.3.16 working for years w/o problems. There are >9M 
> >>> messages in ~9500 mailboxes in two partitions. Daily mail traffic is 
> >>> 20-50K messages.
> >>>
> >>> Some weeks ago we migrated the server to a new one under Oracle Linux 
> >>> 7 with cyrus-imapd 2.4.17 (cyrus-imapd-2.4.17-13.el7.x86_64 rpm) and 
> >>> now have problem. Some times a week one of imapd processes locks an 
> >>> "INBOX" mailbox with corresponding 
> >>> /var/lib/imap/lock/user/<user>.lock file and does not unlock it 
> >>> anymore. Other imapd processes trying to access this mailbox sticks 
> >>> waiting to obtain the lock. The bigger problem is that lmtpd 
> >>> processes trying to deliver new mail to this mailbox hangs too. The 
> >>> number of lmtpd processes is limited (maxchild=32) to limit the 
> >>> server load, so free lmtpd pool become empty after a time, and mail 
> >>> delivery stopsto all the mailboxes. MTA (postfix) mail queue blows up 
> >>> quickly
> >>>
> >>> Example lsof output:
> >>>
> >>> lmtpd?? 8182 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 
> >>> /var/lib/imap/lock/user/smilovsky.lock
> >>> lmtpd?? 8183 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 
> >>> /var/lib/imap/lock/user/smilovsky.lock
> >>> lmtpd?? 8187 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 
> >>> /var/lib/imap/lock/user/smilovsky.lock
> >>> lmtpd?? 8771 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 
> >>> /var/lib/imap/lock/user/smilovsky.lock
> >>> lmtpd?? 8834 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 
> >>> /var/lib/imap/lock/user/smilovsky.lock
> >>> lmtpd?? 9123 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 
> >>> /var/lib/imap/lock/user/smilovsky.lock
> >>> lmtpd?? 9631 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402 
> >>> /var/lib/imap/lock/user/smilovsky.lock
> >>> lmtpd?? 10343 cyrus?? 18uR? REG????????????? 249,0???????? 0 
> >>> 202944402 /var/lib/imap/lock/user/smilovsky.lock
> >>> lmtpd?? 10437 cyrus?? 18uR? REG????????????? 249,0???????? 0 
> >>> 202944402 /var/lib/imap/lock/user/smilovsky.lock
> >>> lmtpd?? 11411 cyrus?? 18uR? REG????????????? 249,0???????? 0 
> >>> 202944402 /var/lib/imap/lock/user/smilovsky.lock
> >>> lmtpd?? 11413 cyrus?? 18uR? REG????????????? 249,0???????? 0 
> >>> 202944402 /var/lib/imap/lock/user/smilovsky.lock
> >>> lmtpd?? 11506 cyrus?? 18uR? REG????????????? 249,0???????? 0 
> >>> 202944402 /var/lib/imap/lock/user/smilovsky.lock
> >>>
> >>> [root at hippo4 bin]# lsof /var/lib/imap/lock/user/smilovsky.lock | grep 
> >>> '^imapd'
> >>> imapd??? 9132 cyrus?? 19uR? REG? 249,0??????? 0 202944402 
> >>> /var/lib/imap/lock/user/smilovsky.lock
> >>> imapd??? 9154 cyrus?? 19uR? REG? 249,0??????? 0 202944402 
> >>> /var/lib/imap/lock/user/smilovsky.lock
> >>> imapd?? 10311 cyrus?? 19uR? REG? 249,0??????? 0 202944402 
> >>> /var/lib/imap/lock/user/smilovsky.lock
> >>> imapd?? 10746 cyrus?? 19uR? REG? 249,0??????? 0 202944402 
> >>> /var/lib/imap/lock/user/smilovsky.lock
> >>> imapd?? 11843 cyrus?? 19uR? REG? 249,0??????? 0 202944402 
> >>> /var/lib/imap/lock/user/smilovsky.lock
> >>> imapd?? 12059 cyrus?? 19uR? REG? 249,0??????? 0 202944402 
> >>> /var/lib/imap/lock/user/smilovsky.lock
> >>> imapd?? 21885 cyrus?? 19uR? REG? 249,0??????? 0 202944402 
> >>> /var/lib/imap/lock/user/smilovsky.lock
> >>>
> >>> In this case the root cause imapd process is 21885. strace shows that 
> >>> the process waits on a futex forever:
> >>>
> >>> [root at hippo4 bin]# strace -tt -f -p 21885
> >>> strace: Process 21885 attached
> >>> 13:07:28.264713 futex(0x7f6dcc5ac760, FUTEX_WAIT_PRIVATE, 2, 
> >>> NULL^Cstrace: Process 21885 detached
> >>> ?<detached ...>
> >>>
> >>> Killing the process frees the lock, all the other stuck processes 
> >>> runs and the problem disappears in a few seconds.
> >>>
> >>> Investigation shows that this process has a long-term connection from 
> >>> a user MUA (Thunderbird) and deletes mail messages time-to-time. This 
> >>> may be a Thunderbird 'filter' thread
> >>>
> >>> [root at hippo4 bin]# grep 'imap\[21885\]' /var/log/maillog
> >>> [...]
> >>> Dec 13 09:59:11 hippo4 imap[21885]: starttls: TLSv1.2 with cipher 
> >>> DHE-RSA-AES256-SHA (256/256 bits new) no authentication
> >>> Dec 13 09:59:11 hippo4 imap[21885]: login: thunder.solvo.ru 
> >>> [172.16.85.69] smilovsky CRAM-MD5+TLS User logged in 
> >>> SESSIONID=<hippo.solvo.ru-21885-1544684350-1>
> >>> Dec 13 09:59:11 hippo4 imap[21885]: client id: "name" "Thunderbird" 
> >>> "version" "52.5.2"
> >>> Dec 13 09:59:11 hippo4 imap[21885]: Expunged 1 messages from 
> >>> user.smilovsky
> >>> Dec 13 09:59:12 hippo4 imap[21885]: Expunged 1 messages from 
> >>> user.smilovsky
> >>> Dec 13 09:59:12 hippo4 imap[21885]: Expunged 1 messages from 
> >>> user.smilovsky
> >>> Dec 13 09:59:12 hippo4 imap[21885]: Expunged 1 messages from 
> >>> user.smilovsky
> >>> Dec 13 10:01:45 hippo4 imap[21885]: Expunged 1 messages from 
> >>> user.smilovsky
> >>> Dec 13 10:01:53 hippo4 imap[21885]: Expunged 1 messages from 
> >>> user.smilovsky
> >>> Dec 13 10:03:17 hippo4 imap[21885]: Expunged 1 messages from 
> >>> user.smilovsky
> >>> Dec 13 10:09:06 hippo4 imap[21885]: Expunged 1 messages from 
> >>> user.smilovsky
> >>> Dec 13 10:09:19 hippo4 imap[21885]: Expunged 1 messages from 
> >>> user.smilovsky
> >>>
> >>> Problem is sporadic, I didn't see any correlation with server load, 
> >>> time of day, user name and so on
> >>>
> >>> Removing "maxchild=" parameter for lmtp just delay the final as the 
> >>> server system resources are limited. Now I wrote a patrol script 
> >>> counting locks and killing imapd if locks number is growing up. The 
> >>> trouble is I can't reliably determine the root cause imapd and kill 
> >>> them one-by-one
> >>>
> >>> Any ideas more?
> >>>
> >>> Ivan Kuznetsov, SOLVO ltd
> >>> ----
> >>> Cyrus Home Page: http://www.cyrusimap.org/
> >>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> >>> To Unsubscribe:
> >>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> >>
> > 
> 
> -- 
> ? ?????????, ???? ????????
> ???????????? ???????????? ??????
> 
> ???????? "?????"
> +7(812)60-60-555
> +7(495)66-83-003
> +7(921)740-72-61
> http://www.solvo.ru
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

From l.schimmer at cgv.tugraz.at  Fri Feb 22 07:33:26 2019
From: l.schimmer at cgv.tugraz.at (Lars Schimmer)
Date: Fri, 22 Feb 2019 13:33:26 +0100
Subject: cyrus update to 3.x, sasldb: usernot found
Message-ID: <b495bfc8-ad20-e4d7-e620-ef82e05180d0@cgv.tugraz.at>

Hi

I setup a new debian buster server with cyrus 3.x and try to migrate old
data/config from a cyrus 2.4 debian system to the new server.

Cyrus itself works, sieve works, just sasldb give me headache.

I did copy over the sasldb file from the old server, cyrus setup config
is the same, but it gives me the error:

badlogin: xyz.test.test [1.2.2.11] PLAIN [SASL(-13): user not found:
Password verification failed]
Feb 22 13:24:15 xyz cyrus/imap[7842]: SASL Password verification failed

If I issue a saslpasswd2 usern - and enter a passwd, this password works
fine.

How can I migrate the old sasldb do be useful with cyrus3.x without the
need to update all users password?

Any infos, tips, help?

(I did create the saslpasswd file >10 years ago, it still works fine
with debian 6.x and cyrus 2.4.x, but not with 3.x on debian buster).

Thank you!


MfG,
Lars Schimmer
-- 
-------------------------------------------------------------
TU Graz, Institut f?r ComputerGraphik & WissensVisualisierung
Tel: +43 316 873-5405       E-Mail: l.schimmer at cgv.tugraz.at
Fax: +43 316 873-5402       PGP-Key-ID: 0x4A9B1723



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190222/7de2ac97/attachment.sig>

From kia at solvo.ru  Fri Feb 22 10:06:03 2019
From: kia at solvo.ru (Ivan Kuznetsov)
Date: Fri, 22 Feb 2019 18:06:03 +0300
Subject: Cyrus-imapd 2.4.17: processes stick on mailbox locking resulting
 in total mailsystem failure
In-Reply-To: <8d6ba9fb-fb2e-4b56-96bf-47032a5980fc@www.fastmail.com>
References: <50036990-66af-bfea-900d-ae9d64428c1c@solvo.ru>
 <87554ee5-c872-eea4-7ef9-cd31f984c7bf@oakton.edu>
 <69869d48-3eb4-ec65-99bc-6de60505c9df@solvo.ru>
 <b3899975-e6da-f493-4249-8fd596183962@solvo.ru>
 <8d6ba9fb-fb2e-4b56-96bf-47032a5980fc@www.fastmail.com>
Message-ID: <dace03ae-0aca-7be7-da74-d86ad7db73b9@solvo.ru>

Hi Ellie

Thanks a lot, I will try to build and test 2.4.20

22.02.2019 04:20, ellie timoney ?????:
> Hi Ivan,
> 
>> #7  0x000056193ed6382e in idle_update ()
> 
>> with cyrus-imapd 2.4.17 (cyrus-imapd-2.4.17-13.el7.x86_64 rpm)
> 
> It looks like this version is old enough to still be using signal handlers in its IMAP IDLE implementation.  This is known to be a problem, and IDLE was rewritten to not use signals for 2.5 and later.
> 
> Thomas Jarosch kindly backported the rewrite for 2.4, and it has been included in releases since 2.4.19 (the current 2.4 release is 2.4.20)
> 
> Cheers,
> 
> ellie
> 
> On Wed, Feb 20, 2019, at 1:15 AM, Ivan Kuznetsov wrote:
>> Hello
>>
>> I have gdb'ed the locked process, backtrace is below. It seems that
>> problem occurs when imapd process catch a signal when is inside malloc()
>> call. The signal handler has a malloc() call too, so finally there is
>> interlock between two mallocs
>>
>> I have only a few time to investigate because locked processes list
>> grows up dramatically. So I didn't found what the signal was. But it
>> seems that there is a bug in imapd code...
>>
>> (gdb) thread apply all bt
>>
>> Thread 1 (Thread 0x7ff6ead5f840 (LWP 22980)):
>> #0  0x00007ff6e83e282c in __lll_lock_wait_private () from /lib64/libc.so.6
>> #1  0x00007ff6e835e7e5 in _L_lock_16773 () from /lib64/libc.so.6
>> #2  0x00007ff6e835b833 in malloc () from /lib64/libc.so.6
>> #3  0x000056193edc9910 in xzmalloc ()
>> #4  0x000056193edb2664 in seqset_init ()
>> #5  0x000056193ed821b0 in index_tellchanges ()
>> #6  0x000056193ed8232b in index_check ()
>> #7  0x000056193ed6382e in idle_update ()
>> #8  <signal handler called>
>> #9  0x00007ff6e83580f0 in _int_malloc () from /lib64/libc.so.6
>> #10 0x00007ff6e835b7dc in malloc () from /lib64/libc.so.6
>> #11 0x00007ff6e956f4b8 in CRYPTO_malloc () from /lib64/libcrypto.so.10
>> #12 0x00007ff6e96397ec in EVP_PKEY_CTX_dup () from /lib64/libcrypto.so.10
>> #13 0x00007ff6e962b360 in EVP_MD_CTX_copy_ex () from /lib64/libcrypto.so.10
>> #14 0x00007ff6e999ab3d in tls1_mac () from /lib64/libssl.so.10
>> #15 0x00007ff6e998db02 in ssl3_read_bytes () from /lib64/libssl.so.10
>> #16 0x00007ff6e998a644 in ssl3_read_internal () from /lib64/libssl.so.10
>> #17 0x000056193edb9745 in prot_fill ()
>> #18 0x000056193eda9ad7 in getword ()
>> #19 0x000056193ed670e7 in cmd_idle ()
>> #20 0x000056193ed7848d in cmdloop ()
>> #21 0x000056193ed79769 in service_main ()
>> #22 0x000056193ed62875 in main ()
>>
>>
>> 13.12.2018 17:50, Ivan Kuznetsov ?????:
>>> Jonk, thank you for the idea. Somewhat looks strange as old mail server
>>> worked w/o this problem 5+ years. But the system environment changed
>>> dramatically, may be some filesystem quircks are significant for this
>>> locks...
>>>
>>> I will try gdb'ing the process when problem occurs once more
>>>
>>> 13.12.2018 17:34, John Wade ?????:
>>>> Without running gdb on the process, I have no idea, but your problem
>>>> sounds similar to something we hit a very long time ago:
>>>>
>>>> See https://www.oakton.edu/user/3/jwade/cyrus/Readme.html
>>>>
>>>> In our cases, the problem was the imapd process that was holding the
>>>> lock was trying to obtain a second lock on the same file.?? What does
>>>> a stack trace look like on the imapd process that is holding the lock?
>>>> It would appear the lock process has changed since I last looked at
>>>> this, so this may not be a help at all.
>>>>
>>>> Good luck,
>>>> John
>>>>
>>>>
>>>>
>>>> On 12/13/2018 5:21 AM, Ivan Kuznetsov wrote:
>>>>> Hello
>>>>>
>>>>> We had a company mail server under Oracle Linux 6 (a clone of RHEL6)
>>>>> with cyrus-imapd 2.3.16 working for years w/o problems. There are >9M
>>>>> messages in ~9500 mailboxes in two partitions. Daily mail traffic is
>>>>> 20-50K messages.
>>>>>
>>>>> Some weeks ago we migrated the server to a new one under Oracle Linux
>>>>> 7 with cyrus-imapd 2.4.17 (cyrus-imapd-2.4.17-13.el7.x86_64 rpm) and
>>>>> now have problem. Some times a week one of imapd processes locks an
>>>>> "INBOX" mailbox with corresponding
>>>>> /var/lib/imap/lock/user/<user>.lock file and does not unlock it
>>>>> anymore. Other imapd processes trying to access this mailbox sticks
>>>>> waiting to obtain the lock. The bigger problem is that lmtpd
>>>>> processes trying to deliver new mail to this mailbox hangs too. The
>>>>> number of lmtpd processes is limited (maxchild=32) to limit the
>>>>> server load, so free lmtpd pool become empty after a time, and mail
>>>>> delivery stopsto all the mailboxes. MTA (postfix) mail queue blows up
>>>>> quickly
>>>>>
>>>>> Example lsof output:
>>>>>
>>>>> lmtpd?? 8182 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402
>>>>> /var/lib/imap/lock/user/smilovsky.lock
>>>>> lmtpd?? 8183 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402
>>>>> /var/lib/imap/lock/user/smilovsky.lock
>>>>> lmtpd?? 8187 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402
>>>>> /var/lib/imap/lock/user/smilovsky.lock
>>>>> lmtpd?? 8771 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402
>>>>> /var/lib/imap/lock/user/smilovsky.lock
>>>>> lmtpd?? 8834 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402
>>>>> /var/lib/imap/lock/user/smilovsky.lock
>>>>> lmtpd?? 9123 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402
>>>>> /var/lib/imap/lock/user/smilovsky.lock
>>>>> lmtpd?? 9631 cyrus?? 18uR? REG????????????? 249,0???????? 0 202944402
>>>>> /var/lib/imap/lock/user/smilovsky.lock
>>>>> lmtpd?? 10343 cyrus?? 18uR? REG????????????? 249,0???????? 0
>>>>> 202944402 /var/lib/imap/lock/user/smilovsky.lock
>>>>> lmtpd?? 10437 cyrus?? 18uR? REG????????????? 249,0???????? 0
>>>>> 202944402 /var/lib/imap/lock/user/smilovsky.lock
>>>>> lmtpd?? 11411 cyrus?? 18uR? REG????????????? 249,0???????? 0
>>>>> 202944402 /var/lib/imap/lock/user/smilovsky.lock
>>>>> lmtpd?? 11413 cyrus?? 18uR? REG????????????? 249,0???????? 0
>>>>> 202944402 /var/lib/imap/lock/user/smilovsky.lock
>>>>> lmtpd?? 11506 cyrus?? 18uR? REG????????????? 249,0???????? 0
>>>>> 202944402 /var/lib/imap/lock/user/smilovsky.lock
>>>>>
>>>>> [root at hippo4 bin]# lsof /var/lib/imap/lock/user/smilovsky.lock | grep
>>>>> '^imapd'
>>>>> imapd??? 9132 cyrus?? 19uR? REG? 249,0??????? 0 202944402
>>>>> /var/lib/imap/lock/user/smilovsky.lock
>>>>> imapd??? 9154 cyrus?? 19uR? REG? 249,0??????? 0 202944402
>>>>> /var/lib/imap/lock/user/smilovsky.lock
>>>>> imapd?? 10311 cyrus?? 19uR? REG? 249,0??????? 0 202944402
>>>>> /var/lib/imap/lock/user/smilovsky.lock
>>>>> imapd?? 10746 cyrus?? 19uR? REG? 249,0??????? 0 202944402
>>>>> /var/lib/imap/lock/user/smilovsky.lock
>>>>> imapd?? 11843 cyrus?? 19uR? REG? 249,0??????? 0 202944402
>>>>> /var/lib/imap/lock/user/smilovsky.lock
>>>>> imapd?? 12059 cyrus?? 19uR? REG? 249,0??????? 0 202944402
>>>>> /var/lib/imap/lock/user/smilovsky.lock
>>>>> imapd?? 21885 cyrus?? 19uR? REG? 249,0??????? 0 202944402
>>>>> /var/lib/imap/lock/user/smilovsky.lock
>>>>>
>>>>> In this case the root cause imapd process is 21885. strace shows that
>>>>> the process waits on a futex forever:
>>>>>
>>>>> [root at hippo4 bin]# strace -tt -f -p 21885
>>>>> strace: Process 21885 attached
>>>>> 13:07:28.264713 futex(0x7f6dcc5ac760, FUTEX_WAIT_PRIVATE, 2,
>>>>> NULL^Cstrace: Process 21885 detached
>>>>>  ?<detached ...>
>>>>>
>>>>> Killing the process frees the lock, all the other stuck processes
>>>>> runs and the problem disappears in a few seconds.
>>>>>
>>>>> Investigation shows that this process has a long-term connection from
>>>>> a user MUA (Thunderbird) and deletes mail messages time-to-time. This
>>>>> may be a Thunderbird 'filter' thread
>>>>>
>>>>> [root at hippo4 bin]# grep 'imap\[21885\]' /var/log/maillog
>>>>> [...]
>>>>> Dec 13 09:59:11 hippo4 imap[21885]: starttls: TLSv1.2 with cipher
>>>>> DHE-RSA-AES256-SHA (256/256 bits new) no authentication
>>>>> Dec 13 09:59:11 hippo4 imap[21885]: login: thunder.solvo.ru
>>>>> [172.16.85.69] smilovsky CRAM-MD5+TLS User logged in
>>>>> SESSIONID=<hippo.solvo.ru-21885-1544684350-1>
>>>>> Dec 13 09:59:11 hippo4 imap[21885]: client id: "name" "Thunderbird"
>>>>> "version" "52.5.2"
>>>>> Dec 13 09:59:11 hippo4 imap[21885]: Expunged 1 messages from
>>>>> user.smilovsky
>>>>> Dec 13 09:59:12 hippo4 imap[21885]: Expunged 1 messages from
>>>>> user.smilovsky
>>>>> Dec 13 09:59:12 hippo4 imap[21885]: Expunged 1 messages from
>>>>> user.smilovsky
>>>>> Dec 13 09:59:12 hippo4 imap[21885]: Expunged 1 messages from
>>>>> user.smilovsky
>>>>> Dec 13 10:01:45 hippo4 imap[21885]: Expunged 1 messages from
>>>>> user.smilovsky
>>>>> Dec 13 10:01:53 hippo4 imap[21885]: Expunged 1 messages from
>>>>> user.smilovsky
>>>>> Dec 13 10:03:17 hippo4 imap[21885]: Expunged 1 messages from
>>>>> user.smilovsky
>>>>> Dec 13 10:09:06 hippo4 imap[21885]: Expunged 1 messages from
>>>>> user.smilovsky
>>>>> Dec 13 10:09:19 hippo4 imap[21885]: Expunged 1 messages from
>>>>> user.smilovsky
>>>>>
>>>>> Problem is sporadic, I didn't see any correlation with server load,
>>>>> time of day, user name and so on
>>>>>
>>>>> Removing "maxchild=" parameter for lmtp just delay the final as the
>>>>> server system resources are limited. Now I wrote a patrol script
>>>>> counting locks and killing imapd if locks number is growing up. The
>>>>> trouble is I can't reliably determine the root cause imapd and kill
>>>>> them one-by-one
>>>>>
>>>>> Any ideas more?
>>>>>
>>>>> Ivan Kuznetsov, SOLVO ltd
>>>>> ----
>>>>> Cyrus Home Page: http://www.cyrusimap.org/
>>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>>>> To Unsubscribe:
>>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>>>
>>>
>>
>> -- 
>> ? ?????????, ???? ????????
>> ???????????? ???????????? ??????
>>
>> ???????? "?????"
>> +7(812)60-60-555
>> +7(495)66-83-003
>> +7(921)740-72-61
>> http://www.solvo.ru
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 

-- 
? ?????????, ???? ????????
???????????? ???????????? ??????

???????? "?????"
+7(812)60-60-555
+7(495)66-83-003
+7(921)740-72-61
http://www.solvo.ru

From simon.matter at invoca.ch  Fri Feb 22 10:20:04 2019
From: simon.matter at invoca.ch (Simon Matter)
Date: Fri, 22 Feb 2019 16:20:04 +0100
Subject: Cyrus-imapd 2.4.17: processes stick on mailbox locking resulting
 in total mailsystem failure
In-Reply-To: <dace03ae-0aca-7be7-da74-d86ad7db73b9@solvo.ru>
References: <50036990-66af-bfea-900d-ae9d64428c1c@solvo.ru>
 <87554ee5-c872-eea4-7ef9-cd31f984c7bf@oakton.edu>
 <69869d48-3eb4-ec65-99bc-6de60505c9df@solvo.ru>
 <b3899975-e6da-f493-4249-8fd596183962@solvo.ru>
 <8d6ba9fb-fb2e-4b56-96bf-47032a5980fc@www.fastmail.com>
 <dace03ae-0aca-7be7-da74-d86ad7db73b9@solvo.ru>
Message-ID: <2f7373fb3eaae4627f35ff749871c275.squirrel@webmail.bi.invoca.ch>

> Hi Ellie
>
> Thanks a lot, I will try to build and test 2.4.20

Maybe try this:

http://www.invoca.ch/pub/packages/cyrus-imapd/RPMS/ils-7/SRPMS/cyrus-imapd-2.4.20-2.el7.src.rpm

Regards,
Simon


From stephane.branchoux at univ-perp.fr  Fri Feb 22 13:41:15 2019
From: stephane.branchoux at univ-perp.fr (=?utf-8?Q?St=C3=A9phane_Branchoux?=)
Date: Fri, 22 Feb 2019 19:41:15 +0100
Subject: Cyrus imap and identity theft
Message-ID: <545F8DAC-C29B-4BE9-9ADA-E2A8250A3C93@univ-perp.fr>

Hello,

Each week , few users respond to phishing mails.
I use rules on firewalls, DNS filters, training program for users , anti spam products , anti virus ?.

I am looking for a way or tools to reduce identity theft on my Cyrus imap server.
For example , scripts to geo localise ip requests , detect and reject bad connexions  ?
Is it possible to authorize few devices for a user and reject other devices  ?

Which tools do you use on your Cyrus imap servers to protect them ?

Many thanks in advance

-- 
St?phane BRANCHOUX 
DSI de l'Universit? de Perpignan. 
Syst?mes / R?seaux / RSSI

mailto:stephane.branchoux at univ-perp.fr
 
04 68 66 21 24 / 07 60 73 38 42




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190222/4a5d696c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3552 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190222/4a5d696c/attachment.p7s>

From l.schimmer at cgv.tugraz.at  Mon Feb 25 05:45:42 2019
From: l.schimmer at cgv.tugraz.at (Lars Schimmer)
Date: Mon, 25 Feb 2019 11:45:42 +0100
Subject: Best way to auth cyrus 3.x to an AD domain setup
Message-ID: <81fed929-5ad6-f1a8-a2de-9d40ef75db74@cgv.tugraz.at>

Hi

Ok, after sasldb2 file is not good anymore, I want to ask user passwords
from our AD Domain setup.

I had a short search and I did find several methosd to let cyrus3 ask
for users/pwasswords from a AD server, but all are kinda old.

E.g. using krb5 service principle in win server 2008, or just using LDAp
against the server.

What is the preferred, easy to use method nowadays, any docs available?

Or how do I use sasl to save passwords encrypted with hash on local
harddrive?

Thank you.

MfG,
Lars Schimmer
-- 
-------------------------------------------------------------
TU Graz, Institut f?r ComputerGraphik & WissensVisualisierung
Tel: +43 316 873-5405       E-Mail: l.schimmer at cgv.tugraz.at
Fax: +43 316 873-5402       PGP-Key-ID: 0x4A9B1723



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190225/29bf6fbb/attachment.sig>

From dwhite at olp.net  Mon Feb 25 09:17:34 2019
From: dwhite at olp.net (Dan White)
Date: Mon, 25 Feb 2019 08:17:34 -0600
Subject: Best way to auth cyrus 3.x to an AD domain setup
In-Reply-To: <81fed929-5ad6-f1a8-a2de-9d40ef75db74@cgv.tugraz.at>
References: <81fed929-5ad6-f1a8-a2de-9d40ef75db74@cgv.tugraz.at>
Message-ID: <20190225141734.GF1945@dan.olp.net>

On 02/25/19?11:45?+0100, Lars Schimmer wrote:
>Ok, after sasldb2 file is not good anymore, I want to ask user passwords
>from our AD Domain setup.
>
>I had a short search and I did find several methosd to let cyrus3 ask
>for users/pwasswords from a AD server, but all are kinda old.
>
>E.g. using krb5 service principle in win server 2008, or just using LDAp
>against the server.
>
>What is the preferred, easy to use method nowadays, any docs available?
>
>Or how do I use sasl to save passwords encrypted with hash on local
>harddrive?

https://www.openldap.org/lists/openldap-technical/201106/msg00198.html

In imapd.conf:

sasl_pwcheck_method: saslauthd

From vladislav.kurz at webstep.net  Mon Feb 25 09:36:13 2019
From: vladislav.kurz at webstep.net (Vladislav Kurz)
Date: Mon, 25 Feb 2019 15:36:13 +0100
Subject: Best way to auth cyrus 3.x to an AD domain setup
In-Reply-To: <20190225141734.GF1945@dan.olp.net>
References: <81fed929-5ad6-f1a8-a2de-9d40ef75db74@cgv.tugraz.at>
 <20190225141734.GF1945@dan.olp.net>
Message-ID: <7e6cd9c4-2c4c-07a9-5f7d-28e980f45777@webstep.net>

On 25/02/2019 15:17, Dan White wrote:
> On 02/25/19?11:45?+0100, Lars Schimmer wrote:
>> Ok, after sasldb2 file is not good anymore, I want to ask user passwords
>> from our AD Domain setup.
>>
>> I had a short search and I did find several methosd to let cyrus3 ask
>> for users/pwasswords from a AD server, but all are kinda old.
>>
>> E.g. using krb5 service principle in win server 2008, or just using LDAp
>> against the server.
>>
>> What is the preferred, easy to use method nowadays, any docs available?

Hello,

I have several cyrus 2.x installation authenticating to AD, like this:

cyrus -> saslauthd -> PAM -> kerberos -> AD.


-- 
Best Regards
        Vladislav Kurz

From l.schimmer at cgv.tugraz.at  Tue Feb 26 05:35:21 2019
From: l.schimmer at cgv.tugraz.at (Lars Schimmer)
Date: Tue, 26 Feb 2019 11:35:21 +0100
Subject: Best way to auth cyrus 3.x to an AD domain setup
In-Reply-To: <20190225141734.GF1945@dan.olp.net>
References: <81fed929-5ad6-f1a8-a2de-9d40ef75db74@cgv.tugraz.at>
 <20190225141734.GF1945@dan.olp.net>
Message-ID: <9feffd69-ad95-fe4c-27ec-612c8df036bf@cgv.tugraz.at>

On 2/25/19 3:17 PM, Dan White wrote:
> On 02/25/19?11:45?+0100, Lars Schimmer wrote:
>> Ok, after sasldb2 file is not good anymore, I want to ask user passwords
>> from our AD Domain setup.
>>
>> I had a short search and I did find several methosd to let cyrus3 ask
>> for users/pwasswords from a AD server, but all are kinda old.
>>
>> E.g. using krb5 service principle in win server 2008, or just using LDAp
>> against the server.
>>
>> What is the preferred, easy to use method nowadays, any docs available?
>>
>> Or how do I use sasl to save passwords encrypted with hash on local
>> harddrive?
> 
> https://www.openldap.org/lists/openldap-technical/201106/msg00198.html
> 
> In imapd.conf:
> 
> sasl_pwcheck_method: saslauthd

Tried, but passwd check bad.
With ldapsearch I get a positiv feedback, but on imap login I get:
SASL(-13): authentication failure: checkpass failed

And no simple way to debug :-(


MfG,
Lars Schimmer
-- 
-------------------------------------------------------------
TU Graz, Institut f?r ComputerGraphik & WissensVisualisierung
Tel: +43 316 873-5405       E-Mail: l.schimmer at cgv.tugraz.at
Fax: +43 316 873-5402       PGP-Key-ID: 0x4A9B1723



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190226/86c6fb47/attachment.sig>

From jpilfold-bagwell at bordengrammar.kent.sch.uk  Tue Feb 26 05:36:31 2019
From: jpilfold-bagwell at bordengrammar.kent.sch.uk (J Pilfold-Bagwell)
Date: Tue, 26 Feb 2019 10:36:31 +0000
Subject: Sieve not working
In-Reply-To: <C6BDF808-73E3-4474-8F49-3531FC0E0D1D@Offermans.Rompen.nl>
References: <eb2fa1ec-3e14-6315-1467-438349b0bb9a@bordengrammar.kent.sch.uk>
 <b6eeff5b17822c1ce8f080e0a3a3c8e6@sarenet.es>
 <C6BDF808-73E3-4474-8F49-3531FC0E0D1D@Offermans.Rompen.nl>
Message-ID: <446a34c6-1129-6838-d379-d43d25d12cc3@bordengrammar.kent.sch.uk>

Hi Willem,

Once I found that you can run sieve on ports 200 and 4190 at the same 
time, I set it up just in case Cyrus was talking to only one of the ports.

If I telnet in, I get this response.

[root at mail rules]# telnet localhost 2000
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
"IMPLEMENTATION" "Cyrus timsieved v2.4.17-Fedora-RPM-2.4.17-13.el7"
"SASL" "PLAIN LOGIN CRAM-MD5 DIGEST-MD5"
"SIEVE" "comparator-i;ascii-numeric fileinto reject vacation imapflags 
notify envelope relational regex subaddress copy"
"STARTTLS"
"UNAUTHENTICATE"
OK
#

and

[root at mail rules]# telnet localhost 4190
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
"IMPLEMENTATION" "Cyrus timsieved v2.4.17-Fedora-RPM-2.4.17-13.el7"
"SASL" "PLAIN LOGIN CRAM-MD5 DIGEST-MD5"
"SIEVE" "comparator-i;ascii-numeric fileinto reject vacation imapflags 
notify envelope relational regex subaddress copy"
"STARTTLS"
"UNAUTHENTICATE"
OK

Replacing localhost with 127.0.0.1 gets rid of the? "telnet: connect to 
address ::1: Connection refused".

I can also log into sieveshell and list and manipulate scripts::

[root at mail rules]# sieveshell --authname=mail-admin --user=jpb localhost
connecting to localhost
Please enter your password:
 > list
mail
sieve-test? <- active script
 > get sieve-test
require ["fileinto"];
if address :is "From" "test-user at gmail.com" {
 ? fileinto "INBOX.Microsoft";
 ? stop;
}




On 18/02/2019 09:33, Willem Offermans wrote:

> Dear Egoitz and Cyrus friends,
>
> Check if sieve is actually listening at port 2000 as well.
>
>
> Wiel Offermans
> Willem at Offermans.Rompen.nl <mailto:Willem at Offermans.Rompen.nl>
>
>
>
>
>> On 18 Feb 2019, at 09:34, egoitz at sarenet.es 
>> <mailto:egoitz at sarenet.es> wrote:
>>
>> Hi!
>>
>>
>> Could you try enabling local6.debug channel in syslog, so that you 
>> could see additional Sieve debugging information?. Can you then post 
>> that log?
>>
>>
>> Cheers!
>>
>>
>> El 2019-02-15 12:32, J Pilfold-Bagwell escribi?:
>>
>>> Hi?All,
>>>
>>> I have a Centos 7 box running with the latest default cyrus install 
>>> from the Centos 7 repo, i.e. cyrus-imapd-2.4.17-13.el7.x86_64 .
>>>
>>> The problem I have is that sieve doesn't seem to pay any attention 
>>> to the scripts.? I have sieve running, I can successfully log in to 
>>> it using sieveshell, create, upload and activate scripts, but they 
>>> don't seem to be applied to the incoming mail.? First I was trying 
>>> the vacation and reject scripts so checked that the correct sendmail 
>>> is in use but it fails on fileinto as well.
>>>
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>
>>> imapd.conf?looks?like?this:
>>>
>>> [root at mail?admin]#?cat?/etc/imapd.conf
>>> configdirectory:?/var/lib/imap
>>> partition-default:?/var/spool/imap
>>> admins:?cyradmin
>>> sieve_admins:?cyradmin
>>> sievedir:?/var/lib/imap/sieve
>>> sendmail:?/usr/sbin/sendmail
>>> hashimapspool:?true
>>> sasl_pwcheck_method:?auxprop
>>> sasl_auxprop_plugin:?sasldb
>>> sasl_mech_list:?PLAIN?LOGIN?CRAM-MD5?DIGEST-MD5
>>> allowplaintext:?yes
>>> allowusermoves:?yes
>>> defaultdomain:?mail
>>> lmtp_downcase_rcpt:?yes
>>>
>>> tls_cert_file:?/etc/ssl/certs/cyrus-imapd/newcert.pem
>>> tls_key_file:?/etc/ssl/certs/cyrus-imapd/newkey.pem
>>> tls_ca_file:?/etc/ssl/certs/cyrus-imapd/cacert.pem
>>> tls_ca_path:?/etc/ssl/certscyrus-imapd
>>>
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>
>>> cyrus.conf:
>>>
>>> #?standard?standalone?server?implementation
>>>
>>> START?{
>>> ??#?do?not?delete?this?entry!
>>> ??recover????cmd="ctl_cyrusdb?-r"
>>>
>>> ??#?this?is?only?necessary?if?using?idled?for?IMAP?IDLE
>>> ??idled????????cmd="idled"
>>> }
>>>
>>> #?UNIX?sockets?start?with?a?slash?and?are?put?into?/var/lib/imap/sockets
>>> SERVICES?{
>>> ??#?add?or?remove?based?on?preferences
>>> ??imap????????cmd="imapd"?listen="imap"?prefork=5
>>> ??imaps????????cmd="imapd?-s"?listen="imaps"?prefork=1
>>> #??pop3????????cmd="pop3d"?listen="pop3"?prefork=3
>>> #??pop3s????????cmd="pop3d?-s"?listen="pop3s"?prefork=1
>>> ??sieve????????cmd="timsieved"?listen="0.0.0.0:2000"?prefork=0
>>> ??sieve?????????cmd="timsieved"?listen="0.0.0.0:4190"?prefork=0
>>> #??managesieve???cmd="timsieved"?listen="localhost:4190"?prefork=0
>>>
>>> ??#?these?are?only?necessary?if?receiving/exporting?usenet?via?NNTP
>>> #??nntp????????cmd="nntpd"?listen="nntp"?prefork=3
>>> #??nntps????????cmd="nntpd?-s"?listen="nntps"?prefork=1
>>>
>>> ??#?at?least?one?LMTP?is?required?for?delivery
>>> #??lmtp????????cmd="lmtpd"?listen="lmtp"?prefork=0
>>> ??lmtpunix????cmd="lmtpd"?listen="/var/lib/imap/socket/lmtp"?prefork=1
>>>
>>> ??#?this?is?only?necessary?if?using?notifications
>>> #? notify??? cmd="notifyd" listen="/var/lib/imap/socket/notify" 
>>> proto="udp" prefork=1
>>> }
>>>
>>> EVENTS?{
>>> ??#?this?is?required
>>> ??checkpoint????cmd="ctl_cyrusdb?-c"?period=30
>>>
>>> ??#?this?is?only?necessary?if?using?duplicate?delivery?suppression,
>>> ??#?Sieve?or?NNTP
>>> ??delprune????cmd="cyr_expire?-E?3"?at=0400
>>>
>>> ??#?this?is?only?necessary?if?caching?TLS?sessions
>>> ??tlsprune????cmd="tls_prune"?at=0400
>>>
>>> ??#?reindex?changed?mailboxes?(fulltext)?approximately?every?three?hours
>>> ? squatter1?????? cmd="/usr/bin/ionice -c idle 
>>> /usr/lib/cyrus/bin/squatter -s" period=180
>>>
>>> ??#?reindex?all?mailboxes?(fulltext)?daily
>>> ??squattera???????cmd="/usr/lib/cyrus/bin/squatter"?at=0117
>>> }
>>>
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>
>>> sieveshell?logs?in?fine:
>>>
>>> [root at mail?admin]#?sieveshell?--authname=cyradmin?--user=testuser1?localhost
>>> connecting?to?localhost
>>> Please?enter?your?password:
>>>> list
>>> mail
>>> sieve-test??<-?active?script
>>>
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~telnet
>>>
>>> Telnet?login?provides:
>>>
>>> [root at mail?admin]#?telnet?192.168.0.6?4190
>>> Trying?192.168.0.6...
>>> Connected?to?192.168.0.6.
>>> Escape?character?is?'^]'.
>>> "IMPLEMENTATION"?"Cyrus?timsieved?v2.4.17-Fedora-RPM-2.4.17-13.el7"
>>> "SASL"?"PLAIN?LOGIN?CRAM-MD5?DIGEST-MD5"
>>> "SIEVE" "comparator-i;ascii-numeric fileinto reject vacation 
>>> imapflags notify envelope relational regex subaddress copy"
>>> "STARTTLS"
>>> "UNAUTHENTICATE"
>>> OK
>>>
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>
>>> And this works for both port 2000 and 4190 on all interfaces. LMTP 
>>> is in use but somewhere, they aren't talking.
>>>
>>> Does anyone have any troubleshooting tips they can feed me or, can 
>>> anyone see a glaringly obvious error I've made because it's all gone 
>>> a bit wood for the trees here.
>>>
>>> The logs are huge but if you'd like to see the contents, let me know 
>>> what you'd like it grep'd for and I'll provide.
>>>
>>> Thanks,
>>>
>>> Julian
>>>
>>> -- 
>>> This?email?is?from?Borden?Grammar?School?Trust.
>>>
>>> This?email,?together?with?any?files?transmitted?with?it,?is?confidential,?and?is?intended?solely?for?the?use?of?the?individual?or?entity?to?whom?it?is?addressed.?Any?unauthorised?dissemination?or?copying?of?this?email?or?its?attachments,?and?any?use?or?disclosure?of?any?information?contained?in?them,?is?strictly?prohibited,?and?may?also?be?illegal.?If?you?are?not?the?intended?recipient?you?may?not?use,?disclose,?distribute,?copy,?print?or?relay?this?email.
>>>
>>> Please?note?that?any?views?expressed?by?an?individual?within?this?email,?do?not?necessarily?reflect?the?views?of?the?Borden?Grammar?School?Trust.
>>>
>>> Borden?Grammar?School?Trust?has?taken?reasonable?precautions?to?ensure?no?viruses?are?present?in?this?email.??The?Academy?cannot?accept?responsibility?for?any?loss?or?damage?arising?from?the?use?of?this?email?and/or?files?attached.
>>>
>>> Registered?Office:?Borden?Grammar?School,?Avenue?of?Remembrance,?Sittingbourne,?Kent?ME10?4DB
>>>
>>> Registered?in?England:?07827591
>>>
>>> ----
>>> Cyrus?Home?Page: http://www.cyrusimap.org/
>>> List?Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>> To?Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>
>
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
This email is from Borden Grammar School Trust.

This email, together with any files transmitted with it, is confidential, and is intended solely for the use of the individual or entity to whom it is addressed. Any unauthorised dissemination or copying of this email or its attachments, and any use or disclosure of any information contained in them, is strictly prohibited, and may also be illegal. If you are not the intended recipient you may not use, disclose, distribute, copy, print or relay this email.

Please note that any views expressed by an individual within this email, do not necessarily reflect the views of the Borden Grammar School Trust.

Borden Grammar School Trust has taken reasonable precautions to ensure no viruses are present in this email.  The Academy cannot accept responsibility for any loss or damage arising from the use of this email and/or files attached.

Registered Office: Borden Grammar School, Avenue of Remembrance, Sittingbourne, Kent ME10 4DB

Registered in England: 07827591

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190226/c03848ab/attachment-0001.html>

From Willem at Offermans.Rompen.nl  Tue Feb 26 05:57:25 2019
From: Willem at Offermans.Rompen.nl (Willem Offermans)
Date: Tue, 26 Feb 2019 11:57:25 +0100
Subject: Sieve not working
In-Reply-To: <446a34c6-1129-6838-d379-d43d25d12cc3@bordengrammar.kent.sch.uk>
References: <eb2fa1ec-3e14-6315-1467-438349b0bb9a@bordengrammar.kent.sch.uk>
 <b6eeff5b17822c1ce8f080e0a3a3c8e6@sarenet.es>
 <C6BDF808-73E3-4474-8F49-3531FC0E0D1D@Offermans.Rompen.nl>
 <446a34c6-1129-6838-d379-d43d25d12cc3@bordengrammar.kent.sch.uk>
Message-ID: <DBF80458-1969-406A-ADE5-EF17FB9F90D6@Offermans.Rompen.nl>

Dear jpilfold-bagwell <mailto:jpilfold-bagwell at bordengrammar.kent.sch.uk> and Cyrus friends,

Now you are sure that sieve is actually listening on the mentioned ports.

How about the main question? 

Are the scripts applied to the incoming mail?



Wiel Offermans
Willem at Offermans.Rompen.nl




> On 26 Feb 2019, at 11:36, J Pilfold-Bagwell <jpilfold-bagwell at bordengrammar.kent.sch.uk> wrote:
> 
> Hi Willem,
> 
> Once I found that you can run sieve on ports 200 and 4190 at the same time, I set it up just in case Cyrus was talking to only one of the ports.
> 
> If I telnet in, I get this response.
> 
> [root at mail rules]# telnet localhost 2000
> Trying ::1...
> telnet: connect to address ::1: Connection refused
> Trying 127.0.0.1...
> Connected to localhost.
> Escape character is '^]'.
> "IMPLEMENTATION" "Cyrus timsieved v2.4.17-Fedora-RPM-2.4.17-13.el7"
> "SASL" "PLAIN LOGIN CRAM-MD5 DIGEST-MD5"
> "SIEVE" "comparator-i;ascii-numeric fileinto reject vacation imapflags notify envelope relational regex subaddress copy"
> "STARTTLS"
> "UNAUTHENTICATE"
> OK
> #
> 
> and 
> 
> [root at mail rules]# telnet localhost 4190
> Trying ::1...
> telnet: connect to address ::1: Connection refused
> Trying 127.0.0.1...
> Connected to localhost.
> Escape character is '^]'.
> "IMPLEMENTATION" "Cyrus timsieved v2.4.17-Fedora-RPM-2.4.17-13.el7"
> "SASL" "PLAIN LOGIN CRAM-MD5 DIGEST-MD5"
> "SIEVE" "comparator-i;ascii-numeric fileinto reject vacation imapflags notify envelope relational regex subaddress copy"
> "STARTTLS"
> "UNAUTHENTICATE"
> OK
> 
> 
> Replacing localhost with 127.0.0.1 gets rid of the  "telnet: connect to address ::1: Connection refused" <telnet:connecttoaddress::1:Connectionrefused>.
> 
> I can also log into sieveshell and list and manipulate scripts::
> 
> [root at mail rules]# sieveshell --authname=mail-admin --user=jpb localhost
> connecting to localhost
> Please enter your password: 
> > list
> mail 
> sieve-test  <- active script
> > get sieve-test
> require ["fileinto"];
> if address :is "From" "test-user at gmail.com" <mailto:test-user at gmail.com> {
>   fileinto "INBOX.Microsoft";
>   stop;
> }
> 
> 
> 
> 
> 
> 
> 
> 
> On 18/02/2019 09:33, Willem Offermans wrote:
> 
>> Dear Egoitz and Cyrus friends,
>> 
>> Check if sieve is actually listening at port 2000 as well.
>> 
>> 
>> Wiel Offermans
>> Willem at Offermans.Rompen.nl <mailto:Willem at Offermans.Rompen.nl>
>> 
>> 
>> 
>> 
>>> On 18 Feb 2019, at 09:34, egoitz at sarenet.es <mailto:egoitz at sarenet.es> wrote:
>>> 
>>> Hi!
>>> 
>>> 
>>> 
>>> Could you try enabling local6.debug channel in syslog, so that you could see additional Sieve debugging information?. Can you then post that log?
>>> 
>>> 
>>> 
>>> Cheers!
>>> 
>>>  
>>> 
>>> 
>>> El 2019-02-15 12:32, J Pilfold-Bagwell escribi?:
>>> 
>>>> Hi All,
>>>> 
>>>> I have a Centos 7 box running with the latest default cyrus install from the Centos 7 repo, i.e. cyrus-imapd-2.4.17-13.el7.x86_64 .
>>>> 
>>>> The problem I have is that sieve doesn't seem to pay any attention to the scripts.  I have sieve running, I can successfully log in to it using sieveshell, create, upload and activate scripts, but they don't seem to be applied to the incoming mail.  First I was trying the vacation and reject scripts so checked that the correct sendmail is in use but it fails on fileinto as well.
>>>> 
>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>> 
>>>> imapd.conf looks like this:
>>>> 
>>>> [root at mail admin]# cat /etc/imapd.conf
>>>> configdirectory: /var/lib/imap
>>>> partition-default: /var/spool/imap
>>>> admins: cyradmin
>>>> sieve_admins: cyradmin
>>>> sievedir: /var/lib/imap/sieve
>>>> sendmail: /usr/sbin/sendmail
>>>> hashimapspool: true
>>>> sasl_pwcheck_method: auxprop
>>>> sasl_auxprop_plugin: sasldb
>>>> sasl_mech_list: PLAIN LOGIN CRAM-MD5 DIGEST-MD5
>>>> allowplaintext: yes
>>>> allowusermoves: yes
>>>> defaultdomain: mail
>>>> lmtp_downcase_rcpt: yes
>>>> 
>>>> tls_cert_file: /etc/ssl/certs/cyrus-imapd/newcert.pem
>>>> tls_key_file: /etc/ssl/certs/cyrus-imapd/newkey.pem
>>>> tls_ca_file: /etc/ssl/certs/cyrus-imapd/cacert.pem
>>>> tls_ca_path: /etc/ssl/certscyrus-imapd
>>>> 
>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>> 
>>>> cyrus.conf:
>>>> 
>>>> # standard standalone server implementation
>>>> 
>>>> START {
>>>>   # do not delete this entry!
>>>>   recover    cmd="ctl_cyrusdb -r"
>>>> 
>>>>   # this is only necessary if using idled for IMAP IDLE
>>>>   idled        cmd="idled"
>>>> }
>>>> 
>>>> # UNIX sockets start with a slash and are put into /var/lib/imap/sockets
>>>> SERVICES {
>>>>   # add or remove based on preferences
>>>>   imap        cmd="imapd" listen="imap" prefork=5
>>>>   imaps        cmd="imapd -s" listen="imaps" prefork=1
>>>> #  pop3        cmd="pop3d" listen="pop3" prefork=3
>>>> #  pop3s        cmd="pop3d -s" listen="pop3s" prefork=1
>>>>   sieve        cmd="timsieved" listen="0.0.0.0:2000" prefork=0
>>>>   sieve         cmd="timsieved" listen="0.0.0.0:4190" prefork=0
>>>> #  managesieve   cmd="timsieved" listen="localhost:4190" prefork=0
>>>> 
>>>>   # these are only necessary if receiving/exporting usenet via NNTP
>>>> #  nntp        cmd="nntpd" listen="nntp" prefork=3
>>>> #  nntps        cmd="nntpd -s" listen="nntps" prefork=1
>>>> 
>>>>   # at least one LMTP is required for delivery
>>>> #  lmtp        cmd="lmtpd" listen="lmtp" prefork=0
>>>>   lmtpunix    cmd="lmtpd" listen="/var/lib/imap/socket/lmtp" prefork=1
>>>> 
>>>>   # this is only necessary if using notifications
>>>> #  notify    cmd="notifyd" listen="/var/lib/imap/socket/notify" proto="udp" prefork=1
>>>> }
>>>> 
>>>> EVENTS {
>>>>   # this is required
>>>>   checkpoint    cmd="ctl_cyrusdb -c" period=30
>>>> 
>>>>   # this is only necessary if using duplicate delivery suppression,
>>>>   # Sieve or NNTP
>>>>   delprune    cmd="cyr_expire -E 3" at=0400
>>>> 
>>>>   # this is only necessary if caching TLS sessions
>>>>   tlsprune    cmd="tls_prune" at=0400
>>>> 
>>>>   # reindex changed mailboxes (fulltext) approximately every three hours
>>>>   squatter1       cmd="/usr/bin/ionice -c idle /usr/lib/cyrus/bin/squatter -s" period=180
>>>> 
>>>>   # reindex all mailboxes (fulltext) daily
>>>>   squattera       cmd="/usr/lib/cyrus/bin/squatter" at=0117
>>>> }
>>>> 
>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>> 
>>>> sieveshell logs in fine:
>>>> 
>>>> [root at mail admin]# sieveshell --authname=cyradmin --user=testuser1 localhost
>>>> connecting to localhost
>>>> Please enter your password:
>>>>> 
>>>>> list
>>>> mail
>>>> sieve-test  <- active script
>>>> 
>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~telnet
>>>> 
>>>> Telnet login provides:
>>>> 
>>>> [root at mail admin]# telnet 192.168.0.6 4190
>>>> Trying 192.168.0.6...
>>>> Connected to 192.168.0.6.
>>>> Escape character is '^]'.
>>>> "IMPLEMENTATION" "Cyrus timsieved v2.4.17-Fedora-RPM-2.4.17-13.el7"
>>>> "SASL" "PLAIN LOGIN CRAM-MD5 DIGEST-MD5"
>>>> "SIEVE" "comparator-i;ascii-numeric fileinto reject vacation imapflags notify envelope relational regex subaddress copy"
>>>> "STARTTLS"
>>>> "UNAUTHENTICATE"
>>>> OK
>>>> 
>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>> 
>>>> And this works for both port 2000 and 4190 on all interfaces. LMTP is in use but somewhere, they aren't talking.
>>>> 
>>>> Does anyone have any troubleshooting tips they can feed me or, can anyone see a glaringly obvious error I've made because it's all gone a bit wood for the trees here.
>>>> 
>>>> The logs are huge but if you'd like to see the contents, let me know what you'd like it grep'd for and I'll provide.
>>>> 
>>>> Thanks,
>>>> 
>>>> Julian
>>>> 
>>>> -- 
>>>> This email is from Borden Grammar School Trust.
>>>> 
>>>> This email, together with any files transmitted with it, is confidential, and is intended solely for the use of the individual or entity to whom it is addressed. Any unauthorised dissemination or copying of this email or its attachments, and any use or disclosure of any information contained in them, is strictly prohibited, and may also be illegal. If you are not the intended recipient you may not use, disclose, distribute, copy, print or relay this email.
>>>> 
>>>> Please note that any views expressed by an individual within this email, do not necessarily reflect the views of the Borden Grammar School Trust.
>>>> 
>>>> Borden Grammar School Trust has taken reasonable precautions to ensure no viruses are present in this email.  The Academy cannot accept responsibility for any loss or damage arising from the use of this email and/or files attached.
>>>> 
>>>> Registered Office: Borden Grammar School, Avenue of Remembrance, Sittingbourne, Kent ME10 4DB
>>>> 
>>>> Registered in England: 07827591
>>>> 
>>>> ----
>>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>>> To Unsubscribe:
>>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>----
>>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>>> To Unsubscribe:
>>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
>> 
>> 
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>-- 
> This email is from Borden Grammar School Trust.
> 
> This email, together with any files transmitted with it, is confidential, and is intended solely for the use of the individual or entity to whom it is addressed. Any unauthorised dissemination or copying of this email or its attachments, and any use or disclosure of any information contained in them, is strictly prohibited, and may also be illegal. If you are not the intended recipient you may not use, disclose, distribute, copy, print or relay this email.
> 
> Please note that any views expressed by an individual within this email, do not necessarily reflect the views of the Borden Grammar School Trust.
> 
> Borden Grammar School Trust has taken reasonable precautions to ensure no viruses are present in this email.  The Academy cannot accept responsibility for any loss or damage arising from the use of this email and/or files attached.
> 
> Registered Office: Borden Grammar School, Avenue of Remembrance, Sittingbourne, Kent ME10 4DB
> 
> Registered in England: 07827591
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190226/09ec4eb6/attachment-0001.html>

From nprice at gibb.co.za  Tue Feb 26 06:00:15 2019
From: nprice at gibb.co.za (Neil Price)
Date: Tue, 26 Feb 2019 13:00:15 +0200
Subject: Cyrus imap and identity theft
In-Reply-To: <545F8DAC-C29B-4BE9-9ADA-E2A8250A3C93@univ-perp.fr>
References: <545F8DAC-C29B-4BE9-9ADA-E2A8250A3C93@univ-perp.fr>
Message-ID: <dade5388-f228-04ea-4a85-dd0b69917ee7@gibb.co.za>


On 22/02/2019 08:41 PM, St?phane Branchoux wrote:
>
> Each week , few users respond to phishing mails.
> I use rules on firewalls, DNS filters, training program for users , 
> anti spam products , anti virus ?.
>
> I am looking for a way or tools to reduce identity theft on my Cyrus 
> imap server.
> For example , scripts to geo localise ip requests , detect and reject 
> bad connexions  ?
> Is it possible to authorize few devices for a user and reject other 
> devices  ?
>
> Which tools do you use on your Cyrus imap servers to protect them ?
>
>

fail2ban and fail2ban-repeater 
https://stuffphilwrites.com/2013/03/permanently-ban-repeat-offenders-fail2ban/
ipset-blacklist https://github.com/trick77/ipset-blacklist (great for 
banning whole countries)
password policies

Plus the usual: SPF, clam, spamassassin, greylisting, etc
Spam check outgoing mail too.

From l.schimmer at cgv.tugraz.at  Tue Feb 26 06:20:53 2019
From: l.schimmer at cgv.tugraz.at (Lars Schimmer)
Date: Tue, 26 Feb 2019 12:20:53 +0100
Subject: Best way to auth cyrus 3.x to an AD domain setup
In-Reply-To: <9feffd69-ad95-fe4c-27ec-612c8df036bf@cgv.tugraz.at>
References: <81fed929-5ad6-f1a8-a2de-9d40ef75db74@cgv.tugraz.at>
 <20190225141734.GF1945@dan.olp.net>
 <9feffd69-ad95-fe4c-27ec-612c8df036bf@cgv.tugraz.at>
Message-ID: <55566020-924b-685c-9797-bb703a938dc4@cgv.tugraz.at>

Ok, here my config saslauthd.conf file:


# Servers
ldap_servers: ldap://a.b.c.d:389/ ldap://a.b.c.e:389/

# Identity
ldap_bind_dn: cn=bind,cn=Users,DC=cgv,DC=tugraz,DC=at
ldap_password: pass
#ldap_auth_method: bind

# Search
ldap_search_base: cn=Users,DC=cgv,DC=tugraz,DC=at
ldap_filter: sAMAccountName=%u


# misc
ldap_use_sasl: yes
ldap_mech: DIGEST-MD5


testsaslauthd -u maya -p pass

shows:
:auth failure: [user=maya] [service=imap] [realm=] [mech=ldap]
[reason=Unknown]
saslauthd[2735] :response: NO


So far it looks like it does not ask the ldap server at all. Hm.


MfG,
Lars Schimmer
-- 
-------------------------------------------------------------
TU Graz, Institut f?r ComputerGraphik & WissensVisualisierung
Tel: +43 316 873-5405       E-Mail: l.schimmer at cgv.tugraz.at
Fax: +43 316 873-5402       PGP-Key-ID: 0x4A9B1723



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190226/905d7f08/attachment.sig>

From stephane.branchoux at univ-perp.fr  Tue Feb 26 09:20:40 2019
From: stephane.branchoux at univ-perp.fr (Stephane Branchoux)
Date: Tue, 26 Feb 2019 15:20:40 +0100
Subject: Cyrus imap and identity theft
In-Reply-To: <dade5388-f228-04ea-4a85-dd0b69917ee7@gibb.co.za>
References: <545F8DAC-C29B-4BE9-9ADA-E2A8250A3C93@univ-perp.fr>
 <dade5388-f228-04ea-4a85-dd0b69917ee7@gibb.co.za>
Message-ID: <ea6929a2-df54-a616-f3b5-c729ff4a15bb@univ-perp.fr>

Hello,

Thanks for the link to ipset-balcklist, i will try it.

fail2ban is not interesting for me because with phishing, connexions are 
succeded !

I would like to detect and block succeed? connections when a user 
connects from multiple

countries the same day.

Thanks

Le 26/02/2019 ? 12:00, Neil Price a ?crit?:
>
> On 22/02/2019 08:41 PM, St?phane Branchoux wrote:
>>
>> Each week , few users respond to phishing mails.
>> I use rules on firewalls, DNS filters, training program for users , 
>> anti spam products , anti virus ?.
>>
>> I am looking for a way or tools to reduce identity theft on my Cyrus 
>> imap server.
>> For example , scripts to geo localise ip requests , detect and reject 
>> bad connexions? ?
>> Is it possible to authorize few devices for a user and reject other 
>> devices? ?
>>
>> Which tools do you use on your Cyrus imap servers to protect them ?
>>
>>
>
> fail2ban and fail2ban-repeater 
> https://stuffphilwrites.com/2013/03/permanently-ban-repeat-offenders-fail2ban/
> ipset-blacklist https://github.com/trick77/ipset-blacklist (great for 
> banning whole countries)
> password policies
>
> Plus the usual: SPF, clam, spamassassin, greylisting, etc
> Spam check outgoing mail too.
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
Stephane BRANCHOUX
Centre de Ressources Informatiques de l'Universit? de Perpignan.
Syst?mes/R?seaux - RSSI
mailto:stephane.branchoux at univ-perp.fr
04 68 66 21 24 / 07 60 73 38 42


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3662 bytes
Desc: Signature cryptographique S/MIME
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190226/15eec5b1/attachment.p7s>

From merlin at mrc-mbu.cam.ac.uk  Tue Feb 26 09:35:48 2019
From: merlin at mrc-mbu.cam.ac.uk (Merlin Hartley)
Date: Tue, 26 Feb 2019 14:35:48 +0000
Subject: Cyrus imap and identity theft
In-Reply-To: <ea6929a2-df54-a616-f3b5-c729ff4a15bb@univ-perp.fr>
References: <545F8DAC-C29B-4BE9-9ADA-E2A8250A3C93@univ-perp.fr>
 <dade5388-f228-04ea-4a85-dd0b69917ee7@gibb.co.za>
 <ea6929a2-df54-a616-f3b5-c729ff4a15bb@univ-perp.fr>
Message-ID: <55A2A1FE-97DE-4F77-BBFF-414A100E0741@mrc-mbu.cam.ac.uk>

fail2ban can do anything you want - including what you describe - you just have tell it what to look for in the logs!


--
Merlin Hartley
Computer Officer
MRC Mitochondrial Biology Unit
University of Cambridge
Cambridge, CB2 0XY
United Kingdom

> On 26 Feb 2019, at 14:20, Stephane Branchoux <stephane.branchoux at univ-perp.fr> wrote:
> 
> Hello,
> 
> Thanks for the link to ipset-balcklist, i will try it.
> 
> fail2ban is not interesting for me because with phishing, connexions are succeded !
> 
> I would like to detect and block succeed  connections when a user connects from multiple
> 
> countries the same day.
> 
> Thanks
> 
> Le 26/02/2019 ? 12:00, Neil Price a ?crit :
>> 
>> On 22/02/2019 08:41 PM, St?phane Branchoux wrote:
>>> 
>>> Each week , few users respond to phishing mails.
>>> I use rules on firewalls, DNS filters, training program for users , anti spam products , anti virus ?.
>>> 
>>> I am looking for a way or tools to reduce identity theft on my Cyrus imap server.
>>> For example , scripts to geo localise ip requests , detect and reject bad connexions  ?
>>> Is it possible to authorize few devices for a user and reject other devices  ?
>>> 
>>> Which tools do you use on your Cyrus imap servers to protect them ?
>>> 
>>> 
>> 
>> fail2ban and fail2ban-repeater https://stuffphilwrites.com/2013/03/permanently-ban-repeat-offenders-fail2ban/
>> ipset-blacklist https://github.com/trick77/ipset-blacklist (great for banning whole countries)
>> password policies
>> 
>> Plus the usual: SPF, clam, spamassassin, greylisting, etc
>> Spam check outgoing mail too.
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> -- 
> Stephane BRANCHOUX
> Centre de Ressources Informatiques de l'Universit? de Perpignan.
> Syst?mes/R?seaux - RSSI
> mailto:stephane.branchoux at univ-perp.fr
> 04 68 66 21 24 / 07 60 73 38 42
> 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190226/d4c004bc/attachment-0001.html>

From l.schimmer at cgv.tugraz.at  Wed Feb 27 05:20:24 2019
From: l.schimmer at cgv.tugraz.at (Lars Schimmer)
Date: Wed, 27 Feb 2019 11:20:24 +0100
Subject: Best way to auth cyrus 3.x to an AD domain setup - works
In-Reply-To: <55566020-924b-685c-9797-bb703a938dc4@cgv.tugraz.at>
References: <81fed929-5ad6-f1a8-a2de-9d40ef75db74@cgv.tugraz.at>
 <20190225141734.GF1945@dan.olp.net>
 <9feffd69-ad95-fe4c-27ec-612c8df036bf@cgv.tugraz.at>
 <55566020-924b-685c-9797-bb703a938dc4@cgv.tugraz.at>
Message-ID: <418ba6c3-b6e6-47da-e623-ad52b0ddd7e4@cgv.tugraz.at>

On 2/26/19 12:20 PM, Lars Schimmer wrote:
> Ok, here my config saslauthd.conf file:

Ok, now, after removing all typos, rebooting, setting a correct user, it
works, out of the box for cyrus 3.x and exim4 on debian buster.
Great.
Thank you.

MfG,
Lars Schimmer
-- 
-------------------------------------------------------------
TU Graz, Institut f?r ComputerGraphik & WissensVisualisierung
Tel: +43 316 873-5405       E-Mail: l.schimmer at cgv.tugraz.at
Fax: +43 316 873-5402       PGP-Key-ID: 0x4A9B1723



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190227/d373cd0c/attachment.sig>

From dennisdavis+info-cyrus at fastmail.fm  Wed Feb 27 17:04:06 2019
From: dennisdavis+info-cyrus at fastmail.fm (Dennis Davis)
Date: Wed, 27 Feb 2019 22:04:06 +0000 (GMT)
Subject: Cyrus imap and identity theft
In-Reply-To: <545F8DAC-C29B-4BE9-9ADA-E2A8250A3C93@univ-perp.fr>
References: <545F8DAC-C29B-4BE9-9ADA-E2A8250A3C93@univ-perp.fr>
Message-ID: <alpine.BSO.2.21.9999.1902272155390.29011@fastmail.fm>

On Fri, 22 Feb 2019, St?phane Branchoux wrote:

> From: St?phane Branchoux <stephane.branchoux at univ-perp.fr>
> To: info-cyrus at lists.andrew.cmu.edu
> Date: Fri, 22 Feb 2019 19:41:15 +0100
> Subject: Cyrus imap and identity theft
>
> Each week , few users respond to phishing mails.  I use rules on
> firewalls, DNS filters, training program for users , anti spam
> products , anti virus ?.
>
> I am looking for a way or tools to reduce identity theft on my
> Cyrus imap server.  For example , scripts to geo localise ip
> requests , detect and reject bad connexions ? Is it possible to
> authorize few devices for a user and reject other devices ?
>
> Which tools do you use on your Cyrus imap servers to protect them ?

As a slightly left-field suggestion, you can sometimes install the
checking and protection in your email relay.  Of course this depends
on the email software you use.

For the exim message transfer agent (MTA) see:

https://github.com/Exim/exim/wiki/BlockCracking

Disclaimer: I've never used the above, but it seems a lot of
work and thought have gone into this suggestion.
-- 
Dennis Davis <dennisdavis at fastmail.fm>