another problem with conversations db

Michael Menge michael.menge at zdv.uni-tuebingen.de
Mon Feb 4 04:17:12 EST 2019 

Hi,

Quoting Bron Gondwana <brong at fastmailteam.com>:

> Hi Michael,
>
> Sorry about the delay in looking at this - I was mad crazy busy  
> getting ready to go overseas. At Fosdem now, about to give a talk  
> about JMAP!
>
> OK, let's start with the things that give me a little bit of hives...
>
> configdirectory: /srv/cyrus-be
> partition-default: /srv/cyrus-be
> partition-ssd: /srv/cyrus-be/ssd-part
>
> Ouch. There's a couple of things I wouldn't do there - having the  
> partition be the same as the config directory, and having a separate  
> partition be a subdirectory of a partition. They're both asking for  
> trouble. I would probably lay my system out like:
>
> configdirectory: /srv/cyrus-be/conf
> partition-default: /srv/cyrus-be/default-part
> partition-ssd: /srv/cyrus-be/ssd-part
>

partition-default isn't used any more. To use the metapartition we moved
all accounts form the default partition to the ssd partition which is the
the new defaultpartition ("defaultpartition: ssd")

> And then each tree would only have one type of thing in it.
>
> Anyway, I don't think that would break anything.
>
> metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part
> metapartition_files: header index cache expunge squat annotations  
> lock dav archivecache
>
> Ooh, I haven't tested having cache and archivecache on the same  
> location. That's really interesting. Again, I'd be in favour of  
> separation here, give them different paths. That might be tricky  
> with ssd though, the way this is laid out. I assume you have some  
> kind of symlink farm going on?
>

I didn't know that there could be a problem with cache and archivecache.
At the time we decided on the configuration for cyrus 3.0 I looked at the
imapd.conf man page and for metapartition_files decided that I want all
meta files on the ssd storage. There was no indication in the man page
that there could be a problem.

How do I separate location of archivecache from the other metapartition path?
And fix the cache and archivecache files?

No there is no sysmlink farm.  We have mounted different iSCSI volumes to
/srv/cyrus-ssd-be, /srv/cyrus-hdd-be and /srv/cyrus-be


> Otherwise it all looks OK. Are you getting other IOERRORs in your  
> log files which could show things aborting? It really looks like  
> your conversations DB is getting out of sync due to other failures.

I found a few instances of 3 related errors.

Feb  4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening  
/srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory
Feb  4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening  
/srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory
Feb  4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR archive  
user.XXXX 2185 failed to copyfile  
(/srv/cyrus-be/ssd-part/L/user/XXXX/2185. =>  
/srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.): Unknown code  
____ 255

The file was already at /srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.

stat 2185.
   File: '2185.'
   Size: 424542    	Blocks: 832        IO Block: 4096   regular file
Device: 860h/2144d	Inode: 4677438462  Links: 1
Access: (0600/-rw-------)  Uid: (   96/   cyrus)   Gid: (   12/    mail)
Context: system_u:object_r:unlabeled_t:s0
Access: 2018-11-27 01:12:46.585864239 +0100
Modify: 2018-11-27 01:12:46.585864239 +0100
Change: 2018-11-27 01:12:46.585864239 +0100
  Birth: -


>
> Cheers,
>
> Bron.
>
>
>
> On Thu, Jan 31, 2019, at 19:36, Michael Menge wrote:
>>
>> Quoting Bron Gondwana <brong at fastmail.fm>:
>>
>> > On Fri, Jan 25, 2019, at 20:00, Michael Menge wrote:
>> >> Hi Bron
>> >>
>> >>
>> >> Quoting Bron Gondwana <brong at fastmail.fm>:
>> >>
>> >> > Sorry I haven't been following along with this earlier. Can you post
>> >> > your imapd.conf and cyrus.conf as well as let her know if you run
>> >> > anything which directly messes with files on disk.
>> >>
>> >> Thanks for looking into it. I have attached the backend and replica
>> >> configs. There should be nothing messing with the files on disk
>> >> directly. Some times we need to restore mails from file based
>> >> backup (bacula) followed by a reconstruct but this was not the
>> >> case this time.
>> >
>> > Do you always rebuild the conversations db after doing the
>> > reconstruct? That will be necessary now. We switched to doing all
>> > our restores using IMAP append a while back so we're never fiddling
>> > the file system under Cyrus.
>> >
>> >> >
>> >> > Also, what operating system is this on and what Cyrus version?
>> >> >
>> >>
>> >> We are running a cyrus 3.0.8 compiled with the following Options
>> >> (./configure --enable-murder --enable-http --enable-calalarmd
>> >> --enable-replication --enable-backup --enable-idled
>> >> --enable-autocreate CFLAGS="-fPIC -g")
>> >> in a murder configuration on a RHEL 7.5 System. As filesystem we use xfs.
>> >
>> > They should be fine. I'll have a read of the config when I'm at a
>> > real computer.
>> >
>>
>> did you find anything in the config that would explain the problems?
>>
>> >>
>> >> > Bron.
>> >> >
>> >> > On Fri, Jan 25, 2019, at 04:08, Michael Menge wrote:
>> >> >> Hi,
>> >> >>
>> >> >> I have discovered an other problem with the conversations db:
>> >> >>
>> >> >> Thousends of lines with "IOERROR: conversations_audit on load:" and
>> >> >> "IOERROR: conversations_audit on store:"
>> >> >> A look at the source code shows that these errors are logged after
>> >> >> "_sanity_check_counts" is called.
>> >> >> The log level LOG_ERR and the prefix IOERROR indicate that I have a
>> >> >> serious problem. Do I?
>> >> >>
>> >> >> This problem occurred for accounts where the rebuild of the
>> >> >> conversations db was successful.
>> >> >>
>> >> >> I don't want to rebuild the conversations db every few days.
>> >> >>
>> >> >> Any help is appreciated.
>> >> >>
>> >> >> Kind regards
>> >> >>
>> >> >>
>> >> >> Michael Menge
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >>  
>> --------------------------------------------------------------------------------
>> >> >> M.Menge Tel.: (49) 7071/29-70316
>> >> >> Universität Tübingen Fax.: (49) 7071/29-5912
>> >> >> Zentrum für Datenverarbeitung mail:
>> >> >> michael.menge at zdv.uni-tuebingen.de
>> >> >> Wächterstraße 76
>> >> >> 72074 Tübingen
>> >> >>
>> >> >> ----
>> >> >> Cyrus Home Page: http://www.cyrusimap.org/
>> >> >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> >> >> To Unsubscribe:
>> >> >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>> >> >
>> >> > --
>> >> > Bron Gondwana
>> >> > brong at fastmail.fm
>> >>
>> >>
>> >>
>> >>  
>> --------------------------------------------------------------------------------
>> >> M.Menge Tel.: (49) 7071/29-70316
>> >> Universität Tübingen Fax.: (49) 7071/29-5912
>> >> Zentrum für Datenverarbeitung mail:
>> >> michael.menge at zdv.uni-tuebingen.de
>> >> Wächterstraße 76
>> >> 72074 Tübingen
>> >>
>> >> ----
>> >> Cyrus Home Page: http://www.cyrusimap.org/
>> >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> >> To Unsubscribe:
>> >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>> >>
>> >> *Attachments:*
>> >> * imapd_be.conf
>> >> * cyrus_be.conf
>> >> * cyrus_re.conf
>> >> * imapd_re.conf
>> >
>> > --
>> > Bron Gondwana
>> > brong at fastmail.fm
>>
>>
>>
>> --------------------------------------------------------------------------------
>> M.Menge Tel.: (49) 7071/29-70316
>> Universität Tübingen Fax.: (49) 7071/29-5912
>> Zentrum für Datenverarbeitung mail:
>> michael.menge at zdv.uni-tuebingen.de
>> Wächterstraße 76
>> 72074 Tübingen
>>
>> ----
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>
> --
>  Bron Gondwana, CEO, FastMail Pty Ltd
>  brong at fastmailteam.com



--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universität Tübingen                   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung          mail:  
michael.menge at zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen

More information about the Info-cyrus mailing list