<!DOCTYPE html><html><head><title></title><style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}</style></head><body><div style="font-family:Arial;">On Mon, Feb 4, 2019, at 22:00, Michael Menge wrote:<br></div><blockquote type="cite" id="fastmail-quoted"><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Quoting Bron Gondwana <brong@fastmail.fm>:<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">> On Mon, Feb 4, 2019, at 20:21, Michael Menge wrote:<br></div><div style="font-family:Arial;">>> Hi,<br></div><div style="font-family:Arial;">>><br></div><div style="font-family:Arial;">>> Quoting Bron Gondwana <brong@fastmailteam.com>:<br></div><div style="font-family:Arial;">>><br></div><div style="font-family:Arial;">>> > Hi Michael,<br></div><div style="font-family:Arial;">>> ><br></div><div style="font-family:Arial;">>> > Sorry about the delay in looking at this - I was mad crazy busy<br></div><div style="font-family:Arial;">>> > getting ready to go overseas. At Fosdem now, about to give a talk<br></div><div style="font-family:Arial;">>> > about JMAP!<br></div><div style="font-family:Arial;">>> ><br></div><div style="font-family:Arial;">>> > OK, let's start with the things that give me a little bit of hives...<br></div><div style="font-family:Arial;">>> ><br></div><div style="font-family:Arial;">>> > configdirectory: /srv/cyrus-be<br></div><div style="font-family:Arial;">>> > partition-default: /srv/cyrus-be<br></div><div style="font-family:Arial;">>> > partition-ssd: /srv/cyrus-be/ssd-part<br></div><div style="font-family:Arial;">>> ><br></div><div style="font-family:Arial;">>> > Ouch. There's a couple of things I wouldn't do there - having the<br></div><div style="font-family:Arial;">>> > partition be the same as the config directory, and having a separate<br></div><div style="font-family:Arial;">>> > partition be a subdirectory of a partition. They're both asking for<br></div><div style="font-family:Arial;">>> > trouble. I would probably lay my system out like:<br></div><div style="font-family:Arial;">>> ><br></div><div style="font-family:Arial;">>> > configdirectory: /srv/cyrus-be/conf<br></div><div style="font-family:Arial;">>> > partition-default: /srv/cyrus-be/default-part<br></div><div style="font-family:Arial;">>> > partition-ssd: /srv/cyrus-be/ssd-part<br></div><div style="font-family:Arial;">>> ><br></div><div style="font-family:Arial;">>><br></div><div style="font-family:Arial;">>> partition-default isn't used any more. To use the metapartition we moved<br></div><div style="font-family:Arial;">>> all accounts form the default partition to the ssd partition which is the<br></div><div style="font-family:Arial;">>> the new defaultpartition ("defaultpartition: ssd")<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> Right - that makes sense.<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">>> > And then each tree would only have one type of thing in it.<br></div><div style="font-family:Arial;">>> ><br></div><div style="font-family:Arial;">>> > Anyway, I don't think that would break anything.<br></div><div style="font-family:Arial;">>> ><br></div><div style="font-family:Arial;">>> > metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part<br></div><div style="font-family:Arial;">>> > metapartition_files: header index cache expunge squat annotations<br></div><div style="font-family:Arial;">>> > lock dav archivecache<br></div><div style="font-family:Arial;">>> ><br></div><div style="font-family:Arial;">>> > Ooh, I haven't tested having cache and archivecache on the same<br></div><div style="font-family:Arial;">>> > location. That's really interesting. Again, I'd be in favour of<br></div><div style="font-family:Arial;">>> > separation here, give them different paths. That might be tricky<br></div><div style="font-family:Arial;">>> > with ssd though, the way this is laid out. I assume you have some<br></div><div style="font-family:Arial;">>> > kind of symlink farm going on?<br></div><div style="font-family:Arial;">>> ><br></div><div style="font-family:Arial;">>><br></div><div style="font-family:Arial;">>> I didn't know that there could be a problem with cache and archivecache.<br></div><div style="font-family:Arial;">>> At the time we decided on the configuration for cyrus 3.0 I looked at the<br></div><div style="font-family:Arial;">>> imapd.conf man page and for metapartition_files decided that I want all<br></div><div style="font-family:Arial;">>> meta files on the ssd storage. There was no indication in the man page<br></div><div style="font-family:Arial;">>> that there could be a problem.<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> Fair. I'd have to test that to see if it works correctly. I would <br></div><div style="font-family:Arial;">> hope so, but I haven't tested that configuration. This is the <br></div><div style="font-family:Arial;">> downside with having lots of different ways to do things!<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">>> How do I separate location of archivecache from the other <br></div><div style="font-family:Arial;">>> metapartition path?<br></div><div style="font-family:Arial;">>> And fix the cache and archivecache files?<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> This I don't know a good answer for. I will test if having the same <br></div><div style="font-family:Arial;">> path for cache and archivecache could fail. I THINK that I made the <br></div><div style="font-family:Arial;">> code safe for it, but I'm not sure that it's been tested.<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">>> No there is no sysmlink farm. We have mounted different iSCSI volumes to<br></div><div style="font-family:Arial;">>> /srv/cyrus-ssd-be, /srv/cyrus-hdd-be and /srv/cyrus-be<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> Right. That makes sense.<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">>> > Otherwise it all looks OK. Are you getting other IOERRORs in your<br></div><div style="font-family:Arial;">>> > log files which could show things aborting? It really looks like<br></div><div style="font-family:Arial;">>> > your conversations DB is getting out of sync due to other failures.<br></div><div style="font-family:Arial;">>><br></div><div style="font-family:Arial;">>> I found a few instances of 3 related errors.<br></div><div style="font-family:Arial;">>><br></div><div style="font-family:Arial;">>> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening<br></div><div style="font-family:Arial;">>> /srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory<br></div><div style="font-family:Arial;">>> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening<br></div><div style="font-family:Arial;">>> /srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory<br></div><div style="font-family:Arial;">>> Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR archive<br></div><div style="font-family:Arial;">>> user.XXXX 2185 failed to copyfile<br></div><div style="font-family:Arial;">>> (/srv/cyrus-be/ssd-part/L/user/XXXX/2185. =><br></div><div style="font-family:Arial;">>> /srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.): Unknown code<br></div><div style="font-family:Arial;">>> ____ 255<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> Ouch. Yeah, that could have been caused by a bug in delivery, and <br></div><div style="font-family:Arial;">> would definitely cause conversations DB corruption if the index file <br></div><div style="font-family:Arial;">> was updated but the conversations DB wasn't or vice versa.<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">>> The file was already at /srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> Right! I do wonder if there are some bugs in 3.0.x which are fixed <br></div><div style="font-family:Arial;">> on master around delivery to archive partition. We definitely had <br></div><div style="font-family:Arial;">> bugs on master, but I thought they were newly introduced on master <br></div><div style="font-family:Arial;">> as well, which is why the fixes weren't backported. But if you're <br></div><div style="font-family:Arial;">> having files be in the wrong location, maybe there are bugs on 3.0.x <br></div><div style="font-family:Arial;">> as well.<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> Do you have the syslog lines at the time that email was delivered?<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">I dont' have the log, for that message, but I will search for a<br></div><div style="font-family:Arial;">more recent example.<br></div></blockquote><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Great.<br></div><div style="font-family:Arial;"><br></div><blockquote type="cite" id="fastmail-quoted"><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">From the mail headers i know that it was not dilivered to the archive <br></div><div style="font-family:Arial;">partition<br></div><div style="font-family:Arial;">but moved by cyr_expire. The conversation db was not used at that time.<br></div></blockquote><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">OK - that shouldn't matter then - because the conversations rebuild should have found it.<br></div><div style="font-family:Arial;"><br></div><blockquote type="cite" id="fastmail-quoted"><div style="font-family:Arial;">PS.: the timesamp of the file is not the internal date but the time<br></div><div style="font-family:Arial;">the mail was moved to the archive partition. I was wondering about the reason.<br></div></blockquote><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Hmm, yeah:<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">r = cyrus_copyfile(srcname, destname, COPYFILE_MKDIR);<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">That's how the file is moved. It only does a hardlink if it's the same filesystem. Interestingly, it does NOT set the timestamp correctly. This is clearly a bug.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"><a href="https://github.com/cyrusimap/cyrus-imapd/issues/2641">https://github.com/cyrusimap/cyrus-imapd/issues/2641</a><br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Bron.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"><br></div><div id="sig56629417"><div class="signature">--<br></div><div class="signature"> Bron Gondwana, CEO, FastMail Pty Ltd<br></div><div class="signature"> brong@fastmailteam.com<br></div><div class="signature"><br></div></div><div style="font-family:Arial;"><br></div></body></html>