<!DOCTYPE html><html><head><title></title><style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}</style></head><body><div style="font-family:Arial;">On Mon, Feb 4, 2019, at 20:21, Michael Menge wrote:<br></div><blockquote type="cite" id="fastmail-quoted"><div style="font-family:Arial;">Hi,<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Quoting Bron Gondwana <brong@fastmailteam.com>:<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">> Hi Michael,<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> Sorry about the delay in looking at this - I was mad crazy busy <br></div><div style="font-family:Arial;">> getting ready to go overseas. At Fosdem now, about to give a talk <br></div><div style="font-family:Arial;">> about JMAP!<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> OK, let's start with the things that give me a little bit of hives...<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> configdirectory: /srv/cyrus-be<br></div><div style="font-family:Arial;">> partition-default: /srv/cyrus-be<br></div><div style="font-family:Arial;">> partition-ssd: /srv/cyrus-be/ssd-part<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> Ouch. There's a couple of things I wouldn't do there - having the <br></div><div style="font-family:Arial;">> partition be the same as the config directory, and having a separate <br></div><div style="font-family:Arial;">> partition be a subdirectory of a partition. They're both asking for <br></div><div style="font-family:Arial;">> trouble. I would probably lay my system out like:<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> configdirectory: /srv/cyrus-be/conf<br></div><div style="font-family:Arial;">> partition-default: /srv/cyrus-be/default-part<br></div><div style="font-family:Arial;">> partition-ssd: /srv/cyrus-be/ssd-part<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">partition-default isn't used any more. To use the metapartition we moved<br></div><div style="font-family:Arial;">all accounts form the default partition to the ssd partition which is the<br></div><div style="font-family:Arial;">the new defaultpartition ("defaultpartition: ssd")<br></div></blockquote><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Right - that makes sense.<br></div><div style="font-family:Arial;"><br></div><blockquote type="cite" id="fastmail-quoted"><div style="font-family:Arial;">> And then each tree would only have one type of thing in it.<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> Anyway, I don't think that would break anything.<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> metapartition-ssd: /srv/cyrus-ssd-be/meta/ssd-part<br></div><div style="font-family:Arial;">> metapartition_files: header index cache expunge squat annotations <br></div><div style="font-family:Arial;">> lock dav archivecache<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> Ooh, I haven't tested having cache and archivecache on the same <br></div><div style="font-family:Arial;">> location. That's really interesting. Again, I'd be in favour of <br></div><div style="font-family:Arial;">> separation here, give them different paths. That might be tricky <br></div><div style="font-family:Arial;">> with ssd though, the way this is laid out. I assume you have some <br></div><div style="font-family:Arial;">> kind of symlink farm going on?<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">I didn't know that there could be a problem with cache and archivecache.<br></div><div style="font-family:Arial;">At the time we decided on the configuration for cyrus 3.0 I looked at the<br></div><div style="font-family:Arial;">imapd.conf man page and for metapartition_files decided that I want all<br></div><div style="font-family:Arial;">meta files on the ssd storage. There was no indication in the man page<br></div><div style="font-family:Arial;">that there could be a problem.<br></div></blockquote><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Fair. I'd have to test that to see if it works correctly. I would hope so, but I haven't tested that configuration. This is the downside with having lots of different ways to do things!<br></div><div style="font-family:Arial;"><br></div><blockquote type="cite" id="fastmail-quoted"><div style="font-family:Arial;">How do I separate location of archivecache from the other metapartition path?<br></div><div style="font-family:Arial;">And fix the cache and archivecache files?<br></div></blockquote><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">This I don't know a good answer for. I will test if having the same path for cache and archivecache could fail. I THINK that I made the code safe for it, but I'm not sure that it's been tested.<br></div><div style="font-family:Arial;"><br></div><blockquote type="cite" id="fastmail-quoted"><div style="font-family:Arial;">No there is no sysmlink farm. We have mounted different iSCSI volumes to<br></div><div style="font-family:Arial;">/srv/cyrus-ssd-be, /srv/cyrus-hdd-be and /srv/cyrus-be<br></div></blockquote><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Right. That makes sense.<br></div><div style="font-family:Arial;"><br></div><blockquote type="cite" id="fastmail-quoted"><div style="font-family:Arial;">> Otherwise it all looks OK. Are you getting other IOERRORs in your <br></div><div style="font-family:Arial;">> log files which could show things aborting? It really looks like <br></div><div style="font-family:Arial;">> your conversations DB is getting out of sync due to other failures.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">I found a few instances of 3 related errors.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening <br></div><div style="font-family:Arial;">/srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory<br></div><div style="font-family:Arial;">Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR: opening <br></div><div style="font-family:Arial;">/srv/cyrus-be/ssd-part/L/user/XXXX/2185.: No such file or directory<br></div><div style="font-family:Arial;">Feb 4 01:10:55 mailserv03 be/cyr_expire[7626]: IOERROR archive <br></div><div style="font-family:Arial;">user.XXXX 2185 failed to copyfile <br></div><div style="font-family:Arial;">(/srv/cyrus-be/ssd-part/L/user/XXXX/2185. => <br></div><div style="font-family:Arial;">/srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.): Unknown code <br></div><div style="font-family:Arial;">____ 255<br></div></blockquote><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Ouch. Yeah, that could have been caused by a bug in delivery, and would definitely cause conversations DB corruption if the index file was updated but the conversations DB wasn't or vice versa.<br></div><div style="font-family:Arial;"><br></div><blockquote type="cite" id="fastmail-quoted"><div style="font-family:Arial;">The file was already at /srv/cyrus-hdd-be/archive/ssd-part/L/user/XXXX/2185.<br></div></blockquote><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Right! I do wonder if there are some bugs in 3.0.x which are fixed on master around delivery to archive partition. We definitely had bugs on master, but I thought they were newly introduced on master as well, which is why the fixes weren't backported. But if you're having files be in the wrong location, maybe there are bugs on 3.0.x as well.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Do you have the syslog lines at the time that email was delivered?<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Bron.<br></div><div id="sig567075"><div class="signature">-- <br></div><div class="signature"> Bron Gondwana<br></div><div class="signature"> brong@fastmail.fm<br></div><div class="signature"><br></div></div><div style="font-family:Arial;"><br></div></body></html>