<div dir="ltr"><div>Hi Bron,</div><div><br>Really sorry for the delay. It took me a while to get back to this.<br>It is harmless (other than a waste of CPU cycle) to call delete "if the record isn't there". <br>It must be found in that specific mailbox for something to happen.</div><div><br>Nevertheless, I made some small logic adjustment.<br>Also cleanup <a href="http://configure.ac">configure.ac</a> regarding OpenIO options</div><div><br></div><div>You can find my changes as usual over here</div><div><br></div><div><a href="https://github.com/RaymondPo/cyrus-imapd">https://github.com/RaymondPo/cyrus-imapd</a><br></div><div><br></div><div>and now </div><div><br></div><div><a href="https://github.com/open-io/cyrus-imapd">https://github.com/open-io/cyrus-imapd</a><br></div><div><br></div><div>Thanks<br></div><div>Raymond<br></div><div><br></div><div><br></div><div><br></div><div class="gmail_extra"><div class="gmail_quote">2016-03-18 6:35 GMT-04:00 Bron Gondwana via Cyrus-devel <span dir="ltr"><<a href="mailto:cyrus-devel@lists.andrew.cmu.edu" target="_blank">cyrus-devel@lists.andrew.cmu.edu</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><u></u>
<div><div style="font-family:Arial">And it turns out, there was a major bug in FastMail style Archiving (non-objectstore) to the point that it deleted emails after archiving them. Ouch.<br></div>
<div style="font-family:Arial"> </div>
<div style="font-family:Arial">Thanksfully we run one replica in archive-immediately mode, and it still had the messages - and we have a script that can rescue them back, so nothing was lost.<br></div>
<div style="font-family:Arial"> </div>
<div style="font-family:Arial">(stage 2 rescue would have been pulling from backups, which also works but is slower)<br></div>
<div style="font-family:Arial"> </div>
<div style="font-family:Arial">Patch on master now. Raymond, you might want to check of the objectstorage logic. It might call 'delete' if the record isn't there. Is that a problem? Presumably the refcount is smart enough to detect this, because otherwise there are failure modes where it could get out of sync as well.<br></div>
<div style="font-family:Arial"> </div>
<div style="font-family:Arial">Bron.<br></div>
<div> </div>
<div> </div>
<div>On Mon, Mar 14, 2016, at 20:14, Bron Gondwana via Cyrus-devel wrote:<br></div>
<blockquote type="cite"><div style="font-family:Arial">There were a couple of nits around style:<br></div>
<div style="font-family:Arial"> <br></div>
<div style="font-family:Arial">+ int remove_temp_spool_file = 0 ;<br></div>
<div style="font-family:Arial">+ int object_storage_enabled = 0 ;<br></div>
<div style="font-family:Arial"> <br></div>
<div style="font-family:Arial">(we don't put a space before the ; anywhere in the rest of the codebase)<br></div>
<div style="font-family:Arial"> <br></div>
<div style="font-family:Arial">And some <br></div>
<div style="font-family:Arial">}<br></div>
<div style="font-family:Arial">else<br></div>
<div style="font-family:Arial">{<br></div>
<div style="font-family:Arial"> <br></div>
<div style="font-family:Arial">or<br></div>
<div style="font-family:Arial"> <br></div>
<div style="font-family:Arial">if (condition)<br></div>
<div style="font-family:Arial">{<br></div>
<div style="font-family:Arial"> <br></div>
<div style="font-family:Arial">We cuddle the brace like this:<br></div>
<div style="font-family:Arial"> <br></div>
<div style="font-family:Arial">}<br></div>
<div style="font-family:Arial">else {<br></div>
<div style="font-family:Arial"> <br></div>
<div style="font-family:Arial">if (condition) {<br></div>
<div style="font-family:Arial"> <br></div>
<div style="font-family:Arial">Pretty much exclusively. But those can be fixed easily enough. The important thing for me was to check that the non-object storage paths are the same, and they read identically to me.<br></div>
<div style="font-family:Arial"> <br></div>
<div style="font-family:Arial">I've done a build and Cassandane test run, and I'm happy with it. Pushing a merge of that branch to master now. Sorry about taking so long to get to this.<br></div>
<div style="font-family:Arial"> <br></div>
<div style="font-family:Arial">Meeting in just under 2 hours, might see you there :)<br></div>
<div> <br></div>
<div style="font-family:Arial">Bron.<br></div>
<div> <br></div>
<div>On Wed, Feb 24, 2016, at 01:39, Raymond Poitras wrote:<br></div>
<blockquote type="cite"><div dir="ltr"><div style="font-family:Arial">Hi Bron<br></div>
<div> <br></div>
<div>Sorry for the delay and thank you for pushing this work to master.<br></div>
<div> <br></div>
<div>A) While doing this, some of the logic was broken. (This is now fix.)<br></div>
<div>B) The new solution works very well with a mix of archive (in object storage) and non archive (on regular spool) emails.<br></div>
<div>C) Unarchive have been fix. (was used previously to remove email from object storage. Now a new function is used for that).<br></div>
<div>D) Include reconstruct support. Works very well if message.db file is reliable.<br></div>
<div> <br></div>
<div>Please let me know if you can update master again from here <a href="https://github.com/RaymondPo/cyrus-imapd/tree/Cyrus-OpenIO" target="_blank">https://github.com/RaymondPo/cyrus-imapd/tree/Cyrus-OpenIO</a><br></div>
<div> <br></div>
<div>When is the next meeting? Jean-Francois and I would like to attend<br></div>
<div> <br></div>
<div>Regards<br></div>
<div>Raymond<br></div>
<div> <br></div>
</div>
<div><div style="font-family:Arial"> <br></div>
<div><div style="font-family:Arial">2015-10-13 8:03 GMT-04:00 Bron Gondwana <span dir="ltr"><<a href="mailto:brong@fastmail.fm" target="_blank">brong@fastmail.fm</a>></span>:<br></div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="font-family:Arial">I've pushed the objectstorage code based on Raymond and Jean-Francois's work to master. It's almost exactly what was on Raymond's branch with just a couple of modifications.<br></div>
<div style="font-family:Arial"> <br></div>
<div style="font-family:Arial">1) updated for some API changes I'd made to archiving in append.c<br></div>
<div style="font-family:Arial">2) fixed compilation under warn-unused-variables<br></div>
<div style="font-family:Arial">3) normalised logic between append_fromstage and append_copy for handling objectstorage errors by creating non-ARCHIVED files.<br></div>
<div style="font-family:Arial">4) fixed file copy in mailbox_archive to work the same way as the file moves, where it correctly passes the source/destination file to objectstorage and gets the file back, so unarchive works.<br></div>
<div style="font-family:Arial"> <br></div>
<div style="font-family:Arial">WARNING WARNING WARNING: while I've tested that normal operation and normal archive file operation (non-objectstorage) works correctly with Cassandane and FastMail experimental branch, I haven't done more than rudimentary testing with the dummy objectstorage.<br></div>
<div style="font-family:Arial"> <br></div>
<div style="font-family:Arial">I'm almost positive that reconstruct won't work correctly with object storage turned on right now.<br></div>
<div style="font-family:Arial"> <br></div>
<div style="font-family:Arial">This is still very much work in progress - but putting it on master means that the code will be maintained and updated as other API changes are made.<br></div>
<div style="font-family:Arial"> <br></div>
<div style="font-family:Arial">Thanks Raymond and Jean-Francois for all your work on this. I hope it gets more testing and use.<br></div>
<div style="font-family:Arial"> <br></div>
<div style="font-family:Arial">Cheers,<br></div>
<div style="font-family:Arial"> <br></div>
<div style="font-family:Arial">Bron.<br></div>
<div style="font-family:Arial"><span><span style="color:rgb(136,136,136)"><br>--<br>
Bron Gondwana<br>
<a href="mailto:brong@fastmail.fm" target="_blank">brong@fastmail.fm</a></span></span></div>
</blockquote></div>
</div>
</blockquote><div style="font-family:Arial"> <span><font color="#888888"><br></font></span></div><span><font color="#888888">
<div><div>-- <br></div>
<div> Bron Gondwana<br></div>
<div> <a href="mailto:brong@fastmail.fm" target="_blank">brong@fastmail.fm</a><br></div>
<div> <br></div>
</div>
<div style="font-family:Arial"> <br></div>
</font></span></blockquote><span><font color="#888888"><div style="font-family:Arial"> </div>
<div><div>-- <br></div>
<div> Bron Gondwana<br></div>
<div> <a href="mailto:brong@fastmail.fm" target="_blank">brong@fastmail.fm</a><br></div>
<div> </div>
</div>
<div style="font-family:Arial"> </div>
</font></span></div>
</blockquote></div><br></div></div>