Object Storage alpha lands on master

Raymond Poitras poitrasray at gmail.com
Thu May 19 16:09:46 EDT 2016


Hi Bron,

Really sorry for the delay. It took me a while to get back to this.
It is harmless (other than a waste of CPU cycle) to call delete "if the
record isn't there".
It must be found in that specific mailbox for something to happen.

Nevertheless, I made some small logic adjustment.
Also cleanup configure.ac regarding OpenIO options

You can find my changes as usual over here

https://github.com/RaymondPo/cyrus-imapd

and now

https://github.com/open-io/cyrus-imapd

Thanks
Raymond



2016-03-18 6:35 GMT-04:00 Bron Gondwana via Cyrus-devel <
cyrus-devel at lists.andrew.cmu.edu>:

> And it turns out, there was a major bug in FastMail style Archiving
> (non-objectstore) to the point that it deleted emails after archiving
> them.  Ouch.
>
> Thanksfully we run one replica in archive-immediately mode, and it still
> had the messages - and we have a script that can rescue them back, so
> nothing was lost.
>
> (stage 2 rescue would have been pulling from backups, which also works but
> is slower)
>
> Patch on master now.  Raymond, you might want to check of the
> objectstorage logic.  It might call 'delete' if the record isn't there.  Is
> that a problem?  Presumably the refcount is smart enough to detect this,
> because otherwise there are failure modes where it could get out of sync as
> well.
>
> Bron.
>
>
> On Mon, Mar 14, 2016, at 20:14, Bron Gondwana via Cyrus-devel wrote:
>
> There were a couple of nits around style:
>
> +    int remove_temp_spool_file = 0 ;
> +    int object_storage_enabled = 0 ;
>
> (we don't put a space before the ; anywhere in the rest of the codebase)
>
> And some
> }
> else
> {
>
> or
>
> if (condition)
> {
>
> We cuddle the brace like this:
>
> }
> else {
>
> if (condition) {
>
> Pretty much exclusively.  But those can be fixed easily enough.  The
> important thing for me was to check that the non-object storage paths are
> the same, and they read identically to me.
>
> I've done a build and Cassandane test run, and I'm happy with it. Pushing
> a merge of that branch to master now.  Sorry about taking so long to get to
> this.
>
> Meeting in just under 2 hours, might see you there :)
>
> Bron.
>
> On Wed, Feb 24, 2016, at 01:39, Raymond Poitras wrote:
>
> Hi Bron
>
> Sorry for the delay and thank you for pushing this work to master.
>
> A) While doing this, some of the logic was broken.  (This is now fix.)
> B) The new solution works very well with a mix of archive (in object
> storage) and non archive (on regular spool) emails.
> C) Unarchive have been fix.  (was used previously to remove email from
> object storage.  Now a new function is used for that).
> D) Include reconstruct support.  Works very well if message.db file is
> reliable.
>
> Please let me know if you can update master again from here
> https://github.com/RaymondPo/cyrus-imapd/tree/Cyrus-OpenIO
>
> When is the next meeting? Jean-Francois and I would like to attend
>
> Regards
> Raymond
>
>
> 2015-10-13 8:03 GMT-04:00 Bron Gondwana <brong at fastmail.fm>:
>
> I've pushed the objectstorage code based on Raymond and Jean-Francois's
> work to master.  It's almost exactly what was on Raymond's branch with just
> a couple of modifications.
>
> 1) updated for some API changes I'd made to archiving in append.c
> 2) fixed compilation under warn-unused-variables
> 3) normalised logic between append_fromstage and append_copy for handling
> objectstorage errors by creating non-ARCHIVED files.
> 4) fixed file copy in mailbox_archive to work the same way as the file
> moves, where it correctly passes the source/destination file to
> objectstorage and gets the file back, so unarchive works.
>
> WARNING WARNING WARNING: while I've tested that normal operation and
> normal archive file operation (non-objectstorage) works correctly with
> Cassandane and FastMail experimental branch, I haven't done more than
> rudimentary testing with the dummy objectstorage.
>
> I'm almost positive that reconstruct won't work correctly with object
> storage turned on right now.
>
> This is still very much work in progress - but putting it on master means
> that the code will be maintained and updated as other API changes are made.
>
> Thanks Raymond and Jean-Francois for all your work on this.  I hope it
> gets more testing and use.
>
> Cheers,
>
> Bron.
>
> --
>   Bron Gondwana
>   brong at fastmail.fm
>
>
> --
>   Bron Gondwana
>   brong at fastmail.fm
>
>
>
>
> --
>   Bron Gondwana
>   brong at fastmail.fm
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20160519/314bd8f6/attachment.html>


More information about the Cyrus-devel mailing list