Object Storage alpha lands on master

[Bron Gondwana]

And it turns out, there was a major bug in FastMail style Archiving
(non-objectstore) to the point that it deleted emails after archiving
them.  Ouch.
 
Thanksfully we run one replica in archive-immediately mode, and it still
had the messages - and we have a script that can rescue them back, so
nothing was lost.
 
(stage 2 rescue would have been pulling from backups, which also works
but is slower)
 
Patch on master now.  Raymond, you might want to check of the
objectstorage logic.  It might call 'delete' if the record isn't there.
Is that a problem?  Presumably the refcount is smart enough to detect
this, because otherwise there are failure modes where it could get out
of sync as well.
 
Bron.
 
 
On Mon, Mar 14, 2016, at 20:14, Bron Gondwana via Cyrus-devel wrote:
> There were a couple of nits around style:
>
> +    int remove_temp_spool_file = 0 ;
> +    int object_storage_enabled = 0 ;
>
> (we don't put a space before the ; anywhere in the rest of the
> codebase)
>
> And some
> }
> else
> {
>
> or
>
> if (condition)
> {
>
> We cuddle the brace like this:
>
> }
> else {
>
> if (condition) {
>
> Pretty much exclusively.  But those can be fixed easily enough.  The
> important thing for me was to check that the non-object storage paths
> are the same, and they read identically to me.
>
> I've done a build and Cassandane test run, and I'm happy with it.
> Pushing a merge of that branch to master now.  Sorry about taking so
> long to get to this.
>
> Meeting in just under 2 hours, might see you there :)
>
> Bron.
>
> On Wed, Feb 24, 2016, at 01:39, Raymond Poitras wrote:
>> Hi Bron
>>
>> Sorry for the delay and thank you for pushing this work to master.
>>
>> A) While doing this, some of the logic was broken.  (This is now
>>    fix.)
>> B) The new solution works very well with a mix of archive (in object
>>    storage) and non archive (on regular spool) emails.
>> C) Unarchive have been fix.  (was used previously to remove email
>>    from object storage.  Now a new function is used for that).
>> D) Include reconstruct support.  Works very well if message.db file
>>    is reliable.
>>
>> Please let me know if you can update master again from here https://github.com/RaymondPo/cyrus-imapd/tree/Cyrus-OpenIO
>>
>> When is the next meeting? Jean-Francois and I would like to attend
>>
>> Regards
>> Raymond
>>
>>
>> 2015-10-13 8:03 GMT-04:00 Bron Gondwana <brong at fastmail.fm>:
>>> I've pushed the objectstorage code based on Raymond and Jean-
>>> Francois's work to master.  It's almost exactly what was on
>>> Raymond's branch with just a couple of modifications.
>>>
>>> 1) updated for some API changes I'd made to archiving in append.c
>>> 2) fixed compilation under warn-unused-variables
>>> 3) normalised logic between append_fromstage and append_copy for
>>>    handling objectstorage errors by creating non-ARCHIVED files.
>>> 4) fixed file copy in mailbox_archive to work the same way as the
>>>    file moves, where it correctly passes the source/destination file
>>>    to objectstorage and gets the file back, so unarchive works.
>>>
>>> WARNING WARNING WARNING: while I've tested that normal operation and normal archive file operation (non-objectstorage) works correctly with Cassandane and FastMail experimental branch, I haven't done more than rudimentary testing with the dummy objectstorage.
>>>
>>> I'm almost positive that reconstruct won't work correctly with
>>> object storage turned on right now.
>>>
>>> This is still very much work in progress - but putting it on master
>>> means that the code will be maintained and updated as other API
>>> changes are made.
>>>
>>> Thanks Raymond and Jean-Francois for all your work on this.  I hope
>>> it gets more testing and use.
>>>
>>> Cheers,
>>>
>>> Bron.
>>>
>>> --
>>>
Bron Gondwana
>>>
brong at fastmail.fm
>
> --
> Bron Gondwana
> brong at fastmail.fm
>
>
 
-- 
Bron Gondwana
brong at fastmail.fm
 
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20160318/e98c4f1f/attachment.html>