Object Storage alpha lands on master

Tue Mar 15 20:16:59 EDT 2016

Hi Sayid,

Cyrus has been updated to reflect OpenIO latest version. The solution evolved a lot since the 1.0.1 you mention!
If you prefer using a release number, the 2.0.0.c1 should be your current choice: it is the first candidate for a release scheduled for 2016/04, destined to be a stable branch on the long term.

Feel free to ask if you need any additional information.

--
Jean-Francois

________________________________
De : Cyrus-devel [cyrus-devel-bounces+jean-francois.smigielski=openio.io at lists.andrew.cmu.edu] de la part de Sayid Munawar via Cyrus-devel [cyrus-devel at lists.andrew.cmu.edu]
Date d'envoi : mardi 15 mars 2016 21:27
À : Cyrus Devel
Objet : Fwd: Object Storage alpha lands on master

Hi, i've just tried to compile cyrus with openio, and it gave this error:

libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I. -I./lib -I. -I./lib -DLIBEXEC_DIR=\"/usr/cyrus/libexec\" -DSBIN_DIR=\"/usr/cyrus/sbin\" -DSYSCONF_DIR=\"/etc\" -DHAVE_CONFIG_H -I/usr/include/libxml2 -I/usr/include/postgresql -fPIC -fvisibility=hidden -g -O2 -I/usr/include -MT imap/imap_libcyrus_imap_la-objectstore_openio.lo -MD -MP -MF imap/.deps/imap_libcyrus_imap_la-objectstore_openio.Tpo -c imap/objectstore_openio.c  -fPIC -DPIC -o imap/.libs/imap_libcyrus_imap_la-objectstore_openio.o
imap/objectstore_openio.c: In function 'objectstore_put':
imap/objectstore_openio.c:195:42: warning: passing argument 2 of 'oio_sds_upload_from_file' from incompatible pointer type
     err = oio_sds_upload_from_file (sds, &ul_dst, fname, 0, 0);
                                          ^
In file included from imap/objectstore_openio.c:49:0:
/usr/include/oio_sds.h:250:21: note: expected 'struct oio_url_s *' but argument is of type 'struct oio_sds_ul_dst_s *'
 struct oio_error_s* oio_sds_upload_from_file (struct oio_sds_s *sds,
                     ^
imap/objectstore_openio.c:195:11: error: too many arguments to function 'oio_sds_upload_from_file'
     err = oio_sds_upload_from_file (sds, &ul_dst, fname, 0, 0);
           ^
In file included from imap/objectstore_openio.c:49:0:
/usr/include/oio_sds.h:250:21: note: declared here
 struct oio_error_s* oio_sds_upload_from_file (struct oio_sds_s *sds,
                     ^
Makefile:4679: recipe for target 'imap/imap_libcyrus_imap_la-objectstore_openio.lo' failed
make[2]: *** [imap/imap_libcyrus_imap_la-objectstore_openio.lo] Error 1
make[2]: Leaving directory '/opt/cyrus-imapd'
Makefile:5800: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/opt/cyrus-imapd'
Makefile:2511: recipe for target 'all' failed
make: *** [all] Error 2
root at jessie:/opt/cyrus-imapd#

any clue how to fix this ?

OS: debian jessie amd64
openio-sds installed version 1.0.1-1

On Mon, Mar 14, 2016 at 4:14 PM, Bron Gondwana via Cyrus-devel <cyrus-devel at lists.andrew.cmu.edu<mailto:cyrus-devel at lists.andrew.cmu.edu>> wrote:
There were a couple of nits around style:

+    int remove_temp_spool_file = 0 ;
+    int object_storage_enabled = 0 ;

(we don't put a space before the ; anywhere in the rest of the codebase)

And some
}
else
{

or

if (condition)
{

We cuddle the brace like this:

}
else {

if (condition) {

Pretty much exclusively.  But those can be fixed easily enough.  The important thing for me was to check that the non-object storage paths are the same, and they read identically to me.

I've done a build and Cassandane test run, and I'm happy with it. Pushing a merge of that branch to master now.  Sorry about taking so long to get to this.

Meeting in just under 2 hours, might see you there :)

Bron.

On Wed, Feb 24, 2016, at 01:39, Raymond Poitras wrote:
Hi Bron

Sorry for the delay and thank you for pushing this work to master.

A) While doing this, some of the logic was broken.  (This is now fix.)
B) The new solution works very well with a mix of archive (in object storage) and non archive (on regular spool) emails.
C) Unarchive have been fix.  (was used previously to remove email from object storage.  Now a new function is used for that).
D) Include reconstruct support.  Works very well if message.db file is reliable.

Please let me know if you can update master again from here https://github.com/RaymondPo/cyrus-imapd/tree/Cyrus-OpenIO

When is the next meeting? Jean-Francois and I would like to attend

Regards
Raymond

2015-10-13 8:03 GMT-04:00 Bron Gondwana <brong at fastmail.fm<mailto:brong at fastmail.fm>>:
I've pushed the objectstorage code based on Raymond and Jean-Francois's work to master.  It's almost exactly what was on Raymond's branch with just a couple of modifications.

1) updated for some API changes I'd made to archiving in append.c
2) fixed compilation under warn-unused-variables
3) normalised logic between append_fromstage and append_copy for handling objectstorage errors by creating non-ARCHIVED files.
4) fixed file copy in mailbox_archive to work the same way as the file moves, where it correctly passes the source/destination file to objectstorage and gets the file back, so unarchive works.

WARNING WARNING WARNING: while I've tested that normal operation and normal archive file operation (non-objectstorage) works correctly with Cassandane and FastMail experimental branch, I haven't done more than rudimentary testing with the dummy objectstorage.

I'm almost positive that reconstruct won't work correctly with object storage turned on right now.

This is still very much work in progress - but putting it on master means that the code will be maintained and updated as other API changes are made.

Thanks Raymond and Jean-Francois for all your work on this.  I hope it gets more testing and use.

Cheers,

Bron.

--
  Bron Gondwana
  brong at fastmail.fm<mailto:brong at fastmail.fm>

--
  Bron Gondwana
  brong at fastmail.fm<mailto:brong at fastmail.fm>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20160316/c57c1bc0/attachment-0001.html>