objectstore: don't build unless asked for

Sat Nov 7 10:22:34 EST 2015

Hi Ellie

I think you got most of it.

The object storage implementation was built on top of the "archive_enabled"
feature.  Both must be enable for the object storage to kick in.  Or in
other words it is only when a file (email) must be archive that it will go
into object storage.

You can enable this with the following in Imap.conf

archive_enabled: yes
archive_days: 0  (this will make sure that all emails will go strait into
archive,  and if object storage is enable it will go into object storage
instead)

object_storage_enabled: yes
object_storage_dummy_spool: /var/objectstore

You can check functionality by cross checking files in
the object_storage_dummy_spool and information in the user database

sqlite3 message.db  (message.db is the database created in each user
directory in the spool directory.)
sqlite> SELECT * FROM user_msg;  (will dump the information)

54d78bfa187279fad2fa93356f0546dbcfd9b6ab|{"mboxs":{"uid":
user.ray.MyStuff.toto.abc.1}}

-in yellow is the message GUID (used as file name in the all object storage
implementations)
-In red is where the file should be if not in object storage.
-In greed is what the file name should be if not in object storage.

Note: you can have many "uid:" tags for one message_guid  if the same email
is in multiple mailboxes.  (Since the delayed deletion and rename of
mailboxes imply copying all emails and removing old ones, this is a huge
gain in performance in object storage solution, as it eliminate all file
movement. It only add and remove tags in the database)
----------------

There was many reason for this supplied dummy solution.
1- All object storage solution are not easy to setup.  The dummy solution
is there to make sure someone can test easily the functionality.
2- I am not a fan of #ifdef.  The alternative was to supply "dummy"
functions calls for the code to compile.  (if object_storage_enabled is not
defined in Imapd.conf this code is simply not used.)
3-This solution is also super lite compare to object storage deployment.
4-It is also a guide to measure performance of the different
implementations.

I may have time in a few weeks to test your changes if you can wait.  If
not you can contact Jean-Francois at OpenIo to help test the OpenIo
solution. He can probably help document the missing dependencies at the
same time.  (At the moment of development of the object storage solution,
OpenIo was in the middle of eliminating dependencies, so the long list of
dependencies was getting shorter)

Important to say that OpenIo have a real test going on now.  Hopefully they
should be able to tell us more about that soon.

As for Caringo it is a bit more complicate.  Caringo is not an open source
solution.  Therefore the SDK is not public.  You need an account to get it.

Finally I plead guilty for the Sqlite dependency.  Sorry for that and
thanks for fixing this.
As you find out the alternative was to include --enable-httpd, (It is using
small portion of code added by Ken for httpd ; the opening/creation of the
database with version support.;  The version support is mandatory if we
need to change the database format one day)

Thanks
Raymond

2015-11-06 2:44 GMT-05:00 ellie timoney via Cyrus-devel <
cyrus-devel at lists.andrew.cmu.edu>:

> At present, our master branch unconditionally builds in the object
> storage functionality (using the objectstore_dummy implementation).
> There's a few ways this isn't great:
>
> * it has an undocumented dependency on lib/sqldb, but doesn't try to
> link it in, so the build fails unless you --enable-httpd, which is
> currently the only thing that tries to link in lib/sqldb.  (This is
> what's causing the HarborMaster build failures, fwiw.)
> * it brings in a bunch of experimental code with no way to compile that
> out
> * there's some configure settings for dealing with it, but they don't
> interact well together
>
> I had a chat with Ken and Nicola today, and have put together some
> changes, with the following goals:
>
> * as light a touch as reasonable
> * add an --enable-objectstore configure option (default: no) to control
> whether objectstore code is built at all
> * keep the existing --with-openio[...] and --with-caringo options for
> selecting the backend
> * ditch the --enable-dummy_objstore option altogether
> * use the dummy implementation only if objectstore is enabled but
> neither backend has been selected
>
> The changes are on the "v30/optional-objectstore" branch on my github,
> for now:
> https://github.com/elliefm/cyrus-imapd/tree/v30/optional-objectstore
>
> This isn't on master yet cause there's some caveats:
>
> The implementation necessitates a bunch of "#ifdef ENABLE_OBJECTSTORE"
> around the code paths that use it (imap/append.c and imap/mailbox.c).  I
> don't much care for this -- it makes maintenance and testing down the
> track fiddly.  But the alternatives seem to be the current state, which
> we already know is broken, or adding a no-op objectstore implementation.
>  The latter seems pretty nasty from the way we use this: there's lots of
> code like "if the option is set in imapd.conf, do it the objectstore
> way, otherwise, do it the normal way".  If we were to just drop in a
> no-op objectstore implementation then cyrus admins would be only a
> config setting away from data loss, and that feels all kinds of wrong.
> So, #ifdefs.
>
> The dummy implementation is not a no-op implementation, it's a flat
> file-based emulation.  "Dummy" is possibly a misleading name here, by
> which I mean, it mislead me until I read the code.  Maybe we should
> rename it, but for now: light touch.
>
> The caringo/openio detection code is pretty frail and needs attention.
> For example, if you --with-caringo=yes without having the dependencies
> installed, it just happily adds the dependencies straight into
> LDFLAGS(!)... which makes subsequent unrelated checks that try to
> compile a code fragment fail unexpectedly.  The openio path has the good
> grace to fail clumsily on its own when the dependencies are missing,
> though it is just as guilty of blatting into stuff it shouldn't touch.
>
> The --enable-objectstore option still has the undocumented dependency on
> --enable-http (in order to bring in lib/sqldb).  Light touch.  I'll end
> up fixing this at some point anyway if no-one else does first, because
> the backup code I'm working on has a similar dependency.
>
> I'm not sure about the terminology here -- "object store" / "object
> storage" / various capitalisations / without spaces.  I've just made
> some wild guesses for now.  If you have an opinion, tell me about it.
>
> The imapd.conf setting for enabling object storage is always present,
> whether the feature is compiled in or not.  There's a fair bit of work
> in making this not the case (changing the scripts that generate
> lib/imapopts.[ch] and man/imapd.conf.5 from lib/imapoptions).  And we
> already have other settings that are nonsensical unless optional
> features are compiled in, so there's precedent in just leaving it as is
> -- at least for now. Light touch.
>
> And for all that, I don't have the dependencies for either Caringo or
> OpenIO handy, so I haven't tested those code paths beyond verifying that
> it fails as expected when I try to enable them.  I've tried not to alter
> what was already there, but I can't verify that I haven't accidentally
> broken something somehow.  If you've been able to get either of these
> actually working, I'd appreciate if you could try again with my branch
> and let me know if it still works.  It does pass our tests without
> objectstore, and with the dummy objectstore implementation.
>
> So, those sure are some caveats hey.
>
> I think the main questions at this point are:
>
> * are we happy with the combination of --enable-objectstore to turn the
> feature on, plus --with-[backend] to select a backend, and default to
> dummy if no other backend was selected?
> * are we happy/comfortable/resigned to the #ifdef'd approach to
> compiling it out?
> * did I manage to avoid breaking anything that used to work?
>
> If those are yes, then I'll merge it onto master, and the rest can be
> figured out as we go.
>
> Cheers,
>
> ellie
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20151107/ee4b182f/attachment-0001.html>