Release plan blog post

Sat Dec 24 01:09:00 EST 2016

Hi Bron, all.

Thanks for the update and for the support of the project. That's great 
we'll see the 3.0 release soon!

Replying to your last paragraph in the blog post about the community 
needs, I believe that what's good for FM is mostly good for the 
community too. The FM team is probably the largest operator of the 
project and has a better view / face issues and special needs more 
frequently than anyone else, so your vision should suit well other 
project users too.

A few areas where I see the FM needs probably don't exactly match the 
needs of the community are the following 3.

*1. **Small (SMB) deployments* with a single server and somehow limited 
physical resources (e.g. disk space).

Here as an example comes the excellent backup mechanism Ellie 
implemented that suits well the needs of medium to large deployments, 
but IMO that's not the best approach for small deployments, as it 
requires a separate server or, if ran at the same server just for the 
safe data-to-disk synchronization, twice the disk space.

A better approach for small deployments, as I see it (and I believe it's 
highly demanded by the community), would be to have an executable that 
would instruct Cyrus daemon to synchronize to disk all the internal 
structures and lock (stop writing to disk) for a defined period. The 
lock could be implemented by hanging on network write requests or by 
writing them to temporary files, or by accumulating the changes in 
memory (the latter approach has a potential for data loss).

Once the flush is performed and the lock is applied, a (custom) backup 
script could create a snapshot of the partition that would hold the 
Cyrus data in a safe-to-backup state. Immediately after creating the 
snapshot, the lock would be released and the daemon would continue its 
normal operation. Then the backup script would be able to safely backup 
the data, e.g. create an incremental backup and upload it to some 
external storage, then destroy the snapshot.

Usage example: cyrus_sync2disk --lock=5 -> returns 0 when the data is 
synced and a lock for 5 seconds is obtained. cyrus_sync2disk --unlock -> 
returns 0 if the lock has been released and 1 if there was no active 
lock (e.g a previous lock has expired), so the backup script knows if it 
performed the required operations with the lock still in place or if it 
should perform the lock-snapshot-unlock operation again. The short 
timeout is to protect the daemon from an infinite lock if a backup 
script fails to unlock it.

*2. **Small sysadmin tasks* for typical configurations that now require 
manual actions or writing one's own scripts. An example: new mailbox 
creation with particular flags (\Sent, \Junk, \Trash) set for 
special-use folders (that could be implemented as an extended 
functionality of the autocreate_inbox_folders option).

At FM you have everything automated for sure with your own customs 
scripts, but sysadmins with little experience with Cyrus or those that 
don't write scripts with ease would find some tasks difficult to 
accomplish, for others that's just an overhead/additional points of 
failure that could be avoided with small built-in automations.

*3. **New deployments* (vs ongoing upgrades/maintenance). How easy and 
straightforward it is to setup a new deployment (possibly migrating from 
other email servers). Here I'm referring to both the initial 
configuration, tools and documentation.

*Push* is an area that is well implemented at FM, but there's no 
considerable advance in the Cyrus repository, and I believe the 
community needs in this area are mostly the same as the FM's.

The 3.0 release includes Apple push notifications support 
(XAPPLEPUSHSERVICE) and that's a good start. I haven't tried it yet and 
I understand that some effort would be required to make it work (the 
part that talks to the APS is not included and should be implemented 
independently). I do wonder why wouldn't FM share the notifier code & 
some documentation about how to make everything work? The only thing 
that'd be different in each deployment are the certificates. And it 
would be really exciting to have working apple push in Cyrus just after 
some typical setup steps.

If there are some impediments for the FM team to share their 
implementation details on mail and caldav/carddav push notifications, 
I'll try to make this feature work in my deployments in the near future 
and contribute to the project a detailed howto and the APS notifier code 
(but your assistance would be great).

And a general area that would benefit everyone, but that wasn't 
specifically mentioned in the blog post, is *Security*.

I don't mean Cyrus is insecure, and I do know that the FM team pays 
special attention to security of their infrastructure as a whole. Rather 
I would like to suggest that a special emphasis could be placed on Cyrus 
security from a development POV, e.g. to document in detail (and keep 
updated) the entire project's code base and its architecture, to follow 
most of the security development best-practices, to re-implement with 
security in mind some old/hacky parts of the system (they would become 
apparent during the documentation phase), to apply general hardening 
tactics (like chroot) or even to re-engineer the overall architecture 
for security, to perform internal security code reviews on a regular basis.

FM already had a security audit in 2014 (according to your previous blog 
posts), but you don't specify any details of how deep it was and what 
aspects it covered. Maybe an independent in-depth security audit with 
public results just for the Cyrus code base could be sponsored in 
collaboration with the community?

As for me as a member of the community, I have an intention to implement 
the chroot functionality for the daemon (late chroot like in OpenVPN). 
I've already discussed it briefly with Ellie and was hoping to make it 
ready for the 3.0 release, but had no time for it yet. To implement it 
correctly, first some important changes should be applied to the 
initialization logic (the moment of dropping the privs, it should be 
inside newly started processes, rather than in the master). This change 
should be carefully analyzed and it's a significant effort, I hope to be 
able to contribute it during the Q1/17. Once this change is implemented 
(which in itself wouldn't change almost any functionality, so it would 
be easy to test and deploy), the chroot functionality would be some 15 
lines of code.

Merry Christmas and Happy New Year!
Anatoli

*From:* Bron Gondwana Via Cyrus-devel
*Sent:* Thursday, December 22, 2016 03:15
*To:* Cyrus Devel, Info Cyrus
*Subject:* Release plan blog post

> I posted on the FastMail advent about our plans for releasing Cyrus 3.0 - it's a bit roundabout doing it this way rather than here first, but hey - we talked about it on Monday night's regular meeting.
>
> Here's the blog post:
>
> https://blog.fastmail.com/2016/12/22/cyrus-development-and-release-plans/
>
> tl;dr, Ellie recently released 3.0beta6.  We're going to do a release candidate on Jan 13th and then release for real soon afterwards, so get testing!
>
> There are no major changes expected before release.  I'll be doing a couple of small JMAP changes to align with the latest spec and possibly to add getMessageListUpdates if I can manage it in time.
>
> Other than that, I'm looking a reverse UniqueId indexing similar to the RACL support - it's already in testing and might get added behind a default-off config switch.
>
> We'll be assessing all the defaults.  I'm really tempted to turn RACL on, but it needs group support if your site uses groups, and that's not done yet, so I'd need someone willing to test it!
>
> Bron.
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20161224/90a4ddaa/attachment.html>