Release plan blog post

Bron Gondwana brong at fastmail.fm
Tue Dec 27 19:04:04 EST 2016


Hi,


Sorry for the delay in responding to this - I left it over Christmas
so I could sit down without distraction and reply when I was back in
the office.


On Sat, 24 Dec 2016, at 17:09, Anatoli via Cyrus-devel wrote:

> Hi Bron, all.
>
>  Thanks for the update and for the support of the project. That's
>  great we'll see the 3.0 release soon!
>
>  Replying to your last paragraph in the blog post about the community
>  needs, I believe that what's good for FM is mostly good for the
>  community too. The FM team is probably the largest operator of the
>  project and has a better view / face issues and special needs more
>  frequently than anyone else, so your vision should suit well other
>  project users too.
>
>  A few areas where I see the FM needs probably don't exactly match the
>  needs of the community are the following 3.
>
> *1. **Small (SMB) deployments* with a single server and somehow
> limited physical resources (e.g. disk space).
>
>  Here as an example comes the excellent backup mechanism Ellie
>  implemented that suits well the needs of medium to large deployments,
>  but IMO that's not the best approach for small deployments, as it
>  requires a separate server or, if ran at the same server just for the
>  safe data-to-disk synchronization, twice the disk space.
>
>  A better approach for small deployments, as I see it (and I believe
>  it's highly demanded by the community), would be to have an
>  executable that would instruct Cyrus daemon to synchronize to disk
>  all the internal structures and lock (stop writing to disk) for a
>  defined period. The lock could be implemented by hanging on network
>  write requests or by writing them to temporary files, or by
>  accumulating the changes in memory (the latter approach has a
>  potential for data loss).
>
>  Once the flush is performed and the lock is applied, a (custom)
>  backup script could create a snapshot of the partition that would
>  hold the Cyrus data in a safe-to-backup state. Immediately after
>  creating the snapshot, the lock would be released and the daemon
>  would continue its normal operation. Then the backup script would be
>  able to safely backup the data, e.g. create an incremental backup and
>  upload it to some external storage, then destroy the snapshot.
>
>  Usage example: cyrus_sync2disk --lock=5 -> returns 0 when the data is
>  synced and a lock for 5 seconds is obtained. cyrus_sync2disk --unlock
>  -> returns 0 if the lock has been released and 1 if there was no
>  active lock (e.g a previous lock has expired), so the backup script
>  knows if it performed the required operations with the lock still in
>  place or if it should perform the lock-snapshot-unlock operation
>  again. The short timeout is to protect the daemon from an infinite
>  lock if a backup script fails to unlock it.
> 



I saw the reponse to this which suggested a "run a command under
exclusive lock' which is definitely a better approach to this.  I
understand what you want here, and I mostly like the idea.

The one thing that gives me pause is that it requires a single lock
against ALL cyrus processes.  Right now, there's no global lock that
processes take while making changes, and we'd need to add one.  I would
want to make it be something that needs to be turned on in config so
that sites which DON'T need it don't have to pay the extra locking cost.


But the design is definitely viable.  I want to do some other things
with locking as well, like a single global lock for moves between
users, renames, etc - so that we don't have lock ordering issues with
those things.


https://github.com/cyrusimap/cyrus-imapd/issues/1763



>
> *2. Small sysadmin tasks* for typical configurations that now require
> manual actions or writing one's own scripts. An example: new mailbox
> creation with particular flags (\Sent, \Junk, \Trash) set for special-
> use folders (that could be implemented as an extended functionality of
> the autocreate_inbox_folders option).
>
>  At FM you have everything automated for sure with your own customs
>  scripts, but sysadmins with little experience with Cyrus or those
>  that don't write scripts with ease would find some tasks difficult to
>  accomplish, for others that's just an overhead/additional points of
>  failure that could be avoided with small built-in automations.
> 



This is a definitely interesting area for enhancement.  The basic
tool here is cyradm, and I think what we're really looking for is
extending cyradm.


https://github.com/cyrusimap/cyrus-imapd/issues/1764



I'd love some more specific details here, including test plans ideally
so that we can build and test these features.  Or pull requests that
do that :)


>
> *3. New deployments* (vs ongoing upgrades/maintenance). How easy and
> straightforward it is to setup a new deployment (possibly migrating
> from other email servers). Here I'm referring to both the initial
> configuration, tools and documentation.
> 



Yeah, we know about this one.  I'm not going to create a specific bug
for it, because it's kind of spread out over lots of different things.
Nicola is working on improving our documentation, but again the best
people to give advice are people who've recently done it.  I haven't
really "installed Cyrus from scratch" for 12 years, certainly not
without the FastMail configuration and build systems.  Except for the
test environment, which has its own special magics.


>
> *Push* is an area that is well implemented at FM, but there's no
> considerable advance in the Cyrus repository, and I believe the
> community needs in this area are mostly the same as the FM's.
>
>  The 3.0 release includes Apple push notifications support
>  (XAPPLEPUSHSERVICE) and that's a good start. I haven't tried it yet
>  and I understand that some effort would be required to make it work
>  (the part that talks to the APS is not included and should be
>  implemented independently). I do wonder why wouldn't FM share the
>  notifier code & some documentation about how to make everything work?
>  The only thing that'd be different in each deployment are the
>  certificates. And it would be really exciting to have working apple
>  push in Cyrus just after some typical setup steps.
>
>  If there are some impediments for the FM team to share their
>  implementation details on mail and caldav/carddav push notifications,
>  I'll try to make this feature work in my deployments in the near
>  future and contribute to the project a detailed howto and the APS
>  notifier code (but your assistance would be great).


Unfortunately part of that is under NDA, so we can't offer much more
support there.  When/if Apple open up their push infrastructure more,
we'll definitely release the other parts of it.


I'm sure we've published at least part of our perl pusher layer before,
though some of the session magic uses our sql infrastructure rather than
storing sessions in Cyrus so that it survives failover between replicas.
If we wanted to store them in Cyrus we'd need to have a replication
protocol for key-value stores or some sort of replicated DB store.


>
>  And a general area that would benefit everyone, but that wasn't
>  specifically mentioned in the blog post, is *Security*.
>
>  I don't mean Cyrus is insecure, and I do know that the FM team pays
>  special attention to security of their infrastructure as a whole.
>  Rather I would like to suggest that a special emphasis could be
>  placed on Cyrus security from a development POV, e.g. to document in
>  detail (and keep updated) the entire project's code base and its
>  architecture, to follow most of the security development best-
>  practices, to re-implement with security in mind some old/hacky parts
>  of the system (they would become apparent during the documentation
>  phase), to apply general hardening tactics (like chroot) or even to
>  re-engineer the overall architecture for security, to perform
>  internal security code reviews on a regular basis.


This is the kind of well meaning plan that leads down a massive rabbit
hole.  "Document in detail (and keep updated)".  Such few words for so
much work.  We do bits and pieces of this as we can, and I've recently
set up coverity to assess the project, and am working my way through
its reports.


Certainly some parts of the code (like sieve) are a fricking mess, and
could very well be hiding security issues because they're just so
horrible.  We fix them up as we have time and deal with them.


>
>  FM already had a security audit in 2014 (according to your previous
>  blog posts), but you don't specify any details of how deep it was and
>  what aspects it covered. Maybe an independent in-depth security audit
>  with public results just for the Cyrus code base could be sponsored
>  in collaboration with the community?


Again, unfortunately NDAs :(



Feel free to sponsor a security audit.  I'd be happy to participate, but
I can't justify funding it.  I have an idea of where likely bugs are
(URLAUTH, FETCH BODY[part] until recently when we rewrote it, maybe even
message structure parsing) and I rewrite them to be safer when I deal
with those bits of code, as do the rest of the team.


> As for me as a member of the community, I have an intention to
> implement the chroot functionality for the daemon (late chroot like in
> OpenVPN). I've already discussed it briefly with Ellie and was hoping
> to make it ready for the 3.0 release, but had no time for it yet. To
> implement it correctly, first some important changes should be applied
> to the initialization logic (the moment of dropping the privs, it
> should be inside newly started processes, rather than in the master).
> This change should be carefully analyzed and it's a significant
> effort, I hope to be able to contribute it during the Q1/17. Once this
> change is implemented (which in itself wouldn't change almost any
> functionality, so it would be easy to test and deploy), the chroot
> functionality would be some 15 lines of code.


Interesting.  I'm looking forward to seeing it.



One thing that I would add here, is that we need to extract the
SNMP code from master and run it in a separate process as well if
we have any hope of making master something that can be allowed to
run with any higher privileges than it currently does in its
mainloop.  Greg explained to me what he had planned for that, but
never had time to do it.


https://github.com/cyrusimap/cyrus-imapd/issues/1765



> 
>  Merry Christmas and Happy New Year!

> 



Thanks, the same to you!  



Regards,



Bron.



> 

> 
>  Anatoli

> 

> *From:* Bron Gondwana Via Cyrus-devel

> *Sent:* Thursday, December 22, 2016 03:15

> *To:* Cyrus Devel, Info Cyrus

> *Subject:* Release plan blog post
>> I posted on the FastMail advent about our plans for releasing Cyrus
>> 3.0 - it's a bit roundabout doing it this way rather than here first,
>> but hey - we talked about it on Monday night's regular meeting.
>> Here's the blog post:

>> https://blog.fastmail.com/2016/12/22/cyrus-development-and-release-plans/
>> tl;dr, Ellie recently released 3.0beta6.  We're going to do a release
>> candidate on Jan 13th and then release for real soon afterwards, so
>> get testing!  There are no major changes expected before release.
>> I'll be doing a couple of small JMAP changes to align with the latest
>> spec and possibly to add getMessageListUpdates if I can manage it in
>> time.  Other than that, I'm looking a reverse UniqueId indexing
>> similar to the RACL support - it's already in testing and might get
>> added behind a default-off config switch.  We'll be assessing all the
>> defaults.  I'm really tempted to turn RACL on, but it needs group
>> support if your site uses groups, and that's not done yet, so I'd
>> need someone willing to test it!  Bron.

>>


--

  Bron Gondwana

  brong at fastmail.fm




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20161228/7ff3dd7a/attachment-0001.html>


More information about the Cyrus-devel mailing list