<!DOCTYPE html>
<html>
<head>
<title></title>
</head>
<body><div style="font-family:Arial;">Hi,<br></div>
<div style="font-family:Arial;"><br>Sorry for the delay in responding to this - I left it over Christmas so I could sit down without distraction and reply when I was back in the office.<br></div>
<div><br></div>
<div>On Sat, 24 Dec 2016, at 17:09, Anatoli via Cyrus-devel wrote:<br></div>
<blockquote type="cite"><div><div style="font-size:10pt;font-family:Verdana,Arial;"><div style="font-family:Arial;"><span class="font" style="font-family:Verdana">Hi Bron, all.<br> <br> Thanks for the update and for the support of the project.
That's great we'll see the 3.0 release soon!<br> <br> Replying to your last paragraph in the blog post about the
community needs, I believe that what's good for FM is mostly
good for the community too. The FM team is probably the
largest operator of the project and has a better view / face
issues and special needs more frequently than anyone else, so
your vision should suit well other project users too.<br> <br> A few areas where I see the FM needs probably don't exactly
match the needs of the community are the following 3.<br> <br> <b>1. </b><b>Small (SMB) deployments</b> with a single server
and somehow limited physical resources (e.g. disk space).<br> <br> Here as an example comes the excellent backup mechanism Ellie
implemented that suits well the needs of medium to large
deployments, but IMO that's not the best approach for small
deployments, as it requires a separate server or, if ran at
the same server just for the safe data-to-disk
synchronization, twice the disk space.<br> <br> A better approach for small deployments</span><span class="font" style="font-family:Verdana"><span class="font" style="font-family:Verdana">, as I see it</span> (and
I believe it's highly demanded by the community), would be to
have an executable that would instruct Cyrus daemon to
synchronize to disk all the internal structures and lock (stop
writing to disk) for a defined period. The lock could be
implemented by hanging on network write requests or by writing
them to temporary files, or by accumulating the changes in
memory (the latter approach has a potential for data loss).<br> <br> Once the flush is performed and the lock is applied, a
(custom) backup script could create a snapshot of the
partition that would hold the Cyrus data in a safe-to-backup
state. Immediately after creating the snapshot, the lock would
be released and the daemon would continue its normal
operation. Then the backup script would be able to safely
backup the data, e.g. create an incremental backup and upload
it to some external storage, then destroy the snapshot.<br> <br> Usage example: <span class="font" style="font-family:"Courier New"">cyrus_sync2disk
--lock=5</span> -> returns 0 when the data is synced and
a lock for 5 seconds is obtained. <span class="font" style="font-family:"Courier New"">cyrus_sync2disk
--unlock</span> -> returns 0 if the lock has been
released and 1 if there was no active lock (e.g a previous
lock has expired), so the backup script knows if it performed
the required operations with the lock still in place or if it
should perform the lock-snapshot-unlock operation again. The
short timeout is to protect the daemon from an infinite lock
if a backup script fails to unlock it.<br> </span></div>
<div style="font-family:Arial;"><br></div>
</div>
</div>
</blockquote><div><div style="font-size:10pt;font-family:Verdana,Arial;"><div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">I saw the reponse to this which suggested a "run a command under exclusive lock' which is definitely a better approach to this. I understand what you want here, and I mostly like the idea.<br></div>
<div style="font-family:Arial;"><br>The one thing that gives me pause is that it requires a single lock against ALL cyrus processes. Right now, there's no global lock that processes take while making changes, and we'd need to add one. I would want to make it be something that needs to be turned on in config so that sites which DON'T need it don't have to pay the extra locking cost.</div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">But the design is definitely viable. I want to do some other things
with locking as well, like a single global lock for moves between users,
renames, etc - so that we don't have lock ordering issues with those
things.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;"><a href="https://github.com/cyrusimap/cyrus-imapd/issues/1763">https://github.com/cyrusimap/cyrus-imapd/issues/1763</a><br></div>
<div style="font-family:Arial;"><br></div>
</div>
</div>
<blockquote type="cite"><div><div style="font-size:10pt;font-family:Verdana,Arial;"><div style="font-family:Arial;"><span class="font" style="font-family:Verdana"><br> <b>2. Small sysadmin tasks</b> for typical
configurations that now require manual actions or writing
one's own scripts. An example: new mailbox creation with
particular flags (\Sent, \Junk, \Trash) set for special-use
folders (that could be implemented as an extended
functionality of the <span class="font" style="font-family:"Courier New"">autocreate_inbox_folders</span> option).<br> <br> At FM you have everything automated for sure with your own
customs scripts, but sysadmins with little experience with
Cyrus or those that don't write scripts with ease would find
some tasks difficult to accomplish, for others that's just an
overhead/additional points of failure that could be avoided
with small built-in automations.<br> </span></div>
<div style="font-family:Arial;"><br></div>
</div>
</div>
</blockquote><div><div style="font-size:10pt;font-family:Verdana,Arial;"><div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">This is a definitely interesting area for enhancement. The basic tool here is cyradm, and I think what we're really looking for is extending cyradm.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;"><a href="https://github.com/cyrusimap/cyrus-imapd/issues/1764">https://github.com/cyrusimap/cyrus-imapd/issues/1764</a><br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">I'd love some more specific details here, including test plans ideally so that we can build and test these features. Or pull requests that do that :)<br></div>
<div style="font-family:Arial;"><br></div>
</div>
</div>
<blockquote type="cite"><div><div style="font-size:10pt;font-family:Verdana,Arial;"><div style="font-family:Arial;"><span class="font" style="font-family:Verdana"><br> <b>3. New deployments</b> (vs ongoing
upgrades/maintenance). How easy and straightforward it is to
setup a new deployment (possibly migrating from other email
servers). Here I'm referring to both the initial
configuration, tools and documentation.<br> </span></div>
<div style="font-family:Arial;"><br></div>
</div>
</div>
</blockquote><div><div style="font-size:10pt;font-family:Verdana,Arial;"><div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Yeah, we know about this one. I'm not going to create a specific bug for it, because it's kind of spread out over lots of different things. Nicola is working on improving our documentation, but again the best people to give advice are people who've recently done it. I haven't really "installed Cyrus from scratch" for 12 years, certainly not without the FastMail configuration and build systems. Except for the test environment, which has its own special magics.<br></div>
<div style="font-family:Arial;"><br></div>
</div>
</div>
<blockquote type="cite"><div><div style="font-size:10pt;font-family:Verdana,Arial;"><div style="font-family:Arial;"><span class="font" style="font-family:Verdana"><br> <b>Push</b> is an area that is well implemented at FM, but
there's no considerable advance in the Cyrus repository, and I
believe the community needs in this area are mostly the same
as the FM's.<br> <br> The 3.0 release includes Apple push notifications support
(XAPPLEPUSHSERVICE) and that's a good start. I haven't tried
it yet and I understand that some effort would be required to
make it work (the part that talks to the APS is not included
and should be implemented independently). I do wonder why
wouldn't FM share the notifier code & some documentation
about how to make everything work? The only thing that'd be
different in each deployment are the certificates. And it
would be really exciting to have working apple push in Cyrus
just after some typical setup steps.<br> </span></div>
<div style="font-family:Arial;"><span class="font" style="font-family:Verdana"><br> If there are some impediments for the FM team to share their
implementation details on mail and caldav/carddav push
notifications, I'll try to make this feature work in my
deployments in the near future and contribute to the project a
detailed howto and the APS notifier code (but your assistance
would be great).<br> </span></div>
</div>
</div>
</blockquote><div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Unfortunately part of that is under NDA, so we can't offer much more support there. When/if Apple open up their push infrastructure more, we'll definitely release the other parts of it.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">I'm sure we've published at least part of our perl pusher layer before, though some of the session magic uses our sql infrastructure rather than storing sessions in Cyrus so that it survives failover between replicas. If we wanted to store them in Cyrus we'd need to have a replication protocol for key-value stores or some sort of replicated DB store.<br></div>
<div style="font-family:Arial;"><br></div>
<blockquote type="cite"><div><div style="font-size:10pt;font-family:Verdana,Arial;"><div style="font-family:Arial;"><span class="font" style="font-family:Verdana"><br> And a general area that would benefit everyone, but that
wasn't specifically mentioned in the blog post, is <b>Security</b>.<br> <br> I don't mean Cyrus is insecure, and I do know that the FM team
pays special attention to security of their infrastructure as
a whole. Rather I would like to suggest that a special
emphasis could be placed on Cyrus security from a development
POV, e.g. to document in detail (and keep updated) the entire
project's code base and its architecture, to follow most of
the security development best-practices, to re-implement with
security in mind some old/hacky parts of the system (they
would become apparent during the documentation phase), to
apply general hardening tactics (like chroot) or even to
re-engineer the overall architecture for security, to perform
internal security code reviews on a regular basis.</span></div>
</div>
</div>
</blockquote><div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">This is the kind of well meaning plan that leads down a massive rabbit hole. "Document in detail (and keep updated)". Such few words for so much work. We do bits and pieces of this as we can, and I've recently set up coverity to assess the project, and am working my way through its reports.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Certainly some parts of the code (like sieve) are a fricking mess, and could very well be hiding security issues because they're just so horrible. We fix them up as we have time and deal with them.<br></div>
<div style="font-family:Arial;"><br></div>
<blockquote type="cite"><div><div style="font-size:10pt;font-family:Verdana,Arial;"><div style="font-family:Arial;"><span class="font" style="font-family:Verdana"><br> FM already had a security audit in 2014 (according to your
previous blog posts), but you don't specify any details of how
deep it was and what aspects it covered. Maybe an independent
in-depth security audit with public results just for the Cyrus
code base could be sponsored in collaboration with the
community?<br> </span></div>
</div>
</div>
</blockquote><div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Again, unfortunately NDAs :(<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Feel free to sponsor a security audit. I'd be happy to participate, but I can't justify funding it. I have an idea of where likely bugs are (URLAUTH, FETCH BODY[part] until recently when we rewrote it, maybe even message structure parsing) and I rewrite them to be safer when I deal with those bits of code, as do the rest of the team.<br></div>
<div style="font-family:Arial;"><br></div>
<blockquote type="cite"><div><div style="font-size:10pt;font-family:Verdana,Arial;"><div style="font-family:Arial;"><span class="font" style="font-family:Verdana">As for me as a member of the community, I have an intention to
implement the chroot functionality for the daemon (late chroot
like in OpenVPN). I've already discussed it briefly with Ellie
and was hoping to make it ready for the 3.0 release, but had
no time for it yet. To implement it correctly, first some
important changes should be applied to the initialization
logic (the moment of dropping the privs, it should be inside
newly started processes, rather than in the master). This
change should be carefully analyzed and it's a significant
effort, I hope to be able to contribute it during the Q1/17.
Once this change is implemented (which in itself wouldn't
change almost any functionality, so it would be easy to test
and deploy), the chroot functionality would be some 15 lines
of code.<br> </span></div>
</div>
</div>
</blockquote><div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Interesting. I'm looking forward to seeing it.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">One thing that I would add here, is that we need to extract the SNMP code from master and run it in a separate process as well if we have any hope of making master something that can be allowed to run with any higher privileges than it currently does in its mainloop. Greg explained to me what he had planned for that, but never had time to do it.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;"><a href="https://github.com/cyrusimap/cyrus-imapd/issues/1765">https://github.com/cyrusimap/cyrus-imapd/issues/1765</a><br></div>
<div style="font-family:Arial;"><br></div>
<blockquote type="cite"><div><div style="font-size:10pt;font-family:Verdana,Arial;"><div style="font-family:Arial;"><span class="font" style="font-family:Verdana"><br> Merry Christmas and Happy New Year!</span></div>
<div style="font-family:Arial;"><br></div>
</div>
</div>
</blockquote><div><div style="font-size:10pt;font-family:Verdana,Arial;"><div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Thanks, the same to you! <br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Regards,<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Bron.<br></div>
<div style="font-family:Arial;"><br></div>
</div>
</div>
<blockquote type="cite"><div><div style="font-size:10pt;font-family:Verdana,Arial;"><div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;"><span class="font" style="font-family:Verdana"><br> Anatoli</span></div>
<div style="font-family:Arial;"> <br></div>
</div>
<div style="border-right-width:medium;border-bottom-width:medium;border-left-width:medium;border-right-style:none;border-bottom-style:none;border-left-style:none;border-right-color:-moz-use-text-color;border-bottom-color:-moz-use-text-color;border-left-color:-moz-use-text-color;-moz-border-top-colors:none;-moz-border-right-colors:none;-moz-border-bottom-colors:none;-moz-border-left-colors:none;border-image-source:none;border-image-slice:100% 100% 100% 100%;border-image-width:1 1 1 1;border-image-outset:0 0 0 0;border-image-repeat:stretch stretch;border-top-width:1pt;border-top-style:solid;border-top-color:rgb(181, 196, 223);padding-top:3pt;padding-right:0cm;padding-bottom:0cm;padding-left:0cm;font-size:10pt;font-family:"Tahoma","sans-serif";"><div style="font-family:Arial;"><b>From:</b> Bron Gondwana Via Cyrus-devel<br></div>
<div style="font-family:Arial;"> <b>Sent:</b> Thursday, December 22, 2016 03:15<br></div>
<div style="font-family:Arial;"> <b>To:</b> Cyrus Devel, Info Cyrus<br></div>
<div style="font-family:Arial;"> <b>Subject:</b> Release plan blog post<br></div>
</div>
</div>
<blockquote type="cite"><pre>I posted on the FastMail advent about our plans for releasing Cyrus 3.0 - it's a bit roundabout doing it this way rather than here first, but hey - we talked about it on Monday night's regular meeting.
Here's the blog post:
<a href="https://blog.fastmail.com/2016/12/22/cyrus-development-and-release-plans/">https://blog.fastmail.com/2016/12/22/cyrus-development-and-release-plans/</a>
tl;dr, Ellie recently released 3.0beta6. We're going to do a release candidate on Jan 13th and then release for real soon afterwards, so get testing!
There are no major changes expected before release. I'll be doing a couple of small JMAP changes to align with the latest spec and possibly to add getMessageListUpdates if I can manage it in time.
Other than that, I'm looking a reverse UniqueId indexing similar to the RACL support - it's already in testing and might get added behind a default-off config switch.
We'll be assessing all the defaults. I'm really tempted to turn RACL on, but it needs group support if your site uses groups, and that's not done yet, so I'd need someone willing to test it!
Bron.
<br></pre></blockquote></blockquote><div style="font-family:Arial;"><br></div>
<div id="sig567075"><div class="signature">--<br></div>
<div class="signature"> Bron Gondwana<br></div>
<div class="signature"> brong@fastmail.fm<br></div>
<div class="signature"><br></div>
</div>
<div style="font-family:Arial;"><br></div>
</body>
</html>