Unifying the Cyrus World

Patrick Goetz pgoetz at mail.utexas.edu
Wed Dec 30 08:01:22 EST 2015


Hi -

Having just recently struggled with a 
still-not-100%-clear-how-it-happened mailboxes.db corruption issue that 
Bron ended up having to help me fix, I have some thoughts on this.  And 
the recent bout of banging my head against the wall over this issue for 
days has left me motivated to write up some of this documentation 
myself, assuming I can find the information necessary.  Here is my list

1. Suppose the /var/imap folder (the one containing the critical *.db 
files) disappears completely:  A step by step tutorial on how to get 
your mail system back up and running.  Such a tutorial alone would go a 
long way towards clearing up the mysteries of how indexing is related 
user mail folders.  This tutorial should include explanations of what 
can't be recovered (e.g. which messages have been viewed, possibly sieve 
scripts, etc.).


2. Documentation on safely backing up and restoring a cyrus mail system. 
There was a discussion about this on the list a year ago:

 
https://lists.andrew.cmu.edu/pipermail/info-cyrus/2014-December/037849.html

(Further up the thread)
 
https://lists.andrew.cmu.edu/pipermail/info-cyrus/2014-December/037799.html

Which I consider inconclusive.  While Nic's suggestion to use 
replication is probably the safest approach, there still needs to be 
something for people who run a single mail server and want to keep it 
that way.  There's something to be said for simplicity.  Also, the 
process of setting up a replication server (with and without murder) 
needs to be documented better.  In the thread above, Nic suggested that 
I look at this documentation:

   http://cyrusimap.web.cmu.edu/docs/cyrus-imapd/2.4.17/ag.php

which I found to be mostly unhelpful.  Tutorials are good even when they 
dont match your exact use case.  Too much abstract hand-waving is just 
confusing to most people.  I'd rather read a common use case tutorial 
and then extrapolate to my own situation from there.  Way too many 
critical details end up getting left out when you talk about these 
issues absractly.


3. The creation, use, and maintenance of sieve scripts.  I dug into this 
earlier this month and found almost nothing available online.  I did get 
some helpful responses on the cyrus list:

 
https://lists.andrew.cmu.edu/pipermail/info-cyrus/2015-December/038701.html

but this should be in the documentation; including a tutorial on how to 
work with sieve scripts and pointers to 3rd party tools, with perhaps 
even a description of how to set up and use them.


4. Organization of the documentation.  It looks like the worst case 
scenario of having docs spread out over 2 domains with completely 
different organizational schemes, style, etc.. is being addressed, but 
some thought should nevertheless be given to this very difficult task. 
My suggestion is to look at the organization of the documentation from 
different perspectives:

  - a new or potentially new cyrus user who knows absolutely nothing

  - a somewhat experienced cyrus admin looking to quickly get an answer 
to a specific question

  - a cyrus admin who wants to implement some new functionality, say 
moving from a single server to a murder, or start using sieve scripts

I use both cyrus and dovecot (the first by choice, the latter because I 
got outvoted in a larger organization).  Arch, Red Hat, and Ubuntu all 
package dovecot as the default IMAP server; on Arch I have to install 
cyrus from the AUR.  I'm convinced that the one and only one reason for 
this is that the dovecot guy has had pretty good and thorough 
documentation from the very beginning.  (That, and the cyrus project was 
dormant for a while.)  Technically, cyrus is cleaner, faster, more 
scalable, etc..

Soapbox:  having worked with open source software for decades, it's very 
clear to me that open source projects become successful not on the 
quality of the code, but rather the quality of the documentation.  Good 
projects with poor documentation generally die if there is an 
alternative, poorly conceived projects with good documentation succeed. 
  Django is (in my opinion) pretty crappy software compared to say, 
bottle.py (which most people haven't even heard of).  Why is Django so 
popular?  Because they took the time to write good documentation.  In 
fact, if you want to use an MVC python web framework, the trick is to 
read the Django documentation first, and then use something like 
bottle.py or flask for your web app.  Most developers don't bother to 
follow up with step 2.  They read the Django documentation ... and then 
use Django.



On 12/30/2015 03:02 AM, Nicola Nye via Cyrus-devel wrote:
> Hi Patrick,
>
> Just before Christmas we got all the access sorted out and there's now a
> copy of docs.cyrus.foundation on cyrusimap.org/new, and it's
> auto-updating. It all took longer than we had hoped given FastMail's
> exciting few weeks with the DDoS situation.
>
> When I'm back in the office after the New Year I'll look at what's the
> best way to go moving the old site out of the way and whether we
> maintain any redirects so any old bookmarks still work for common entry
> points.
>
> Then we can start on the larger task of integrating the docs source back
> into the cyrus code source.
>
> If there's areas that you think need immediate and obvious updating, do
> let me know (especially if you can provide pointers to where I can get
> the information from, even if it's just links to list threads); I'd
> rather ensure that my time is spent easing definite pain points.
>
> Cheers,
>
>     Nicola
>
> On Mon, Dec 28, 2015, at 01:45 AM, Patrick Goetz via Cyrus-devel wrote:
>> Was this issue ever resolved?  I'm still finding the documentation split
>> between cyrus.foundation and cyrusimap.org, and it's incomplete in both
>> places.
>>
>> On 09/16/2015 08:50 PM, Nicola Nye wrote:
>>> Dear Cyrus folks,
>>> After nominally completing the migration of the docs from the wiki at
>>> cyrusimap.org onto the sphinx repository at docs.cyrus.foundation, I
>>> worked out there was a whole bunch of existing services at cyrusimap.org
>>> that were non-trivial to relocate. Which, after some discussion at the
>>> Cyrus hangout, we felt that retaining cyrusimap.org as our primary (and
>>> sole) presence made more sense.
>>> Equally, there are some documentation issues that are affected by our
>>> current straddling of two worlds. Read on for a discussion of the issues
>>> and down to the options and recommendation.
>>> I'd love to have your input (particularly Nic Bernstein on the docs
>>> angle!) if you agree or disagree so we can make for a more unified Cyrus
>>> presence.
>>> *Issues*
>>> 1) Branding
>>> We are currently split across two internet presences: cyrus.foundation
>>> and cyrusimap.org. This is confusing. We should only have one.
>>> The cyrus.foundation domain name is not of real use as there is a
>>> pre-existing entity (The Cyrus Foundation) of the same name.
>>> Swapping our internet domain name presence from cyrusimap.org to
>>> cyrus.foundation is no longer a significant driver as a result,
>>> especially as cyrusimap.org is already indexed and linked to around the
>>> net and is unambiguous. (Yes, Cyrus does so much more than just imap,
>>> but it's the primary association)
>>> Currently services are split across the two domains - git repos,
>>> phabricator and authoritative online docs on cyrus.foundation, while
>>> downloads, bugzilla and generated docs are on cyrusimap.org.
>>> 2) Docs logistics
>>> The docs repo is separate to source which is causing friction keeping
>>> man pages updated.
>>> We want man pages in two formats: *nix for actual man pages to ship and
>>> install with cyrus, and html for web presentation.
>>> Nice to have: ship a snapshot of all current docs (not just man pages)
>>> along with source distribution.
>>> 3) Sphinx vs wiki
>>> We have been working towards making the content on cyrus.foundation to
>>> be the most up to date. This is using Sphinx, which allows us to
>>> generate, from the same source, man pages as well as web pages. (And
>>> even pdf or single page html). So we get nicer output format options.
>>> Clearer history in the git log than you get from wiki.
>>> Cyrusimap docs are held in mediawiki. Not so great for man pages. But
>>> it's much easier for third party contributor to pitch in and help out:
>>> they don't need to learn rst, they don't need a phabricator account,
>>> they don't need to navigate git.
>>> *Possible solutions*
>>> In all cases it seems clear that we should stop using the
>>> cyrus.foundation domain and ideally services that are hosted there be
>>> shifted to live alongside cyrusimap.org for ease of access.
>>> There are a number of @cyrus.foundation email accounts that would need
>>> to be dealt with for Chris Davies' testing.
>>> Option 1: Single domain, unify git repos, use sphinx everywhere
>>> --------------------------------------------------------------------------------
>>> Moving the git repo to cyrusimap (or anywhere) isn't hard. (a job for Bron)
>>> Get rid of separate cyrus-docs repo. Put cyrus-docs stuff back inside
>>> cyrus-imapd/docs for easier man page generation and tagging of docs
>>> versions with source, and shipping of current docs with source.
>>> Put sphinx on cyrusimap to replace the wiki. This requires either:
>>>
>>>   1.
>>>      working out how to set up the existing generated docs for use with
>>>      sphinx,
>>>   2.
>>>      or make all the stuff in cyrus-imapd/doc into rst so it works with
>>>      sphinx. We can still ship the built html.
>>>
>>> Option 2: Single domain, unify git repos, use wiki for docs
>>> -----------------------------------------------------------------------------
>>> Put sphinx on cyrusimap.org just for generating man page html. Leave the
>>> existing wiki where it is for documentation. (Requires porting the page
>>> updates I made from cyrus.foundation onto the wiki).
>>> This means the only thing left in the cyrus-docs git repo would be the
>>> man pages, at which point it makes more sense to put them back into the
>>> cyrus-source repo.
>>> We can still ship docs with the release if we tie in a wiki export into
>>> the release-building process. (A job for Ellie and I)
>>> Option 3: Single domain, separate git repos, use sphinx for docs
>>> ----------------------------------------------------------------------------------
>>> Move all services to cyrusimap.org, but still leaves us with all the
>>> docs logistics and sphinx and wiki chafe points. Not recommended.
>>> *Recommendation*
>>> I am leaning towards suggesting option 2. Anything that makes
>>> documentation support easier is a good! But we'd still like to retain
>>> the usefulness of sphinx for generating two output formats from single
>>> source man pages.
>>> *What Now?
>>> *
>>> Discuss! I imagine those folks who come to the Hangout in the next few
>>> meetings will kick it around and we'll come to a decision next week.
>>>      Nicola
>


More information about the Cyrus-devel mailing list