Unifying the Cyrus World

Wed Jan 6 20:00:45 EST 2016

Hi Patrick,

I've logged all your recommendations (great feedback, thanks!) in the
Phabricator system so they're not lost in the dusty archives of our
collective inboxes.

https://git.cyrus.foundation/maniphest/
https://git.cyrus.foundation/T226
https://git.cyrus.foundation/T225
https://git.cyrus.foundation/T224
https://git.cyrus.foundation/T223

There's also my large placeholder for all the missing outstanding doc
items that Bron wrote up for me last year at
https://git.cyrus.foundation/T219

Our documentation contribution guide is here:
http://cyrusimap.org/new/imap/developer/documentation.html

If you end up making a Phabricator account, let me know and I can
reassign the relevant tasks to you. But if you'd rather just send
through some files/patches on the mailing list, I can merge it all back
in.

I'm going to take a look at the overall structure and come up with A
Plan. While we do currently have the contributor section (for
developers) and the administrator section (for folks running cyrus), I
agree the navigation is confusing and the content within those sections
feels buried in a maze of twisty passages all alike. Takes too many
clicks to find what you're looking for.

   Nicola

On Thu, Dec 31, 2015, at 12:01 AM, Patrick Goetz via Cyrus-devel wrote:
> Hi -
> 
> Having just recently struggled with a 
> still-not-100%-clear-how-it-happened mailboxes.db corruption issue that 
> Bron ended up having to help me fix, I have some thoughts on this.  And 
> the recent bout of banging my head against the wall over this issue for 
> days has left me motivated to write up some of this documentation 
> myself, assuming I can find the information necessary.  Here is my list
> 
> 1. Suppose the /var/imap folder (the one containing the critical *.db 
> files) disappears completely:  A step by step tutorial on how to get 
> your mail system back up and running.  Such a tutorial alone would go a 
> long way towards clearing up the mysteries of how indexing is related 
> user mail folders.  This tutorial should include explanations of what 
> can't be recovered (e.g. which messages have been viewed, possibly sieve 
> scripts, etc.).
> 
> 
> 2. Documentation on safely backing up and restoring a cyrus mail system. 
> There was a discussion about this on the list a year ago:
> 
>  
> https://lists.andrew.cmu.edu/pipermail/info-cyrus/2014-December/037849.html
> 
> (Further up the thread)
>  
> https://lists.andrew.cmu.edu/pipermail/info-cyrus/2014-December/037799.html
> 
> Which I consider inconclusive.  While Nic's suggestion to use 
> replication is probably the safest approach, there still needs to be 
> something for people who run a single mail server and want to keep it 
> that way.  There's something to be said for simplicity.  Also, the 
> process of setting up a replication server (with and without murder) 
> needs to be documented better.  In the thread above, Nic suggested that 
> I look at this documentation:
> 
>    http://cyrusimap.web.cmu.edu/docs/cyrus-imapd/2.4.17/ag.php
> 
> which I found to be mostly unhelpful.  Tutorials are good even when they 
> dont match your exact use case.  Too much abstract hand-waving is just 
> confusing to most people.  I'd rather read a common use case tutorial 
> and then extrapolate to my own situation from there.  Way too many 
> critical details end up getting left out when you talk about these 
> issues absractly.
> 
> 
> 3. The creation, use, and maintenance of sieve scripts.  I dug into this 
> earlier this month and found almost nothing available online.  I did get 
> some helpful responses on the cyrus list:
> 
>  
> https://lists.andrew.cmu.edu/pipermail/info-cyrus/2015-December/038701.html
> 
> but this should be in the documentation; including a tutorial on how to 
> work with sieve scripts and pointers to 3rd party tools, with perhaps 
> even a description of how to set up and use them.
> 
> 
> 4. Organization of the documentation.  It looks like the worst case 
> scenario of having docs spread out over 2 domains with completely 
> different organizational schemes, style, etc.. is being addressed, but 
> some thought should nevertheless be given to this very difficult task. 
> My suggestion is to look at the organization of the documentation from 
> different perspectives:
> 
>   - a new or potentially new cyrus user who knows absolutely nothing
> 
>   - a somewhat experienced cyrus admin looking to quickly get an answer 
> to a specific question
> 
>   - a cyrus admin who wants to implement some new functionality, say 
> moving from a single server to a murder, or start using sieve scripts
> 
> I use both cyrus and dovecot (the first by choice, the latter because I 
> got outvoted in a larger organization).  Arch, Red Hat, and Ubuntu all 
> package dovecot as the default IMAP server; on Arch I have to install 
> cyrus from the AUR.  I'm convinced that the one and only one reason for 
> this is that the dovecot guy has had pretty good and thorough 
> documentation from the very beginning.  (That, and the cyrus project was 
> dormant for a while.)  Technically, cyrus is cleaner, faster, more 
> scalable, etc..
> 
> Soapbox:  having worked with open source software for decades, it's very 
> clear to me that open source projects become successful not on the 
> quality of the code, but rather the quality of the documentation.  Good 
> projects with poor documentation generally die if there is an 
> alternative, poorly conceived projects with good documentation succeed. 
>   Django is (in my opinion) pretty crappy software compared to say, 
> bottle.py (which most people haven't even heard of).  Why is Django so 
> popular?  Because they took the time to write good documentation.  In 
> fact, if you want to use an MVC python web framework, the trick is to 
> read the Django documentation first, and then use something like 
> bottle.py or flask for your web app.  Most developers don't bother to 
> follow up with step 2.  They read the Django documentation ... and then 
> use Django.
> 
> 
> 
> On 12/30/2015 03:02 AM, Nicola Nye via Cyrus-devel wrote:
> > Hi Patrick,
> >
> > Just before Christmas we got all the access sorted out and there's now a
> > copy of docs.cyrus.foundation on cyrusimap.org/new, and it's
> > auto-updating. It all took longer than we had hoped given FastMail's
> > exciting few weeks with the DDoS situation.
> >
> > When I'm back in the office after the New Year I'll look at what's the
> > best way to go moving the old site out of the way and whether we
> > maintain any redirects so any old bookmarks still work for common entry
> > points.
> >
> > Then we can start on the larger task of integrating the docs source back
> > into the cyrus code source.
> >
> > If there's areas that you think need immediate and obvious updating, do
> > let me know (especially if you can provide pointers to where I can get
> > the information from, even if it's just links to list threads); I'd
> > rather ensure that my time is spent easing definite pain points.
> >
> > Cheers,
> >
> >     Nicola
> >
> > On Mon, Dec 28, 2015, at 01:45 AM, Patrick Goetz via Cyrus-devel wrote:
> >> Was this issue ever resolved?  I'm still finding the documentation split
> >> between cyrus.foundation and cyrusimap.org, and it's incomplete in both
> >> places.
> >>
> >> On 09/16/2015 08:50 PM, Nicola Nye wrote:
> >>> Dear Cyrus folks,
> >>> After nominally completing the migration of the docs from the wiki at
> >>> cyrusimap.org onto the sphinx repository at docs.cyrus.foundation, I
> >>> worked out there was a whole bunch of existing services at cyrusimap.org
> >>> that were non-trivial to relocate. Which, after some discussion at the
> >>> Cyrus hangout, we felt that retaining cyrusimap.org as our primary (and
> >>> sole) presence made more sense.
> >>> Equally, there are some documentation issues that are affected by our
> >>> current straddling of two worlds. Read on for a discussion of the issues
> >>> and down to the options and recommendation.
> >>> I'd love to have your input (particularly Nic Bernstein on the docs
> >>> angle!) if you agree or disagree so we can make for a more unified Cyrus
> >>> presence.
> >>> *Issues*
> >>> 1) Branding
> >>> We are currently split across two internet presences: cyrus.foundation
> >>> and cyrusimap.org. This is confusing. We should only have one.
> >>> The cyrus.foundation domain name is not of real use as there is a
> >>> pre-existing entity (The Cyrus Foundation) of the same name.
> >>> Swapping our internet domain name presence from cyrusimap.org to
> >>> cyrus.foundation is no longer a significant driver as a result,
> >>> especially as cyrusimap.org is already indexed and linked to around the
> >>> net and is unambiguous. (Yes, Cyrus does so much more than just imap,
> >>> but it's the primary association)
> >>> Currently services are split across the two domains - git repos,
> >>> phabricator and authoritative online docs on cyrus.foundation, while
> >>> downloads, bugzilla and generated docs are on cyrusimap.org.
> >>> 2) Docs logistics
> >>> The docs repo is separate to source which is causing friction keeping
> >>> man pages updated.
> >>> We want man pages in two formats: *nix for actual man pages to ship and
> >>> install with cyrus, and html for web presentation.
> >>> Nice to have: ship a snapshot of all current docs (not just man pages)
> >>> along with source distribution.
> >>> 3) Sphinx vs wiki
> >>> We have been working towards making the content on cyrus.foundation to
> >>> be the most up to date. This is using Sphinx, which allows us to
> >>> generate, from the same source, man pages as well as web pages. (And
> >>> even pdf or single page html). So we get nicer output format options.
> >>> Clearer history in the git log than you get from wiki.
> >>> Cyrusimap docs are held in mediawiki. Not so great for man pages. But
> >>> it's much easier for third party contributor to pitch in and help out:
> >>> they don't need to learn rst, they don't need a phabricator account,
> >>> they don't need to navigate git.
> >>> *Possible solutions*
> >>> In all cases it seems clear that we should stop using the
> >>> cyrus.foundation domain and ideally services that are hosted there be
> >>> shifted to live alongside cyrusimap.org for ease of access.
> >>> There are a number of @cyrus.foundation email accounts that would need
> >>> to be dealt with for Chris Davies' testing.
> >>> Option 1: Single domain, unify git repos, use sphinx everywhere
> >>> --------------------------------------------------------------------------------
> >>> Moving the git repo to cyrusimap (or anywhere) isn't hard. (a job for Bron)
> >>> Get rid of separate cyrus-docs repo. Put cyrus-docs stuff back inside
> >>> cyrus-imapd/docs for easier man page generation and tagging of docs
> >>> versions with source, and shipping of current docs with source.
> >>> Put sphinx on cyrusimap to replace the wiki. This requires either:
> >>>
> >>>   1.
> >>>      working out how to set up the existing generated docs for use with
> >>>      sphinx,
> >>>   2.
> >>>      or make all the stuff in cyrus-imapd/doc into rst so it works with
> >>>      sphinx. We can still ship the built html.
> >>>
> >>> Option 2: Single domain, unify git repos, use wiki for docs
> >>> -----------------------------------------------------------------------------
> >>> Put sphinx on cyrusimap.org just for generating man page html. Leave the
> >>> existing wiki where it is for documentation. (Requires porting the page
> >>> updates I made from cyrus.foundation onto the wiki).
> >>> This means the only thing left in the cyrus-docs git repo would be the
> >>> man pages, at which point it makes more sense to put them back into the
> >>> cyrus-source repo.
> >>> We can still ship docs with the release if we tie in a wiki export into
> >>> the release-building process. (A job for Ellie and I)
> >>> Option 3: Single domain, separate git repos, use sphinx for docs
> >>> ----------------------------------------------------------------------------------
> >>> Move all services to cyrusimap.org, but still leaves us with all the
> >>> docs logistics and sphinx and wiki chafe points. Not recommended.
> >>> *Recommendation*
> >>> I am leaning towards suggesting option 2. Anything that makes
> >>> documentation support easier is a good! But we'd still like to retain
> >>> the usefulness of sphinx for generating two output formats from single
> >>> source man pages.
> >>> *What Now?
> >>> *
> >>> Discuss! I imagine those folks who come to the Hangout in the next few
> >>> meetings will kick it around and we'll come to a decision next week.
> >>>      Nicola
> >