Future Ideas wiki page

[Bron Gondwana]

On Wed, Jan 13, 2010 at 04:02:43PM -0800, Wil Cooley wrote:
> Bron Gondwana wrote:
> 
> > Anyway, I'd love feedback on any or all of it, and if there
> > are other things that you feel are really important for the
> > future viability of Cyrus I'd love to hear about them as well.
> > I haven't yet had a chance to look at the QRESYNC stuff that
> > Ken's already done for 2.4, and we might wind up releasing
> > a 2.4 without a lot of these changes just because there's a
> > lot of work in there!
> 
> Thanks for asking! I'll just mention a few of mine here and if you like them or
> see them as useful, I will add them to the wiki. I think I kinda got carried
> away when I started trying to remember things I've wished for; sorry this is so
> long.
> 
> 1. It would be nice from a provisioning standpoint to be able to enumerate
> available partitions and their capacities within the IMAP protocol itself. This
> way, provisioning scripts could decide intelligently which backend and partition
> to use without resorting to SNMP or SSH or other. Is this something that could
> be delivered in an annotation or "ID" response perhaps?

Some of the work Ken did with this (auto-choose-partition) would probably be
extendable.  I like the idea.
 
> 2. Being able to run "(cyr)quota -f" on individual partitions easily would be
> nice. 'squatter' too. There may be other maintenance utilities that could
> benefit from being easily isolated to a particular partition, like the
> consistency checker in item #4 on your list.

Definitely.

> How about setting a server shutdown or motd message that applies only to a
> particular partition? We've had cases where we had to restore a whole partition
> after some SAN buggery. It would have been nice if we could have locked out or
> at least provided a message directly to the affected users.

Hmm... interesting.  Would you do this via the user's INBOX partition?

> 3. Another idea related to partitioning: Would there be any value in
> partitioning the user config data parallel to the mailbox spools? By that I mean
> that a user foo's mailbox in /var/spool/imap/01/**/foo has sub/seen data in
> /var/lib/imap/user/01/**/foo.{seen,sub} and likewise for the sieve data. With
> this, you could move/restore/recover partitions on different hosts. I can think
> of a few cases where that would make sense, but maybe they're unlikely cases.

Now, you see - this is why we just run separate instances on the same machine rather
than using partitions at Fastmail - you get all this for free, and besides you can
have the replicas go different places.

> Somewhat orthogonally, what about being able to configure separate
> metapartitions per metadata type for a partition? For example, as you mention on
> #5 on your list, one metapartition to put the cyrus.index files on a fast SSD
> and another to put the large squatter files on cheaper, slower storage?

Yeah, this and database file locations even more so.  It would be great to have
the statuscache and deliver dbs on tmpfs or similar to avoid the disk IO.

> 4. If you're going to be computing SHA1s for messages anyway, would there be any
> value in hashing the message files across sub-directories, so that directories
> wouldn't get to be so large? Are there cases where imapd would do a directory
> listing or is all of that done through the index and cache files? Otherwise, I
> guess only local admins and backup software would benefit. And anyway, would it
> be faster to open and list 1,000 files in 23 directories than to open one
> directory and list 23,000 files? Would that be overshadowed by the cost of
> opening all 23,000 files (which I presume it would need to if it were resorting
> to listing them).

Ick.  You could hash by UID just as easily really, least-significant bit(s).
I saw the response to this - anything that doesn't let you open by explicit
filename quickly will suck.  See, our backup software reads the cyrus.index
to choose which files to back up as well - so we never[tm] enumerate the
directory.  Except for reconstruct of course.

> 5. What about toggling whatever CYRUS_VERBOSE enables by sending a signal or
> something like that? On my test systems I have gotten some good info from that
> which I would not have otherwise gotten but it is not feasible to enable it on
> my production systems.

Hey, that's a cool idea.  Pretty easy too.

> 6. sieveshell: Command to report supported extensions (a little more intuitive
> than 'nc localhost sieve').

Sounds great.

> 7. Doc additions: (I could do some of these, if I'd make the time...) There are
> some cyradm commands that, in a Murder setup, need to be run from a front-end or
> back-end (sometimes a particular back-end). For example, it took me a while to
> figure out that 'xfer' needs to be run from the source back-end and not from a
> front-end. (Maybe that is a feature request itself?) For some of the maintenance
> utilities that check and modify the spools or config data, it's not always clear
> if they can be run with the server live or if it should be shut down, such as
> 'ctl_cyrusdb -r' or 'quota -f'.

Yeah, it would be fantastic if you can do some of these - particularly the ones
that I don't use myself or understand particularly well.  Good docs are very
important for a polished product :)

> 8. Last one, I promise: lmtpproxyd to use local mupdate instead of having to hit
> the master. Maybe this has changed? I don't see it mentioned in the change log.

Don't have a clue sorry.

It would be fantastic if you could put these ideas on the wiki.  I'd be tempted
to say "put items in Bugzilla", but honestly it's a bit of a jungle in there at
the moment!  Maybe both, and cross link to the bug IDs from the wiki.

Bron.