Re-designing cyrus.cache format

Greg Banks gnb at fastmail.fm
Tue Feb 12 19:58:25 EST 2013



On Tue, Feb 12, 2013, at 08:44 PM, Bron Gondwana wrote:

> As you can see, there are some normalised things from some headers.  The
> same information normalised in a DIFFERENT way in the ENVELOPE and then a
> BODYSTRUCTURE and a BODY response.
> 
> We have already changed the normalisation rules here a couple of times.
> 
> There are two benefits to doing this.
> 
> 1: reduced CPU usage re-parsing the fields for fast responses.

Maybe this was true once, but current fastmail code has to reparse
everything from cache, and it's a crawling horror of bizarre special
cases.  See for example message2_parse_cbodystructure() at
https://github.com/gnb/cyrus-imapd/blob/fastmail/imap/message.c#L4150

> So - I would propose this:
> 
> 1) keep the BODYSTRUCTURE, it's the result of parsing the entire message,
> and can't be calculated cheaply again
> 2) keep the SECTION data (possibly along with the bodystructure) - it's
> the offsets for the various parts of the message, same issue

These two are inconsistent with each other, full of bizarre edge cases,
and don't map well onto MIME structures or what the modern and flexible
fastmail message2 code needs for its internal representation.  They need
to be 

a) unified into one
b) redesigned to be more easily parseable into data structures and not
to be compatible with IMAP wire format
c) redesigned to be more MIME-friendly

> 3) add a list of "SUPPRESSED HEADERS".  This would list any header which
> is present in the file, but NOT in the cache.

Easily discovered from the field_desc_t structures in the mesage2 code.

> 4) cache every other header, including all the To:, From:, Subject:, etc
> - in as close to raw form as possible.

This could be trivially achieved as a binary dump of the part_t tree in
the message2 code.

-- 
Greg.


More information about the Cyrus-devel mailing list