Put only cyrus.index and cyrus.cache in metadata partitions ?

[Eric Luyten]

On Thu, August 26, 2010 9:36 pm, Bron Gondwana wrote:
> On Thu, Aug 26, 2010 at 05:19:25PM +0200, Eric Luyten wrote:
>
>> Hello,
>>
>>
>>
>> I understand the cyrus.header file is the only irreplacable one, i.e.
>> cyrus.index and cyrus.cache can be regenerated using 'reconstruct' while
>> cyrus.header, a very small (150 bytes or thereabout) and stable file,
>> contains some vital mailbox identification data.
>
> cyrus.index contains both the "INTERNALDATE" (re-creatable from the file
> timestamp if you're lucky) and all your flags.  You'll be sad if you lose it.
> It also contains UIDVALIDITY.
>
>
> cyrus.header contains the only copy of the mailbox UNIQUEID - which is what
> the \Seen flag uses, so you'll be sad if you lose that.
>
> cyrus.cache is totally re-generatable using reconstruct.  You won't care too
> much if you lose that.
>
> Mind you - cyrus.index is only 96 bytes per message or so (88 if you're
> not using my patches)
>
>> I observe a cyrus.header file only gets modified when the mailbox ACL is
>> changed or when annotations are added. Am I correct in assuming this ?
>
> Does it get changed for annotations?  It gets changed if you add a new flag,
> or if you update the ACL (though it's only a backup copy of the ACL.  The
> active version is in mailboxes.db)
>
>> So most accesses to cyrus.header files are read operations.
>>
>
> Yeah, true.
>
>
>> Does it therefore make (some) sense to leave the cyrus.header files with
>> the message files and not with the other metadata ? I am not considering
>> cyrus.expunge and cyrus.squat here for the sake of simplicity.
>
> No, not really.  It makes more sense to leave cyrus.cache with the message
> files because it's large, and put your meta files on small fast disk.


Bron (and others),



I just ran some tests on an empty test configuration and obtained surprising
results with different data/metadata split situations.


Server : SunFire X4170 with 72 GB of RAM, 2 x 300 GB internal disks in ZFS
mirror configuration, 32 GB "LogZilla" SSD, 2 x 1 Gbps iSCSI SAN connection
with multisession failover and load balancing capabilities.


Test : start four parallel streams invoking /usr/cyrus/bin/deliver on a 50 KB
message and send it 5000 (five thousand) times to a different mailbox.

Metadata merged in data (on iSCSI SAN) : 114 seconds elapsed time, 175 msgs/s
Metadata on SAS disks                  : 200 seconds elapsed time, 100 msgs/s
Metadata on SSD                        :  97 seconds elapsed time, 206 msgs/s

The ZFS 'atime' attribute was 'off' on all devices.

The SSD is also used for the Cyrus DB's (mailboxes.db, deliver.db) and 'proc'
directory.

Dtrace shows that the Cyrus metadata files attract many synchronous writes.


Eric Luyten, Computing Centre VUB/ULB.