<!DOCTYPE html>
<html>
<head>
<title></title>
</head>
<body><div>I'm publishing this to give an idea of the relative sizes of the data usage on our systems, for deciding how to change internal datastructures for JMAP support and simplification of the internal database structures to better support object storage/distributed architecture.<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">Here's what exists on my mail server for my user:<br></div>
<div> <br></div>
<div><b>config directory (ssd):</b><br></div>
<div> <br></div>
<div>./sieve/domain/f/fastmail.fm/b/brong/websieve.bc<br></div>
<div>./sieve/domain/f/fastmail.fm/b/brong/defaultbc<br></div>
<div>./sieve/domain/f/fastmail.fm/b/brong/websieve.script<br></div>
<div> <br></div>
<div>./domain/f/fastmail.fm/user/b/brong.xapianactive<br></div>
<div>./domain/f/fastmail.fm/user/b/brong.seen<br></div>
<div>./domain/f/fastmail.fm/user/b/brong.sub<br></div>
<div>./domain/f/fastmail.fm/user/b/brong.dav-journal<br></div>
<div>./domain/f/fastmail.fm/user/b/brong.conversations<br></div>
<div>./domain/f/fastmail.fm/user/b/brong.counters<br></div>
<div>./domain/f/fastmail.fm/user/b/brong.dav<br></div>
<div> <br></div>
<div>./domain/f/fastmail.fm/quota/b/user.brong<br></div>
<div> <br></div>
<div><b>search partition (disk: raid1):</b><br></div>
<div> <br></div>
<div>./domain/f/fastmail.fm/b/user/brong/xapian.55/flintlock<br></div>
<div>./domain/f/fastmail.fm/b/user/brong/xapian.55/record.baseB<br></div>
<div>./domain/f/fastmail.fm/b/user/brong/xapian.55/iamchert<br></div>
<div>./domain/f/fastmail.fm/b/user/brong/xapian.55/position.baseB<br></div>
<div>./domain/f/fastmail.fm/b/user/brong/xapian.55/postlist.baseA<br></div>
<div>./domain/f/fastmail.fm/b/user/brong/xapian.55/position.DB<br></div>
<div>./domain/f/fastmail.fm/b/user/brong/xapian.55/cyrus.indexed.db<br></div>
<div>./domain/f/fastmail.fm/b/user/brong/xapian.55/postlist.DB<br></div>
<div>./domain/f/fastmail.fm/b/user/brong/xapian.55/termlist.baseA<br></div>
<div>./domain/f/fastmail.fm/b/user/brong/xapian.55/termlist.baseB<br></div>
<div>./domain/f/fastmail.fm/b/user/brong/xapian.55/postlist.baseB<br></div>
<div>./domain/f/fastmail.fm/b/user/brong/xapian.55/record.baseA<br></div>
<div>./domain/f/fastmail.fm/b/user/brong/xapian.55/termlist.DB<br></div>
<div>./domain/f/fastmail.fm/b/user/brong/xapian.55/record.DB<br></div>
<div>./domain/f/fastmail.fm/b/user/brong/xapian.55/position.baseA<br></div>
<div>[...]<br></div>
<div> <br></div>
<div><b>hot data partition (ssd):</b><br></div>
<div> <br></div>
<div>./domain/f/fastmail.fm/b/user/brong/cyrus.annotations<br></div>
<div>./domain/f/fastmail.fm/b/user/brong/cyrus.cache<br></div>
<div>./domain/f/fastmail.fm/b/user/brong/cyrus.header<br></div>
<div>./domain/f/fastmail.fm/b/user/brong/cyrus.index<br></div>
<div>./domain/f/fastmail.fm/b/user/brong/1.<br></div>
<div>[...]<br></div>
<div> <br></div>
<div><b>Long term data partition (disk: raid6):</b><br></div>
<div> <br></div>
<div>./domain/f/fastmail.fm/b/user/brong/cyrus.cache<br></div>
<div>./domain/f/fastmail.fm/b/user/brong/1.<br></div>
<div>[...]<br></div>
<div> <br></div>
<div> <br></div>
<div><b>There is also data related to me in the following global files on the ssd:</b><br></div>
<div> <br></div>
<div>./annotations.db (calendar metadata and specialuse ~= 30 entries)<br></div>
<div>./caldav_alarm.sqlite3 (none right now, one entry per future event with an alarm)<br></div>
<div>./mailboxes.db (one entry per mailbox plus $RACL entires for shared mailboxes ~= 200 entries)<br></div>
<div> <br></div>
<div>================<br></div>
<div> <br></div>
<div><b>metadata disk usage:</b><br></div>
<div> <br></div>
<div>lrwxrwxrwx 1 cyrus mail 11 Feb 15 2014 defaultbc -> websieve.bc<br></div>
<div>-rw------- 1 cyrus mail 16K May 19 20:24 websieve.bc<br></div>
<div>-rw------- 1 cyrus mail 13K May 19 20:24 websieve.script<br></div>
<div>-rw------- 1 cyrus mail 335M Jun 7 19:55 ./domain/f/fastmail.fm/user/b/brong.conversations<br></div>
<div>-rw------- 1 cyrus mail 88 Jun 7 19:55 ./domain/f/fastmail.fm/user/b/brong.counters<br></div>
<div>-rw------- 1 cyrus mail 1.1M Jun 7 19:39 ./domain/f/fastmail.fm/user/b/brong.dav<br></div>
<div>-rw------- 1 cyrus mail 52K Jun 7 08:50 ./domain/f/fastmail.fm/user/b/brong.seen<br></div>
<div>-rw------- 1 cyrus mail 1.2K Apr 13 22:20 ./domain/f/fastmail.fm/user/b/brong.sub<br></div>
<div>-rw------- 1 cyrus mail 41 Jun 7 01:48 ./domain/f/fastmail.fm/user/b/brong.xapianactive<br></div>
<div>-rw------- 1 cyrus mail 63 Jun 7 19:57 ./domain/f/fastmail.fm/quota/b/user.brong<br></div>
<div> <br></div>
<div>counters, seen, sub and xapianactive are all tiny. They are all flat files except seen, which is twoskip.<br></div>
<div> <br></div>
<div>dav is somewhat significantly sized, and is an sqlite database.<br></div>
<div> <br></div>
<div>conversations is massive. It's a twoskip database.<br></div>
<div> <br></div>
<div>sieve script is pretty tiny and it's a flat file/single blob.<br></div>
<div> <br></div>
<div><b>My entire search archive:</b><br></div>
<div>4.0G search-archive/domain/f/fastmail.fm/b/user/brong/<br></div>
<div>766M search/domain/f/fastmail.fm/b/user/brong/<br></div>
<div> <br></div>
<div>That's all Xapian Chert files plus a small twoskip file per index (similar format to the seen db):<br></div>
<div>-rw------- 1 cyrus mail 3.8K Jun 7 01:48 search/./domain/f/fastmail.fm/b/user/brong/xapian.55/cyrus.indexed.db<br></div>
<div> <br></div>
<div><b>SSD:</b><br></div>
<div>Total usage: 893M<br></div>
<div>cyrus.index: 71M<br></div>
<div>cyrus.cache: 64M (only has cache for spool files on the SSD)<br></div>
<div>cyrus.header: 504K<br></div>
<div>cyrus.annotations: 376M (includes previews for EVERYTHING)<br></div>
<div>emails: 382M<br></div>
<div> <br></div>
<div><b>Disk:</b><br></div>
<div>Total usage: 13G<br></div>
<div>cyrus.cache: 1.1G<br></div>
<div>emails: 12G<br></div>
<div style="font-family:Arial;"> <br></div>
<div><b>Entire shared files on global ssd:</b><br></div>
<div>1.6M annotations.db<br></div>
<div>320K caldav_alarm.sqlite3<br></div>
<div>26M mailboxes.db<br></div>
<div><i>(but barely any of that is mine)</i><br></div>
<div> <br></div>
<div>The most surprising thing standing out to me is the massive amount of space used by my search archives. This isn't normal, because for my entire store, we have:<br></div>
<div> <br></div>
<div>791G /mnt/i30t01 (entire disk usage from df -h)<br></div>
<div>4.2G /mnt/ssd30/sloti30t01/store23/conf (all the non-spool per-user and shared files)<br></div>
<div>15G /mnt/ssd30/sloti30t01/store23/spool (this week's SSD spool)<br></div>
<div>33G /mnt/i30search/sloti30t01/store23/search<br></div>
<div>77G /mnt/i30search/sloti30t01/store23/search-archive<br></div>
<div> <br></div>
<div>Which means that my search-archive is more than 1/20 of the entire usage over 752 total users on that store. For the average use I would expect that search is closer to 15% of their total spool size.<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">Anyway, here's my chart:<br></div>
<div> <br></div>
<div style="position: relative; margin: 1em auto; max-width: 605px;" contenteditable="false"><img style="display: block; margin: 0px auto; max-width: 100%; height: auto;" src="cid:1465346414.184972.53ce577c6321d4f34822b278bcbaa51fee53ad9e.13609712@content.messagingengine.com" id="inlineImage1465346411196"><br></div>
<div> <br></div>
<div style="font-family:Arial;">By far the bulk is the actual email, which is good. These are raw RFC822 blobs. The next biggest thing is cyrus.cache, which accounts for about 10% of the size of the emails. cyrus.annotations is 3%, conversations 2%, cyrus.index 0.5% and the rest is basically nothing. <br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;"><b>Data Churn</b><br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">rfc822 files are written once and never changed - they are immutable after creation<br></div>
<div style="font-family:Arial;">search and cache are both "cache" really - they're generated from the RFC822 files and are disposable.<br></div>
<div style="font-family:Arial;">cyrus.annotations is an interesting thing, because in theory it can store things that aren't cache (I use it for the last-alarm-time in caldav alarm now), but in practice at FastMail it's mostly a preview that's generated from the message file at delivery, aka - cache. It would be good to split these two things out and store the preview in the cache to keep cyrus.annotations small.<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">Conversations is entirely generated from the message files, and is basically cache. You can recreate it ctl_conversationsdb -R.<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">Which means that the next thing that's actually "real user-generated information" is cyrus.index at 0.5% or 71G. Let's look inside cyrus.index:<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">#define OFFSET_UID 0<br></div>
<div style="font-family:Arial;">#define OFFSET_INTERNALDATE 4<br></div>
<div style="font-family:Arial;">#define OFFSET_SENTDATE 8<br></div>
<div style="font-family:Arial;">#define OFFSET_SIZE 12<br></div>
<div style="font-family:Arial;">#define OFFSET_HEADER_SIZE 16<br></div>
<div style="font-family:Arial;">#define OFFSET_GMTIME 20<br></div>
<div style="font-family:Arial;">#define OFFSET_CACHE_OFFSET 24<br></div>
<div style="font-family:Arial;">#define OFFSET_LAST_UPDATED 28<br></div>
<div style="font-family:Arial;">#define OFFSET_SYSTEM_FLAGS 32<br></div>
<div style="font-family:Arial;">#define OFFSET_USER_FLAGS 36<br></div>
<div style="font-family:Arial;">#define OFFSET_CONTENT_LINES 52 /* added for nntpd */<br></div>
<div style="font-family:Arial;">#define OFFSET_CACHE_VERSION 56<br></div>
<div style="font-family:Arial;">#define OFFSET_MESSAGE_GUID 60<br></div>
<div style="font-family:Arial;">#define OFFSET_MODSEQ 80 /* CONDSTORE (64-bit modseq) */<br></div>
<div style="font-family:Arial;">#define OFFSET_THRID 88 /* conversation id, added in v13 */<br></div>
<div style="font-family:Arial;">#define OFFSET_CACHE_CRC 96 /* CRC32 of cache record */<br></div>
<div style="font-family:Arial;">#define OFFSET_RECORD_CRC 100<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">each record is 104 bytes long, of which:<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">UID is the key<br></div>
<div style="font-family:Arial;">INTERNALDATE is somewhat primary (we also store it as mtime on the spool file - it's immutable at least in theory)<br></div>
<div style="font-family:Arial;">SENTDATE is cache (taken from rfc822 header)<br></div>
<div style="font-family:Arial;">SIZE is cache (taken from rfc822 file size)<br></div>
<div style="font-family:Arial;">HEADER_SIZE is cache<br></div>
<div style="font-family:Arial;">GMTIME is cache<br></div>
<div style="font-family:Arial;">CACHE_OFFSET is mutable, but it's just a pointer<br></div>
<div style="font-family:Arial;">LAST_UPDATED is primary, but not user visible - it's internal recordkeeping<br></div>
<div style="font-family:Arial;">SYSTEM_FLAGS is primary and important user facts (plus some recordkeeping, the bitspace is partitioned)<br></div>
<div style="font-family:Arial;">USER_FLAGS is primary and important user facts<br></div>
<div style="font-family:Arial;">CONTENT_LINES is cache<br></div>
<div style="font-family:Arial;">CACHE_VERSION is internal recordkeeping<br></div>
<div style="font-family:Arial;">MESSAGE_GUID is cache/sanity checking - it's the sha1sum of the rfc822 file<br></div>
<div style="font-family:Arial;">MODSEQ is primary / recordkeeping.<br></div>
<div style="font-family:Arial;">THRID is primary / recordkeeping but immutable (allocated at delivery based on conversationsdb)<br></div>
<div style="font-family:Arial;">CACHE_CRC is recordkeeping<br></div>
<div style="font-family:Arial;">RECORD_CRC is recordkeeping<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">So of 104 bytes, actually (4 + 4 + 16 + 8 + 8) => 40 bytes per message are really important. UID, FLAGS, MODSEQ, THRID. We're down to about 0.2% of the data now, or maybe 30Mb for the entire server. Nice.<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">============<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">Let's look inside the conversations database, because that's a high churn random access file. Remember it used 335 Mb. A dump as a flat text file is 149Mb, and segmenting into three datasets:<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">149M all.txt<br></div>
<div style="font-family:Arial;">82M byconv.txt<br></div>
<div style="font-family:Arial;">20K byfolder.txt<br></div>
<div style="font-family:Arial;">67M msgids.txt<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">the folder counts are noise. The "msgids" file is everything starting with '<' and looks like this:<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;"><!&!aaaaaaaaaaauaaaaaaaaagayjud6695iv/kkipqn4imbanltncjhprtfudq2lhcbs8ebacqa//8aabaaaabwur+oaqihrjg9v4un3qk2aqaaaaa=@spamreducer.eu> 0 d0ca2202f453ad68 1460600647<br></div>
<div style="font-family:Arial;"><!&!aaaaaaaaaaauaaaaaaaaaloc/y79rhhcoseanz1scukbamdcnoymubjpiolnqodg6d8aaaaavv8aabaaaabstmhn7+e+tyjbrspmov1raqaaaaa=@gunasekera.com.au> 0 d69877672a8c1a15 1460600464<br></div>
<div style="font-family:Arial;"><!&!aaaaaaaaaaauaaaaaaaaaloc/y79rhhcoseanz1scukbamdcnoymubjpiolnqodg6d8aaaaavv8aabaaaadxvsnegyytqbeoardfdwptaqaaaaa=@gunasekera.com.au> 0 d69877672a8c1a15 1460600464<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">(that's msgid, version, thrid, last-seen)<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">This file gets cleaned out of records more than 6 months old.<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">The byconv file is all the keys starting with 'B'.<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">B00000741084435a6 0 (295339132740792 1 1 0 (0 0 0 0 0 0) ((78 295339132740792 1 1 0)) (("Subversion Commits" NIL fm-cvs list.krot.org 1276694410 1)) "*brong-bah-needstocreateanobject" 3731)<br></div>
<div style="font-family:Arial;">B0000087576508d8a 0 (295339132715691 1 1 0 (0 0 0 0 0 0) ((100 295339132715691 1 1 0)) ((NIL NIL sales softlayer.com 1384495901 1)) SoftLayerDutchHoldingsB.V.BillingAdvanceNotification 4343)<br></div>
<div style="font-family:Arial;">B000011630568958c 0 (295339132740628 1 1 0 (0 0 1 0 0 0) ((90 295339132740628 1 1 0)) (("Matt Rosser" NIL notification+[...] facebookmail.com 1383195915 1)) Afternoonteam, 8568)<br></div>
<div style="font-family:Arial;">B00004e266b006ec2 0 (295339134881905 2 2 2 (0 0 2 0 0 0) ((59 295339134881905 2 2 2)) (("Dilger, Andreas" NIL andreas.dilger intel.com 1447465224 2)) fixe2fsck-fDdirectorytruncation 54816)<br></div>
<div style="font-family:Arial;">B00004f3a24fd8403 0 (295339132807848 4 4 0 (0 0 4 0 0 0) ((57 295339132807848 4 4 0)) (("Timo Sirainen" NIL tss iki.fi 1329786670 2) (Robin NIL dovecot r.paypc.com 1329446783 1) (NIL NIL manuel.bertrand gmail.com 1328617214 1)) "Possiblebrokenindexer(lucene/solr)?" 16883)<br></div>
<div style="font-family:Arial;">[...]<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">For every thrid this is a mapping to modseq, counts, perl-flag counts, per folder counts, a complete list of senders in the conversation, a compressed version of the subject, and the sum of the sizes of all the messages.<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">Obviously any time a counted flag changes on any message, the conversations database needs to update for it.<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;"><b>THE PLAN[tm]</b><b></b><br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">For JMAP support, I'm going to discard the existing conversations DB and create a sqlite database per user which contains everything of value. I'm also going to subsume everything else (or at least the facts about it) into this database, which will be called 'brong.userdb' and be stored on the meta partition.<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">msgids: there's no need to keep the whole thing. Normalise it, hash it, store a 64 bit hash as the key in a table. Potentially this will be used to replace duplicate.db as well, since it's basically a duplicate of the data. I didn't mention duplicate.db above, but it's yet another twoskip file (87Mb for everyone) containing a map from messageid to delivery information. It's disposable, so stored on tmpfs. The metadata won't change - thrid, timestamp.<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">folders:<br></div>
<div style="font-family:Arial;"> id INTEGER PRIMARY KEY, -- similar to the $FOLDER_NAMES in conversations.db<br></div>
<div style="font-family:Arial;"> uniqueid TEXT (UNIQUE INDEX)<br></div>
<div style="font-family:Arial;"> mboxname TEXT (INDEX)<br></div>
<div style="font-family:Arial;"> deletedmodseq INTEGER -- 0 for not deleted<br></div>
<div style="font-family:Arial;"> UNIQUE INDEX (mboxname, deletedmodseq)<br></div>
<div style="font-family:Arial;"> -- F-key fields and statuscache fields:<br></div>
<div style="font-family:Arial;"> modseq INTEGER (INDEX)<br></div>
<div style="font-family:Arial;"> countsmodseq INTEGER (INDEX)<br></div>
<div style="font-family:Arial;"> exists INTEGER<br></div>
<div style="font-family:Arial;"> unseen INTEGER<br></div>
<div style="font-family:Arial;"> convmodseq INTEGER<br></div>
<div style="font-family:Arial;"> convexists INTEGER<br></div>
<div style="font-family:Arial;"> convunseen INTEGER<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">messages:<br></div>
<div style="font-family:Arial;"> folderid INTEGER<br></div>
<div style="font-family:Arial;"> uid INTEGER<br></div>
<div style="font-family:Arial;"> PRIMARY KEY (folderid, uid)<br></div>
<div style="font-family:Arial;"> guid TEXT (INDEX)<br></div>
<div style="font-family:Arial;"> thrid INTEGER (INDEX)<br></div>
<div style="font-family:Arial;"> system_flags INTEGER (bitmask)<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">flagnames: (from all the cyrus.headers)<br></div>
<div style="font-family:Arial;"> id INTEGER PRIMARY KEY<br></div>
<div style="font-family:Arial;"> flag TEXT<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">messageflags: (many to many)<br></div>
<div style="font-family:Arial;"> folderid INTEGER<br></div>
<div style="font-family:Arial;"> uid INTEGER<br></div>
<div style="font-family:Arial;"> flagid INTEGER<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;">...<br></div>
<div style="font-family:Arial;"> </div>
<div style="font-family:Arial;">Anyway, this will be landing soon as YET ANOTHER DATABASE that can be enabled and will be created from mailbox_append_index_record and mailbox_update_index_record. To start, it will only be used for JMAP, but maybe the performance will be good enough to move everything from cyrus.header and cyrus.index files into this one database and remove them entirely. Even if not, this might become the primary source of truth, with the cyrus.header and cyrus.index files existing purely as cached copies of the data in here.<br></div>
<div style="font-family:Arial;"> </div>
<div style="font-family:Arial;">"messages" is still quite denormalised at the moment to be similar to struct index_record. All the bits which are identical between messages (aka, gmtime) could be stored keyed by GUID instead, and if we were willing to enforce shared flags between instances, we could even store everything keyed by GUID and have a table that just mapped folderid/uid to guid.<br></div>
<div style="font-family:Arial;"> </div>
<div style="font-family:Arial;">Bron.<br></div>
<div style="font-family:Arial;"> <br></div>
<div>--<br></div>
<div> Bron Gondwana<br></div>
<div> brong@fastmail.fm<br></div>
</body>
</html>