Statistics on message sizes, folders, etc

[Bron Gondwana]

Well, I figure this is interesting anyway - rough
statistics on how many folders users have, how
big their messages are, how much "churn" the folders
have.

Source code attached (including a copy of 
IndexFile.pm which it requires to read the index
files)

I've tried to make it compatible with different hashing
layouts and mailbox setups, but no guarantees, it's
pretty untested!

Here's a dump for the store that my mailbox is on.  300Gb
total mail storage space, sitting at about 85% full.  Note
that the output is totally unsorted because I'm lazy.

Some notes:

* we autocreate INBOX, INBOX.Drafts, INBOX.Trash and
  'INBOX.Sent Items' - hence the huge '4' spike in
  folders per user.

* lots of empty folders, but they don't explain all
  the 'NONE' for gappyness, because it checks for a
  gap between the end and 'LastUid' as well, so
  they'll actually end up with 'ALL' if there was ever
  a message in the folder.

* don't ask me what that 12 byte "message" is.
  Obviously some imap client puked it into a folder.
  I'm not going searching for it!

* if you want different buckets, the format is pretty
  simple, and the specification is at the top of the
  file!  Enjoy.

Bron ( yes, I'd love to see some other people's stats.
       Thanks for asking.  I'm planning to feed these
       stats into my massively realistic IMAP server
       stresstester... )

STATS for /etc/imapd-slots10a1p3-master.conf
Partitions: 1
Users: 3237
Folders: 25619

Folders per user:
 10 - 30              273
 100 - 300            19
 3                    1
 30 - 100             70
 300 - 1000           1
 4                    1974
 5                    491
 6                    190
 7                    89
 8                    71
 9                    57
 > 1000               1

Messages per folder:
 0                    8856
 1                    2055
 1,000 - 3,000        343
 10 - 30              3368
 10,000 - 30,000      34
 100 - 300            1548
 2                    1476
 20,000 - 100,000     15
 3                    958
 3,000 - 10,000       142
 30 - 100             2885
 300 - 1,000          853
 4                    759
 5                    595
 6                    535
 7                    444
 8                    421
 9                    330
 > 100,000            2

Message sizes:
 1 - 3 KB             1094127
 1 - 3 MB             22340
 10 - 30 KB           1073445
 100 - 300 B          40
 100 - 300 KB         89560
 12                   1
 3 - 10 KB            1555633
 3 - 10 MB            13849
 30 - 100 KB          307347
 300 B - 1 KB         124574
 300 KB - 1 MB        46155
 > 10 MB              2335

Ratio of gaps between UIDs: (expunge tracking)
 ALL                  4143
 HIGH                 3494
 LOW                  3399
 MEDIUM               2056
 NONE                 12527

-------------- next part --------------
A non-text attachment was scrubbed...
Name: IndexFile.pm
Type: text/x-perl
Size: 28737 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090930/0fa67a3d/attachment-0002.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cyr_stats.pl
Type: text/x-perl
Size: 4834 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20090930/0fa67a3d/attachment-0003.bin