Cyrus 2.5, xapian, Sphinx and index sizes

Sebastian Hagedorn Hagedorn at
Wed Sep 24 10:23:38 EDT 2014

Hi Bron,

thanks a lot for this detailed description of your setup! I have added a 
few questions inline below ...

--On 23. September 2014 21:32:04 +1000 Bron Gondwana <brong at> 

> On Tue, Sep 23, 2014, at 06:58 PM, Sebastian Hagedorn wrote:
>> Hi,
>> as I mentioned a few days ago, we're considering metapartions on SSD
>> drives  in order to optimize IMAP search performance. We have yet to run
>> a full  analysis on how much storage that would require, but a first
>> guesstimate  points towards about 20% of the net mail data for all the
>> cyrus.* files  when using SQUAT.
>> Bron mentioned support for xapian in 2.5, so I took a look at the branch
>> and noticed that there isn't only support for xapian, but actually a
>> choice  of SQUAT, xapian and Sphinx. Eventually I'd like to learn the
>> pros and cons  of the various choices, but right now I have mainly one
>> concern:
>> Will index files be larger with xapian or Sphinx? Will they also be
>> stored  on the metapartions? My concern is that we might run out of
>> space on those  metapartitions if we choose a different indexer ...
>> what's the operational  experience regarding that at Fastmail?
> So Sphinx was just too IO intensive, we had to ditch it entirely, but we
> didn't kill the code.  It's probably stale though - I wouldn't use it
> without doing a ton of testing.


> 10% is a reasonable estimate for search.  We run a 3Tb search partition
> for 20T of email storage, and it's nowhere near full.  Here's one with 20
> slots, 18 of which are in use:
> /dev/mapper/md2                                         2.7T  988G  1.8T
> 36% /mnt/i32d2search /dev/mapper/sdb1
> 917G  573G  298G  66% /mnt/i32d2t01 /dev/mapper/sdb2
> 917G  576G  295G  67% /mnt/i32d2t02 /dev/mapper/sdb3
> 917G  571G  300G  66% /mnt/i32d2t03 /dev/mapper/sdb4
> 917G  573G  298G  66% /mnt/i32d2t04 /dev/mapper/sdb5
> 917G  702G  169G  81% /mnt/i32d2t05 /dev/mapper/sdb6
> 917G  743G  128G  86% /mnt/i32d2t06 /dev/mapper/sdb7
> 917G  697G  174G  81% /mnt/i32d2t07 /dev/mapper/sdb8
> 917G  760G  111G  88% /mnt/i32d2t08 /dev/mapper/sdb9
> 917G  763G  108G  88% /mnt/i32d2t09 /dev/mapper/sdb10
> 917G  727G  144G  84% /mnt/i32d2t10 /dev/mapper/sdb11
> 917G  754G  117G  87% /mnt/i32d2t11 /dev/mapper/sdb12
> 917G  757G  114G  87% /mnt/i32d2t12 /dev/mapper/sdb13
> 917G  706G  165G  82% /mnt/i32d2t13 /dev/mapper/sdb14
> 917G  746G  125G  86% /mnt/i32d2t14 /dev/mapper/sdb15
> 917G   72M  870G   1% /mnt/i32d2t15 /dev/mapper/sdb16
> 917G   72M  870G   1% /mnt/i32d2t16 /dev/mapper/sdb17
> 917G  704G  167G  81% /mnt/i32d2t17 /dev/mapper/sdb18
> 917G  774G   97G  89% /mnt/i32d2t18 /dev/mapper/sdb19
> 917G  722G  149G  83% /mnt/i32d2t19 /dev/mapper/sdb20
> 917G  741G  130G  86% /mnt/i32d2t20 /dev/md1
> 367G  249G  118G  68% /mnt/ssd32d2
> sdb1-20 are LUKS encrypted partitions on a single hardware RAID6 volume
> with 12 x 2Tb WD RE4 drives. md2 is also LUKS encrypted, but it's a
> software RAID1e with 3 x 2Tb WD RE4 drives. md1 is 400Gb Intel DC3700
> drives in software RAID1.  It's not using LUKS because the drives support
> encryption on-disk, so we're using that.

What do you use LUKS for? My best guess would be to make it easier to toss 
out broken drives without having to worry about personal data remaining on 

> So how do we structure our search?  It's complicated.  There are 4
> "tiers" of storage.  The first tier is tmpfs, the second is ssd (it's not
> used much though), the third is on the search partition, and the 4th is
> ALSO on the search partition, but it's there for archive purposes, so we
> can compact most of the long-term search down to a single index without
> having to rewrite it every week.

So you only use fast storage for writing? Isn't there a big performance hit 
for searches on the data and archive partitions? I wonder why you don't use 
SSDs for those.

> Xapian supports reading from multiple databases.
> So the config on my server (we're moving to another machine here) is:
> search_engine: xapian
> search_index_headers: no
> search_batchsize: 8192
> defaultpartition: default
> defaultsearchtier: temp
> tempsearchpartition-default: /var/run/cyrus/search-sloti30t01
> metasearchpartition-default: /mnt/ssd30/sloti30t01/store23/search
> datasearchpartition-default: /mnt/i30search/sloti30t01/store23/search
> archivesearchpartition-default:
> /mnt/i30search/sloti30t01/store23/search-archive
> (layout is similar, but imap30 is a smaller machine, with just a single
> set)
> So by default it always indexes to temp, which gets us close-to-realtime
> indexing with a squatter that watches the sync_log directory for changes,
> and without causing too much random IO.
> Compress is run from cron:
># Any time the disk gets over 50%, compress -o single down to data
> 13 *  * * * /home/mod_perl/hm/scripts/ -a -o -d 50 temp
> data
># Copy the temporary search databases down to data during the week
> 43 1  * * 1,2,3,4,5,6 /home/mod_perl/hm/scripts/ -a
> temp,meta data
># Sundays repack the entire data directory with filtering of deleted
># messages
> 43 1  * * 0 /home/mod_perl/hm/scripts/ -a -F
> temp,meta,data data
> I'll attach the script to this email.

Why is there no job for archiving? You don't really do that manually, I 

> ($Slot->RunCommand is pretty much system with a ton of magic around it)
> With this layout, we get a few different search indexes throughout the
> week, we check every hour that we don't waste too much memory on tmpfs,
> and we get IO efficiency with the compacts being in the quieter times.
> The xapian compact code in Cyrus does clever locking to allow it to
> compact all the existing databases while creating a brand new temp
> database to index new messages.
> [brong at imap30 hm]$ du -s /var/run/cyrus/search-sloti30t01/b/user/brong/*
> 79944	/var/run/cyrus/search-sloti30t01/b/user/brong/xapian.225
> [brong at imap30 hm]$ du -s
> /mnt/i30search/sloti30t01/store23/search*/b/user/brong/* 1739980
> 	/mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian
> 21516
> 	/mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.1
> 1392840	/mnt/i30search/sloti30t01/store23/search/b/user/brong/xapian.218
> 63676	/mnt/i30search/sloti30t01/store23/search/b/user/brong/xapian.219
> 385936	/mnt/i30search/sloti30t01/store23/search/b/user/brong/xapian.220
> Wow, it looks like I'm due for an archiving!
> [brong at imap30 hm]$ sudo -u cyrus /usr/cyrus/bin/squatter -C
> /etc/cyrus/imapd-sloti30t01.conf -v -i -z archive -t
> temp,meta,data,archive -u brong compressing
> temp:225,archive:0,archive:1,data:218,data:219,data:220 to archive:2 for
> user.brong (active
> temp:225,archive:0,archive:1,data:218,data:219,data:220) adding new
> initial search location temp:226
> compacting databases
> sloti30t01/squatter[2365398]: twoskip: checkpointed
> /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.2.NE
> W/cyrus.indexed.db (107 records, 17240 => 10600 bytes) in 0.003 seconds
> Compressing messages for brong
> done
> /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.2.NEW
> renaming tempdir into place
> finished compact of user.brong (active temp:226,archive:2)
> That took a few minutes, and now:
> [brong at imap30 hm]$ du -s
> /mnt/i30search/sloti30t01/store23/search*/b/user/brong/* 3365336
> 	/mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.2
> [brong at imap30 hm]$ du -s /var/run/cyrus/search-sloti30t01/b/user/brong/*
> 168	/var/run/cyrus/search-sloti30t01/b/user/brong/xapian.226
> I just have the one search index, nicely and efficiently compacted - plus
> a tiny new one with new messages being indexed.

    .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:.
                 .:.Regionales Rechenzentrum (RRZK).:.
   .:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pkcs7-signature
Size: 5313 bytes
Desc: not available
Url : 

More information about the Info-cyrus mailing list