Cyrus 2.5, xapian, Sphinx and index sizes

Bron Gondwana brong at
Tue Sep 23 07:32:04 EDT 2014

On Tue, Sep 23, 2014, at 06:58 PM, Sebastian Hagedorn wrote:
> Hi,
> as I mentioned a few days ago, we're considering metapartions on SSD drives 
> in order to optimize IMAP search performance. We have yet to run a full 
> analysis on how much storage that would require, but a first guesstimate 
> points towards about 20% of the net mail data for all the cyrus.* files 
> when using SQUAT.
> Bron mentioned support for xapian in 2.5, so I took a look at the branch 
> and noticed that there isn't only support for xapian, but actually a choice 
> of SQUAT, xapian and Sphinx. Eventually I'd like to learn the pros and cons 
> of the various choices, but right now I have mainly one concern:
> Will index files be larger with xapian or Sphinx? Will they also be stored 
> on the metapartions? My concern is that we might run out of space on those 
> metapartitions if we choose a different indexer ... what's the operational 
> experience regarding that at Fastmail?

So Sphinx was just too IO intensive, we had to ditch it entirely, but we didn't kill the code.  It's probably stale though - I wouldn't use it without doing a ton of testing.

10% is a reasonable estimate for search.  We run a 3Tb search partition for 20T of email storage, and it's nowhere near full.  Here's one with 20 slots, 18 of which are in use:

/dev/mapper/md2                                         2.7T  988G  1.8T  36% /mnt/i32d2search
/dev/mapper/sdb1                                        917G  573G  298G  66% /mnt/i32d2t01
/dev/mapper/sdb2                                        917G  576G  295G  67% /mnt/i32d2t02
/dev/mapper/sdb3                                        917G  571G  300G  66% /mnt/i32d2t03
/dev/mapper/sdb4                                        917G  573G  298G  66% /mnt/i32d2t04
/dev/mapper/sdb5                                        917G  702G  169G  81% /mnt/i32d2t05
/dev/mapper/sdb6                                        917G  743G  128G  86% /mnt/i32d2t06
/dev/mapper/sdb7                                        917G  697G  174G  81% /mnt/i32d2t07
/dev/mapper/sdb8                                        917G  760G  111G  88% /mnt/i32d2t08
/dev/mapper/sdb9                                        917G  763G  108G  88% /mnt/i32d2t09
/dev/mapper/sdb10                                       917G  727G  144G  84% /mnt/i32d2t10
/dev/mapper/sdb11                                       917G  754G  117G  87% /mnt/i32d2t11
/dev/mapper/sdb12                                       917G  757G  114G  87% /mnt/i32d2t12
/dev/mapper/sdb13                                       917G  706G  165G  82% /mnt/i32d2t13
/dev/mapper/sdb14                                       917G  746G  125G  86% /mnt/i32d2t14
/dev/mapper/sdb15                                       917G   72M  870G   1% /mnt/i32d2t15
/dev/mapper/sdb16                                       917G   72M  870G   1% /mnt/i32d2t16
/dev/mapper/sdb17                                       917G  704G  167G  81% /mnt/i32d2t17
/dev/mapper/sdb18                                       917G  774G   97G  89% /mnt/i32d2t18
/dev/mapper/sdb19                                       917G  722G  149G  83% /mnt/i32d2t19
/dev/mapper/sdb20                                       917G  741G  130G  86% /mnt/i32d2t20
/dev/md1                                                367G  249G  118G  68% /mnt/ssd32d2

sdb1-20 are LUKS encrypted partitions on a single hardware RAID6 volume with 12 x 2Tb WD RE4 drives.
md2 is also LUKS encrypted, but it's a software RAID1e with 3 x 2Tb WD RE4 drives.
md1 is 400Gb Intel DC3700 drives in software RAID1.  It's not using LUKS because the drives support encryption on-disk, so we're using that.


So how do we structure our search?  It's complicated.  There are 4 "tiers" of storage.  The first tier is tmpfs, the second is ssd (it's not used much though), the third is on the search partition, and the 4th is ALSO on the search partition, but it's there for archive purposes, so we can compact most of the long-term search down to a single index without having to rewrite it every week.

Xapian supports reading from multiple databases.

So the config on my server (we're moving to another machine here) is:

search_engine: xapian
search_index_headers: no
search_batchsize: 8192
defaultpartition: default
defaultsearchtier: temp
tempsearchpartition-default: /var/run/cyrus/search-sloti30t01
metasearchpartition-default: /mnt/ssd30/sloti30t01/store23/search
datasearchpartition-default: /mnt/i30search/sloti30t01/store23/search
archivesearchpartition-default: /mnt/i30search/sloti30t01/store23/search-archive

(layout is similar, but imap30 is a smaller machine, with just a single set)

So by default it always indexes to temp, which gets us close-to-realtime indexing with a squatter that watches the sync_log directory for changes, and without causing too much random IO.

Compress is run from cron:

# Any time the disk gets over 50%, compress -o single down to data
13 *  * * * /home/mod_perl/hm/scripts/ -a -o -d 50 temp data
# Copy the temporary search databases down to data during the week
43 1  * * 1,2,3,4,5,6 /home/mod_perl/hm/scripts/ -a temp,meta data
# Sundays repack the entire data directory with filtering of deleted messages
43 1  * * 0 /home/mod_perl/hm/scripts/ -a -F temp,meta,data data

I'll attach the script to this email.

($Slot->RunCommand is pretty much system with a ton of magic around it)

With this layout, we get a few different search indexes throughout the week, we check every hour that we don't waste too much memory on tmpfs, and we get IO efficiency with the compacts being in the quieter times.

The xapian compact code in Cyrus does clever locking to allow it to compact all the existing databases while creating a brand new temp database to index new messages.

[brong at imap30 hm]$ du -s /var/run/cyrus/search-sloti30t01/b/user/brong/*
79944	/var/run/cyrus/search-sloti30t01/b/user/brong/xapian.225

[brong at imap30 hm]$ du -s /mnt/i30search/sloti30t01/store23/search*/b/user/brong/*
1739980	/mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian
21516	/mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.1
1392840	/mnt/i30search/sloti30t01/store23/search/b/user/brong/xapian.218
63676	/mnt/i30search/sloti30t01/store23/search/b/user/brong/xapian.219
385936	/mnt/i30search/sloti30t01/store23/search/b/user/brong/xapian.220

Wow, it looks like I'm due for an archiving!

[brong at imap30 hm]$ sudo -u cyrus /usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -i -z archive -t temp,meta,data,archive -u brong
compressing temp:225,archive:0,archive:1,data:218,data:219,data:220 to archive:2 for user.brong (active temp:225,archive:0,archive:1,data:218,data:219,data:220)
adding new initial search location temp:226
compacting databases
sloti30t01/squatter[2365398]: twoskip: checkpointed /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.2.NEW/cyrus.indexed.db (107 records, 17240 => 10600 bytes) in 0.003 seconds
Compressing messages for brong
done /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.2.NEW
renaming tempdir into place
finished compact of user.brong (active temp:226,archive:2)

That took a few minutes, and now:

[brong at imap30 hm]$ du -s /mnt/i30search/sloti30t01/store23/search*/b/user/brong/*
3365336	/mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.2
[brong at imap30 hm]$ du -s /var/run/cyrus/search-sloti30t01/b/user/brong/*
168	/var/run/cyrus/search-sloti30t01/b/user/brong/xapian.226

I just have the one search index, nicely and efficiently compacted - plus a tiny new one with new messages being indexed.


  Bron Gondwana
  brong at
-------------- next part --------------
A non-text attachment was scrubbed...
Type: application/x-perl
Size: 1771 bytes
Desc: not available
Url : 

More information about the Info-cyrus mailing list