Cyrus 2.5, xapian, Sphinx and index sizes
Bron Gondwana
brong at fastmail.fm
Tue Sep 23 07:32:04 EDT 2014
On Tue, Sep 23, 2014, at 06:58 PM, Sebastian Hagedorn wrote:
> Hi,
>
> as I mentioned a few days ago, we're considering metapartions on SSD drives
> in order to optimize IMAP search performance. We have yet to run a full
> analysis on how much storage that would require, but a first guesstimate
> points towards about 20% of the net mail data for all the cyrus.* files
> when using SQUAT.
>
> Bron mentioned support for xapian in 2.5, so I took a look at the branch
> and noticed that there isn't only support for xapian, but actually a choice
> of SQUAT, xapian and Sphinx. Eventually I'd like to learn the pros and cons
> of the various choices, but right now I have mainly one concern:
>
> Will index files be larger with xapian or Sphinx? Will they also be stored
> on the metapartions? My concern is that we might run out of space on those
> metapartitions if we choose a different indexer ... what's the operational
> experience regarding that at Fastmail?
So Sphinx was just too IO intensive, we had to ditch it entirely, but we didn't kill the code. It's probably stale though - I wouldn't use it without doing a ton of testing.
10% is a reasonable estimate for search. We run a 3Tb search partition for 20T of email storage, and it's nowhere near full. Here's one with 20 slots, 18 of which are in use:
/dev/mapper/md2 2.7T 988G 1.8T 36% /mnt/i32d2search
/dev/mapper/sdb1 917G 573G 298G 66% /mnt/i32d2t01
/dev/mapper/sdb2 917G 576G 295G 67% /mnt/i32d2t02
/dev/mapper/sdb3 917G 571G 300G 66% /mnt/i32d2t03
/dev/mapper/sdb4 917G 573G 298G 66% /mnt/i32d2t04
/dev/mapper/sdb5 917G 702G 169G 81% /mnt/i32d2t05
/dev/mapper/sdb6 917G 743G 128G 86% /mnt/i32d2t06
/dev/mapper/sdb7 917G 697G 174G 81% /mnt/i32d2t07
/dev/mapper/sdb8 917G 760G 111G 88% /mnt/i32d2t08
/dev/mapper/sdb9 917G 763G 108G 88% /mnt/i32d2t09
/dev/mapper/sdb10 917G 727G 144G 84% /mnt/i32d2t10
/dev/mapper/sdb11 917G 754G 117G 87% /mnt/i32d2t11
/dev/mapper/sdb12 917G 757G 114G 87% /mnt/i32d2t12
/dev/mapper/sdb13 917G 706G 165G 82% /mnt/i32d2t13
/dev/mapper/sdb14 917G 746G 125G 86% /mnt/i32d2t14
/dev/mapper/sdb15 917G 72M 870G 1% /mnt/i32d2t15
/dev/mapper/sdb16 917G 72M 870G 1% /mnt/i32d2t16
/dev/mapper/sdb17 917G 704G 167G 81% /mnt/i32d2t17
/dev/mapper/sdb18 917G 774G 97G 89% /mnt/i32d2t18
/dev/mapper/sdb19 917G 722G 149G 83% /mnt/i32d2t19
/dev/mapper/sdb20 917G 741G 130G 86% /mnt/i32d2t20
/dev/md1 367G 249G 118G 68% /mnt/ssd32d2
sdb1-20 are LUKS encrypted partitions on a single hardware RAID6 volume with 12 x 2Tb WD RE4 drives.
md2 is also LUKS encrypted, but it's a software RAID1e with 3 x 2Tb WD RE4 drives.
md1 is 400Gb Intel DC3700 drives in software RAID1. It's not using LUKS because the drives support encryption on-disk, so we're using that.
....
So how do we structure our search? It's complicated. There are 4 "tiers" of storage. The first tier is tmpfs, the second is ssd (it's not used much though), the third is on the search partition, and the 4th is ALSO on the search partition, but it's there for archive purposes, so we can compact most of the long-term search down to a single index without having to rewrite it every week.
Xapian supports reading from multiple databases.
So the config on my server (we're moving to another machine here) is:
search_engine: xapian
search_index_headers: no
search_batchsize: 8192
defaultpartition: default
defaultsearchtier: temp
tempsearchpartition-default: /var/run/cyrus/search-sloti30t01
metasearchpartition-default: /mnt/ssd30/sloti30t01/store23/search
datasearchpartition-default: /mnt/i30search/sloti30t01/store23/search
archivesearchpartition-default: /mnt/i30search/sloti30t01/store23/search-archive
(layout is similar, but imap30 is a smaller machine, with just a single set)
So by default it always indexes to temp, which gets us close-to-realtime indexing with a squatter that watches the sync_log directory for changes, and without causing too much random IO.
Compress is run from cron:
# Any time the disk gets over 50%, compress -o single down to data
13 * * * * /home/mod_perl/hm/scripts/xapian_compact.pl -a -o -d 50 temp data
# Copy the temporary search databases down to data during the week
43 1 * * 1,2,3,4,5,6 /home/mod_perl/hm/scripts/xapian_compact.pl -a temp,meta data
# Sundays repack the entire data directory with filtering of deleted messages
43 1 * * 0 /home/mod_perl/hm/scripts/xapian_compact.pl -a -F temp,meta,data data
I'll attach the xapian_compact.pl script to this email.
($Slot->RunCommand is pretty much system with a ton of magic around it)
With this layout, we get a few different search indexes throughout the week, we check every hour that we don't waste too much memory on tmpfs, and we get IO efficiency with the compacts being in the quieter times.
The xapian compact code in Cyrus does clever locking to allow it to compact all the existing databases while creating a brand new temp database to index new messages.
[brong at imap30 hm]$ du -s /var/run/cyrus/search-sloti30t01/b/user/brong/*
79944 /var/run/cyrus/search-sloti30t01/b/user/brong/xapian.225
[brong at imap30 hm]$ du -s /mnt/i30search/sloti30t01/store23/search*/b/user/brong/*
1739980 /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian
21516 /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.1
1392840 /mnt/i30search/sloti30t01/store23/search/b/user/brong/xapian.218
63676 /mnt/i30search/sloti30t01/store23/search/b/user/brong/xapian.219
385936 /mnt/i30search/sloti30t01/store23/search/b/user/brong/xapian.220
Wow, it looks like I'm due for an archiving!
[brong at imap30 hm]$ sudo -u cyrus /usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -i -z archive -t temp,meta,data,archive -u brong
compressing temp:225,archive:0,archive:1,data:218,data:219,data:220 to archive:2 for user.brong (active temp:225,archive:0,archive:1,data:218,data:219,data:220)
adding new initial search location temp:226
compacting databases
sloti30t01/squatter[2365398]: twoskip: checkpointed /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.2.NEW/cyrus.indexed.db (107 records, 17240 => 10600 bytes) in 0.003 seconds
Compressing messages for brong
done /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.2.NEW
renaming tempdir into place
finished compact of user.brong (active temp:226,archive:2)
That took a few minutes, and now:
[brong at imap30 hm]$ du -s /mnt/i30search/sloti30t01/store23/search*/b/user/brong/*
3365336 /mnt/i30search/sloti30t01/store23/search-archive/b/user/brong/xapian.2
[brong at imap30 hm]$ du -s /var/run/cyrus/search-sloti30t01/b/user/brong/*
168 /var/run/cyrus/search-sloti30t01/b/user/brong/xapian.226
I just have the one search index, nicely and efficiently compacted - plus a tiny new one with new messages being indexed.
Bron.
--
Bron Gondwana
brong at fastmail.fm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: xapian_compact.pl
Type: application/x-perl
Size: 1771 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20140923/ef535d69/attachment-0001.bin
More information about the Info-cyrus
mailing list