Xapian partition definition when using archiving?
Nic Bernstein
nic at onlight.com
Thu Dec 28 12:47:03 EST 2017
Bron, et al.,
Okay, one more time around on this. When I try a version of the
partition layout listed below, in my original post, and run "cyr_info
conf-lint" against it, I get all of the "/yadda/searchtier: blah" lines
thrown back at me. I had assumed that "/default/searchtier: blah" was
to specify to which tier to index the default partition, but that's
wrong, isn't it? It's really the default search tier to use, and has no
direct relationship to the "default" partition.
Am I correct in this new interpretation? It is implied by the man page
for imapd.conf (derived from lib/imapoptions):
defaultsearchtier: <empty string>
Name of the default tier that messages will be indexed to. Search indexes can be organized in tiers to allow index
storage in different directories and physical media. See the man page of squatter for details. The default search
tier also requires the definition of an according searchtierpartition-name entry.
This option MUST be specified for xapian search.
...
searchtierpartition-name: <none>
The pathname where to store the xapian search indexes of searchtier for mailboxes of partition name. This must be
configured for the defaultsearchtier and any additional search tier (see squatter for details).
For example: if defaultpartition is defined as part1 and defaultsearchtier as tier1 then the configuration must
contain an entry tier1partitionname-part1 that defines the path where to store this tier1's search index for the
part1 partition.
This option MUST be specified for xapian search.
This is all so muddled, due to the use of the word "default" as an
embedded string within so many settings, some of which refer to a
default value and some of which refer to a partition called "default".
I really think we need to go through the documentation from top to
bottom and weed out such confusing language. The same is true for the
various uses of the word "archive" and some others. This sort of thing
is clearly leading to some folks just giving up on complex Cyrus
features, the implementation of which depend upon piercing the veil of
confusion surround this language (he says in frustration).
Oh, and that imapd.conf(5) comment "(see sqautter for details)" is
useless, as the squatter(8) man page says nothing about this (damn
manpage writers!).
Thanks in advance,
-nic
On 12/20/2017 04:04 PM, Bron Gondwana wrote:
> Totally! The name "archive" is overused. It could be called
> something else easily enough.
>
> Bron.
>
>
> On Thu, 21 Dec 2017, at 01:05, Nic Bernstein wrote:
>> Bron,
>> Thanks for the swift reply. So if I understand this correctly, the
>> "archivesearchpartition-default" is named such because it's for the
>> archive location of Xapian search data, not because it's Xapian
>> search data from an Archive partition. Is that correct? In other
>> words, this is just another circumstance where seemingly obvious
>> partition names (like default-default) get us into documentation
>> trouble. Right?
>>
>> Thanks again,
>> -nic
>>
>>
>> On 12/20/2017 05:39 AM, Bron Gondwana wrote:
>>> Hi Nic,
>>>
>>> The Xapian partitions are entirely separate from archive
>>> partitions. The indexing code will find the file in the correct
>>> location if it needs to read it.
>>>
>>> Here's an example from one of our servers:
>>>
>>> defaultpartition: default
>>> defaultsearchtier: temp
>>> partition-default: /mnt/ssd21d2/sloti21d2t40/store254/spool
>>> tempsearchpartition-default: /tmpfs-search/sloti21d2t40
>>> metasearchpartition-default: /mnt/ssd21d2/sloti21d2t40/store254/search
>>> datasearchpartition-default:
>>> /mnt/i21d2search/sloti21d2t40/store254/search
>>> archivesearchpartition-default:
>>> /mnt/i21d2search/sloti21d2t40/store254/search-archive
>>> archivepartition-default:
>>> /mnt/i21d2t40/sloti21d2t40/store254/spool-archive
>>>
>>> (clearly auto-generated!)
>>>
>>> Bron.
>>>
>>>
>>> On Wed, 20 Dec 2017, at 07:32, Nic Bernstein wrote:
>>>> Bron, et al.,
>>>> We're about to set up a whole new bunch of partitions in support of
>>>> Xapian indexing, for a 2.5.10-to-3.0.4 upgrade, and then will be
>>>> introducing archive functionality, too. How do archive partitions
>>>> and Xapian partitions interact?
>>>>
>>>> For example, the server currently has the following in imapd.conf:
>>>>
>>>> defaultpartition: default
>>>> partition-default: /var/mailstores/default
>>>> metapartition-default: /var/imapmeta/default
>>>> partition-1: /var/mailstores/1
>>>> partition-2: /var/mailstores/2
>>>> partition-3: /var/mailstores/3
>>>> partition-4: /var/mailstores/4
>>>> ...
>>>> partition-29: /var/mailstores/29
>>>> partition-30: /var/mailstores/30
>>>> partition-100: /var/mailstores/100
>>>> partition-temp: /var/mailstores/temp
>>>> ...
>>>> # non-default metapartitions
>>>> metapartition-1: /var/imapmeta/1
>>>> metapartition-2: /var/imapmeta/2
>>>> metapartition-3: /var/imapmeta/3
>>>> metapartition-4: /var/imapmeta/4
>>>> ...
>>>> metapartition-29: /var/imapmeta/29
>>>> metapartition-30: /var/imapmeta/30
>>>> metapartition-100: /var/imapmeta/100
>>>> metapartition-temp: /var/imapmeta/temp
>>>>
>>>>
>>>> Going by the documentation, which I wrote with help from you good
>>>> folk at Fastmail, the Archive partition scheme might look something
>>>> like this:
>>>> https://www.cyrusimap.org/imap/reference/admin/locations/archive-partitions.html
>>>>
>>>> archivepartition-default: /var/mailarchives/default
>>>> archivepartition-1: /var/mailarchives/1
>>>> archivepartition-2: /var/mailarchives/2
>>>> archivepartition-3: /var/mailarchives/3
>>>> archiveartition-4: /var/mailarchives/4
>>>> ...
>>>> archivepartition-29: /var/mailarchives/29
>>>> archivepartition-30: /var/mailarchives/30
>>>> archivepartition-100: /var/mailarchives/100
>>>> archivepartition-temp: /var/mailarchives/temp
>>>>
>>>>
>>>> And the Xapian partition structure to mate with this would look
>>>> something like this (again, from the docs):
>>>> https://www.cyrusimap.org/imap/developer/install-xapian.html
>>>>
>>>> defaultpartition: default
>>>> partition-default: /var/mailstores/default
>>>> search_engine:xapian
>>>> search_index_headers: no
>>>> search_batchsize: 8192
>>>> defaultsearchtier: t1
>>>> 1searchtier: t1
>>>> 2searchtier: t1
>>>> 3searchtier: t1
>>>> 4searchtier: t1
>>>> ...
>>>> 29searchtier: t1
>>>> 30searchtier: t1
>>>> 100searchtier: t1
>>>> tempsearchtier: t1
>>>> ...
>>>> t1searchpartition-default: /var/search/default
>>>> t1searchpartition-1: /var/search/1
>>>> t1searchpartition-2: /var/search/2
>>>> t1searchpartition-3: /var/search/3
>>>> t1searchpartition-4: /var/search/4
>>>> ...
>>>> t1searchpartition-29: /var/search/29
>>>> t1searchpartition-30: /var/search/30
>>>> t1searchpartition-100: /var/search/100
>>>> t1searchpartition-temp: /var/search/temp
>>>>
>>>> First question, since there's no examples to work from; Is this
>>>> Xapian layout correct?
>>>>
>>>> Do I need to define & create Xapian partitions for the metadata
>>>> partitions, as is indirectly implied in Bron's original email on
>>>> this topic:
>>>> https://lists.tartarus.org/pipermail/xapian-discuss/2014-October/009112.html
>>>>
>>>> Also, how do these Xapian and Archive, interact? Do I need to add
>>>> a separate Xapian partition for each Archive partition, or will the
>>>> Archive partition be treated like a child of the non-Archive
>>>> partition? (again, implied but not directly addressed in that email).
>>>>
>>>> Any guidance gladly accepted, and whatever I learn will be
>>>> repackaged into more complete documentation on same.
>>>>
>>>> Cheers,
>>>> -nic
>>>>
>>>> --
>>>> Nic Bernsteinnic at onlight.com <mailto:nic at onlight.com>
>>>> Onlight, Inc.www.onlight.com <http://www.onlight.com>
>>>> 6525 W Bluemound Road, Suite 24 v. 414.272.4477
>>>> Milwaukee, Wisconsin 53213-4073
>>>>
>>>>
>>>> Email had 1 attachment:
>>>>
>>>> *
>>>> |nic.vcf|
>>>> 1k (text/x-vcard)
>>>>
>>>
>>> --
>>> Bron Gondwana
>>> brong at fastmail.fm <mailto:brong at fastmail.fm>
>>>
>>>
>>
>> --
>> Nic Bernsteinnic at onlight.com <mailto:nic at onlight.com>
>> Onlight Inc.www.onlight.com <http://www.onlight.com>
>> 6525 W Bluemound Rd., Ste 24 v. 414.272.4477
>> Milwaukee, Wisconsin 53213-4073 f. 414.290.0335
>>
>
> --
> Bron Gondwana
> brong at fastmail.fm
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20171228/e5ec8ae5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nic.vcf
Type: text/x-vcard
Size: 278 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20171228/e5ec8ae5/attachment.vcf>
More information about the Cyrus-devel
mailing list