Xapian partition definition when using archiving?
Nic Bernstein
nic at onlight.com
Fri Dec 29 10:22:37 EST 2017
Ellie,
Thanks for weighing in on this, and for doing so on a weekend! :-)
I've just pushed some documentation edits in PR #2224 as a first attempt
at clarifying this stuff.
Cheers,
-nic
On 12/28/2017 09:22 PM, ellie timoney wrote:
> > It's really the default search tier to use, and has no direct
> relationship to the "default" partition.
>
> Yep, or at least, this is my understanding too.
>
> > In other words, this is just another circumstance where seemingly
> obvious partition names (like default-default) get us into
> documentation trouble. Right?
>
> We trip over this all the time... and it's hard to document because
> the trivial case and the complicated cases are so divergent. Even at
> FM, where our setup is pretty advanced, it's still relatively simple,
> in that we only have a single partition-/name/ entry ("default") per
> Cyrus instance. So all of our /foo/partition-/name/ settings are
> /foo/partition-default!
>
> I guess what I'm getting at is there's a few dimensions of complexity
> here. I think the "multiple-named-partitions" dimension of complexity
> might be reasonably well documented, but then each additional
> dimension (e.g. archive partitions, search tiers, etc) is only
> documented for the single-named-default-partition case.
>
> It'd be great if the documentation could move away from the "single
> partition is named 'default'" scheme -- I think that would ease a LOT
> of confusion for people trying to understand the different dimensions
> all at once. But I don't really have any ideas for what might be a
> better name for it. I haven't seen real-world multi-partition Cyrus
> instances, so I don't know the use cases for having multiple
> partitions, so it's hard to suggest a good name for someone's first
> and maybe only partition.
>
> Cheers,
>
> ellie
>
> On Fri, Dec 29, 2017, at 4:47 AM, Nic Bernstein wrote:
>> Bron, et al.,
>> Okay, one more time around on this. When I try a version of the
>> partition layout listed below, in my original post, and run "cyr_info
>> conf-lint" against it, I get all of the "/yadda/searchtier: blah"
>> lines thrown back at me. I had assumed that "/default/searchtier:
>> blah" was to specify to which tier to index the default partition,
>> but that's wrong, isn't it? It's really the default search tier to
>> use, and has no direct relationship to the "default" partition.
>>
>> Am I correct in this new interpretation? It is implied by the man
>> page for imapd.conf (derived from lib/imapoptions):
>>
>> defaultsearchtier: <empty string>
>> Name of the default tier that messages will be indexed to. Search indexes can be organized in tiers to allow index
>> storage in different directories and physical media. See the man page of squatter for details. The default search
>> tier also requires the definition of an according searchtierpartition-name entry.
>>
>> This option MUST be specified for xapian search.
>> ...
>> searchtierpartition-name: <none>
>> The pathname where to store the xapian search indexes of searchtier for mailboxes of partition name. This must be
>> configured for the defaultsearchtier and any additional search tier (see squatter for details).
>>
>> For example: if defaultpartition is defined as part1 and defaultsearchtier as tier1 then the configuration must
>> contain an entry tier1partitionname-part1 that defines the path where to store this tier1's search index for the
>> part1 partition.
>>
>> This option MUST be specified for xapian search.
>>
>> This is all so muddled, due to the use of the word "default" as an
>> embedded string within so many settings, some of which refer to a
>> default value and some of which refer to a partition called
>> "default". I really think we need to go through the documentation
>> from top to bottom and weed out such confusing language. The same is
>> true for the various uses of the word "archive" and some others.
>> This sort of thing is clearly leading to some folks just giving up on
>> complex Cyrus features, the implementation of which depend upon
>> piercing the veil of confusion surround this language (he says in
>> frustration).
>>
>> Oh, and that imapd.conf(5) comment "(see sqautter for details)" is
>> useless, as the squatter(8) man page says nothing about this (damn
>> manpage writers!).
>>
>> Thanks in advance,
>> -nic
>>
>>
>> On 12/20/2017 04:04 PM, Bron Gondwana wrote:
>>> Totally! The name "archive" is overused. It could be called
>>> something else easily enough.
>>>
>>> Bron.
>>>
>>>
>>> On Thu, 21 Dec 2017, at 01:05, Nic Bernstein wrote:
>>>> Bron,
>>>> Thanks for the swift reply. So if I understand this correctly, the
>>>> "archivesearchpartition-default" is named such because it's for the
>>>> archive location of Xapian search data, not because it's Xapian
>>>> search data from an Archive partition. Is that correct? In other
>>>> words, this is just another circumstance where seemingly obvious
>>>> partition names (like default-default) get us into documentation
>>>> trouble. Right?
>>>>
>>>> Thanks again,
>>>> -nic
>>>>
>>>>
>>>> On 12/20/2017 05:39 AM, Bron Gondwana wrote:
>>>>> Hi Nic,
>>>>>
>>>>> The Xapian partitions are entirely separate from archive
>>>>> partitions. The indexing code will find the file in the correct
>>>>> location if it needs to read it.
>>>>>
>>>>> Here's an example from one of our servers:
>>>>>
>>>>> defaultpartition: default
>>>>> defaultsearchtier: temp
>>>>> partition-default: /mnt/ssd21d2/sloti21d2t40/store254/spool
>>>>> tempsearchpartition-default: /tmpfs-search/sloti21d2t40
>>>>> metasearchpartition-default: /mnt/ssd21d2/sloti21d2t40/store254/search
>>>>> datasearchpartition-default:
>>>>> /mnt/i21d2search/sloti21d2t40/store254/search
>>>>> archivesearchpartition-default:
>>>>> /mnt/i21d2search/sloti21d2t40/store254/search-archive
>>>>> archivepartition-default:
>>>>> /mnt/i21d2t40/sloti21d2t40/store254/spool-archive
>>>>>
>>>>> (clearly auto-generated!)
>>>>>
>>>>> Bron.
>>>>>
>>>>>
>>>>> On Wed, 20 Dec 2017, at 07:32, Nic Bernstein wrote:
>>>>>> Bron, et al.,
>>>>>> We're about to set up a whole new bunch of partitions in support
>>>>>> of Xapian indexing, for a 2.5.10-to-3.0.4 upgrade, and then will
>>>>>> be introducing archive functionality, too. How do archive
>>>>>> partitions and Xapian partitions interact?
>>>>>>
>>>>>> For example, the server currently has the following in imapd.conf:
>>>>>>
>>>>>> defaultpartition: default
>>>>>> partition-default: /var/mailstores/default
>>>>>> metapartition-default: /var/imapmeta/default
>>>>>> partition-1: /var/mailstores/1
>>>>>> partition-2: /var/mailstores/2
>>>>>> partition-3: /var/mailstores/3
>>>>>> partition-4: /var/mailstores/4
>>>>>> ...
>>>>>> partition-29: /var/mailstores/29
>>>>>> partition-30: /var/mailstores/30
>>>>>> partition-100: /var/mailstores/100
>>>>>> partition-temp: /var/mailstores/temp
>>>>>> ...
>>>>>> # non-default metapartitions
>>>>>> metapartition-1: /var/imapmeta/1
>>>>>> metapartition-2: /var/imapmeta/2
>>>>>> metapartition-3: /var/imapmeta/3
>>>>>> metapartition-4: /var/imapmeta/4
>>>>>> ...
>>>>>> metapartition-29: /var/imapmeta/29
>>>>>> metapartition-30: /var/imapmeta/30
>>>>>> metapartition-100: /var/imapmeta/100
>>>>>> metapartition-temp: /var/imapmeta/temp
>>>>>>
>>>>>>
>>>>>>
>>>>>> Going by the documentation, which I wrote with help from you good
>>>>>> folk at Fastmail, the Archive partition scheme might look
>>>>>> something like this:
>>>>>> https://www.cyrusimap.org/imap/reference/admin/locations/archive-partitions.html
>>>>>>
>>>>>> archivepartition-default: /var/mailarchives/default
>>>>>> archivepartition-1: /var/mailarchives/1
>>>>>> archivepartition-2: /var/mailarchives/2
>>>>>> archivepartition-3: /var/mailarchives/3
>>>>>> archiveartition-4: /var/mailarchives/4
>>>>>> ...
>>>>>> archivepartition-29: /var/mailarchives/29
>>>>>> archivepartition-30: /var/mailarchives/30
>>>>>> archivepartition-100: /var/mailarchives/100
>>>>>> archivepartition-temp: /var/mailarchives/temp
>>>>>>
>>>>>>
>>>>>>
>>>>>> And the Xapian partition structure to mate with this would look
>>>>>> something like this (again, from the docs):
>>>>>> https://www.cyrusimap.org/imap/developer/install-xapian.html
>>>>>>
>>>>>> defaultpartition: default
>>>>>> partition-default: /var/mailstores/default
>>>>>> search_engine:xapian
>>>>>> search_index_headers: no
>>>>>> search_batchsize: 8192
>>>>>> defaultsearchtier: t1
>>>>>> 1searchtier: t1
>>>>>> 2searchtier: t1
>>>>>> 3searchtier: t1
>>>>>> 4searchtier: t1
>>>>>> ...
>>>>>> 29searchtier: t1
>>>>>> 30searchtier: t1
>>>>>> 100searchtier: t1
>>>>>> tempsearchtier: t1
>>>>>> ...
>>>>>> t1searchpartition-default: /var/search/default
>>>>>> t1searchpartition-1: /var/search/1
>>>>>> t1searchpartition-2: /var/search/2
>>>>>> t1searchpartition-3: /var/search/3
>>>>>> t1searchpartition-4: /var/search/4
>>>>>> ...
>>>>>> t1searchpartition-29: /var/search/29
>>>>>> t1searchpartition-30: /var/search/30
>>>>>> t1searchpartition-100: /var/search/100
>>>>>> t1searchpartition-temp: /var/search/temp
>>>>>>
>>>>>>
>>>>>> First question, since there's no examples to work from; Is this
>>>>>> Xapian layout correct?
>>>>>>
>>>>>> Do I need to define & create Xapian partitions for the metadata
>>>>>> partitions, as is indirectly implied in Bron's original email on
>>>>>> this topic:
>>>>>> https://lists.tartarus.org/pipermail/xapian-discuss/2014-October/009112.html
>>>>>>
>>>>>> Also, how do these Xapian and Archive, interact? Do I need to
>>>>>> add a separate Xapian partition for each Archive partition, or
>>>>>> will the Archive partition be treated like a child of the
>>>>>> non-Archive partition? (again, implied but not directly
>>>>>> addressed in that email).
>>>>>>
>>>>>> Any guidance gladly accepted, and whatever I learn will be
>>>>>> repackaged into more complete documentation on same.
>>>>>>
>>>>>> Cheers,
>>>>>> -nic
>>>>>>
>>>>>> --
>>>>>> Nic Bernsteinnic at onlight.com <mailto:nic at onlight.com>
>>>>>> Onlight, Inc.www.onlight.com <http://www.onlight.com>
>>>>>> 6525 W Bluemound Road, Suite 24 v. 414.272.4477
>>>>>> Milwaukee, Wisconsin 53213-4073
>>>>>>
>>>>>>
>>>>>>
>>>>>> Email had 1 attachment:
>>>>>>
>>>>>> *
>>>>>> |nic.vcf|
>>>>>> 1k (text/x-vcard)
>>>>>>
>>>>>
>>>>> --
>>>>> Bron Gondwana
>>>>> brong at fastmail.fm <mailto:brong at fastmail.fm>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Nic Bernsteinnic at onlight.com <mailto:nic at onlight.com>
>>>> Onlight Inc.www.onlight.com <http://www.onlight.com>
>>>> 6525 W Bluemound Rd., Ste 24 v. 414.272.4477
>>>> Milwaukee, Wisconsin 53213-4073 f. 414.290.0335
>>>>
>>>>
>>>
>>> --
>>> Bron Gondwana
>>> brong at fastmail.fm <mailto:brong at fastmail.fm>
>>>
>>>
>>
>> Email had 1 attachment:
>>
>> *
>> |nic.vcf|
>> 1k (text/x-vcard)
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20171229/555d4a70/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nic.vcf
Type: text/x-vcard
Size: 278 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20171229/555d4a70/attachment-0001.vcf>
More information about the Cyrus-devel
mailing list