Initial use of Xapian

[Nic Bernstein]

On 02/14/2018 11:04 AM, Per olof Ljungmark wrote:
> Quoting Nic Bernstein <nic at onlight.com>:
>
>> On 02/14/2018 10:35 AM, Lists Nethead wrote:
>>> Quoting Nic Bernstein <nic at onlight.com>:
>>>
>>>> The advice to move it to START is actually based on a recently 
>>>> discovered bug, referred to in that issue report (#2234).  It 
>>>> /should/ be in DAEMON, but for that bug, which has been 
>>>> fixed.  The fix will be in the next release.
>>>>
>>>> In general, the mailing is a grand place to start, and IRC is also 
>>>> your friend (#cyrus on freenode).
>>>>
>>>> Cheers,
>>>>     -nic
>>>>
>>>> On 02/14/2018 04:58 AM, Lists Nethead wrote:
>>>>> Quoting Sebastian Hagedorn <Hagedorn at uni-koeln.de>:
>>>>>
>>>>>> --On 14. Februar 2018 um 11:13:15 +0100 Lists Nethead 
>>>>>> <lists at nethead.se> wrote:
>>>>>>
>>>>>>> I am testing Xapian in a 3.0.5 setup on FreeBSD using most default
>>>>>>> setting. If I start imapd with "squatter cmd="squatter -R", more 
>>>>>>> and more
>>>>>>> "squatter" processes are created and eventually grabs all 
>>>>>>> resources.
>>>>>>
>>>>>> Where did you put that statement? You can't put it in the DAEMON 
>>>>>> section with 3.0.5. Put it in START instead. See this issue for 
>>>>>> more information:
>>>>>>
>>>>>> <https://github.com/cyrusimap/cyrus-imapd/issues/2234>
>>>>>
>>>>> A-ha. So it is better to look at Github instead of the mailing 
>>>>> list apparently.
>>>>>
>>>>> https://www.cyrusimap.org/imap/reference/manpages/systemcommands/squatter.html 
>>>>> still advises DAEMON, that is why it ended up there.
>>>
>>> Thanks to the advise above and I believe I got it working now.
>>>
>>> One more thing though, if replication is involved, it appears one 
>>> needs one log for squatter and another for replication, am I 
>>> correct? I got replication to work only after I added another log, 
>>> like,
>>>
>>> sync_log_channels: squatter replog
>>> and
>>> syncclient       cmd="/usr/local/cyrus/sbin/sync_client -r -n replog"
>>>
>>> Not sure if this is mentioned in the docs somewhere.
>>>
>>> Cheers,
>>>
>>> //per
>>
>> In general, for each consumer of a sync_log, a new log should be 
>> defined in `sync_log_channels:`.  I use the word "consumer" there 
>> intentionally, as the processes which use these logs actually consume 
>> them, and leave nothing behind (assuming it all works as expected).  
>> So if more than one process tries to eat up the same log, you'll have 
>> problems.
>>
>> The overloading of the sync_log framework for purposes beyond 
>> replication is new in 3.0, so we're still getting the documentation 
>> up to snuff in that regard.  However, the documentation already makes 
>> this concept clear in that when using more than one replica you need 
>> to specify more than one sync_log via the `sync_log_channels:` 
>> directive (see 
>> https://www.cyrusimap.org/imap/reference/admin/sop/replication.html#channels 
>> for details).
>>
>> We obviously do need to produce more generalized documentation for 
>> this whole scheme, and I'll be using this discussion as a guide in 
>> that regard.  sync_log, as the name implies, started life as a way 
>> for the "master" server to provide a list of "units" -- either users 
>> or mailboxes -- it has touched, so that a replica knows what to 
>> request in updates.  This is such a useful concept, however, that it 
>> is spreading to other subsystems which need to know what might have 
>> changed in a potentially large data set (the typical mail store) and 
>> so we need to explain this not just in the Replication documentation, 
>> but in a more general way.
>>
>> Note also that there is a `sync_log_unsuppressable_channels:` 
>> directive, which defaults to "squatter".  This is defined as:
>>
>>        If specified, the named channels are exempt from the effect of
>>        setting sync_log_chain:off, i.e. they are always logged to by
>>        the sync_server process. This is only really useful to allow
>>        rolling search indexing on a replica.
>>
>> If you are going to use a name other than "squatter" for your rolling 
>> indexing sync_log channel, then you need to update this as well.
>>
>
> Nic, thank you for the clarification, it makes real sense.
>
> I do understand that managing a project like this is quite a task and 
> v.3 is still young. Apart from the log issue and xapian I also 
> stumbled over other undocumented changes related to FreeBSD only I 
> think(?) in that binaries that before existed in bin/ are now 
> separated into sbin/ and libexec/, meaning that your scripts will of 
> course fail after the upgrade. I should produce a short writeup for 
> the FreeBSD camp.
>
> Cheers,
>
> //per

Per,
I've created an issue #2248 for this on GitHub, and have just created PR 
#2249 which addresses it.  Thanks for your feedback.

As for the FreeBSD path changing issues, that's up to the package 
creator.  Please let them know.

Cheers,
     -nic

1: https://github.com/cyrusimap/cyrus-imapd/issues/2248
2: https://github.com/cyrusimap/cyrus-imapd/pull/2249
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nic.vcf
Type: text/x-vcard
Size: 278 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20180214/6c80d01e/attachment.vcf>