Initial use of Xapian

[Nic Bernstein]

On 02/14/2018 10:35 AM, Lists Nethead wrote:
> Quoting Nic Bernstein <nic at onlight.com>:
>
>> The advice to move it to START is actually based on a recently 
>> discovered bug, referred to in that issue report (#2234).  It 
>> /should/ be in DAEMON, but for that bug, which has been fixed.  
>> The fix will be in the next release.
>>
>> In general, the mailing is a grand place to start, and IRC is also 
>> your friend (#cyrus on freenode).
>>
>> Cheers,
>>     -nic
>>
>> On 02/14/2018 04:58 AM, Lists Nethead wrote:
>>> Quoting Sebastian Hagedorn <Hagedorn at uni-koeln.de>:
>>>
>>>> --On 14. Februar 2018 um 11:13:15 +0100 Lists Nethead 
>>>> <lists at nethead.se> wrote:
>>>>
>>>>> I am testing Xapian in a 3.0.5 setup on FreeBSD using most default
>>>>> setting. If I start imapd with "squatter cmd="squatter -R", more 
>>>>> and more
>>>>> "squatter" processes are created and eventually grabs all resources.
>>>>
>>>> Where did you put that statement? You can't put it in the DAEMON 
>>>> section with 3.0.5. Put it in START instead. See this issue for 
>>>> more information:
>>>>
>>>> <https://github.com/cyrusimap/cyrus-imapd/issues/2234>
>>>
>>> A-ha. So it is better to look at Github instead of the mailing list 
>>> apparently.
>>>
>>> https://www.cyrusimap.org/imap/reference/manpages/systemcommands/squatter.html 
>>> still advises DAEMON, that is why it ended up there.
>
> Thanks to the advise above and I believe I got it working now.
>
> One more thing though, if replication is involved, it appears one 
> needs one log for squatter and another for replication, am I correct? 
> I got replication to work only after I added another log, like,
>
> sync_log_channels: squatter replog
> and
> syncclient       cmd="/usr/local/cyrus/sbin/sync_client -r -n replog"
>
> Not sure if this is mentioned in the docs somewhere.
>
> Cheers,
>
> //per

In general, for each consumer of a sync_log, a new log should be defined 
in `sync_log_channels:`.  I use the word "consumer" there intentionally, 
as the processes which use these logs actually consume them, and leave 
nothing behind (assuming it all works as expected).  So if more than one 
process tries to eat up the same log, you'll have problems.

The overloading of the sync_log framework for purposes beyond 
replication is new in 3.0, so we're still getting the documentation up 
to snuff in that regard.  However, the documentation already makes this 
concept clear in that when using more than one replica you need to 
specify more than one sync_log via the `sync_log_channels:` directive 
(see 
https://www.cyrusimap.org/imap/reference/admin/sop/replication.html#channels 
for details).

We obviously do need to produce more generalized documentation for this 
whole scheme, and I'll be using this discussion as a guide in that 
regard.  sync_log, as the name implies, started life as a way for the 
"master" server to provide a list of "units" -- either users or 
mailboxes -- it has touched, so that a replica knows what to request in 
updates.  This is such a useful concept, however, that it is spreading 
to other subsystems which need to know what might have changed in a 
potentially large data set (the typical mail store) and so we need to 
explain this not just in the Replication documentation, but in a more 
general way.

Note also that there is a `sync_log_unsuppressable_channels:` directive, 
which defaults to "squatter".  This is defined as:

        If specified, the named channels are exempt from the effect of
        setting sync_log_chain:off, i.e. they are always logged to by
        the sync_server process. This is only really useful to allow
        rolling search indexing on a replica.

If you are going to use a name other than "squatter" for your rolling 
indexing sync_log channel, then you need to update this as well.

Cheers,
     -nic
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20180214/bc79c52d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nic.vcf
Type: text/x-vcard
Size: 278 bytes
Desc: not available
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20180214/bc79c52d/attachment-0001.vcf>