<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    Ellie,<br>
    Thanks for weighing in on this, and for doing so on a weekend! :-)<br>
    <br>
    I've just pushed some documentation edits in PR #2224 as a first
    attempt at clarifying this stuff.<br>
    <br>
    Cheers,<br>
        -nic<br>
    <br>
    <div class="moz-cite-prefix">On 12/28/2017 09:22 PM, ellie timoney
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:1514517737.3774119.1218442136.6D24A64C@webmail.messagingengine.com">
      <title></title>
      <style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}</style>
      <div><span class="highlight" style="background-color:rgb(255, 255,
          255)"><span class="colour" style="color:rgb(31, 31, 31)"><span
              class="font" style="font-family:'Source Sans Pro',
              sans-serif"><span class="size" style="font-size:14px"><span>> </span>It's
                really the default search tier to use, and has no direct
                relationship to the "default" partition.</span></span></span></span><br>
      </div>
      <div><br>
      </div>
      <div>Yep, or at least, this is my understanding too.<br>
      </div>
      <div><br>
      </div>
      <div>> In other words, this is just another circumstance where
        seemingly obvious partition names (like default-default) get us
        into documentation trouble.  Right?<br>
      </div>
      <div><br>
      </div>
      <div>We trip over this all the time... and it's hard to document
        because the trivial case and the complicated cases are so
        divergent.  Even at FM, where our setup is pretty advanced, it's
        still relatively simple, in that we only have a single
        partition-<i>name</i> entry ("default") per Cyrus instance.  So
        all of our <i>foo</i>partition-<i>name</i> settings are <i>foo</i>partition-default!<br>
      </div>
      <div><br>
      </div>
      <div>I guess what I'm getting at is there's a few dimensions of
        complexity here.  I think the "multiple-named-partitions"
        dimension of complexity might be reasonably well documented, but
        then each additional dimension (e.g. archive partitions, search
        tiers, etc) is only documented for the
        single-named-default-partition case.<br>
      </div>
      <div><br>
      </div>
      <div>It'd be great if the documentation could move away from the
        "single partition is named 'default'" scheme -- I think that
        would ease a LOT of confusion for people trying to understand
        the different dimensions all at once.  But I don't really have
        any ideas for what might be a better name for it.  I haven't
        seen real-world multi-partition Cyrus instances, so I don't know
        the use cases for having multiple partitions, so it's hard to
        suggest a good name for someone's first and maybe only
        partition.<br>
      </div>
      <div><br>
      </div>
      <div>Cheers,<br>
      </div>
      <div><br>
      </div>
      <div>ellie<br>
      </div>
      <div><br>
      </div>
      <div>On Fri, Dec 29, 2017, at 4:47 AM, Nic Bernstein wrote:<br>
      </div>
      <blockquote type="cite">
        <div>Bron, et al.,<br>
        </div>
        <div>Okay, one more time around on this.  When I try a version
          of the partition layout listed below, in my original post, and
          run "cyr_info conf-lint" against it, I get all of the "<i>yadda</i>searchtier:
          blah" lines thrown back at me.  I had assumed that "<i>default</i>searchtier:
          blah" was to specify to which tier to index the default
          partition, but that's wrong, isn't it?  It's really the
          default search tier to use, and has no direct relationship to
          the "default" partition.<br>
        </div>
        <div><br>
        </div>
        <div>Am I correct in this new interpretation?  It is implied by
          the man page for imapd.conf (derived from lib/imapoptions):<br>
        </div>
        <div><br>
        </div>
        <pre>          defaultsearchtier: <empty string>
              Name of the default tier that messages will be indexed to. Search indexes can be organized in tiers to allow index
              storage in different directories and physical media. See the man page of squatter for details. The default  search
              tier also requires the definition of an according searchtierpartition-name entry.

              This option MUST be specified for xapian search.
...
          searchtierpartition-name: <none>
              The  pathname where to store the xapian search indexes of searchtier for mailboxes of partition name. This must be
              configured for the defaultsearchtier and any additional search tier (see squatter for details).

              For example: if defaultpartition is defined as part1 and defaultsearchtier as tier1 then  the  configuration  must
              contain  an  entry tier1partitionname-part1 that defines the path where to store this tier1's search index for the
              part1 partition.

              This option MUST be specified for xapian search.

</pre>
        <div>This is all so muddled, due to the use of the word
          "default" as an embedded string within so many settings, some
          of which refer to a default value and some of which refer to a
          partition called "default".  I really think we need to go
          through the documentation from top to bottom and weed out such
          confusing language.  The same is true for the various uses of
          the word "archive" and some others.  This sort of thing is
          clearly leading to some folks just giving up on complex Cyrus
          features, the implementation of which depend upon piercing the
          veil of confusion surround this language (he says in
          frustration).<br>
        </div>
        <div><br>
        </div>
        <div>Oh, and that imapd.conf(5) comment "(see sqautter for
          details)" is useless, as the squatter(8) man page says nothing
          about this (damn manpage writers!).<br>
        </div>
        <div><br>
        </div>
        <div>Thanks in advance,<br>
        </div>
        <div>    -nic<br>
        </div>
        <div><br>
        </div>
        <div><br>
        </div>
        <div>On 12/20/2017 04:04 PM, Bron Gondwana wrote:<br>
        </div>
        <blockquote type="cite"
cite="mid:1513807486.1214759.1211724200.107B9D9E@webmail.messagingengine.com">
          <div style="font-family:Arial;">Totally!  The name "archive"
            is overused.  It could be called something else easily
            enough.<br>
          </div>
          <div style="font-family:Arial;"><br>
          </div>
          <div style="font-family:Arial;">Bron.<br>
          </div>
          <div><br>
          </div>
          <div><br>
          </div>
          <div>On Thu, 21 Dec 2017, at 01:05, Nic Bernstein wrote:<br>
          </div>
          <blockquote type="cite">
            <div style="font-family:Arial;">Bron,<br>
            </div>
            <div style="font-family:Arial;">Thanks for the swift reply. 
              So if I understand this correctly, the
              "archivesearchpartition-default" is named such because
              it's for the archive location of Xapian search data, not
              because it's Xapian search data from an Archive
              partition.  Is that correct?  In other words, this is just
              another circumstance where seemingly obvious partition
              names (like default-default) get us into documentation
              trouble.  Right?<br>
            </div>
            <div style="font-family:Arial;"><br>
            </div>
            <div style="font-family:Arial;">Thanks again,<br>
            </div>
            <div style="font-family:Arial;">    -nic<br>
            </div>
            <div style="font-family:Arial;"><br>
            </div>
            <div style="font-family:Arial;"><br>
            </div>
            <div>On 12/20/2017 05:39 AM, Bron Gondwana wrote:<br>
            </div>
            <blockquote type="cite"
cite="mid:1513769948.2909243.1211084184.71773DBD@webmail.messagingengine.com">
              <div style="font-family:Arial;">Hi Nic,<br>
              </div>
              <div style="font-family:Arial;"><br>
              </div>
              <div style="font-family:Arial;">The Xapian partitions are
                entirely separate from archive partitions.  The indexing
                code will find the file in the correct location if it
                needs to read it.<br>
              </div>
              <div style="font-family:Arial;"><br>
              </div>
              <div style="font-family:Arial;">Here's an example from one
                of our servers:<br>
              </div>
              <div style="font-family:Arial;"><br>
              </div>
              <div style="font-family:Arial;">defaultpartition: default<br>
              </div>
              <div style="font-family:Arial;">defaultsearchtier: temp<br>
              </div>
              <div style="font-family:Arial;">partition-default:
                /mnt/ssd21d2/sloti21d2t40/store254/spool<br>
              </div>
              <div style="font-family:Arial;">tempsearchpartition-default:
                /tmpfs-search/sloti21d2t40<br>
              </div>
              <div style="font-family:Arial;">metasearchpartition-default:
                /mnt/ssd21d2/sloti21d2t40/store254/search<br>
              </div>
              <div style="font-family:Arial;">datasearchpartition-default:
                /mnt/i21d2search/sloti21d2t40/store254/search<br>
              </div>
              <div style="font-family:Arial;">archivesearchpartition-default:
                /mnt/i21d2search/sloti21d2t40/store254/search-archive<br>
              </div>
              <div style="font-family:Arial;">archivepartition-default:
                /mnt/i21d2t40/sloti21d2t40/store254/spool-archive<br>
              </div>
              <div style="font-family:Arial;"><br>
              </div>
              <div style="font-family:Arial;">(clearly auto-generated!)<br>
              </div>
              <div style="font-family:Arial;"><br>
              </div>
              <div style="font-family:Arial;">Bron.<br>
              </div>
              <div><br>
              </div>
              <div><br>
              </div>
              <div>On Wed, 20 Dec 2017, at 07:32, Nic Bernstein wrote:<br>
              </div>
              <blockquote type="cite">
                <div style="font-family:Arial;">Bron, et al.,<br>
                </div>
                <div style="font-family:Arial;">We're about to set up a
                  whole new bunch of partitions in support of Xapian
                  indexing, for a 2.5.10-to-3.0.4 upgrade, and then will
                  be introducing archive functionality, too.  How do
                  archive partitions and Xapian partitions interact?<br>
                </div>
                <div style="font-family:Arial;"><br>
                </div>
                <div style="font-family:Arial;">For example, the server
                  currently has the following in imapd.conf:<br>
                </div>
                <div style="font-family:Arial;"><br>
                </div>
                <blockquote>
                  <pre>defaultpartition: default
partition-default: /var/mailstores/default
metapartition-default: /var/imapmeta/default
partition-1: /var/mailstores/1
partition-2: /var/mailstores/2
partition-3: /var/mailstores/3
partition-4: /var/mailstores/4
...
partition-29: /var/mailstores/29
partition-30: /var/mailstores/30
partition-100: /var/mailstores/100
partition-temp: /var/mailstores/temp
...
# non-default metapartitions
metapartition-1: /var/imapmeta/1
metapartition-2: /var/imapmeta/2
metapartition-3: /var/imapmeta/3
metapartition-4: /var/imapmeta/4
...
metapartition-29: /var/imapmeta/29
metapartition-30: /var/imapmeta/30
metapartition-100: /var/imapmeta/100
metapartition-temp: /var/imapmeta/temp



</pre>
                </blockquote>
                <div style="font-family:Arial;">Going by the
                  documentation, which I wrote with help from you good
                  folk at Fastmail, the Archive partition scheme might
                  look something like this:<br>
                </div>
                <div style="font-family:Arial;"><a
href="https://www.cyrusimap.org/imap/reference/admin/locations/archive-partitions.html"
                    moz-do-not-send="true">https://www.cyrusimap.org/imap/reference/admin/locations/archive-partitions.html</a><br>
                </div>
                <div style="font-family:Arial;"><br>
                </div>
                <blockquote>
                  <pre>archivepartition-default: /var/mailarchives/default
archivepartition-1: /var/mailarchives/1
archivepartition-2: /var/mailarchives/2
archivepartition-3: /var/mailarchives/3
archiveartition-4: /var/mailarchives/4
...
archivepartition-29: /var/mailarchives/29
archivepartition-30: /var/mailarchives/30
archivepartition-100: /var/mailarchives/100
archivepartition-temp: /var/mailarchives/temp



</pre>
                </blockquote>
                <div style="font-family:Arial;">And the Xapian partition
                  structure to mate with this would look something like
                  this (again, from the docs):<br>
                </div>
                <div style="font-family:Arial;"><a
                    href="https://www.cyrusimap.org/imap/developer/install-xapian.html"
                    moz-do-not-send="true">https://www.cyrusimap.org/imap/developer/install-xapian.html</a><br>
                </div>
                <div style="font-family:Arial;"><br>
                </div>
                <blockquote>
                  <pre>defaultpartition: default
partition-default: /var/mailstores/default
search_engine: <span>xapian</span>
search_index_headers: no
search_batchsize: 8192
defaultsearchtier: t1
1searchtier: t1
2searchtier: t1
3searchtier: t1
4searchtier: t1
...
29searchtier: t1
30searchtier: t1
100searchtier: t1
tempsearchtier: t1
...
t1searchpartition-default: /var/search/default
t1searchpartition-1: /var/search/1
t1searchpartition-2: /var/search/2
t1searchpartition-3: /var/search/3
t1searchpartition-4: /var/search/4
...
t1searchpartition-29: /var/search/29
t1searchpartition-30: /var/search/30
t1searchpartition-100: /var/search/100
t1searchpartition-temp: /var/search/temp


</pre>
                </blockquote>
                <div style="font-family:Arial;">First question, since
                  there's no examples to work from; Is this Xapian
                  layout correct? <br>
                </div>
                <div style="font-family:Arial;"><br>
                </div>
                <div style="font-family:Arial;">Do I need to define
                  & create Xapian partitions for the metadata
                  partitions, as is indirectly implied in Bron's
                  original email on this topic:<br>
                </div>
                <div style="font-family:Arial;">    <a
href="https://lists.tartarus.org/pipermail/xapian-discuss/2014-October/009112.html"
                    moz-do-not-send="true">https://lists.tartarus.org/pipermail/xapian-discuss/2014-October/009112.html</a><br>
                </div>
                <div style="font-family:Arial;"><br>
                </div>
                <div style="font-family:Arial;">Also, how do these
                  Xapian and Archive, interact?  Do I need to add a
                  separate Xapian partition for each Archive partition,
                  or will the Archive partition be treated like a child
                  of the non-Archive partition?  (again, implied but not
                  directly addressed in that email).<br>
                </div>
                <div style="font-family:Arial;"><br>
                </div>
                <div style="font-family:Arial;">Any guidance gladly
                  accepted, and whatever I learn will be repackaged into
                  more complete documentation on same.<br>
                </div>
                <div style="font-family:Arial;"><br>
                </div>
                <div style="font-family:Arial;">Cheers,<br>
                </div>
                <div style="font-family:Arial;">    -nic<br>
                </div>
                <div style="font-family:Arial;"><br>
                </div>
                <pre>-- 
Nic Bernstein                             <a href="mailto:nic@onlight.com" moz-do-not-send="true">nic@onlight.com</a>
Onlight, Inc.                             <a href="http://www.onlight.com" moz-do-not-send="true">www.onlight.com</a>
6525 W Bluemound Road, Suite 24           v. 414.272.4477
Milwaukee, Wisconsin  53213-4073



</pre>
                <p>Email had 1 attachment:<br>
                </p>
                <ul>
                  <li>
                    <div style="font-family:Arial;"><code>nic.vcf</code><br>
                    </div>
                    <div style="font-family:Arial;">  1k (text/x-vcard)<br>
                    </div>
                  </li>
                </ul>
              </blockquote>
              <div style="font-family:Arial;"><br>
              </div>
              <div>
                <div>--<br>
                </div>
                <div>  Bron Gondwana<br>
                </div>
                <div>  <a href="mailto:brong@fastmail.fm"
                    moz-do-not-send="true">brong@fastmail.fm</a><br>
                </div>
                <div><br>
                </div>
              </div>
              <div style="font-family:Arial;"><br>
              </div>
            </blockquote>
            <div style="font-family:Arial;"><br>
            </div>
            <pre>-- 
Nic Bernstein                             <a href="mailto:nic@onlight.com" moz-do-not-send="true">nic@onlight.com</a>
Onlight Inc.                              <a href="http://www.onlight.com" moz-do-not-send="true">www.onlight.com</a>
6525 W Bluemound Rd., Ste 24              v. 414.272.4477
Milwaukee, Wisconsin  53213-4073          f. 414.290.0335


</pre>
          </blockquote>
          <div style="font-family:Arial;"><br>
          </div>
          <div>
            <div>--<br>
            </div>
            <div>  Bron Gondwana<br>
            </div>
            <div>  <a href="mailto:brong@fastmail.fm"
                moz-do-not-send="true">brong@fastmail.fm</a><br>
            </div>
            <div><br>
            </div>
          </div>
          <div style="font-family:Arial;"><br>
          </div>
        </blockquote>
        <div><br>
        </div>
        <p>Email had 1 attachment:<br>
        </p>
        <ul>
          <li>
            <div><code>nic.vcf</code><br>
            </div>
            <div>  1k (text/x-vcard)<br>
            </div>
          </li>
        </ul>
      </blockquote>
      <div><br>
      </div>
    </blockquote>
    <br>
  </body>
</html>