<!DOCTYPE html><html><head><title></title><style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}</style></head><body><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"><br></div><div>On Tue, May 21, 2019, at 18:41, Dilyan Palauzov wrote:<br></div><blockquote type="cite" id="qt"><div style="font-family:Arial;">Hello,<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">thanks, Bron, for your answer.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">I gave it a try.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">squatter does not remove .NEW directories when aborted (SIGINT), the <br></div><div style="font-family:Arial;">directories have to be removed manually<br></div></blockquote><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"><a href="https://github.com/cyrusimap/cyrus-imapd/issues/2765">https://github.com/cyrusimap/cyrus-imapd/issues/2765</a><br></div><div style="font-family:Arial;"><br></div><blockquote type="cite" id="qt"><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">squatter -t X -z X -o recognizes, when the directory structure behind <br></div><div style="font-family:Arial;">tier X exists, that nothing has to be done, prints “Skipping X for <br></div><div style="font-family:Arial;">user.ABC, only one” and quits, without updating the .xapianactive files.<br></div></blockquote><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">yeah right, that won't work. Glad to know :)<br></div><div style="font-family:Arial;"><br></div><blockquote type="cite" id="qt"><div style="font-family:Arial;">squatter -t Y -z Y -o, when the directory structructure behind tier Y <br></div><div style="font-family:Arial;">does not exist, prints “compressing Y:1,Y:0 to Y:2 for user... (active <br></div><div style="font-family:Arial;">Y:1,Y:0)”. As far as I remember this has not updated the xapianactive <br></div><div style="font-family:Arial;">files.<br></div></blockquote><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Yeah right, it won't add a new target unless you are compressing the current first item in xapianactive.<br></div><div style="font-family:Arial;"><br></div><blockquote type="cite" id="qt"><div style="font-family:Arial;">squatter -t X -z Y -o does add to the .xapianactive files the <br></div><div style="font-family:Arial;">defaultsearhtier, but first has to duplicate with rsync all existing <br></div><div style="font-family:Arial;">files. This takes a while… But at the end did what I wanted. <br></div><div style="font-family:Arial;">Afterwards the directory structure for the new tier was not created. <br></div><div style="font-family:Arial;">The directory structure was created once I started all the cyrus <br></div><div style="font-family:Arial;">processes again.<br></div></blockquote><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">That makes sense. We don't create a directory structure until a document gets created in there.<br></div><div style="font-family:Arial;"><br></div><blockquote type="cite" id="qt"><div style="font-family:Arial;">squatter -t X -z Y -o emits the message “undefined search partition <br></div><div style="font-family:Arial;">X,Ysearchpartition-default” and then “compressing X:0,X,Y:0 to Y:2 for <br></div><div style="font-family:Arial;">... (active Y:0,X:0,X,Y:0,Y:1)”.<br></div></blockquote><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">That sounds like a sanity checking failure! Good catch:<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"><a href="https://github.com/cyrusimap/cyrus-imapd/issues/2764">https://github.com/cyrusimap/cyrus-imapd/issues/2764</a><br></div><div style="font-family:Arial;"><br></div><blockquote type="cite" id="qt"><div style="font-family:Arial;">Does squatter -t X -z Y append X to Y, or it deletes Y and copies X to <br></div><div style="font-family:Arial;">Y? In the latter case, is there any (performance) difference between <br></div><div style="font-family:Arial;">"squatter -t X,Y -z Y" and “squatter -t Y,X -z Y”?<br></div></blockquote><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">There's no difference in what order you add items to -t. -t is a comma separated list of selectors for source items. You can even explicitly say:<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">squatter -t X:0,X:2,Y:45 -z Y and it will compact just those three sources into a new target in Y.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">What it does under the hood is creates a new database and copy all the documents over from the source databases, then compress the end result into the most compact and fastest xapian format which is designed to never write again. This compressed file is then stored into the target database name, and in an exclusively locked operation the new database is moved into place and the old tiers are removed from the xapianactive, such that all new searches look into the single destination database instead of the multiple source databases.<br></div><div style="font-family:Arial;"><br></div><blockquote type="cite" id="qt"><div style="font-family:Arial;">Can one xapian tier store a document, and another tier store the <br></div><div style="font-family:Arial;">information, that the address of the document has changed?<br></div></blockquote><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">It doesn't work like that. The addresses of the documents never change (they are the sha1 of the document contents, and Cyrus documents are all immutable). The xapian engine searches across the full set of databases listed in xapianactive in order to find document ids, then maps them through the conversations.db file to find the actual emails. A copy/move of an email updates the conversations.db lookups, so the next search will find the new location without anything changing in xapian.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">the cyrus.indexed.db file is just a convenience to allow rolling squatter to avoid having to re-scan records that it knows are already indexed.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Bron.<br></div><div style="font-family:Arial;"><br></div><blockquote type="cite" id="qt"><div style="font-family:Arial;">Regards<br></div><div style="font-family:Arial;"> Дилян<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">----- Message from Bron Gondwana <brong@fastmailteam.com> ---------<br></div><div style="font-family:Arial;"> Date: Mon, 20 May 2019 18:52:07 +1000<br></div><div style="font-family:Arial;"> From: Bron Gondwana <brong@fastmailteam.com><br></div><div style="font-family:Arial;">Subject: Re: Prepending Xapian Tiers<br></div><div style="font-family:Arial;"> To: Cyrus Devel <cyrus-devel@lists.andrew.cmu.edu><br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">> On Fri, May 17, 2019, at 23:52, Дилян Палаузов wrote:<br></div><div style="font-family:Arial;">>> Hello,<br></div><div style="font-family:Arial;">>><br></div><div style="font-family:Arial;">>> I set up a Cyrus system with one tier. I think it works. The <br></div><div style="font-family:Arial;">>> .xapianactive files contain 'tiername: 0'.<br></div><div style="font-family:Arial;">>><br></div><div style="font-family:Arial;">>> How can I insert a second tier?<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> I have never tried this on a live server! Clearly the right thing to <br></div><div style="font-family:Arial;">> do is to build a cassandane search which implements doing this so <br></div><div style="font-family:Arial;">> that we can make sure it works.<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">>> Adding a XYZsearchpartition-default to imapd.conf, together with <br></div><div style="font-family:Arial;">>> defaultsearchtier: XYZ does not utilize the new directory: it stays <br></div><div style="font-family:Arial;">>> empty and the .xapianactive files do not get updated to mention the <br></div><div style="font-family:Arial;">>> new tier.<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> That looks like it should work. I assume you have restarted your <br></div><div style="font-family:Arial;">> cyrus since making the change? I'm not certain that a rolling <br></div><div style="font-family:Arial;">> squatter will discover a new config in the way that imapd does.<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> Also - you'll need to run squatter in compact mode in order to add a <br></div><div style="font-family:Arial;">> new xapianactive entry. The simplest could be:<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> squatter -z tiername -t tiername -o<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> I believe that given your current setup, this will just copy the <br></div><div style="font-family:Arial;">> entry from tiername:0 to tirename:1 and also create XYZ:0 in the <br></div><div style="font-family:Arial;">> xapianactive file at the same time.<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">>> Besides, if a message is MOVEd over IMAP, is any optimization <br></div><div style="font-family:Arial;">>> utilized, to avoid reindexing the message, but just change the <br></div><div style="font-family:Arial;">>> address of the document?<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> Yes, both XAPINDEXED mode where the GUID is read from xapian, and <br></div><div style="font-family:Arial;">> CONVINDEXED mode where the GUID is looked up via user.conversations <br></div><div style="font-family:Arial;">> and then mapped into the cyrus.indexed.db files in each xapian tier <br></div><div style="font-family:Arial;">> allow Xapian to skip reindexing when a message is already indexed. <br></div><div style="font-family:Arial;">> This works for both MOVE and for re-uploading of an identical <br></div><div style="font-family:Arial;">> message file via IMAP.<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> Cheers,<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> Bron.<br></div><div style="font-family:Arial;">><br></div><div style="font-family:Arial;">> --<br></div><div style="font-family:Arial;">> Bron Gondwana, CEO, FastMail Pty Ltd<br></div><div style="font-family:Arial;">> brong@fastmailteam.com<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">----- End message from Bron Gondwana <brong@fastmailteam.com> -----<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"><br></div></blockquote><div style="font-family:Arial;"><br></div><div id="sig56629417"><div class="signature">--<br></div><div class="signature"> Bron Gondwana, CEO, FastMail Pty Ltd<br></div><div class="signature"> brong@fastmailteam.com<br></div><div class="signature"><br></div></div><div style="font-family:Arial;"><br></div></body></html>