<!DOCTYPE html><html><head><title></title><style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}</style></head><body><div style="font-family:Arial;">It's definitely safe to have one rolling mode writing and one repacking. I wouldn't run multiple repacks in parallel, as they can wind up doing duplicate work (though the end result should always be correct and safe).<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Here's what we run:<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"># Any time the disk gets over 50%, compress -o single down to data<br></div><div style="font-family:Arial;">13 * * * * [% INCLUDE cronjob c='/home/mod_perl/hm/scripts/xapian_compact.pl -a -o -d 50 temp data' %]<br></div><div style="font-family:Arial;"># Copy the temporary search databases down to data during the week<br></div><div style="font-family:Arial;">43 1 * * 1,2,3,4,5,6 [% INCLUDE cronjob c='/home/mod_perl/hm/scripts/xapian_compact.pl -a temp,meta data' %]<br></div><div style="font-family:Arial;"># Sundays repack the entire data directory<br></div><div style="font-family:Arial;">43 1 * * 0 [% INCLUDE cronjob c='/home/mod_perl/hm/scripts/xapian_compact.pl -a temp,meta,data data' %]<br></div><div style="font-family:Arial;"># Late on Sundays, pack any oversized data directories down to archive<br></div><div style="font-family:Arial;">0 15 * * 0 [% INCLUDE cronjob c='/home/mod_perl/hm/scripts/xapian_archive.pl -a' %]<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">And here's the interesting logic. In xapian_compact.pl:<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> if ($Opts{d}) {<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> my $Path = $Slot->SearchPath();<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> my $Usage = df($Path);<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> my $RunUsage = df("/run/cyrus");<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> return Process::Status->new(0) if ($Usage->{per} < $Opts{d} and $RunUsage->{per} < $Opts{d});<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> }<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"><br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> my @args = (-z => $dest, -t => $src);<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> push @args, '-v' if $Opts{v};<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> push @args, '-o' if $Opts{o};<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> push @args, '-F' if $Opts{F};<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> push @args, '-X' if $Opts{X};<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> push @args, ('-T' => $Opts{T}) if $Opts{T};<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> push @args, ('-u' => $Opts{u}) if $Opts{u};<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> my %RunOpts = (<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> PrintOutput => 1,<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> );<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> $RunOpts{Nice} = 1 unless $Opts{N};<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> $RunOpts{Daemon} = 1 if $Opts{D};<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"><br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> $0 = "xapian_compact: $SN";<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> $Slot->RunCommand(\%RunOpts, 'squatter', @args);<br></span></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">And in xapian_archive.pl:<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font">my $Percent = $Opts{P} || 20;</span><br></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font">[...]<br></span></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> foreach my $user (sort keys %$DataUsage) {<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> my $au = $ArchiveUsage->{$user} || 1;<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> my $du = $DataUsage->{$user} || 1;<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> if ($du < 5000) {<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> print "Too small $user ($du)\n";<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> next;<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> }<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> my $This = int($du * 100 / $au);<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> if ($This < $Percent) {<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> print "Not enough dirty $user: ($du, $au)\n";<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> next;<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> }<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> print "Recompacting $user: ($du, $au)\n";<br></span></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font"> my @args = (-z => 'archive', -t => 'data,archive');</span><br></div><div style="font-family:Arial;"><span style="font-family: menlo, consolas, monospace, sans-serif;" class="font">[...]<br></span></div><div style="font-family:Arial;"> <br></div><div style="font-family:Arial;">In summary, repack data down to archive if data is more than 1/5 size of existing archive. So each of these scripts is a wrapper around squatter to help it run automatically.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Bron.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">On Mon, Feb 11, 2019, at 21:55, Egoitz Aurrekoetxea wrote:<br></div><blockquote type="cite" id="fastmail-quoted"><p>Now I'm noticing for instance, for moving data between Xapian databases.. you need to launch something like :<br></p><pre><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">sudo -u cyrus /usr/cyrus/bin/squatter -C /usr/local/etc/imapd.conf -v -z archive -t temp,meta,data,archive -u user/egoitz@sarenet.es<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">perhaps would be better to do :<br></div></pre><pre><div style="font-family:Arial;">sudo -u cyrus /usr/cyrus/bin/squatter -C /usr/local/etc/imapd.conf <u><b>-F</b></u> -v -z archive -t temp,meta,data,archive -u user/egoitz@sarenet.es<br></div></pre><pre><div style="font-family:Arial;">But then, having two Squatter processes running at same time, one for rolling mode and one for moving/repacking data, should not be an issue?.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Thanks mates!!<br></div><div style="font-family:Arial;"><br></div></pre><div><div style="font-family:Arial;">---<br></div><div style="font-family:Arial;"> <br></div><div style="clear:both;text-align:left;" class="fastmail-quoted-firmasarenet"><div style="margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;" class="fastmail-quoted-imgsarenet"><img alt="sarenet" src="https://www.sarenet.es/estaticos/LogoSarenetEmails.png?newlogo"><br></div><div style="font-family:Helvetica, Arial, sans-serif;font-weight:bold;font-size:14px;color:rgb(51, 51, 51);" class="fastmail-quoted-titulosarenet"><b>Egoitz Aurrekoetxea</b><br></div><div style="font-family:Helvetica, Arial, sans-serif;font-weight:normal;font-size:12px;color:rgb(51, 51, 51);" class="fastmail-quoted-dptosarenet">Departamento de sistemas<br></div><div style="font-family:Helvetica, Arial, sans-serif;font-weight:normal;font-size:12px;color:rgb(51, 51, 51);" class="fastmail-quoted-textosarenet"><div style="font-family:Arial;">944 209 470<br></div><div style="font-family:Arial;">Parque Tecnológico. Edificio 103<br></div><div style="font-family:Arial;">48170 Zamudio (Bizkaia)<br></div></div><div style="font-family:Helvetica, Arial, sans-serif;font-weight:normal;font-size:12px;color:rgb(0, 122, 196);line-height:1.2;" class="fastmail-quoted-lnksarenet"><label id="fastmail-quoted-label_email">egoitz@sarenet.es<br></label></div><div style="font-family:Helvetica, Arial, sans-serif;font-weight:normal;font-size:12px;color:rgb(0, 122, 196);line-height:1.2;" class="fastmail-quoted-lnksarenet"><a href="http://www.sarenet.es" style="font-size:12px;color:rgb(0, 122, 196);text-decoration-line:underline;text-decoration-style:solid;text-decoration-color:currentcolor;">www.sarenet.es</a><br></div><div style="font-family:Arial;"><br></div><div style="font-family:Helvetica, Arial, sans-serif;font-weight:normal;font-size:10px;color:rgb(160, 195, 97);" class="fastmail-quoted-imprimirsarenet">Antes de imprimir este correo electrónico piense si es necesario hacerlo.<br></div></div></div><p><br></p><p>El 11-02-2019 11:22, Egoitz Aurrekoetxea escribió:<br></p><blockquote style="padding-top:0px;padding-right:0.4em;padding-bottom:0px;padding-left:0.4em;border-left-color:rgb(16, 16, 255);border-left-style:solid;border-left-width:2px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;" type="cite"><p>Hi Bron,<br></p><p><br></p><p>So, it would be interesting to run once a day... for instance in cyrus.conf in events section :<br></p><p>repack_xapian cmd="squatter -F" at=0200<br></p><p>Is it needed top stop the other rolling Squatter we run, in same cyrus.conf as :<br></p><p></p><div style="font-family:Arial;">START {<br></div><div style="font-family:Arial;"> # do not delete this entry!<br></div><div style="font-family:Arial;"> recover cmd="ctl_cyrusdb -r"<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"> squatter cmd="squatter -R"<br></div><div style="font-family:Arial;">}<br></div><p></p><p><br></p><p>Thank you so much for all the clarifications mate :) really :)<br></p><p><br></p><p>Cheers!<br></p><div><div style="font-family:Arial;">---<br></div><div style="font-family:Arial;"> <br></div><div style="clear:both;text-align:left;" class="fastmail-quoted-firmasarenet"><div style="margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;" class="fastmail-quoted-imgsarenet"><img alt="sarenet" src="https://www.sarenet.es/estaticos/LogoSarenetEmails.png?newlogo"><br></div><div style="font-family:Helvetica, Arial, sans-serif;font-weight:bold;font-size:14px;color:rgb(51, 51, 51);" class="fastmail-quoted-titulosarenet"><b>Egoitz Aurrekoetxea</b><br></div><div style="font-family:Helvetica, Arial, sans-serif;font-weight:normal;font-size:12px;color:rgb(51, 51, 51);" class="fastmail-quoted-dptosarenet">Departamento de sistemas<br></div><div style="font-family:Helvetica, Arial, sans-serif;font-weight:normal;font-size:12px;color:rgb(51, 51, 51);" class="fastmail-quoted-textosarenet"><div style="font-family:Arial;">944 209 470<br></div><div style="font-family:Arial;">Parque Tecnológico. Edificio 103<br></div><div style="font-family:Arial;">48170 Zamudio (Bizkaia)<br></div></div><div style="font-family:Helvetica, Arial, sans-serif;font-weight:normal;font-size:12px;color:rgb(0, 122, 196);line-height:1.2;" class="fastmail-quoted-lnksarenet"><label id="fastmail-quoted-label_email">egoitz@sarenet.es<br></label></div><div style="font-family:Helvetica, Arial, sans-serif;font-weight:normal;font-size:12px;color:rgb(0, 122, 196);line-height:1.2;" class="fastmail-quoted-lnksarenet"><a rel="noopener noreferrer" href="http://www.sarenet.es" style="font-size:12px;color:rgb(0, 122, 196);text-decoration-line:underline;text-decoration-style:solid;text-decoration-color:currentcolor;">www.sarenet.es</a><br></div><div style="font-family:Arial;"><br></div><div style="font-family:Helvetica, Arial, sans-serif;font-weight:normal;font-size:10px;color:rgb(160, 195, 97);" class="fastmail-quoted-imprimirsarenet">Antes de imprimir este correo electrónico piense si es necesario hacerlo.<br></div></div></div><p><br></p><p>El 11-02-2019 10:23, Bron Gondwana escribió:<br></p><blockquote style="padding-top:0px;padding-right:0.4em;padding-bottom:0px;padding-left:0.4em;border-left-color:rgb(16, 16, 255);border-left-style:solid;border-left-width:2px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;"><div style="font-family:Arial;">Conversations.db is an index over lots of interesting bits of the message, but the key part that's used by Xapian is the mapping from G key (aka: GUID, aka: sha1 of the message RFC822 data) to individual email. It's used for deduplication and for mapping from results to messages.<br></div><div style="font-family:Arial;"> <br></div><div style="font-family:Arial;">The data in conversations.db is added and removed in real time as messages are appended and updated in the cyrus.index.<br></div><div style="font-family:Arial;"> <br></div><div style="font-family:Arial;">The data in the xapian databases on the other hand is append only - so you can wind up with hits that no longer map to existing emails. The way to solve that is with a xapian repack that filters messages - which can be done using the -F flag to squatter.<br></div><div style="font-family:Arial;"> <br></div><div style="font-family:Arial;">Cheers,<br></div><div style="font-family:Arial;"> <br></div><div style="font-family:Arial;">Bron.<br></div><div style="font-family:Arial;"> <br></div><div style="font-family:Arial;">On Sat, Feb 9, 2019, at 23:04, Egoitz Aurrekoetxea wrote:<br></div><blockquote id="fastmail-quoted-fastmail-quoted"><p>Good morning,<br></p><p><br></p><p>As far as I understood, for Xapian you first create it's conversation database in order to work. Later you create database(s) for each mailbox where Xapian can search in. You can move data between them, new mails become indexed for instance Squatter in rolling mode... that's ok... and understood I think. I was wondering, what happens when mail indexed in the archive database in removed and then does not exist any more in the database... does Squatter rolling log manage that too?.<br></p><p><br></p><p>By the way. I was wondering if mail gets indexed in the tier databases (for instance in Fastmail in temp, meta, data, archine...) what's the role or function of conversations databases you create with ctl_conversationsdb -b -r ?.<br></p><p><br></p><p>Cheers!<br></p><div><div style="font-family:Arial;">--<br></div><div style="font-family:Arial;"> <br></div><div style="clear:both;text-align:left;" class="fastmail-quoted-fastmail-quoted-firmasarenet"><div style="margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;" class="fastmail-quoted-fastmail-quoted-imgsarenet"><img alt="sarenet" src="https://www.sarenet.es/estaticos/LogoSarenetEmails.png?newlogo"><br></div><div style="font-family:Helvetica, Arial, sans-serif;font-weight:bold;font-size:14px;color:rgb(51, 51, 51);" class="fastmail-quoted-fastmail-quoted-titulosarenet"><b>Egoitz Aurrekoetxea</b><br></div><div style="font-family:Helvetica, Arial, sans-serif;font-weight:normal;font-size:12px;color:rgb(51, 51, 51);" class="fastmail-quoted-fastmail-quoted-dptosarenet">Departamento de sistemas<br></div><div style="font-family:Helvetica, Arial, sans-serif;font-weight:normal;font-size:12px;color:rgb(51, 51, 51);" class="fastmail-quoted-fastmail-quoted-textosarenet"><div style="font-family:Arial;">944 209 470<br></div><div style="font-family:Arial;">Parque Tecnológico. Edificio 103<br></div><div style="font-family:Arial;">48170 Zamudio (Bizkaia)<br></div></div><div style="font-family:Helvetica, Arial, sans-serif;font-weight:normal;font-size:12px;color:rgb(0, 122, 196);line-height:1.2;" class="fastmail-quoted-fastmail-quoted-lnksarenet"><label id="fastmail-quoted-fastmail-quoted-label_email">egoitz@sarenet.es<br></label></div><div style="font-family:Helvetica, Arial, sans-serif;font-weight:normal;font-size:12px;color:rgb(0, 122, 196);line-height:1.2;" class="fastmail-quoted-fastmail-quoted-lnksarenet"><a rel="noopener noreferrer" href="http://www.sarenet.es" style="font-size:12px;color:rgb(0, 122, 196);text-decoration-line:underline;text-decoration-style:solid;text-decoration-color:currentcolor;">www.sarenet.es</a><br></div><div style="font-family:Arial;"> <br></div><div style="font-family:Helvetica, Arial, sans-serif;font-weight:normal;font-size:10px;color:rgb(160, 195, 97);" class="fastmail-quoted-fastmail-quoted-imprimirsarenet">Antes de imprimir este correo electrónico piense si es necesario hacerlo.<br></div></div></div><div>----<br></div><div>Cyrus Home Page: http://www.cyrusimap.org/<br></div><div>List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/<br></div><div>To Unsubscribe:<br></div><div>https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus<br></div></blockquote><div style="font-family:Arial;"> <br></div><div id="fastmail-quoted-sig56629417"><div class="fastmail-quoted-signature">--<br></div><div class="fastmail-quoted-signature"> Bron Gondwana, CEO, FastMail Pty Ltd<br></div><div class="fastmail-quoted-signature"> brong@fastmailteam.com<br></div><div class="fastmail-quoted-signature"> <br></div></div><div style="font-family:Arial;"> <br></div><div style="font-family:Arial;"><br></div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;font-family:monospace;" class="fastmail-quoted-pre"><div style="font-family:Arial;">----<br></div><div style="font-family:Arial;"> Cyrus Home Page: <a rel="noopener noreferrer" href="http://www.cyrusimap.org/">http://www.cyrusimap.org/</a><br></div><div style="font-family:Arial;"> List Archives/Info: <a rel="noopener noreferrer" href="http://lists.andrew.cmu.edu/pipermail/info-cyrus/">http://lists.andrew.cmu.edu/pipermail/info-cyrus/</a><br></div><div style="font-family:Arial;"> To Unsubscribe:<br></div><div style="font-family:Arial;"> <a rel="noopener noreferrer" href="https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus">https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus</a><br></div></div></blockquote></blockquote><div>----<br></div><div>Cyrus Home Page: http://www.cyrusimap.org/<br></div><div>List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/<br></div><div>To Unsubscribe:<br></div><div>https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus<br></div></blockquote><div style="font-family:Arial;"><br></div><div id="sig56629417"><div class="signature">--<br></div><div class="signature"> Bron Gondwana, CEO, FastMail Pty Ltd<br></div><div class="signature"> brong@fastmailteam.com<br></div><div class="signature"><br></div></div><div style="font-family:Arial;"><br></div></body></html>