Removing email from Xapian tier databases

[Egoitz Aurrekoetxea]

Now I'm noticing for instance, for moving data between Xapian
databases.. you need to launch something like : 

sudo -u cyrus /usr/cyrus/bin/squatter -C /usr/local/etc/imapd.conf -v -z
archive -t temp,meta,data,archive -u user/egoitz at sarenet.es

perhaps would be better to do :

sudo -u cyrus /usr/cyrus/bin/squatter -C /usr/local/etc/imapd.conf -F -v
-z archive -t temp,meta,data,archive -u user/egoitz at sarenet.es

But then, having two Squatter processes running at same time, one for
rolling mode and one for moving/repacking data, should not be an issue?.

Thanks mates!!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
egoitz at sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 11-02-2019 11:22, Egoitz Aurrekoetxea escribió:

> Hi Bron, 
> 
> So, it would be interesting to run once a day... for instance in cyrus.conf in events section : 
> 
> repack_xapian      cmd="squatter -F" at=0200 
> 
> Is it needed top stop the other rolling Squatter we run, in same cyrus.conf as : 
> 
> START {
> # do not delete this entry!
> recover       cmd="ctl_cyrusdb -r"
> 
> squatter cmd="squatter -R"
> } 
> 
> Thank you so much for all the clarifications mate :) really :) 
> 
> Cheers!
> 
> ---
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia) 
> egoitz at sarenet.es 
> www.sarenet.es [1] 
> Antes de imprimir este correo electrónico piense si es necesario hacerlo. 
> 
> El 11-02-2019 10:23, Bron Gondwana escribió: 
> Conversations.db is an index over lots of interesting bits of the message, but the key part that's used by Xapian is the mapping from G key (aka: GUID, aka: sha1 of the message RFC822 data) to individual email.  It's used for deduplication and for mapping from results to messages. 
> 
> The data in conversations.db is added and removed in real time as messages are appended and updated in the cyrus.index. 
> 
> The data in the xapian databases on the other hand is append only - so you can wind up with hits that no longer map to existing emails.  The way to solve that is with a xapian repack that filters messages - which can be done using the -F flag to squatter. 
> 
> Cheers, 
> 
> Bron. 
> 
> On Sat, Feb 9, 2019, at 23:04, Egoitz Aurrekoetxea wrote: 
> 
> Good morning, 
> 
> As far as I understood, for Xapian you first create it's conversation database in order to work. Later you create database(s) for each mailbox where Xapian can search in. You can move data between them, new mails become indexed for instance Squatter in rolling mode... that's ok... and understood I  think. I was wondering, what happens when mail indexed in the archive database in removed and then does not exist any more in the database... does Squatter rolling log manage that too?. 
> 
> By the way. I was wondering if mail gets indexed in the tier databases (for instance in Fastmail in temp, meta, data, archine...) what's the role or function of conversations databases you create with ctl_conversationsdb -b -r ?. 
> 
> Cheers!
> 
> -- 
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 
> 944 209 470 
> Parque Tecnológico. Edificio 103 
> 48170 Zamudio (Bizkaia) 
> egoitz at sarenet.es
> 
> www.sarenet.es [1] 
> 
> Antes de imprimir este correo electrónico piense si es necesario hacerlo. 
> ---- 
> Cyrus Home Page: http://www.cyrusimap.org/ 
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 
> To Unsubscribe: 
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 
> 
> -- 
> Bron Gondwana, CEO, FastMail Pty Ltd 
> brong at fastmailteam.com 
> 
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 

Links:
------
[1] http://www.sarenet.es
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190211/61e22080/attachment-0001.html>