Help with few questions

Wed Jan 9 07:37:02 EST 2019

Hi Sebastian!! 

Thank you so much mate!!. Point 2 is the most uncertain for me. I answer
your comments below :) 

Point 1 -> It's clear... yes I think so too... I was planning to force
all searches as FUZZY with search_fuzzy_always: 1 as it seems really
nice :) 

Point 2 -> So, you do a df every two minutes to see free space in
tmpfs?... and when you see it starts growing you launch a "sudo -u cyrus
/usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z
archive -t temp,meta,data,archive -u brong" equivalent command?. You
would launch it without -u I suppose, for doing Squatter for all users
(no sense of doing it for just one user... isn't it?)... I assume this
command would move all items from temp, meta and data to archive
database. So, for instance if you launch a "sudo -u cyrus
/usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z data
-t temp,meta,data -u brong" it moves (and then empties temp and meta)
temp and meta to data and even compacts the own data database?. We
usually have 2000 imap concurrent connections, 150 concurrent pop3 and
receive 120 messages/min more or less... and SSD array of disks yes..
for each server... 

Point 3 -> Totally agree. Just for confirming. 

Point 4 -> If having a crash in a server... the Cyrus log file, would
throw which mailboxes need a ctl_conversations -z and -b or "it's
better" doing it for all accounts (with -r and without -u?)?. When
talking about Xapian databases (not conversations) I assume some error
would appear for them surely... the same way as for instance if
cyrus.seen would be corrupt... isn't it?. 

Thanks a lot again. Your help is really really important for at this
stage... 

 Cheers!!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
egoitz at sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 09-01-2019 12:59, Sebastian Hagedorn escribió:

> Hi,
> 
> I will try to answer what I can ... see below.
> 
> --On 8. Januar 2019 um 18:40:17 +0100 Egoitz Aurrekoetxea <egoitz at sarenet.es> wrote:
> 
>> - search_fuzzy_always: 1 , causes all searches to go through Xapian
>> engine. Being so good as it seems (and the way it speeds in my testing
>> env search operations, it's nice!!), what could be the reason for not
>> having it enabled by default?. Can it have some kind of problem?. I
>> can't see them... Just for avoiding surprises.
> 
> With FUZZY you may get more matches than without. Look here for an explanation:
> 
> <https://tools.ietf.org/html/rfc6203>
> 
> The example in 3. is a good one:
> 
> 3.  The FUZZY Search Key
> 
> The FUZZY search key takes another search key as its argument.  The
> server is allowed to perform all matching in an implementation-
> defined manner for this search key, including ignoring the active
> comparator as defined by [RFC5255].  Typically, this would be used to
> search for strings.  For example:
> 
> C: A1 SEARCH FUZZY (SUBJECT "IMAP break")
> S: * SEARCH 1 5 10
> S: A1 OK Search completed.
> 
> Besides matching messages with a subject of "IMAP break", the above
> search may also match messages with subjects "broken IMAP", "IMAP is
> broken", or anything else the server decides that might be a good
> match.
> 
> Note that the *server* decides what "might be a good match". When all searches become FUZZY it might confuse users, but on the other hand I doubt there is a single IMAP client that lets the user choose whether a particular search should be FUZZY or not ...
> 
>> - Bron, in this post
>> https://fastmail.blog/2014/12/01/email-search-system/ told that Fastmail
>> was not able to handle with just one Xapian default database (even on
>> SSD disks) all traffic. So he said Fastmail was using a in-memory
>> filesystem for a database (called temp) for new email. Later another for
>> cleaning up that in memory filesystem. And later one more, for keeping
>> definitively the content. You seemed to use a Squatter command for
>> moving elements between databases. Concretely (sudo -u cyrus
>> /usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z
>> archive -t temp,meta,data,archive -u brong). I assume that compacts all
>> elements from all databases to archive?. If I wanted to compact elements
>> from temp to data, the command should be "sudo -u cyrus
>> /usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z data
>> -t temp -u brong" (in this example for user brong) ??. I assume you
>> launch something like it with a cron weekly and it's done?.
> 
> That depends entirely on your system. Without knowing the number of users you have, the number of mails that arrive per hour, what kind of storage systems you have, how much free RAM your servers have, it is impossible to say how often you need to squatter in compact mode. On a test server I set up I run a cron job every 2 minutes to check if tmpfs is getting tight ...
> 
>> - If something went wrong when the upgrade proccess from 2.4 to 3.0,
>> could I setup 3.0 as master of the 2.4 and later make 2.4 master again?.
>> Could that cause info loosing?. I assume yes but just
>> for knowing posibilities.
> 
> You have to describe in more detail what you mean. The point of no return is usually when you start to deliver new mail to the new server. You cannot sync from 3.0 to 2.4. That means if you start to deliver new mail to the 3.0 server, you can't go back to 2.4 without losing the messages that have arrived n the meantime. Strictly speaking to could manually copy the mailboxes from the 3.0 server to the 2.4 and run reconstruct, but I don't consider that a viable option.
> 
>> - When a Xapian or conversation index becomes broken, a reconstruct
>> could recover?. What could be the repairing procedure?.
> 
> ctl_conversationsdb -z "zaps" the conversationsdb - it basically empties it. Then you can recreate it with ctl_conversationsdb -b. The "reconstruct" command does not touch the conversationsdb. If the actual Xapian index should be broken, I guess you'd have to delete the index files and run squatter again.
> 
> Cheers
> Sebastian

Links:
------
[1] http://www.sarenet.es
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190109/4ca003d1/attachment-0001.html>