Various questions about databases (upgrade and migration)

[Egoitz Aurrekoetxea]

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
egoitz at sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 14-02-2019 15:46, Raphaël Halimi escribió:

> Hi Egoitz,
> 
> Thank you for your quick answer.
> 
> Le 14/02/2019 à 14:46, Egoitz Aurrekoetxea a écrit : Now for the databases themselves. In /var/lib/cyrus the global databases
> were converted on-the-fly:
> 
> # file /var/lib/cyrus/*.db
> /var/lib/cyrus/annotations.db:  Cyrus twoskip DB
> /var/lib/cyrus/deliver.db:      Cyrus twoskip DB
> /var/lib/cyrus/mailboxes.db:    Cyrus twoskip DB
> /var/lib/cyrus/statuscache.db:  Cyrus twoskip DB
> /var/lib/cyrus/tls_sessions.db: Cyrus twoskip DB
> /var/lib/cyrus/user_deny.db:    empty
> 
> However, the user databases were not converted:
> 
> # file /var/lib/cyrus/user/*/*
> /var/lib/cyrus/user/u/user1.seen: Cyrus skiplist DB
> /var/lib/cyrus/user/u/user1.sub:  ASCII text
> /var/lib/cyrus/user/u/user2.seen:   Cyrus skiplist DB
> /var/lib/cyrus/user/u/user2.sub:    ASCII text
> /var/lib/cyrus/user/u/user3.seen:   Cyrus skiplist DB
> /var/lib/cyrus/user/u/user3.sub:    ASCII text
> *Cyrus 2.4 converted databases on the fly. Cyrus 2.5 and newer don't.
> You should launch a "reconstruct -r -V max" for that purpose.* 
> So my next questions are: why are the databases still in skiplist
> format, whereas according to /usr/lib/cyrus/cyrus-db-types.txt, they
> should be twoskip ? Why didn't Cyrus convert them on-the-fly like the
> global databases ? Do I have to manually do it myself ? And if I do
> convert them, will it change anything (performance, reliability, etc
> etc) ?
> 
> *As said perhaps is a Debian derived config for the package. Yes you
> should with the command above.* 
> Also, what about the various databases in the mail directories
> (cyrus.cache, cyrus.header, cyrus.index) ? For most of them, the "file"
> command only reports "data". What format are they actually in ? Do I
> have to convert them too ?
> 
> *Sure... Just launch a reconstruct -r -V max...*

I just ran this command as cyrus user on my server (after reading the
manual page). Unfortunately, the "seen" databases in /var/lib/cyrus/user
are still reported by "file" as skiplist, and the "cyrus.cache",
"cyrus.header" and "cyrus.index" in the various (sub)mailboxes, are
still reported as simply "data".

It did create "cyrus.annotations" databases in each subfolder, though
(in twoskip format).

Also,I'm a bit worried. I did see in the logs lines that said:

repacking mailbox user.<user>.<folder>.<subfolder>

and

reconstructing user.<user>.<folder>.<subfolder> 

THOSE TWO LINES ARE PRETTY NORMAL... 

...but also some more worrying lines that said:

uniqueid clash with user.<user>.<folder>.<subfolder> for <some hex
value> - changing user.<user>.<folder>.<subfolder> 

I HAVEN'T EVER SEEN IT.... CAN'T SAY... I SHOULD TAKE A LOT AT THE CODE
FOR ANSWERING.....

Is it something I should worry about ?

Regarding the fact that they're still not in the twoskip format, should
I use cvt_cyrusdb instead ? That would be unfortunate, since I'll have
to create a script fed to the "find" command to mass-convert all
databases; plus, I still don't know what the input format (the "data"
that file talks about) is. 

RECONSTRUCT -R -V MAX SHOULD HANDLE ALL CONVERSIONS...

>> When I mill migrate, will I have to convert the databases through the
>> flat format and back, or can I blindly copy the whole contents of
>> /var/spool/cyrus and /var/lib/cyrus to the new server and expect it to
>> work out of the box ?
>> 
>> *When doing such a migration, it would be better to setup a
>> replication between the 2.4 and the new 2.5 in the hosting. You should
>> encrypt that communication. You could use the own cyrus encription for
>> replication or something like OpenVPN. Although it should work, I
>> wouldn't copy directly (with an rsync or scp) the files.*

Yes, both servers communicate through a VPN, but since both will have
the same Cyrus version, I thought I could just copy the files. Why is it
a bad idea ? 

IT'S NOT A HORRIBLE IDEA... BUT WITH THE REPLICATION YOU CREATE ALL
INDEXES AND SO FROM 0, WITH THE CORRECT VERSION, WITHOUT NEEDING
CONVERSIONS AND WITH THE PROPER VERSION AND, GIVES YOU CLEANER
DATABASES... FOR INSTANCE... AND YOU COULD EVEN COPY CONSTANTLY CONTENT
TO THE DATACENTER FOR JUST FAILING-OVER TO THE DATACENTER WHEN YOU ARE
READY... MOVING TO THE NEW SERVER IS CLEANER AND EASIER... THAT'S WHAT I
WOULD DO :) 

Regards, 

CHEERS! 

Links:
------
[1] http://www.sarenet.es
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20190214/075c7ef4/attachment.html>