Is skiplist dependant on byte order?

[Bron Gondwana]

On Mon, 16 Jun 2008 03:29:25 -0700, "Scott Likens" <damm at yazzy.org> said:
> I'm going to take a shot in the dark,
> 
> BIG Endian vs. Little Endian?

Skiplist has had quite a lot of care taken to use network order for
all values.  I don't _believe_ there are any issues.

> Unfortunately I do believe bdb databases do care if it was big or  
> little... and going from Sparc (BIG) to x86 (little)...
>
> Would not work very well :(
 
Yeah - they are pretty version and system specific.  I would always
dump BDB databases for a transfer.  Certainly crossing architectures.

> I am going to guess that a reconstruct may not be a bad idea, your  
> seen databases may or may not work, and pretty much guess that any and  
> all databases related to Cyrus will need to be re-worked... I'm sure  
> someone like Bron (fastmail.fm) might have something already whipped  
> up for this.

Seen databases are probably skiplists too.  You'll be hearing something
else exciting about skiplists from me in the very near future I suspect.
I've been spending the last few days examining them in great detail
again.  Woot.  Appears to be Linux specific though, so you don't need to
worry about this one!

No, I'm afraid I don't have a script already written for this situation,
it's something we've never had to do.  I would use cyr_dbtool for all
the dump and restores, because - well - I wrote it ;)  I know all the
corners.  I'm happy to have a look at any script you write and see if
you've missed anything.  Probably don't have the time to write one from
scratch.

> Seen should be in /var/imap/user
> ... I would check on sieve (it is compiled also) ... (/var/imap/sieve?)

Yeah, recompiling your sieve files doesn't sound like a bad idea at all.
 
> as well as /var/imap/db/*

Ouch.  We don't keep any BDB stuff across reboots thankfully, so we tend
to just ditch that on startup.

> then the /var/spool/imap/a/user/.. and you can more then likely just  
> do a reconstruct -rf and be fine...

That's a serious amount of IO.  All the index and cache files are
supposed to be endian-clean as well.  It's all htonl and ntohl everywhere.

> This is at least what I would do, I might have overstated how much  
> "fun" it will be, or under.

I really hope it's not that bad.  I've never done an endian transition, but
I have read the code an awful lot (points at the slashdot thing about open
source code not being understandable - at least it's _available_ - and I
understand the skiplist code pretty well now... *sigh*)

> On Jun 15, 2008, at 6:38 PM, Gary Mills wrote:

(there's more below)
 
> > I recently upgraded a murder front end server from Solaris 9 SPARC to
> > Solaris 10 x86 by copying the /imap directory.  I did dump the
> > mailboxes database before the copy.  It's a skiplist database.  I'm
> > running cyrus-imapd-2.3.8 on both systems.  As a test, I first checked
> > on the mailboxes database like this:
> >
> >    # su cyrus -c ksh
> >    # /usr/local/cyrus/bin/ctl_mboxlist -d | wc -l
> >           0

Do you have cyr_dbtool in that version?  I can't remember when it got
take upstream.  dbtool is nice because it dumps all the cyrus databases,
not just the mailboxes.db.

> > This message appeared in the log:
> >
> >    Jun 11 16:24:43 setup01 ctl_mboxlist[14082]: [ID 864961  
> > local6.crit] DBERROR: critical database situation

That sounds like BDB to me.  Are you running BDB mailboxes.db?  That would
certainly explain it.

> > After I reloaded it, I got the correct output:
> >
> >    # /usr/local/cyrus/bin/ctl_mboxlist -u < mailboxes.txt
> >    # /usr/local/cyrus/bin/ctl_mboxlist -d | wc -l
> >          77

Wow, tiny.  Actually, the hurt won't be so bad then even if you do
have to dump and reload every file.  Of course, it costs about the
same amount of time to write a script no matter how many values you
have to process (aside: I told a story at lunch the other day of
the time I spent about a month writing a really nice OCR and FAX
system for my previous job, and we got a grand total of 2 pages
of data faxed in for that project.  Scripts are sometimes overkill)

> > This is a test server with only a few mailboxes.  I'll upgrade the
> > production server later.

Ahh, so here comes the pain then.  Script time!

> > I'm assuming that skiplist is dependant on the machine's byte order,
> > and that a dump and reload is necessary in this case.

No, it really shouldn't matter.  One of the good things about the skiplist
design.  There are other bits that aren't so good - but the byte order
part is nice.

> > Are there any other databases that I should also dump and reload?  As
> > far as I can tell, the annotation_db, duplicate_db, and tlscache_db
> > are empty and can simply be removed.  Are there any others on a murder
> > front end that I've missed?  Where do they reside?

Yeah, we nuke all those on restart.  duplicate_db is the most interesting
of that lot - but not a giant concern.  It will cause vacation messages to
be repeated, and duplicate messageids to be delivered if it's gone - that's
about all.  For a once-off I wouldn't be at all concerned.

mailboxes.db really is the big one.  Anything else with berkeley named in it
that's either in your imapd.conf or defaulted that way in lib/imapoptions.

Bron.
-- 
  Bron Gondwana
  brong at fastmail.fm