Upgrade from 2.1 to 2.2 issue...

[Rob Mueller]

Hi

I'm just testing out upgrading from cyrus 2.1.9 to cyrus 2.2.3. So far
everything seems to work pretty much fine, except for one thing. On some
existing folders, when you try to 'expunge' them, imapd segfaults. It's not
all folders, only some folders, though it is reproduceable for the "broken"
folders.

Now running a reconstruct on these folders fixes the problem, though I'm
keen not to have to do this as part of the upgrade process (finding the
folders is a pain, and it might take a while to run the reconstruct). To get
an idea of the spread of the problem...

[root at server3 user]# find . -type d | wc -l
   9001
[root at server3 user]# find . -name "core.*" | sed 's/core.*/core/' | uniq |
wc -l
     29

So after running on stress test (which uses all the folders) out of 9001
folders on our test server, 29 are affected.

Some more details:

[root imap]# find . -name "core.*"
./i/user/*username*/core.13280

There's a core dump for this user, so we know it's repeatable.

[root *user*]# telnet localhost 143
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
* OK ... Cyrus IMAP4 v2.2.3 server ready
. login *username* password
. OK User logged in
. select inbox
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]
* 13 EXISTS
* 0 RECENT
* OK [UNSEEN 3]
* OK [UIDVALIDITY 1058258370]
* OK [UIDNEXT 79]
. OK [READ-WRITE] Completed
. fetch 1:* flags
* 1 FETCH (FLAGS (\Seen))
* 2 FETCH (FLAGS (\Deleted \Seen))
* 3 FETCH (FLAGS ())
* 4 FETCH (FLAGS (\Seen))
* 5 FETCH (FLAGS (\Seen))
* 6 FETCH (FLAGS (\Seen))
* 7 FETCH (FLAGS (\Seen))
* 8 FETCH (FLAGS (\Seen))
* 9 FETCH (FLAGS (\Seen))
* 10 FETCH (FLAGS (\Seen))
* 11 FETCH (FLAGS (\Deleted))
* 12 FETCH (FLAGS ())
* 13 FETCH (FLAGS ())
. OK Completed (0.000 sec)

Meanwhile in another window...

[root at server3 imapstresstest308]# gdb /usr/cyrus/bin/imapd 5029
...
Attaching to program: /usr/cyrus/bin/imapd, process 5029
...
(gdb) c
Continuing.

Back in our IMAP session:

. expunge

And back to gdb... Ooops....


Program received signal SIGSEGV, Segmentation fault.
0x08074b41 in mailbox_expunge (mailbox=0x81349a0, iscurrentdir=1,
decideproc=0, deciderock=0x0) at mailbox.c:2018
2018                    cacheitem = CACHE_ITEM_NEXT(cacheitem);
(gdb) bt
#0  0x08074b41 in mailbox_expunge (mailbox=0x81349a0, iscurrentdir=1,
decideproc=0, deciderock=0x0) at mailbox.c:2018
#1  0x08056e2d in cmd_expunge (tag=0x814e528 ".", sequence=0x0) at
imapd.c:3434
#2  0x0804fa18 in cmdloop () at imapd.c:915
#3  0x0804e828 in service_main (argc=1, argv=0x8143e10, envp=0xbfffe88c) at
imapd.c:570
#4  0x0804d1a7 in main (argc=1, argv=0xbfffe884, envp=0xbfffe88c) at
service.c:546
#5  0x42017499 in __libc_start_main () from /lib/i686/libc.so.6
(gdb) p cacheitem
$1 = 0x59da4884 <Address 0x59da4884 out of bounds>
(gdb) l 2016,2019
2016                cacheitembegin = cacheitem = mailbox->cache_base +
cache_offset;
2017                for(cache_ent = 0; cache_ent < NUM_CACHE_FIELDS;
cache_ent++) {
2018                    cacheitem = CACHE_ITEM_NEXT(cacheitem);
2019                }
(gdb) p mailbox->cache_base
$2 = 0x41456000 "A"
(gdb) p cache_offset
$3 = 4294961280
(gdb) p cacheitembegin
$4 = 0x41454880
"\030\224ÿÿ\211EØ\203Ä\020\205À\017\204\226þÿÿ\203ì\fWè±\223ÿÿ\211EÔ\203Ä\02
0\205À\017\204\177þÿÿ\215´&"
(gdb) p cache_ent
$5 = 1
(gdb) p *mailbox
$6 = {header_fd = 11, index_fd = 12, cache_fd = 13,
  header_base = 0x4129e000 "¡\002\213\rCyrus mailbox header\n\"The best
thing about this system was that it had lots of goals.\"\n\t--Jim Morris on
Andrew\nuser.*username*\t6da7138e3f13bdc2\n\n*username*\tlrswipcda\tadmin\tl
rswipcda\t"...,
  header_len = 210, index_base = 0x41445000 "A",
  index_len = 16384, cache_base = 0x41456000 "A",
  cache_len = 16384, cache_size = 5164,
  header_lock_count = 1, index_lock_count = 1,
  seen_lock_count = 0, pop_lock_count = 1,
  header_ino = 4328652, index_mtime = 1074665767,
  index_ino = 4388307, index_size = 856,
  name = 0x814ded8 "user.*username*",
  path = 0x814def8 "/var/spool/imap/i/user/*username*",
  acl = 0x8155508 "*username*\tlrswipcda\tadmin\tlrswipcda\tanyone\tp\t",
  myrights = 511, uniqueid = 0x81556e8 "6da7138e3f13bdc2",
  flagname = {0x0 <repeats 128 times>},
  generation_no = 1090519040, format = 0,
  minor_version = 6, start_offset = 76,
  record_size = 60, exists = 13,
  last_appenddate = 1074665664, last_uid = 78,
  quota_mailbox_used = 2761812, pop3_last_login = 0,
  uidvalidity = 1058258370, deleted = 2,
  answered = 0, flagged = 0, dirty = 0,
  pop3_new_uidl = 1, leaked_cache_records = 1967064584,
  spares = {0, 0}, quota = {fd = 15, lock_count = 0,
    root = 0x8155548 "user.*username*",
  used = 10693530, limit = 50000}}

If you need any more information, let me know. I can reproduce it, and I can
probably send you the cyrus.* files as well if it would help.

Rob