Build failed in Jenkins: cyrus-imapd-master #449

Greg Banks gnb at fastmail.fm
Mon Mar 12 20:54:44 EDT 2012



On Mon, Mar 12, 2012, at 10:10 PM, Bron Gondwana wrote:
> On Mon, Mar 12, 2012 at 05:03:06PM -0400, Jenkins wrote:
> > See <http://ci.cyrusimap.org/job/cyrus-imapd-master/449/changes>
> 
> No idea what actually went wrong there - I don't know how to read
> these stack dumps.  It's uglier than Java I tell you!
> 

Cassandane is run in -v (verbose) mode, which generates enormous amounts
of output, which is shoved aside into the file cass.errs, which you can
view from
the web interface thusly:

http://ci.cyrusimap.org/job/cyrus-imapd-master/ws/cassandane/cass.errs/*view*/

The last few lines are

...
=====> Instance[688] start main instance for test
test_cve_2011_3208_list_active: basedir /var/tmp/cass/21021782
=====> Instance[1063] Running: "/usr/sbin/saslpasswd2" "-f"
"/var/tmp/cass/21021782/conf/sasldb2" "-c" "-p" "admin"
=====> Instance[526] _start_master: logging to
/var/tmp/cass/21021782/conf/master.log
=====> Instance[1063] Running:
"/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/master"
"-C" "/var/tmp/cass/21021782/conf/imapd.conf" "-l" "255" "-p"
"/var/tmp/cass/21021782/run/cyrus.pid" "-d" "-M"
"/var/tmp/cass/21021782/conf/cyrus.conf" "-L"
"/var/tmp/cass/21021782/conf/master.log"
=====> Instance[535] _start_master: waiting for PID file
=====> Instance[538] _start_master: PID file present and correct
=====> Instance[544] _start_master: PID waiting for services
=====> Service[269] is_listening: service nntp is listening on
127.0.0.1:9101
=====> Instance[555] _start_master: all services listening
=====> Instance[652] create user cassandane
=====> Instance[1063] Running: "/usr/sbin/saslpasswd2" "-f"
"/var/tmp/cass/21021782/conf/sasldb2" "-c" "-p" "cassandane"
News::NNTPClient::SOCK1 connecting to localhost.localdomain:9101
News::NNTPClient unexpected EOF on News::NNTPClient::SOCK1
 at Cassandane/Cyrus/Nntp.pm line 66
News::NNTPClient::SOCK1 command: AUTHINFO USER cassandane

That looks like the NNTPClient Perl code failed because the nntpd it was
talking
to dropped the connection, probably a result of terminating prematurely.

Also, every other test seems to follow a pattern like

=====> Instance[688] start main instance for test
test_set_system_flag_deliver: basedir /var/tmp/cass/2102173
=====> Instance[1063] Running: "/usr/sbin/saslpasswd2" "-f"
"/var/tmp/cass/2102173/conf/sasldb2" "-c" "-p" "admin"
=====> Instance[526] _start_master: logging to
/var/tmp/cass/2102173/conf/master.log
=====> Instance[1063] Running:
"/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/master"
"-C" "/var/tmp/cass/2102173/conf/imapd.conf" "-l" "255" "-p"
"/var/tmp/cass/2102173/run/cyrus.pid" "-d" "-M"
"/var/tmp/cass/2102173/conf/cyrus.conf" "-L"
"/var/tmp/cass/2102173/conf/master.log"
=====> Instance[535] _start_master: waiting for PID file
=====> Instance[538] _start_master: PID file present and correct
=====> Instance[544] _start_master: PID waiting for services
WARNING: Port 9102 never freed.  Allocated  at Cassandane/Service.pm
line 100
	Cassandane::Service::port('Cassandane::IMAPService=HASH(0xa7b84c0)')
	called at Cassandane/Service.pm line 142
	[...Perl stack trace...]

This is missing the bulk of the test xlog() calls, and seems to have
failed early in instance startup.

Let's have a look to see if there's any cores.  Normally, tests will be
failed if we get
to the end of them and Cassandane notices there are cores in the
instance directories,
but in this case because of the "Port 9102 never freed" warning I
suspect instances
are not being shut down properly.

root at hudson-01 2004 cd /var/tmp/cass
root at hudson-01 2004 find * -name core\* -type f | while read f ; do echo
-n "$f: " ; gdb -q --batch -c $f | sed -ne 's|Core was generated by ||p'
; done
2102171/conf/cores/core.20475:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021710/conf/cores/core.20536:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021711/conf/cores/core.20543:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021712/conf/cores/core.20550:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021713/conf/cores/core.20557:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021714/conf/cores/core.20564:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021715/conf/cores/core.20570:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021716/conf/cores/core.20577:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021717/conf/cores/core.20583:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021718/conf/cores/core.20589:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021719/conf/cores/core.20594:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
2102172/conf/cores/core.20483:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021720/conf/cores/core.20601:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021721/conf/cores/core.20606:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021722/conf/cores/core.20612:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021723/conf/cores/core.20618:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021724/conf/cores/core.20624:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021725/conf/cores/core.20630:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021726/conf/cores/core.20636:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021727/conf/cores/core.20642:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021728/conf/cores/core.20648:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021729/conf/cores/core.20654:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
2102173/conf/cores/core.20489:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
2102174/conf/cores/core.20497:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021746/conf/cores/core.21098:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021747/conf/cores/core.21104:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021748/conf/cores/core.21111:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021749/conf/cores/core.21117:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
2102175/conf/cores/core.20502:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021750/conf/cores/core.21122:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021751/conf/cores/core.21128:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021752/conf/cores/core.21134:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021753/conf/cores/core.21140:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021754/conf/cores/core.21146:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021755/conf/cores/core.21152:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021756/conf/cores/core.21158:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021757/conf/cores/core.21164:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021758/conf/cores/core.21170:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021759/conf/cores/core.21176:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
2102176/conf/cores/core.20508:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021760/conf/cores/core.21182:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021761/conf/cores/core.21188:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021762/conf/cores/core.21194:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021763/conf/cores/core.21200:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021764/conf/cores/core.21206:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021765/conf/cores/core.21212:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021766/conf/cores/core.21218:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021767/conf/cores/core.21224:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021768/conf/cores/core.21230:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021769/conf/cores/core.21236:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
2102177/conf/cores/core.20516:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021770/conf/cores/core.21242:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021771/conf/cores/core.21248:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021772/conf/cores/core.21254:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021773/conf/cores/core.21260:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021774/conf/cores/core.21266:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021775/conf/cores/core.21272:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021776/conf/cores/core.21278:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021777/conf/cores/core.21284:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021778/conf/cores/core.21290:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021779/conf/cores/core.21296:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
2102178/conf/cores/core.20522:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021780/conf/cores/core.21302:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021781/conf/cores/core.21310:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.
21021782/conf/cores/core.21318:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/nntpd
-C'.
21021782/conf/cores/core.21320:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/nntpd
-C'.
21021782/conf/cores/core.21322:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/nntpd
-C'.
21021782/conf/cores/core.21319:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/nntpd
-C'.
21021782/conf/cores/core.21321:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/nntpd
-C'.
2102179/conf/cores/core.20529:
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/ctl_cyrus'.

My, that's a lot of cores, most from ctl_cyrus but some from nntpd.

The `ctl_cyrus' cores are actually from ctl_cyrusdb and are doing

Program terminated with signal 6, Aborted.
#0  0x0000003665830265 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x0000003665830265 in raise () from /lib64/libc.so.6
#1  0x0000003665831d10 in abort () from /lib64/libc.so.6
#2  0x00000036658296e6 in __assert_fail () from /lib64/libc.so.6
#3  0x00000000004a4b1b in config_getenum (opt=IMAPOPT_VIRTDOMAINS)
    at libconfig.c:138
#4  0x00000000004a5c85 in config_read (
    alt_config=0x7fff871b47b5 "/var/tmp/cass/21021775/conf/imapd.conf")
    at libconfig.c:386
#5  0x000000000041df80 in cyrus_init (
    alt_config=0x7fff871b47b5 "/var/tmp/cass/21021775/conf/imapd.conf", 
    ident=0x4acbf2 "ctl_cyrusdb", flags=0) at global.c:152
#6  0x0000000000406ada in main (argc=4, argv=0x7fff871b4268)
    at ctl_cyrusdb.c:272

The nntpd cores are doing this

root at hudson-01 2005 gdb -q
/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/nntpd
21021782/conf/cores/core.21321
.Core was generated by
`/var/lib/jenkins/jobs/cyrus-imapd-master/workspace/inst/usr/cyrus/bin/nntpd
-C'.
Program terminated with signal 6, Aborted.
#0  0x0000003665830265 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x0000003665830265 in raise () from /lib64/libc.so.6
#1  0x0000003665831d10 in abort () from /lib64/libc.so.6
#2  0x00000036658296e6 in __assert_fail () from /lib64/libc.so.6
#3  0x00000000004dd7ff in config_getenum (opt=IMAPOPT_VIRTDOMAINS)
    at libconfig.c:138
#4  0x00000000004de969 in config_read (
    alt_config=0x7fff726407a0 "/var/tmp/cass/21021782/conf/imapd.conf")
    at libconfig.c:386
#5  0x0000000000471170 in cyrus_init (
    alt_config=0x7fff726407a0 "/var/tmp/cass/21021782/conf/imapd.conf", 
    ident=0x129f2530 "nntp", flags=0) at global.c:152
#6  0x0000000000438cf6 in main (argc=3, argv=0x7fff7263eed8, 
    envp=0x7fff7263eef8) at service.c:337
..

(gdb) up 3
#3  0x00000000004dd7ff in config_getenum (opt=IMAPOPT_VIRTDOMAINS)
    at libconfig.c:138
138         assert(imapopts[opt].t == OPT_ENUM);

(gdb) p opt
$1 = IMAPOPT_VIRTDOMAINS

(gdb) p imapopts[opt]
$2 = {opt = IMAPOPT_VIRTDOMAINS, optname = 0x5efbbf "unixhierarchysep", 
  seen = 0, t = OPT_SWITCH, val = {s = 0x0, i = 0, b = 0, e =
  IMAP_ENUM_ZERO, 
    x = 0}, def = {s = 0x0, i = 0, b = 0, e = IMAP_ENUM_ZERO, x = 0}, 
  enum_options = {{name = 0x0, val = IMAP_ENUM_ZERO} <repeats 22
  times>}}

Woops.  It looks like there's at least one commit

> Commit a95d47894f08539f794e020f334d8327015f13ef by brong
> Bug #3642 - add 'specialusealways' switch to imapd.conf

that added a new imapd.conf option, but the changes didn't flow through
to the
object code properly.  Clearly there's a Makefile bug here.  Aha:

-rw-r--r-- 1 tomcat tomcat 21063 Mar 12 05:01 libconfig.c
-rw-rw-r-- 1 tomcat tomcat 63616 Mar 12 05:01 libconfig.o
-rw-r--r-- 1 tomcat tomcat   62904 Mar 12 17:01 imapoptions
-rw-rw-r-- 1 tomcat tomcat  132440 Mar 12 17:01 imapopts.o
-rw-rw-r-- 1 tomcat tomcat   10129 Mar 12 17:01 imapopts.h
-rw-rw-r-- 1 tomcat tomcat   55749 Mar 12 17:01 imapopts.c

So on this last run, make rebuilt:

imapoptions -> imapopts.c, imapopts.h
imapopts.c -> imapopts.o

but it did NOT build

libconfig,c -> libconfig.o

presumably because it lack the dependency information to know that it
should, as libconfig.a #includes imapopts.h.

Stupid dependency crap...we need automake!!

Anyway, the easiest workaround is probably to do a "make clean" in the
autobuild.sh script
to remove all the .o files.


-- 
Greg.


More information about the Cyrus-devel mailing list