Build failed in Jenkins: cyrus-imapd-master #1069

Greg Banks gnb at fastmail.fm
Tue Feb 12 21:08:19 EST 2013 


On Mon, Jan 7, 2013, at 09:13 AM, Jenkins wrote:
> See <http://ci.cyrusimap.org/job/cyrus-imapd-master/1069/>
> 
> ------------------------------------------
> [...truncated 4455 lines...]
> + '[' -d .git ']'
> ++ git ls-files
> ++ wc -l
> + nfiles=99
> + '[' 99 -gt 0 ']'
> + git ls-files -o
> broken.log
> cass.errs
> cassandane.ini
> coverage.xml
> e.fpl
> errs
> errs2
> find-process-leak.sh
> reports.old/TEST-Cassandane.Cyrus.Annotator.xml
> reports.old/TEST-Cassandane.Cyrus.Bug3072.xml
> reports.old/TEST-Cassandane.Cyrus.Bug3463.xml
> reports.old/TEST-Cassandane.Cyrus.Bug3470.xml
> reports.old/TEST-Cassandane.Cyrus.Bug3649.xml
> reports.old/TEST-Cassandane.Cyrus.Delete.xml
> reports.old/TEST-Cassandane.Cyrus.Delivery.xml
> reports.old/TEST-Cassandane.Cyrus.Fetch.xml
> reports.old/TEST-Cassandane.Cyrus.Flags.xml
> reports.old/TEST-Cassandane.Cyrus.Idle.xml
> reports.old/TEST-Cassandane.Cyrus.Info.xml
> reports.old/TEST-Cassandane.Cyrus.Lsub.xml
> reports.old/TEST-Cassandane.Cyrus.Master.xml
> reports.old/TEST-Cassandane.Cyrus.Metadata.xml
> reports.old/TEST-Cassandane.Cyrus.Nntp.xml
> reports.old/TEST-Cassandane.Cyrus.Pop3.xml
> reports.old/TEST-Cassandane.Cyrus.Quota.xml
> reports.old/TEST-Cassandane.Cyrus.Rename.xml
> reports.old/TEST-Cassandane.Cyrus.Replication.xml
> reports.old/TEST-Cassandane.Cyrus.Search.xml
> reports.old/TEST-Cassandane.Cyrus.Sieve.xml
> reports.old/TEST-Cassandane.Cyrus.Simple.xml
> reports.old/TEST-Cassandane.Test.Address.xml
> reports.old/TEST-Cassandane.Test.Cassini.xml
> reports.old/TEST-Cassandane.Test.Clone.xml
> reports.old/TEST-Cassandane.Test.Config.xml
> reports.old/TEST-Cassandane.Test.DateTime.xml
> reports.old/TEST-Cassandane.Test.Message.xml
> reports.old/TEST-Cassandane.Test.MessageStoreFactory.xml
> reports.old/TEST-Cassandane.Test.Metronome.xml
> reports.old/TEST-Cassandane.Test.Sample.xml
> reports/TEST-Cassandane.Cyrus.Annotator.xml
> reports/TEST-Cassandane.Cyrus.Bug3072.xml
> reports/TEST-Cassandane.Cyrus.Bug3463.xml
> reports/TEST-Cassandane.Cyrus.Bug3470.xml
> reports/TEST-Cassandane.Cyrus.Bug3649.xml
> reports/TEST-Cassandane.Cyrus.Delete.xml
> reports/TEST-Cassandane.Cyrus.Delivery.xml
> reports/TEST-Cassandane.Cyrus.Fetch.xml
> reports/TEST-Cassandane.Cyrus.Flags.xml
> reports/TEST-Cassandane.Cyrus.Idle.xml
> reports/TEST-Cassandane.Cyrus.Info.xml
> reports/TEST-Cassandane.Cyrus.Lsub.xml
> reports/TEST-Cassandane.Cyrus.Master.xml
> reports/TEST-Cassandane.Cyrus.Metadata.xml
> reports/TEST-Cassandane.Cyrus.Nntp.xml
> reports/TEST-Cassandane.Cyrus.Pop3.xml
> reports/TEST-Cassandane.Cyrus.Quota.xml
> reports/TEST-Cassandane.Cyrus.Rename.xml
> reports/TEST-Cassandane.Cyrus.Replication.xml
> reports/TEST-Cassandane.Cyrus.Search.xml
> reports/TEST-Cassandane.Cyrus.Sieve.xml
> reports/TEST-Cassandane.Cyrus.Simple.xml
> reports/TEST-Cassandane.Test.Address.xml
> reports/TEST-Cassandane.Test.Cassini.xml
> reports/TEST-Cassandane.Test.Clone.xml
> reports/TEST-Cassandane.Test.Config.xml
> reports/TEST-Cassandane.Test.DateTime.xml
> reports/TEST-Cassandane.Test.Message.xml
> reports/TEST-Cassandane.Test.MessageStoreFactory.xml
> reports/TEST-Cassandane.Test.Metronome.xml
> reports/TEST-Cassandane.Test.Sample.xml
> utils/gdbtramp
> utils/gdbtramp.o
> utils/lemming
> utils/lemming.o
> working.log
> + git status
> # On branch master
> # Untracked files:
> #   (use "git add <file>..." to include in what will be committed)
> #
> #       broken.log
> #       coverage.xml
> #       e.fpl
> #       errs
> #       errs2
> #       find-process-leak.sh
> #       working.log
> nothing added to commit but untracked files present (use "git add" to
> track)
> + make
> make[1]: Entering directory
> `<http://ci.cyrusimap.org/job/cyrus-imapd-master/ws/cassandane/utils'>
> make[1]: Nothing to be done for `all'.
> make[1]: Leaving directory
> `<http://ci.cyrusimap.org/job/cyrus-imapd-master/ws/cassandane/utils'>
> testrunner.pl syntax OK
> Cassandane/ThreadedGenerator.pm syntax OK
> Cassandane/Config.pm syntax OK
> Cassandane/MasterEntry.pm syntax OK
> Cassandane/Cyrus/Quota.pm syntax OK
> Cassandane/Cyrus/Delivery.pm syntax OK
> Cassandane/Cyrus/Conversations.pm syntax OK
> Cassandane/Cyrus/Rename.pm syntax OK
> Cassandane/Cyrus/Bug3463.pm syntax OK
> Cassandane/Cyrus/Fetch.pm syntax OK
> Cassandane/Cyrus/Bug3470.pm syntax OK
> Cassandane/Cyrus/Delete.pm syntax OK
> Cassandane/Cyrus/Pop3.pm syntax OK
> Cassandane/Cyrus/Metadata.pm syntax OK
> Cassandane/Cyrus/Info.pm syntax OK
> Cassandane/Cyrus/Nntp.pm syntax OK
> Cassandane/Cyrus/Bug3649.pm syntax OK
> Cassandane/Cyrus/TestCase.pm syntax OK
> Cassandane/Cyrus/Sieve.pm syntax OK
> Cassandane/Cyrus/Replication.pm syntax OK
> Cassandane/Cyrus/Master.pm syntax OK
> Cassandane/Cyrus/Flags.pm syntax OK
> Cassandane/Cyrus/Lsub.pm syntax OK
> Cassandane/Cyrus/Idle.pm syntax OK
> Cassandane/Cyrus/Bug3072.pm syntax OK
> Cassandane/Cyrus/Search.pm syntax OK
> Cassandane/Cyrus/Annotator.pm syntax OK
> Cassandane/Cyrus/Simple.pm syntax OK
> Cassandane/Test/Config.pm syntax OK
> Cassandane/Test/Sample.pm syntax OK
> Cassandane/Test/Message.pm syntax OK
> Cassandane/Test/DateTime.pm syntax OK
> Cassandane/Test/MessageStoreFactory.pm syntax OK
> Cassandane/Test/Metronome.pm syntax OK
> Cassandane/Test/Clone.pm syntax OK
> Cassandane/Test/Address.pm syntax OK
> Cassandane/Test/Cassini.pm syntax OK
> Cassandane/Instance.pm syntax OK
> Cassandane/Message.pm syntax OK
> Cassandane/POP3MessageStore.pm syntax OK
> Cassandane/SequenceGenerator.pm syntax OK
> Cassandane/PortManager.pm syntax OK
> Cassandane/IMAPService.pm syntax OK
> Cassandane/MessageStoreFactory.pm syntax OK
> Cassandane/IMAPMessageStore.pm syntax OK
> Cassandane/MaildirMessageStore.pm syntax OK
> Cassandane/Unit/SyslogListener.pm syntax OK
> Cassandane/Unit/RunnerPretty.pm syntax OK
> Cassandane/Unit/RunnerXML.pm syntax OK
> Cassandane/Unit/TestCase.pm syntax OK
> Cassandane/Unit/TestPlan.pm syntax OK
> Cassandane/Unit/Runner.pm syntax OK
> Cassandane/MessageStore.pm syntax OK
> Cassandane/Generator.pm syntax OK
> Cassandane/ServiceFactory.pm syntax OK
> Cassandane/Address.pm syntax OK
> Cassandane/MboxMessageStore.pm syntax OK
> Cassandane/Service.pm syntax OK
> Cassandane/Util/SHA.pm syntax OK
> Cassandane/Util/Sample.pm syntax OK
> Cassandane/Util/Wait.pm syntax OK
> Cassandane/Util/DateTime.pm syntax OK
> Cassandane/Util/Words.pm syntax OK
> Cassandane/Util/Setup.pm syntax OK
> Cassandane/Util/Metronome.pm syntax OK
> Cassandane/Util/Log.pm syntax OK
> Cassandane/Util/NetString.pm syntax OK
> Cassandane/Util/Socket.pm syntax OK
> Cassandane/MasterEvent.pm syntax OK
> Cassandane/Cassini.pm syntax OK
> Cassandane/MasterStart.pm syntax OK
> + sed -e 's|^##destdir =.*$|destdir =
> <http://ci.cyrusimap.org/job/cyrus-imapd-master/ws/inst|'> -e
> 's|^##pwcheck = .*$|pwcheck = sasldb|'
> + rm -rf reports.old
> + mv -f reports reports.old
> + mkdir -m 0777 reports
> + ./testrunner.pl --cleanup -f xml -v
> + exitcode=1
> + '[' -x jenkins-xml-summary.pl ']'
> + ./jenkins-xml-summary.pl
> --build-url=http://ci.cyrusimap.org/job/cyrus-imapd-master/1069/
> Test failures and errors summary
> ================================
> 
> Cassandane::Cyrus::Nntp.cve_2011_3208_newnews
>     http://ci.cyrusimap.org/job/cyrus-imapd-master/1069//testReport/%28root%29/Cassandane__Cyrus__Nntp/test_cve_2011_3208_newnews/

Error Message

Perl exception: Cannot shut down master pid 28883     

Stacktrace

test_cve_2011_3208_newnews(Cassandane::Cyrus::Nntp)
     Perl exception: Cannot shut down master pid 28883
     at Cassandane/Instance.pm line 831
    	Cassandane::Instance::stop('Cassandane::Instance=HASH(0x18d313c0)')
    	called at Cassandane/Cyrus/TestCase.pm line 267
    	Cassandane::Cyrus::TestCase::tear_down('Cassandane::Cyrus::Nntp=HASH(0x18490550)')
    	called at Cassandane/Cyrus/Nntp.pm line 83
    	Cassandane::Cyrus::Nntp::tear_down('Cassandane::Cyrus::Nntp=HASH(0x18490550)')
    	called at /usr/lib/perl5/vendor_perl/5.8.8/Test/Unit/TestCase.pm
    	line 65

What happened here is most likely:

 * the Cassandane Perl code initiated shutdown by sending SIGTERM to the
 master process
 * the Perl code waits, for up to 20 sec
 * master resent the signal to it's process group
 * master waits (with no timeout) to be notified of the graceful exit of
 its children via SIGCHLD
 * but a child (in this case, probably the imapd that was used to issue
 admin commands to set up the instance at the start of the test) failed
 to notice the SIGTERM and never died
 * eventually the Perl code wakes up and notices that the master process
 has not exited
 * the Perl code wreaks terrible vengeance with SIGILL or SIGKILL and
 reports the test failed.

I've recently fixed several Cyrus bugs which caused this class of
failure mode, commits

59b181c Also use signals_select() waiting for connection
10c21a1 Also use signals_select() in prot_*() functions
76f5e8e idle: close SIGTERM/select race

The rest of these were spurious cascading failures like

Error Message

Perl exception: Some process is already listening on 127.0.0.1:9107     

which should be solved by Cassandane commit

468fbd9 Kill stray processes on test exit


-- 
Greg.

More information about the Cyrus-devel mailing list