My cassandane tests break. Yours too?

[ellie timoney]

On Wed, Dec 2, 2015, at 09:15 PM, rost via Cyrus-devel wrote:
> On Wed, Dec 2, 2015, at 05:21 AM, Bron Gondwana via Cyrus-devel wrote:
> > I've added a copy of Cyrus::DList directly to the repo for now.  I'll get
> > it (and all our other Cyrus:: modules) into the Cyrus codebase directly I
> > think.
> >
> > On Wed, Dec 2, 2015, at 10:34, ellie timoney via Cyrus-devel wrote:
> > > I had problems with that commit the other week.  The problem turned out
> > > to be the lack of Cyrus::DList (needed by AnyEvent::Handle to process
> > > part of the work -- see line 1666 of Cassandane/Instance.pm).  
> 
> Cool. Tests work now. Thank you both.

They're weird for me (cass master, cyrus master)

If I run with the -j switch (multiple parallel jobs), then I get a lot
of syslog noise spammed onto my terminals(!), but the tests eventually
all report OK.  The syslog noise is a mixture of 

* perl/cassandane backtraces squashed into single lines (starting with
"cannot write to pipe: bad file descriptor [...]"
* perl/cassandane backtraces dumped as base64 (when decoded, looks like
process receiving SIGTERM while in the middle of AnyEvent work)

If I run without the -j switch (single process) then the first test
(ACL.delete) takes several minutes to run, reports OK, but then
Cassandane crashes out in annotate_from_file because it can't find the
/tmp file anymore

If I run without -j and with --log-directory (which also stops it trying
to clean up its temp files) it starts looking really weird.  It takes
several minutes for ACL.delete to report success (but it does, and its
log file looks unremarkable), and then it carries on to eventually
report success for ACL.move (ditto re log file), but looking in the log
file directory there is already log output appearing for many of the
subsequent tests that haven't yet reported anything.  I see one (very
busy) "cassandane notifyd" process at a time in ps output -- stracing
and waiting till it finishes seems to show that "current" test reports
success when its cassandane notifyd process eventually finishes, and
then the next test's notifyd starts/runs/finishes after that...

It's just (minutes later) reported success for ACL.reconstruct.  Syslog
hasn't yet spammed anything to my terminals, but going by the output
it's only completed three tests so far, so that's not surprising.  I was
hoping that running in single thread mode would show me which tests are
leading to the syslog'd backtraces, so that I could dig in further.  But
at this rate running it single threaded will take the rest of the day.