Slipstream of #3553: Execute Cassandane functional tests against deployment

[Jeroen van Meeuwen (Kolab Systems)]

Quoting me and Greg from the slipstream of;

   https://bugzilla.cyrusimap.org/show_bug.cgi?id=3553#c3

Greg Banks wrote:
> Jeroen van Meeuwen wrote:
> > That said, I'm going to want to be able to have Cassandane run 
> against an
> > existing Murder deployment...
>
> Cassandane is designed to set up it's own Cyrus instances rather than 
> run
> against existing ones.  This was entirely deliberate and there's a 
> bunch of
> reasons for it, but I think that predictable and reproducible setups, 
> and the
> avoidance of state leakage between tests, are sufficient reasons.
>
> Cassandane test cases are by design capable of starting multiple 
> Cyrus
> instances and controlling their configuration files to connect the 
> instances in
> any way necessary; this is how we test replication scenarios.  It was 
> always my
> intention to add murder support.
>
> The tricky bit is "how do we convince Cassandane to go run test X 
> once for
> single instance and then once again for a murder"?

I think we could take a different approach. Excuse my French as I'm not 
a terminology guru, and apologies if the entire concept is crazy / way 
out of scope; I'm just thinking out loud.

> Because the alternative is
> to have two copies of every test that needs to be run against a 
> murder, which
> would be just nuts.  I'm not entirely sure what the best approach is 
> for this,
> but I'm betting that something like parameterised tests is the Right
> Way...except that Test::Unit doesn't seem to support those.

The tests that can be executed against a deployed environment are a 
type of tests usually qualified as functional tests, or at least a 
sub-set thereof, right?

If running those tests could be provided the configuration of a 
deployed instance of Cyrus IMAP(single-server?),

  - admin credentials,
  - sample user credentials, possibly a set of login names for the same 
user (uid, mail, alias, ...)
  - Cyrus IMAP server IP / DNS name (or 'localhost')

a variety of (functional) tests could be executed for a deployed 
instance rather then a snapshot of the codebase.

I would just execute this against a deployed frontend. The unit tests 
can be performed using a local checkout of the tag used to built the 
deployed package.

Later on perhaps such set of functional tests could be extended to 
perform murder specific stuff (MUPDATE capability would display part of 
the murder topology?) such as transfer, rename, setquota, reconstruct 
against frontend, and many, many other things.

-- 
Kind regards,

Jeroen van Meeuwen

-- 
Senior Engineer, Kolab Systems AG

e: vanmeeuwen at kolabsys.com
t: +44 144 340 9500
m: +44 74 2516 3817
w: http://www.kolabsys.com

pgp: 9342 BF08