Cyrus IMAPd 2.3.4 Released

[Robert Mueller]

> I agree that a test suite is something that we need.  I'll add this to
> our TODO list. As always, code contributions are always welcome.

See below and attached, a copy of what I sent april '04. It's not great, but 
it's a start...

Rob

----

Hi Ken & Rob

Given that there's currently no regression test for cyrus, I thought it
might be worth creating one. The basic idea being that the tests are kept up
to date, and any new features get a basic set of tests. Also anytime a bug
is discovered, a test is added that shows the bug, and left to make sure it
doesn't come back in the future. We're willing to fund the starting of this
project (geared towards the most recent patches for our needs), but hope CMU
will keep it up to date in the future as well. I'm not quite sure how any of
this would work (both funding and maintenance wise), but maybe you can make
some suggestions?

Anyway, I think there should be a generic imap component, and cyrus specific
component, in that it does some tests that rely on cyrus and/or it's
utilities (eg check location of spool directory and meta files, corrupt a
mailbox and run reconstruct, check quota and quota fixing, etc)

I think it's probably best to write the script in perl, and here's a couple
of modules that should help:
http://search.cpan.org/~robm/Mail-IMAPTalk-1.00/
http://search.cpan.org/~lhoward/Net-LMTP-0.02/

I tend to like creating two main types of tests:
1. A very structured set that test very specific set of actions. I tend to
make it so each 'batch' of tests is a function and is independent, and that
it leaves everything exactly how it was beforehand so you can easily run the
same test suite over and over, or a particular test function. eg

sub TestBasicFolder {
$Imap->create('user.test1')
  || die "Could not create user.test1";
my $ACL = $Imap->getacl('user.test1')
  || die "Could not get user.test1 ACL 1";
$Imap->setacl('user.test1', 'admin', 'lrswipcda')
  || die "Could not set user.test1 ACL";
$ACL = $Imap->getacl('user.test1')
  || die "Could not get user.test1 ACL 2";
... some more tests ...
$Imap->delete('user.test1')
  || die "Could not delete user.test1";
}

2. Ones that pick a bunch of random commands from a set and executes them,
somehow keeping internally what it expects the state to be and making sure
that it is. eg something like:

sub TestRandomFolderCreateDelete {
my @Folders;
for (1 .. 100000) {
  my $Rand = rand();
  # Create a random folder
  if ($Rand < 0.7) {
    my $Folder = "user." . RandomFolderName(int(rand(20)));
    $Imap->create()
      || die "Could not create folder $Folder: $@";
    push @Folders, $Folder;
  # Delete a random folder
  } elsif ($Rand < 0.9) {
    my $Folder = splice @Folders, int(rand(@Folders));, 1;
    $Imap->delete($Folder)
      || die "Could not delete folder $Folder: $@";
  # Check mailbox list
  } else {
    my %List = map { $_->[2] => 1 } $Imap->list('*', '*');
    for (@Folders) {
      delete $List{$_} || die "Folder $_ not found";
    }
    !%List || die "Not all folders found";
  }
}
for (@Folders) {
    $Imap->delete($_)
      || die "Could not delete folder $Folder: $@";
}
my @List = $Imap->list('*', '*');
!@List || die "Unexpectedly found folders";
}

I find tests like this quite important to find the more obscure bugs. Once
it does find a bug, I then find that I try differnet values of srand() at
the start (which then makes rand generate the same sequence each time the
program is run) until the bug is reproducible so I can track it down.

So basic things I can think of:
1. Login as 'administrator' user and create a bunch of user.* mailboxes and
sub-folders and listing
2. Test deleting some folders (eg test problem of deleting user.adminxxx
folder name) and listing
3. Set/get/myrights tests ACL's on the mailboxes (eg try read only, post
only, sharing other users folder, etc)
4. Set quotas on the mailboxes (different quota roots, etc)
5. Login as a user for one of the mailboxes. Test different ACL's set above
(eg try read only, post only, sharing other users folder, etc)
6. Appending messages to a mailbox
7. Reading messages (and seen flag)
8. Setting/getting other flags
9. Delivering email via LMTP (and of course + addressing)
10. LMTP new wildcard addressing
11. POP access to a mailbox
12. POP access to sub-folder
13. Group ACLs
14. Sieve actions

A couple of tricky things:
1. User logins. Since these are separate to the actual mailbox names, I'm
not sure how to easily handle this... maybe force usage of a saslauthd
daemon that always authorises all username/password combinations...
2. imapd.conf will need to be changed and then the server restarted from the
code

To get the ball rolling, I looked at some existing code I had, and started
the outlines of how I think things might work. I've attached the script,
which currently has 3 tests which I've just tested at home and verified that
it works.

Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cyrus_regression_test.pl
Type: application/octet-stream
Size: 46149 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20060526/610815b6/cyrus_regression_test-0001.obj