How I set up my test/development environment

Tue Sep 15 20:58:46 EDT 2015

So I promised I would give a guide to what _I_ do :)  Unfortunately this is delayed a bit because some genius left his laptop at work last night.

First things - I'm on Ubuntu/Debian.  Most recent version usually, and I'll fix Cyrus to build there if it doesn't already.

Next we need some dependencies.  I use the packages as documented in the various READMEs - it won't compile if you don't have them.  They're all installed from apt-get, and I don't even think about them.

I have a bunch of software downloaded from git repositories:

~/src/cyrus-imapd  == ssh://git@git.cyrus.foundation/diffusion/I/cyrus-imapd.git (git)
~/src/cassandane == ssh://git@git.cyrus.foundation/diffusion/C/cassandane.git (git)
~/src/cyrusdevtools == ssh://git@github.com/brong/cyrusdevtools.git (git)

~/src/imaptest/dovecot-2.2 == http://hg.dovecot.org/dovecot-2.2/ (hg)
~/src/imaptest/imaptest == http://hg.dovecot.org/imaptest/ (hg)

~/src/cyruslibs/icu/release-55-1 == http://source.icu-project.org/repos/icu/icu/tags/release-55-1/ (svn)
~/src/cyruslibs/libical == https://github.com/libical/libical (git)
~/src/cyruslibs/opendkim == git://git.code.sf.net/p/opendkim/git (git)

I have aliases to cd directly into some of these:

alias cdt='cd /home/brong/src/cyrusdevtools'
alias ce='cd /home/brong/src/cyrus-imapd'
alias it='cd /home/brong/src/imaptest/imaptest'

I build libical, libicu and opendkim from source and install them into /usr/local/cyruslibs-fastmail-v1/

libicu:
% cd /home/brong/src/cyruslibs/icu/release-55-1/source
% ./configure --prefix=/usr/local/cyruslibs-fastmail-v1
% make
% sudo make install

opendkim:
% cd /home/brong/src/cyruslibs/opendkim
% autoreconf -v -i
%  ./configure --prefix=/usr/local/cyruslibs-fastmail-v1
% make
% sudo make install

libical:
% cd /home/brong/src/cyruslibs/libical
% mkdir build
% cd build
% cmake -DCMAKE_INSTALL_PREFIX=/usr/local/cyruslibs-fastmail-v1 ..
% make
% sudo make install

We then need to have these libs in the path for all Cyrus builds...  so this goes into .bashrc:

export LDFLAGS="-L/usr/local/cyruslibs-fastmail-v1/lib/x86_64-linux-gnu -L/usr/local/cyruslibs-fastmail-v1/lib -Wl,-rpath,/usr/local/cyruslibs-fastmail-v1/lib/x86_64-linux-gnu -Wl,-rpath,/usr/local/cyruslibs-fastmail-v1/lib"
export PKG_CONFIG_PATH=/usr/local/cyruslibs-fastmail-v1/lib/x86_64-linux-gnu/pkgconfig:/usr/local/cyruslibs-fastmail-v1/lib/pkgconfig

And finally we're ready to start building Cyrus:

OK, so Cyrus builds.  I have an alias 'cbcalse'.  It's a little dangerous, you MUST commit all new files to git before running it, or it will clean them up.

alias cbcalse='cd /home/brong/src/cyrus-imapd; make clean; git clean -f -x -d; autoreconf -v -i; CFLAGS="-g -fPIC -W -Wall -Wextra -Werror" ./configure --prefix=/usr/cyrus --with-cyrus-prefix=/usr/cyrus --enable-autocreate --enable-http --enable-unit-tests --enable-replication --enable-nntp --enable-murder --enable-idled --enable-xapian && make sieve/sieve-lex.c && perl -p -i -e "s/int yyl;/yy_size_t yyl;/" sieve/sieve-lex.c && make -j16 && make -j16 check && sudo make install && sudo cp tools/mkimap /usr/cyrus/bin/mkimap'

Let's look at this in detail.

First it changes into the directory, runs 'make clean' and 'git clean -f -x -d', which nukes anything that's a build product.  It's like doing a brand new checkout.  This is the dangerous bit, it also nukes any file that git doesn't know about, including your new source files.

autoreconf -v -i

That's part of the standard "from git" build.

CFLAGS="-g -fPIC -W -Wall -Wextra -Werror"

We're talking about adding --std=gnu99 here.  That does TONS of warnings, and -g enables debug mode.  We build with debug mode for production at FastMail as well, and with a patch that turns every assert() into a core dump.  That way we can debug production issues more quickly.

./configure --prefix=/usr/cyrus --with-cyrus-prefix=/usr/cyrus --enable-autocreate --enable-http --enable-unit-tests --enable-replication --enable-nntp --enable-murder --enable-idled --enable-xapian

Actually there are a couple more that should be here, like--enable-calalarmd and --enable-apple-push-service .  I really want most of this to not be optional - build should build everything, and just don't ship some binaries if you don't want them.  The alternative is that half our stuff doesn't get compiled by people part of the time.

I also REALLY want to strip all the crappy #ifdef HAVE_LIBRARY ... code ... #else ... stubs ... #endif that leads to different behaviour depending on what you've got.  Dependencies are not hard to download, and the cognative load/testing complexity/behaviour delta is just too great to be worth it.  Likewise with library versions (mostly) - we should just require the version we need.  If a system must support an older one as well, they can package that dependency alongside Cyrus, it's what I'm doing with libical for our production servers now.

make sieve/sieve-lex.c && perl -p -i -e "s/int yyl;/yy_size_t yyl;/" sieve/sieve-lex.c

This is a workaround for a buggy non-Werror-safe bit of junk in this version of Ubuntu's flex.  *sigh*.  There's a 'make lex-fix' now, but I left my alias in place.

make -j16 && make -j16 check && sudo make install && sudo make install-binsymlinks && sudo cp tools/mkimap /usr/cyrus/bin/mkimap'

Hammering the CPUs plenty, but hey.  I have 16 virtual cores.  Note that I install the binsymlinks, because I test across versions all the time, and older Cyrus doesn't have the binaries in the new locations.  Easier than making all the support tooling handle both.

And I use mkimap in bootstrap :)

Note that the default install path is /usr/cyrus/ - and I use that.  I also have /usr/cyrus24, /usr/cyrus25, /usr/cyrus23 and so on.  They don't get updates as often, but some of my murder test stuff can use them.  There are aliases to build these versions as well, that know about what flags they supported.

After running cbcalse (assuming tests passed and it installed) we have the latest Cyrus, now to test it.

In cyrusdevtools are some various scripts which can launch an instance or many and set up debug environments and saslauthds and everything - feel free to clone that repo and have a play, it's all very custom and for-me though, so don't be surprised if they're all kinda bogus and tricky to understand.  You don't need to do tests that way, I just find it easy if I want to run up a murder or replication instance and fiddle with it.  I wrote all this stuff pre-Cassandane.

......

OK, so Cassandane.  You need to pull in the dependencies.  All the Debian and Perl dependencies as documented in the repo, and we also need to set up imaptest:

dovecot:
% cd /home/brong/src/imaptest/dovecot-2.2
% ./configure
% make

imaptest:
% ./configure --dovecot=/home/brong/src/imaptest/dovecot-2.2
% make

NOTE: no "make install" on either of those.

That's all that's required, then to configure cassandane.  I have a run script:

brong at bat:~/src/cassandane$ cat run 
#!/bin/bash

ps ax | fgrep /tmpfs/cas | awk '{print $1}' | xargs sudo kill -9
sudo mkdir -p /tmpfs/cas
sudo chown cyrus /tmpfs/cas
sudo -u cyrus ./testrunner.pl -f pretty -j 8 "$@"

Note that we're murdering anything with this path in the name (all binaries will be started with -C /tmpfs/cas/$dir/conf/imapd.conf, so they will match this expression) - that cleans up after any previous run.

I use a tmpfs mounted on /tmpfs - it's way faster, and I have tons of RAM.

Finally I run with -f pretty, because it's nice to read.

You can run with --valgrind to use valgrind (if it's installed) - way slower, but finds memory leaks.

You can name individual test suites of even individual tests on the command line - running with -v -v is very noisy, but gives a lot more data.  All IMAP telemetry for example.  I also run "sudo tail -f /var/log/syslog", and do some stuff in /tmpfs/cas as root to examine log files and disk structures for failed tests.

Config:
% grep -v '^#' cassandane.ini

[cassandane]
rootdir = /tmpfs/cas

[valgrind]

[cyrus default]
prefix = /usr/cyrus

[gdb]

[config]

[imaptest]
basedir=/home/brong/src/imaptest/imaptest
suppress=append-binary urlauth-binary fetch-binary-mime fetch-binary-mime-qp

===

As you can see, I suppress some tests.  The binary tests are, I think, buggy upstream still.  I need to get back to talking to Timo about that.  Either that or relax Cyrus a bit, since the commands it's running kinda make sense, even if they aren't specifically allowed by the spec.

The other thing is that imaptest has tests for RFC features that we don't have - but as we add them, it will see the capability and run those tests automatically :)

===

Hopefully this helps give an insight into my development process.  I tend to work against fm/future mainly, which is the branch that runs just a couple of users at FastMail and gets all the new stuff first.  As things are ready for upstream, I rebase that on top of cf/master, and then push those commits upstream and force push the new head to fm/future after seeing that it's identical in content.

Right now we're a few commits ahead of cf/master:

brong at wot:~/src/cyrus-imapd$ git log --oneline cf/master..fm/future | cat
2f9aab4 fix memory leaks in ACL (found with valgrind)
5d868eb lmtpd: repair fuzzy match of "_ -"
2da5864 mboxname: improve safety around invalid names, INBOX case
db16a14 imapd: make sure rename things actually exist
2567585 mboxlist: auditlog all mailboxes.db changes
bf4cb38 add imip notifier via notifyd
276c7c3 syslog the remote scheduling request...
b5e2402 configure: require up-to-date libs and remove wrappers
eb034bd free some memory leaked in the Reverts above
d1d3aa5 Revert "http_dav.c: DAV:acl property doesn't have to include privileges contained in an aggregated privilege"
bdf7cdd Revert "http_dav.c: reintroduced handling of <deny> (negative rights) and properly handle all/authenticated/unauthenticated identifiers"
e5ddfb0 Revert "http_dav.c: added IMAP-like semantics to ACL method as documented in the comment"
9cc151a Revert "http_caldav.js: use new IMAP-like semantics for changing ACL; build XML documents using existing JS methods"
bd26219 mkdebian: use cyruslibs-fastmail-v1 package
8a15daf Fastmail Secrets (no rated)
3101fc8 Remove sieve action string
b59dabb Fatal -> abort in common daemons
7312df1 assert: make it core dump
0b61da2 hammer_cyrusdb: now with more generic
1f94f67 conversations doc, kinda
6050926 httpd: free memory at the end of every command
dceec91 horrible hack to make non-owner addressbooks sort lexically later than owner
0138d0e sync_support: skip annot CRC again -XXX temporary

The top 5 of these or so are candidates for upstream after a bit more testing.  The "Revert" commits are just me not finding the time to fix our ACL handling code to match upstream's more correct working of the ACL command.  Ken patched that a few weeks ago, but I wasn't ready to switch back yet...

The iMIP stuff will go upstream once I've got the notify side working nicely and done a bunch more testing, it's buggy as anything right now and not being used in production yet.

...

Cheers,

Bron.
-- 
  Bron Gondwana
  brong at fastmail.fm