how many users can Cyrus server support? -- is 5.5 msg/sec sl ow?

[Jeremy Rumpf]

>
> I had an eye on Su's postings the last days because of own interest in
> getting performance messurement data examples. I then started to do my
> own tests because I just wanted to see how my installation performs and
> was surprised that I got a much better performance without knowing that
> much about filesystems and BerkleyDB than he actually does. Maybe
> someone can handle me a script or something that shows me how you
> messure things so that I can be shure that I do the same messurements
> than you all do! Would be great to see how other people are testing
> theire email performance...
>
> ---Christian---

Perhaps the best thing to do for folks setting up cyrus high end mail systems 
would be to describe our own setups and let folks (like Su) look them over to 
see what they can pick off for his own setup.

The stuff I'm currently working on is based on Linux, so most of my setup is 
linux centric, but the generic basics would apply to most systems. My system 
would look roughtly as follows (pardon if my ascii art sucks):

+------------+  +------------+   +------------+ +------------+
| mta server |  | mta server |   | imap proxy | | imap proxy |
| postfix    |  | postfix    |   | perdition  | | perdition  |
+------------+  +------------+   +------------+ +------------+
      |               |    \           |    \          |  \
      |               |     \          |     \         |   \
      \               /      \         \      \        /    \
       \             /        \         \      \      /      \
        \           /          \         \      \    /        \
         -----------            \         -----------          \
	/           \            \      /         \             \ +----------+
       /             \            \----/--------------------------| ldap dir | 
+------------+  +------------+        /                           | openldap |
| imap server|  | imap server|       /                            +----------+
| cyrus      |--| cyrus      |-------
+------------+  +------------+ 


Optimization for the mta servers is as follows:

	Two sets of disk mirrors at RAID 0
	  First set is the system disks
	  Second set is the postfix mail queues

	Use a nice hardware RAID controller with
	a fat slice of cache (128MB cache on an
	Mylex ExtremeRaid here). Set the cache to
	run in writeback mode.

	Setup the mail queus on ext3 fs with full
	journaling  data=journal (note there are 
	some details Stephen Tweedie is working on
	with regards to data=journal, they've been
	fixed but in patches, not mainline yet)

	Create the ext3 journal on a separate set 
	of disks, specifically allocate a partition
	to hold the journal (I put the journal on a
	dedication partition on the system disks as
	their load normally isn't all that
	high). Make the journal HUGE, ie 250 + MB.

	Mount the mail queues with the noatime mount 
	option. Also mount /var/log with noatime.

	chattr -S /var/log and the mail queues.

	Disable syncronous logging in syslogd or 
	use network syslogging to another host.

	Consider using tmpfs for lmtpd temp directory
	(I haven't done this yet, but I really like
	the idea).

	Bump the number of open file descriptors 
	available to the system to something like
	16k - 32k.

	Use direct TCP lmtp delivery to the cyrus
	backends.

Optimization for the cyrus servers is as follows:

	All the same optimizations for the mta's.

	Don't run a specific mta on the backend
	(use TCP lmtp). Note: I do run an mta
	but it's only used for local system 
	mail and notifications, not for cyrus 
	delivery.

	Allocate multiple cyrus partitions. Keep
	each partition on it's own RAID 0 disk set.
	Keep the partitions on a single disk pair 
	(don't stripe 6 drives together, break them
	up into 3 two disk stripe sets, one set for each
	partition). Maximize the number of independent
	spindles. Allocate the journals on separate
	disks.

	chattr -S the mail partitions (I do keep the
	mailboxes.db, etc, in the conf dir syncronous)

	Disable duplicate delivery suppression.

Theory of operation:
	
	Operation of the system hinges on LDAP. LDAP is used by postfix
	for all the mail routing (to get the message to the 
	correct cyrus backend). It's also used for authentication 
	(saslauthd). It's also used by the perdition proxies to 
	determine the correct cyrus backend to route connections
	to.

	An LDAP entry looks as follows:

	dn: cn=john.doe,ou=real,ou=addresses,ou=mail,dc=somecompany,dc=com
	sn: Doe
	givenName: John
	userPassword: {MD5}+/xxxxxxxxxxxxDZgg==
	description: mail user account
	uid: john.doe at doeworld.com
	cn: john.doe
	mail: john.doe at doeworld.com
	mail: jdoe at doeworld.com
	mail: supafreeza at doeworld.com
	mailLocalAddress: john.doe at mail-store1.somecompany.com
	smtpauthAccess: enabled
	popimapAccess: enabled
	smtprouteAccess: enabled

	I have postfix transport maps for all the cyrus backends in 
	postfix (mail-store1, mail-store2, .....) to route the messages
	via ltmp to the correct backend. There's an LDAP virtual map on
	the postfix mtas that looks like this:

	virtual_maps = ldap:ldapvirtual, hash:/etc/postfix/virtual
	ldapvirtual_server_host = ldap1.somecompany.com
	ldapvirtual_server_port = 389
	ldapvirtual_search_base = ou=addresses,ou=mail,dc=somecompany,dc=com
	ldapvirtual_timeout = 20
	ldapvirtual_query_filter = (&(mail=%s)(smtprouteAccess=enabled))
	ldapvirtual_result_attribute = mailLocalAddress, mailRoutingAddress
	ldapvirtual_scope = sub
	ldapvirtual_bind = yes
	ldapvirtual_bind_dn = cn=mailro,dc=somecompany,dc=com
	ldapvirtual_bind_pw = xxxxxxx

	Then, when a message arrives for john.doe at doeworld.com or
	supafreeza at doeworld.com the virtual map rewrites the 
	_envelope_ address (the original To: address is preserved)
	to john.doe at mail-store1.somecompany.com. From there lmtp
	delivery kicks it to mail-store1 which has a mailbox for
	user/john.doe (unix separator).

	If the user connects to perdition to check his mail. He logs in
	as the user john.doe at doeworld.com (aka the uid). Perdition does
	an LDAP query to resolve the true mail backend. Specifically it
	uses the mailLocalAddress with the mailbox being the username 
	and the domain being the host. Something like 
	<username>@<mail backend>. It then connects to the correct mail 
	backend (mail-store1.somecompany.com in this case) as the 
	user john.doe. Note: There _is_ a mailbox john.doe on
	mail-store1, which is legal.

	Cyrus then attemps to authenticate the user john.doe against 
	LDAP via saslauthd. Saslauthd is setup as:

	ldap_servers: ldap://ldap1.somecompany.com/
	ldap_bind_dn: cn=mailro,dc=somecompany,dc=com
	ldap_bind_pw: xxxxxxx
	ldap_timeout: 20
	ldap_scope: one
	ldap_search_base: ou=real,ou=addresses,ou=mail,dc=somecompany,dc=com
	ldap_auth_method: bind
	ldap_filter: (&(cn=%u)(popimapAccess=enabled))
	ldap_debug: 2
	ldap_verbose: on
	ldap_ssl: no
	ldap_start_tls: no

	Saslauthd resolves the user via the cn attribute (hence why it is
	john.doe) and uses the userPassword attribute for the password.

	Postfix is also tweaked to allow SMTP AUTH access as well, using
	a running instance of saslauthd on the mtas for password
	resolution. The SMTP AUTH username is the same john.doe at doeworld.com 
	(the uid attribute). Access to SMTP AUTH is also controlled via 
	the smtpauthAccess attribute.

	Of course, since saslauthd is in use (thus requiring plain text
	passwords everywhere), IMAP connections to the perdition
	proxies as well as the SMTP AUTH connections to the mtas are
	protected via SSL/TLS.

	Management of the system is performed via a custom MySQL database
	backend fronted by some perl scripts. The perl scripts can put 
	entries into LDAP as well as connect to cyrus (via the cyradm module)
	to create the required mailboxes.

	From this setup, the cyrus backends are horizontally scaleable 
	(just add more backends) and the postfix mtas are horizontally
	scaleable (add more if load demands it). Mtas, proxies, etc,
	are either load balanced via round robin DNS or some hardware
	based solution.


Whew, okay, I'm done typing for now. Hope this helps :)

Cheers,
Jeremy