how many users can Cyrus server support? -- is 5.5 msg/sec sl ow?

[Aidan Evans]

On 27 Sep 2002 at 08:24 Adam Williams wrote to Su Li and
info-cyrus at andrew....

>>I think you are right. I don't think the bottle neck is Cyrus IMAP. Because
>>when I move syslog to the network the performance get 2 times better and
>>when I change to a disk striped by 3 disks, I 2 times faster. Right now I
>>can get 10 msg/sec. If I can get 15-20 msg/sec, I will be very happy.
>
>If adjusting syslog makes that much of a difference you probably haven't
>disabled fsync() on the /var/log/maillog entry (or whatever your
>relevant log file is).  Prepending the filename in /etc/syslog.conf with
>a "-", thus /var/log/maillog becomes -/var/log/maillog with disables
>syslog's fsync() call after each log message.  If you generate alot of
>logs those fsync()s will kill your I/O throughput.

  This is so, so right.  This comment has saved our Cyrus installation.
We'd been having continual problems where users would experience long, long
delays to login, tens or even hundreds of seconds.  As long as I leave the
high volume logs as, e.g., "-/var/log/maillog", we never have this
problem.  Put it back the "old" way and the problem returns.

  I wrote a simple test program:

	#include <syslog.h>
	main () {
	  int j;
	  openlog("loadtest", LOG_PID, LOG_LOCAL2);
	  for (j = 1; j <= 1000; j++) {
	    syslog(LOG_DEBUG, "test syslog message %d", j);
	  }
          closelog();
	}

to generate a thousand syslog messages and then timed it.  With synchronous
logging it took about 18 seconds; without "time" timing says "0.05s" which
I take to mean essentially not a measurable amount of time.

  I don't know how Linux syslog processing works internally, but the
observed behaviour suggests that with synchronous logging and bunches of
Cyrus processes all trying to write log messages, the syslog daemon gets
behind syncing and is not able to accept new log messages so all the Cyrus
processes queue up waiting to syslog, and the clients on the other end
think they've been put on ignore.

  If someone at CMU has the opportunity, it would be helpful to future
Linux installers to update the documentation in "Operating System Specific
Notes" or "Performance Notes" under "Installing and Configuring the Cyrus
IMAP Server" to have a note like

	Depending on your syslog configuration and usage volume, Cyrus may
	generate thousands of syslog messages.  On Linux, syslog
	performance can be greatly improved by disabling synchronous
	logging (disabling fsync() after each message).  Prepending
	filenames in /etc/syslog.conf with a "-", e.g., "/var/log/maillog"
	becomes "-/var/log/maillog", disables syslog's fsync() call after
	each log message.  If you log many messages those fsync()s will
	kill your I/O throughput.

Aidan Evans   | Networks & Systems
(902)494-3332 | University Computing & Information Services
              | Dalhousie University, Halifax, N.S., Canada