Cyrus replication performance improvement

[DEMBEK, Adam (Adam)]

Welcome

We are working on improving performance of our Cyrus server replication.

We are using Cyrus 2.3.7. When installed our software on server with
slower 10 RPM disks we notice that one sync_client is not able to
synchronize all data in real time. On servers with 15 RPM disk
performance was better. 

We prepared few modifications that allow to run 3 sync_client that will
replicate different mailboxes at the same time. We would like to get
opinion of other Cyrus developers about these modifications to identify
if they can introduce any additional risks (Details below). Our tests
show 30 - 50 % improvement of Cyrus replication speed on server with 10
RPM disks. We tested that no data is lost during synchronization for
CREATE, APPEND, STORE, RENAME and SETANNOTATION commands. 

Our changes:

sync_log.c
Write log to one of 3 files depending on mailbox name. The same mailbox
is always sent to the same log file. 

sync_client.c
Read from different sync/log file. Path to correct log file is
configured in imapd.conf
Each sync_client is run with different imapd.conf. 

cyrus.conf
Start 3 syncservers on different ports

syncserver       cmd="/apsw/ms/cyrus/imapd/bin/sync_server -C
/etc/imapd_s.conf" listen="50055" maxchild=1 
syncserver1 cmd="/apsw/ms/cyrus/imapd/bin/sync_server -C
/etc/imapd_s1conf" listen="50056" maxchild=1
syncserver2 cmd="/apsw/ms/cyrus/imapd/bin/sync_server -C
/etc/imapd_s2.conf" listen="50057" maxchild=1
                                 
Code changes in sync_log.c

void sync_log_mailbox(char *name)
{
    int sync_file_id = getFileId(name + 5);
    sync_log(sync_file_id, "MAILBOX %s\n", name);
}

int getFileId(const char *name)
{
    if (multi_sync_log > 1)
    {
        unsigned int result = 0;
        int pos = 0;
        for (; name[pos] != 0; ++pos)
        {
            result += name[pos];
        }
        return (result % multi_sync_log);
    }
    return 0;
}

static void sync_log_base(int sync_file_id, const char *string, int len)
{
    int fd, rc;
    struct stat sbuffile, sbuffd;
    int retries = 0;
    char *file_name;

    if (!sync_log_enabled) return;

    if (sync_file_id == 0)
    {
        file_name = sync_log_file;
    }
    else if (sync_file_id == 1)
    {
        file_name = sync_log_file1;
    }
    else if (sync_file_id == 2)
    {
        file_name = sync_log_file2;
    }

    while (retries++ < SYNC_LOG_RETRIES) {
        fd = open(file_name, O_WRONLY|O_APPEND|O_CREAT, 0640);
        if (fd < 0 && errno == ENOENT) {
            if (!cyrus_mkdir(file_name, 0755)) {
                fd = open(file_name, O_WRONLY|O_APPEND|O_CREAT, 0640);
            }
        }
        if (fd < 0) {
            syslog(LOG_ERR, "sync_log(): Unable to write to log file %s:
%s",
                   file_name, strerror(errno));
            return;
        }

        if (lock_blocking(fd) == -1) {
            syslog(LOG_ERR, "sync_log(): Failed to lock %s for %s: %m",
                   file_name, string);
            close(fd);
            return;
        }
...
}

Regards, 

Adam Dembek
Messaging Applications Poland
phone: +48 52 349 1908

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20081203/834f15f5/attachment-0001.html