ctl_mboxlist virtual memory exhausted !

Wed Mar 26 04:10:17 EDT 2008

thanks for your answer !

* the previous "large installation" we have is 30Millions users with lots of backends and store (cyrus + netapp filer ) , and an internaly developped pop3 proxy (usage was only pop3 at this time ;-) (we also have an imap proxy for a fraction of users), at the time we build it, things like nginx was not there ;-)

* this one was "low usage", but large number of users, so we thought only one backend was needed, with the ability to move users one more backend if load dictate it. So we tought murder architecture was the way to go ...
seems to be a bad idea ?
note : storage is Hitachi HDS SAN on RAID5 volumes, replicated across 2 datacenters.

* and yes, I applied the patches last week, and checking now if there is no more "bugs".

note : I also got some "hangs" with berkeley DB (4.3.27) :

#4  0xb7d89edd in ___newselect_nocancel () from /lib/libc.so.6
#5  0x080e4a08 in __os_sleep_cyrus_imapd_db (dbenv=0x0, secs=1, usecs=0) at ../dist/../os/os_sleep.c:84
#6  0x080e4b36 in __os_yield_cyrus_imapd_db (dbenv=0x0, usecs=1000000) at ../dist/../os/os_spin.c:110
#7  0x080f0927 in __db_fcntl_mutex_lock_cyrus_imapd_db (dbenv=0x823dac0, mutexp=0xb7721360)
    at ../dist/../mutex/mut_fcntl.c:95
#8  0x080c5eb4 in __lock_get_internal_cyrus_imapd_db (lt=0x823ddb8, locker=2149230760, flags=0, obj=0x823fecc,
    lock_mode=DB_LOCK_WRITE, timeout=0, lock=0xbfcc4bf0) at ../dist/../lock/lock.c:820
#9  0x080c45b8 in __lock_vec_cyrus_imapd_db (dbenv=0x823dac0, locker=2149230760, flags=0, list=0xbfcc4be0, nlist=2,

-----Message d'origine-----
De : Bron Gondwana [mailto:brong at fastmail.fm]
Envoyé : mardi 25 mars 2008 23:28
À : Brasseur Valery; info-cyrus at lists.andrew.cmu.edu
Objet : Re: ctl_mboxlist virtual memory exhausted !

On Tue, 25 Mar 2008 09:31:40 +0100, "Brasseur Valery" <Valery.Brasseur at atosorigin.com> said:
> Hi,
>
> I am running 2.3.11, with 2 Millions users (4M mailboxes ;-)
>
> when trying to do a ctl_mboxlist -m, after some time (a few second !) I
> got a "virtual memory exhausted", and i can see that the process is
> allocating more than 3Gb of memory !

Ouch. That hurts

> did some of you encontered this ?
> any way to bypass ?

We split our Cyrus instance up into 300Gb data partitions.  We currently have
56 stores (112 partitions thanks to replication).  Obviously you need infrastructure
to manage this, and some form of frontend proxy to direct user logins to the correct
store (we use nginx).  Further, any users who need to share mailboxes must be on the
same store.

Still, things are a lot faster when your average mailboxes.db is only 4.5Mb in size
(having just checked the one for the store my mailbox is on)

> I also got a lot's of skiplist corruption when file size is around 700Mb
> for mailboxes.db, and mupdate process getting 100% of CPU when it's
> arrive !!!
> any ideas are welcome !

Are you using:

http://cyrus.brong.fastmail.fm/patches/cyrus-skiplist-safelock-2.3.11.diff
http://cyrus.brong.fastmail.fm/patches/cyrus-skiplist-state-2.3.11.diff
http://cyrus.brong.fastmail.fm/patches/cyrus-skiplist-transactions-2.3.10.diff

Finally, are you running a 32 bit operating system?  With a 700Mb mailboxes.db
being mmaped, you might be pushing close to the available process memory space.
Running a 64 bit kernel would probably help a lot there (you will of course need
to have 64 bit hardware!)

I would seriously recommend against having 2 million users all on one machine
on disaster recovery principles - it takes far too long to copy that much data
onto modern drives, so if you lose your drive unit then getting users back up
and running looks like about a week's sitting there watching data copy.  Yes,
I have done that before.  That's why we run partitions that can rebuild from
scratch in 6 hours now.

Bron.
--
  Bron Gondwana
  brong at fastmail.fm

Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité du groupe Atos Origin ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis.

This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos Origin group liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.