prevent stuck processes with large folder manipulations

[Brian Awood]

Hi Paul

On Sunday 03 January 2010 @ 13:35, Paul Dekkers wrote:
>
> Are you running 32 or 64-bits? We run 64-bits, and I realized that this
> allows a single imapd process to consume a considerable amount of
> memory (eg. all) instead of "just" 2G or so per process. (The server
> I'm talking about has 6G of memory and 2G swap, should be ok for ~50
> active users.) But now there's nothing limiting the memory consumption
> by just one single user/process, I guess.

We're running 32-bit, with 8G of ram and 8G of swap but around ~4k active 
users per backend machine.  It suspect it would probably harmless to 
limit the cyrus processes to 2G with ulimit, I doubt any one imapd would 
ever legitimately need to be larger than that.

> We are not running any proxies, well, other than stunnel doing SSL
> offloading on another box. So apart from SSL the clients are directly
> talking to imapd. I do believe it's just Thunderbird timing-out. This
> box is running 2.3.13 (Simon's RPM on 64-bit RHEL 4), but I recall
> seeing it with olders versions before (of both Cyrus and TB).
>
> Actually; the mozilla bugtracker reference sounds very similar. It
> happens with large deletions too, just like with large copies, as
> described in this bug. But we see this with much more recent versions,
> Thunderbird 3 and the recent 2.3 cyrus.

The behavior I observed was that TB was hitting a timeout, even though the 
server was responding almost immediately.  TB just wasn't recognizing the 
response and it seemed related to the length of that response.  If you 
still see the behavior with Thunderbird 3, and your able to reproduce the 
problem, you may be able to provide more info to the TB developers.  They 
seem pretty responsive if they get good feedback, so there's a good 
chance it could be solved once and for all if you can give provide them 
some debugging info.

> Regarding the last suggestion in this bug; for the deletes I did
> consider the delayed expunge and/or delete, but that wouldn't help with
> the large copies.

Even large copies should occur fairly quickly, unless you have disabled 
singleinstancestore.  singleinstancestore is enabled by default, and 
causes cyrus to create hardlinks rather than actually creating copies.  

-Brian