BUG? File descriptor use in cmd_append (for MULTIAPPEND) results in many open files

[Andrew Morgan]

On Mon, 11 Dec 2006, Nik Conwell wrote:

> I'm using the UW mailutil to transfer mailboxes from UW to Cyrus (2.3.7).  It 
> uses APPEND, specifically multiappend (single APPEND with multiple messages 
> being appended).  Cyrus-imapd handles this multiappend by creating stage 
> files for each appended message and leaving the file descriptor open.  The 
> problem is that after 240 messages, we run out of file descriptors and so an 
> open() of the next stage file fails with EMFILE.  I updated /etc/cyrus.conf 
> to make the max fds be 1024 (AFAICT kernel MAX) which helped somewhat but not 
> for larger mailboxes with > 1008 messages.
>
> Shouldn't the multiappend/append be closing the FD for each stage file and 
> then reopening it later as it needs it?
>
> Do people just tweak their kernels to have some insane number of FDs 
> available in order to compensate for this?
>
> Or, do people not use mailutil and instead use something that issues multiple 
> append commands rather than a single append with multiple e-mails?

We run with a much, much larger number of file descriptors here.  I've 
increased the system limit to around 200k (/proc/sys/fs/file-max on 
linux).  This is for the day-to-day running of Cyrus, so I don't know if 
you would need a higher limit for running mailutil (but I doubt it).

In practice, each of my backends has only used a maximum of around 12k 
file descriptors, but I'd hate to run out!  :)

 	Andy