BUG? File descriptor use in cmd_append (for MULTIAPPEND) results in many open files

[Nik Conwell]

On Dec 11, 2006, at 12:29 PM, Andrew Morgan wrote:

> On Mon, 11 Dec 2006, Nik Conwell wrote:
>
>> I'm using the UW mailutil to transfer mailboxes from UW to Cyrus  
>> (2.3.7).  It uses APPEND, specifically multiappend (single APPEND  
>> with multiple messages being appended).  Cyrus-imapd handles this  
>> multiappend by creating stage files for each appended message and  
>> leaving the file descriptor open.  The problem is that after 240  
>> messages, we run out of file descriptors and so an open() of the  
>> next stage file fails with EMFILE.  I updated /etc/cyrus.conf to  
>> make the max fds be 1024 (AFAICT kernel MAX) which helped somewhat  
>> but not for larger mailboxes with > 1008 messages.
>>
>> Shouldn't the multiappend/append be closing the FD for each stage  
>> file and then reopening it later as it needs it?
>>
>> Do people just tweak their kernels to have some insane number of  
>> FDs available in order to compensate for this?
>>
>> Or, do people not use mailutil and instead use something that  
>> issues multiple append commands rather than a single append with  
>> multiple e-mails?
>
> We run with a much, much larger number of file descriptors here.   
> I've increased the system limit to around 200k (/proc/sys/fs/file- 
> max on linux).  This is for the day-to-day running of Cyrus, so I  
> don't know if you would need a higher limit for running mailutil  
> (but I doubt it).
>
> In practice, each of my backends has only used a maximum of around  
> 12k file descriptors, but I'd hate to run out!  :)

That's for the entire system though, right?  I'm running into a 1024  
limit per process, namely the cyrus imap server process has all the  
appended stage files open.  Am I missing something fundamental here?   
(I probably am because I would have figured people would have run  
into this issue already...)

BTW - my /proc/sys/fs/file-max has 406572.