Squatter segfault on big mailbox

[Vladislav Kurz]

On 10/11/17 07:37, Mark wrote:
> On 2017-10-10 16:05, Vladislav Kurz wrote:
>> Hello everyone,
>>
>> we have recently migrated our mail server from cyrus 2.2 to 2.5 (both
>> debian packages).
>>
>> We have a problem that squatter segfaults on some quite big mailboxes.
>> I had to add -i (--incremental) to at least index new mails. But users
>> cannot search in old mails (before migration). I used strace to find out
>> if it is some particular mail, but it parsed everything fine, and failed
>> when dealing with some of the many temporary files squatXXXX.
>>
>> The problematic mailboxes have just inbox > 10 GB (and >15 GB with all
>> subfolders) and >20000 emails in inbox.
>>
>> Has anyone else hit similar problem and solved it somehow?
> 
> I'm not sure about fixes, but there are some reports of similar
> problems.  Have a look at the thread with the subject "Cyrus 2.5.10 IMAP
> search" from the beginning of June this year. 
> 
> Apparently the format of the squatter database changed between 2.4 and
> 2.5, and some people (at least) have seen a 10x increase in the size of
> the squatter database for the same mailbox.  For systems that only allow
> 2GB mmap, this can be a problem.
> 
> I'm not sure if this increase in size is a bug or a feature.
> 
> Mark
> 

Hello Mark,

Thanks for the hint about old discussion. Unfortunately, there was no
solution to this issue mentioned. I found out that this was already
reported to debian, I wonder whetrher the maintainer forwarded the bug
report upstream.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871921

This is on 64-bit Linux, so mmap should not be limited to 2 GB.

And I also have noticed the extreme increase in squat file sizes.


-- 
Best Regards
        Vladislav Kurz