Squatter segfault on big mailbox

[Gabriele Bulfon]

Tried looking into the source of squatter and verbosing on a specific folder that always shows the problem.
Here is how it ends:
 
Processing index character 128, 9 total words, temp file size is 41
Processing index character 130, 8 total words, temp file size is 52
Processing index character 147, 2 total words, temp file size is 12
Processing index character 152, 2 total words, temp file size is 12
Processing index character 153, 5 total words, temp file size is 26
Processing index character 172, 10 total words, temp file size is 66
Processing index character 176, 72 total words, temp file size is 498
Processing index character 191, 417128205 total words, temp file size is 1251444163
Closing index: Not enough space
 
I will look and change the source to give more info on the backtrace of the error, but maybe someone here alreay has any idea?
 
Gabriele
 
 
Sonicle S.r.l. 
: 
http://www.sonicle.com
Music: 
http://www.gabrielebulfon.com
Quantum Mechanics : 
http://www.cdbaby.com/cd/gabrielebulfon
Da:
Gabriele Bulfon
A:
Vladislav Kurz
Mark
info-cyrus
Data:
17 dicembre 2019 7.26.35 CET
Oggetto:
Re: Squatter segfault on big mailbox
 
Hello, we're having the same problem with Cyrus 2.5.11 on XStreamOS/illumos
Any idea how to fix this?
Same problem both on 32 bit large files build, and 64 bit.
Looks like the Debian issue has no solution.
Can't believe this has never been addressed.
 
Gabriele
 
 
Sonicle S.r.l. 
: 
http://www.sonicle.com
Music: 
http://www.gabrielebulfon.com
Quantum Mechanics : 
http://www.cdbaby.com/cd/gabrielebulfon
----------------------------------------------------------------------------------
Da: Vladislav Kurz
A: Mark
info-cyrus
Data: 11 ottobre 2017 9.37.21 CEST
Oggetto: Re: Squatter segfault on big mailbox
On 10/11/17 07:37, Mark wrote:
On 2017-10-10 16:05, Vladislav Kurz wrote:
Hello everyone,
we have recently migrated our mail server from cyrus 2.2 to 2.5 (both
debian packages).
We have a problem that squatter segfaults on some quite big mailboxes.
I had to add -i (--incremental) to at least index new mails. But users
cannot search in old mails (before migration). I used strace to find out
if it is some particular mail, but it parsed everything fine, and failed
when dealing with some of the many temporary files squatXXXX.
The problematic mailboxes have just inbox10 GB (and15 GB with all
subfolders) and20000 emails in inbox.
Has anyone else hit similar problem and solved it somehow?
I'm not sure about fixes, but there are some reports of similar
problems.  Have a look at the thread with the subject "Cyrus 2.5.10 IMAP
search" from the beginning of June this year. 
Apparently the format of the squatter database changed between 2.4 and
2.5, and some people (at least) have seen a 10x increase in the size of
the squatter database for the same mailbox.  For systems that only allow
2GB mmap, this can be a problem.
I'm not sure if this increase in size is a bug or a feature.
Mark
Hello Mark,
Thanks for the hint about old discussion. Unfortunately, there was no
solution to this issue mentioned. I found out that this was already
reported to debian, I wonder whetrher the maintainer forwarded the bug
report upstream.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871921
This is on 64-bit Linux, so mmap should not be limited to 2 GB.
And I also have noticed the extreme increase in squat file sizes.
--
Best Regards
Vladislav Kurz
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
----Cyrus Home Page: http://www.cyrusimap.org/List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/To Unsubscribe:https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20200102/4aae66fe/attachment.html>