Speed-differences in listing mailboxes between 2.4 and 3.0

[Felix Schumacher]

Hi all,

we are trying to migrate our mailservers from cyrus imapd 2.4 to 3.0 and 
are currently looking at speed differences in listing mailboxes. Both 
the imap command 'list "" ""' for a user and a ctl_mboxlist -d seem to 
be quite a bit slower.

At the moment the 3.0 environment has around around 170k mail folders 
listed by ctl_mboxlist -d and takes about 0.8s to list all those entries 
(piping the output to /dev/null). This system has no real load except 
for the list command).

In the 2.4 environment we have 244k mail folders listed and it takes 
about 0.5s to list those entries (again piped to /dev/null). This system 
is our productive mail server and had some load to handle while listing 
was done.

mailboxes.db is in skiplist format on the 2.4 environment. We tried 
twoskip and skiplist on the 3.0 environment, but it didn't change that 
much.

Using perf record -F 99 -g ctl_mboxlist -d > /dev/null and then perf 
report I get for 3.0

+   41.67%     0.00%  ctl_mboxlist  libcyrus.so.0.0.0       [.] 
cyrusdb_foreach
+   41.67%     0.00%  ctl_mboxlist  libcyrus_imap.so.0.0.0  [.] 
mboxlist_allmbox
+   41.67%     0.00%  ctl_mboxlist  ctl_mboxlist            [.] do_dump
+   34.52%     0.00%  ctl_mboxlist  [kernel.kallsyms]       [k] 
entry_SYSCALL_64_fastpath
+   22.62%     5.95%  ctl_mboxlist  libc-2.22.so            [.] do_fcntl
+   22.62%     2.38%  ctl_mboxlist  libc-2.22.so            [.] 
__xstat64
...

and for 2.4 I get

+     11.63%  ctl_mboxlist  [kernel.kallsyms]  [k] kmem_cache_alloc
+      9.30%  ctl_mboxlist  [kernel.kallsyms]  [k] 
system_call_after_swapgs
+      6.98%  ctl_mboxlist  [kernel.kallsyms]  [k] kmem_cache_free
+      6.98%  ctl_mboxlist  [kernel.kallsyms]  [k] fcntl_setlk
+      4.65%  ctl_mboxlist  [kernel.kallsyms]  [k] fget_raw
+      4.65%  ctl_mboxlist  [kernel.kallsyms]  [k] locks_copy_lock
+      4.65%  ctl_mboxlist  [kernel.kallsyms]  [k] _raw_spin_lock
+      2.33%  ctl_mboxlist  libc-2.11.3.so     [.] 
__GI_____strtoll_l_internal
+      2.33%  ctl_mboxlist  libc-2.11.3.so     [.] __free
...

Has anyone an idea, why cyrusdb_foreach and mboxlist_allmbox is so 
promiment in the 3.0 recording or rather why 3.0 appears to be so much 
slower compared to 2.4?

Felix