locking problems with 2.1.9

[Peter Krotkov]

Our locking issues have revisited us.  We are running cyrus 2.1.9 on
a Solaris 8 system using fcntl and skiplist (except flat for subscriptions).

Our MTA (pmdf) is unable to deliver mail to a local user via lmtp.
Except for this particular user cyrus is doing its thing just as
expected.

The stage. directory for this user's partition is as follows and
all files are related to delivery attempts to this user:

-rw-------   1 cyrus    mail        3101 Nov  8 09:37 14279-1036766256
-rw-------   1 cyrus    mail       21476 Nov  8 07:05 14521-1036757103
-rw-------   1 cyrus    mail       21476 Nov  8 08:00 20731-1036760410

Following is the stack for the three processes listed above (as
reported by the Solaris pstack command):

14279:  lmtpd -a
 ff098958 fcntl    (b, 7, ffbe9440)
 ff092f00 s_fcntl  (b, 7, 7, ffbe9440, b, 58d5c) + 1ec
 00058d5c lock_reopen (b, ffbe9570, ffbe94e8, ffbe94e4, e393e, ffbe9470) + 4c
 00035988 mailbox_lock_header (ffbeb658, 0, ffbeb658, 7efefeff, 81010100, ff0000) + cc
 00032d34 append_setup (0, ffbea658, 0, 0, 0, 0) + 54
 0002c284 deliver_mailbox (0, 1b7de8, c1d, 0, 0, 0) + 124
 0002c7fc deliver  (1b32d0, 0, 0, 0, ffffefcc, ffbecbe8) + 410
 0002f6b0 lmtpmode (1841e0, 1a8530, 1a8588, 1a8e70, 1a3d78, e10d8) + d2c
 0002ab8c service_main (2, ffbefd7c, ffbefd88, ff1f8000, 8, 2a5ac) + b8
 0002a6f4 main     (184000, ffbefd7c, ffbefd88, 2, e0c00, 0) + 618
 00029c1c _start   (0, 0, 0, 0, 0, 0) + 5c


20731:  lmtpd -a
 ff098958 fcntl    (b, 7, ffbe9440)
 ff092f00 s_fcntl  (b, 7, 7, ffbe9440, b, 58d5c) + 1ec
 00058d5c lock_reopen (b, ffbe9570, ffbe94e8, ffbe94e4, e393e, ffbe9470) + 4c
 00035988 mailbox_lock_header (ffbeb658, 0, ffbeb658, 7efefeff, 81010100, ff0000) + cc
 00032d34 append_setup (0, ffbea658, 0, 0, 0, 0) + 54
 0002c284 deliver_mailbox (0, 1a8530, 53e4, 0, 0, 0) + 124
 0002c7fc deliver  (1b22c8, 0, 0, 0, ffffefcc, ffbecbe8) + 410
 0002f6b0 lmtpmode (1841e0, 1c4b80, 1c4bd8, 1a93f0, 1a3d78, e10d8) + d2c
 0002ab8c service_main (2, ffbefd7c, ffbefd88, 2, 0, 2a5ac) + b8
 0002a6f4 main     (184000, ffbefd7c, ffbefd88, 2, e0c00, 0) + 618
 00029c1c _start   (0, 0, 0, 0, 0, 0) + 5c


14521:  lmtpd -a
 ff098958 fcntl    (b, 7, ffbe9440)
 ff092f00 s_fcntl  (b, 7, 7, ffbe9440, b, 58d5c) + 1ec
 00058d5c lock_reopen (b, ffbe9570, ffbe94e8, ffbe94e4, e393e, ffbe9470) + 4c
 00035988 mailbox_lock_header (ffbeb658, 0, ffbeb658, 7efefeff, 81010100, ff0000) + cc
 00032d34 append_setup (0, ffbea658, 0, 0, 0, 0) + 54
 0002c284 deliver_mailbox (0, 1c06a8, 53e4, 0, 0, 0) + 124
 0002c7fc deliver  (1b22c8, 0, 0, 0, ffffefcc, ffbecbe8) + 410
 0002f6b0 lmtpmode (1841e0, 1c15e0, 1c1638, 1a8eb0, 1a3d78, e10d8) + d2c
 0002ab8c service_main (2, ffbefd7c, ffbefd88, 2, 0, 2a5ac) + b8
 0002a6f4 main     (184000, ffbefd7c, ffbefd88, 2, e0c00, 0) + 618
 00029c1c _start   (0, 0, 0, 0, 0, 0) + 5c


There is one imapd (according to the cyrus proc/) for this user.
Over the course of several minutes of observation the stack
remains exactly the same:

11885:  imapd -s
 ff098958 fcntl    (b, 7, ffbebc10)
 ff092f00 s_fcntl  (b, 7, 7, ffbebc10, b, 633d4) + 1ec
 000633d4 lock_reopen (b, ffbebd40, ffbebcb8, ffbebcb4, ef566, 0) + 4c
 00046444 mailbox_lock_header (1986cc, 1aa610, e, ffbeedbc, ff0bd194, ffbeed28) + cc
 000479c0 mailbox_expunge (1986cc, 1, 0, 0, 0, ff09c648) + 1c
 00032580 cmd_expunge (1bb8e0, 0, 1a17a8, 1bb950, 6e676500, 6e676500) + 54
 0002c65c cmdloop  (18c800, 198400, 0, 0, 0, 0) + b40
 0002b8c4 service_main (0, ffbefd7c, ffbefd88, ff1f8000, 7, 2a1ec) + 290
 0002a334 main     (18c400, ffbefd7c, ffbefd88, 2, eb400, 0) + 618
 0002985c _start   (0, 0, 0, 0, 0, 0) + 5c


Following are processes reported as having the user's cyrus.* files
open:

# fuser cyrus.*
cyrus.cache:    11885om   22335om
cyrus.header:    14279om   11885om    9273om   20731om   14521om   22335om
cyrus.index:    11885om   22335om
cyrus.seen:

And following is pstack for all of the above except for 11885 (the imapd
already listed above) and 14279, 20731, 14521 (the ltmpd listed above):

22335:  imapd -s
 ff09b3bc read     (0, 1c4bc8, 6f5)
 0008e8c8 sock_read (0, 1c4bc8, 6f5, 8e8a0, 18edf8, 1) + 28
 0008d670 BIO_read (1bd090, 1c4bc8, 6f5, 1c32a8, 1bccf0, 0) + d0
 0007dec8 ssl3_read_n (5, 2010, 2010, 191b, 0, 0) + 148
 0007e140 ssl3_get_record (1bb7e0, 1bccf0, 0, 0, 23138, ff0941d8) + 1e0
 0007e8d4 ssl3_read_bytes (1bb7e0, 17, 1aa610, 1000, 0, 1bccf0) + 1d4
 0007c6e8 ssl3_read (1bb7e0, 1aa610, 1000, 7c6a0, 19a1ac, 0) + 48
 0006e730 SSL_read (1bb7e0, 1aa610, 1000, 1, ff0bd194, ffbec4b8) + 70
 00060524 prot_fill (ffffffff, 0, 1000, 19cfb0, ffbec7b8, 1) + 340
 00060d8c prot_read (0, ffbec7b8, 1000, 19cfb0, 1, ffbec7b8) + 6c
 00050894 message_copy_strict (0, 19cfb0, 8008c, ffffeff8, 1a17a8, ff09c648) + 64
 00044584 append_fromstream (ffbed830, 1a17a8, 9408c, 3cbce214, 1d0520, 1) + 14c
 0002fcec cmd_append (0, 0, ffbee8f0, 18c400, a, 0) + 4c0
 0002bf04 cmdloop  (18c800, 198400, 0, 0, 0, 0) + 3e8
 0002b8c4 service_main (0, ffbefd7c, ffbefd88, ff1f8000, 7, 2a1ec) + 290
 0002a334 main     (18c400, ffbefd7c, ffbefd88, 2, eb400, 0) + 618
 0002985c _start   (0, 0, 0, 0, 0, 0) + 5c

9273:   lmtpd -a
 ff098958 fcntl    (b, 7, ffbe9440)
 ff092f00 s_fcntl  (b, 7, 7, ffbe9440, b, 58d5c) + 1ec
 00058d5c lock_reopen (b, ffbe9570, ffbe94e8, ffbe94e4, e393e, ffbe9470) + 4c
 00035988 mailbox_lock_header (ffbeb658, 0, ffbeb658, 7efefeff, 81010100, ff0000) + cc
 00032d34 append_setup (0, ffbea658, 0, 0, 0, 0) + 54
 0002c284 deliver_mailbox (0, 1ca620, 2893, 0, 0, 0) + 124
 0002c7fc deliver  (1b12c0, 0, 0, 0, ffffefcc, ffbecbe8) + 410
 0002f6b0 lmtpmode (1841e0, 1b59d8, 1b5a30, 1d7380, 1a3d78, e10d8) + d2c
 0002ab8c service_main (2, ffbefd7c, ffbefd88, 2, 0, 2a5ac) + b8
 0002a6f4 main     (184000, ffbefd7c, ffbefd88, 2, e0c00, 0) + 618
 00029c1c _start   (0, 0, 0, 0, 0, 0) + 5c



Cyrus is configured as follows:

./configure
--with-com_err
--prefix=/var/cyrus/local
--with-cyrus-prefix=/var/cyrus/local/cyrus
--with-cyrus-group=mail
--with-sasl=/var/cyrus/local
--with-openssl=/usr/local/ssl2
--without-ucdsnmp
--with-dbdir=/usr/local/BerkeleyDB.3.3
--with-libwrap=/usr/local
--with-duplicate-db=skiplist
--with-mboxlist-db=skiplist
--with-seen-db=skiplist
--with-subs-db=flat
--with-tls-db=skiplist


Any advice?  (lsof is being installed now).

Thanks,

Pete