IMAP4 IDLE does not work. imapd_index == NULL ?

Tue Feb 3 21:36:44 EST 2015

Hi,

I've some very annoying bit of a problem with the cyrus imapd server and
hope that someone here has a good grasp on it's internals and can tell
me what's going on, confirm it's a bug, a matter lacking documentation
or something else entirely. I'm currently out of ideas (and freedom to
spend more time on this).

Thanks!

Problem:
The IMAP4 IDLE Command does not work as expected.
I.e.
- The cyrus imapd server advertised the IDLE CAPABILITY
- after sending IDLE it responds with "idling"
- when I send an e-mail to myself I see the e-mail arrive and being
filed in /var/log/mail.log
- but imapd does not ever notify me about the new e-mail. (rather my
client times out and recheck after >10 minutes)

Version: Cyrus IMAP v2.4.16-Debian-2.4.16-4+deb7u2
(which is the currently latest stable Debian Package)

I spent quite a few hours debugging this problem and learned the following:

1. I tried disabling idled and rely on just imapd's polling setting.
- I disabled idle in cyrus.conf
- i deleted the stale/remaining socket file idlesocket:
/var/run/cyrus/socket/idle
- I set imapidlepoll: 30

Still nothing happened, imapd never did notify me about my incoming test
messages.

2. I re-enabled idled and set CYRUS_VERBOSE=1
- verified that idled was working, socket was reachable, permissions ok

On connection to the imapd and start of any IDLE session, mail.log would
contain:

<logdate> mail cyrus/idled[1477]: imapd[1524]: IDLE_INIT '.'

On receipt of a new e-mail, mail.log would contain:

<logdate> mail cyrus/idled[1477]: IDLE_NOTIFY 'user.xyz'
<logdate> mail cyrus/idled[1477]: IDLE_NOTIFY 'user.uvw'

After playing around with strace, I discovered that idled was being
correctly notified about mailbox updates,
but DID NOT send signals to the idling imapd instances.

3. I had a look at idled.c and imapd.c from the latest stable sources.

After glancing over the code, it seems obvious that idled will only
notify those instances of imapd
about a mailbox update that have registered themselves for that very
mailbox.
i.e. the logmessage
  IDLE_INIT '.'
should have read
  IDLE_INIT 'user.xyz'

Since it did not, idled does not feel any imapd has registered for
updates to that mailbox thus we do not the signals.

Why is that ?

It seems that imapd purposefully DOES NOT send its mailbox name to
idled, if the test "if(imapd_index)" fails.

This is where I stopped browsing the code and tried to rebuild / recover
the index in case it became corrupted by the recent debian distribution
upgrade.
I tried running squatter and recover -r user.xyz serveral times as well
as restarted cyrus, but to no avail.

4. Looking to the Maillinglist for Help

.. any ideas or help is appreciated. :-)

config files attached.

-------------- next part --------------
# Debian Cyrus imapd.conf
# See imapd.conf(5) for more information and more options

# Configuration directory
configdirectory: /var/lib/cyrus

# Directories for proc and lock files
proc_path: /run/cyrus/proc
mboxname_lockpath: /run/cyrus/lock

# Which partition to use for default mailboxes
defaultpartition: default
partition-default: /var/spool/cyrus/mail

# News setup
partition-news: /var/spool/cyrus/news
newsspool: /var/spool/news

# Alternate namespace
# If enabled, activate the alternate namespace as documented in
# /usr/share/doc/cyrus-doc-2.4/html/altnamespace.html, where an user's
# subfolders are in the same level as the INBOX
# See also userprefix and sharedprefix on imapd.conf(5)
altnamespace: no

# UNIX Hierarchy Convention
# Set to yes, and cyrus will accept dots in names, and use the forward
# slash "/" to delimit levels of the hierarchy. This is done by converting
# internally all dots to "^", and all "/" to dots. So the "rabbit.holes"
# mailbox of user "helmer.fudd" is stored in "user.elmer^fud.rabbit^holes"
unixhierarchysep: no

# Rejecting illegal characters in headers
# Headers of RFC2882 messages must not have characters with the 8th bit
# set. However, too many badly-written MUAs generate this, including most
# spamware. Enable this to reject such messages.
#reject8bit: yes

# Munging illegal characters in headers
# Headers of RFC2882 messages must not have characters with the 8th bit
# set. However, too many badly-written MUAs generate this, including most
# spamware. If you kept reject8bit disabled, you can choose to leave the
# crappage untouched by disabling this (if you don't care that IMAP SEARCH
# won't work right anymore.
#munge8bit: no

# Forcing recipient user to lowercase
# Cyrus IMAPD is case-sensitive.  If all your mail users are in lowercase, it is
# probably a very good idea to set lmtp_downcase_rcpt to true.  This is set by 
# default, per RFC2821. This was not set by default in debian versions up to
# and including 2.2.12-4.
lmtp_downcase_rcpt: yes

# Uncomment the following and add the space-separated users who 
# have admin rights for all services.
admins: cyrus

# Space-separated list of users that have lmtp "admin" status (i.e. that
# can deliver email through TCP/IP lmtp). If specified, this parameter
# overrides the "admins" parameter above
#lmtp_admins: postman

# Space-separated list of users that have mupdate "admin" status, in
# addition to those in the admins: entry above. Note that mupdate slaves and 
# backends in a Murder cluster need to autenticate against the mupdate master
# as admin users.
#mupdate_admins: mupdateman

# Space-separated list of users that have imapd "admin" status, in
# addition to those in the admins: entry above
#imap_admins: cyrus

# Space-separated list of users that have sieve "admin" status, in
# addition to those in the admins: entry above
#sieve_admins: cyrus

# List of users and groups that are allowed to proxy for other users,
# seperated by spaces.  Any user listed in this will be allowed to login
# for any other user.  Like "admins:" above, you can have imap_proxyservers
# and sieve_proxyservers.
#proxyservers: cyrus

# No anonymous logins
allowanonymouslogin: no

# Minimum time between POP mail fetches in minutes
popminpoll: 0

# If nonzero, normal users may create their own IMAP accounts by creating
# the mailbox INBOX.  The user's quota is set to the value if it is positive,
# otherwise the user has unlimited quota.
autocreatequota: 0

# umask used by Cyrus programs
#essential for smooth debian operation to be 077
umask: 077

# Sendmail binary location
# DUE TO A BUG, Cyrus sends CRLF EOLs to this program. This breaks Exim 3. 
# For now, to work around the bug, set this to a wrapper that calls 
# /usr/sbin/sendmail -dropcr instead if you use Exim 3.
#sendmail: /usr/sbin/sendmail

# If enabled, cyrdeliver will look for Sieve scripts in user's home
# directories: ~user/.sieve.
sieveusehomedir: false

# If sieveusehomedir is false, this directory is searched for Sieve scripts.
sievedir: /var/spool/sieve

# notifyd(8) method to use for "MAIL" notifications.  If not set, "MAIL"
# notifications are disabled.  Valid methods are: null, log, zephyr
#mailnotifier: zephyr

# notifyd(8) method to use for "SIEVE" notifications.  If not set, "SIEVE"
# notifications are disabled.  This method is only used when no method is
# specified in the script.  Valid methods are null, log, zephyr, mailto
#sievenotifier: zephyr

# If enabled, the partitions will also be hashed, in addition to the hashing
# done on configuration directories. This is recommended if one partition has a
# very bushy mailbox tree.
hashimapspool: true

# Allow plaintext logins by default (SASL PLAIN)
allowplaintext: yes

# Force PLAIN/LOGIN authentication only
# (you need to uncomment this if you are not using an auxprop-based SASL
# mechanism.  saslauthd users, that means you!). And pay attention to
# sasl_minimum_layer and allowapop below, too.
sasl_mech_list: PLAIN LOGIN

# Allow use of the POP3 APOP authentication command.
# Note that this command requires that the plaintext passwords are 
# available in a SASL auxprop backend (eg. sasldb), and that the system
# can provide enough entropy (eg. from /dev/urandom) to create a challenge
# in the banner.
#allowapop: no

# The minimum SSF that the server will allow a client to negotiate. A
# value of 1 requires integrity protection; any higher value requires some
# amount of encryption.
sasl_minimum_layer: 0

# The maximum SSF that the server will allow a client to negotiate. A
# value of 1 requires integrity protection; any higher value requires some
# amount of encryption.
#sasl_maximum_layer: 256

# List of remote realms whose users may log in using cross-realm
# authentications. Seperate each realm name by a space. A cross-realm
# identity is considered any identity returned by SASL with an "@" in it.
# NOTE: To support multiple virtual domains on the same interface/IP,
# you need to list them all as loginreals. If you don't list them here,
# (most of) your users probably won't be able to log in.
#loginrealms: example.com
loginrealms: <------redacted-------->

# Enable virtual domain support.  If enabled, the user's domain will
# be determined by splitting a fully qualified userid at the last '@'
# or '%' symbol.  If the userid is unqualified, and the virtdomains
# option is set to "on", then the domain will be determined by doing
# a reverse lookup on the IP address of the incoming network
# interface, otherwise the user is assumed to be in the default
# domain (if set).
#virtdomains: userid

# The default domain for virtual domain support
# If the domain of a user can't be taken from its login and it can't
# be determined by doing a reverse lookup on the interface IP, this
# domain is used.
#defaultdomain: 
defaultdomain: <------redacted-------->

#
# SASL library options (these are handled directly by the SASL libraries,
# refer to SASL documentation for an up-to-date list of these)
#

# The mechanism(s) used by the server to verify plaintext passwords. Possible
# values are "saslauthd", "auxprop", "pwcheck" and "alwaystrue".  They
# are tried in order, you can specify more than one, separated by spaces.
#
# Do note that, since sasl will be run as user cyrus, you may have a lot of
# trouble to set this up right.
sasl_pwcheck_method: saslauthd

allowusermoves: yes

# What auxpropd plugins to load, if using sasl_pwcheck_method: auxprop
# by default, all plugins are tried (which is probably NOT what you want).
#sasl_auxprop_plugin: sasldb

# If enabled, the SASL library will automatically create authentication secrets
# when given a plaintext password. Refer to SASL documentation 
sasl_auto_transition: no

<------ SSL Certs redacted -------->

# The length of time (in minutes) that a TLS session will be cached for later
# reuse.  The maximum value is 1440 (24 hours), the default.  A value of 0 will
# disable session caching.
#tls_session_timeout: 1440
tls_session_timeout: 0

# The list of SSL/TLS ciphers to allow, in decreasing order of precedence.  
# The format of the string is described in ciphers(1).  The Debian default
# selects TLSv1 high-security ciphers only, and removes all anonymous ciphers
# from the list (because they provide no defense against man-in-the-middle
# attacks).  It also orders the list so that stronger ciphers come first.
tls_cipher_list: TLSv1+HIGH:!aNULL:@STRENGTH

# Require a client certificate for ALL services (imap, pop3, lmtp, sieve).
#tls_require_cert: false

# Require a client certificate for imap ONLY.
#imap_tls_require_cert: false

# Require a client certificate for pop3 ONLY.
#pop3_tls_require_cert: false

# Require a client certificate for lmtp ONLY.
#lmtp_tls_require_cert: false

# Require a client certificate for sieve ONLY.
#sieve_tls_require_cert: false

#
# Cyrus Murder cluster configuration
#
# Set the following options to the values needed for this server to
# autenticate against the mupdate master server:
# mupdate_server
# mupdate_port
# mupdate_username
# mupdate_authname
# mupdate_realm
# mupdate_password
# mupdate_retry_delay

##
## KEEP THESE IN SYNC WITH cyrus.conf
##
# Unix domain socket that lmtpd listens on.
lmtpsocket: /var/run/cyrus/socket/lmtp

# Unix domain socket that idled listens on.
idlesocket: /var/run/cyrus/socket/idle

# Unix domain socket that the new mail notification daemon listens on.
notifysocket: /var/run/cyrus/socket/notify

# Syslog prefix. Defaults to cyrus (so logging is done as cyrus/imap etc.)
syslog_prefix: cyrus

imapidlepoll: 30
-------------- next part --------------
# Debian defaults for Cyrus IMAP server/cluster implementation
# see cyrus.conf(5) for more information
#
# All the tcp services are tcpd-wrapped. see hosts_access(5)

START {
	# do not delete this entry!
	recover		cmd="/usr/sbin/cyrus ctl_cyrusdb -r"

	# this is only necessary if idlemethod is set to "idled" in imapd.conf
	idled		cmd="idled"

	# this is useful on backend nodes of a Murder cluster
	# it causes the backend to syncronize its mailbox list with
	# the mupdate master upon startup
	#mupdatepush   cmd="/usr/sbin/cyrus ctl_mboxlist -m"

	# this is recommended if using duplicate delivery suppression
	delprune	cmd="/usr/sbin/cyrus expire -E 3"
	# this is recommended if caching TLS sessions
	tlsprune	cmd="/usr/sbin/cyrus tls_prune"
}

# UNIX sockets start with a slash and are absolute paths
# you can use a maxchild=# to limit the maximum number of forks of a service
# you can use babysit=true and maxforkrate=# to keep tight tabs on the service
# most services also accept -U (limit number of reuses) and -T (timeout)
SERVICES {
	# --- Normal cyrus spool, or Murder backends ---
	# add or remove based on preferences
	imap		cmd="imapd -U 30" listen="imap" prefork=0 maxchild=100
	imaps		cmd="imapd -s -U 30" listen="imaps" prefork=0 maxchild=100
	pop3		cmd="pop3d -U 30" listen="pop3" prefork=0 maxchild=50
	pop3s		cmd="pop3d -s -U 30" listen="pop3s" prefork=0 maxchild=50
	nntp		cmd="nntpd -U 30" listen="nntp" prefork=0 maxchild=100
	#nntps		cmd="nntpd -s -U 30" listen="nntps" prefork=0 maxchild=100

	# At least one form of LMTP is required for delivery
	# (you must keep the Unix socket name in sync with imap.conf)
	#lmtp		cmd="lmtpd" listen="localhost:lmtp" prefork=0 maxchild=20
	lmtpunix	cmd="lmtpd" listen="/var/run/cyrus/socket/lmtp" prefork=0 maxchild=20
	# ----------------------------------------------

	# useful if you need to give users remote access to sieve
	# by default, we limit this to localhost in Debian
  	sieve		cmd="timsieved" listen="localhost:sieve" prefork=0 maxchild=100

	# this one is needed for the notification services
	notify		cmd="notifyd" listen="/var/run/cyrus/socket/notify" proto="udp" prefork=1

	# --- Murder frontends -------------------------
	# enable these and disable the matching services above, 
	# except for sieve (which deals automatically with Murder)

	# mupdate database service - must prefork at least 1
	# (mupdate slaves)
	#mupdate       cmd="mupdate" listen=3905 prefork=1
	# (mupdate master, only one in the entire cluster)
	#mupdate       cmd="mupdate -m" listen=3905 prefork=1

	# proxies that will connect to the backends
	#imap		cmd="proxyd" listen="imap" prefork=0 maxchild=100
	#imaps		cmd="proxyd -s" listen="imaps" prefork=0 maxchild=100
	#pop3		cmd="pop3proxyd" listen="pop3" prefork=0 maxchild=50
	#pop3s		cmd="pop3proxyd -s" listen="pop3s" prefork=0 maxchild=50
	#lmtp		cmd="lmtpproxyd" listen="lmtp" prefork=1 maxchild=20
	# ----------------------------------------------
}

EVENTS {
	# this is required
	checkpoint	cmd="/usr/sbin/cyrus ctl_cyrusdb -c" period=30

	# this is only necessary if using duplicate delivery suppression
	delprune	cmd="/usr/sbin/cyrus expire -E 3" at=0401

	# this is only necessary if caching TLS sessions
	tlsprune	cmd="/usr/sbin/cyrus tls_prune" at=0401

	# indexing of mailboxes for server side fulltext searches

	# reindex changed mailboxes (fulltext) approximately every other hour
	squatter_1	cmd="/usr/bin/nice -n 19 /usr/sbin/cyrus squatter -s" period=120

	# reindex all mailboxes (fulltext) daily
	squatter_a	cmd="/usr/sbin/cyrus squatter" at=0517
}