IMAP4 IDLE does not work. imapd_index == NULL ?

Bernhard T cyrus at
Tue Feb 3 21:36:44 EST 2015


I've some very annoying bit of a problem with the cyrus imapd server and
hope that someone here has a good grasp on it's internals and can tell
me what's going on, confirm it's a bug, a matter lacking documentation
or something else entirely. I'm currently out of ideas (and freedom to
spend more time on this).


The IMAP4 IDLE Command does not work as expected.
- The cyrus imapd server advertised the IDLE CAPABILITY
- after sending IDLE it responds with "idling"
- when I send an e-mail to myself I see the e-mail arrive and being
filed in /var/log/mail.log
- but imapd does not ever notify me about the new e-mail. (rather my
client times out and recheck after >10 minutes)

Version: Cyrus IMAP v2.4.16-Debian-2.4.16-4+deb7u2
(which is the currently latest stable Debian Package)

I spent quite a few hours debugging this problem and learned the following:

1. I tried disabling idled and rely on just imapd's polling setting.
- I disabled idle in cyrus.conf
- i deleted the stale/remaining socket file idlesocket:
- I set imapidlepoll: 30

Still nothing happened, imapd never did notify me about my incoming test

2. I re-enabled idled and set CYRUS_VERBOSE=1
- verified that idled was working, socket was reachable, permissions ok

On connection to the imapd and start of any IDLE session, mail.log would

<logdate> mail cyrus/idled[1477]: imapd[1524]: IDLE_INIT '.'

On receipt of a new e-mail, mail.log would contain:

<logdate> mail cyrus/idled[1477]: IDLE_NOTIFY ''
<logdate> mail cyrus/idled[1477]: IDLE_NOTIFY 'user.uvw'

After playing around with strace, I discovered that idled was being
correctly notified about mailbox updates,
but DID NOT send signals to the idling imapd instances.

3. I had a look at idled.c and imapd.c from the latest stable sources.

After glancing over the code, it seems obvious that idled will only
notify those instances of imapd
about a mailbox update that have registered themselves for that very
i.e. the logmessage
should have read

Since it did not, idled does not feel any imapd has registered for
updates to that mailbox thus we do not the signals.

Why is that ?

It seems that imapd purposefully DOES NOT send its mailbox name to
idled, if the test "if(imapd_index)" fails.

This is where I stopped browsing the code and tried to rebuild / recover
the index in case it became corrupted by the recent debian distribution
I tried running squatter and recover -r serveral times as well
as restarted cyrus, but to no avail.

4. Looking to the Maillinglist for Help

.. any ideas or help is appreciated. :-)

config files attached.

-------------- next part --------------
# Debian Cyrus imapd.conf
# See imapd.conf(5) for more information and more options

# Configuration directory
configdirectory: /var/lib/cyrus

# Directories for proc and lock files
proc_path: /run/cyrus/proc
mboxname_lockpath: /run/cyrus/lock

# Which partition to use for default mailboxes
defaultpartition: default
partition-default: /var/spool/cyrus/mail

# News setup
partition-news: /var/spool/cyrus/news
newsspool: /var/spool/news

# Alternate namespace
# If enabled, activate the alternate namespace as documented in
# /usr/share/doc/cyrus-doc-2.4/html/altnamespace.html, where an user's
# subfolders are in the same level as the INBOX
# See also userprefix and sharedprefix on imapd.conf(5)
altnamespace: no

# UNIX Hierarchy Convention
# Set to yes, and cyrus will accept dots in names, and use the forward
# slash "/" to delimit levels of the hierarchy. This is done by converting
# internally all dots to "^", and all "/" to dots. So the "rabbit.holes"
# mailbox of user "helmer.fudd" is stored in "user.elmer^fud.rabbit^holes"
unixhierarchysep: no

# Rejecting illegal characters in headers
# Headers of RFC2882 messages must not have characters with the 8th bit
# set. However, too many badly-written MUAs generate this, including most
# spamware. Enable this to reject such messages.
#reject8bit: yes

# Munging illegal characters in headers
# Headers of RFC2882 messages must not have characters with the 8th bit
# set. However, too many badly-written MUAs generate this, including most
# spamware. If you kept reject8bit disabled, you can choose to leave the
# crappage untouched by disabling this (if you don't care that IMAP SEARCH
# won't work right anymore.
#munge8bit: no

# Forcing recipient user to lowercase
# Cyrus IMAPD is case-sensitive.  If all your mail users are in lowercase, it is
# probably a very good idea to set lmtp_downcase_rcpt to true.  This is set by 
# default, per RFC2821. This was not set by default in debian versions up to
# and including 2.2.12-4.
lmtp_downcase_rcpt: yes

# Uncomment the following and add the space-separated users who 
# have admin rights for all services.
admins: cyrus

# Space-separated list of users that have lmtp "admin" status (i.e. that
# can deliver email through TCP/IP lmtp). If specified, this parameter
# overrides the "admins" parameter above
#lmtp_admins: postman

# Space-separated list of users that have mupdate "admin" status, in
# addition to those in the admins: entry above. Note that mupdate slaves and 
# backends in a Murder cluster need to autenticate against the mupdate master
# as admin users.
#mupdate_admins: mupdateman

# Space-separated list of users that have imapd "admin" status, in
# addition to those in the admins: entry above
#imap_admins: cyrus

# Space-separated list of users that have sieve "admin" status, in
# addition to those in the admins: entry above
#sieve_admins: cyrus

# List of users and groups that are allowed to proxy for other users,
# seperated by spaces.  Any user listed in this will be allowed to login
# for any other user.  Like "admins:" above, you can have imap_proxyservers
# and sieve_proxyservers.
#proxyservers: cyrus

# No anonymous logins
allowanonymouslogin: no

# Minimum time between POP mail fetches in minutes
popminpoll: 0

# If nonzero, normal users may create their own IMAP accounts by creating
# the mailbox INBOX.  The user's quota is set to the value if it is positive,
# otherwise the user has unlimited quota.
autocreatequota: 0

# umask used by Cyrus programs
#essential for smooth debian operation to be 077
umask: 077

# Sendmail binary location
# DUE TO A BUG, Cyrus sends CRLF EOLs to this program. This breaks Exim 3. 
# For now, to work around the bug, set this to a wrapper that calls 
# /usr/sbin/sendmail -dropcr instead if you use Exim 3.
#sendmail: /usr/sbin/sendmail

# If enabled, cyrdeliver will look for Sieve scripts in user's home
# directories: ~user/.sieve.
sieveusehomedir: false

# If sieveusehomedir is false, this directory is searched for Sieve scripts.
sievedir: /var/spool/sieve

# notifyd(8) method to use for "MAIL" notifications.  If not set, "MAIL"
# notifications are disabled.  Valid methods are: null, log, zephyr
#mailnotifier: zephyr

# notifyd(8) method to use for "SIEVE" notifications.  If not set, "SIEVE"
# notifications are disabled.  This method is only used when no method is
# specified in the script.  Valid methods are null, log, zephyr, mailto
#sievenotifier: zephyr

# If enabled, the partitions will also be hashed, in addition to the hashing
# done on configuration directories. This is recommended if one partition has a
# very bushy mailbox tree.
hashimapspool: true

# Allow plaintext logins by default (SASL PLAIN)
allowplaintext: yes

# Force PLAIN/LOGIN authentication only
# (you need to uncomment this if you are not using an auxprop-based SASL
# mechanism.  saslauthd users, that means you!). And pay attention to
# sasl_minimum_layer and allowapop below, too.
sasl_mech_list: PLAIN LOGIN

# Allow use of the POP3 APOP authentication command.
# Note that this command requires that the plaintext passwords are 
# available in a SASL auxprop backend (eg. sasldb), and that the system
# can provide enough entropy (eg. from /dev/urandom) to create a challenge
# in the banner.
#allowapop: no

# The minimum SSF that the server will allow a client to negotiate. A
# value of 1 requires integrity protection; any higher value requires some
# amount of encryption.
sasl_minimum_layer: 0

# The maximum SSF that the server will allow a client to negotiate. A
# value of 1 requires integrity protection; any higher value requires some
# amount of encryption.
#sasl_maximum_layer: 256

# List of remote realms whose users may log in using cross-realm
# authentications. Seperate each realm name by a space. A cross-realm
# identity is considered any identity returned by SASL with an "@" in it.
# NOTE: To support multiple virtual domains on the same interface/IP,
# you need to list them all as loginreals. If you don't list them here,
# (most of) your users probably won't be able to log in.
loginrealms: <------redacted-------->

# Enable virtual domain support.  If enabled, the user's domain will
# be determined by splitting a fully qualified userid at the last '@'
# or '%' symbol.  If the userid is unqualified, and the virtdomains
# option is set to "on", then the domain will be determined by doing
# a reverse lookup on the IP address of the incoming network
# interface, otherwise the user is assumed to be in the default
# domain (if set).
#virtdomains: userid

# The default domain for virtual domain support
# If the domain of a user can't be taken from its login and it can't
# be determined by doing a reverse lookup on the interface IP, this
# domain is used.
defaultdomain: <------redacted-------->

# SASL library options (these are handled directly by the SASL libraries,
# refer to SASL documentation for an up-to-date list of these)

# The mechanism(s) used by the server to verify plaintext passwords. Possible
# values are "saslauthd", "auxprop", "pwcheck" and "alwaystrue".  They
# are tried in order, you can specify more than one, separated by spaces.
# Do note that, since sasl will be run as user cyrus, you may have a lot of
# trouble to set this up right.
sasl_pwcheck_method: saslauthd

allowusermoves: yes

# What auxpropd plugins to load, if using sasl_pwcheck_method: auxprop
# by default, all plugins are tried (which is probably NOT what you want).
#sasl_auxprop_plugin: sasldb

# If enabled, the SASL library will automatically create authentication secrets
# when given a plaintext password. Refer to SASL documentation 
sasl_auto_transition: no

<------ SSL Certs redacted -------->

# The length of time (in minutes) that a TLS session will be cached for later
# reuse.  The maximum value is 1440 (24 hours), the default.  A value of 0 will
# disable session caching.
#tls_session_timeout: 1440
tls_session_timeout: 0

# The list of SSL/TLS ciphers to allow, in decreasing order of precedence.  
# The format of the string is described in ciphers(1).  The Debian default
# selects TLSv1 high-security ciphers only, and removes all anonymous ciphers
# from the list (because they provide no defense against man-in-the-middle
# attacks).  It also orders the list so that stronger ciphers come first.
tls_cipher_list: TLSv1+HIGH:!aNULL:@STRENGTH

# Require a client certificate for ALL services (imap, pop3, lmtp, sieve).
#tls_require_cert: false

# Require a client certificate for imap ONLY.
#imap_tls_require_cert: false

# Require a client certificate for pop3 ONLY.
#pop3_tls_require_cert: false

# Require a client certificate for lmtp ONLY.
#lmtp_tls_require_cert: false

# Require a client certificate for sieve ONLY.
#sieve_tls_require_cert: false

# Cyrus Murder cluster configuration
# Set the following options to the values needed for this server to
# autenticate against the mupdate master server:
# mupdate_server
# mupdate_port
# mupdate_username
# mupdate_authname
# mupdate_realm
# mupdate_password
# mupdate_retry_delay

# Unix domain socket that lmtpd listens on.
lmtpsocket: /var/run/cyrus/socket/lmtp

# Unix domain socket that idled listens on.
idlesocket: /var/run/cyrus/socket/idle

# Unix domain socket that the new mail notification daemon listens on.
notifysocket: /var/run/cyrus/socket/notify

# Syslog prefix. Defaults to cyrus (so logging is done as cyrus/imap etc.)
syslog_prefix: cyrus

imapidlepoll: 30
-------------- next part --------------
# Debian defaults for Cyrus IMAP server/cluster implementation
# see cyrus.conf(5) for more information
# All the tcp services are tcpd-wrapped. see hosts_access(5)

	# do not delete this entry!
	recover		cmd="/usr/sbin/cyrus ctl_cyrusdb -r"
	# this is only necessary if idlemethod is set to "idled" in imapd.conf
	idled		cmd="idled"

	# this is useful on backend nodes of a Murder cluster
	# it causes the backend to syncronize its mailbox list with
	# the mupdate master upon startup
	#mupdatepush   cmd="/usr/sbin/cyrus ctl_mboxlist -m"

	# this is recommended if using duplicate delivery suppression
	delprune	cmd="/usr/sbin/cyrus expire -E 3"
	# this is recommended if caching TLS sessions
	tlsprune	cmd="/usr/sbin/cyrus tls_prune"

# UNIX sockets start with a slash and are absolute paths
# you can use a maxchild=# to limit the maximum number of forks of a service
# you can use babysit=true and maxforkrate=# to keep tight tabs on the service
# most services also accept -U (limit number of reuses) and -T (timeout)
	# --- Normal cyrus spool, or Murder backends ---
	# add or remove based on preferences
	imap		cmd="imapd -U 30" listen="imap" prefork=0 maxchild=100
	imaps		cmd="imapd -s -U 30" listen="imaps" prefork=0 maxchild=100
	pop3		cmd="pop3d -U 30" listen="pop3" prefork=0 maxchild=50
	pop3s		cmd="pop3d -s -U 30" listen="pop3s" prefork=0 maxchild=50
	nntp		cmd="nntpd -U 30" listen="nntp" prefork=0 maxchild=100
	#nntps		cmd="nntpd -s -U 30" listen="nntps" prefork=0 maxchild=100

	# At least one form of LMTP is required for delivery
	# (you must keep the Unix socket name in sync with imap.conf)
	#lmtp		cmd="lmtpd" listen="localhost:lmtp" prefork=0 maxchild=20
	lmtpunix	cmd="lmtpd" listen="/var/run/cyrus/socket/lmtp" prefork=0 maxchild=20
	# ----------------------------------------------

	# useful if you need to give users remote access to sieve
	# by default, we limit this to localhost in Debian
  	sieve		cmd="timsieved" listen="localhost:sieve" prefork=0 maxchild=100

	# this one is needed for the notification services
	notify		cmd="notifyd" listen="/var/run/cyrus/socket/notify" proto="udp" prefork=1

	# --- Murder frontends -------------------------
	# enable these and disable the matching services above, 
	# except for sieve (which deals automatically with Murder)

	# mupdate database service - must prefork at least 1
	# (mupdate slaves)
	#mupdate       cmd="mupdate" listen=3905 prefork=1
	# (mupdate master, only one in the entire cluster)
	#mupdate       cmd="mupdate -m" listen=3905 prefork=1

	# proxies that will connect to the backends
	#imap		cmd="proxyd" listen="imap" prefork=0 maxchild=100
	#imaps		cmd="proxyd -s" listen="imaps" prefork=0 maxchild=100
	#pop3		cmd="pop3proxyd" listen="pop3" prefork=0 maxchild=50
	#pop3s		cmd="pop3proxyd -s" listen="pop3s" prefork=0 maxchild=50
	#lmtp		cmd="lmtpproxyd" listen="lmtp" prefork=1 maxchild=20
	# ----------------------------------------------

	# this is required
	checkpoint	cmd="/usr/sbin/cyrus ctl_cyrusdb -c" period=30

	# this is only necessary if using duplicate delivery suppression
	delprune	cmd="/usr/sbin/cyrus expire -E 3" at=0401

	# this is only necessary if caching TLS sessions
	tlsprune	cmd="/usr/sbin/cyrus tls_prune" at=0401
	# indexing of mailboxes for server side fulltext searches

	# reindex changed mailboxes (fulltext) approximately every other hour
	squatter_1	cmd="/usr/bin/nice -n 19 /usr/sbin/cyrus squatter -s" period=120

	# reindex all mailboxes (fulltext) daily
	squatter_a	cmd="/usr/sbin/cyrus squatter" at=0517

More information about the Cyrus-devel mailing list