Message store events notification

[Sébastien Michel]

Hi,
Below an update of my work to support out-band event notification in
Cyrus based on RFC 5423.
All event types together with their parameters defined in the RFC are
fully supported. Implementation is done to avoid any I/O overhead.

1) event types
MessageAppend, MessageNew
 - parameters: mailboxID
 - optional parameters: bodyStructure, messageContent, messages,
messageSize, modseq, service, timestamp, uidnext, vnd.cmu.midset,
vnd.cmu.newMessages
 - comment: One notification is send per message. Several
notifications in case of IMAP MULTIAPPEND

MessageExpire
 - parameters: mailboxID, uidset
 - optional parameters: messages, modseq, service, timestamp, uidnext,
vnd.cmu.midset, vnd.cmu.newMessages
 - comment: modseq may be included if referring to a specific message.
uidset if referring to a set of messages

MessageExpunge
 - parameters: mailboxID, uidset
 - optional parameters: messages, modseq, service, timestamp, uidnext,
vnd.cmu.midset, vnd.cmu.newMessages
 - comment: modseq may be included if referring to a specific message.
uidset if referring to a set of messages

QuotaExceed, QuotaWithin
 - parameters: diskQuota, diskUsed, mailboxID, maxMessages, messages
 - optional parameters: service, timestamp
 - comment: include diskQuota and diskUsed if quota STORAGE exceed.
maxMessages and messages if quota MESSAGE exceed. I don't support
quota ANNOTSTORAGE, however we could define proprietary parameters for
that like vnd.cmu.annotationQuota and vnd.cmu.annotationUsed.

QuotaChange
 - parameters: diskQuota, mailboxID, maxMessages
 - optional parameters: diskUsed, messages, service, timestamp
 - comment: diskQuota and maxMessages may be included together
depending the SETQUOTA arguments. QuotaExceed, QuotaWithin and
QuotaChange are not enabled by default.

MessageRead, MessageTrash
 - parameters: mailboxID, uidset
 - optional parameters: messages, modseq, service, timestamp, uidnext,
vnd.cmu.midset, vnd.cmu.newMessages
 - comment: MessageRead and MessageTrash replace FlagsSet for \Seen
and \Deleted flags. MessageRead notification is also send on IMAP
FETCH.

FlagsSet, FlagsClear
 - parameters: mailboxID, uidset, flagNames
 - optional parameters: messages, modseq, service, timestamp, uidnext,
vnd.cmu.midset, vnd.cmu.newMessages
 - comment: support +FLAGS, -FLAGS and FLAGS arguments to IMAP STORE.
May send FLagsSet and FlagsClear for FLAGS. May also send MessageTrash
and MessageRead depending the flags in argument.

Login, Logout
 - parameters: serverDomain, serverPort, user
 - optional parameters: timestamp, service, clientIP, clientPort
 - comment: Login and Logout are not enabled by default

MailboxCreate, MailboxDelete
 - parameters: mailboxID
 - optional parameters: service, timestamp

MailboxRename
 - parameters: mailboxID, oldMailboxID
 - optional parameters: service, timestamp

MailboxSubscribe, MailboxUnSubscribe
 - parameters: mailboxID, user
 - optional parameters: service, timestamp

and additional vnd.cmu.MessageCopy.
 - parameters: mailboxID, oldMailboxID, vnd.cmu.oldUidset, uidset
 - optional parameters: messages, modseq, service, timestamp, uidnext,
vnd.cmu.midset, vnd.cmu.newMessages
 - comment: modseq may be included if referring to a specific message.
uidset if referring to a set of messages

2) configuration
The setting is lighter than our current code in Cyrus 2.3 with I hope
good defaults :
- eventnotifier (NULL) :
        Notifyd(8) method to use for "EVENT" notifications which are
based on the RFC 5423.  If not set, "EVENT" notifications are
disabled.

- event_groups ("message mailbox") :
        Space-separated list of groups of related events to turn on
notification: message", "quota", "flags", "access", "mailbox"

- event_extra_params ("timestamp") :
        Space-separated list of extra parameters to add to any
appropriated event : "bodyStructure", "clientAddress", "diskUsed",
"flagNames", "messageContent", "messageSize", "messages", "modseq",
"service", "timestamp", "uidnext", "vnd.cmu.host", "vnd.cmu.midset",
"vnd.cmu.newMessages"

- event_exclude_flags (NULL) :
        Don't send event notification for given IMAP flag(s)

- event_exclude_folders (NULL) :
        Don't send event notification for given folder(s). Set ALL for
any folder

- event_content_inclusion_mode ("standard") :
        The mode in which message content may be included with
MessageAppend and MessageNew. "standard" mode is the default behavior
in which message is included up to a size with the notification. In
"message" mode, the message is included and may be truncated to a
size. In "header" mode, it includes headers truncated to a size. In
"body" mode, it includes body truncated to a size. In "headerbody"
mode, it includes full headers and body truncated to a size

- event_content_size (0) :
        Truncate the message content that may be included with
MessageAppend and MessageNew. Set 0 to include the entire message
itself

- event_timestamp_format ("iso8601") :
        Format event timestamp in ISO 8601 format or print the number
of seconds since the Epoch (UTC) both with fractions of second

3) message format
Notification format is JSON.
examples:
"{"event":"Login","timestamp":"2011-11-22T17:19:36.849Z","service":"imappublic","serverDomain":"127.0.0.3","serverPort":143,"clientIP":"127.0.0.1","clientPort":56197,"user":"testtmpuser"}"

"{"event":"MessageRead","timestamp":"2011-11-22T17:19:36.854Z","service":"imappublic","mailboxID":"imap:\/\/127.0.0.3\/user.testtmpuser;UIDVALIDITY=1321982376","messages":5,"vnd.cmu.newMessages":2,"uidnext":6,"uidset":"1
2","vnd.cmu.midset":"<cmu-lmtpd-3821-1321982376-0 at 127.0.0.3>
<cmu-lmtpd-3821-1321982376-1 at 127.0.0.3>"}"

 "{"event":"FlagsSet","timestamp":"2011-11-22T17:19:36.854Z","service":"imappublic","mailboxID":"imap:\/\/127.0.0.3\/user.testtmpuser;UIDVALIDITY=1321982376","messages":5,"vnd.cmu.newMessages":2,"uidnext":6,"uidset":"1
2","vnd.cmu.midset":"<cmu-lmtpd-3821-1321982376-0 at 127.0.0.3>
<cmu-lmtpd-3821-1321982376-1 at 127.0.0.3>","flagNames":"\\Draft"}"

 "{"event":"FlagsClear","timestamp":"2011-11-22T17:19:36.866Z","service":"imappublic","mailboxID":"imap:\/\/127.0.0.3\/user.testtmpuser;UIDVALIDITY=1321982376\/;UID=1","messages":5,"vnd.cmu.newMessages":3,"uidnext":6,"vnd.cmu.midset":"<cmu-lmtpd-3821-1321982376-0 at 127.0.0.3>","flagNames":"\\Seen","modseq":8}"

 "{"event":"MessageTrash","timestamp":"2011-11-22T17:19:36.866Z","service":"imappublic","mailboxID":"imap:\/\/127.0.0.3\/user.testtmpuser;UIDVALIDITY=1321982376\/;UID=1","messages":5,"vnd.cmu.newMessages":3,"uidnext":6,"vnd.cmu.midset":"<cmu-lmtpd-3821-1321982376-0 at 127.0.0.3>","modseq":8}"

"{"event":"MessageExpunge","timestamp":"2011-11-22T17:19:36.873Z","service":"imappublic","mailboxID":"imap:\/\/127.0.0.3\/user.testtmpuser;UIDVALIDITY=1321982376\/;UID=1","messages":4,"vnd.cmu.newMessages":2,"uidnext":6,"vnd.cmu.midset":"<cmu-lmtpd-3821-1321982376-0 at 127.0.0.3>","modseq":9}"
  "{"event":"Logout","timestamp":"2011-11-22T17:19:36.880Z","service":"imappublic","serverDomain":"127.0.0.3","serverPort":143,"user":"testtmpuser"}"


4) choices of design
- There is no event notification on internal cyrus behaviors :
rollback on mailbox creation, replication on cyrus slave, deletion of
delayed message, murder proxy, and so on.
- I wonder about the IMAP URL format of the messageID parameter. I
made use of cyrus' internal format. But I should rather use namespace
translation to get the external format.
I encounter some issues to access to namespace structure in some
places in the code (QuotaExceed in quota_db.c, MessageExpunge in
mailbox.c and so on). It may be centralized in mboxevent.c before
sending the event, but namespace dependencies are unknown here. Thus,
code to handle namespace should be rewritten :(
- I chose JSON as default format. It may not be the best choice to
implement in-band event notification like IMAP NOTIFY

I will write some documentation based on informations above.
I rebased the msgevent branch on current master :
https://github.com/worldline-messaging/cyrus-imapd/compare/msgevent

any comments or suggestions are welcome.

Thanks,
Sébastien

--
Atos Worldline