Design of Event notification

[Sébastien Michel]

>> => We have defined lot of configuration settings (may be too much for
>> common usages) :
>>  - eventnotifier, "off", STRING : Enables event notification with the
>> given notification method (see notifyd). i.e mail, log, activemq
>>  - event_subfolder, 0, SWITCH : Enables event notification for subfolders too
>>  - event_exclude_folders, "", STRING : Don't send any notification for
>> the given folders (i.e Spam folder)
>>  - event_ignore_flags, "", STRING : Don't send notification on event
>> FlagsClear/FlagsSet for these flags (i.e \Deleted)
>
> Are these space separated?  Comma separated?  Tab separated?  how do you
> handle spaces in folder names?

This is comma separated. space and tab around comma are ignored. This
not conform to the modified UTF-7 and needs to be rewritten. It was
not a problem for us since we exclude only us-ascii folders.
it makes sense to exclude folders from their SPECIAL-USE metadata as
proposed below.

>>
>> version=1
>> event=MessageNew
>> host=host0302
>> timestamp=1190272093123
>> mailboxID=user.mailbox1.Personal
>> messages=3
>> newMessages=2
>> uidnext=124
>> uidset=123
>
> Seems sane enough.  I'd be tempted to do it with either a standard
> (XML/JSON as suggested below) or even as an IMAP parameter list.
> Whatever you choose, something that supports full encodings of
> everything.  In particular mailboxID... is that IMAP-UTF7 as
> in the internal namespace?

My preference is for JSON because it is more compact than XML.
MailboxID has the value stored in mailbox list database.

>> - Discuss about our proprietary event types/parameters. There are two
>> options (or a mix of both) :
>>     -> keep it private in our source code.
>>     -> prefix with vnd. to be RFC compliant
>
> prefix with vnd. looks good to me, or even just have them disabled by
> default in the config.  A site that's willing to step ourside RFC
> complience can then just toggle them on.

I agree.

>> - FlagsSet and FlagsClear are not well supported. In particular the
>> support of the keyword FLAGS to replace existing flags (OK for +FLAGS
>> and -FLAGS)
>
> Is that required?  +FLAGS and -FLAGS are generally considered to be enough
> these days, with everyone agreeing that "FLAGS" as an explicit set to this
> value is annoying and unnecessary.

We made the same assumption !

>> - Need tests and probably bugs fix on shared folders and public
>> folders. In particular the management of the flag \Seen.
>
> Yes, probably.  I can help you a bit with that, because I know the
> shared \Seen stuff pretty well.

thanks. It could be helpful.

>> 4) redesign the code
>> - Most of the code is located in two new files : msgevent.h and
>> msgevent.c. Function calls are scattered throughout the Cyrus's code.
>> Most of the calls are done from a suitable location to avoid I/O
>> overhead. ie when the mailbox is open to get event parameter values.
>> However, calls should be made with the aim of eventually implementing
>> the RFC 5465 (The IMAP NOTIFY Extension).
>
> Sounds good to me.  Again, I can probably help here, because I know
> the mailbox layer inside out, and I also have an idea where I want to
> take it!

I would not fail to ask some questions if needed :)

>> - We have choose to use the Cyrus's internal form of the mailbox for
>> mailboxID parameter in order to be independent of the current
>> namespace and folder separator settings. To be discussed.
>
> No discussion required.  This is the only sane way to do it.  There
> may be some value in an optional externalName parameter which does
> namespace translation in Cyrus, but otherwise the unique ID is the
> internal name.  Make sure virtdomains are supported too.

This choice seems to be not conform to the RFC :
     "mailboxID
           Included in events that affect mailboxes.  A URI describing the
           mailbox.  In the case of MailboxRename, this refers to the new
           name."
but it does not define the URI format of any kind... Given that the
RFC defines the 'uri' parameter which is based on IMAP URL

>> - Simplify the behavior. Do people need to define a context (queue)
>> per event type ? Moreover several contexts per event type (and thus
>> send each notification in several queues) ?
>
> How much of this can be done outside Cyrus?  I don't know the queue
> stuff well enough to be sure, but I would think you can feed events
> to an external daemon which then multiplexes and filters them to
> the consumers.

Yes. Moreover we provide this setting but we don't use it!

> This external daemon could also do any protocol
> translation into XML, JSON, whatever.  It's a prime candidate for
> being written in a non-blocking style, since it's stateless other
> than the filter/target definitions.

You are right !


>> 6) and later provide more connectors to notifyd like support of AMQP protocol
>
> Again - a candidate for external daemon?  I think notifyd and idled
> should be non-blocking a bit more anyway.  And I'd really REALLY like
> for 'signal 10' from idled to die.  Every time we lose a frontend and
> drop 50k connections across our backends, we get a bunch of random
> processes being attacked with signal 10 as idled spews out its
> connection attempts.

Yeah. Our activemq connector is non-blocking.
The daemon notifyd we thought appropriate. Which external daemon ? One
new Cyrus's daemon for this feature. Or a chain of daemons like this :
imapd -> Notifyd -> Other daemon ?
And a default protocol must be choosen. I don't think that 'mail' or
'log' connectors are good candidates.

>> I will create a new branch (based on Cyrus master) in our github for
>> this feature (https://github.com/worldline-messaging)
>
> Fantastic, I look forward to seeing the code!
>

Thanks a lot for your reply!