Design of Event notification

[Bron Gondwana]

On Tue, Jun 28, 2011 at 06:50:53PM +0200, Sébastien Michel wrote:
> We have developed some time ago a notification mechanism on mailbox
> events. This piece of software is based on the RFC 5423 - Internet
> Message Store Events - that defines a number of event types and event
> parameters.

This looks awesome!

> => We have defined lot of configuration settings (may be too much for
> common usages) :
>  - eventnotifier, "off", STRING : Enables event notification with the
> given notification method (see notifyd). i.e mail, log, activemq
>  - event_subfolder, 0, SWITCH : Enables event notification for subfolders too
>  - event_exclude_folders, "", STRING : Don't send any notification for
> the given folders (i.e Spam folder)
>  - event_ignore_flags, "", STRING : Don't send notification on event
> FlagsClear/FlagsSet for these flags (i.e \Deleted)

Are these space separated?  Comma separated?  Tab separated?  how do you
handle spaces in folder names?

>  - event_notifspam, 1, SWITCH : Enables event notification for email
> rated as spam
>  - event_XXX_params, "", STRING : Enables event notification of the
> type XXX and adds non standard parameters or those that are optional
> in the RFC for this event.
>
>  - event_XXX_ctx, "", STRING : Defines one or several contexts for the
> notification of type XXX. Name(s) of the queue(s) for activemq.

It seems to me that it makes sense to have good defaults, and offer
just about everything as tunable here.  Don't worry too much about
creating extra names - particularly of the "XXX" variety.

> => the RFC doesn't define any notification format. Today we send
> key/value pairs of parameters like this : (example for new message
> delivered by LMTP in folder Personal)
>
> version=1
> event=MessageNew
> host=host0302
> timestamp=1190272093123
> mailboxID=user.mailbox1.Personal
> messages=3
> newMessages=2
> uidnext=124
> uidset=123

Seems sane enough.  I'd be tempted to do it with either a standard
(XML/JSON as suggested below) or even as an IMAP parameter list.
Whatever you choose, something that supports full encodings of
everything.  In particular mailboxID... is that IMAP-UTF7 as
in the internal namespace?

> => how to donate the code ?
> 
> 1) be RFC compliant
> - Some event types are missing, and some mandatory event parameters too.

It would be nice to add the mandatory event parameters if possible.
What are they?

> - Discuss about our proprietary event types/parameters. There are two
> options (or a mix of both) :
>     -> keep it private in our source code.
>     -> prefix with vnd. to be RFC compliant

prefix with vnd. looks good to me, or even just have them disabled by
default in the config.  A site that's willing to step ourside RFC
complience can then just toggle them on.

> - FlagsSet and FlagsClear are not well supported. In particular the
> support of the keyword FLAGS to replace existing flags (OK for +FLAGS
> and -FLAGS)

Is that required?  +FLAGS and -FLAGS are generally considered to be enough
these days, with everyone agreeing that "FLAGS" as an explicit set to this
value is annoying and unnecessary.

> - Need tests and probably bugs fix on shared folders and public
> folders. In particular the management of the flag \Seen.

Yes, probably.  I can help you a bit with that, because I know the
shared \Seen stuff pretty well.

> 2) possibly rename or simplify the configuration settings. Possibly
> use IMAP METADATA for some settings. To be discussed.

Only use IMAP METADATA if it makes sense to be per-folder, otherwise
it just becomes an extra headache.  The one place this might make sense
is for suppressing individual folders rather than the "ignore these
folder names".  I'm thinking particularly of multi-lingual servers
where people might be using SPECIAL-USE metadata rather than well
known folder names.

> 3) provide more notification format like XML and JSON

Surely one good format, and if people want others they can whack
a translation daemon in the mix.

> 4) redesign the code
> - Most of the code is located in two new files : msgevent.h and
> msgevent.c. Function calls are scattered throughout the Cyrus's code.
> Most of the calls are done from a suitable location to avoid I/O
> overhead. ie when the mailbox is open to get event parameter values.
> However, calls should be made with the aim of eventually implementing
> the RFC 5465 (The IMAP NOTIFY Extension).

Sounds good to me.  Again, I can probably help here, because I know
the mailbox layer inside out, and I also have an idea where I want to
take it!

> - We have choose to use the Cyrus's internal form of the mailbox for
> mailboxID parameter in order to be independent of the current
> namespace and folder separator settings. To be discussed.

No discussion required.  This is the only sane way to do it.  There
may be some value in an optional externalName parameter which does
namespace translation in Cyrus, but otherwise the unique ID is the
internal name.  Make sure virtdomains are supported too.

> - Simplify the behavior. Do people need to define a context (queue)
> per event type ? Moreover several contexts per event type (and thus
> send each notification in several queues) ?

How much of this can be done outside Cyrus?  I don't know the queue
stuff well enough to be sure, but I would think you can feed events
to an external daemon which then multiplexes and filters them to
the consumers.  This external daemon could also do any protocol
translation into XML, JSON, whatever.  It's a prime candidate for
being written in a non-blocking style, since it's stateless other
than the filter/target definitions.

> 5) provide full documentation of the feature

Yeah, that's popular with the punters ;)

> 6) and later provide more connectors to notifyd like support of AMQP protocol

Again - a candidate for external daemon?  I think notifyd and idled
should be non-blocking a bit more anyway.  And I'd really REALLY like
for 'signal 10' from idled to die.  Every time we lose a frontend and
drop 50k connections across our backends, we get a bunch of random
processes being attacked with signal 10 as idled spews out its
connection attempts.

> I will create a new branch (based on Cyrus master) in our github for
> this feature (https://github.com/worldline-messaging)

Fantastic, I look forward to seeing the code!

Regards,

Bron.