JMAP: Current status

[Nicola Nye]

Hi Robert,

Given a couple of the recent questions on the list about Cyrus and JMAP
support, I think it'd be good to have a quick guide in the docs for
people wanting to get started with it.

We can then also keep it updated with the state of implementation(such
as you've outlined below) which will let people know what functionality
is and isn't yet available.

Are you able to write up a brief set of instructions on getting the
sample client running against a Cyrus installation and I can fold them
into the docs? (We can discuss at the Cyrus meeting later today)

Cheers,

   Nicola

----- Original message -----
From: "rost via Cyrus-devel" <cyrus-devel at lists.andrew.cmu.edu>
To: cyrus-devel at lists.andrew.cmu.edu
Subject: JMAP: Current status
Date: Tue, 05 Jan 2016 18:38:32 +0100

Hi,

as promised in our last Hangout, here is my writeup of the current state
of JMAP in Cyrus. Since I’ll be off from Cyrus development for a couple
of weeks, this double-serves as my TODO list. But first things first:
all my Cyrus contributions outlined below are 100% sponsored by
FastMail.

What works already
===============

Contacts
------------
Mostly. But see next section.

Calendars
--------------
Mostly. But see next section.

Messages 
--------------
Are work in progress and my current focus.

  - getMessages: works mostly. Not supported: message threads, inReplyTo
  or proper html-to-text body conversion (see
  [http_jmap.c:2248](https://github.com/rsto/cyrus-imapd/blob/jmap/imap/http_jmap.c#L2248)).
  - setMessages: supports to create drafts, send mails. Does not support
  creation of messages in multiple mailboxes, or any mailbox moves.
  - getMailboxes: mostly, except threads.
  - setMailboxes: mostly
  - getMessageList: supports filters

Cassandane tests
-------------------------
There are currently 59 JMAP tests in Cassandane.

What doesn’t work, yet
==================

Attachments
-----------------
I have worked on a proof-concept of supporting the JMAP File API (aka
“blobs”) in Cyrus, but it’s far from complete and it didn’t seem
promising enough to merge it into the main JMAP branch. As discussed
with Bron, I rather would like to get back to the design stage. For
that, I welcome any input! 

My current assumptions are that the majority of blobs will belong to
exactly one JMAP object, and most of the blobs uploaded by an account
will only be attached to one object. Also: while I assume most blobs
belong to mails, some users may have (rarely changing) avatar blobs in
most of their contacts. I assume calendar event attachments to be rarely
used.

Under these assumptions, I plan the blob service to keep a blob in its
containing object as long as possible. E.g. if a JMAP client wants to
download a mail attachment, the blob service should be able to use the
blobId to locate the containing message in the mailboxes. It should not
create a copy of the blob. That’s very similar to the current JMAP proxy
implementation. Things get tricky though, once such a blob is used as an
attachment for another object or even another entity (e.g. from message
blob to avatar blob). In order to better understand what a JMAP client
might want to do, I’ve summarised all use cases I could come up with in
a [Google
spreadsheet](https://docs.google.com/spreadsheets/d/15CvwT-aYw8ks3PbCS3Svm2sPfKWTbIB_JNBEBLMoGSc/edit?pref=2&pli=1#gid=0).
Note that the prioritisation is solely mine and some of these use cases
probably shouldn’t be supported at all. Also, in a sane blob service,
one control flow should over multiple use cases.

The JMAP blob ref-counting will require a tight grip on what happens to
mails, calendars and contacts. Maybe the notify service could be used to
support asynchronous ref-counting? Also: Since calendar managed
attachments are almost a subset of the more generic JMAP blobs (except
for deletions!), these two features might even be merged. The blob
service might deserve its own httpd and I could imagine it being
decoupled from the JMAP code. The blob HTTP service should also be
configurable to only redirect to “safe” URLs for custom calendar or
contact URL attachments.

Multi-user accounts
---------------------------
All the current JMAP code operates on the userid currently authenticated
to Cyrus httpd. That is, the ‘accountId’ property in JMAP requests is
not really supported.

Authentication
--------------------
All the JMAP methods in httpd currently require an account logged in via
Basic Authentication.

Remote mailboxes
--------------------------
As discussed with Bron, we don’t take remote mailboxes into account,
yet.

Events
---------
The JMAP event service hooks into notifications, so that’s almost done.
What’s AFAIK missing is the service layer (Bron knows more).

Messages
---------------
Anything not mentioned in the “What works” section doesn’t works.
Notably, that’s search snippets and threads.

Phrase-Matching
--------------------
The JMAP filters require phrase matching for text properties, but as a
placeholder we currently only support case-insensitive substring search.

What could work better
==================

Lookup message by guid
----------------------------------
We use message guids as JMAP message ids. Currently, that requires O(n),
where n is the number of records across all a users mailboxes. That
really should become O(1) or O(lgN)

Lookup mailbox by unique-id
------------------------------------------
We use mailbox unique-ids for JMAP mailbox ids. To look them up,  I
added already a stub in
[mboxlist](https://github.com/rsto/cyrus-imapd/blob/jmap/imap/mboxlist.c#L598),
but it’s O(n) (n is the number of a users mailboxes). Should be O(1) or
O(lgN) 

Filters
---------
The contacts and calendar code reuses the same JMAP filter code. We
agreed to first implement naive search for these objects. Currently, we
match every record in a a users calendar or contacts mailboxes against
the JMAP filter. At least for calendar events, this could be sped up,
e.g. it would be straight-forward to only pull out calendar events for a
custom time range or even create the SQL statements on demand based on
the filter contents.
Also, for message filters, there is a very naive filter implementation.
That’s just meant as a placeholder, and should be refactored to make use
of Xapian.

Error reporting
--------------------
The JMAP spec requires all invalid properties of a request to be
reported. But contacts fail at the first property error. Calendars and
Messages try hard to report all erroneous properties. None of the JMAP
error handlers report an error description.
  
Code organization
-------------------------
While code reuse is OK, contacts, calendars and messages still use
slightly different idioms. Note to me: Possibly use jmap_req as context,
and pass around sub-requests, e.g. jmap_setMessages_req embeds jmap_req.
That could help unify control and data flow. 

<EOF>

Cheers,
Robert