update AS API to merge in feedback from the pull request

drafts/application_services.rst

@@ -87,6 +87,11 @@ In the registration process, the AS provides:
  * Credentials to identify itself as an approved application service for that HS
  * Details of the namespaces of users and rooms the AS is acting on behalf of and
    "subscribing to"
+ * Namespaces are defined as a list of regexps against which to match room aliases,
+   room IDs, and user IDs.
+ * There is overlap between selecting events via the csv2 Filter API and subscribing
+   to events here - perhaps subscription involves passing a filter token into the
+   registration API.
  * A URL base for receiving requests from the HS (as the AS is a server,
     implementers expect to receive data via inbound requests rather than
     long-poll outbound requests)
@@ -101,8 +106,13 @@ On HS handling events to unknown users:
    @.irc.freenode.Arathorn:matrix.org.  This lets Matrix users communicate with
    foreign users who are not yet mapped into Matrix via 3PID mappings or through
    an existing non-virtual Matrix user by trying to talk to them via a gateway.
- * The AS can preprovision virtual users using the existing CS API rather than
-   lazy-loading them in this manner.
+ * The AS can alternatively preprovision virtual users using the existing CS API
+   rather than lazy-loading them in this manner.
+ * The AS may want to link the matrix ID of the sender through to their 3PID in
+   the remote ecosystem.  E.g. a message sent from @matthew:matrix.org may wish
+   to originate from Arathorn on irc.freenode.net in the case of an IRC bridge.
+   It's left as an AS implementation detail as to how the user should authorise
+   the AS to act on its behalf.
 
 On HS handling events to unknown rooms:
 
@@ -116,12 +126,19 @@ On HS handling events to unknown rooms:
    from the point of m.room.create, we will not be able to back-populate
    arbitrary history for rooms which are lazy-created in this manner, and instead
    have to chose the amount of history to be synchronised into the AS as a one-off.
- * If exposing arbitrary history is required, then either the room history must be
-   preemptively provisioned in the HS by the AS via the CS API (TODO: meaning the
-   CS API needs to support massaged timestamps), or the HS must delegate conversation
-   storage entirely to the AS using a Storage API (not defined here) which allows
-   the existing conversation store to back the HS, complete with all necessary
-   Matrix metadata (e.g. hashes, signatures, federation DAG, etc).
+ * If exposing arbitrary history is required, then:
+   
+   * either: the room history must be preemptively provisioned in the HS by the AS via
+     the CS API (TODO: meaning the CS API needs to support massaged
+     timestamps), resulting in conversation history being replicated between
+     the HS and the source store.
+   * or: the HS must delegate conversation storage entirely to the
+     AS using a Storage API (not defined here) which allows the existing
+     conversation store to back the HS, complete with all necessary Matrix
+     metadata (e.g. hashes, signatures, federation DAG, etc).  This obviously
+     increases the burden of implementing an AS considerably, but is the only
+     option if the implementer wants to avoid duplicating conversation history
+     between the external data source and the HS.
 
 On HS handling events to existing users and rooms:
 
@@ -130,7 +147,20 @@ On HS handling events to existing users and rooms:
    is handled as normal.
  * Events in the namespaces of rooms and users that the AS has subscribed to
    are pushed to the AS using the same pattern as the federation API (without
-   any of the encryption or federation metadata).  TODO: are they linearised?
+   any of the encryption or federation metadata).  This serves precisely the
+   same purpose as the CS event stream and has the same data flow semantics
+   (and indeed an AS implementer could chose to use the CS event stream instead)
+   
+   * Events are linearised to avoid the AS having to handle the complexity of
+     linearisation, and because if linearisation is good enough for CS, it
+     should be good enough for AS. Should the AS require non-linearised events
+     from Matrix, it should implement the federation API rather than the AS API
+     instead.
+   * HS->AS event pushes are retried for reliability with sequence numbers
+     (or logical timestamping?) to presereve the linearisation order and ensure
+     a reliable event stream.
+   * Clustered HSes must linearise just as they do for the CS API.  Clustered
+     ASes must loadbalance the inbound stream across the cluster as required.
 
 On AS relaying events from unknown-to-HS users:
 
@@ -146,6 +176,12 @@ On AS relaying events from unknown-to-HS users:
    * TODO: or do we maintain a separate access_token mapping?  It seems like
      unnecessary overhead for the AS developer; easier to just use a single
      privileged access_token and just track which userid is emitting events?
+   * If the AS is spoofing the identity of a real (not virtual) matrix user,
+     we should actually let them log themselves in via OAuth2 to give permission
+     to the AS to act on their behalf.
+   * We can't auth gatewayed virtual users from 3rd party systems who are being
+     relayed into Matrix, as the relaying is happening whether the user likes it
+     or not.  Therefore we do need to be able to spoof sender ID for virtual users.
  
 On AS relaying events in unknown-to-HS rooms:
 
@@ -180,11 +216,18 @@ AS Visibility:
    being physically seen in the room.  In this scenario, the user should set
    its presence to 'invisible', a state that HSes should only allow AS-authed
    users to set.
+   
+E2E Encryption
+
+ * The AS obviously has no visibility to E2E encrypted messages, unless it is
+   explicitly added to an encrypted room and participates in the group chat
+   itself.
 
 Extensions to CS API
 ====================
 
  * Ability to assert the identity of the virtual user for all methods.
  * Ability to massage timestamps when prepopulating historical state and
-   messages of virtual rooms.
+   messages of virtual rooms (either by overriding origin_server_ts (preferred) or
+   adding an as_ts which we expect clients to honour)
  * Ability to delete aliases (including from the directory) as well as create them.