wholetrans/docs-matrix-spec
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Merge branch 'master' into human-id-rules

Browse source
This commit is contained in:
Kegan Dougal 2015-10-13 15:16:41 +01:00
commit c9f6534d84
212 changed files with 14928 additions and 2324 deletions
Download patch file Download diff file Expand all files Collapse all files

14
drafts/address-book-repo.rst Normal file
View file

@ -0,0 +1,14 @@
Address book repository
=======================
.. NOTE::
This section is a work in progress.
.. TODO-spec
Do we even need it? Clients can use out-of-band addressbook servers for now;
this should definitely not be core.
- format: POST(?) wodges of json, some possible processing, then return wodges of json on GET.
- processing may remove dupes, merge contacts, pepper with extra info (e.g. matrix-ability of
contacts), etc.
- Standard json format for contacts? Piggy back off vcards?

244
drafts/application_services.rst Normal file
View file

@ -0,0 +1,244 @@
.. TODO
Sometimes application services need to create rooms (e.g. when lazy loading
from room aliases). Created rooms need to have a user that created them, so
federation works (as it relies on an entry existing in m.room.member). We should
be able to add metadata to m.room.member to state that this user is an application
service, a virtual user, etc.
Application Services
====================
Overview
========
Application services provide a way of implementing custom serverside functionality
on top of Matrix without the complexity of implementing the full federation API.
By acting as a trusted service logically located behind an existing homeserver,
Application services are decoupled from:
* Signing or validating federated traffic or conversation history
* Validating authorisation constraints on federated traffic
* Managing routing or retry schemes to the rest of the Matrix federation
As such, developers can focus entirely on implementing application logic rather
than being concerned with the details of managing Matrix federation.
Features available to application services include:
* Privileged subscription to any events available to the homeserver
* Synthesising virtual users
* Synthesising virtual rooms
* Injecting message history for virtual rooms
Features not provided by application services include:
* Intercepting and filtering/modifying message or behaviour within a room
(this is a job for a Policy Server, as it requires a single logical focal
point for messages in order to consistently apply the custom business logic)
Example use cases for application services include:
* Exposing existing communication services in Matrix
* Gateways to/from standards-based protocols (SIP, XMPP, IRC, RCS (MSRP), SIMPLE, Lync, etc)
* Gateways to/from closed services (e.g. WhatsApp)
* Gateways could be architected as:
* Act as a virtual client on the non-Matrix network
(e.g. connect as multiple virtual clients to an IRC or XMPP server)
* Act as a server on the non-Matrix network
(e.g. speak s2s XMPP federation, or IRC link protocol)
* Act as an application service on the non-Matrix network
(e.g. link up as IRC services, or an XMPP component)
* Exposing a non-Matrix client interface listener from the AS
(e.g. listen on port 6667 for IRC clients, or port 5222 for XMPP clients)
* Bridging existing APIs into Matrix
* e.g. SMS/MMS aggregator APIs
* Domain-specific APIs such as SABRE
* Integrating more exotic content into Matrix
* e.g. MIDI<->Matrix gateway/bridge
* 3D world <-> Matrix bridge
* Application services:
* Search engines (e.g. elasticsearch search indices)
* Notification systems (e.g. send custom pushes for various hooks)
* VoIP Conference services
* Text-to-speech and Speech-to-text services
* Signal processing
* IVR
* Server-machine translation
* Censorship service
* Multi-User Gaming (Dungeons etc)
* Other "constrained worlds" (e.g. 3D geometry representations)
* applying physics to a 3D world on the serverside
* (applying gravity and friction and air resistance... collision detection)
* domain-specific merge conflict resolution of events
* Payment style transactional usecases with transactional guarantees
Architecture Outline
====================
The application service registers with its host homeserver to offer its services.
In the registration process, the AS provides:
* Credentials to identify itself as an approved application service for that HS
* Details of the namespaces of users and rooms the AS is acting on behalf of and
"subscribing to"
* Namespaces are defined as a list of regexps against which to match room aliases,
room IDs, and user IDs. Regexps give the flexibility to say, sub-domain MSISDN
ranges per AS, whereas a blunt prefix string does not. These namespaces are further
configured by setting whether they are ``exclusive`` or not. An exclusive namespace
prevents entities other than the aforementioned AS from creating/editing/deleting
entries within that namespace. This does not affect the visibility/readability of
entries within that namespace (e.g. it doesn't prevent users joining exclusive
aliases, or ASes from listening to exclusive aliases, but does prevent both users
and ASes from creating/editing/deleting aliases within that namespace).
* There is overlap between selecting events via the csv2 Filter API and subscribing
to events here - perhaps subscription involves passing a filter token into the
registration API.
* A URL base for receiving requests from the HS (as the AS is a server,
implementers expect to receive data via inbound requests rather than
long-poll outbound requests)
On HS handling events to unknown users:
* If the HS receives an event for an unknown user who is in the namespace delegated to
the AS, then the HS queries the AS for the profile of that user. If the AS
confirms the existence of that user (from its perspective), then the HS
creates an account to represent the virtual user.
* The namespace of virtual user accounts should conform to a structure like
``@.irc.freenode.Arathorn:matrix.org``. This lets Matrix users communicate with
foreign users who are not yet mapped into Matrix via 3PID mappings or through
an existing non-virtual Matrix user by trying to talk to them via a gateway.
* The AS can alternatively preprovision virtual users using the existing CS API
rather than lazy-loading them in this manner.
* The AS may want to link the matrix ID of the sender through to their 3PID in
the remote ecosystem. E.g. a message sent from ``@matthew:matrix.org`` may wish
to originate from Arathorn on irc.freenode.net in the case of an IRC bridge.
It's left as an AS implementation detail as to how the user should authorise
the AS to act on its behalf.
On HS handling events to unknown rooms:
* If the HS receives an invite to an unknown room which is in the namespace
delegated to the AS, then the HS queries the AS for the existence of that room.
If the AS confirms its existence (from its perspective), then the HS creates
the room.
* The initial state of the room may be populated by the AS by querying an
initialSync API (probably a subset of the CS initialSync API, to reuse the
same pattern for the equivalent function). As messages have to be signed
from the point of ``m.room.create``, we will not be able to back-populate
arbitrary history for rooms which are lazy-created in this manner, and instead
have to chose the amount of history to be synchronised into the AS as a one-off.
* If exposing arbitrary history is required, then:
* either: the room history must be preemptively provisioned in the HS by the AS via
the CS API (TODO: meaning the CS API needs to support massaged
timestamps), resulting in conversation history being replicated between
the HS and the source store.
* or: the HS must delegate conversation storage entirely to the
AS using a Storage API (not defined here) which allows the existing
conversation store to back the HS, complete with all necessary Matrix
metadata (e.g. hashes, signatures, federation DAG, etc). This obviously
increases the burden of implementing an AS considerably, but is the only
option if the implementer wants to avoid duplicating conversation history
between the external data source and the HS.
On HS handling events to existing users and rooms:
* If the HS receives an event for a user or room that already exists (either
provisioned by the AS or by normal client interactions), then the message
is handled as normal.
* Events in the namespaces of rooms and users that the AS has subscribed to
are pushed to the AS using the same pattern as the federation API (without
any of the encryption or federation metadata). This serves precisely the
same purpose as the CS event stream and has the same data flow semantics
(and indeed an AS implementer could chose to use the CS event stream instead)
* Events are linearised to avoid the AS having to handle the complexity of
linearisation, and because if linearisation is good enough for CS, it
should be good enough for AS. Should the AS require non-linearised events
from Matrix, it should implement the federation API rather than the AS API
instead.
* HS->AS event pushes are retried for reliability with sequence numbers
(or logical timestamping?) to preserve the linearisation order and ensure
a reliable event stream.
* Clustered HSes must linearise just as they do for the CS API. Clustered
ASes must loadbalance the inbound stream across the cluster as required.
On AS relaying events from unknown-to-HS users:
* AS injects the event to the HS using the CS API, irrespective of whether the
target user or room is known to the HS or not. If the HS doesn't recognise
the target it goes through the same lazy-load provisioning as per above.
* The reason for not using a subset of the federation API here is because it
allows AS developers to reuse existing CS SDKs and benefit from the more
meaningful error handling of the CS API. The sending user ID must be
explicitly specified, as it cannot be inferred from the access_token, which
will be the same for all AS requests.
* TODO: or do we maintain a separate ``access_token`` mapping? It seems like
unnecessary overhead for the AS developer; easier to just use a single
privileged ``access_token`` and just track which ``user_id`` is emitting events?
* If the AS is spoofing the identity of a real (not virtual) matrix user,
we should actually let them log themselves in via OAuth2 to give permission
to the AS to act on their behalf.
* We can't auth gatewayed virtual users from 3rd party systems who are being
relayed into Matrix, as the relaying is happening whether the user likes it
or not. Therefore we do need to be able to spoof sender ID for virtual users.
On AS relaying events in unknown-to-HS rooms:
* See above.
On AS publishing aliases for virtual rooms:
* AS uses the normal alias management API to preemptively create/delete public
directory entries for aliases for virtual rooms provided by the AS.
* In order to create these aliases, the underlying room ID must also exist, so
at least the ``m.room.create`` of that room must also be prepopulated. It seems
sensible to prepopulate the required initial state and history of the room to
avoid a two-phase prepopulation process.
On unregistering the AS from the HS:
* An AS must tell the HS when it is going offline in order to stop receiving
requests from the HS. It does this by hitting an API on the HS.
AS Visibility:
* If an AS needs to sniff events in a room in order to operate on them (e.g.
to act as a search engine) but not inject traffic into the room, it should
do so by subscribing to the relevant events without actually joining the room.
* If the AS needs to participate in the room as a virtual user (e.g. an IVR
service, or a bot, or a gatewayed virtual user), it should join the room
normally.
* There are rare instances where an AS may wish to participate in a room
(including inserting messages), but be hidden from the room list - e.g. a
conferencing server focus bot may wish to join many rooms as the focus and
both listen to VoIP setups and inject its own VoIP answers, without ever
being physically seen in the room. In this scenario, the user should set
its presence to 'invisible', a state that HSes should only allow AS-authed
users to set.
E2E Encryption
* The AS obviously has no visibility to E2E encrypted messages, unless it is
explicitly added to an encrypted room and participates in the group chat
itself.
Extensions to CS API
====================
* Ability to assert the identity of the virtual user for all methods.
* Ability to massage timestamps when prepopulating historical state and
messages of virtual rooms (either by overriding ``origin_server_ts`` (preferred) or
adding an ``as_ts`` which we expect clients to honour)
* Ability to delete aliases (including from the directory) as well as create them.

574
drafts/as-http-api.rst Normal file
View file

@ -0,0 +1,574 @@
.. TODO
Sometimes application services need to create rooms (e.g. when lazy loading
from room aliases). Created rooms need to have a user that created them, so
federation works (as it relies on an entry existing in m.room.member). We
should be able to add metadata to m.room.member to state that this user is an
application service, a virtual user, etc.
Application Services HTTP API
=============================
.. contents:: Table of Contents
.. sectnum::
Application Service -> Home Server
----------------------------------
This contains home server APIs which are used by the application service.
Registration API ``[Draft]``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This API registers the application service with its host homeserver to offer its
services.
Inputs:
- Credentials (e.g. some kind of string token)
- Namespace[users]
- Namespace[room aliases]
- URL base to receive inbound comms
Output:
- The credentials the HS will use to query the AS with in return. (e.g. some
kind of string token)
Side effects:
- The HS will start delivering events to the URL base specified if this 200s.
API called when:
- The application service wants to register with a brand new home server.
Notes:
- An application service can state whether they should be the only ones who
can manage a specified namespace. This is referred to as an "exclusive"
namespace. An exclusive namespace prevents humans and other application
services from creating/deleting entities in that namespace. Typically,
exclusive namespaces are used when the rooms represent real rooms on
another service (e.g. IRC). Non-exclusive namespaces are used when the
application service is merely augmenting the room itself (e.g. providing
logging or searching facilities).
- Namespaces are represented by POSIX extended regular expressions in JSON.
They look like::
users: [
{
"exclusive": true,
"regex": "@irc\.freenode\.net/.*"
}
]
::
POST /register
Request format
{
url: "https://my.application.service.com/matrix/",
as_token: "some_AS_token",
namespaces: {
users: [
{
"exclusive": true,
"regex": "@irc\.freenode\.net/.*"
}
],
aliases: [
{
"exclusive": true,
"regex": "#irc\.freenode\.net/.*"
}
],
rooms: [
{
"exclusive": true,
"regex": "!irc\.freenode\.net/.*"
}
]
}
}
Returns:
200 : Registration accepted.
400 : Namespaces do not conform to regex
401 : Credentials need to be supplied.
403 : AS credentials rejected.
200 OK response format
{
hs_token: "string"
}
Unregister API ``[Draft]``
~~~~~~~~~~~~~~~~~~~~~~~~~~
This API unregisters a previously registered AS from the home server.
Inputs:
- AS token
Output:
- None.
Side effects:
- The HS will stop delivering events to the URL base specified for this AS if
this 200s.
API called when:
- The application service wants to stop receiving all events from the HS.
::
POST /unregister
Request format
{
as_token: "string"
}
Home Server -> Application Service
----------------------------------
This contains application service APIs which are used by the home server.
User Query ``[Draft]``
~~~~~~~~~~~~~~~~~~~~~~
This API is called by the HS to query the existence of a user on the Application
Service's namespace.
Inputs:
- User ID
- HS Credentials
Output:
- Whether the user exists.
Side effects:
- User is created on the HS by the AS via CS APIs during the processing of this request.
API called when:
- HS receives an event for an unknown user ID in the AS's namespace, e.g. an
invite event to a room.
Notes:
- When the AS receives this request, if the user exists, it must create the user via
the CS API.
- It can also set arbitrary information about the user (e.g. display name, join rooms, etc)
using the CS API.
- When this setup is complete, the AS should respond to the HS request. This means the AS
blocks the HS until the user is created.
- This is deemed more flexible than alternative methods (e.g. returning a JSON blob with the
user's display name and get the HS to provision the user).
Retry notes:
- The home server cannot respond to the client's request until the response to
this API is obtained from the AS.
- Recommended that home servers try a few times then time out, returning a
408 Request Timeout to the client.
::
GET /users/$user_id?access_token=$hs_token
Returns:
200 : User is recognised.
404 : User not found.
401 : Credentials need to be supplied.
403 : HS credentials rejected.
200 OK response format
{}
Room Alias Query ``[Draft]``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This API is called by the HS to query the existence of a room alias on the
Application Service's namespace.
Inputs:
- Room alias
- HS Credentials
Output:
- Whether the room exists.
Side effects:
- Room is created on the HS by the AS via CS APIs during the processing of
this request.
API called when:
- HS receives an event to join a room alias in the AS's namespace.
Notes:
- When the AS receives this request, if the room exists, it must create the room via
the CS API.
- It can also set arbitrary information about the room (e.g. name, topic, etc)
using the CS API.
- It can send messages as other users in order to populate scrollback.
- When this setup is complete, the AS should respond to the HS request. This means the AS
blocks the HS until the room is created and configured.
- This is deemed more flexible than alternative methods (e.g. returning an initial sync
style JSON blob and get the HS to provision the room). It also means that the AS knows
the room ID -> alias mapping.
Retry notes:
- The home server cannot respond to the client's request until the response to
this API is obtained from the AS.
- Recommended that home servers try a few times then time out, returning a
408 Request Timeout to the client.
::
GET /rooms/$room_alias?access_token=$hs_token
Returns:
200 : Room is recognised.
404 : Room not found.
401 : Credentials need to be supplied.
403 : HS credentials rejected.
200 OK response format
{}
Pushing ``[Draft]``
~~~~~~~~~~~~~~~~~~~
This API is called by the HS when the HS wants to push an event (or batch of
events) to the AS.
Inputs:
- HS Credentials
- Event(s) to give to the AS
- HS-generated transaction ID
Output:
- None.
Data flows:
::
Typical
HS ---> AS : Home server sends events with transaction ID T.
<--- : AS sends back 200 OK.
AS ACK Lost
HS ---> AS : Home server sends events with transaction ID T.
<-/- : AS 200 OK is lost.
HS ---> AS : Home server retries with the same transaction ID of T.
<--- : AS sends back 200 OK. If the AS had processed these events
already, it can NO-OP this request (and it knows if it is the same
events based on the transacton ID).
Retry notes:
- If the HS fails to pass on the events to the AS, it must retry the request.
- Since ASes by definition cannot alter the traffic being passed to it (unlike
say, a Policy Server), these requests can be done in parallel to general HS
processing; the HS doesn't need to block whilst doing this.
- Home servers should use exponential backoff as their retry algorithm.
- Home servers MUST NOT alter (e.g. add more) events they were going to
send within that transaction ID on retries, as the AS may have already
processed the events.
Ordering notes:
- The events sent to the AS should be linearised, as they are from the event
stream.
- The home server will need to maintain a queue of transactions to send to
the AS.
::
PUT /transactions/$transaction_id?access_token=$hs_token
Request format
{
events: [
...
]
}
Client-Server v2 API Extensions
-------------------------------
Identity assertion
~~~~~~~~~~~~~~~~~~
The client-server API infers the user ID from the ``access_token`` provided in
every request. It would be an annoying amount of book-keeping to maintain tokens
for every virtual user. It would be preferable if the application service could
use the CS API with its own ``as_token`` instead, and specify the virtual user
they wish to be acting on behalf of. For real users, this would require
additional permissions granting the AS permission to masquerade as a matrix user.
Inputs:
- Application service token (``access_token``)
Either:
- User ID in the AS namespace to act as.
Or:
- OAuth2 token of real user (which may end up being an access token)
Notes:
- This will apply on all aspects of the CS API, except for Account Management.
- The ``as_token`` is inserted into ``access_token`` which is usually where the
client token is. This is done on purpose to allow application services to
reuse client SDKs.
::
/path?access_token=$token&user_id=$userid
Query Parameters:
access_token: The application service token
user_id: The desired user ID to act as.
/path?access_token=$token&user_token=$token
Query Parameters:
access_token: The application service token
user_token: The token granted to the AS by the real user
Timestamp massaging
~~~~~~~~~~~~~~~~~~~
The application service may want to inject events at a certain time (reflecting
the time on the network they are tracking e.g. irc, xmpp). Application services
need to be able to adjust the ``origin_server_ts`` value to do this.
Inputs:
- Application service token (``as_token``)
- Desired timestamp
Notes:
- This will only apply when sending events.
::
/path?access_token=$token&ts=$timestamp
Query Parameters added to the send event APIs only:
access_token: The application service token
ts: The desired timestamp
Server admin style permissions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The home server needs to give the application service *full control* over its
namespace, both for users and for room aliases. This means that the AS should
be able to create/edit/delete any room alias in its namespace, as well as
create/delete any user in its namespace. No additional API changes need to be
made in order for control of room aliases to be granted to the AS. Creation of
users needs API changes in order to:
- Work around captchas.
- Have a 'passwordless' user.
This involves bypassing the registration flows entirely. This is achieved by
including the AS token on a ``/register`` request, along with a login type of
``m.login.application_service`` to set the desired user ID without a password.
::
/register?access_token=$as_token
Content:
{
type: "m.login.application_service",
user: "<desired user localpart in AS namespace>"
}
Application services which attempt to create users or aliases *outside* of
their defined namespaces will receive an error code ``M_EXCLUSIVE``. Similarly,
normal users who attempt to create users or alises *inside* an application
service-defined namespace will receive the same ``M_EXCLUSIVE`` error code.
ID conventions ``[Draft]``
--------------------------
.. NOTE::
- Giving HSes the freedom to namespace still feels like the Right Thing here.
- Exposing a public API provides the consistency which was the main complaint
against namespacing.
- This may have knock-on effects for the AS registration API. E.g. why don't
we let ASes specify the *URI* regex they want?
This concerns the well-defined conventions for mapping 3P network IDs to matrix
IDs, which we expect clients to be able to do by themselves.
User IDs
~~~~~~~~
Matrix users may wish to directly contact a virtual user, e.g. to send an email.
The URI format is a well-structured way to represent a number of different ID
types, including:
- MSISDNs (``tel``)
- Email addresses (``mailto``)
- IRC nicks (``irc`` - https://tools.ietf.org/html/draft-butcher-irc-url-04)
- XMPP (xep-0032)
- SIP URIs (RFC 3261)
As a result, virtual user IDs SHOULD relate to their URI counterpart. This
mapping from URI to user ID can be expressed in a number of ways:
- Expose a C-S API on the HS which takes URIs and responds with user IDs.
- Munge the URI with the user ID.
Exposing an API would allow HSes to internally map user IDs however they like,
at the cost of an extra round trip (of which the response can be cached).
Munging the URI would allow clients to apply the mapping locally, but would force
user X on service Y to always map to the same munged user ID. Considering the
exposed API could just be applying this munging, there is more flexibility if
an API is exposed.
::
GET /_matrix/app/v1/user?uri=$url_encoded_uri
Returns 200 OK:
{
user_id: <complete user ID on local HS>
}
Room Aliases
~~~~~~~~~~~~
We may want to expose some 3P network rooms so Matrix users can join them directly,
e.g. IRC rooms. We don't want to expose every 3P network room though, e.g. mailto,
tel. Rooms which are publicly accessible (e.g. IRC rooms) can be exposed as an alias by
the application service. Private rooms (e.g. sending an email to someone) should not
be exposed in this way, but should instead operate using normal invite/join semantics.
Therefore, the ID conventions discussed below are only valid for public rooms which
expose room aliases.
Matrix users may wish to join XMPP rooms (e.g. using XEP-0045) or IRC rooms. In both
cases, these rooms can be expressed as URIs. For consistency, these "room" URIs
SHOULD be mapped in the same way as "user" URIs.
::
GET /_matrix/app/v1/alias?uri=$url_encoded_uri
Returns 200 OK:
{
alias: <complete room alias on local HS>
}
Event fields
~~~~~~~~~~~~
We recommend that any gatewayed events should include an ``external_url`` field
in their content to provide a way for Matrix clients to link into the 'native'
client from which the event originated. For instance, this could contain the
message-ID for emails/nntp posts, or a link to a blog comment when gatewaying
blog comment traffic in & out of matrix
Examples
--------
.. NOTE::
- User/Alias namespaces are subject to change depending on ID conventions.
IRC
~~~
Pre-conditions:
- Server admin stores the AS token "T_a" on the home server.
- Home server has a token "T_h".
- Home server has the domain "hsdomain.com"
1. Application service registration
::
AS -> HS: Registers itself with the home server
POST /register
{
url: "https://someapp.com/matrix",
as_token: "T_a",
namespaces: {
users: [
{
"exclusive": true,
"regex": "@irc\.freenode\.net/.*"
}
],
aliases: [
{
"exclusive": true,
"regex": "#irc\.freenode\.net/.*"
}
]
}
}
Returns 200 OK:
{
hs_token: "T_h"
}
2. IRC user "Bob" says "hello?" on "#matrix" at timestamp 1421416883133:
::
- AS stores message as potential scrollback.
- Nothing happens as no Matrix users are in the room.
3. Matrix user "@alice:hsdomain.com" wants to join "#matrix":
::
User -> HS: Request to join "#irc.freenode.net/#matrix:hsdomain.com"
HS -> AS: Room Query "#irc.freenode.net/#matrix:hsdomain.com"
GET /rooms/%23irc.freenode.net%2F%23matrix%3Ahsdomain.com?access_token=T_h
[Starts blocking]
AS -> HS: Creates room. Gets room ID "!aasaasasa:hsdomain.com".
AS -> HS: Sets room name to "#matrix".
AS -> HS: Sends message as ""@irc.freenode.net/Bob:hsdomain.com"
PUT /rooms/%21aasaasasa%3Ahsdomain.com/send/m.room.message
?access_token=T_a
&user_id=%40irc.freenode.net%2FBob%3Ahsdomain.com
&ts=1421416883133
{
body: "hello?"
msgtype: "m.text"
}
HS -> AS: User Query "@irc.freenode.net/Bob:hsdomain.com"
GET /users/%40irc.freenode.net%2FBob%3Ahsdomain.com?access_token=T_h
[Starts blocking]
AS -> HS: Creates user using CS API extension.
POST /register?access_token=T_a
{
type: "m.login.application_service",
user: "irc.freenode.net/Bob"
}
AS -> HS: Set user display name to "Bob".
[Finishes blocking]
[Finished blocking]
- HS sends room information back to client.
4. @alice:hsdomain.com says "hi!" in this room:
::
User -> HS: Send message "hi!" in room !aasaasasa:hsdomain.com
- HS sends message.
- HS sees the room ID is in the AS namespace and pushes it to the AS.
HS -> AS: Push event
PUT /transactions/1?access_token=T_h
{
events: [
{
content: {
body: "hi!",
msgtype: "m.text"
},
origin_server_ts: <generated by hs>,
user_id: "@alice:hsdomain.com",
room_id: "!aasaasasa:hsdomain.com",
type: "m.room.message"
}
]
}
- AS passes this through to IRC.
5. IRC user "Bob" says "what's up?" on "#matrix" at timestamp 1421418084816:
::
IRC -> AS: "what's up?"
AS -> HS: Send message via CS API extension
PUT /rooms/%21aasaasasa%3Ahsdomain.com/send/m.room.message
?access_token=T_a
&user_id=%40irc.freenode.net%2FBob%3Ahsdomain.com
&ts=1421418084816
{
body: "what's up?"
msgtype: "m.text"
}
- HS modifies the user_id and origin_server_ts on the event and sends it.

4
drafts/definitions.rst
View file

@ -5,7 +5,7 @@ Definitions
# *Event* -- A JSON object that represents a piece of information to be
distributed to the the room. The object includes a payload and metadata,
including a `type` used to indicate what the payload is for and how to process
including a ``type`` used to indicate what the payload is for and how to process
them. It also includes one or more references to previous events.
# *Event graph* -- Events and their references to previous events form a
@ -13,7 +13,7 @@ directed acyclic graph. All events must be a descendant of the first event in a
room, except for a few special circumstances.
# *State event* -- A state event is an event that has a non-null string valued
`state_key` field. It may also include a `prev_state` key referencing exactly
`state_key` field. It may also include a ``prev_state`` key referencing exactly
one state event with the same type and state key, in the same event graph.
# *State tree* -- A state tree is a tree formed by a collection of state events

5
drafts/erik-model.rst
View file

@ -1,4 +1,7 @@
This is a standalone description of the data architecture of Synapse. There is a lot of overlap with the currennt specification, so it has been split out here for posterity. Hopefully all the important bits have been merged into the relevant places in the main spec.
This is a standalone description of the data architecture of Synapse. There is a
lot of overlap with the current specification, so it has been split out here for
posterity. Hopefully all the important bits have been merged into the relevant
places in the main spec.
Model

65
drafts/general_api.rst
View file

@ -555,7 +555,7 @@ signature. Requesting the "raw" federation event will have to return these keys.
Account Management API ``[Draft]``
----------------------------------
The registration and login APIs in v2 do not support specifying device IDs. In v2,
The registration and login APIs in v1 do not support specifying device IDs. In v2,
this will become *mandatory* when sending your initial request. Access tokens will
be scoped per device, so using the same device ID twice when logging in will
clobber the old access token.
@ -810,6 +810,11 @@ Notes:
Presence API ``[Draft]``
------------------------
.. FIXME
this seems to be ignoring activity timers entirely, which were present on
the planning etherpad and are present in the actual HTTP API. Needs attention.
The goals of presence are to:
- Let other users know if someone is "online".
@ -817,22 +822,23 @@ The goals of presence are to:
- Let other users know specific status information (e.g. "In a Meeting").
"Online" state can be detected by inspecting when the last time the client made
a request to the server. This could be any request, or a specific kind of request.
For connection-orientated protocols, detecting "online" state can be determined by
the state of this connection stream. For HTTP, this can be detected via requests
to the event stream.
a request to the server. This could be any request, or a specific kind of
request. For connection-orientated protocols, detecting "online" state can be
determined by the state of this connection stream. For HTTP, this can be
detected via requests to the event stream.
Online state is separate from letting other users know if someone is *likely to
respond* to messages. This introduces the concept of an "idle" flag, which is
set when the user has not done any "interaction" with the app. The definition of
"interaction" varies based on the app, so it is up to the app to set this "idle"
flag.
respond* to messages. This introduces the concept of being "idle", which is
when the user has not done any "interaction" with the app for a while. The
definition of "interaction" and "for a while" varies based on the app, so it is
up to the app to set when the user is idle.
Letting users know specific status information can be achieved via the same method
as v1. Status information should be scoped per *user* and not device as determining
a union algorithm between statuses is nonsensical. Passing status information per
device to all other users just redirects the union problem to the client, which
will commonly be presenting this information as an icon alongside the user.
Letting users know specific status information can be achieved via the same
method as v1. Status information should be scoped per *user* and not device as
determining a union algorithm between statuses is nonsensical. Passing status
information per device to all other users just redirects the union problem to
the client, which will commonly be presenting this information as an icon
alongside the user.
When a client hits the event stream, the home server can treat the user as
"online". This behaviour should be able to be overridden to avoid flicker
@ -841,11 +847,11 @@ appear offline > goes into a tunnel > server times out > device regains
connection and hits the event stream forcing the device online before the
"appear offline" state can be set). When the client has not hit the event
stream for a certain period of time, the home server can treat the user as
"offline".
"offline". The user can also set a global *per-user* appear offline flag.
The user should also be able to set their presence via a direct API, without
having to hit the event stream. The home server will set a timer when the
connection ends, after which it will set that device to offline.
The user should also be able to set their presence state via a direct API,
without having to hit the event stream. The home server will set a timer when
the connection ends, after which it will set that device to offline.
As the idle flag and online state is determined per device, there needs to be a
union algorithm to merge these into a single state and flag per user, which will
@ -859,22 +865,33 @@ Changing presence status:
Inputs:
- User ID
- Presence status (online, away, busy, do not disturb, etc)
Outputs:
- Presence status (busy, do not disturb, in a meeting, etc)
Output:
- None.
Setting the idle flag:
Setting presence state:
Inputs:
- User ID
- Is idle
Outputs:
- Device ID
- Presence state (online|idle|offline)
Output:
- None.
Setting global appear offline:
Inputs:
- User ID
- Should appear offline (boolean)
Output:
- None.
Extra parameters associated with the event stream:
Inputs:
- Presence state (online, appear offline)
- Presence state (online, idle, offline)
Notes:
- Scoped per device just like the above API, e.g. from the access_token.
Typing API ``[Final]``

34
drafts/macaroons_caveats.rst Normal file
View file

@ -0,0 +1,34 @@
Macaroon Caveats
================
`Macaroons`_ are issued by Matrix servers as authorization tokens. Macaroons may be restricted by adding caveats to them.
.. _Macaroons: http://theory.stanford.edu/~ataly/Papers/macaroons.pdf
Caveats can only be used for reducing the scope of a token, never for increasing it. Servers are required to reject any macroon with a caveat that they do not understand.
Some caveats are specified in this specification, and must be understood by all servers. The use of non-standard caveats is allowed.
All caveats must take the form:
`key` `operator` `value`
where `key` is a non-empty string drawn from the character set [A-Za-z0-9_]
`operator` is a non-empty string which does not contain whitespace
`value` is a non-empty string
And these are joined by single space characters.
Specified caveats:
+-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+
| Caveat name | Description | Legal Values |
+-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+
| gen | Generation of the macaroon caveat spec. | 1 |
| user_id | ID of the user for which this macaroon is valid. | Pure equality check. Operator must be =. |
| type | The purpose of this macaroon. | access - used to authorize any action except token refresh |
| refresh - only used to authorize a token refresh |
| time | Time before/after which this macaroon is valid. | A POSIX timestamp in milliseconds (in UTC). |
| Operator < means the macaroon is valid before the timestamp, as interpreted by the server. |
| Operator > means the macaroon is valid after the timestamp, as interpreted by the server. |
| Operator == means the macaroon is valid at exactly the timestamp, as interpreted by the server.|
| Note that exact equality of time is largely meaningless. |
+-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+

76
drafts/media_repository.rst
View file

@ -1,76 +0,0 @@
Media Repository
================
File uploading and downloading.
HTTP API
--------
Uploads are POSTed to a resource which returns a token which is used to GET
the download. Uploads are POSTed to the sender's local homeserver, but are
downloaded from the recipient's local homeserver, which must thus first transfer
the content from the origin homeserver using the same API (unless the origin
and destination homeservers are the same). The upload/download API is::
=> POST /_matrix/media/v1/upload HTTP/1.1
Content-Type: <media-type>
<media>
<= HTTP/1.1 200 OK
Content-Type: application/json
{ "content-uri": "mxc://<server-name>/<media-id>" }
=> GET /_matrix/media/v1/download/<server-name>/<media-id> HTTP/1.1
<= HTTP/1.1 200 OK
Content-Type: <media-type>
Content-Disposition: attachment;filename=<upload-filename>
<media>
Clients can get thumbnails by supplying a desired width and height and
thumbnailing method::
=> GET /_matrix/media/v1/thumbnail/<server_name>
/<media-id>?width=<w>&height=<h>&method=<m> HTTP/1.1
<= HTTP/1.1 200 OK
Content-Type: image/jpeg or image/png
<thumbnail>
The thumbnail methods are "crop" and "scale". "scale" trys to return an
image where either the width or the height is smaller than the requested
size. The client should then scale and letterbox the image if it needs to
fit within a given rectangle. "crop" trys to return an image where the
width and height are close to the requested size and the aspect matches
the requested size. The client should scale the image if it needs to fit
within a given rectangle.
Homeservers may generate thumbnails for content uploaded to remote
homeservers themselves or may rely on the remote homeserver to thumbnail
the content. Homeservers may return thumbnails of a different size to that
requested. However homeservers should provide extact matches where reasonable.
Security
--------
Clients may try to upload very large files. Homeservers should not store files
that are too large and should not serve them to clients.
Clients may try to upload very large images. Homeservers should not attempt to
generate thumbnails for images that are too large.
Remote homeservers may host very large files or images. Homeserver should not
proxy or thumbnail large files or images from remote homeservers.
Clients may try to upload a large number of files. Homeservers should limit the
number and total size of media that can be uploaded by clients.
Clients may try to access a large number of remote files through a homeserver.
Homeservers should restrict the number and size of remote files that it caches.
Clients or remote homeservers may try to upload malicious files targeting
vunerabilities in either the homeserver thumbnailing or the client decoders.

2
drafts/model/protocol_examples.rst
View file

@ -32,7 +32,7 @@ Content-Type: application/json
}
HTTP/1.1 200 OK
...
======================================

5
drafts/object_model.rst
View file

@ -1,6 +1,5 @@
..TODO
What are the start & end tokens doing here?!
::

48
drafts/pstn_gatewaying.txt Normal file
View file

@ -0,0 +1,48 @@
Gatewaying to the PSTN via Matrix Application Services
======================================================
Matrix Application Services (AS) provides a way for PSTN users to interact
with Matrix via an AS acting as a gateway. Each PSTN user is represented as a
virtual user on a specific homeserver maintained by the AS. Typically the AS
is provisioned on a well-known AS-supplier HS (e.g. matrix.openmarket.com) or
is a service provisioned on the user's local HS.
In either scenario, the AS maintains virtual users of form
@.tel.e164:homeserver. These are lazily created (as per the AS spec) when
matrix users try to contact a user id of form @.tel.*:homeserver, or when the
AS needs to inject traffic into the HS on behalf of the PSTN user. The reason
for these being a visible virtual user rather than an invisible user or an
invisible sniffing AS is because they do represent real physical 3rd party
endpoints in the PSTN, and need to be able to send return messages.
Communication with an actual PSTN user happens in a normal Matrix room, which
for 1:1 matrix<->pstn contact will typically store all conversation history
with that user. On first contact, the matrix user invites the virtual user
into the room (or vice versa). In the event of switching to another AS-enabled
HS, the matrix user would kick the old AS and invite the new one. In the event
of needing loadbalancing between two SMS gateways (for instance), the user
would set visibility flags (TODO: specify per-message ACLs, or use crypto to
only sign messages so they're visible to certain other users?) to adjust which
virtual AS users could see which messages in the room.
For group chat, one or more AS virtual users may be invited to a group chat,
where-upon they will relay all the traffic in that group chat through to their
PSTN counterpart (and vice versa). This behaviour requires no additional
functionality beyond that required to support 1:1 chat.
When contacting a user, Matrix clients should check whether a given E.164
number is already mapped to a real Matrix user by querying the identity
servers (or subscribing to identity updates for a given E.164 number. TODO: ID
server subscriptions). If the E.164 number has a validated mapping in the ID
server to a Matrix ID, then this target ID should be used instead of
contacting the virtual user.
It's likely that PSTN gateway ASes will need to charge the end-user for use of
the gateway. The AS must therefore track credit per matrix ID it interacts
with, and stop gatewaying as desired once credit is exhausted. The task of
extracting credit from the end-user and adding it to the AS is not covered by
the Matrix specification.
For SMS routing, options are:
* Terminate traffic only (from a shared shortcode originator)
* Two-way traffic via a VMN. To save allocating huge numbers of VMNs to Matrix users, the VMN can be allocated from a pool such that each {caller,callee} tuple is unique (but the caller number will only work from that specific callee).

57
drafts/typing_notifications.rst
View file

@ -1,57 +0,0 @@
Typing Notifications
====================
Client APIs
-----------
To set "I am typing for the next N msec"::
PUT .../rooms/:room_id/typing/:user_id
Content: { "typing": true, "timeout": N }
# timeout is in msec; I suggest no more than 20 or 30 seconds
This should be re-sent by the client to continue informing the server the user
is still typing; I suggest a safety margin of 5 seconds before the expected
timeout runs out. Just keep declaring a new timeout, it will replace the old
one.
To set "I am no longer typing"::
PUT ../rooms/:room_id/typing/:user_id
Content: { "typing": false }
Client Events
-------------
All room members will receive an event on the event stream::
{
"type": "m.typing",
"room_id": "!room-id-here:matrix.org",
"content": {
"user_ids": ["list of", "every user", "who is", "currently typing"]
}
}
The client must use this list to *REPLACE* its knowledge of every user who is
currently typing. The reason for this is that the server DOES NOT remember
users who are not currently typing, as that list gets big quickly. The client
should mark as not typing, any user ID who is not in that list.
Server APIs
-----------
Servers will emit EDUs in the following form::
{
"type": "m.typing",
"content": {
"room_id": "!room-id-here:matrix.org",
"user_id": "@user-id-here:matrix.org",
"typing": true/false,
}
}
Server EDUs don't (currently) contain timing information; it is up to
originating HSes to ensure they eventually send "stop" notifications.
((This will eventually need addressing, as part of the wider typing/presence
timer addition work))