wholetrans/docs-matrix-spec
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Merge branch 'master' into human-id-rules

Browse source
This commit is contained in:
Richard van der Hoff 2017-09-26 14:54:14 +01:00
commit aebfcda320
491 changed files with 21635 additions and 10852 deletions
Download patch file Download diff file Expand all files Collapse all files

14
drafts/address-book-repo.rst
View file

@ -1,14 +0,0 @@
Address book repository
=======================
.. NOTE::
This section is a work in progress.
.. TODO-spec
Do we even need it? Clients can use out-of-band addressbook servers for now;
this should definitely not be core.
- format: POST(?) wodges of json, some possible processing, then return wodges of json on GET.
- processing may remove dupes, merge contacts, pepper with extra info (e.g. matrix-ability of
contacts), etc.
- Standard json format for contacts? Piggy back off vcards?

8
drafts/ancient_federated_versioning_design_notes.rst
View file

@ -1,11 +1,11 @@
Versioning is, like, hard for backfilling backwards because of the number of Home Servers involved.
Versioning is, like, hard for backfilling backwards because of the number of homeservers involved.
The way we solve this is by doing versioning as an acyclic directed graph of PDUs. For backfilling purposes, this is done on a per context basis.
The way we solve this is by doing versioning as an acyclic directed graph of PDUs. For backfilling purposes, this is done on a per context basis.
When we send a PDU we include all PDUs that have been received for that context that hasn't been subsequently listed in a later PDU. The trivial case is a simple list of PDUs, e.g. A <- B <- C. However, if two servers send out a PDU at the same to, both B and C would point at A - a later PDU would then list both B and C.
Problems with opaque version strings:
- How do you do clustering without mandating that a cluster can only have one transaction in flight to a given remote home server at a time.
- How do you do clustering without mandating that a cluster can only have one transaction in flight to a given remote homeserver at a time.
If you have multiple transactions sent at once, then you might drop one transaction, receive another with a version that is later than the dropped transaction and which point ARGH WE LOST A TRANSACTION.
- How do you do backfilling? A version string defines a point in a stream w.r.t. a single home server, not a point in the context.
- How do you do backfilling? A version string defines a point in a stream w.r.t. a single homeserver, not a point in the context.
We only need to store the ends of the directed graph, we DO NOT need to do the whole one table of nodes and one of edges.

574
drafts/as-http-api.rst
View file

@ -1,574 +0,0 @@
.. TODO
Sometimes application services need to create rooms (e.g. when lazy loading
from room aliases). Created rooms need to have a user that created them, so
federation works (as it relies on an entry existing in m.room.member). We
should be able to add metadata to m.room.member to state that this user is an
application service, a virtual user, etc.
Application Services HTTP API
=============================
.. contents:: Table of Contents
.. sectnum::
Application Service -> Home Server
----------------------------------
This contains home server APIs which are used by the application service.
Registration API ``[Draft]``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This API registers the application service with its host homeserver to offer its
services.
Inputs:
- Credentials (e.g. some kind of string token)
- Namespace[users]
- Namespace[room aliases]
- URL base to receive inbound comms
Output:
- The credentials the HS will use to query the AS with in return. (e.g. some
kind of string token)
Side effects:
- The HS will start delivering events to the URL base specified if this 200s.
API called when:
- The application service wants to register with a brand new home server.
Notes:
- An application service can state whether they should be the only ones who
can manage a specified namespace. This is referred to as an "exclusive"
namespace. An exclusive namespace prevents humans and other application
services from creating/deleting entities in that namespace. Typically,
exclusive namespaces are used when the rooms represent real rooms on
another service (e.g. IRC). Non-exclusive namespaces are used when the
application service is merely augmenting the room itself (e.g. providing
logging or searching facilities).
- Namespaces are represented by POSIX extended regular expressions in JSON.
They look like::
users: [
{
"exclusive": true,
"regex": "@irc\.freenode\.net/.*"
}
]
::
POST /register
Request format
{
url: "https://my.application.service.com/matrix/",
as_token: "some_AS_token",
namespaces: {
users: [
{
"exclusive": true,
"regex": "@irc\.freenode\.net/.*"
}
],
aliases: [
{
"exclusive": true,
"regex": "#irc\.freenode\.net/.*"
}
],
rooms: [
{
"exclusive": true,
"regex": "!irc\.freenode\.net/.*"
}
]
}
}
Returns:
200 : Registration accepted.
400 : Namespaces do not conform to regex
401 : Credentials need to be supplied.
403 : AS credentials rejected.
200 OK response format
{
hs_token: "string"
}
Unregister API ``[Draft]``
~~~~~~~~~~~~~~~~~~~~~~~~~~
This API unregisters a previously registered AS from the home server.
Inputs:
- AS token
Output:
- None.
Side effects:
- The HS will stop delivering events to the URL base specified for this AS if
this 200s.
API called when:
- The application service wants to stop receiving all events from the HS.
::
POST /unregister
Request format
{
as_token: "string"
}
Home Server -> Application Service
----------------------------------
This contains application service APIs which are used by the home server.
User Query ``[Draft]``
~~~~~~~~~~~~~~~~~~~~~~
This API is called by the HS to query the existence of a user on the Application
Service's namespace.
Inputs:
- User ID
- HS Credentials
Output:
- Whether the user exists.
Side effects:
- User is created on the HS by the AS via CS APIs during the processing of this request.
API called when:
- HS receives an event for an unknown user ID in the AS's namespace, e.g. an
invite event to a room.
Notes:
- When the AS receives this request, if the user exists, it must create the user via
the CS API.
- It can also set arbitrary information about the user (e.g. display name, join rooms, etc)
using the CS API.
- When this setup is complete, the AS should respond to the HS request. This means the AS
blocks the HS until the user is created.
- This is deemed more flexible than alternative methods (e.g. returning a JSON blob with the
user's display name and get the HS to provision the user).
Retry notes:
- The home server cannot respond to the client's request until the response to
this API is obtained from the AS.
- Recommended that home servers try a few times then time out, returning a
408 Request Timeout to the client.
::
GET /users/$user_id?access_token=$hs_token
Returns:
200 : User is recognised.
404 : User not found.
401 : Credentials need to be supplied.
403 : HS credentials rejected.
200 OK response format
{}
Room Alias Query ``[Draft]``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This API is called by the HS to query the existence of a room alias on the
Application Service's namespace.
Inputs:
- Room alias
- HS Credentials
Output:
- Whether the room exists.
Side effects:
- Room is created on the HS by the AS via CS APIs during the processing of
this request.
API called when:
- HS receives an event to join a room alias in the AS's namespace.
Notes:
- When the AS receives this request, if the room exists, it must create the room via
the CS API.
- It can also set arbitrary information about the room (e.g. name, topic, etc)
using the CS API.
- It can send messages as other users in order to populate scrollback.
- When this setup is complete, the AS should respond to the HS request. This means the AS
blocks the HS until the room is created and configured.
- This is deemed more flexible than alternative methods (e.g. returning an initial sync
style JSON blob and get the HS to provision the room). It also means that the AS knows
the room ID -> alias mapping.
Retry notes:
- The home server cannot respond to the client's request until the response to
this API is obtained from the AS.
- Recommended that home servers try a few times then time out, returning a
408 Request Timeout to the client.
::
GET /rooms/$room_alias?access_token=$hs_token
Returns:
200 : Room is recognised.
404 : Room not found.
401 : Credentials need to be supplied.
403 : HS credentials rejected.
200 OK response format
{}
Pushing ``[Draft]``
~~~~~~~~~~~~~~~~~~~
This API is called by the HS when the HS wants to push an event (or batch of
events) to the AS.
Inputs:
- HS Credentials
- Event(s) to give to the AS
- HS-generated transaction ID
Output:
- None.
Data flows:
::
Typical
HS ---> AS : Home server sends events with transaction ID T.
<--- : AS sends back 200 OK.
AS ACK Lost
HS ---> AS : Home server sends events with transaction ID T.
<-/- : AS 200 OK is lost.
HS ---> AS : Home server retries with the same transaction ID of T.
<--- : AS sends back 200 OK. If the AS had processed these events
already, it can NO-OP this request (and it knows if it is the same
events based on the transacton ID).
Retry notes:
- If the HS fails to pass on the events to the AS, it must retry the request.
- Since ASes by definition cannot alter the traffic being passed to it (unlike
say, a Policy Server), these requests can be done in parallel to general HS
processing; the HS doesn't need to block whilst doing this.
- Home servers should use exponential backoff as their retry algorithm.
- Home servers MUST NOT alter (e.g. add more) events they were going to
send within that transaction ID on retries, as the AS may have already
processed the events.
Ordering notes:
- The events sent to the AS should be linearised, as they are from the event
stream.
- The home server will need to maintain a queue of transactions to send to
the AS.
::
PUT /transactions/$transaction_id?access_token=$hs_token
Request format
{
events: [
...
]
}
Client-Server v2 API Extensions
-------------------------------
Identity assertion
~~~~~~~~~~~~~~~~~~
The client-server API infers the user ID from the ``access_token`` provided in
every request. It would be an annoying amount of book-keeping to maintain tokens
for every virtual user. It would be preferable if the application service could
use the CS API with its own ``as_token`` instead, and specify the virtual user
they wish to be acting on behalf of. For real users, this would require
additional permissions granting the AS permission to masquerade as a matrix user.
Inputs:
- Application service token (``access_token``)
Either:
- User ID in the AS namespace to act as.
Or:
- OAuth2 token of real user (which may end up being an access token)
Notes:
- This will apply on all aspects of the CS API, except for Account Management.
- The ``as_token`` is inserted into ``access_token`` which is usually where the
client token is. This is done on purpose to allow application services to
reuse client SDKs.
::
/path?access_token=$token&user_id=$userid
Query Parameters:
access_token: The application service token
user_id: The desired user ID to act as.
/path?access_token=$token&user_token=$token
Query Parameters:
access_token: The application service token
user_token: The token granted to the AS by the real user
Timestamp massaging
~~~~~~~~~~~~~~~~~~~
The application service may want to inject events at a certain time (reflecting
the time on the network they are tracking e.g. irc, xmpp). Application services
need to be able to adjust the ``origin_server_ts`` value to do this.
Inputs:
- Application service token (``as_token``)
- Desired timestamp
Notes:
- This will only apply when sending events.
::
/path?access_token=$token&ts=$timestamp
Query Parameters added to the send event APIs only:
access_token: The application service token
ts: The desired timestamp
Server admin style permissions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The home server needs to give the application service *full control* over its
namespace, both for users and for room aliases. This means that the AS should
be able to create/edit/delete any room alias in its namespace, as well as
create/delete any user in its namespace. No additional API changes need to be
made in order for control of room aliases to be granted to the AS. Creation of
users needs API changes in order to:
- Work around captchas.
- Have a 'passwordless' user.
This involves bypassing the registration flows entirely. This is achieved by
including the AS token on a ``/register`` request, along with a login type of
``m.login.application_service`` to set the desired user ID without a password.
::
/register?access_token=$as_token
Content:
{
type: "m.login.application_service",
user: "<desired user localpart in AS namespace>"
}
Application services which attempt to create users or aliases *outside* of
their defined namespaces will receive an error code ``M_EXCLUSIVE``. Similarly,
normal users who attempt to create users or alises *inside* an application
service-defined namespace will receive the same ``M_EXCLUSIVE`` error code.
ID conventions ``[Draft]``
--------------------------
.. NOTE::
- Giving HSes the freedom to namespace still feels like the Right Thing here.
- Exposing a public API provides the consistency which was the main complaint
against namespacing.
- This may have knock-on effects for the AS registration API. E.g. why don't
we let ASes specify the *URI* regex they want?
This concerns the well-defined conventions for mapping 3P network IDs to matrix
IDs, which we expect clients to be able to do by themselves.
User IDs
~~~~~~~~
Matrix users may wish to directly contact a virtual user, e.g. to send an email.
The URI format is a well-structured way to represent a number of different ID
types, including:
- MSISDNs (``tel``)
- Email addresses (``mailto``)
- IRC nicks (``irc`` - https://tools.ietf.org/html/draft-butcher-irc-url-04)
- XMPP (xep-0032)
- SIP URIs (RFC 3261)
As a result, virtual user IDs SHOULD relate to their URI counterpart. This
mapping from URI to user ID can be expressed in a number of ways:
- Expose a C-S API on the HS which takes URIs and responds with user IDs.
- Munge the URI with the user ID.
Exposing an API would allow HSes to internally map user IDs however they like,
at the cost of an extra round trip (of which the response can be cached).
Munging the URI would allow clients to apply the mapping locally, but would force
user X on service Y to always map to the same munged user ID. Considering the
exposed API could just be applying this munging, there is more flexibility if
an API is exposed.
::
GET /_matrix/app/v1/user?uri=$url_encoded_uri
Returns 200 OK:
{
user_id: <complete user ID on local HS>
}
Room Aliases
~~~~~~~~~~~~
We may want to expose some 3P network rooms so Matrix users can join them directly,
e.g. IRC rooms. We don't want to expose every 3P network room though, e.g. mailto,
tel. Rooms which are publicly accessible (e.g. IRC rooms) can be exposed as an alias by
the application service. Private rooms (e.g. sending an email to someone) should not
be exposed in this way, but should instead operate using normal invite/join semantics.
Therefore, the ID conventions discussed below are only valid for public rooms which
expose room aliases.
Matrix users may wish to join XMPP rooms (e.g. using XEP-0045) or IRC rooms. In both
cases, these rooms can be expressed as URIs. For consistency, these "room" URIs
SHOULD be mapped in the same way as "user" URIs.
::
GET /_matrix/app/v1/alias?uri=$url_encoded_uri
Returns 200 OK:
{
alias: <complete room alias on local HS>
}
Event fields
~~~~~~~~~~~~
We recommend that any gatewayed events should include an ``external_url`` field
in their content to provide a way for Matrix clients to link into the 'native'
client from which the event originated. For instance, this could contain the
message-ID for emails/nntp posts, or a link to a blog comment when gatewaying
blog comment traffic in & out of matrix
Examples
--------
.. NOTE::
- User/Alias namespaces are subject to change depending on ID conventions.
IRC
~~~
Pre-conditions:
- Server admin stores the AS token "T_a" on the home server.
- Home server has a token "T_h".
- Home server has the domain "hsdomain.com"
1. Application service registration
::
AS -> HS: Registers itself with the home server
POST /register
{
url: "https://someapp.com/matrix",
as_token: "T_a",
namespaces: {
users: [
{
"exclusive": true,
"regex": "@irc\.freenode\.net/.*"
}
],
aliases: [
{
"exclusive": true,
"regex": "#irc\.freenode\.net/.*"
}
]
}
}
Returns 200 OK:
{
hs_token: "T_h"
}
2. IRC user "Bob" says "hello?" on "#matrix" at timestamp 1421416883133:
::
- AS stores message as potential scrollback.
- Nothing happens as no Matrix users are in the room.
3. Matrix user "@alice:hsdomain.com" wants to join "#matrix":
::
User -> HS: Request to join "#irc.freenode.net/#matrix:hsdomain.com"
HS -> AS: Room Query "#irc.freenode.net/#matrix:hsdomain.com"
GET /rooms/%23irc.freenode.net%2F%23matrix%3Ahsdomain.com?access_token=T_h
[Starts blocking]
AS -> HS: Creates room. Gets room ID "!aasaasasa:hsdomain.com".
AS -> HS: Sets room name to "#matrix".
AS -> HS: Sends message as ""@irc.freenode.net/Bob:hsdomain.com"
PUT /rooms/%21aasaasasa%3Ahsdomain.com/send/m.room.message
?access_token=T_a
&user_id=%40irc.freenode.net%2FBob%3Ahsdomain.com
&ts=1421416883133
{
body: "hello?"
msgtype: "m.text"
}
HS -> AS: User Query "@irc.freenode.net/Bob:hsdomain.com"
GET /users/%40irc.freenode.net%2FBob%3Ahsdomain.com?access_token=T_h
[Starts blocking]
AS -> HS: Creates user using CS API extension.
POST /register?access_token=T_a
{
type: "m.login.application_service",
user: "irc.freenode.net/Bob"
}
AS -> HS: Set user display name to "Bob".
[Finishes blocking]
[Finished blocking]
- HS sends room information back to client.
4. @alice:hsdomain.com says "hi!" in this room:
::
User -> HS: Send message "hi!" in room !aasaasasa:hsdomain.com
- HS sends message.
- HS sees the room ID is in the AS namespace and pushes it to the AS.
HS -> AS: Push event
PUT /transactions/1?access_token=T_h
{
events: [
{
content: {
body: "hi!",
msgtype: "m.text"
},
origin_server_ts: <generated by hs>,
user_id: "@alice:hsdomain.com",
room_id: "!aasaasasa:hsdomain.com",
type: "m.room.message"
}
]
}
- AS passes this through to IRC.
5. IRC user "Bob" says "what's up?" on "#matrix" at timestamp 1421418084816:
::
IRC -> AS: "what's up?"
AS -> HS: Send message via CS API extension
PUT /rooms/%21aasaasasa%3Ahsdomain.com/send/m.room.message
?access_token=T_a
&user_id=%40irc.freenode.net%2FBob%3Ahsdomain.com
&ts=1421418084816
{
body: "what's up?"
msgtype: "m.text"
}
- HS modifies the user_id and origin_server_ts on the event and sends it.

55
drafts/definitions.rst
View file

@ -1,55 +0,0 @@
Definitions
===========
(a relatively recent (Oct 2014) new list of definitions from Erik)
# *Event* -- A JSON object that represents a piece of information to be
distributed to the the room. The object includes a payload and metadata,
including a ``type`` used to indicate what the payload is for and how to process
them. It also includes one or more references to previous events.
# *Event graph* -- Events and their references to previous events form a
directed acyclic graph. All events must be a descendant of the first event in a
room, except for a few special circumstances.
# *State event* -- A state event is an event that has a non-null string valued
`state_key` field. It may also include a ``prev_state`` key referencing exactly
one state event with the same type and state key, in the same event graph.
# *State tree* -- A state tree is a tree formed by a collection of state events
that have the same type and state key (all in the same event graph.
# *State resolution algorithm* -- An algorithm that takes a state tree as input
and selects a single leaf node.
# *Current state event* -- The leaf node of a given state tree that has been
selected by the state resolution algorithm.
# *Room state* / *state dictionary* / *current state* -- A mapping of the pair
(event type, state key) to the current state event for that pair.
# *Room* -- An event graph and its associated state dictionary. An event is in
the room if it is part of the event graph.
# *Topological ordering* -- The partial ordering that can be extracted from the
event graph due to it being a DAG.
(The state definitions are purposely slightly ill-defined, since if we allow
deleting events we might end up with multiple state trees for a given event
type and state key pair.)
Federation specific
-------------------
# *(Persistent data unit) PDU* -- An encoding of an event for distribution of
the server to server protocol.
# *(Ephemeral data unit) EDU* -- A piece of information that is sent between
servers and doesn't encode an event.
Client specific
---------------
# *Child events* -- Events that reference a single event in the same room
independently of the event graph.
# *Collapsed events* -- Events that have all child events that reference it
included in the JSON object.

14
drafts/design_workflow.rst
View file

@ -1,14 +0,0 @@
Matrix Spec Design Workflow
===========================
1. Write use cases
2. Design data flows for use cases
3. Design generic API (factoring out commonalities where possible)
4. Design transport-specific API with justifications
5. Formalise transport-specific API as swagger or similar
6. Evolve the generic API design doc and transport-specific API into the actual spec.

6
drafts/erikj_federation.rst
View file

@ -237,7 +237,7 @@ Domain specific string
``room_id``
A domain specific string with prefix ``!`` that is static across all events
in a graph and uniquely identifies it. The ``domain`` should be that of the
home server that created the room (i.e., the server that generated the
homeserver that created the room (i.e., the server that generated the
first ``m.room.create`` event).
``sender``
@ -246,7 +246,7 @@ Domain specific string
User Id
A domain specific string with prefix ``@`` representing a user account. The
``domain`` is the home server of the user and is the server used to contact
``domain`` is the homeserver of the user and is the server used to contact
the user.
Joining a room
@ -280,7 +280,7 @@ can then process the join event itself.
Inviting a user
---------------
To invite a remote user to a room we need their home server to sign the invite
To invite a remote user to a room we need their homeserver to sign the invite
event. This is done by sending the event to the remote server, which then signs
the event, before distributing the invite to other servers.

123
drafts/event_schema.yaml
View file

@ -1,123 +0,0 @@
# JSON Schema for Matrix events. http://json-schema.org/
title: "Matrix Event"
type: object
properties:
auth_events:
description: The events needed to authenticate this event
$ref: "#/definitions/event_reference_list"
content:
type: object
description: The body of the event.
depth:
type: integer
description: A number one greater than that of any preceeding event.
minimum: 0
event_id:
$ref: "#/definitions/event_id"
hashes:
$ref: "#/definitions/hashes"
origin:
$ref: "#/definitions/server_id"
origin_server_ts:
type: integer
description: Posix timestamp on the originating server
minimum: 0
prev_events:
description: The event(s) this event came after.
$ref: "#/definitions/event_reference_list"
prev_state:
description: The state event(s) this event replaces.
$ref: "#/definitions/event_reference_list"
room_id:
$ref: "#/definitions/room_id"
sender:
oneOf:
- $ref: "#/definitions/user_id"
- $ref: "#/definitions/server_id"
state_key:
type: string
type:
type: string
unsigned:
type: object
required:
- auth_events
- content
- depth
- event_id
- hashes
- origin
- origin_server_ts
- prev_events
- room_id
- sender
- type
dependencies:
state_key:
- prev_state
prev_state:
- state_key
definitions:
server_id:
type: string
description: Identifies a server.
pattern: "^(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\\-]*[a-zA-Z0-9])\\.)*\
([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\\-]*[a-zA-Z0-9])\
(:[0-9]*[0-9])?$"
event_id:
type: string
description: Identifies an event.
pattern: "^\\$[^:]+:\
(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\\-]*[a-zA-Z0-9])\\.)*\
([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\\-]*[a-zA-Z0-9])\
(:[0-9]*[0-9])?$"
room_id:
type: string
description: Identifies a room.
pattern: "^\\![^:]+:\
(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\\-]*[a-zA-Z0-9])\\.)*\
([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\\-]*[a-zA-Z0-9])\
(:[0-9]*[0-9])?$"
user_id:
type: string
description: Identifies a user.
pattern: "^\\@[^:]+:\
(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\\-]*[a-zA-Z0-9])\\.)*\
([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\\-]*[a-zA-Z0-9])\
(:[0-9]*[0-9])?$"
base64:
type: string
description: Base64 string without padding.
pattern: "^[a-zA-Z0-9/+]*$"
hashes:
type: object
description: Hashes
minProperties: 1
additionalProperties:
$ref: "#/definitions/base64"
signatures:
type: object
description: Signatures
additionalProperties:
type: object
minProperties: 1
additionalProperties:
$ref: "#/definitions/base64"
event_reference:
type: array
minItems: 2
maxItems: 2
items:
- type: string
description: Event id of the referenced event.
$ref: "#/definitions/event_id"
- type: object
description: Reference hashes of the referenced event.
$ref: "#/definitions/hashes"
event_reference_list:
type: array
items:
$ref: "#/definitions/event_reference"

54
drafts/general_api.rst
View file

@ -61,7 +61,7 @@ This version will change the path prefix for HTTP:
- Version 2: ``/_matrix/client/v2``
Note the lack of the ``api`` segment. This is for consistency between other
home server path prefixes.
homeserver path prefixes.
Terminology:
- ``Chunk token`` : An opaque string which can be used to return another chunk
@ -169,16 +169,16 @@ Outputs:
``content`` key. Deleted message events are ``m.room.redaction`` events.
- New position in the stream. (chunk token)
State Events Ordering Notes:
- Home servers may receive state events over federation that are superceded by
state events previously sent to the client. The home server *cannot* send
- Homeservers may receive state events over federation that are superceded by
state events previously sent to the client. The homeserver *cannot* send
these events to the client else they would end up erroneously clobbering the
superceding state event.
- As a result, the home server reserves the right to omit sending state events
- As a result, the homeserver reserves the right to omit sending state events
which are known to be superceded already.
- This may result in missed *state* events. However, the state of the room will
always be eventually consistent.
Message Events Ordering Notes:
- Home servers may receive message events over federation that happened a long
- Homeservers may receive message events over federation that happened a long
time ago. The client may or may not be interested in these message events.
- For clients which do not store scrollback for a room (they discard events
after processing them), this is not a problem as they only care about the
@ -191,11 +191,11 @@ Message Events Ordering Notes:
- The event, when it comes down the stream, will indicate which event it comes
after.
Rejected events:
- A home server may find out via federation that it should not have accepted
- A homeserver may find out via federation that it should not have accepted
an event (e.g. to send a message/state event in a room). For example, it may
send an event to another home server and receive an auth event stating
send an event to another homeserver and receive an auth event stating
that the event should not have been sent.
- If this happens, the home server will send a ``m.room.redaction`` for the
- If this happens, the homeserver will send a ``m.room.redaction`` for the
event in question. This will be a local server event (not shared with other
servers).
- If the event was a state event, it will synthesise a new state event to
@ -206,7 +206,7 @@ Unknown rooms:
- You could receive events for rooms you are unaware of (e.g. you didn't do an
initial sync, or your HS lost its database and is told from another HS that
they are in this room). How do you handle this?
- The simplest option would be to redo the initial sync with a filter on the
- The simplest option would be to redo the initial sync with a filter on the
room ID you're unaware of. This would retrieve the room state so you can
display the room.
What data flows does it address:
@ -291,7 +291,7 @@ Scrollback API ``[Draft]``
but as a purely informational display thing it would be nice.
Additional Inputs:
- flag to say if the home server should do a backfill over federation
- flag to say if the homeserver should do a backfill over federation
Additional Outputs:
- whether there are more events on the local HS / over federation.
What data flows does it address:
@ -313,7 +313,7 @@ Additional Outputs:
Room Alias API ``[Draft]``
--------------------------
This provides mechanisms for creating and removing room aliases for a room on a
home server. Typically, any user in a room can make an alias for that room. The
homeserver. Typically, any user in a room can make an alias for that room. The
alias creator (or anyone in the room?) can delete that alias. Server admins can
also delete any alias on their server.
@ -323,7 +323,7 @@ Inputs:
- Room Alias
Output:
- Room ID
- List of home servers to join via.
- List of homeservers to join via.
Mapping a room to an alias:
@ -334,7 +334,7 @@ Inputs:
Output:
- Room alias
Notes:
- The home server may add restrictions e.g. the user must be in the room.
- The homeserver may add restrictions e.g. the user must be in the room.
Deleting a mapping:
@ -347,11 +347,11 @@ Output:
Published room list API ``[Draft]``
-----------------------------------
This provides mechanisms for searching for published rooms on a home server.
This provides mechanisms for searching for published rooms on a homeserver.
Inputs:
- Search text (e.g. room alias/name/topic to search on)
- Home server to search on (this may just be the URL hit for HTTP)
- Homeserver to search on (this may just be the URL hit for HTTP)
- Any existing pagination token, can be missing if this is the first hit.
- Limit for pagination
Output:
@ -378,7 +378,7 @@ Notes:
User Profile API ``[Draft]``
----------------------------
Every user on a home server has a profile. This profile is effectively a
Every user on a homeserver has a profile. This profile is effectively a
key-value store scoped to a user ID. It can include an ``avatar_url``,
``displayname`` and other metadata. Updates to a profile should propagate to
other interested users.
@ -435,7 +435,7 @@ This had a number of problems associated with it:
flicker.
- Name/avatar changes created more ``m.room.member`` events which meant
they needed to be included in the auth chains for federation. This
created long auth chains which is suboptimal since home servers need
created long auth chains which is suboptimal since homeservers need
to store the auth chains forever.
These problems can be resolved by creating an ``m.room.member.profile``
@ -448,7 +448,7 @@ However, this introduces its own set of problems, namely flicker. The
client would receive the ``m.room.member`` event first, followed by
the ``m.room.member.profile`` event, which could cause a flicker. In
addition, federation may not send both events in a single transaction,
resulting in missing information on the receiving home server.
resulting in missing information on the receiving homeserver.
For federation, these problems can be resolved by sending the
``m.room.member`` event as they are in v1 (with ``displayname`` and
@ -457,7 +457,7 @@ they cannot be in the ``unsigned`` part of the event. The receiving home
server will then extract these keys and create a server-generated
``m.room.member.profile`` event. To avoid confusion with duplicate
information, the ``avatar_url`` and ``displayname`` keys should be
removed from the ``m.room.member`` event by the receiving home server.
removed from the ``m.room.member`` event by the receiving homeserver.
When a client requests these events (either from the event stream
or from an initial sync), the server will send the generated
``m.room.member.profile`` event under the ``unsigned.profile`` key of the
@ -840,17 +840,17 @@ information per device to all other users just redirects the union problem to
the client, which will commonly be presenting this information as an icon
alongside the user.
When a client hits the event stream, the home server can treat the user as
When a client hits the event stream, the homeserver can treat the user as
"online". This behaviour should be able to be overridden to avoid flicker
during connection losses when the client is appear offline (e.g. device is
appear offline > goes into a tunnel > server times out > device regains
connection and hits the event stream forcing the device online before the
"appear offline" state can be set). When the client has not hit the event
stream for a certain period of time, the home server can treat the user as
stream for a certain period of time, the homeserver can treat the user as
"offline". The user can also set a global *per-user* appear offline flag.
The user should also be able to set their presence state via a direct API,
without having to hit the event stream. The home server will set a timer when
without having to hit the event stream. The homeserver will set a timer when
the connection ends, after which it will set that device to offline.
As the idle flag and online state is determined per device, there needs to be a
@ -970,12 +970,12 @@ the hashes the same is the best as that means clients do not need to request
the capabilities for the given hash.
On first signup, the client will attempt to send the hash and be most likely
refused by the home server as it does not know the full capability set for that
refused by the homeserver as it does not know the full capability set for that
hash. The client will then have to upload the full capability set to the home
server. The client will then be able to send the hash as normal.
When a client receives a hash, the client will either recognise the hash or
will have to request the capability set from their home server:
will have to request the capability set from their homeserver:
Inputs:
- Hash
@ -1070,7 +1070,7 @@ Main use cases for ``updates``:
- Call signalling (child events are ICE candidates, answer to the offer, and
termination)
- *Local* Delivery/Read receipts : "Local" means they are not shared with other
users on the same home server or via federation but *are* shared between
users on the same homeserver or via federation but *are* shared between
clients for the same user; useful for push notifications, read count markers,
etc. This is done to avoid the ``n^2`` problem for sending receipts, where
the vast majority of traffic tends towards sending more receipts.
@ -1168,11 +1168,11 @@ Events (breaking changes; event version 2) ``[Draft]``
when dealing with custom event types. E.g. ``_custom.event`` would allow
anything in the state key, ``_@custom.event`` would only allow user IDs in
the state key, etc.
- s/user_id/sender/g given that home servers can send events, not just users.
- s/user_id/sender/g given that homeservers can send events, not just users.
Server-generated events ``[Draft]``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Home servers may want to send events to their local clients or to other home
Homeservers may want to send events to their local clients or to other home
servers e.g. for server status notifications.
These events look like regular events but have a server domain name as the

14
drafts/human-id-rules.rst
View file

@ -32,7 +32,7 @@ These checks are:
User ID Localparts:
- MUST NOT contain a ``:`` or start with a ``@`` or ``.``
- MUST NOT contain one of the 107 blacklisted characters on this list:
- MUST NOT contain one of the 107 blacklisted characters on this list:
http://kb.mozillazine.org/Network.IDN.blacklist_chars
- After stripping " 0-9, +, -, [, ], _, and the space character it MUST NOT
contain characters from >1 language, defined by the `exemplar characters`_
@ -42,7 +42,7 @@ User ID Localparts:
Room Alias Localparts:
- MUST NOT contain a ``:``
- MUST NOT contain one of the 107 blacklisted characters on this list:
- MUST NOT contain one of the 107 blacklisted characters on this list:
http://kb.mozillazine.org/Network.IDN.blacklist_chars
- After stripping " 0-9, +, -, [, ], _, and the space character it MUST NOT
contain characters from >1 language, defined by the `exemplar characters`_
@ -74,14 +74,14 @@ http://cldr.unicode.org/
This check SHOULD be applied when the user ID is created, in order to prevent
registration with the same name and different capitalisations, e.g.
``@foo:bar`` vs ``@Foo:bar`` vs ``@FOO:bar``. Home servers MAY canonicalise
``@foo:bar`` vs ``@Foo:bar`` vs ``@FOO:bar``. Homeservers MAY canonicalise
the user ID to be completely lower-case if desired.
Rationale
=========
Each ID is split into segments (localpart/domain) around the ``:``. For
this reason, ``:`` is a reserved character and cannot be a localpart character.
Each ID is split into segments (localpart/domain) around the ``:``. For
this reason, ``:`` is a reserved character and cannot be a localpart character.
The 107 blacklisted characters are used to prevent non-printable characters and
spaces from being used. The decision to ban characters from more than 1 language
matches the behaviour of `Google Chrome for IDN handling`_. This is to protect
@ -98,7 +98,7 @@ fail the ID checks. A failed ID could look like ``@@xn--c1yn36f:domain.com``.
If a user ID fails the check, the user ID on the event is renamed. This doesn't
require extra work for clients, and users will see an odd user ID rather than a
spoofed name. Renaming is done in order to protect users of a given HS, so if a
spoofed name. Renaming is done in order to protect users of a given HS, so if a
malicious HS doesn't rename their IDs, it doesn't affect any other HS.
Room aliases cannot be rewritten as punycode and sent to the HS the alias is
@ -115,7 +115,7 @@ Other rejected solutions for failed checks
- Reject event: Outright rejection of the ID at the point of creation /
receiving event. Point of creation rejection is preferable to avoid the ID
entering the system in the first place. However, malicious HSes can just
allow the ID. Hence, other home servers must reject them if they see them in
allow the ID. Hence, other homeservers must reject them if they see them in
events. Client never sees the problem ID, provided the HS is correctly
implemented. However, it is difficult to ensure that ALL HSes will come to the
same conclusion (given the CLDR dataset does come out with new versions).

34
drafts/macaroons_caveats.rst
View file

@ -1,34 +0,0 @@
Macaroon Caveats
================
`Macaroons`_ are issued by Matrix servers as authorization tokens. Macaroons may be restricted by adding caveats to them.
.. _Macaroons: http://theory.stanford.edu/~ataly/Papers/macaroons.pdf
Caveats can only be used for reducing the scope of a token, never for increasing it. Servers are required to reject any macroon with a caveat that they do not understand.
Some caveats are specified in this specification, and must be understood by all servers. The use of non-standard caveats is allowed.
All caveats must take the form:
`key` `operator` `value`
where `key` is a non-empty string drawn from the character set [A-Za-z0-9_]
`operator` is a non-empty string which does not contain whitespace
`value` is a non-empty string
And these are joined by single space characters.
Specified caveats:
+-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+
| Caveat name | Description | Legal Values |
+-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+
| gen | Generation of the macaroon caveat spec. | 1 |
| user_id | ID of the user for which this macaroon is valid. | Pure equality check. Operator must be =. |
| type | The purpose of this macaroon. | access - used to authorize any action except token refresh |
| refresh - only used to authorize a token refresh |
| time | Time before/after which this macaroon is valid. | A POSIX timestamp in milliseconds (in UTC). |
| Operator < means the macaroon is valid before the timestamp, as interpreted by the server. |
| Operator > means the macaroon is valid after the timestamp, as interpreted by the server. |
| Operator == means the macaroon is valid at exactly the timestamp, as interpreted by the server.|
| Note that exact equality of time is largely meaningless. |
+-------------+--------------------------------------------------+------------------------------------------------------------------------------------------------+

12
drafts/model/presence.rst
View file

@ -5,14 +5,14 @@ A simple implementation of presence messaging has the ability to cause a large
amount of Internet traffic relating to presence updates. In order to minimise
the impact of such a feature, the following observations can be made:
* There is no point in a Home Server polling status for peers in a user's
* There is no point in a homeserver polling status for peers in a user's
presence list if the user has no clients connected that care about it.
* It is highly likely that most presence subscriptions will be symmetric - a
given user watching another is likely to in turn be watched by that user.
* It is likely that most subscription pairings will be between users who share
at least one Room in common, and so their Home Servers are actively
at least one Room in common, and so their homeservers are actively
exchanging message PDUs or transactions relating to that Room.
* Presence update messages do not need realtime guarantees. It is acceptable to
@ -25,7 +25,7 @@ promise to send them when required. Rather than actively polling for the
current state all the time, HSes can rely on their relative stability to only
push updates when required.
A Home Server should not rely on the longterm validity of this presence
A homeserver should not rely on the longterm validity of this presence
information, however, as this would not cover such cases as a user's server
crashing and thus failing to inform their peers that users it used to host are
no longer available online. Therefore, each promise of future updates should
@ -33,7 +33,7 @@ carry with a timeout value (whether explicit in the message, or implicit as some
defined default in the protocol), after which the receiving HS should consider
the information potentially stale and request it again.
However, because of the likelihood that two home servers are exchanging messages
However, because of the likelihood that two homeservers are exchanging messages
relating to chat traffic in a room common to both of them, the ongoing receipt
of these messages can be taken by each server as an implicit notification that
the sending server is still up and running, and therefore that no status changes
@ -98,7 +98,7 @@ The data model presented here puts the following requirements on the APIs:
Client-Server
-------------
Requests that a client can make to its Home Server
Requests that a client can make to its homeserver
* get/set current presence state
Basic enumeration + ability to set a custom piece of text
@ -128,7 +128,7 @@ Requests that a client can make to its Home Server
Server-Server
-------------
Requests that Home Servers make to others
Requests that homeservers make to others
* request permission to add a user to presence list

16
drafts/model/profiles.rst
View file

@ -9,7 +9,7 @@ Overview
========
Internally within Synapse users are referred to by an opaque ID, which consists
of some opaque localpart combined with the domain name of their home server.
of some opaque localpart combined with the domain name of their homeserver.
Obviously this does not yield a very nice user experience; users would like to
see readable names for other users that are in some way meaningful to them.
Additionally, users like to be able to publish "profile" details to inform other
@ -59,7 +59,7 @@ servers should be accounted for here.]]
Visibility Permissions
======================
A home server implementation could offer the ability to set permissions on
A homeserver implementation could offer the ability to set permissions on
limited visibility of those fields. When another user requests access to the
target user's profile, their own identity should form part of that request. The
HS implementation can then decide which fields to make available to the
@ -130,12 +130,12 @@ namespace to allocate names into.
It would also be nice from a user experience perspective if the profile that a
given name links to can also declare that name as part of its metadata.
Furthermore as a security and consistency perspective it would be nice if each
end (the directory server and the user's home server) check the validity of the
end (the directory server and the user's homeserver) check the validity of the
mapping in some way. This needs investigation from a security perspective to
ensure against spoofing.
One such model may be that the user starts by declaring their intent to use a
given user name link to their home server, which then contacts the directory
given user name link to their homeserver, which then contacts the directory
service. At some point later (maybe immediately for "public open FCFS servers",
maybe after some kind of human intervention for verification) the DS decides to
honour this link, and includes it in its served output. It should also tell the
@ -170,7 +170,7 @@ balancing choice on behalf of the user who would choose, or not, to make use of
such a feature to publish their information.
Additionally, unless some form of strong end-to-end user-based encryption is
used, a user of ACLs for information privacy has to trust other home servers not
used, a user of ACLs for information privacy has to trust other homeservers not
to lie about the identify of the user requesting access to the Profile.
@ -182,7 +182,7 @@ The data model presented here puts the following requirements on the APIs:
Client-Server
-------------
Requests that a client can make to its Home Server
Requests that a client can make to its homeserver
* get/set my Display Name
This should return/take a simple "text/plain" field
@ -207,7 +207,7 @@ TODO(paul): At some later stage we should consider the API for:
Server-Server
-------------
Requests that Home Servers make to others
Requests that homeservers make to others
* get a user's Display Name / Avatar
@ -221,7 +221,7 @@ Requests that Home Servers make to others
Room Event PDU Types
--------------------
Events that are pushed from Home Servers to other Home Servers or clients.
Events that are pushed from homeservers to other homeservers or clients.
* user Display Name change

12
drafts/model/room-join-workflow.rst
View file

@ -8,7 +8,7 @@ Discovery
=========
To join a room, a user has to discover the room by some mechanism in order to
obtain the (opaque) Room ID and a candidate list of likely home servers that
obtain the (opaque) Room ID and a candidate list of likely homeservers that
contain it.
Sending an Invitation
@ -21,7 +21,7 @@ The inviter's HS sets the membership status of the invitee to "invited" in the
"m.members" state key by sending a state update PDU. The HS then broadcasts this
PDU among the existing members in the usual way. An invitation message is also
sent to the invited user, containing the Room ID and the PDU ID of this
invitation state change and potentially a list of some other home servers to use
invitation state change and potentially a list of some other homeservers to use
to accept the invite. The user's client can then choose to display it in some
way to alert the user.
@ -34,7 +34,7 @@ Directory Service
Alternatively, the user may discover the channel via a directory service; either
by performing a name lookup, or some kind of browse or search acitivty. However
this is performed, the end result is that the user's home server requests the
this is performed, the end result is that the user's homeserver requests the
Room ID and candidate list from the directory service.
[[TODO(paul): At present, no API has been designed or described for this
@ -44,14 +44,14 @@ directory service]]
Joining
=======
Once the ID and home servers are obtained, the user can then actually join the
Once the ID and homeservers are obtained, the user can then actually join the
room.
Accepting an Invite
-------------------
If a user has received and accepted an invitation to join a room, the invitee's
home server can now send an invite acceptance message to a chosen candidate
homeserver can now send an invite acceptance message to a chosen candidate
server from the list given in the invitation, citing also the PDU ID of the
invitation as "proof" of their invite. (This is required as due to late message
propagation it could be the case that the acceptance is received before the
@ -85,7 +85,7 @@ can instead post a "knock" message, which informs other members of the room that
the would-be joiner wishes to become a member and sets their membership value to
"knocked". If any of them wish to accept this, they can then send an invitation
in the usual way described above. Knowing that the user has already knocked and
expressed an interest in joining, the invited user's home server should
expressed an interest in joining, the invited user's homeserver should
immediately accept that invitation on the user's behalf, and go on to join the
room in the usual way.

20
drafts/model/rooms.rst
View file

@ -18,19 +18,19 @@ users, and other management and miscellaneous metadata), and a message history.
Room Identity and Naming
========================
Rooms can be arbitrarily created by any user on any home server; at which point
the home server will sign the message that creates the channel, and the
Rooms can be arbitrarily created by any user on any homeserver; at which point
the homeserver will sign the message that creates the channel, and the
fingerprint of this signature becomes the strong persistent identify of the
room. This now identifies the room to any home server in the network regardless
room. This now identifies the room to any homeserver in the network regardless
of its original origin. This allows the identify of the room to outlive any
particular server. Subject to appropriate permissions [to be discussed later],
any current member of a room can invite others to join it, can post messages
that become part of its history, and can change the persistent state of the room
(including its current set of permissions).
Home servers can provide a directory service, allowing a lookup from a
Homeservers can provide a directory service, allowing a lookup from a
convenient human-readable form of room label to a room ID. This mapping is
scoped to the particular home server domain and so simply represents that server
scoped to the particular homeserver domain and so simply represents that server
administrator's opinion of what room should take that label; it does not have to
be globally replicated and does not form part of the stored state of that room.
@ -156,7 +156,7 @@ m.public_history
to be a member of the room.
m.archive_servers
For "public" rooms with public history, gives a list of home servers that
For "public" rooms with public history, gives a list of homeservers that
should be included in message distribution to the room, even if no users on
that server are present. These ensure that a public room can still persist
even if no users are currently members of it. This list should be consulted by
@ -179,7 +179,7 @@ m.topic
Room Creation Templates
=======================
A client (or maybe home server?) could offer a few templates for the creation of
A client (or maybe homeserver?) could offer a few templates for the creation of
new rooms. For example, for a simple private one-to-one chat the channel could
assign the creator a power-level of 1, requiring a level of 1 to invite, and
needing an invite before members can join. An invite is then sent to the other
@ -215,7 +215,7 @@ permissions to allow this direct messaging.
Between any given pair of user IDs that wish to exchange private messages, there
will exist a single shared Room, created lazily by either side. These rooms will
need a certain amount of special handling in both home servers and display on
need a certain amount of special handling in both homeservers and display on
clients, but as much as possible should be treated by the lower layers of code
the same as other rooms.
@ -226,7 +226,7 @@ clients should display these in a special way too as the room name is not
important; instead it should distinguish them on the Display Name of the other
party.
Home Servers will need a client-API option to request setting up a new user-user
Homeservers will need a client-API option to request setting up a new user-user
chat room, which will then need special handling within the server. It will
create a new room with the following
@ -260,7 +260,7 @@ history with each other simultaneously create a room and invite the other to it.
This is called a "glare" situation. There are two possible ideas for how to
resolve this:
* Each Home Server should persist the mapping of (user ID pair) to room ID, so
* Each homeserver should persist the mapping of (user ID pair) to room ID, so
that duplicate requests can be suppressed. On receipt of a room creation
request that the HS thinks there already exists a room for, the invitation to
join can be rejected if:

2
drafts/model/third-party-id.rst
View file

@ -66,7 +66,7 @@ Privacy
A User may publish the association between their phone number and Matrix User ID
on the Identity Server without publishing the number in their Profile hosted on
their Home Server.
their homeserver.
Identity Servers should refrain from publishing reverse mappings and should
take steps, such as rate limiting, to prevent attackers enumerating the space of

76
drafts/reputation_thoughts.rst Normal file
View file

@ -0,0 +1,76 @@
(a stream of incoherent consciousness from matthew which should go somewhere)
Invite-graph based reputation data:
* Users need a reputation score to issue invites or join public rooms.
* A user can have many reputation scores in different audiences (and perhaps a global average?)
* A room (degenerate case: user) can align itself with a given audience in order to consume the reputation data for that audience.
* The people that a user invites inherits a proportion of their reputation.
* If your reputation in an audience is ever reduced, it similarly reduces the reputation you have ever conveyed to anyone else (which propagates through the invite graph).
* Users increase reputation by:
* Inviting someone.
* Upvoting their messages in a room (i.e. for the suitability of that audience)
* Users decrease reputation by:
* Blocking them.
* Downvoting their messages in a room (i.e. for the suitability of that audience)
Need to ensure the accounts are of a decent quality - making it harder to create sockpuppet accounts and associating them with real people is more important than the actual reputation problem.
Build a war game simulation to test?
Problems:
* How are audiences defined? Just a given unique set of users? Which then makes inheriting reputation easy between audiences - if the overlap is significant, the chances are the reputation rules are the same.
* But is it possible to have the same set of users in two different rooms have different rules for reputation? Probably yes, as the potential audience may include future invitees or indeed the general public, so history visibility rules should probably also contribute to this. But given privacy rules can change over time, each room should effectively define its own audience. So in the end, an audience === a room.
* Create a large network of fake users, and go and have them all vote up each other's score for a given audience.
* This can be solved if the root inviter is penalised, which then destroys all the reputation they conveyed to their graph.
Could Reputation == Power Level (!?!?!)
Inheritence semantics for reputation between different audiences is hard.
* You should base the reputation of a stranger on their reputation in other communities that you or your communities have some overlap with.
* Do you consider 2nd hand reputation data at all from private rooms? Or do you look only at the public reputation data?
How do you do these calculations in a byzantine world?
How do you do these calculations whilst preserving privacy?
* Only consider reputation data from rooms you are actually in?
* Store reputation data in room state?
* Have a function (HS? client? AS? spider?) that aggregates reputation data (and proves that the aggregation is accurate, almost like blockchain mining?)?
* Or have a separate reputation global db seperate from room state that people contribute metrics into (which gathers the aggregate data into a single place, and makes it easier to query reputation data for strangers)
How do you avoid backstabbing? (People maliciously ganging up on someone to downvote them)?
How do you avoid a voting war? (Community fragments; different factions turn up and try to downvote the other)?
* This is effectively two different audiences emerging in a single room.
* Perhaps this means we should model audiences separately from rooms.
* Perhaps audiences are literally ACL groups? And eventually, one might change the ACLs of a room to eject one of the groups?
* Or do you just synthesise audiences based on cliques of people who support each other? The act of upvoting someone is effectively aligning yourself as being part of the same audience?
So:
* Gather all public upvote/downvotes/invites/blocks in a global DB.
* Partition this into audiences based on who votes on who. Stuff which is read and not complained about could provide a small implicit approval? Although this makes it easy to flood content to boost your reputation, so bad idea.
* Partitioning algorithm could be quite subtle.
* You could end up with lots of small audiences (including invalid ones), and it's fairly unclear how they get aggregated into a single view. How should you treat a stranger who you have no audience-overlap with at all? Treat them as effectively having zero reputation from your perspective?
Problem:
* If the douchebag who invites spammers never says anything, how do you go vote on their reputation? Should there be some kind of back-propagation? Or is there explicitly a "this person invited a douchebag" downvote? Or hang on - how can they ever get reputation in the first place to invite their sockpuppets if they don't say anything (beyond the initial invite)?
* What if users simply don't talk in public? Is it right that we prevent them issuing invites just because they stick to private rooms? What about inviting people into those private rooms? I guess the point is that if these are public invites, then they need to have some kind of public reputation, or rely on out-of-band private invitation to establish trust?
* Are we rewarding people who don't change their habits? There's no time component considered here, and we punish people's entire history of invites and rep if they misbehave. The only way to escape is to create a new identity atm. Is this a feature or a bug?
* How does this handle people's accounts getting 0wn3d and doing things which wipe out their reputation? => This is always a risk; ignore it.
* Do you need a particular level of reputation to be able to vote on people?
Summary?
* Partition the global population into multiple overlapping clusters called 'audiences' based on mutual(?) upvote/downvote relationships in public rooms.
* Clusters of the same people but in different rooms could be modelled as separate (but overlapping) clusters.
* Each audience builds up a reputation score for the global population, blending in damped scores from overlapping audiences.
* Anyone can upvote/downvote, but the votes will not contribute to your personal opinion unless the voter overlaps with your audience's scoresheet.
* A room could adopt a given audience (that of the moderators'?) for considering the reputation of who can join, invite people, etc.
* A user uses their own 'audience of one' scoresheet to put a threshold on filtering out contact from other users (invites, messages, etc).
* Their personal scoresheet is presumably a blend of all the audiences they are already in.
* The act of inviting someone gives them some reputation, within your audiences, proportional to your own. Similarly blocking reduces reputation.
* If you are downvoted, it retrospectively reduces the weight of all of your upvote/downvotes (at least for audiences that the downvoter's opinion contributes to). Similarly for upvoting.
* This penalisation process is transitive.
Do we even need the penalisation stuff if audience partitioning works?

288
drafts/websockets.rst Normal file
View file

@ -0,0 +1,288 @@
WebSockets API
==============
Introduction
------------
This document is a proposal for a WebSockets-based client-server API. It is not
intended to replace the REST API, but rather to complement it and provide an
alternative interface for certain operations.
The primary goal is to offer a more efficient interface than the REST API: by
using a bidirectional protocol such as WebSockets we can avoid the overheads
involved in long-polling (SSL negotiation, HTTP headers, etc). In doing so we
will reduce the latency between server and client by allowing the server to
send events as soon as they arrive, rather than having to wait for a poll from
the client.
Note: This proposal got continued in a google document you can find here:
https://docs.google.com/document/d/104ClehFBgqLQbf4s-AKX2ijr8sOAxcizfcRs_atsB0g
Handshake
---------
1. Instead of calling ``/sync``, the client makes a websocket request to
``/_matrix/client/rN/stream``, passing the query parameters ``access_token``
and ``since``, and optionally ``filter`` - all of which have the same
meaning as for ``/sync``.
* The client sets the ``Sec-WebSocket-Protocol`` to ``m.json``. (Servers may
offer alternative encodings; at present only the JSON encoding is
specified but in future we will specify alternative encodings.)
#. The server returns the websocket handshake; the socket is then connected.
If the server does not return a valid websocket handshake, this indicates that
the server or an intermediate proxy does not support WebSockets. In this case,
the client should fall back to polling the ``/sync`` REST endpoint.
Example
~~~~~~~
Client request:
.. code:: http
GET /_matrix/client/v2_alpha/stream?access_token=123456&since=s72594_4483_1934 HTTP/1.1
Host: matrix.org
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: x3JJHMbDL1EzLkh9GBhXDw==
Sec-WebSocket-Protocol: m.json
Sec-WebSocket-Version: 13
Origin: https://matrix.org
Server response:
.. code:: http
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: HSmrc0sMlYUkAGmm5OPpG2HaGWk=
Sec-WebSocket-Protocol: m.json
Update Notifications
--------------------
Once the socket is connected, the server begins streaming updates over the
websocket. The server sends Update notifications about new messages or state
changes. To make it easy for clients to parse, Update notifications have the
same structure as the response to ``/sync``: an object with the following
members:
============= ========== ===================================================
Key Type Description
============= ========== ===================================================
next_batch string The batch token to supply in the ``since`` param of
the next /sync request. This is not required for
streaming of events over the WebSocket, but is
provided so that clients can reconnect if the
socket is disconnected.
presence Presence The updates to the presence status of other users.
rooms Rooms Updates to rooms.
============= ========== ===================================================
Example
~~~~~~~
Message from the server:
.. code:: json
{
"next_batch": "s72595_4483_1934",
"presence": {
"events": []
},
"rooms": {
"join": {},
"invite": {},
"leave": {}
}
}
Client-initiated operations
---------------------------
The client can perform certain operations by sending a websocket message to
the server. Such a "Request" message should be a JSON-encoded object with
the following members:
============= ========== ===================================================
Key Type Description
============= ========== ===================================================
id string A unique identifier for this request
method string Specifies the name of the operation to be
performed; see below for available operations
param object The parameters for the requested operation.
============= ========== ===================================================
The server responds to a client Request with a Response message. This is a
JSON-encoded object with the following members:
============= ========== ===================================================
Key Type Description
============= ========== ===================================================
id string The same as the value in the corresponding Request
object. The presence of the ``id`` field
distinguishes a Response message from an Update
notification.
result object On success, the results of the request.
error object On error, an object giving the resons for the
error. This has the same structure as the "standard
error response" for the Matrix API: an object with
the fields ``errcode`` and ``error``.
============= ========== ===================================================
Request methods
~~~~~~~~~~~~~~~
It is not intended that all operations which are available via the REST API
will be available via the WebSockets API, but a few simple, common operations
will be exposed. The initial operations will be as follows.
``ping``
^^^^^^^^
This is a no-op which clients may use to keep their connection alive.
The request ``params`` and the response ``result`` should be empty.
``send``
^^^^^^^^
Send a message event to a room. The parameters are as follows:
============= ========== ===================================================
Parameter Type Description
============= ========== ===================================================
room_id string **Required.** The room to send the event to
event_type string **Required.** The type of event to send.
content object **Required.** The content of the event.
============= ========== ===================================================
The result is as follows:
============= ========== ===================================================
Key Type Description
============= ========== ===================================================
event_id string A unique identifier for the event.
============= ========== ===================================================
The ``id`` from the Request message is used as the transaction ID by the
server.
``state``
^^^^^^^^^
Update the state on a room.
============= ========== ===================================================
Parameter Type Description
============= ========== ===================================================
room_id string **Required.** The room to set the state in
event_type string **Required.** The type of event to send.
state_key string **Required.** The state_key for the state to send.
content object **Required.** The content of the event.
============= ========== ===================================================
The result is as follows:
============= ========== ===================================================
Key Type Description
============= ========== ===================================================
event_id string A unique identifier for the event.
============= ========== ===================================================
Example
~~~~~~~
Client request:
.. code:: json
{
"id": "12345",
"method": "send",
"params": {
"room_id": "!d41d8cd:matrix.org",
"event_type": "m.room.message",
"content": {
"msgtype": "m.text",
"body": "hello"
}
}
}
Server response:
.. code:: json
{
"id": "12345",
"result": {
"event_id": "$66697273743031:matrix.org"
}
}
Alternative server response, in case of error:
.. code:: json
{
"id": "12345",
"error": {
"errcode": "M_MISSING_PARAM",
"error": "Missing parameter: event_type"
}
}
Rationale
---------
Alternatives to WebSockets include HTTP/2, CoAP, and simply rolling our own
protocol over raw TCP sockets. However, the need to implement browser-based
clients essentially reduces our choice to WebSockets. HTTP/2 streams will
probably provide an interesting alternative in the future, but current browsers
do not appear to give javascript applications low-level access to the protocol.
Concerning the continued use of the JSON encoding: we prefer to focus on the
transition to WebSockets initially. Replacing JSON with a compact
representation such as CBOR, MessagePack, or even just compressed JSON will be
a likely extension for the future. The support for negotiation of subprotocols
within WebSockets should make this a simple transition once time permits.
The number of methods available for client requests is deliberately limited, as
each method requires code to be written to map it onto the equivalent REST
implementation. Some REST methods - for instance, user registration and login -
would be pointless to expose via WebSockets. It is likely, however, that we
will increate the number of methods available via the WebSockets API as it
becomes clear which would be most useful.
Open questions
--------------
Throttling
~~~~~~~~~~
At least in v2 sync, clients are inherently self-throttling - if they do not
poll quickly enough, events will be dropped from the next result. This proposal
raises the possibility that events will be produced more quickly than they can
be sent to the client; backlogs will build up on the server and/or in the
intermediate network, which will not only lead to high latency on events being
delivered, but will lead to responses to client requests also being delayed.
We may need to implement some sort of throttling mechanism by which the server
can start to drop events. The difficulty is in knowing when to start dropping
events. A few ideas:
* Use websocket pings to measure the RTT; if it starts to increase, start
dropping events. But this requires knowledge of the base RTT, and a useful
model of what constitutes an excessive increase.
* Have the client acknowledge each batch of events, and use a window to ensure
the number of outstanding batches is limited. This is annoying as it requires
the client to have to acknowledge batches - and it's not clear what the right
window size is: we want a big window for long fat networks (think of mobile
clients), but a small one for one with lower latency.
* Start dropping events if the server's TCP buffer starts filling up. This has
the advantage of delegating the congestion-detection to TCP (which already
has a number of algorithms to deal with it, to greater or lesser
effectiveness), but relies on homeservers being hosted on OSes which use
sensible TCP congestion-avoidance algorithms, and more critically, an ability
to read the fill level of the TCP send buffer.