diff --git a/drafts/erik-model.rst b/drafts/erik-model.rst index c21c558d..b2f9908d 100644 --- a/drafts/erik-model.rst +++ b/drafts/erik-model.rst @@ -1,4 +1,7 @@ -This is a standalone description of the data architecture of Synapse. There is a lot of overlap with the currennt specification, so it has been split out here for posterity. Hopefully all the important bits have been merged into the relevant places in the main spec. +This is a standalone description of the data architecture of Synapse. There is a +lot of overlap with the current specification, so it has been split out here for +posterity. Hopefully all the important bits have been merged into the relevant +places in the main spec. Model diff --git a/drafts/general_api.rst b/drafts/general_api.rst index 2b42b8b0..6c87c0f6 100644 --- a/drafts/general_api.rst +++ b/drafts/general_api.rst @@ -555,7 +555,7 @@ signature. Requesting the "raw" federation event will have to return these keys. Account Management API ``[Draft]`` ---------------------------------- -The registration and login APIs in v2 do not support specifying device IDs. In v2, +The registration and login APIs in v1 do not support specifying device IDs. In v2, this will become *mandatory* when sending your initial request. Access tokens will be scoped per device, so using the same device ID twice when logging in will clobber the old access token. @@ -810,6 +810,10 @@ Notes: Presence API ``[Draft]`` ------------------------ + +FIXME: this seems to be ignoring activity timers entirely, which were present on +the planning etherpad and are present in the actual HTTP API. Needs attention. + The goals of presence are to: - Let other users know if someone is "online". diff --git a/drafts/media_repository.rst b/drafts/media_repository.rst deleted file mode 100644 index 879e1b85..00000000 --- a/drafts/media_repository.rst +++ /dev/null @@ -1,77 +0,0 @@ -Media Repository -================ - -File uploading and downloading. - -HTTP API --------- - -Uploads are POSTed to a resource which returns a token which is used to GET -the download. Uploads are POSTed to the sender's local homeserver, but are -downloaded from the recipient's local homeserver, which must thus first transfer -the content from the origin homeserver using the same API (unless the origin -and destination homeservers are the same). The upload/download API is:: - - => POST /_matrix/media/v1/upload HTTP/1.1 - Content-Type: - - - - <= HTTP/1.1 200 OK - Content-Type: application/json - - { "content-uri": "mxc:///" } - - => GET /_matrix/media/v1/download// HTTP/1.1 - - <= HTTP/1.1 200 OK - Content-Type: - Content-Disposition: attachment;filename= - - - -Clients can get thumbnails by supplying a desired width and height and -thumbnailing method:: - - => GET /_matrix/media/v1/thumbnail/ - /?width=&height=&method= HTTP/1.1 - - <= HTTP/1.1 200 OK - Content-Type: image/jpeg or image/png - - - -The thumbnail methods are "crop" and "scale". "scale" trys to return an -image where either the width or the height is smaller than the requested -size. The client should then scale and letterbox the image if it needs to -fit within a given rectangle. "crop" trys to return an image where the -width and height are close to the requested size and the aspect matches -the requested size. The client should scale the image if it needs to fit -within a given rectangle. - -Homeservers may generate thumbnails for content uploaded to remote -homeservers themselves or may rely on the remote homeserver to thumbnail -the content. Homeservers may return thumbnails of a different size to that -requested. However homeservers should provide exact matches where reasonable. -Homeservers must never upscale images. - -Security --------- - -Clients may try to upload very large files. Homeservers should not store files -that are too large and should not serve them to clients. - -Clients may try to upload very large images. Homeservers should not attempt to -generate thumbnails for images that are too large. - -Remote homeservers may host very large files or images. Homeserver should not -proxy or thumbnail large files or images from remote homeservers. - -Clients may try to upload a large number of files. Homeservers should limit the -number and total size of media that can be uploaded by clients. - -Clients may try to access a large number of remote files through a homeserver. -Homeservers should restrict the number and size of remote files that it caches. - -Clients or remote homeservers may try to upload malicious files targeting -vulnerabilities in either the homeserver thumbnailing or the client decoders. diff --git a/drafts/object_model.rst b/drafts/object_model.rst index 1ae60b2c..0341d6a5 100644 --- a/drafts/object_model.rst +++ b/drafts/object_model.rst @@ -1,6 +1,5 @@ - - - +..TODO + What are the start & end tokens doing here?! :: diff --git a/scripts/gendoc.py b/scripts/gendoc.py index 1b008685..431b1710 100755 --- a/scripts/gendoc.py +++ b/scripts/gendoc.py @@ -101,7 +101,8 @@ def prepare_env(): pass def cleanup_env(): - shutil.rmtree("./tmp") + pass + #shutil.rmtree("./tmp") def main(): prepare_env() diff --git a/specification/00_basis.rst b/specification/00_basis.rst index e97d3406..37d1b9ef 100644 --- a/specification/00_basis.rst +++ b/specification/00_basis.rst @@ -1,8 +1,8 @@ Matrix Specification ==================== -.. NOTE:: - The git version of this document is ``$GIT_VERSION`` +Version: ``$GIT_VERSION`` +------------------------------------------- Table of Contents ================= @@ -13,11 +13,14 @@ Table of Contents Introduction ============ -Matrix is a new set of open APIs for open-federated Instant Messaging and VoIP -functionality, designed to create and support a new global real-time -communication ecosystem on the internet. This specification is the ongoing -result of standardising the APIs used by the various components of the Matrix -ecosystem to communicate with one another. +Matrix is a set of open APIs for open-federated Instant Messaging, VoIP and +Internet of Things communication, designed to create and support a new global +real-time communication ecosystem. The intention is to provide an open +decentralised pubsub fabric for the internet for securely persisting and +publishing/subscribing JSON objects. + +This specification is the ongoing result of standardising the APIs used by the +various components of the Matrix ecosystem to communicate with one another. .. WARNING:: The Matrix specification is still evolving: the APIs are not yet frozen @@ -97,8 +100,34 @@ Overview Architecture ------------ -Clients transmit data to other clients through home servers (HSes). Clients do -not communicate with each other directly. +Matrix defines APIs for synchronising extensible JSON objects known as +`events` between compatible clients, servers and services. Clients are +typically messaging/VoIP applications or IoT devices/hubs and communicate by +synchronising communication history with their `homeserver` using the +`Client-Server API`. Each homeserver stores the communication history and +account information for all of its clients, and shares data with the wider +Matrix ecosystem by synchronising communication history with other homeservers +and their clients. + +Clients typically communicate with each other by emitting events in the +context of a virtual `room`. Room data is replicated across *all of the +homeservers* whose users are participating in a given room. As such, *no +single homeserver has control or ownership over a given room*. Homeservers +model communication history as a partially ordered graph of events known as +the room's `event graph`, which is synchronised with eventual consistency +between the participating servers using the Server-Server API. Matrix optimises +for the the Availability and Partitioned properties of CAP theorem at the +expense of Consistency. + +For example, for client A to send a message to client B, client A performs an +HTTP PUT of the required JSON event on its homeserver (HS) using the +client-server API. A's HS appends this event to its copy of the room's event +graph, signing the message in the context of the graph for integrity. A's HS +then replicates the message to B's HS by performing an HTTP PUT using the +server-server API. B's HS authenticates the request, validates the event's +signature, authorises the event's contents and then adds it to its copy of the +room's event graph. Client B then receives the message from his homeserver via +a long-lived GET request. :: @@ -107,21 +136,22 @@ not communicate with each other directly. { Matrix client A } { Matrix client B } ^ | ^ | - | events | | events | + | events | Client-Server API | events | | V | V +------------------+ +------------------+ | |---------( HTTPS )--------->| | | Home Server | | Home Server | | |<--------( HTTPS )----------| | - +------------------+ Federation +------------------+ + +------------------+ Server-Server API +------------------+ + History Synchronisation -A "Client" typically represents a human using a web application or mobile app. -Clients use the "Client-to-Server" (C-S) API to communicate with their home -server, which stores their profile data and their record of the conversations -in which they participate. Each client is associated with a user account (and -may optionally support multiple user accounts). A user account is represented -by a unique "User ID". This ID is namespaced to the home server which allocated -the account and looks like:: + +Users +~~~~~ + +Each client is associated with a user account, which is identified in Matrix +using a unique "User ID". This ID is namespaced to the home server which +allocated the account and has the form:: @localpart:domain @@ -131,19 +161,14 @@ this user. They are case-insensitive. .. TODO-spec - Need to specify precise grammar for Matrix IDs -A "Home Server" is a server which provides C-S APIs and has the ability to -federate with other HSes. It is typically responsible for multiple clients. -"Federation" is the term used to describe the sharing of data between two or -more home servers. - Events ~~~~~~ -Data in Matrix is encapsulated in an "event". An event is an action within the -system. Typically each action (e.g. sending a message) correlates with exactly -one event. Each event has a ``type`` which is used to differentiate different -kinds of data. ``type`` values MUST be uniquely globally namespaced following -Java's `package naming conventions +All data exchanged over Matrix is expressed as an "event". Typically each client +action (e.g. sending a message) correlates with exactly one event. Each event +has a ``type`` which is used to differentiate different kinds of data. ``type`` +values MUST be uniquely globally namespaced following Java's `package naming +conventions `, e.g. ``com.example.myapp.event``. The special top-level namespace ``m.`` is reserved for events defined in the Matrix specification - for instance ``m.room.message`` @@ -153,36 +178,39 @@ of a "Room". Event Graphs ~~~~~~~~~~~~ -Each event has a list of zero or more `parent` events. These relations form -directed acyclic graphs of events called `event graphs`. Every event graph has a single root event, and each event graph forms the -basis of the history of a matrix room. +Events exchanged in the context of a room are stored in a directed acyclic graph +(DAG) called an `event graph`. The partial ordering of this graph gives the +chronological ordering of events within the room. Each event in the graph has a +list of zero or more `parent` events, which refer to any preceeding events which +have no chronological successor from the perspective of the homeserver which +created the event. -Event graphs give a partial ordering of events, i.e. given two events one may -be considered to have come before the other if one is an ancestor of the other. -Since two events may be on separate branches, not all events can be compared in -this manner. +Typically an event has a single parent: the most recent message in the room at +the point it was sent. However, homeservers may legitimately race with each +other when sending messages, resulting in a single event having multiple +successors. The next event added to the graph thus will have multiple parents. +Every event graph has a single root event with no parent. -Every event has a metadata `depth` field that is a positive integer that is -strictly greater than the depths of any of its parents. The root event should -have a depth of 1. - -[Note: if one event is before another, then it must have a strictly smaller -depth] +To order and ease chronological comparison between the events within the graph, +homeservers maintain a `depth` metadata field on each event. An event's `depth` +is a positive integer that is strictly greater than the depths of any of its +parents. The root event should have a depth of 1. Thus if one event is before +another, then it must have a strictly smaller depth. Room structure ~~~~~~~~~~~~~~ -A room is a conceptual place where users can send and receive events. -Events are sent to a room, and all participants in -that room with sufficient access will receive the event. Rooms are uniquely -identified internally via a "Room ID", which look like:: +A room is a conceptual place where users can send and receive events. Events are +sent to a room, and all participants in that room with sufficient access will +receive the event. Rooms are uniquely identified internally via "Room IDs", +which have the form:: !opaque_id:domain There is exactly one room ID for each room. Whilst the room ID does contain a domain, it is simply for globally namespacing room IDs. The room does NOT reside on the domain specified. Room IDs are not meant to be human readable. -They ARE case-sensitive. +They are case-sensitive. The following conceptual diagram shows an ``m.room.message`` event being sent to the room ``!qporfwt:matrix.org``:: @@ -190,6 +218,7 @@ the room ``!qporfwt:matrix.org``:: { @alice:matrix.org } { @bob:domain.com } | ^ | | + [HTTP POST] [HTTP GET] Room ID: !qporfwt:matrix.org Room ID: !qporfwt:matrix.org Event type: m.room.message Event type: m.room.message Content: { JSON object } Content: { JSON object } @@ -200,7 +229,7 @@ the room ``!qporfwt:matrix.org``:: | matrix.org | | domain.com | +------------------+ +------------------+ | ^ - | | + | [HTTP POST] | | Room ID: !qporfwt:matrix.org | | Event type: m.room.message | | Content: { JSON object } | @@ -222,7 +251,7 @@ the room ``!qporfwt:matrix.org``:: | Content: { JSON object } | |...................................| -Federation maintains shared data structures per-room between multiple home +Federation maintains *shared data structures* per-room between multiple home servers. The data is split into ``message events`` and ``state events``. ``Message events`` describe transient 'once-off' activity in a room such as an @@ -252,7 +281,7 @@ participating in a room. Room Aliases ++++++++++++ -Each room can also have multiple "Room Aliases", which looks like:: +Each room can also have multiple "Room Aliases", which look like:: #room_alias:domain @@ -272,7 +301,7 @@ that are in the room that can be used to join via. :: - GET + HTTP GET #matrix:domain.com !aaabaa:matrix.org | ^ | | @@ -285,7 +314,7 @@ that are in the room that can be used to join via. |________________________________| Identity -++++++++ +~~~~~~~~ Users in Matrix are identified via their matrix user ID (MXID). However, existing 3rd party ID namespaces can also be used in order to identify Matrix @@ -306,47 +335,37 @@ the Matrix ecosystem. However, without one clients will not be able to look up user IDs using 3PIDs. Presence -++++++++ +~~~~~~~~ -Each user has the concept of presence information. This encodes the -"availability" of that user, suitable for display on other user's clients. This -is transmitted as an ``m.presence`` event and is one of the few events which -are sent *outside the context of a room*. The basic piece of presence -information is represented by the ``presence`` key, which is an enum of one of -the following: +Each user has the concept of presence information. This encodes: - - ``online`` : The default state when the user is connected to an event - stream. - - ``unavailable`` : The user is not reachable at this time. - - ``offline`` : The user is not connected to an event stream. - - ``free_for_chat`` : The user is generally willing to receive messages - moreso than default. - - ``hidden`` : Behaves as offline, but allows the user to see the client - state anyway and generally interact with client features. (Not yet - implemented in synapse). - -.. TODO-spec - This seems like a very constrained list of states - surely presence states - should be extensible, with us providing a baseline, and possibly a scale of - availability? For instance, do-not-disturb is missing here, as well as a - distinction between 'away' and 'busy'. + * Whether the user is currently online + * How recently the user was last active (as seen by the server) + * Whether a given client considers the user to be currently idle + * Arbitrary information about the user's current status (e.g. "in a meeting"). -This basic ``presence`` field applies to the user as a whole, regardless of how -many client devices they have connected. The presence state is pushed by the homeserver to all connected clients for a user to ensure a consistent experience for the user. +This information is collated from both per-device (online; idle; last_active) and +per-user (status) data, aggregated by the user's homeserver and transmitted as +an ``m.presence`` event. This is one of the few events which are sent *outside +the context of a room*. Presence events are sent to all users who subscribe to +this user's presence through a presence list or by sharing membership of a room. -.. TODO-spec - We need per-device presence in order to handle push notification semantics and similar. +.. TODO + How do we let users hide their presence information? -In addition, the server maintains a timestamp of the last time it saw a -pro-active event from the user; either sending a message to a room, or changing -presence state from a lower to a higher level of availability (thus: changing -state from ``unavailable`` to ``online`` counts as a proactive event, whereas in -the other direction it will not). This timestamp is presented via a key called -``last_active_ago``, which gives the relative number of milliseconds since the -message is generated/emitted that the user was last seen active. +.. TODO + The last_active specifics should be moved to the detailed presence event section + +Last activity is tracked by the server maintaining a timestamp of the last time +it saw a pro-active event from the user; either sending a message to a room, +coming online or back from idle, etc. This timestamp is presented via a key +called ``last_active_ago``, which gives the relative number of milliseconds +since the message is generated/emitted that the user was last seen active. -Presence List -~~~~~~~~~~~~~ +N.B. in v1 API, status/online/idle state are muxed into a single 'presence' field on the m.presence event. + +Presence Lists +~~~~~~~~~~~~~~ Each user's home server stores a "presence list". This stores a list of user IDs whose presence the user wants to follow. @@ -355,38 +374,31 @@ To be added to this list, the user being added must be invited by the list owner and accept the invitation. Once accepted, both user's HSes track the subscription. -Presence and Permissions -~~~~~~~~~~~~~~~~~~~~~~~~ - -For a viewing user to be allowed to see the presence information of a target -user, either: - - - The target user has allowed the viewing user to add them to their presence - list, or - - The two users share at least one room in common - -In the latter case, this allows for clients to display some minimal sense of -presence information in a user list for a room. - Profiles -++++++++ +~~~~~~~~ -.. TODO-spec - - Metadata extensibility +Users may publish arbitrary key/value data associated with their account - such +as a human readable ``display name``, a profile photo URL, contact information +(email address, phone nubers, website URLs etc). -Internally within Matrix users are referred to by their user ID, which is -typically a compact unique identifier. Profiles grant users the ability to see -human-readable names for other users that are in some way meaningful to them. -Additionally, profiles can publish additional information, such as the user's -age or location. +Profile data is typed using namespaced keys for interoperability, much like +events - e.g. ``m.profile.display_name``. -A Profile consists of a display name, an avatar picture, and a set of other -metadata fields that the user may wish to publish (email address, phone -numbers, website URLs, etc...). This specification puts no requirements on the -display name other than it being a valid unicode string. Avatar images are not -stored directly; instead the home server stores an ``http``-scheme URL from which clients may fetch the image. +..TODO + Actually specify the different types of data - e.g. what format are display + names allowed to be? +Private User Data +~~~~~~~~~~~~~~~~~ + +Users may also store arbitrary private key/value data in their account - such as +client preferences, or server configuration settings which lack any other +dedicated API. The API is symmetrical to managing Profile data. + +..TODO + Would it really be overengineered to use the same API for both profile & + private user data, but with different ACLs? API Standards ------------- @@ -430,6 +442,9 @@ For example, if there was a custom namespace ``com.mydomain.here``, and a ``COM.MYDOMAIN.HERE_FORBIDDEN``. There may be additional keys depending on the error, but the keys ``error`` and ``errcode`` MUST always be present. +..TODO + Why the weird mix of underscore and dots? + Some standard error codes are below: :``M_FORBIDDEN``: @@ -502,81 +517,3 @@ In contrast, these are invalid requests:: "key": "This is a put but it is missing a txnId." } -Glossary --------- - -Backfilling: - The process of synchronising historic state from one home server to another, - to backfill the event storage so that scrollback can be presented to the - client(s). Not to be confused with pagination. - -Context: - A single human-level entity of interest (currently, a chat room) - -EDU (Ephemeral Data Unit): - A message that relates directly to a given pair of home servers that are - exchanging it. EDUs are short-lived messages that related only to one single - pair of servers; they are not persisted for a long time and are not forwarded - on to other servers. Because of this, they have no internal ID nor previous - EDUs reference chain. - -Event: - A record of activity that records a single thing that happened on to a context - (currently, a chat room). These are the "chat messages" that Synapse makes - available. - -PDU (Persistent Data Unit): - A message that relates to a single context, irrespective of the server that - is communicating it. PDUs either encode a single Event, or a single State - change. A PDU is referred to by its PDU ID; the pair of its origin server - and local reference from that server. - -PDU ID: - The pair of PDU Origin and PDU Reference, that together globally uniquely - refers to a specific PDU. - -PDU Origin: - The name of the origin server that generated a given PDU. This may not be the - server from which it has been received, due to the way they are copied around - from server to server. The origin always records the original server that - created it. - -PDU Reference: - A local ID used to refer to a specific PDU from a given origin server. These - references are opaque at the protocol level, but may optionally have some - structured meaning within a given origin server or implementation. - -Presence: - The concept of whether a user is currently online, how available they declare - they are, and so on. See also: doc/model/presence - -Profile: - A set of metadata about a user, such as a display name, provided for the - benefit of other users. See also: doc/model/profiles - -Room ID: - An opaque string (of as-yet undecided format) that identifies a particular - room and used in PDUs referring to it. - -Room Alias: - A human-readable string of the form #name:some.domain that users can use as a - pointer to identify a room; a Directory Server will map this to its Room ID - -State: - A set of metadata maintained about a Context, which is replicated among the - servers in addition to the history of Events. - -User ID: - A string of the form @localpart:domain.name that identifies a user for - wire-protocol purposes. The localpart is meaningless outside of a particular - home server. This takes a human-readable form that end-users can use directly - if they so wish, avoiding the 3PIDs. - -Transaction: - A message which relates to the communication between a given pair of servers. - A transaction contains possibly-empty lists of PDUs and EDUs. - -.. TODO - This glossary contradicts the terms used above - especially on State Events v. "State" - and Non-State Events v. "Events". We need better consistent names. - diff --git a/specification/20_client_server_api.rst b/specification/10_client_server_api.rst similarity index 81% rename from specification/20_client_server_api.rst rename to specification/10_client_server_api.rst index af071d6f..ebea7bb9 100644 --- a/specification/20_client_server_api.rst +++ b/specification/10_client_server_api.rst @@ -1,7 +1,752 @@ Client-Server API v1 ==================== -This outlines version 1 of the client-server API. +Overview +-------- + +The client-server API provides a simple lightweight API to let clients send +messages, control rooms and synchronise conversation history. It is designed to +support both lightweight clients which store no state and lazy-load data from +the server as required - as well as heavyweight clients which maintain a full +local peristent copy of server state. + +This describes v1 of the Client-Server API as featured in the original September +2014 launch of Matrix. Version 2 is currently in development (as of Jan-March +2015) as an incremental but backwards-incompatible refinement of Version 1 and +will be released shortly. + +Pagination +---------- + +Querying large datasets in Matrix always uses the same pagination API pattern to +to give clients a consistent way of selecting subsets of a potentially changing +dataset. Requests pass in `from`, `to` and `limit` parameters which describe +where to read from the stream. `from` and `to` are opaque textual 'stream +tokens' which describe positions in the dataset. The response returns new +`start` and `end` stream token values which can then be passed to subsequent +requests to continue pagination. + +Pagination Request Query Parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Query parameters: + from: + $streamtoken - The opaque token to start streaming from. + to: + $streamtoken - The opaque token to end streaming at. Typically, + clients will not know the item of data to end at, so this will usually be + START or END. + limit: + integer - An integer representing the maximum number of items to + return. + +'START' and 'END' are magic token values which specify the start and end of the +dataset respectively. + +Unless specified, the default pagination parameters are from=START, to=END, +without a limit set. This allows you to hit an API like +/events without any query parameters to get everything. + +For example, the event stream has events E1 -> E15. The client wants the last 5 +events and doesn't know any previous events:: + + S E + |-E1-E2-E3-E4-E5-E6-E7-E8-E9-E10-E11-E12-E13-E14-E15-| + | | | + | _____| | + |__________________ | ___________________| + | | | + GET /events?to=START&limit=5&from=END + Returns: + E15,E14,E13,E12,E11 + + +Another example: a public room list has rooms R1 -> R17. The client is showing 5 +rooms at a time on screen, and is on page 2. They want to +now show page 3 (rooms R11 -> 15):: + + S E + | 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | stream token + |-R1-R2-R3-R4-R5-R6-R7-R8-R9-R10-R11-R12-R13-R14-R15-R16-R17| room + |____________| |________________| + | | + Currently | + viewing | + | + GET /rooms/list?from=9&to=END&limit=5 + Returns: R11,R12,R13,R14,R15 + +Note that tokens are treated in an *exclusive*, not inclusive, manner. The end +token from the intial request was '9' which corresponded to R10. When the 2nd +request was made, R10 did not appear again, even though from=9 was specified. If +you know the token, you already have the data. + +Pagination Response +~~~~~~~~~~~~~~~~~~~ + +Responses to pagination requests MUST follow the format:: + + { + "chunk": [ ... , Responses , ... ], + "start" : $streamtoken, + "end" : $streamtoken + } + +Where $streamtoken is an opaque token which can be used in another query to +get the next set of results. The "start" and "end" keys can only be omitted if +the complete dataset is provided in "chunk". + +If the client wants earlier results, they should use from=$start_streamtoken, +to=START. Likewise, if the client wants later results, they should use +from=$end_streamtoken, to=END. + + +Events +------ + +Overview +~~~~~~~~ + +The model of conversation history exposed by the client-server API can be +considered as a list of events. The server 'linearises' the +eventually-consistent event graph of events into an 'event stream' at any given +point in time:: + + [E0]->[E1]->[E2]->[E3]->[E4]->[E5]->[E6]->[E7]->[E8]->[E9] + +Clients can add to the stream by POSTing message or state events, and can read +from the stream via the |initialSync|_, |/rooms//initialSync|_, `Event +Stream`_ and |/rooms//messages|_ APIs. + +For reading events, the intended flow of operation is to call +$PREFIX/initialSync, which returns all of the state and the last N events in the +event stream for each room, including `start` and `end` values describing the +pagination of each room's event stream. For instance, +$PREFIX/initialSync?limit=5 might return the events for a room in the +rooms[0].messages.chunk[] array, with tokens describing the start and end of the +range in rooms[0].messages.start as '1-2-3' and rooms[0].messages.end as +'a-b-c'. + +You can visualise the range of events being returned as:: + + [E0]->[E1]->[E2]->[E3]->[E4]->[E5]->[E6]->[E7]->[E8]->[E9] + ^ ^ + | | + start: '1-2-3' end: 'a-b-c' + +Now, to receive future events in realtime on the eventstream, you simply GET +$PREFIX/events with a `from` parameter of 'a-b-c': in other words passing in the +`end` token returned by initialsync. The request blocks until new events are +available or until your specified timeout elapses, and then returns a +new paginatable chunk of events alongside new start and end parameters:: + + [E0]->[E1]->[E2]->[E3]->[E4]->[E5]->[E6]->[E7]->[E8]->[E9]->[E10] + ^ ^ + | | + | end: 'x-y-z' + start: 'a-b-c' + +To resume polling the events stream, you pass in the new `end` token as the +`from` parameter of $PREFIX/events and poll again. + +Similarly, to paginate events backwards in order to lazy-load in previous +history from the room, you simply GET $PREFIX/rooms//messages +specifying the `from` token to paginate backwards from and a limit of the number +of messages to retrieve. For instance, calling this API with a `from` parameter +of '1-2-3' and a limit of 5 would return: + + [E0]->[E1]->[E2]->[E3]->[E4]->[E5]->[E6]->[E7]->[E8]->[E9]->[E10] + ^ ^ + | | + start: 'u-v-w' end: '1-2-3' + +To continue paginating backwards, one calls the /messages API again, supplying +the new `start` value as the `from` parameter. + + +Receiving live updates on a client +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Clients receive new events by long-polling the home server via the +$PREFIX/events API, specifying a timeout in milliseconds in the timeout +parameter. This will hold open the HTTP connection for a short period of time +waiting for new events, returning early if an event occurs. This is called the +`Event Stream`_. All events which are visible to the client will appear in the +event stream. When the request returns, an ``end`` token is included in the +response. This token can be used in the next request to continue where the +last request left off. + +All events must be deduplicated based on their event ID (TODO: is this actually +a hard requirement in CS v2?) + +.. TODO-spec + Do we ever return multiple events in a single request? + Don't we get lots of request setup RTT latency if we only do one event per request? + Do we ever support streaming requests? Why not websockets? + +When the client first logs in, they will need to initially synchronise with +their home server. This is achieved via the |initialSync|_ API. This API also +returns an ``end`` token which can be used with the event stream. See the 'Room Sync' section below. + +Events in a room +~~~~~~~~~~~~~~~~ + +Room events are split into two categories: + +:Message events: + These are events which describe transient "once-off" activity in a room: + typically communication such as sending an instant messaage or setting up a + VoIP call. These used to be called 'non-state' events. + +:State Events: + These are events which update the metadata state of the room (e.g. room topic, + room membership etc). State is keyed by a tuple of event ``type`` and a + ``state_key``. State in the room with the same key-tuple will be overwritten. + +This specification outlines several events, all with the event type prefix +``m.``. However, applications may wish to add their own type of event, and this +can be achieved using the REST API detailed in the following sections. If new +events are added, the event ``type`` key SHOULD follow the Java package naming +convention, e.g. ``com.example.myapp.event``. This ensures event types are +suitably namespaced for each application and reduces the risk of clashes. + +State events +~~~~~~~~~~~~ + +State events can be sent by ``PUT`` ing to +|/rooms//state//|_. These events will be +overwritten if ````, ```` and ```` all match. +If the state event has no ``state_key``, it can be omitted from the path. These +requests **cannot use transaction IDs** like other ``PUT`` paths because they +cannot be differentiated from the ``state_key``. Furthermore, ``POST`` is +unsupported on state paths. Valid requests look like:: + + PUT /rooms/!roomid:domain/state/m.example.event + { "key" : "without a state key" } + + PUT /rooms/!roomid:domain/state/m.another.example.event/foo + { "key" : "with 'foo' as the state key" } + +In contrast, these requests are invalid:: + + POST /rooms/!roomid:domain/state/m.example.event/ + { "key" : "cannot use POST here" } + + PUT /rooms/!roomid:domain/state/m.another.example.event/foo/11 + { "key" : "txnIds are not supported" } + +Care should be taken to avoid setting the wrong ``state key``:: + + PUT /rooms/!roomid:domain/state/m.another.example.event/11 + { "key" : "with '11' as the state key, but was probably intended to be a txnId" } + +The ``state_key`` is often used to store state about individual users, by using +the user ID as the ``state_key`` value. For example:: + + PUT /rooms/!roomid:domain/state/m.favorite.animal.event/%40my_user%3Adomain.com + { "animal" : "cat", "reason": "fluffy" } + +In some cases, there may be no need for a ``state_key``, so it can be omitted:: + + PUT /rooms/!roomid:domain/state/m.room.bgd.color + { "color": "red", "hex": "#ff0000" } + +See `Room Events`_ for the ``m.`` event specification. + +Message events +~~~~~~~~~~~~~~ + +Message events can be sent by sending a request to +|/rooms//send/|_. These requests *can* use transaction +IDs and ``PUT``/``POST`` methods. Message events allow access to historical +events and pagination, making it best suited for sending messages. For +example:: + + POST /rooms/!roomid:domain/send/m.custom.example.message + { "text": "Hello world!" } + + PUT /rooms/!roomid:domain/send/m.custom.example.message/11 + { "text": "Goodbye world!" } + +See `Room Events`_ for the ``m.`` event specification. + +Syncing rooms +~~~~~~~~~~~~~ + +.. NOTE:: + This section is a work in progress. + +When a client logs in, they may have a list of rooms which they have already +joined. These rooms may also have a list of events associated with them. The +purpose of 'syncing' is to present the current room and event information in a +convenient, compact manner. The events returned are not limited to room events; +presence events will also be returned. A single syncing API is provided: + + - |initialSync|_ : A global sync which will present room and event information + for all rooms the user has joined. + +.. TODO-spec room-scoped initial sync + - |/rooms//initialSync|_ : A sync scoped to a single room. Presents + room and event information for this room only. + - Room-scoped initial sync is Very Tricky because typically people would + want to sync the room then listen for any new content from that point + onwards. The event stream cannot do this for a single room currently. + As a result, commenting room-scoped initial sync at this time. + +The |initialSync|_ API contains the following keys: + +``presence`` + Description: + Contains a list of presence information for users the client is interested + in. + Format: + A JSON array of ``m.presence`` events. + +``end`` + Description: + Contains an event stream token which can be used with the `Event Stream`_. + Format: + A string containing the event stream token. + +``rooms`` + Description: + Contains a list of room information for all rooms the client has joined, + and limited room information on rooms the client has been invited to. + Format: + A JSON array containing Room Information JSON objects. + +Room Information: + Description: + Contains all state events for the room, along with a limited amount of + the most recent events, configured via the ``limit`` query + parameter. Also contains additional keys with room metadata, such as the + ``room_id`` and the client's ``membership`` to the room. + Format: + A JSON object with the following keys: + ``room_id`` + A string containing the ID of the room being described. + ``membership`` + A string representing the client's membership status in this room. + ``messages`` + An event stream JSON object containing a ``chunk`` of recent + events (both state events and non-state events), along with an ``end`` token. + ``state`` + A JSON array containing all the current state events for this room. + +Getting events for a room +~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are several APIs provided to ``GET`` events for a room: + +``/rooms//state//`` + Description: + Get the state event identified. + Response format: + A JSON object representing the state event **content**. + Example: + ``/rooms/!room:domain.com/state/m.room.name`` returns ``{ "name": "Room name" }`` + +|/rooms//state|_ + Description: + Get all state events for a room. + Response format: + ``[ { state event }, { state event }, ... ]`` + Example: + TODO-doc + +|/rooms//members|_ + Description: + Get all ``m.room.member`` state events. + Response format: + ``{ "start": "", "end": "", "chunk": [ { m.room.member event }, ... ] }`` + Example: + TODO-doc + +|/rooms//messages|_ + Description: + Get all ``m.room.message`` and ``m.room.member`` events. This API supports + pagination using ``from`` and ``to`` query parameters, coupled with the + ``start`` and ``end`` tokens from an |initialSync|_ API. + + XXX: Is this accurate? Doesn't it return all events - not just m.room.message/member? + + Response format: + ``{ "start": "", "end": "" }`` + Example: + TODO-doc + +|/rooms//initialSync|_ + Description: + Get all relevant events for a room. This includes state events, paginated + non-state events and presence events. + Response format: + `` { TODO-doc } `` + Example: + TODO-doc + +Redactions +~~~~~~~~~~ +Since events are extensible it is possible for malicious users and/or servers +to add keys that are, for example offensive or illegal. Since some events +cannot be simply deleted, e.g. membership events, we instead 'redact' events. +This involves removing all keys from an event that are not required by the +protocol. This stripped down event is thereafter returned anytime a client or +remote server requests it. + +Events that have been redacted include a ``redacted_because`` key whose value +is the event that caused it to be redacted, which may include a reason. + +Redacting an event cannot be undone, allowing server owners to delete the +offending content from the databases. + +Currently, only room admins can redact events by sending a ``m.room.redaction`` +event, but server admins also need to be able to redact events by a similar +mechanism. + +Upon receipt of a redaction event, the server should strip off any keys not in +the following list: + + - ``event_id`` + - ``type`` + - ``room_id`` + - ``user_id`` + - ``state_key`` + - ``prev_state`` + - ``content`` + +The content object should also be stripped of all keys, unless it is one of +one of the following event types: + + - ``m.room.member`` allows key ``membership`` + - ``m.room.create`` allows key ``creator`` + - ``m.room.join_rules`` allows key ``join_rule`` + - ``m.room.power_levels`` allows keys that are user ids or ``default`` + - ``m.room.add_state_level`` allows key ``level`` + - ``m.room.send_event_level`` allows key ``level`` + - ``m.room.ops_levels`` allows keys ``kick_level``, ``ban_level`` + and ``redact_level`` + - ``m.room.aliases`` allows key ``aliases`` + +The redaction event should be added under the key ``redacted_because``. + + +When a client receives a redaction event it should change the redacted event +in the same way a server does. + + +Rooms +----- + +Creation +~~~~~~~~ +To create a room, a client has to use the |createRoom|_ API. There are various +options which can be set when creating a room: + +``visibility`` + Type: + String + Optional: + Yes + Value: + Either ``public`` or ``private``. + Description: + A ``public`` visibility indicates that the room will be shown in the public + room list. A ``private`` visibility will hide the room from the public room + list. Rooms default to ``private`` visibility if this key is not included. + +``room_alias_name`` + Type: + String + Optional: + Yes + Value: + The room alias localpart. + Description: + If this is included, a room alias will be created and mapped to the newly + created room. The alias will belong on the same home server which created + the room, e.g. ``!qadnasoi:domain.com >>> #room_alias_name:domain.com`` + +``name`` + Type: + String + Optional: + Yes + Value: + The ``name`` value for the ``m.room.name`` state event. + Description: + If this is included, an ``m.room.name`` event will be sent into the room to + indicate the name of the room. See `Room Events`_ for more information on + ``m.room.name``. + +``topic`` + Type: + String + Optional: + Yes + Value: + The ``topic`` value for the ``m.room.topic`` state event. + Description: + If this is included, an ``m.room.topic`` event will be sent into the room + to indicate the topic for the room. See `Room Events`_ for more information + on ``m.room.topic``. + +``invite`` + Type: + List + Optional: + Yes + Value: + A list of user ids to invite. + Description: + This will tell the server to invite everyone in the list to the newly + created room. + +Example:: + + { + "visibility": "public", + "room_alias_name": "thepub", + "name": "The Grand Duke Pub", + "topic": "All about happy hour" + } + +The home server will create a ``m.room.create`` event when the room is created, +which serves as the root of the PDU graph for this room. This event also has a +``creator`` key which contains the user ID of the room creator. It will also +generate several other events in order to manage permissions in this room. This +includes: + + - ``m.room.power_levels`` : Sets the power levels of users. + - ``m.room.join_rules`` : Whether the room is "invite-only" or not. + - ``m.room.add_state_level``: The power level required in order to add new + state to the room (as opposed to updating exisiting state) + - ``m.room.send_event_level`` : The power level required in order to send a + message in this room. + - ``m.room.ops_level`` : The power level required in order to kick or ban a + user from the room or redact an event in the room. + +See `Room Events`_ for more information on these events. + +Room aliases +~~~~~~~~~~~~ +.. NOTE:: + This section is a work in progress. + +Room aliases can be created by sending a ``PUT /directory/room/``:: + + { + "room_id": + } + +They can be deleted by sending a ``DELETE /directory/room/`` with +no content. Only some privileged users may be able to delete room aliases, e.g. +server admins, the creator of the room alias, etc. This specification does not +outline the privilege level required for deleting room aliases. + +As room aliases are scoped to a particular home server domain name, it is +likely that a home server will reject attempts to maintain aliases on other +domain names. This specification does not provide a way for home servers to +send update requests to other servers. + +Rooms store a *partial* list of room aliases via the ``m.room.aliases`` state +event. This alias list is partial because it cannot guarantee that the alias +list is in any way accurate or up-to-date, as room aliases can point to +different room IDs over time. Crucially, the aliases in this event are +**purely informational** and SHOULD NOT be treated as accurate. They SHOULD +be checked before they are used or shared with another user. If a room +appears to have a room alias of ``#alias:example.com``, this SHOULD be checked +to make sure that the room's ID matches the ``room_id`` returned from the +request. + +Room aliases can be checked in the same way they are resolved; by sending a +``GET /directory/room/``:: + + { + "room_id": , + "servers": [ , , ] + } + +Home servers can respond to resolve requests for aliases on other domains than +their own by using the federation API to ask other domain name home servers. + + +Permissions +~~~~~~~~~~~ +.. NOTE:: + This section is a work in progress. + +Permissions for rooms are done via the concept of power levels - to do any +action in a room a user must have a suitable power level. Power levels are +stored as state events in a given room. + +Power levels for users are defined in ``m.room.power_levels``, where both a +default and specific users' power levels can be set:: + + { + "": , + "": , + "default": 0 + } + +By default all users have a power level of 0, other than the room creator whose +power level defaults to 100. Users can grant other users increased power levels +up to their own power level. For example, user A with a power level of 50 could +increase the power level of user B to a maximum of level 50. Power levels for +users are tracked per-room even if the user is not present in the room. + +State events may contain a ``required_power_level`` key, which indicates the +minimum power a user must have before they can update that state key. The only +exception to this is when a user leaves a room, which revokes the user's right +to update state events in that room. + +To perform certain actions there are additional power level requirements +defined in the following state events: + +- ``m.room.send_event_level`` defines the minimum ``level`` for sending + non-state events. Defaults to 50. +- ``m.room.add_state_level`` defines the minimum ``level`` for adding new + state, rather than updating existing state. Defaults to 50. +- ``m.room.ops_level`` defines the minimum ``ban_level`` and ``kick_level`` to + ban and kick other users respectively. This defaults to a kick and ban levels + of 50 each. + + +Joining rooms +~~~~~~~~~~~~~ +.. TODO-doc What does the home server have to do to join a user to a room? + - See SPEC-30. + +Users need to join a room in order to send and receive events in that room. A +user can join a room by making a request to |/join/|_ with:: + + {} + +Alternatively, a user can make a request to |/rooms//join|_ with the +same request content. This is only provided for symmetry with the other +membership APIs: ``/rooms//invite`` and ``/rooms//leave``. If +a room alias was specified, it will be automatically resolved to a room ID, +which will then be joined. The room ID that was joined will be returned in +response:: + + { + "room_id": "!roomid:domain" + } + +The membership state for the joining user can also be modified directly to be +``join`` by sending the following request to +``/rooms//state/m.room.member/``:: + + { + "membership": "join" + } + +See the `Room events`_ section for more information on ``m.room.member``. + +After the user has joined a room, they will receive subsequent events in that +room. This room will now appear as an entry in the |initialSync|_ API. + +Some rooms enforce that a user is *invited* to a room before they can join that +room. Other rooms will allow anyone to join the room even if they have not +received an invite. + +Inviting users +~~~~~~~~~~~~~~ +.. TODO-doc Invite-join dance + - Outline invite join dance. What is it? Why is it required? How does it work? + - What does the home server have to do? + +The purpose of inviting users to a room is to notify them that the room exists +so they can choose to become a member of that room. Some rooms require that all +users who join a room are previously invited to it (an "invite-only" room). +Whether a given room is an "invite-only" room is determined by the room config +key ``m.room.join_rules``. It can have one of the following values: + +``public`` + This room is free for anyone to join without an invite. + +``invite`` + This room can only be joined if you were invited. + +Only users who have a membership state of ``join`` in a room can invite new +users to said room. The person being invited must not be in the ``join`` state +in the room. The fully-qualified user ID must be specified when inviting a +user, as the user may reside on a different home server. To invite a user, send +the following request to |/rooms//invite|_, which will manage the +entire invitation process:: + + { + "user_id": "" + } + +Alternatively, the membership state for this user in this room can be modified +directly by sending the following request to +``/rooms//state/m.room.member/``:: + + { + "membership": "invite" + } + +See the `Room events`_ section for more information on ``m.room.member``. + +Leaving rooms +~~~~~~~~~~~~~ +.. TODO-spec - HS deleting rooms they are no longer a part of. Not implemented. + - This is actually Very Tricky. If all clients a HS is serving leave a room, + the HS will no longer get any new events for that room, because the servers + who get the events are determined on the *membership list*. There should + probably be a way for a HS to lurk on a room even if there are 0 of their + members in the room. + - Grace period before deletion? + - Under what conditions should a room NOT be purged? + + +A user can leave a room to stop receiving events for that room. A user must +have joined the room before they are eligible to leave the room. If the room is +an "invite-only" room, they will need to be re-invited before they can re-join +the room. To leave a room, a request should be made to +|/rooms//leave|_ with:: + + {} + +Alternatively, the membership state for this user in this room can be modified +directly by sending the following request to +``/rooms//state/m.room.member/``:: + + { + "membership": "leave" + } + +See the `Room events`_ section for more information on ``m.room.member``. + +Once a user has left a room, that room will no longer appear on the +|initialSync|_ API. + +If all members in a room leave, that room becomes eligible for deletion. + +Banning users in a room +~~~~~~~~~~~~~~~~~~~~~~~ +A user may decide to ban another user in a room. 'Banning' forces the target +user to leave the room and prevents them from re-joining the room. A banned +user will not be treated as a joined user, and so will not be able to send or +receive events in the room. In order to ban someone, the user performing the +ban MUST have the required power level. To ban a user, a request should be made +to |/rooms//ban|_ with:: + + { + "user_id": "" + } + +Banning a user adjusts the banned member's membership state to ``ban`` and +adjusts the power level of this event to a level higher than the banned person. +Like with other membership changes, a user can directly adjust the target +member's state, by making a request to +``/rooms//state/m.room.member/``:: + + { + "membership": "ban" + } + Registration and Login ---------------------- @@ -347,587 +1092,6 @@ it should request a login fallback page:: This MUST return an HTML page which can perform the entire login process. -Events ------- - -Receiving live updates on a client -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Clients can receive new events by long-polling the home server. This will hold -open the HTTP connection for a short period of time waiting for new events, -returning early if an event occurs. This is called the `Event Stream`_. All -events which are visible to the client will appear in the event stream. When -the request returns, an ``end`` token is included in the response. This token -can be used in the next request to continue where the client left off. - -All events must be deduplicated based on their event ID (TODO: is this actually a -hard requirement in CS v2?) - -.. TODO-spec - How do we filter the event stream? - Do we ever return multiple events in a single request? Don't we get lots of request - setup RTT latency if we only do one event per request? Do we ever support streaming - requests? Why not websockets? - -When the client first logs in, they will need to initially synchronise with -their home server. This is achieved via the |initialSync|_ API. This API also -returns an ``end`` token which can be used with the event stream. - - -Rooms ------ - -Creation -~~~~~~~~ -To create a room, a client has to use the |createRoom|_ API. There are various -options which can be set when creating a room: - -``visibility`` - Type: - String - Optional: - Yes - Value: - Either ``public`` or ``private``. - Description: - A ``public`` visibility indicates that the room will be shown in the public - room list. A ``private`` visibility will hide the room from the public room - list. Rooms default to ``private`` visibility if this key is not included. - -``room_alias_name`` - Type: - String - Optional: - Yes - Value: - The room alias localpart. - Description: - If this is included, a room alias will be created and mapped to the newly - created room. The alias will belong on the same home server which created - the room, e.g. ``!qadnasoi:domain.com >>> #room_alias_name:domain.com`` - -``name`` - Type: - String - Optional: - Yes - Value: - The ``name`` value for the ``m.room.name`` state event. - Description: - If this is included, an ``m.room.name`` event will be sent into the room to - indicate the name of the room. See `Room Events`_ for more information on - ``m.room.name``. - -``topic`` - Type: - String - Optional: - Yes - Value: - The ``topic`` value for the ``m.room.topic`` state event. - Description: - If this is included, an ``m.room.topic`` event will be sent into the room - to indicate the topic for the room. See `Room Events`_ for more information - on ``m.room.topic``. - -``invite`` - Type: - List - Optional: - Yes - Value: - A list of user ids to invite. - Description: - This will tell the server to invite everyone in the list to the newly - created room. - -Example:: - - { - "visibility": "public", - "room_alias_name": "thepub", - "name": "The Grand Duke Pub", - "topic": "All about happy hour" - } - -The home server will create a ``m.room.create`` event when the room is created, -which serves as the root of the PDU graph for this room. This event also has a -``creator`` key which contains the user ID of the room creator. It will also -generate several other events in order to manage permissions in this room. This -includes: - - - ``m.room.power_levels`` : Sets the power levels of users. - - ``m.room.join_rules`` : Whether the room is "invite-only" or not. - - ``m.room.add_state_level``: The power level required in order to add new - state to the room (as opposed to updating exisiting state) - - ``m.room.send_event_level`` : The power level required in order to send a - message in this room. - - ``m.room.ops_level`` : The power level required in order to kick or ban a - user from the room or redact an event in the room. - -See `Room Events`_ for more information on these events. - -Room aliases -~~~~~~~~~~~~ -.. NOTE:: - This section is a work in progress. - -Room aliases can be created by sending a ``PUT /directory/room/``:: - - { - "room_id": - } - -They can be deleted by sending a ``DELETE /directory/room/`` with -no content. Only some privileged users may be able to delete room aliases, e.g. -server admins, the creator of the room alias, etc. This specification does not -outline the privilege level required for deleting room aliases. - -As room aliases are scoped to a particular home server domain name, it is -likely that a home server will reject attempts to maintain aliases on other -domain names. This specification does not provide a way for home servers to -send update requests to other servers. - -Rooms store a *partial* list of room aliases via the ``m.room.aliases`` state -event. This alias list is partial because it cannot guarantee that the alias -list is in any way accurate or up-to-date, as room aliases can point to -different room IDs over time. Crucially, the aliases in this event are -**purely informational** and SHOULD NOT be treated as accurate. They SHOULD -be checked before they are used or shared with another user. If a room -appears to have a room alias of ``#alias:example.com``, this SHOULD be checked -to make sure that the room's ID matches the ``room_id`` returned from the -request. - -Room aliases can be checked in the same way they are resolved; by sending a -``GET /directory/room/``:: - - { - "room_id": , - "servers": [ , , ] - } - -Home servers can respond to resolve requests for aliases on other domains than -their own by using the federation API to ask other domain name home servers. - - -Permissions -~~~~~~~~~~~ -.. NOTE:: - This section is a work in progress. - -Permissions for rooms are done via the concept of power levels - to do any -action in a room a user must have a suitable power level. Power levels are -stored as state events in a given room. - -Power levels for users are defined in ``m.room.power_levels``, where both a -default and specific users' power levels can be set:: - - { - "": , - "": , - "default": 0 - } - -By default all users have a power level of 0, other than the room creator whose -power level defaults to 100. Users can grant other users increased power levels -up to their own power level. For example, user A with a power level of 50 could -increase the power level of user B to a maximum of level 50. Power levels for -users are tracked per-room even if the user is not present in the room. - -State events may contain a ``required_power_level`` key, which indicates the -minimum power a user must have before they can update that state key. The only -exception to this is when a user leaves a room, which revokes the user's right -to update state events in that room. - -To perform certain actions there are additional power level requirements -defined in the following state events: - -- ``m.room.send_event_level`` defines the minimum ``level`` for sending - non-state events. Defaults to 50. -- ``m.room.add_state_level`` defines the minimum ``level`` for adding new - state, rather than updating existing state. Defaults to 50. -- ``m.room.ops_level`` defines the minimum ``ban_level`` and ``kick_level`` to - ban and kick other users respectively. This defaults to a kick and ban levels - of 50 each. - - -Joining rooms -~~~~~~~~~~~~~ -.. TODO-doc What does the home server have to do to join a user to a room? - - See SPEC-30. - -Users need to join a room in order to send and receive events in that room. A -user can join a room by making a request to |/join/|_ with:: - - {} - -Alternatively, a user can make a request to |/rooms//join|_ with the -same request content. This is only provided for symmetry with the other -membership APIs: ``/rooms//invite`` and ``/rooms//leave``. If -a room alias was specified, it will be automatically resolved to a room ID, -which will then be joined. The room ID that was joined will be returned in -response:: - - { - "room_id": "!roomid:domain" - } - -The membership state for the joining user can also be modified directly to be -``join`` by sending the following request to -``/rooms//state/m.room.member/``:: - - { - "membership": "join" - } - -See the `Room events`_ section for more information on ``m.room.member``. - -After the user has joined a room, they will receive subsequent events in that -room. This room will now appear as an entry in the |initialSync|_ API. - -Some rooms enforce that a user is *invited* to a room before they can join that -room. Other rooms will allow anyone to join the room even if they have not -received an invite. - -Inviting users -~~~~~~~~~~~~~~ -.. TODO-doc Invite-join dance - - Outline invite join dance. What is it? Why is it required? How does it work? - - What does the home server have to do? - -The purpose of inviting users to a room is to notify them that the room exists -so they can choose to become a member of that room. Some rooms require that all -users who join a room are previously invited to it (an "invite-only" room). -Whether a given room is an "invite-only" room is determined by the room config -key ``m.room.join_rules``. It can have one of the following values: - -``public`` - This room is free for anyone to join without an invite. - -``invite`` - This room can only be joined if you were invited. - -Only users who have a membership state of ``join`` in a room can invite new -users to said room. The person being invited must not be in the ``join`` state -in the room. The fully-qualified user ID must be specified when inviting a -user, as the user may reside on a different home server. To invite a user, send -the following request to |/rooms//invite|_, which will manage the -entire invitation process:: - - { - "user_id": "" - } - -Alternatively, the membership state for this user in this room can be modified -directly by sending the following request to -``/rooms//state/m.room.member/``:: - - { - "membership": "invite" - } - -See the `Room events`_ section for more information on ``m.room.member``. - -Leaving rooms -~~~~~~~~~~~~~ -.. TODO-spec - HS deleting rooms they are no longer a part of. Not implemented. - - This is actually Very Tricky. If all clients a HS is serving leave a room, - the HS will no longer get any new events for that room, because the servers - who get the events are determined on the *membership list*. There should - probably be a way for a HS to lurk on a room even if there are 0 of their - members in the room. - - Grace period before deletion? - - Under what conditions should a room NOT be purged? - - -A user can leave a room to stop receiving events for that room. A user must -have joined the room before they are eligible to leave the room. If the room is -an "invite-only" room, they will need to be re-invited before they can re-join -the room. To leave a room, a request should be made to -|/rooms//leave|_ with:: - - {} - -Alternatively, the membership state for this user in this room can be modified -directly by sending the following request to -``/rooms//state/m.room.member/``:: - - { - "membership": "leave" - } - -See the `Room events`_ section for more information on ``m.room.member``. - -Once a user has left a room, that room will no longer appear on the -|initialSync|_ API. - -If all members in a room leave, that room becomes eligible for deletion. - -Banning users in a room -~~~~~~~~~~~~~~~~~~~~~~~ -A user may decide to ban another user in a room. 'Banning' forces the target -user to leave the room and prevents them from re-joining the room. A banned -user will not be treated as a joined user, and so will not be able to send or -receive events in the room. In order to ban someone, the user performing the -ban MUST have the required power level. To ban a user, a request should be made -to |/rooms//ban|_ with:: - - { - "user_id": "" - } - -Banning a user adjusts the banned member's membership state to ``ban`` and -adjusts the power level of this event to a level higher than the banned person. -Like with other membership changes, a user can directly adjust the target -member's state, by making a request to -``/rooms//state/m.room.member/``:: - - { - "membership": "ban" - } - -Events in a room -~~~~~~~~~~~~~~~~ -Room events can be split into two categories: - -:State Events: - These are events which replace events that came before it, depending on a set - of unique keys. These keys are the event ``type`` and a ``state_key``. - Events with the same set of keys will be overwritten. Typically, state events - are used to store state, hence their name. - -:Non-state events: - These are events which cannot be overwritten after sending. The list of - events continues to grow as more events are sent. As this list grows, it - becomes necessary to provide a mechanism for navigating this list. Pagination - APIs are used to view the list of historical non-state events. Typically, - non-state events are used to send messages. - -This specification outlines several events, all with the event type prefix -``m.``. However, applications may wish to add their own type of event, and this -can be achieved using the REST API detailed in the following sections. If new -events are added, the event ``type`` key SHOULD follow the Java package naming -convention, e.g. ``com.example.myapp.event``. This ensures event types are -suitably namespaced for each application and reduces the risk of clashes. - -State events -~~~~~~~~~~~~ -State events can be sent by ``PUT`` ing to -|/rooms//state//|_. These events will be -overwritten if ````, ```` and ```` all match. -If the state event has no ``state_key``, it can be omitted from the path. These -requests **cannot use transaction IDs** like other ``PUT`` paths because they -cannot be differentiated from the ``state_key``. Furthermore, ``POST`` is -unsupported on state paths. Valid requests look like:: - - PUT /rooms/!roomid:domain/state/m.example.event - { "key" : "without a state key" } - - PUT /rooms/!roomid:domain/state/m.another.example.event/foo - { "key" : "with 'foo' as the state key" } - -In contrast, these requests are invalid:: - - POST /rooms/!roomid:domain/state/m.example.event/ - { "key" : "cannot use POST here" } - - PUT /rooms/!roomid:domain/state/m.another.example.event/foo/11 - { "key" : "txnIds are not supported" } - -Care should be taken to avoid setting the wrong ``state key``:: - - PUT /rooms/!roomid:domain/state/m.another.example.event/11 - { "key" : "with '11' as the state key, but was probably intended to be a txnId" } - -The ``state_key`` is often used to store state about individual users, by using -the user ID as the ``state_key`` value. For example:: - - PUT /rooms/!roomid:domain/state/m.favorite.animal.event/%40my_user%3Adomain.com - { "animal" : "cat", "reason": "fluffy" } - -In some cases, there may be no need for a ``state_key``, so it can be omitted:: - - PUT /rooms/!roomid:domain/state/m.room.bgd.color - { "color": "red", "hex": "#ff0000" } - -See `Room Events`_ for the ``m.`` event specification. - -Non-state events -~~~~~~~~~~~~~~~~ -Non-state events can be sent by sending a request to -|/rooms//send/|_. These requests *can* use transaction -IDs and ``PUT``/``POST`` methods. Non-state events allow access to historical -events and pagination, making it best suited for sending messages. For -example:: - - POST /rooms/!roomid:domain/send/m.custom.example.message - { "text": "Hello world!" } - - PUT /rooms/!roomid:domain/send/m.custom.example.message/11 - { "text": "Goodbye world!" } - -See `Room Events`_ for the ``m.`` event specification. - -Syncing rooms -~~~~~~~~~~~~~ -.. NOTE:: - This section is a work in progress. - -When a client logs in, they may have a list of rooms which they have already -joined. These rooms may also have a list of events associated with them. The -purpose of 'syncing' is to present the current room and event information in a -convenient, compact manner. The events returned are not limited to room events; -presence events will also be returned. A single syncing API is provided: - - - |initialSync|_ : A global sync which will present room and event information - for all rooms the user has joined. - -.. TODO-spec room-scoped initial sync - - |/rooms//initialSync|_ : A sync scoped to a single room. Presents - room and event information for this room only. - - Room-scoped initial sync is Very Tricky because typically people would - want to sync the room then listen for any new content from that point - onwards. The event stream cannot do this for a single room currently. - As a result, commenting room-scoped initial sync at this time. - -The |initialSync|_ API contains the following keys: - -``presence`` - Description: - Contains a list of presence information for users the client is interested - in. - Format: - A JSON array of ``m.presence`` events. - -``end`` - Description: - Contains an event stream token which can be used with the `Event Stream`_. - Format: - A string containing the event stream token. - -``rooms`` - Description: - Contains a list of room information for all rooms the client has joined, - and limited room information on rooms the client has been invited to. - Format: - A JSON array containing Room Information JSON objects. - -Room Information: - Description: - Contains all state events for the room, along with a limited amount of - the most recent non-state events, configured via the ``limit`` query - parameter. Also contains additional keys with room metadata, such as the - ``room_id`` and the client's ``membership`` to the room. - Format: - A JSON object with the following keys: - ``room_id`` - A string containing the ID of the room being described. - ``membership`` - A string representing the client's membership status in this room. - ``messages`` - An event stream JSON object containing a ``chunk`` of recent non-state - events, along with an ``end`` token. *NB: The name of this key will be - changed in a later version.* - ``state`` - A JSON array containing all the current state events for this room. - -Getting events for a room -~~~~~~~~~~~~~~~~~~~~~~~~~ -There are several APIs provided to ``GET`` events for a room: - -``/rooms//state//`` - Description: - Get the state event identified. - Response format: - A JSON object representing the state event **content**. - Example: - ``/rooms/!room:domain.com/state/m.room.name`` returns ``{ "name": "Room name" }`` - -|/rooms//state|_ - Description: - Get all state events for a room. - Response format: - ``[ { state event }, { state event }, ... ]`` - Example: - TODO-doc - - -|/rooms//members|_ - Description: - Get all ``m.room.member`` state events. - Response format: - ``{ "start": "", "end": "", "chunk": [ { m.room.member event }, ... ] }`` - Example: - TODO-doc - -|/rooms//messages|_ - Description: - Get all ``m.room.message`` and ``m.room.member`` events. This API supports - pagination using ``from`` and ``to`` query parameters, coupled with the - ``start`` and ``end`` tokens from an |initialSync|_ API. - Response format: - ``{ "start": "", "end": "" }`` - Example: - TODO-doc - -|/rooms//initialSync|_ - Description: - Get all relevant events for a room. This includes state events, paginated - non-state events and presence events. - Response format: - `` { TODO-doc } `` - Example: - TODO-doc - -Redactions -~~~~~~~~~~ -Since events are extensible it is possible for malicious users and/or servers -to add keys that are, for example offensive or illegal. Since some events -cannot be simply deleted, e.g. membership events, we instead 'redact' events. -This involves removing all keys from an event that are not required by the -protocol. This stripped down event is thereafter returned anytime a client or -remote server requests it. - -Events that have been redacted include a ``redacted_because`` key whose value -is the event that caused it to be redacted, which may include a reason. - -Redacting an event cannot be undone, allowing server owners to delete the -offending content from the databases. - -Currently, only room admins can redact events by sending a ``m.room.redaction`` -event, but server admins also need to be able to redact events by a similar -mechanism. - -Upon receipt of a redaction event, the server should strip off any keys not in -the following list: - - - ``event_id`` - - ``type`` - - ``room_id`` - - ``user_id`` - - ``state_key`` - - ``prev_state`` - - ``content`` - -The content object should also be stripped of all keys, unless it is one of -one of the following event types: - - - ``m.room.member`` allows key ``membership`` - - ``m.room.create`` allows key ``creator`` - - ``m.room.join_rules`` allows key ``join_rule`` - - ``m.room.power_levels`` allows keys that are user ids or ``default`` - - ``m.room.add_state_level`` allows key ``level`` - - ``m.room.send_event_level`` allows key ``level`` - - ``m.room.ops_levels`` allows keys ``kick_level``, ``ban_level`` - and ``redact_level`` - - ``m.room.aliases`` allows key ``aliases`` - -The redaction event should be added under the key ``redacted_because``. - - -When a client receives a redaction event it should change the redacted event -in the same way a server does. Presence ~~~~~~~~ @@ -1050,36 +1214,6 @@ have to wait in milliseconds before they can try again. homeserver come up with their own idea, causing totally unpredictable performance over federated rooms? -End-to-End Encryption -~~~~~~~~~~~~~~~~~~~~~ - -.. TODO-doc - - Why is this needed. - - Overview of process - - Implementation - -Content repository ------------------- -.. NOTE:: - This section is a work in progress. - -.. TODO-spec - - path to upload - - format for thumbnail paths, mention what it is protecting against. - - content size limit and associated M_ERROR. - - -Address book repository ------------------------ -.. NOTE:: - This section is a work in progress. - -.. TODO-spec - - format: POST(?) wodges of json, some possible processing, then return wodges of json on GET. - - processing may remove dupes, merge contacts, pepper with extra info (e.g. matrix-ability of - contacts), etc. - - Standard json format for contacts? Piggy back off vcards? - .. Links through the external API docs are below .. ============================================= diff --git a/specification/10_events.rst b/specification/20_events.rst similarity index 89% rename from specification/10_events.rst rename to specification/20_events.rst index 20dccb6b..84a40a11 100644 --- a/specification/10_events.rst +++ b/specification/20_events.rst @@ -250,7 +250,7 @@ prefixed with ``m.`` Summary: A message. Type: - Non-state event + Message event JSON format: ``{ "msgtype": "string" }`` Example: @@ -266,8 +266,9 @@ prefixed with ``m.`` ``m.room.message.feedback`` Summary: A receipt for a message. + N.B. not implemented in Synapse, and superceded in v2 CS API by the 'relates_to' event field. Type: - Non-state event + Message event JSON format: ``{ "type": "enum [ delivered|read ]", "target_event_id": "string" }`` Example: @@ -283,7 +284,7 @@ prefixed with ``m.`` Summary: Indicates a previous event has been redacted. Type: - Non-state event + Message event JSON format: ``{ "reason": "string" }`` Description: @@ -292,7 +293,7 @@ prefixed with ``m.`` admins to remove offensive or illegal content that may have been attached to any event. This cannot be undone, allowing server owners to physically delete the offending data. There is also a concept of a moderator hiding a - non-state event, which can be undone, but cannot be applied to state + message event, which can be undone, but cannot be applied to state events. The event that has been redacted is specified in the ``redacts`` event level key. @@ -447,6 +448,48 @@ outlined below: .. TODO-spec Make the definitions "inherit" from FileInfo where necessary... +Presence Events (v1) +~~~~~~~~~~~~~~~~~~~~ + +``m.presence`` + Summary: + Informs you of a user's presence state changes. + Type: + Presence event + JSON format:: + { "displayname": "utf-8 string", + "avatar_url": "url", + "presence": "enum [ online|unavailable|offline|free_for_chat|hidden ]", + "last_active_ago": "milliseconds" } + Example: + ``{ "displayname": "Matthew", "avatar_url": "mxc://domain/id", "presence": "online", "last_active_ago": 10000 }`` + Description: + Each user has the concept of presence information. This encodes the + "availability" of that user, suitable for display on other user's clients. + This is transmitted as an ``m.presence`` event and is one of the few events + which are sent *outside the context of a room*. The basic piece of presence + information is represented by the ``presence`` key, which is an enum of one + of the following: + + - ``online`` : The default state when the user is connected to an event + stream. + - ``unavailable`` : The user is not reachable at this time. + - ``offline`` : The user is not connected to an event stream. + - ``free_for_chat`` : The user is generally willing to receive messages + moreso than default. + - ``hidden`` : Behaves as offline, but allows the user to see the client + state anyway and generally interact with client features. (Not yet + implemented in synapse). + + In addition, the server maintains a timestamp of the last time it saw a + pro-active event from the user; either sending a message to a room, or + changing presence state from a lower to a higher level of availability + (thus: changing state from ``unavailable`` to ``online`` counts as a + proactive event, whereas in the other direction it will not). This timestamp + is presented via a key called ``last_active_ago``, which gives the relative + number of milliseconds since the message is generated/emitted that the user + was last seen active. + Events on Change of Profile Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -457,7 +500,7 @@ values. This change is conveyed using two separate mechanisms: - a ``m.room.member`` event is sent to every room the user is a member of, to update the ``displayname`` and ``avatar_url``. - - a presence status update is sent, again containing the new values of the + - a ``m.presence`` presence status update is sent, again containing the new values of the ``displayname`` and ``avatar_url`` keys, in addition to the required ``presence`` key containing the current presence state of the user. diff --git a/specification/40_application_service_api.rst b/specification/25_application_service_api.rst similarity index 100% rename from specification/40_application_service_api.rst rename to specification/25_application_service_api.rst diff --git a/specification/11_event_signing.rst b/specification/31_event_signing.rst similarity index 100% rename from specification/11_event_signing.rst rename to specification/31_event_signing.rst diff --git a/drafts/push_overview.rst b/specification/42_push_overview.rst similarity index 98% rename from drafts/push_overview.rst rename to specification/42_push_overview.rst index c84fc8d2..913b149f 100644 --- a/drafts/push_overview.rst +++ b/specification/42_push_overview.rst @@ -1,5 +1,5 @@ -Push Notifications -================== +Push Notifications Overview +=========================== :: @@ -72,3 +72,4 @@ Push Gateway For information on the client-server API for setting pushers and push rules, see the Client Server API section. For more information on the format of HTTP notifications, see the HTTP Notification Protocol section. + diff --git a/drafts/push_csapi.rst b/specification/43_push_cs_api.rst similarity index 99% rename from drafts/push_csapi.rst rename to specification/43_push_cs_api.rst index c5d9f280..3f235010 100644 --- a/drafts/push_csapi.rst +++ b/specification/43_push_cs_api.rst @@ -1,5 +1,5 @@ -Push Notifications -================== +Push Notifications HTTP API +=========================== Pushers ------- @@ -415,3 +415,5 @@ Rules can be enabled or disabled with a PUT operation to the 'enabled' component beneath the rule's URI with a content of 'true' or 'false':: curl -X PUT -H "Content-Type: application/json" -d 'false' "http://localhost:8008/_matrix/client/api/v1/pushrules/global/sender/%40spambot%3Amatrix.org/enabled?access_token=123456" + + diff --git a/drafts/push_pgwapi.rst b/specification/44_push_push_gw_api.rst similarity index 99% rename from drafts/push_pgwapi.rst rename to specification/44_push_push_gw_api.rst index d3da9ab2..8890ce5a 100644 --- a/drafts/push_pgwapi.rst +++ b/specification/44_push_push_gw_api.rst @@ -140,3 +140,4 @@ gateway). However, Matrix strongly recommends: * That APNS push gateways do not attempt to wait for errors from the APNS gateway before returning and instead to store failures and return 'rejected' responses next time that pushkey is used. + diff --git a/drafts/typing_notifications.rst b/specification/45_typing_notifications.rst similarity index 99% rename from drafts/typing_notifications.rst rename to specification/45_typing_notifications.rst index 048eba98..4c7d9b72 100644 --- a/drafts/typing_notifications.rst +++ b/specification/45_typing_notifications.rst @@ -55,3 +55,4 @@ originating HSes to ensure they eventually send "stop" notifications. ((This will eventually need addressing, as part of the wider typing/presence timer addition work)) +