wholetrans/docs-matrix-spec
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Add page content as raw Pandoc output

Browse source
This commit is contained in:
Will 2021-01-19 15:14:52 -08:00
parent ebc6db233b
commit c924b3246f
No known key found for this signature in database
GPG key ID: 385872BB265E8BF8
13 changed files with 6469 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

988
content/appendices.md
View file

@ -3,3 +3,991 @@ title: "Appendices"
weight: 70
type: docs
---
# Unpadded Base64
*Unpadded* Base64 refers to 'standard' Base64 encoding as defined in
[RFC 4648](https://tools.ietf.org/html/rfc4648), without "=" padding.
Specifically, where RFC 4648 requires that encoded data be padded to a
multiple of four characters using `=` characters, unpadded Base64 omits
this padding.
For reference, RFC 4648 uses the following alphabet for Base 64:
Value Encoding Value Encoding Value Encoding Value Encoding
0 A 17 R 34 i 51 z
1 B 18 S 35 j 52 0
2 C 19 T 36 k 53 1
3 D 20 U 37 l 54 2
4 E 21 V 38 m 55 3
5 F 22 W 39 n 56 4
6 G 23 X 40 o 57 5
7 H 24 Y 41 p 58 6
8 I 25 Z 42 q 59 7
9 J 26 a 43 r 60 8
10 K 27 b 44 s 61 9
11 L 28 c 45 t 62 +
12 M 29 d 46 u 63 /
13 N 30 e 47 v
14 O 31 f 48 w
15 P 32 g 49 x
16 Q 33 h 50 y
Examples of strings encoded using unpadded Base64:
UNPADDED_BASE64("") = ""
UNPADDED_BASE64("f") = "Zg"
UNPADDED_BASE64("fo") = "Zm8"
UNPADDED_BASE64("foo") = "Zm9v"
UNPADDED_BASE64("foob") = "Zm9vYg"
UNPADDED_BASE64("fooba") = "Zm9vYmE"
UNPADDED_BASE64("foobar") = "Zm9vYmFy"
When decoding Base64, implementations SHOULD accept input with or
without padding characters wherever possible, to ensure maximum
interoperability.
# Signing JSON
Various points in the Matrix specification require JSON objects to be
cryptographically signed. This requires us to encode the JSON as a
binary string. Unfortunately the same JSON can be encoded in different
ways by changing how much white space is used or by changing the order
of keys within objects.
Signing an object therefore requires it to be encoded as a sequence of
bytes using [Canonical JSON](#canonical-json), computing the signature
for that sequence and then adding the signature to the original JSON
object.
## Canonical JSON
We define the canonical JSON encoding for a value to be the shortest
UTF-8 JSON encoding with dictionary keys lexicographically sorted by
Unicode codepoint. Numbers in the JSON must be integers in the range
`[-(2**53)+1, (2**53)-1]`.
We pick UTF-8 as the encoding as it should be available to all platforms
and JSON received from the network is likely to be already encoded using
UTF-8. We sort the keys to give a consistent ordering. We force integers
to be in the range where they can be accurately represented using IEEE
double precision floating point numbers since a number of JSON libraries
represent all numbers using this representation.
Warning
Events in room versions 1, 2, 3, 4, and 5 might not be fully compliant
with these restrictions. Servers SHOULD be capable of handling JSON
which is considered invalid by these restrictions where possible.
The most notable consideration is that integers might not be in the
range specified above.
Note
Float values are not permitted by this encoding.
import json
def canonical_json(value):
return json.dumps(
value,
# Encode code-points outside of ASCII as UTF-8 rather than \u escapes
ensure_ascii=False,
# Remove unnecessary white space.
separators=(',',':'),
# Sort the keys of dictionaries.
sort_keys=True,
# Encode the resulting Unicode as UTF-8 bytes.
).encode("UTF-8")
### Grammar
Adapted from the grammar in <http://tools.ietf.org/html/rfc7159>
removing insignificant whitespace, fractions, exponents and redundant
character escapes.
value = false / null / true / object / array / number / string
false = %x66.61.6c.73.65
null = %x6e.75.6c.6c
true = %x74.72.75.65
object = %x7B [ member *( %x2C member ) ] %7D
member = string %x3A value
array = %x5B [ value *( %x2C value ) ] %5B
number = [ %x2D ] int
int = %x30 / ( %x31-39 *digit )
digit = %x30-39
string = %x22 *char %x22
char = unescaped / %x5C escaped
unescaped = %x20-21 / %x23-5B / %x5D-10FFFF
escaped = %x22 ; " quotation mark U+0022
/ %x5C ; \ reverse solidus U+005C
/ %x62 ; b backspace U+0008
/ %x66 ; f form feed U+000C
/ %x6E ; n line feed U+000A
/ %x72 ; r carriage return U+000D
/ %x74 ; t tab U+0009
/ %x75.30.30.30 (%x30-37 / %x62 / %x65-66) ; u000X
/ %x75.30.30.31 (%x30-39 / %x61-66) ; u001X
### Examples
To assist in the development of compatible implementations, the
following test values may be useful for verifying the canonical
transformation code.
Given the following JSON object:
{}
The following canonical JSON should be produced:
{}
Given the following JSON object:
{
"one": 1,
"two": "Two"
}
The following canonical JSON should be produced:
{"one":1,"two":"Two"}
Given the following JSON object:
{
"b": "2",
"a": "1"
}
The following canonical JSON should be produced:
{"a":"1","b":"2"}
Given the following JSON object:
{"b":"2","a":"1"}
The following canonical JSON should be produced:
{"a":"1","b":"2"}
Given the following JSON object:
{
"auth": {
"success": true,
"mxid": "@john.doe:example.com",
"profile": {
"display_name": "John Doe",
"three_pids": [
{
"medium": "email",
"address": "john.doe@example.org"
},
{
"medium": "msisdn",
"address": "123456789"
}
]
}
}
}
The following canonical JSON should be produced:
{"auth":{"mxid":"@john.doe:example.com","profile":{"display_name":"John Doe","three_pids":[{"address":"john.doe@example.org","medium":"email"},{"address":"123456789","medium":"msisdn"}]},"success":true}}
Given the following JSON object:
{
"a": "日本語"
}
The following canonical JSON should be produced:
{"a":"日本語"}
Given the following JSON object:
{
"本": 2,
"日": 1
}
The following canonical JSON should be produced:
{"日":1,"本":2}
Given the following JSON object:
{
"a": "\u65E5"
}
The following canonical JSON should be produced:
{"a":"日"}
Given the following JSON object:
{
"a": null
}
The following canonical JSON should be produced:
{"a":null}
## Signing Details
JSON is signed by encoding the JSON object without `signatures` or keys
grouped as `unsigned`, using the canonical encoding described above. The
JSON bytes are then signed using the signature algorithm and the
signature is encoded using [unpadded Base64](). The resulting base64
signature is added to an object under the *signing key identifier* which
is added to the `signatures` object under the name of the entity signing
it which is added back to the original JSON object along with the
`unsigned` object.
The *signing key identifier* is the concatenation of the *signing
algorithm* and a *key identifier*. The *signing algorithm* identifies
the algorithm used to sign the JSON. The currently supported value for
*signing algorithm* is `ed25519` as implemented by NACL
(<http://nacl.cr.yp.to/>). The *key identifier* is used to distinguish
between different signing keys used by the same entity.
The `unsigned` object and the `signatures` object are not covered by the
signature. Therefore intermediate entities can add unsigned data such as
timestamps and additional signatures.
{
"name": "example.org",
"signing_keys": {
"ed25519:1": "XSl0kuyvrXNj6A+7/tkrB9sxSbRi08Of5uRhxOqZtEQ"
},
"unsigned": {
"age_ts": 922834800000
},
"signatures": {
"example.org": {
"ed25519:1": "s76RUgajp8w172am0zQb/iPTHsRnb4SkrzGoeCOSFfcBY2V/1c8QfrmdXHpvnc2jK5BD1WiJIxiMW95fMjK7Bw"
}
}
}
def sign_json(json_object, signing_key, signing_name):
signatures = json_object.pop("signatures", {})
unsigned = json_object.pop("unsigned", None)
signed = signing_key.sign(encode_canonical_json(json_object))
signature_base64 = encode_base64(signed.signature)
key_id = "%s:%s" % (signing_key.alg, signing_key.version)
signatures.setdefault(signing_name, {})[key_id] = signature_base64
json_object["signatures"] = signatures
if unsigned is not None:
json_object["unsigned"] = unsigned
return json_object
## Checking for a Signature
To check if an entity has signed a JSON object an implementation does
the following:
1. Checks if the `signatures` member of the object contains an entry
with the name of the entity. If the entry is missing then the check
fails.
2. Removes any *signing key identifiers* from the entry with algorithms
it doesn't understand. If there are no *signing key identifiers*
left then the check fails.
3. Looks up *verification keys* for the remaining *signing key
identifiers* either from a local cache or by consulting a trusted
key server. If it cannot find a *verification key* then the check
fails.
4. Decodes the base64 encoded signature bytes. If base64 decoding fails
then the check fails.
5. Removes the `signatures` and `unsigned` members of the object.
6. Encodes the remainder of the JSON object using the [Canonical
JSON](#canonical-json) encoding.
7. Checks the signature bytes against the encoded object using the
*verification key*. If this fails then the check fails. Otherwise
the check succeeds.
# Identifier Grammar
Some identifiers are specific to given room versions, please refer to
the [room versions specification](index.html#room-versions) for more
information.
## Server Name
A homeserver is uniquely identified by its server name. This value is
used in a number of identifiers, as described below.
The server name represents the address at which the homeserver in
question can be reached by other homeservers. All valid server names are
included by the following grammar:
server_name = hostname [ ":" port ]
port = 1*5DIGIT
hostname = IPv4address / "[" IPv6address "]" / dns-name
IPv4address = 1*3DIGIT "." 1*3DIGIT "." 1*3DIGIT "." 1*3DIGIT
IPv6address = 2*45IPv6char
IPv6char = DIGIT / %x41-46 / %x61-66 / ":" / "."
; 0-9, A-F, a-f, :, .
dns-name = 1*255dns-char
dns-char = DIGIT / ALPHA / "-" / "."
— in other words, the server name is the hostname, followed by an
optional numeric port specifier. The hostname may be a dotted-quad IPv4
address literal, an IPv6 address literal surrounded with square
brackets, or a DNS name.
IPv4 literals must be a sequence of four decimal numbers in the range 0
to 255, separated by `.`. IPv6 literals must be as specified by
[RFC3513, section 2.2](https://tools.ietf.org/html/rfc3513#section-2.2).
DNS names for use with Matrix should follow the conventional
restrictions for internet hostnames: they should consist of a series of
labels separated by `.`, where each label consists of the alphanumeric
characters or hyphens.
Examples of valid server names are:
- `matrix.org`
- `matrix.org:8888`
- `1.2.3.4` (IPv4 literal)
- `1.2.3.4:1234` (IPv4 literal with explicit port)
- `[1234:5678::abcd]` (IPv6 literal)
- `[1234:5678::abcd]:5678` (IPv6 literal with explicit port)
Note
This grammar is based on the standard for internet host names, as
specified by [RFC1123, section
2.1](https://tools.ietf.org/html/rfc1123#page-13), with an extension for
IPv6 literals.
Server names must be treated case-sensitively: in other words,
`@user:matrix.org` is a different person from `@user:MATRIX.ORG`.
Some recommendations for a choice of server name follow:
- The length of the complete server name should not exceed 230
characters.
- Server names should not use upper-case characters.
## Common Identifier Format
The Matrix protocol uses a common format to assign unique identifiers to
a number of entities, including users, events and rooms. Each identifier
takes the form:
&string
where `&` represents a 'sigil' character; `string` is the string which
makes up the identifier.
The sigil characters are as follows:
- `@`: User ID
- `!`: Room ID
- `$`: Event ID
- `+`: Group ID
- `#`: Room alias
User IDs, group IDs, room IDs, room aliases, and sometimes event IDs
take the form:
&localpart:domain
where `domain` is the [server name](#server-name) of the homeserver
which allocated the identifier, and `localpart` is an identifier
allocated by that homeserver.
The precise grammar defining the allowable format of an identifier
depends on the type of identifier. For example, event IDs can sometimes
be represented with a `domain` component under some conditions - see the
[Event IDs](#room-ids-and-event-ids) section below for more information.
### User Identifiers
Users within Matrix are uniquely identified by their Matrix user ID. The
user ID is namespaced to the homeserver which allocated the account and
has the form:
@localpart:domain
The `localpart` of a user ID is an opaque identifier for that user. It
MUST NOT be empty, and MUST contain only the characters `a-z`, `0-9`,
`.`, `_`, `=`, `-`, and `/`.
The `domain` of a user ID is the [server name](#server-name) of the
homeserver which allocated the account.
The length of a user ID, including the `@` sigil and the domain, MUST
NOT exceed 255 characters.
The complete grammar for a legal user ID is:
user_id = "@" user_id_localpart ":" server_name
user_id_localpart = 1*user_id_char
user_id_char = DIGIT
/ %x61-7A ; a-z
/ "-" / "." / "=" / "_" / "/"
Rationale
A number of factors were considered when defining the allowable
characters for a user ID.
Firstly, we chose to exclude characters outside the basic US-ASCII
character set. User IDs are primarily intended for use as an identifier
at the protocol level, and their use as a human-readable handle is of
secondary benefit. Furthermore, they are useful as a last-resort
differentiator between users with similar display names. Allowing the
full Unicode character set would make very difficult for a human to
distinguish two similar user IDs. The limited character set used has the
advantage that even a user unfamiliar with the Latin alphabet should be
able to distinguish similar user IDs manually, if somewhat laboriously.
We chose to disallow upper-case characters because we do not consider it
valid to have two user IDs which differ only in case: indeed it should
be possible to reach `@user:matrix.org` as `@USER:matrix.org`. However,
user IDs are necessarily used in a number of situations which are
inherently case-sensitive (notably in the `state_key` of `m.room.member`
events). Forbidding upper-case characters (and requiring homeservers to
downcase usernames when creating user IDs for new users) is a relatively
simple way to ensure that `@USER:matrix.org` cannot refer to a different
user to `@user:matrix.org`.
Finally, we decided to restrict the allowable punctuation to a very
basic set to reduce the possibility of conflicts with special characters
in various situations. For example, "\*" is used as a wildcard in some
APIs (notably the filter API), so it cannot be a legal user ID
character.
The length restriction is derived from the limit on the length of the
`sender` key on events; since the user ID appears in every event sent by
the user, it is limited to ensure that the user ID does not dominate
over the actual content of the events.
Matrix user IDs are sometimes informally referred to as MXIDs.
#### Historical User IDs
Older versions of this specification were more tolerant of the
characters permitted in user ID localparts. There are currently active
users whose user IDs do not conform to the permitted character set, and
a number of rooms whose history includes events with a `sender` which
does not conform. In order to handle these rooms successfully, clients
and servers MUST accept user IDs with localparts from the expanded
character set:
extended_user_id_char = %x21-39 / %x3B-7E ; all ASCII printing chars except :
#### Mapping from other character sets
In certain circumstances it will be desirable to map from a wider
character set onto the limited character set allowed in a user ID
localpart. Examples include a homeserver creating a user ID for a new
user based on the username passed to `/register`, or a bridge mapping
user ids from another protocol.
Implementations are free to do this mapping however they choose. Since
the user ID is opaque except to the implementation which created it, the
only requirement is that the implementation can perform the mapping
consistently. However, we suggest the following algorithm:
1. Encode character strings as UTF-8.
2. Convert the bytes `A-Z` to lower-case.
- In the case where a bridge must be able to distinguish two
different users with ids which differ only by case, escape
upper-case characters by prefixing with `_` before downcasing.
For example, `A` becomes `_a`. Escape a real `_` with a second
`_`.
3. Encode any remaining bytes outside the allowed character set, as
well as `=`, as their hexadecimal value, prefixed with `=`. For
example, `#` becomes `=23`; `á` becomes `=c3=a1`.
Rationale
The suggested mapping is an attempt to preserve human-readability of
simple ASCII identifiers (unlike, for example, base-32), whilst still
allowing representation of *any* character (unlike punycode, which
provides no way to encode ASCII punctuation).
### Room IDs and Event IDs
A room has exactly one room ID. A room ID has the format:
!opaque_id:domain
An event has exactly one event ID. The format of an event ID depends
upon the [room version specification](index.html#room-versions).
The `domain` of a room ID is the [server name](#server-name) of the
homeserver which created the room/event. The domain is used only for
namespacing to avoid the risk of clashes of identifiers between
different homeservers. There is no implication that the room or event in
question is still available at the corresponding homeserver.
Event IDs and Room IDs are case-sensitive. They are not meant to be
human-readable. They are intended to be treated as fully opaque strings
by clients.
### Group Identifiers
Groups within Matrix are uniquely identified by their group ID. The
group ID is namespaced to the group server which hosts this group and
has the form:
+localpart:domain
The `localpart` of a group ID is an opaque identifier for that group. It
MUST NOT be empty, and MUST contain only the characters `a-z`, `0-9`,
`.`, `_`, `=`, `-`, and `/`.
The `domain` of a group ID is the [server name](#server-name) of the
group server which hosts this group.
The length of a group ID, including the `+` sigil and the domain, MUST
NOT exceed 255 characters.
The complete grammar for a legal group ID is:
group_id = "+" group_id_localpart ":" server_name
group_id_localpart = 1*group_id_char
group_id_char = DIGIT
/ %x61-7A ; a-z
/ "-" / "." / "=" / "_" / "/"
### Room Aliases
A room may have zero or more aliases. A room alias has the format:
#room_alias:domain
The `domain` of a room alias is the [server name](#server-name) of the
homeserver which created the alias. Other servers may contact this
homeserver to look up the alias.
Room aliases MUST NOT exceed 255 bytes (including the `#` sigil and the
domain).
### matrix.to navigation
Note
This namespacing is in place pending a `matrix://` (or similar) URI
scheme. This is **not** meant to be interpreted as an available web
service - see below for more details.
Rooms, users, aliases, and groups may be represented as a "matrix.to"
URI. This URI can be used to reference particular objects in a given
context, such as mentioning a user in a message or linking someone to a
particular point in the room's history (a permalink).
A matrix.to URI has the following format, based upon the specification
defined in RFC 3986:
> <https://matrix.to/#/>&lt;identifier&gt;/&lt;extra
> parameter&gt;?&lt;additional arguments&gt;
The identifier may be a room ID, room alias, user ID, or group ID. The
extra parameter is only used in the case of permalinks where an event ID
is referenced. The matrix.to URI, when referenced, must always start
with `https://matrix.to/#/` followed by the identifier.
The `<additional arguments>` and the preceding question mark are
optional and only apply in certain circumstances, documented below.
Clients should not rely on matrix.to URIs falling back to a web server
if accessed and instead should perform some sort of action within the
client. For example, if the user were to click on a matrix.to URI for a
room alias, the client may open a view for the user to participate in
the room.
The components of the matrix.to URI (`<identifier>` and
`<extra parameter>`) are to be percent-encoded as per RFC 3986.
Examples of matrix.to URIs are:
- Room alias: `https://matrix.to/#/%23somewhere%3Aexample.org`
- Room: `https://matrix.to/#/!somewhere%3Aexample.org`
- Permalink by room:
`https://matrix.to/#/!somewhere%3Aexample.org/%24event%3Aexample.org`
- Permalink by room alias:
`https://matrix.to/#/%23somewhere:example.org/%24event%3Aexample.org`
- User: `https://matrix.to/#/%40alice%3Aexample.org`
- Group: `https://matrix.to/#/%2Bexample%3Aexample.org`
Note
Historically, clients have not produced URIs which are fully encoded.
Clients should try to interpret these cases to the best of their
ability. For example, an unencoded room alias should still work within
the client if possible.
Note
Clients should be aware that decoding a matrix.to URI may result in
extra slashes appearing due to some [room
versions](index.html#room-versions). These slashes should normally be
encoded when producing matrix.to URIs, however.
#### Routing
Room IDs are not routable on their own as there is no reliable domain to
send requests to. This is partially mitigated with the addition of a
`via` argument on a matrix.to URI, however the problem of routability is
still present. Clients should do their best to route Room IDs to where
they need to go, however they should also be aware of [issue
\#1579](https://github.com/matrix-org/matrix-doc/issues/1579).
A room (or room permalink) which isn't using a room alias should supply
at least one server using `via` in the `<additional arguments>`, like
so:
`https://matrix.to/!somewhere%3Aexample.org?via=example.org&via=alt.example.org`.
The parameter can be supplied multiple times to specify multiple servers
to try.
The values of `via` are intended to be passed along as the `server_name`
parameters on the Client Server `/join` API.
When generating room links and permalinks, the application should pick
servers which have a high probability of being in the room in the
distant future. How these servers are picked is left as an
implementation detail, however the current recommendation is to pick 3
unique servers based on the following criteria:
- The first server should be the server of the highest power level
user in the room, provided they are at least power level 50. If no
user meets this criterion, pick the most popular server in the room
(most joined users). The rationale for not picking users with power
levels under 50 is that they are unlikely to be around into the
distant future while higher ranking users (and therefore servers)
are less likely to give up their power and move somewhere else. Most
rooms in the public federation have a power level 100 user and have
not deviated from the default structure where power level 50 users
have moderator-style privileges.
- The second server should be the next highest server by population,
or the first highest by population if the first server was based on
a user's power level. The rationale for picking popular servers is
that the server is unlikely to be removed as the room naturally
grows in membership due to that server joining users. The server
could be refused participation in the future due to server ACLs or
similar, however the chance of that happening to a server which is
organically joining the room is unlikely.
- The third server should be the next highest server by population.
- Servers which are blocked due to server ACLs should never be chosen.
- Servers which are IP addresses should never be chosen. Servers which
use a domain name are less likely to be unroutable in the future
whereas IP addresses cannot be pointed to a different location and
therefore higher risk options.
- All 3 servers should be unique from each other. If the room does not
have enough users to supply 3 servers, the application should only
specify the servers it can. For example, a room with only 2 users in
it would result in maximum 2 `via` parameters.
# 3PID Types
Third Party Identifiers (3PIDs) represent identifiers on other
namespaces that might be associated with a particular person. They
comprise a tuple of `medium` which is a string that identifies the
namespace in which the identifier exists, and an `address`: a string
representing the identifier in that namespace. This must be a canonical
form of the identifier, *i.e.* if multiple strings could represent the
same identifier, only one of these strings must be used in a 3PID
address, in a well-defined manner.
For example, for e-mail, the `medium` is 'email' and the `address` would
be the email address, *e.g.* the string `bob@example.com`. Since domain
resolution is case-insensitive, the email address `bob@Example.com` is
also has the 3PID address of `bob@example.com` (without the capital 'e')
rather than `bob@Example.com`.
The namespaces defined by this specification are listed below. More
namespaces may be defined in future versions of this specification.
## E-Mail
Medium: `email`
Represents E-Mail addresses. The `address` is the raw email address in
`user@domain` form with the domain in lowercase. It must not contain
other text such as real name, angle brackets or a mailto: prefix.
## PSTN Phone numbers
Medium: `msisdn`
Represents telephone numbers on the public switched telephone network.
The `address` is the telephone number represented as a MSISDN (Mobile
Station International Subscriber Directory Number) as defined by the
E.164 numbering plan. Note that MSISDNs do not include a leading '+'.
# Security Threat Model
## Denial of Service
The attacker could attempt to prevent delivery of messages to or from
the victim in order to:
- Disrupt service or marketing campaign of a commercial competitor.
- Censor a discussion or censor a participant in a discussion.
- Perform general vandalism.
### Threat: Resource Exhaustion
An attacker could cause the victim's server to exhaust a particular
resource (e.g. open TCP connections, CPU, memory, disk storage)
### Threat: Unrecoverable Consistency Violations
An attacker could send messages which created an unrecoverable
"split-brain" state in the cluster such that the victim's servers could
no longer derive a consistent view of the chatroom state.
### Threat: Bad History
An attacker could convince the victim to accept invalid messages which
the victim would then include in their view of the chatroom history.
Other servers in the chatroom would reject the invalid messages and
potentially reject the victims messages as well since they depended on
the invalid messages.
### Threat: Block Network Traffic
An attacker could try to firewall traffic between the victim's server
and some or all of the other servers in the chatroom.
### Threat: High Volume of Messages
An attacker could send large volumes of messages to a chatroom with the
victim making the chatroom unusable.
### Threat: Banning users without necessary authorisation
An attacker could attempt to ban a user from a chatroom without the
necessary authorisation.
## Spoofing
An attacker could try to send a message claiming to be from the victim
without the victim having sent the message in order to:
- Impersonate the victim while performing illicit activity.
- Obtain privileges of the victim.
### Threat: Altering Message Contents
An attacker could try to alter the contents of an existing message from
the victim.
### Threat: Fake Message "origin" Field
An attacker could try to send a new message purporting to be from the
victim with a phony "origin" field.
## Spamming
The attacker could try to send a high volume of solicited or unsolicited
messages to the victim in order to:
- Find victims for scams.
- Market unwanted products.
### Threat: Unsolicited Messages
An attacker could try to send messages to victims who do not wish to
receive them.
### Threat: Abusive Messages
An attacker could send abusive or threatening messages to the victim
## Spying
The attacker could try to access message contents or metadata for
messages sent by the victim or to the victim that were not intended to
reach the attacker in order to:
- Gain sensitive personal or commercial information.
- Impersonate the victim using credentials contained in the messages.
(e.g. password reset messages)
- Discover who the victim was talking to and when.
### Threat: Disclosure during Transmission
An attacker could try to expose the message contents or metadata during
transmission between the servers.
### Threat: Disclosure to Servers Outside Chatroom
An attacker could try to convince servers within a chatroom to send
messages to a server it controls that was not authorised to be within
the chatroom.
### Threat: Disclosure to Servers Within Chatroom
An attacker could take control of a server within a chatroom to expose
message contents or metadata for messages in that room.
# Cryptographic Test Vectors
To assist in the development of compatible implementations, the
following test values may be useful for verifying the cryptographic
event signing code.
## Signing Key
The following test vectors all use the 32-byte value given by the
following Base64-encoded string as the seed for generating the `ed25519`
signing key:
SIGNING_KEY_SEED = decode_base64(
"YJDBA9Xnr2sVqXD9Vj7XVUnmFZcZrlw8Md7kMW+3XA1"
)
In each case, the server name and key ID are as follows:
SERVER_NAME = "domain"
KEY_ID = "ed25519:1"
## JSON Signing
Given an empty JSON object:
{}
The JSON signing algorithm should emit the following signed data:
{
"signatures": {
"domain": {
"ed25519:1": "K8280/U9SSy9IVtjBuVeLr+HpOB4BQFWbg+UZaADMtTdGYI7Geitb76LTrr5QV/7Xg4ahLwYGYZzuHGZKM5ZAQ"
}
}
}
Given the following JSON object with data values in it:
{
"one": 1,
"two": "Two"
}
The JSON signing algorithm should emit the following signed JSON:
{
"one": 1,
"signatures": {
"domain": {
"ed25519:1": "KqmLSbO39/Bzb0QIYE82zqLwsA+PDzYIpIRA2sRQ4sL53+sN6/fpNSoqE7BP7vBZhG6kYdD13EIMJpvhJI+6Bw"
}
},
"two": "Two"
}
## Event Signing
Given the following minimally-sized event:
{
"room_id": "!x:domain",
"sender": "@a:domain",
"origin": "domain",
"origin_server_ts": 1000000,
"signatures": {},
"hashes": {},
"type": "X",
"content": {},
"prev_events": [],
"auth_events": [],
"depth": 3,
"unsigned": {
"age_ts": 1000000
}
}
The event signing algorithm should emit the following signed event:
{
"auth_events": [],
"content": {},
"depth": 3,
"hashes": {
"sha256": "5jM4wQpv6lnBo7CLIghJuHdW+s2CMBJPUOGOC89ncos"
},
"origin": "domain",
"origin_server_ts": 1000000,
"prev_events": [],
"room_id": "!x:domain",
"sender": "@a:domain",
"signatures": {
"domain": {
"ed25519:1": "KxwGjPSDEtvnFgU00fwFz+l6d2pJM6XBIaMEn81SXPTRl16AqLAYqfIReFGZlHi5KLjAWbOoMszkwsQma+lYAg"
}
},
"type": "X",
"unsigned": {
"age_ts": 1000000
}
}
Given the following event containing redactable content:
{
"content": {
"body": "Here is the message content"
},
"event_id": "$0:domain",
"origin": "domain",
"origin_server_ts": 1000000,
"type": "m.room.message",
"room_id": "!r:domain",
"sender": "@u:domain",
"signatures": {},
"unsigned": {
"age_ts": 1000000
}
}
The event signing algorithm should emit the following signed event:
{
"content": {
"body": "Here is the message content"
},
"event_id": "$0:domain",
"hashes": {
"sha256": "onLKD1bGljeBWQhWZ1kaP9SorVmRQNdN5aM2JYU2n/g"
},
"origin": "domain",
"origin_server_ts": 1000000,
"type": "m.room.message",
"room_id": "!r:domain",
"sender": "@u:domain",
"signatures": {
"domain": {
"ed25519:1": "Wm+VzmOUOz08Ds+0NTWb1d4CZrVsJSikkeRxh6aCcUwu6pNC78FunoD7KNWzqFn241eYHYMGCA5McEiVPdhzBA"
}
},
"unsigned": {
"age_ts": 1000000
}
}

470
content/application-service-api.md
View file

@ -3,3 +3,473 @@ title: "Application Service API"
weight: 30
type: docs
---
# Application Service API
{{unstable\_warning\_block\_APPSERVICE\_RELEASE\_LABEL}}
The Matrix client-server API and server-server APIs provide the means to
implement a consistent self-contained federated messaging fabric.
However, they provide limited means of implementing custom server-side
behaviour in Matrix (e.g. gateways, filters, extensible hooks etc). The
Application Service API (AS API) defines a standard API to allow such
extensible functionality to be implemented irrespective of the
underlying homeserver implementation.
Table of Contents
## Changelog
**Version: %APPSERVICE\_RELEASE\_LABEL%**
{{application\_service\_changelog}}
This version of the specification is generated from
[matrix-doc](https://github.com/matrix-org/matrix-doc) as of Git commit
[{{git\_version}}](https://github.com/matrix-org/matrix-doc/tree/%7B%7Bgit_rev%7D%7D).
For the full historical changelog, see
<https://github.com/matrix-org/matrix-doc/blob/master/changelogs/application_service.rst>
### Other versions of this specification
The following other versions are also available, in reverse
chronological order:
- [HEAD](https://matrix.org/docs/spec/application_service/unstable.html):
Includes all changes since the latest versioned release.
- [r0.1.1](https://matrix.org/docs/spec/application_service/r0.1.1.html)
- [r0.1.0](https://matrix.org/docs/spec/application_service/r0.1.0.html)
## Application Services
Application services are passive and can only observe events from
homeserver. They can inject events into rooms they are participating in.
They cannot prevent events from being sent, nor can they modify the
content of the event being sent. In order to observe events from a
homeserver, the homeserver needs to be configured to pass certain types
of traffic to the application service. This is achieved by manually
configuring the homeserver with information about the application
service.
### Registration
Note
Previously, application services could register with a homeserver via
HTTP APIs. This was removed as it was seen as a security risk. A
compromised application service could re-register for a global `*` regex
and sniff *all* traffic on the homeserver. To protect against this,
application services now have to register via configuration files which
are linked to the homeserver configuration file. The addition of
configuration files allows homeserver admins to sanity check the
registration for suspicious regex strings.
Application services register "namespaces" of user IDs, room aliases and
room IDs. These namespaces are represented as regular expressions. An
application service is said to be "interested" in a given event if one
of the IDs in the event match the regular expression provided by the
application service, such as the room having an alias or ID in the
relevant namespaces. Similarly, the application service is said to be
interested in a given event if one of the application service's
namespaced users is the target of the event, or is a joined member of
the room where the event occurred.
An application service can also state whether they should be the only
ones who can manage a specified namespace. This is referred to as an
"exclusive" namespace. An exclusive namespace prevents humans and other
application services from creating/deleting entities in that namespace.
Typically, exclusive namespaces are used when the rooms represent real
rooms on another service (e.g. IRC). Non-exclusive namespaces are used
when the application service is merely augmenting the room itself (e.g.
providing logging or searching facilities). Namespaces are represented
by POSIX extended regular expressions and look like:
users:
- exclusive: true
regex: "@_irc_bridge_.*"
Application services may define the following namespaces (with none
being explicitly required):
<table>
<colgroup>
<col style="width: 24%" />
<col style="width: 75%" />
</colgroup>
<thead>
<tr class="header">
<th>Name</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>users</td>
<td>Events which are sent from certain users.</td>
</tr>
<tr class="even">
<td>aliases</td>
<td>Events which are sent in rooms with certain room aliases.</td>
</tr>
<tr class="odd">
<td>rooms</td>
<td>Events which are sent in rooms with certain room IDs.</td>
</tr>
</tbody>
</table>
Each individual namespace MUST declare the following fields:
<table>
<colgroup>
<col style="width: 12%" />
<col style="width: 87%" />
</colgroup>
<thead>
<tr class="header">
<th>Name</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>exclusive</td>
<td><strong>Required</strong> A true or false value stating whether this application service has exclusive access to events within this namespace.</td>
</tr>
<tr class="even">
<td>regex</td>
<td><strong>Required</strong> A regular expression defining which values this namespace includes.</td>
</tr>
</tbody>
</table>
Exclusive user and alias namespaces should begin with an underscore
after the sigil to avoid collisions with other users on the homeserver.
Application services should additionally attempt to identify the service
they represent in the reserved namespace. For example, `@_irc_.*` would
be a good namespace to register for an application service which deals
with IRC.
The registration is represented by a series of key-value pairs, which
this specification will present as YAML. See below for the possible
options along with their explanation:
<table>
<colgroup>
<col style="width: 11%" />
<col style="width: 88%" />
</colgroup>
<thead>
<tr class="header">
<th>Name</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>id</td>
<td><strong>Required.</strong> A unique, user-defined ID of the application service which will never change.</td>
</tr>
<tr class="even">
<td>url</td>
<td><strong>Required.</strong> The URL for the application service. May include a path after the domain name. Optionally set to <code>null</code> if no traffic is required.</td>
</tr>
<tr class="odd">
<td>as_token</td>
<td><strong>Required.</strong> A unique token for application services to use to authenticate requests to Homeservers.</td>
</tr>
<tr class="even">
<td>hs_token</td>
<td><strong>Required.</strong> A unique token for Homeservers to use to authenticate requests to application services.</td>
</tr>
<tr class="odd">
<td>sender_localpart</td>
<td><strong>Required.</strong> The localpart of the user associated with the application service.</td>
</tr>
<tr class="even">
<td>namespaces</td>
<td><strong>Required.</strong> A list of <code>users</code>, <code>aliases</code> and <code>rooms</code> namespaces that the application service controls.</td>
</tr>
<tr class="odd">
<td>rate_limited</td>
<td>Whether requests from masqueraded users are rate-limited. The sender is excluded.</td>
</tr>
<tr class="even">
<td>protocols</td>
<td>The external protocols which the application service provides (e.g. IRC).</td>
</tr>
</tbody>
</table>
An example registration file for an IRC-bridging application service is
below:
id: "IRC Bridge"
url: "http://127.0.0.1:1234"
as_token: "30c05ae90a248a4188e620216fa72e349803310ec83e2a77b34fe90be6081f46"
hs_token: "312df522183efd404ec1cd22d2ffa4bbc76a8c1ccf541dd692eef281356bb74e"
sender_localpart: "_irc_bot" # Will result in @_irc_bot:example.org
namespaces:
users:
- exclusive: true
regex: "@_irc_bridge_.*"
aliases:
- exclusive: false
regex: "#_irc_bridge_.*"
rooms: []
Warning
If the homeserver in question has multiple application services, each
`as_token` and `id` MUST be unique per application service as these are
used to identify the application service. The homeserver MUST enforce
this.
### Homeserver -&gt; Application Service API
#### Authorization
Homeservers MUST include a query parameter named `access_token`
containing the `hs_token` from the application service's registration
when making requests to the application service. Application services
MUST verify the provided `access_token` matches their known `hs_token`,
failing the request with an `M_FORBIDDEN` error if it does not match.
#### Legacy routes
Previous drafts of the application service specification had a mix of
endpoints that have been used in the wild for a significant amount of
time. The application service specification now defines a version on all
endpoints to be more compatible with the rest of the Matrix
specification and the future.
Homeservers should attempt to use the specified endpoints first when
communicating with application services. However, if the application
service receives an HTTP status code that does not indicate success
(i.e.: 404, 500, 501, etc) then the homeserver should fall back to the
older endpoints for the application service.
The older endpoints have the exact same request body and response
format, they just belong at a different path. The equivalent path for
each is as follows:
- `/_matrix/app/v1/transactions/{txnId}` should fall back to
`/transactions/{txnId}`
- `/_matrix/app/v1/users/{userId}` should fall back to
`/users/{userId}`
- `/_matrix/app/v1/rooms/{roomAlias}` should fall back to
`/rooms/{roomAlias}`
- `/_matrix/app/v1/thirdparty/protocol/{protocol}` should fall back to
`/_matrix/app/unstable/thirdparty/protocol/{protocol}`
- `/_matrix/app/v1/thirdparty/user/{user}` should fall back to
`/_matrix/app/unstable/thirdparty/user/{user}`
- `/_matrix/app/v1/thirdparty/location/{location}` should fall back to
`/_matrix/app/unstable/thirdparty/location/{location}`
- `/_matrix/app/v1/thirdparty/user` should fall back to
`/_matrix/app/unstable/thirdparty/user`
- `/_matrix/app/v1/thirdparty/location` should fall back to
`/_matrix/app/unstable/thirdparty/location`
Homeservers should periodically try again for the newer endpoints
because the application service may have been updated.
#### Pushing events
The application service API provides a transaction API for sending a
list of events. Each list of events includes a transaction ID, which
works as follows:
Typical
HS ---> AS : Homeserver sends events with transaction ID T.
<--- : Application Service sends back 200 OK.
AS ACK Lost
HS ---> AS : Homeserver sends events with transaction ID T.
<-/- : AS 200 OK is lost.
HS ---> AS : Homeserver retries with the same transaction ID of T.
<--- : Application Service sends back 200 OK. If the AS had processed these
events already, it can NO-OP this request (and it knows if it is the
same events based on the transaction ID).
The events sent to the application service should be linearised, as if
they were from the event stream. The homeserver MUST maintain a queue of
transactions to send to the application service. If the application
service cannot be reached, the homeserver SHOULD backoff exponentially
until the application service is reachable again. As application
services cannot *modify* the events in any way, these requests can be
made without blocking other aspects of the homeserver. Homeservers MUST
NOT alter (e.g. add more) events they were going to send within that
transaction ID on retries, as the application service may have already
processed the events.
{{transactions\_as\_http\_api}}
#### Querying
The application service API includes two querying APIs: for room aliases
and for user IDs. The application service SHOULD create the queried
entity if it desires. During this process, the application service is
blocking the homeserver until the entity is created and configured. If
the homeserver does not receive a response to this request, the
homeserver should retry several times before timing out. This should
result in an HTTP status 408 "Request Timeout" on the client which
initiated this request (e.g. to join a room alias).
Rationale
Blocking the homeserver and expecting the application service to create
the entity using the client-server API is simpler and more flexible than
alternative methods such as returning an initial sync style JSON blob
and get the HS to provision the room/user. This also meant that there
didn't need to be a "backchannel" to inform the application service
about information about the entity such as room ID to room alias
mappings.
{{query\_user\_as\_http\_api}}
{{query\_room\_as\_http\_api}}
#### Third party networks
Application services may declare which protocols they support via their
registration configuration for the homeserver. These networks are
generally for third party services such as IRC that the application
service is managing. Application services may populate a Matrix room
directory for their registered protocols, as defined in the
Client-Server API Extensions.
Each protocol may have several "locations" (also known as "third party
locations" or "3PLs"). A location within a protocol is a place in the
third party network, such as an IRC channel. Users of the third party
network may also be represented by the application service.
Locations and users can be searched by fields defined by the application
service, such as by display name or other attribute. When clients
request the homeserver to search in a particular "network" (protocol),
the search fields will be passed along to the application service for
filtering.
{{protocols\_as\_http\_api}}
### Client-Server API Extensions
Application services can use a more powerful version of the
client-server API by identifying itself as an application service to the
homeserver.
Endpoints defined in this section MUST be supported by homeservers in
the client-server API as accessible only by application services.
#### Identity assertion
The client-server API infers the user ID from the `access_token`
provided in every request. To avoid the application service from having
to keep track of each user's access token, the application service
should identify itself to the Client-Server API by providing its
`as_token` for the `access_token` alongside the user the application
service would like to masquerade as.
Inputs:
- Application service token (`as_token`)
- User ID in the AS namespace to act as.
Notes:
- This applies to all aspects of the Client-Server API, except for
Account Management.
- The `as_token` is inserted into `access_token` which is usually
where the client token is, such as via the query string or
`Authorization` header. This is done on purpose to allow application
services to reuse client SDKs.
- The `access_token` should be supplied through the `Authorization`
header where possible to prevent the token appearing in HTTP request
logs by accident.
The application service may specify the virtual user to act as through
use of a `user_id` query string parameter on the request. The user
specified in the query string must be covered by one of the application
service's `user` namespaces. If the parameter is missing, the homeserver
is to assume the application service intends to act as the user implied
by the `sender_localpart` property of the registration.
An example request would be:
GET /_matrix/client/%CLIENT_MAJOR_VERSION%/account/whoami?user_id=@_irc_user:example.org
Authorization: Bearer YourApplicationServiceTokenHere
#### Timestamp massaging
Previous drafts of the Application Service API permitted application
services to alter the timestamp of their sent events by providing a `ts`
query parameter when sending an event. This API has been excluded from
the first release due to design concerns, however some servers may still
support the feature. Please visit [issue
\#1585](https://github.com/matrix-org/matrix-doc/issues/1585) for more
information.
#### Server admin style permissions
The homeserver needs to give the application service *full control* over
its namespace, both for users and for room aliases. This means that the
AS should be able to create/edit/delete any room alias in its namespace,
as well as create/delete any user in its namespace. No additional API
changes need to be made in order for control of room aliases to be
granted to the AS. Creation of users needs API changes in order to:
- Work around captchas.
- Have a 'passwordless' user.
This involves bypassing the registration flows entirely. This is
achieved by including the `as_token` on a `/register` request, along
with a login type of `m.login.application_service` to set the desired
user ID without a password.
POST /_matrix/client/%CLIENT_MAJOR_VERSION%/register
Authorization: Bearer YourApplicationServiceTokenHere
Content:
{
type: "m.login.application_service",
username: "_irc_example"
}
Application services which attempt to create users or aliases *outside*
of their defined namespaces will receive an error code `M_EXCLUSIVE`.
Similarly, normal users who attempt to create users or aliases *inside*
an application service-defined namespace will receive the same
`M_EXCLUSIVE` error code, but only if the application service has
defined the namespace as `exclusive`.
#### Using `/sync` and `/events`
Application services wishing to use `/sync` or `/events` from the
Client-Server API MUST do so with a virtual user (provide a `user_id`
via the query string). It is expected that the application service use
the transactions pushed to it to handle events rather than syncing with
the user implied by `sender_localpart`.
#### Application service room directories
Application services can maintain their own room directories for their
defined third party protocols. These room directories may be accessed by
clients through additional parameters on the `/publicRooms`
client-server endpoint.
{{appservice\_room\_directory\_cs\_http\_api}}
### Referencing messages from a third party network
Application services should include an `external_url` in the `content`
of events it emits to indicate where the message came from. This
typically applies to application services that bridge other networks
into Matrix, such as IRC, where an HTTP URL may be available to
reference.
Clients should provide users with a way to access the `external_url` if
it is present. Clients should additionally ensure the URL has a scheme
of `https` or `http` before making use of it.
The presence of an `external_url` on an event does not necessarily mean
the event was sent from an application service. Clients should be wary
of the URL contained within, as it may not be a legitimate reference to
the event's source.

1941
content/client-server-api/_index.md
View file

File diff suppressed because it is too large Load diff

467
content/identity-service-api.md
View file

@ -3,3 +3,470 @@ title: "Identity Service API"
weight: 40
type: docs
---
# Identity Service API
{{unstable\_warning\_block\_IDENTITY\_RELEASE\_LABEL}}
The Matrix client-server and server-server APIs are largely expressed in
Matrix user identifiers. From time to time, it is useful to refer to
users by other ("third-party") identifiers, or "3PID"s, e.g. their email
address or phone number. This Identity Service Specification describes
how mappings between third-party identifiers and Matrix user identifiers
can be established, validated, and used. This description technically
may apply to any 3PID, but in practice has only been applied
specifically to email addresses and phone numbers.
Table of Contents
## Changelog
**Version: %IDENTITY\_RELEASE\_LABEL%**
{{identity\_service\_changelog}}
This version of the specification is generated from
[matrix-doc](https://github.com/matrix-org/matrix-doc) as of Git commit
[{{git\_version}}](https://github.com/matrix-org/matrix-doc/tree/%7B%7Bgit_rev%7D%7D).
For the full historical changelog, see
<https://github.com/matrix-org/matrix-doc/blob/master/changelogs/identity_service.rst>
### Other versions of this specification
The following other versions are also available, in reverse
chronological order:
- [HEAD](https://matrix.org/docs/spec/identity_service/unstable.html):
Includes all changes since the latest versioned release.
- [r0.3.0](https://matrix.org/docs/spec/identity_service/r0.3.0.html)
- [r0.2.1](https://matrix.org/docs/spec/identity_service/r0.2.1.html)
- [r0.2.0](https://matrix.org/docs/spec/identity_service/r0.2.0.html)
- [r0.1.0](https://matrix.org/docs/spec/identity_service/r0.1.0.html)
## General principles
The purpose of an identity server is to validate, store, and answer
questions about the identities of users. In particular, it stores
associations of the form "identifier X represents the same user as
identifier Y", where identities may exist on different systems (such as
email addresses, phone numbers, Matrix user IDs, etc).
The identity server has some private-public keypairs. When asked about
an association, it will sign details of the association with its private
key. Clients may validate the assertions about associations by verifying
the signature with the public key of the identity server.
In general, identity servers are treated as reliable oracles. They do
not necessarily provide evidence that they have validated associations,
but claim to have done so. Establishing the trustworthiness of an
individual identity server is left as an exercise for the client.
3PID types are described in [3PID Types](../appendices.html#pid-types)
Appendix.
## API standards
The mandatory baseline for identity server communication in Matrix is
exchanging JSON objects over HTTP APIs. HTTPS is required for
communication, and all API calls use a Content-Type of
`application/json`. In addition, strings MUST be encoded as UTF-8.
Any errors which occur at the Matrix API level MUST return a "standard
error response". This is a JSON object which looks like:
{
"errcode": "<error code>",
"error": "<error message>"
}
The `error` string will be a human-readable error message, usually a
sentence explaining what went wrong. The `errcode` string will be a
unique string which can be used to handle an error message e.g.
`M_FORBIDDEN`. There may be additional keys depending on the error, but
the keys `error` and `errcode` MUST always be present.
Some standard error codes are below:
`M_NOT_FOUND`
The resource requested could not be located.
`M_MISSING_PARAMS`
The request was missing one or more parameters.
`M_INVALID_PARAM`
The request contained one or more invalid parameters.
`M_SESSION_NOT_VALIDATED`
The session has not been validated.
`M_NO_VALID_SESSION`
A session could not be located for the given parameters.
`M_SESSION_EXPIRED`
The session has expired and must be renewed.
`M_INVALID_EMAIL`
The email address provided was not valid.
`M_EMAIL_SEND_ERROR`
There was an error sending an email. Typically seen when attempting to
verify ownership of a given email address.
`M_INVALID_ADDRESS`
The provided third party address was not valid.
`M_SEND_ERROR`
There was an error sending a notification. Typically seen when
attempting to verify ownership of a given third party address.
`M_UNRECOGNIZED`
The request contained an unrecognised value, such as an unknown token or
medium.
`M_THREEPID_IN_USE`
The third party identifier is already in use by another user. Typically
this error will have an additional `mxid` property to indicate who owns
the third party identifier.
`M_UNKNOWN`
An unknown error has occurred.
## Privacy
Identity is a privacy-sensitive issue. While the identity server exists
to provide identity information, access should be restricted to avoid
leaking potentially sensitive data. In particular, being able to
construct large-scale connections between identities should be avoided.
To this end, in general APIs should allow a 3PID to be mapped to a
Matrix user identity, but not in the other direction (i.e. one should
not be able to get all 3PIDs associated with a Matrix user ID, or get
all 3PIDs associated with a 3PID).
## Version 1 API deprecation
As described on each of the version 1 endpoints, the v1 API is
deprecated in favour of the v2 API described here. The major difference,
with the exception of a few isolated cases, is that the v2 API requires
authentication to ensure the user has given permission for the identity
server to operate on their data.
The v1 API is planned to be removed from the specification in a future
version.
Clients SHOULD attempt the v2 endpoints first, and if they receive a
`404`, `400`, or similar error they should try the v1 endpoint or fail
the operation. Clients are strongly encouraged to warn the user of the
risks in using the v1 API, if they are planning on using it.
## Web browser clients
It is realistic to expect that some clients will be written to be run
within a web browser or similar environment. In these cases, the
identity server should respond to pre-flight requests and supply
Cross-Origin Resource Sharing (CORS) headers on all requests.
When a client approaches the server with a pre-flight (OPTIONS) request,
the server should respond with the CORS headers for that route. The
recommended CORS headers to be returned by servers on all requests are:
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, Authorization
## Authentication
Most `v2` endpoints in the Identity Service API require authentication
in order to ensure that the requesting user has accepted all relevant
policies and is otherwise permitted to make the request. The `v1` API
(currently deprecated) does not require this authentication, however
using `v1` is strongly discouraged as it will be removed in a future
release.
Identity Servers use a scheme similar to the Client-Server API's concept
of access tokens to authenticate users. The access tokens provided by an
Identity Server cannot be used to authenticate Client-Server API
requests.
An access token is provided to an endpoint in one of two ways:
1. Via a query string parameter, `access_token=TheTokenHere`.
2. Via a request header, `Authorization: Bearer TheTokenHere`.
Clients are encouraged to the use the `Authorization` header where
possible to prevent the access token being leaked in access/HTTP logs.
The query string should only be used in cases where the `Authorization`
header is inaccessible for the client.
When credentials are required but missing or invalid, the HTTP call will
return with a status of 401 and the error code `M_UNAUTHORIZED`.
{{v2\_auth\_is\_http\_api}}
## Terms of service
Identity Servers are encouraged to have terms of service (or similar
policies) to ensure that users have agreed to their data being processed
by the server. To facilitate this, an identity server can respond to
almost any authenticated API endpoint with an HTTP 403 and the error
code `M_TERMS_NOT_SIGNED`. The error code is used to indicate that the
user must accept new terms of service before being able to continue.
All endpoints which support authentication can return the
`M_TERMS_NOT_SIGNED` error. When clients receive the error, they are
expected to make a call to `GET /terms` to find out what terms the
server offers. The client compares this to the `m.accepted_terms`
account data for the user (described later) and presents the user with
option to accept the still-missing terms of service. After the user has
made their selection, if applicable, the client sends a request to
`POST /terms` to indicate the user's acceptance. The server cannot
expect that the client will send acceptance for all pending terms, and
the client should not expect that the server will not respond with
another `M_TERMS_NOT_SIGNED` on their next request. The terms the user
has just accepted are appended to `m.accepted_terms`.
{{m\_accepted\_terms\_event}}
{{v2\_terms\_is\_http\_api}}
## Status check
{{ping\_is\_http\_api}}
{{v2\_ping\_is\_http\_api}}
## Key management
An identity server has some long-term public-private keypairs. These are
named in a scheme `algorithm:identifier`, e.g. `ed25519:0`. When signing
an association, the standard [Signing
JSON](../appendices.html#signing-json) algorithm applies.
The identity server may also keep track of some short-term
public-private keypairs, which may have different usage and lifetime
characteristics than the service's long-term keys.
{{pubkey\_is\_http\_api}}
{{v2\_pubkey\_is\_http\_api}}
## Association lookup
{{lookup\_is\_http\_api}}
{{v2\_lookup\_is\_http\_api}}
### Client behaviour
Note
This section only covers the v2 lookup endpoint. The v1 endpoint is
described in isolation above.
Prior to performing a lookup clients SHOULD make a request to the
`/hash_details` endpoint to determine what algorithms the server
supports (described in more detail below). The client then uses this
information to form a `/lookup` request and receive known bindings from
the server.
Clients MUST support at least the `sha256` algorithm.
### Server behaviour
Note
This section only covers the v2 lookup endpoint. The v1 endpoint is
described in isolation above.
Servers, upon receipt of a `/lookup` request, will compare the query
against known bindings it has, hashing the identifiers it knows about as
needed to verify exact matches to the request.
Servers MUST support at least the `sha256` algorithm.
### Algorithms
Some algorithms are defined as part of the specification, however other
formats can be negotiated between the client and server using
`/hash_details`.
#### `sha256`
This algorithm MUST be supported by clients and servers at a minimum. It
is additionally the preferred algorithm for lookups.
When using this algorithm, the client converts the query first into
strings separated by spaces in the format `<address> <medium> <pepper>`.
The `<pepper>` is retrieved from `/hash_details`, the `<medium>` is
typically `email` or `msisdn` (both lowercase), and the `<address>` is
the 3PID to search for. For example, if the client wanted to know about
`alice@example.org`'s bindings, it would first format the query as
`alice@example.org email ThePepperGoesHere`.
Rationale
Mediums and peppers are appended to the address to prevent a common
prefix for each 3PID, helping prevent attackers from pre-computing the
internal state of the hash function.
After formatting each query, the string is run through SHA-256 as
defined by [RFC 4634](https://tools.ietf.org/html/rfc4634). The
resulting bytes are then encoded using URL-Safe [Unpadded
Base64](../appendices.html#unpadded-base64) (similar to [room version
4's event ID format](../rooms/v4.html#event-ids)).
An example set of queries when using the pepper `matrixrocks` would be:
"alice@example.com email matrixrocks" -> "4kenr7N9drpCJ4AfalmlGQVsOn3o2RHjkADUpXJWZUc"
"bob@example.com email matrixrocks" -> "LJwSazmv46n0hlMlsb_iYxI0_HXEqy_yj6Jm636cdT8"
"18005552067 msisdn matrixrocks" -> "nlo35_T5fzSGZzJApqu8lgIudJvmOQtDaHtr-I4rU7I"
The set of hashes is then given as the `addresses` array in `/lookup`.
Note that the pepper used MUST be supplied as `pepper` in the `/lookup`
request.
#### `none`
This algorithm performs plaintext lookups on the identity server.
Typically this algorithm should not be used due to the security concerns
of unhashed identifiers, however some scenarios (such as LDAP-backed
identity servers) prevent the use of hashed identifiers. Identity
servers (and optionally clients) can use this algorithm to perform those
kinds of lookups.
Similar to the `sha256` algorithm, the client converts the queries into
strings separated by spaces in the format `<address> <medium>` - note
the lack of `<pepper>`. For example, if the client wanted to know about
`alice@example.org`'s bindings, it would format the query as
`alice@example.org email`.
The formatted strings are then given as the `addresses` in `/lookup`.
Note that the `pepper` is still required, and must be provided to ensure
the client has made an appropriate request to `/hash_details` first.
### Security considerations
Note
[MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134) has much
more information about the security considerations made for this section
of the specification. This section covers the high-level details for why
the specification is the way it is.
Typically the lookup endpoint is used when a client has an unknown 3PID
it wants to find a Matrix User ID for. Clients normally do this kind of
lookup when inviting new users to a room or searching a user's address
book to find any Matrix users they may not have discovered yet. Rogue or
malicious identity servers could harvest this unknown information and do
nefarious things with it if it were sent in plain text. In order to
protect the privacy of users who might not have a Matrix identifier
bound to their 3PID addresses, the specification attempts to make it
difficult to harvest 3PIDs.
Rationale
Hashing identifiers, while not perfect, helps make the effort required
to harvest identifiers significantly higher. Phone numbers in particular
are still difficult to protect with hashing, however hashing is
objectively better than not.
An alternative to hashing would be using bcrypt or similar with many
rounds, however by nature of needing to serve mobile clients and clients
on limited hardware the solution needs be kept relatively lightweight.
Clients should be cautious of servers not rotating their pepper very
often, and potentially of servers which use a weak pepper - these
servers may be attempting to brute force the identifiers or use rainbow
tables to mine the addresses. Similarly, clients which support the
`none` algorithm should consider at least warning the user of the risks
in sending identifiers in plain text to the identity server.
Addresses are still potentially reversable using a calculated rainbow
table given some identifiers, such as phone numbers, common email
address domains, and leaked addresses are easily calculated. For
example, phone numbers can have roughly 12 digits to them, making them
an easier target for attack than email addresses.
## Establishing associations
The flow for creating an association is session-based.
Within a session, one may prove that one has ownership of a 3PID. Once
this has been established, the user can form an association between that
3PID and a Matrix user ID. Note that this association is only proved one
way; a user can associate *any* Matrix user ID with a validated 3PID,
i.e. I can claim that any email address I own is associated with
@billg:microsoft.com.
Sessions are time-limited; a session is considered to have been modified
when it was created, and then when a validation is performed within it.
A session can only be checked for validation, and validation can only be
performed within a session, within a 24-hour period since its most
recent modification. Any attempts to perform these actions after the
expiry will be rejected, and a new session should be created and used
instead.
To start a session, the client makes a request to the appropriate
`/requestToken` endpoint. The identity server then sends a validation
token to the user, and the user provides the token to the client. The
client then provides the token to the appropriate `/submitToken`
endpoint, completing the session. At this point, the client should
`/bind` the third party identifier or leave it for another entity to
bind.
### Format of a validation token
The format of the validation token is left up to the identity server: it
should choose one appropriate to the 3PID type. (For example, it would
be inappropriate to expect a user to copy a long passphrase including
punctuation from an SMS message into a client.)
Whatever format the identity server uses, the validation token must
consist of at most 255 Unicode codepoints. Clients must pass the token
through without modification.
### Email associations
{{email\_associations\_is\_http\_api}}
{{v2\_email\_associations\_is\_http\_api}}
### Phone number associations
{{phone\_associations\_is\_http\_api}}
{{v2\_phone\_associations\_is\_http\_api}}
### General
{{associations\_is\_http\_api}}
{{v2\_associations\_is\_http\_api}}
## Invitation storage
An identity server can store pending invitations to a user's 3PID, which
will be retrieved and can be either notified on or look up when the 3PID
is associated with a Matrix user ID.
At a later point, if the owner of that particular 3PID binds it with a
Matrix user ID, the identity server will attempt to make an HTTP POST to
the Matrix user's homeserver via the
[/3pid/onbind](../server_server/%SERVER_RELEASE_LABEL%.html#put-matrix-federation-v1-3pid-onbind)
endpoint. The request MUST be signed with a long-term private key for
the identity server.
{{store\_invite\_is\_http\_api}}
{{v2\_store\_invite\_is\_http\_api}}
## Ephemeral invitation signing
To aid clients who may not be able to perform crypto themselves, the
identity server offers some crypto functionality to help in accepting
invitations. This is less secure than the client doing it itself, but
may be useful where this isn't possible.
{{invitation\_signing\_is\_http\_api}}
{{v2\_invitation\_signing\_is\_http\_api}}

594
content/proposals.md
View file

@ -3,3 +3,597 @@ title: "Spec Change Proposals"
weight: 60
type: docs
---
%proposalscssinjection%
Proposals for Spec Changes to Matrix
Table of Contents
# Proposals for Spec Changes to Matrix
If you are interested in submitting a change to the Matrix
Specification, please take note of the following guidelines.
Most changes to the Specification require a formal proposal. Bug fixes,
typos, and clarifications to existing behaviour do not need proposals -
see the [contributing
guide](https://github.com/matrix-org/matrix-doc/blob/master/CONTRIBUTING.rst)
for more information on what does and does not need a proposal.
The proposal process involves some technical writing, having it reviewed
by everyone, having the proposal being accepted, then actually having
your ideas implemented as committed changes to the [Specification
repository](https://github.com/matrix-org/matrix-doc).
Meet the [members of the Core Team](https://matrix.org/foundation), a
group of individuals tasked with ensuring the spec process is as smooth
and painless as possible. Members of the Spec Core Team will do their
best to participate in discussion, summarise when things become
long-winded, and generally try to act towards the benefit of everyone.
As a majority, team members have the ability to change the state of a
proposal, and individually have the final say in proposal discussion.
# Guiding Principles
Proposals **must** act to the greater benefit of the entire Matrix
ecosystem, rather than benefiting or privileging any single player or
subset of players -and must not contain any patent encumbered
intellectual property. Members of the Core Team pledge to act as a
neutral custodian for Matrix on behalf of the whole ecosystem.
For clarity: the Matrix ecosystem is anyone who uses the Matrix
protocol. That includes client users, server admins, client developers,
bot developers, bridge and application service developers, users and
admins who are indirectly using Matrix via 3rd party networks which
happen to be bridged, server developers, room moderators and admins,
companies/projects building products or services on Matrix, spec
contributors, translators, and those who created it in the first place.
"Greater benefit" could include maximising:
- the number of end-users reachable on the open Matrix network
- the number of regular users on the Matrix network (e.g. 30-day
retained federated users)
- the number of online servers in the open federation
- the number of developers building on Matrix
- the number of independent implementations which use Matrix
- the number of bridged end-users reachable on the open Matrix network
- the signal-to-noise ratio of the content on the open Matrix network
(i.e. minimising spam)
- the ability for users to discover content on their terms (empowering
them to select what to see and what not to see)
- the quality and utility of the Matrix spec (as defined by ease and
ability with which a developer can implement spec-compliant clients,
servers, bots, bridges, and other integrations without needing to
refer to any other external material)
In addition, proposal authors are expected to uphold the following
values in their proposed changes to the Matrix protocol:
- Supporting the whole long-term ecosystem rather than individual
stakeholder gain
- Openness rather than proprietary lock-in
- Interoperability rather than fragmentation
- Cross-platform rather than platform-specific
- Collaboration rather than competition
- Accessibility rather than elitism
- Transparency rather than stealth
- Empathy rather than contrariness
- Pragmatism rather than perfection
- Proof rather than conjecture
Please [see
MSC1779](https://github.com/matrix-org/matrix-doc/blob/master/proposals/1779-open-governance.md)
for full details of the project's Guiding Principles.
# Technical notes
Proposals **must** develop Matrix as a layered protocol: with new
features building on layers of shared abstractions rather than
introducing tight vertical coupling within the stack. This ensures that
new features can evolve rapidly by building on existing layers and
swapping out old features without impacting the rest of the stack or
requiring substantial upgrades to the whole ecosystem. This is critical
for Matrix to rapidly evolve and compete effectively with centralised
systems, despite being a federated protocol.
For instance, new features should be implemented using the highest layer
abstractions possible (e.g. new event types, which layer on top of the
existing room semantics, and so don't even require any API changes).
Failing that, the next recourse would be backwards-compatible changes to
the next layer down (e.g. room APIs); failing that, considering changes
to the format of events or the DAG; etc. It would be a very unusual
feature which doesn't build on the existing infrastructure provided by
the spec and instead created new primitives or low level APIs.
Backwards compatibility is very important for Matrix, but not at the
expense of hindering the protocol's evolution. Backwards incompatible
changes to endpoints are allowed when no other alternative exists, and
must be versioned under a new major release of the API. Backwards
incompatible changes to the room algorithm are also allowed when no
other alternative exists, and must be versioned under a new version of
the room algorithm.
There is sometimes a dilemma over where to include higher level
features: for instance, should video conferencing be formalised in the
spec, or should it be implemented via widgets? Should reputation systems
be specified? Should search engine behaviour be specified?
There is no universal answer to this, but the following guidelines
should be applied:
1. If the feature would benefit the whole Matrix ecosystem and is
aligned with the guiding principles above, then it should be
supported by the spec.
2. If the spec already makes the feature possible without changing any
of the implementations and spec, then it may not need to be added to
the spec.
3. However, if the best user experience for a feature does require
custom implementation behaviour then the behaviour should be defined
in the spec such that all implementations may implement it.
4. However, the spec must never add dependencies on
unspecified/nonstandardised 3rd party behaviour.
As a worked example:
1. Video conferencing is clearly a feature which would benefit the
whole ecosystem, and so the spec should find a way to make it
happen.
2. Video conferencing can be achieved by widgets without requiring any
compulsory changes to clients nor servers to work, and so could be
omitted from the spec.
3. A better experience could be achieved by embedding Jitsi natively
into clients rather than using a widget...
4. ...except that would add a dependency on unspecified/nonstandardised
3rd party behaviour, so must not be added to the spec.
Therefore, our two options in the specific case of video conferencing
are either to spec SFU conferencing semantics for WebRTC (or refer to an
existing spec for doing so), or to keep it as a widget-based approach
(optionally with widget extensions specific for more deeply integrating
video conferencing use cases).
As an alternative example: it's very unlikely that "how to visualise
Magnetic Resonance Imaging data over Matrix" would ever be added to the
Matrix spec (other than perhaps a custom event type in a wider
standardised Matrix event registry) given that the spec's existing
primitives of file transfer and extensible events (MSC1767) give
excellent tools for transferring and visualising arbitrary rich data.
Supporting public search engines are likely to not require custom spec
features (other than possibly better bulk access APIs), given they can
be implemented as clients using the existing CS API. An exception could
be API features required by decentralised search infrastructure
(avoiding centralisation of power by a centralised search engine).
Features such as reactions, threaded messages, editable messages,
spam/abuse/content filtering (and reputation systems), are all features
which would clearly benefit the whole Matrix ecosystem, and cannot be
implemented in an interoperable way using the current spec; so they
necessitate a spec change.
# Process
The process for submitting a Matrix Spec Change (MSC) Proposal in detail
is as follows:
- Create a first draft of your proposal using [GitHub-flavored
Markdown](https://help.github.com/articles/basic-writing-and-formatting-syntax/)
- In the document, clearly state the problem being solved, and the
possible solutions being proposed for solving it and their
respective trade-offs.
- Proposal documents are intended to be as lightweight and
flexible as the author desires; there is no formal template; the
intention is to iterate as quickly as possible to get to a good
design.
- However, a [template with suggested
headers](https://github.com/matrix-org/matrix-doc/blob/master/proposals/0000-proposal-template.md)
is available to get you started if necessary.
- Take care in creating your proposal. Specify your intended
changes, and give reasoning to back them up. Changes without
justification will likely be poorly received by the community.
- Fork and make a PR to the
[matrix-doc](https://github.com/matrix-org/matrix-doc) repository.
The ID of your PR will become the MSC ID for the lifetime of your
proposal.
- The proposal must live in the `proposals/` directory with a
filename that follows the format `1234-my-new-proposal.md` where
`1234` is the MSC ID.
- Your PR description must include a link to the rendered Markdown
document and a summary of the proposal.
- It is often very helpful to link any related MSCs or [matrix-doc
issues](https://github.com/matrix-org/matrix-doc/issues) to give
context for the proposal.
- Additionally, please be sure to sign off your proposal PR as per
the guidelines listed on
[CONTRIBUTING.rst](https://github.com/matrix-org/matrix-doc/blob/master/CONTRIBUTING.rst).
- Gather feedback as widely as possible.
- The aim is to get maximum consensus towards an optimal solution.
Sometimes trade-offs are required to meet this goal. Decisions
should be made to the benefit of all major use cases.
- A good place to ask for feedback on a specific proposal is
[\#matrix-spec:matrix.org](https://matrix.to/#/#matrix-spec:matrix.org).
If preferred, an alternative room can be created and advertised
in \#matrix-spec:matrix.org. Please also link to the room in
your PR description.
- For additional discussion areas, know that
\#matrix-dev:matrix.org is for developers using existing Matrix
APIs, \#matrix:matrix.org is for users trying to run Matrix apps
(clients & servers) and \#matrix-architecture:matrix.org is for
cross-cutting discussion of Matrix's architectural design.
- The point of the spec proposal process is to be collaborative
rather than competitive, and to try to solve the problem in
question with the optimal set of trade-offs. The author should
neutrally gather the various viewpoints and get consensus, but
this can sometimes be time-consuming (or the author may be
biased), in which case an impartial 'shepherd' can be assigned
to help guide the proposal through this process instead. A
shepherd is typically a neutral party from the Spec Core Team or
an experienced member of the community. There is no formal
process for assignment. Simply ask for a shepherd to help get
your proposal through and one will be assigned based on
availability. Having a shepherd is not a requirement for
proposal acceptance.
- Members of the Spec Core Team and community will review and discuss
the PR in the comments and in relevant rooms on Matrix. Discussion
outside of GitHub should be summarised in a comment on the PR.
- When a member of the Spec Core Team believes that no new discussion
points are being made, and the proposal has suitable evidence of
working (see [implementing a proposal](#implementing-a-proposal)
below), they will propose a motion for a final comment period (FCP),
along with a *disposition* of either merge, close or postpone. This
FCP is provided to allow a short period of time for any invested
party to provide a final objection before a major decision is made.
If sufficient reasoning is given, an FCP can be cancelled. It is
often preceded by a comment summarising the current state of the
discussion, along with reasoning for its occurrence.
- A concern can be raised by a Spec Core Team member at any time,
which will block an FCP from beginning. An FCP will only begin when
75% of the members of the Spec Core Team agree on its outcome, and
all existing concerns have been resolved.
- The FCP will then begin and last for 5 days, giving anyone else some
time to speak up before it concludes. On its conclusion, the
disposition of the FCP will be carried out. If sufficient reasoning
against the disposition is raised, the FCP can be cancelled and the
MSC will continue to evolve accordingly.
- Once the proposal has been accepted and merged, it is time to submit
the actual change to the Specification that your proposal reasoned
about. This is known as a spec PR. However in order for the spec PR
to be accepted, an implementation **must** be shown to prove that it
works well in practice. A link to the implementation should be
included in the PR description. In addition, any significant
unforeseen changes to the original idea found during this process
will warrant another MSC. Any minor, non-fundamental changes are
allowed but **must** be documented in the original proposal
document. This ensures that someone reading a proposal in the future
doesn't assume old information wasn't merged into the spec.
- Similar to the proposal PR, please sign off the spec PR as per
the guidelines on
[CONTRIBUTING.rst](https://github.com/matrix-org/matrix-doc/blob/master/CONTRIBUTING.rst).
- Your PR will then be reviewed and hopefully merged on the grounds it
is implemented sufficiently. If so, then give yourself a pat on the
back knowing you've contributed to the Matrix protocol for the
benefit of users and developers alike :)
The process for handling proposals is shown visually in the following
diagram. Note that the lifetime of a proposal is tracked through the
corresponding labels for each stage on the
[matrix-doc](https://github.com/matrix-org/matrix-doc) issue and pull
request trackers.
+ +
Proposals | Spec PRs | Additional States
+-------+ | +------+ | +---------------+
| |
+----------------------+ | +---------+ | +-----------+
| | | | | | | |
| Proposal | | +------= Spec PR | | | Postponed |
| Drafting and Initial | | | | Missing | | | |
| Feedback Gathering | | | | | | +-----------+
| | | | +----+----+ |
+----------+-----------+ | | | | +----------+
| | | v | | |
v | | +-----------------+ | | Closed |
+-------------------+ | | | | | | |
| | | | | Spec PR Created | | +----------+
| Proposal PR | | | | and In Review | |
| In Review | | | | | |
| | | | +--------+--------+ |
+---------+---------+ | | | |
| | | v |
v | | +-----------+ |
+----------------------+ | | | | |
| | | | | Spec PR | |
| Proposed Final | | | | Merged! | |
| Comment Period | | | | | |
| | | | +-----------+ |
+----------+-----------+ | | |
| | | |
v | | |
+----------------------+ | | |
| | | | |
| Final Comment Period | | | |
| | | | |
+----------+-----------+ | | |
| | | |
v | | |
+----------------------+ | | |
| | | | |
| Final Comment Period | | | |
| Complete | | | |
| | | | |
+----------+-----------+ | | |
| | | |
+-----------------+ |
| |
+ +
# Lifetime States
**Note:** All labels are to be placed on the proposal PR.
<table>
<thead>
<tr class="header">
<th>Name</th>
<th>GitHub Label</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Proposal Drafting and Feedback</td>
<td>N/A</td>
<td>A proposal document which is still work-in-progress but is being shared to incorporate feedback. Please prefix your proposal's title with <code>[WIP]</code> to make it easier for reviewers to skim their notifications list.</td>
</tr>
<tr class="even">
<td>Proposal In Review</td>
<td>proposal-in-review</td>
<td>A proposal document which is now ready and waiting for review by the Spec Core Team and community</td>
</tr>
<tr class="odd">
<td>Proposed Final Comment Period</td>
<td>proposed-final-comment-period</td>
<td>Currently awaiting signoff of a 75% majority of team members in order to enter the final comment period</td>
</tr>
<tr class="even">
<td>Final Comment Period</td>
<td>final-comment-period</td>
<td>A proposal document which has reached final comment period either for merge, closure or postponement</td>
</tr>
<tr class="odd">
<td>Final Comment Period Complete</td>
<td>finished-final-comment-period</td>
<td>The final comment period has been completed. Waiting for a demonstration implementation</td>
</tr>
<tr class="even">
<td>Spec PR Missing</td>
<td>spec-pr-missing</td>
<td>The proposal has been agreed, and proven with a demonstration implementation. Waiting for a PR against the Spec</td>
</tr>
<tr class="odd">
<td>Spec PR In Review</td>
<td>spec-pr-in-review</td>
<td>The spec PR has been written, and is currently under review</td>
</tr>
<tr class="even">
<td>Spec PR Merged</td>
<td>merged</td>
<td>A proposal with a sufficient working implementation and whose Spec PR has been merged!</td>
</tr>
<tr class="odd">
<td><p>Postponed</p></td>
<td><p>proposal-postponed</p></td>
<td><p>A proposal that is temporarily blocked or a feature that may not be useful currently but perhaps sometime in the future</p></td>
</tr>
<tr class="even">
<td>Closed</td>
<td>proposal-closed</td>
<td>A proposal which has been reviewed and deemed unsuitable for acceptance</td>
</tr>
<tr class="odd">
<td>Obsolete</td>
<td>obsolete</td>
<td>A proposal which has been made obsolete by another proposal or decision elsewhere.</td>
</tr>
</tbody>
</table>
# Categories
We use category labels on MSCs to place them into a track of work. The
Spec Core Team decides which of the tracks they are focusing on for the
next while and generally makes an effort to pull MSCs out of that
category when possible.
The current categories are:
<table>
<thead>
<tr class="header">
<th>Name</th>
<th>GitHub Label</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Core</td>
<td>kind:core</td>
<td>Important for the protocol's success.</td>
</tr>
<tr class="even">
<td>Feature</td>
<td>kind:feature</td>
<td>Nice to have additions to the spec.</td>
</tr>
<tr class="odd">
<td>Maintenance</td>
<td>kind:maintenance</td>
<td>Fixes or clarifies existing spec.</td>
</tr>
</tbody>
</table>
Some examples of core MSCs would be aggregations, cross-signing, and
groups/communities. These are the sorts of things that if not
implemented could cause the protocol to fail or become second-class.
Features would be areas like enhanced media APIs, new transports, and
bookmarks in comparison. Finally, maintenance MSCs would include
improving error codes, clarifying what is required of an API, and adding
properties to an API which makes it easier to use.
The Spec Core Team assigns a category to each MSC based on the
descriptions above. This can mean that new MSCs get categorized into an
area the team isn't focused on, though that can always change as
priorities evolve. We still encourage that MSCs be opened, even if not
the focus for the time being, as they can still make progress and even
be merged without the Spec Core Team focusing on them specifically.
# Implementing a proposal
As part of the proposal process the spec core team will require evidence
of the MSC working in order for it to move into FCP. This can usually be
a branch/pull request to whichever implementation of choice that proves
the MSC works in practice, though in some cases the MSC itself will be
small enough to be considered proven. Where it's unclear if an MSC will
require an implementation proof, ask in
[\#matrix-spec:matrix.org](https://matrix.to/#/#matrix-spec:matrix.org).
## Early release of an MSC/idea
To help facilitate early releases of software dependent on a spec
release, implementations are required to use the following process to
ensure that the official Matrix namespace is not cluttered with
development or testing data.
Note
Unreleased implementations (including proofs-of-concept demonstrating
that a particular MSC works) do not have to follow this process.
1. Have an idea for a feature.
2. Implement the feature using unstable endpoints, vendor prefixes, and
unstable feature flags as appropriate.
- When using unstable endpoints, they MUST include a vendor
prefix. For example:
`/_matrix/client/unstable/com.example/login`. Vendor prefixes
throughout Matrix always use the Java package naming convention.
The MSC for the feature should identify which preferred vendor
prefix is to be used by early adopters.
- Note that unstable namespaces do not automatically inherit
endpoints from stable namespaces: for example, the fact that
`/_matrix/client/r0/sync` exists does not imply that
`/_matrix/client/unstable/com.example/sync` exists.
- If the client needs to be sure the server supports the feature,
an unstable feature flag that MUST be vendor prefixed is to be
used. This kind of flag shows up in the `unstable_features`
section of `/versions` as, for example, `com.example.new_login`.
The MSC for the feature should identify which preferred feature
flag is to be used by early adopters.
- When using this approach correctly, the implementation can
ship/release the feature at any time, so long as the
implementation is able to accept the technical debt that results
from needing to provide adequate backwards and forwards
compatibility. The implementation MUST support the flag (and
server-side implementation) disappearing and be generally safe
for users. Note that implementations early in the MSC review
process may also be required to provide backwards compatibility
with earlier editions of the proposal.
- If the implementation cannot support the technical debt (or if
it's impossible to provide forwards/backwards compatibility -
e.g. a user authentication change which can't be safely rolled
back), the implementation should not attempt to implement the
feature and should instead wait for a spec release.
- If at any point after early release, the idea changes in a
backwards-incompatible way, the feature flag should also change
so that implementations can adapt as needed.
3. In parallel, or ahead of implementation, open an MSC and solicit
review per above.
4. Before FCP can be called, the Spec Core Team will require evidence
of the MSC working as proposed. A typical example of this is an
implementation of the MSC, though the implementation does not need
to be shipped anywhere and can therefore avoid the
forwards/backwards compatibility concerns mentioned here.
5. The FCP process is completed, and assuming nothing is flagged the
MSC lands.
6. A spec PR is written to incorporate the changes into Matrix.
7. A spec release happens.
8. Implementations switch to using stable prefixes (e.g.: `/r0`) if the
server supports the specification version released. If the server
doesn't advertise the specification version, but does have the
feature flag, unstable prefixes should still be used.
9. A transition period of about 2 months starts immediately after the
spec release, before implementations start to encourage other
implementations to switch to stable endpoints. For example, a server
implementation should start asking client implementations to support
the stable endpoints 2 months after the spec release, if they
haven't already. The same applies in the reverse: if clients cannot
switch to stable prefixes because server implementations haven't
started supporting the new spec release, some noise should be raised
in the general direction of the implementation.
Note
MSCs MUST still describe what the stable endpoints/feature looks like
with a note towards the bottom for what the unstable feature
flag/prefixes are. For example, an MSC would propose <span
class="title-ref">/\_matrix/client/r0/new/endpoint</span>, not <span
class="title-ref">/\_matrix/client/unstable/
com.example/new/endpoint</span>.
In summary:
- Implementations MUST NOT use stable endpoints before the MSC is in
the spec. This includes NOT using stable endpoints in the period
between completion of FCP and release of the spec. passed.
- Implementations are able to ship features that are exposed to users
by default before an MSC has been merged to the spec, provided they
follow the process above.
- Implementations SHOULD be wary of the technical debt they are
incurring by moving faster than the spec.
- The vendor prefix is chosen by the developer of the feature, using
the Java package naming convention. The foundation's preferred
vendor prefix is <span class="title-ref">org.matrix</span>.
- The vendor prefixes, unstable feature flags, and unstable endpoints
should be included in the MSC, though the MSC MUST be written in a
way that proposes new stable endpoints. Typically this is solved by
a small table at the bottom mapping the various values from stable
to unstable.
# Proposal Tracking
This is a living document generated from the list of proposals on the
issue and pull request trackers of the
[matrix-doc](https://github.com/matrix-org/matrix-doc) repo.
We use labels and some metadata in MSC PR descriptions to generate this
page. Labels are assigned by the Spec Core Team whilst triaging the
proposals based on those which exist in the
[matrix-doc](https://github.com/matrix-org/matrix-doc) repo already.
It is worth mentioning that a previous version of the MSC process used a
mixture of GitHub issues and PRs, leading to some MSC numbers deriving
from GitHub issue IDs instead. A useful feature of GitHub is that it
does automatically resolve to an issue, if an issue ID is placed in a
pull URL. This means that
<https://github.com/matrix-org/matrix-doc/pull/$MSCID> will correctly
resolve to the desired MSC, whether it started as an issue or a PR.
Other metadata:
- The MSC number is taken from the GitHub Pull Request ID. This is
carried for the lifetime of the proposal. These IDs do not
necessarily represent a chronological order.
- The GitHub PR title will act as the MSC's title.
- Please link to the spec PR (if any) by adding a "PRs: \#1234" line
in the issue description.
- The creation date is taken from the GitHub PR, but can be overridden
by adding a "Date: yyyy-mm-dd" line in the PR description.
- Updated Date is taken from GitHub.
- Author is the creator of the MSC PR, but can be overridden by adding
an "Author: @username" line in the body of the issue description.
Please make sure @username is a GitHub user (include the @!)
- A shepherd can be assigned by adding a "Shepherd: @username" line in
the issue description. Again, make sure this is a real GitHub user.

78
content/push-gateway-api.md
View file

@ -3,3 +3,81 @@ title: "Push Gateway API"
weight: 50
type: docs
---
# Push Gateway API
{{unstable\_warning\_block\_PUSH\_GATEWAY\_RELEASE\_LABEL}}
Clients may want to receive push notifications when events are received
at the homeserver. This is managed by a distinct entity called the Push
Gateway.
Table of Contents
## Changelog
**Version: %PUSH\_GATEWAY\_RELEASE\_LABEL%**
{{push\_gateway\_changelog}}
This version of the specification is generated from
[matrix-doc](https://github.com/matrix-org/matrix-doc) as of Git commit
[{{git\_version}}](https://github.com/matrix-org/matrix-doc/tree/%7B%7Bgit_rev%7D%7D).
For the full historical changelog, see
<https://github.com/matrix-org/matrix-doc/blob/master/changelogs/push_gateway.rst>
### Other versions of this specification
The following other versions are also available, in reverse
chronological order:
- [HEAD](https://matrix.org/docs/spec/push_gateway/unstable.html):
Includes all changes since the latest versioned release.
- [r0.1.0](https://matrix.org/docs/spec/push_gateway/r0.1.0.html)
## Overview
A client's homeserver forwards information about received events to the
push gateway. The gateway then submits a push notification to the push
notification provider (e.g. APNS, GCM).
+--------------------+ +-------------------+
Matrix HTTP | | | |
Notification Protocol | App Developer | | Device Vendor |
| | | |
+-------------------+ | +----------------+ | | +---------------+ |
| | | | | | | | | |
| Matrix homeserver +-----> Push Gateway +------> Push Provider | |
| | | | | | | | | |
+-^-----------------+ | +----------------+ | | +----+----------+ |
| | | | | |
Matrix | | | | | |
Client/Server API + | | | | |
| | +--------------------+ +-------------------+
| +--+-+ |
| | <-------------------------------------------+
+---+ |
| | Provider Push Protocol
+----+
Mobile Device or Client
## Homeserver behaviour
This describes the format used by "HTTP" pushers to send notifications
of events to Push Gateways. If the endpoint returns an HTTP error code,
the homeserver SHOULD retry for a reasonable amount of time using
exponential backoff.
When pushing notifications for events, the homeserver is expected to
include all of the event-related fields in the `/notify` request. When
the homeserver is performing a push where the `format` is
`"event_id_only"`, only the `event_id`, `room_id`, `counts`, and
`devices` are required to be populated.
Note that most of the values and behaviour of this endpoint is described
by the Client-Server API's [Push
Module](../client_server/%CLIENT_RELEASE_LABEL%.html#module-push).
{{push\_notifier\_push\_http\_api}}

284
content/rooms/v1.md
View file

@ -3,3 +3,287 @@ title: Room Version 1
type: docs
weight: 10
---
# Room Version 1
This room version is the first ever version for rooms, and contains the
building blocks for other room versions.
Table of Contents
## Client considerations
Clients may need to consider some algorithms performed by the server for
their own implementation.
### Redactions
Upon receipt of a redaction event, the server must strip off any keys
not in the following list:
- `event_id`
- `type`
- `room_id`
- `sender`
- `state_key`
- `content`
- `hashes`
- `signatures`
- `depth`
- `prev_events`
- `prev_state`
- `auth_events`
- `origin`
- `origin_server_ts`
- `membership`
The content object must also be stripped of all keys, unless it is one
of one of the following event types:
- `m.room.member` allows key `membership`.
- `m.room.create` allows key `creator`.
- `m.room.join_rules` allows key `join_rule`.
- `m.room.power_levels` allows keys `ban`, `events`, `events_default`,
`kick`, `redact`, `state_default`, `users`, `users_default`.
- `m.room.aliases` allows key `aliases`.
- `m.room.history_visibility` allows key `history_visibility`.
## Server implementation components
Warning
The information contained in this section is strictly for server
implementors. Applications which use the Client-Server API are generally
unaffected by the intricacies contained here. The section above
regarding client considerations is the resource that Client-Server API
use cases should reference.
The algorithms defined here should only apply to version 1 rooms. Other
algorithms may be used by other room versions, and as such servers
should be aware of which version room they are dealing with prior to
executing a given algorithm.
Warning
Although there are many rooms using room version 1, it is known to have
undesirable effects. Servers implementing support for room version 1
should be aware that restrictions should be generally relaxed and that
inconsistencies may occur.
### State resolution
Warning
Room version 1 is known to have bugs that can cause the state of rooms
to reset to older versions of the room's state. For example this could
mean that users who had joined the room may be removed from the room,
admins and moderators could lose their power level, and users who have
been banned from the room may be able to rejoin. Other state events such
as the the room's name or topic could also reset to a previous version.
This is fixed in the state resolution algorithm introduced in room
version 2.
The room state *S*′(*E*) after an event *E* is defined in terms of the
room state *S*(*E*) before *E*, and depends on whether *E* is a state
event or a message event:
- If *E* is a message event, then *S*′(*E*) = *S*(*E*).
- If *E* is a state event, then *S*′(*E*) is *S*(*E*), except that its
entry corresponding to *E*'s `event_type` and `state_key` is
replaced by *E*'s `event_id`.
The room state *S*(*E*) before *E* is the *resolution* of the set of
states {*S*′(*E*′), *S*′(*E*″), …} consisting of the states after each
of *E*'s `prev_event`s {*E*′, *E*″, …}.
The *resolution* of a set of states is defined as follows. The resolved
state is built up in a number of passes; here we use *R* to refer to the
results of the resolution so far.
- Start by setting *R* to the union of the states to be resolved,
excluding any *conflicting* events.
- First we resolve conflicts between `m.room.power_levels` events. If
there is no conflict, this step is skipped, otherwise:
- Assemble all the `m.room.power_levels` events from the states to
be resolved into a list.
- Sort the list by ascending `depth` then descending
`sha1(event_id)`.
- Add the first event in the list to *R*.
- For each subsequent event in the list, check that the event
would be allowed by the authorization rules for a room in state
*R*. If the event would be allowed, then update *R* with the
event and continue with the next event in the list. If it would
not be allowed, stop and continue below with `m.room.join_rules`
events.
- Repeat the above process for conflicts between `m.room.join_rules`
events.
- Repeat the above process for conflicts between `m.room.member`
events.
- No other events affect the authorization rules, so for all other
conflicts, just pick the event with the highest depth and lowest
`sha1(event_id)` that passes authentication in *R* and add it to
*R*.
A *conflict* occurs between states where those states have different
`event_ids` for the same `(event_type, state_key)`. The events thus
affected are said to be *conflicting* events.
### Authorization rules
The types of state events that affect authorization are:
- `m.room.create`
- `m.room.member`
- `m.room.join_rules`
- `m.room.power_levels`
- `m.room.third_party_invite`
Note
Power levels are inferred from defaults when not explicitly supplied.
For example, mentions of the `sender`'s power level can also refer to
the default power level for users in the room.
The rules are as follows:
1. If type is `m.room.create`:
1. If it has any previous events, reject.
2. If the domain of the `room_id` does not match the domain of the
`sender`, reject.
3. If `content.room_version` is present and is not a recognised
version, reject.
4. If `content` has no `creator` field, reject.
5. Otherwise, allow.
2. Reject if event has `auth_events` that:
1. have duplicate entries for a given `type` and `state_key` pair
2. have entries whose `type` and `state_key` don't match those
specified by the [auth events
selection](../server_server/%SERVER_RELEASE_LABEL%.html#auth-events-selection)
algorithm described in the server specification.
3. If event does not have a `m.room.create` in its `auth_events`,
reject.
4. If type is `m.room.aliases`:
1. If event has no `state_key`, reject.
2. If sender's domain doesn't matches `state_key`, reject.
3. Otherwise, allow.
5. If type is `m.room.member`:
1. If no `state_key` key or `membership` key in `content`, reject.
2. If `membership` is `join`:
1. If the only previous event is an `m.room.create` and the
`state_key` is the creator, allow.
2. If the `sender` does not match `state_key`, reject.
3. If the `sender` is banned, reject.
4. If the `join_rule` is `invite` then allow if membership
state is `invite` or `join`.
5. If the `join_rule` is `public`, allow.
6. Otherwise, reject.
3. If `membership` is `invite`:
1. If `content` has `third_party_invite` key:
1. If *target user* is banned, reject.
2. If `content.third_party_invite` does not have a `signed`
key, reject.
3. If `signed` does not have `mxid` and `token` keys,
reject.
4. If `mxid` does not match `state_key`, reject.
5. If there is no `m.room.third_party_invite` event in the
current room state with `state_key` matching `token`,
reject.
6. If `sender` does not match `sender` of the
`m.room.third_party_invite`, reject.
7. If any signature in `signed` matches any public key in
the `m.room.third_party_invite` event, allow. The public
keys are in `content` of `m.room.third_party_invite` as:
1. A single public key in the `public_key` field.
2. A list of public keys in the `public_keys` field.
8. Otherwise, reject.
2. If the `sender`'s current membership state is not `join`,
reject.
3. If *target user*'s current membership state is `join` or
`ban`, reject.
4. If the `sender`'s power level is greater than or equal to
the *invite level*, allow.
5. Otherwise, reject.
4. If `membership` is `leave`:
1. If the `sender` matches `state_key`, allow if and only if
that user's current membership state is `invite` or `join`.
2. If the `sender`'s current membership state is not `join`,
reject.
3. If the *target user*'s current membership state is `ban`,
and the `sender`'s power level is less than the *ban level*,
reject.
4. If the `sender`'s power level is greater than or equal to
the *kick level*, and the *target user*'s power level is
less than the `sender`'s power level, allow.
5. Otherwise, reject.
5. If `membership` is `ban`:
1. If the `sender`'s current membership state is not `join`,
reject.
2. If the `sender`'s power level is greater than or equal to
the *ban level*, and the *target user*'s power level is less
than the `sender`'s power level, allow.
3. Otherwise, reject.
6. Otherwise, the membership is unknown. Reject.
6. If the `sender`'s current membership state is not `join`, reject.
7. If type is `m.room.third_party_invite`:
1. Allow if and only if `sender`'s current power level is greater
than or equal to the *invite level*.
8. If the event type's *required power level* is greater than the
`sender`'s power level, reject.
9. If the event has a `state_key` that starts with an `@` and does not
match the `sender`, reject.
10. If type is `m.room.power_levels`:
1. If `users` key in `content` is not a dictionary with keys that
are valid user IDs with values that are integers (or a string
that is an integer), reject.
2. If there is no previous `m.room.power_levels` event in the room,
allow.
3. For the keys `users_default`, `events_default`, `state_default`,
`ban`, `redact`, `kick`, `invite` check if they were added,
changed or removed. For each found alteration:
1. If the current value is higher than the `sender`'s current
power level, reject.
2. If the new value is higher than the `sender`'s current power
level, reject.
4. For each entry being added, changed or removed in both the
`events` and `users` keys:
1. If the current value is higher than the `sender`'s current
power level, reject.
2. If the new value is higher than the `sender`'s current power
level, reject.
5. For each entry being changed under the `users` key, other than
the `sender`'s own entry:
1. If the current value is equal to the `sender`'s current
power level, reject.
6. Otherwise, allow.
11. If type is `m.room.redaction`:
1. If the `sender`'s power level is greater than or equal to the
*redact level*, allow.
2. If the domain of the `event_id` of the event being redacted is
the same as the domain of the `event_id` of the
`m.room.redaction`, allow.
3. Otherwise, reject.
12. Otherwise, allow.
Note
Some consequences of these rules:
- Unless you are a member of the room, the only permitted operations
(apart from the initial create/join) are: joining a public room;
accepting or rejecting an invitation to a room.
- To unban somebody, you must have power level greater than or equal
to both the kick *and* ban levels, *and* greater than the target
user's power level.
### Event format
Events in version 1 rooms have the following structure:
{{definition\_ss\_pdu}}
### Canonical JSON
Servers MUST NOT strictly enforce the JSON format specified in the
[appendices](../appendices.html#canonical-json) for the reasons
described there.

187
content/rooms/v2.md
View file

@ -3,3 +3,190 @@ title: Room Version 2
type: docs
weight: 20
---
# Room Version 2
This room version builds off of [version 1](v1.html) with an improved
state resolution algorithm.
Table of Contents
## Server implementation components
Warning
The information contained in this section is strictly for server
implementors. Applications which use the Client-Server API are generally
unaffected by the details contained here, and can safely ignore their
presence.
Room version 2 uses the base components of [room version 1](v1.html),
changing only the state resolution algorithm.
### State resolution
The room state *S*′(*E*) after an event *E* is defined in terms of the
room state *S*(*E*) before *E*, and depends on whether *E* is a state
event or a message event:
- If *E* is a message event, then *S*′(*E*) = *S*(*E*).
- If *E* is a state event, then *S*′(*E*) is *S*(*E*), except that its
entry corresponding to *E*'s `event_type` and `state_key` is
replaced by *E*'s `event_id`.
The room state *S*(*E*) before *E* is the *resolution* of the set of
states {*S*′(*E*<sub>1</sub>), *S*′(*E*<sub>2</sub>), …} consisting of
the states after each of *E*'s `prev_event`s
{*E*<sub>1</sub>, *E*<sub>2</sub>, …}, where the resolution of a set of
states is given in the algorithm below.
#### Definitions
The state resolution algorithm for version 2 rooms uses the following
definitions, given the set of room states
{*S*<sub>1</sub>, *S*<sub>2</sub>, …}:
Power events
A *power event* is a state event with type `m.room.power_levels` or
`m.room.join_rules`, or a state event with type `m.room.member` where
the `membership` is `leave` or `ban` and the `sender` does not match the
`state_key`. The idea behind this is that power events are events that
might remove someone's ability to do something in the room.
Unconflicted state map and conflicted state set
The *unconflicted state map* is the state where the value of each key
exists and is the same in each state *S*<sub>*i*</sub>. The *conflicted
state set* is the set of all other state events. Note that the
unconflicted state map only has one event per `(event_type, state_key)`,
whereas the conflicted state set may have multiple events.
Auth difference
The *auth difference* is calculated by first calculating the full auth
chain for each state *S*<sub>*i*</sub>, that is the union of the auth
chains for each event in *S*<sub>*i*</sub>, and then taking every event
that doesn't appear in every auth chain. If *C*<sub>*i*</sub> is the
full auth chain of *S*<sub>*i*</sub>, then the auth difference is
 ∪ *C*<sub>*i*</sub> −  ∩ *C*<sub>*i*</sub>.
Full conflicted set
The *full conflicted set* is the union of the conflicted state set and
the auth difference.
Reverse topological power ordering
The *reverse topological power ordering* of a set of events is the
lexicographically smallest topological ordering based on the DAG formed
by auth events. The reverse topological power ordering is ordered from
earliest event to latest. For comparing two topological orderings to
determine which is the lexicographically smallest, the following
comparison relation on events is used: for events *x* and *y*,
*x* &lt; *y* if
1. *x*'s sender has *greater* power level than *y*'s sender, when
looking at their respective `auth_event`s; or
2. the senders have the same power level, but *x*'s `origin_server_ts`
is *less* than *y*'s `origin_server_ts`; or
3. the senders have the same power level and the events have the same
`origin_server_ts`, but *x*'s `event_id` is *less* than *y*'s
`event_id`.
The reverse topological power ordering can be found by sorting the
events using Kahn's algorithm for topological sorting, and at each step
selecting, among all the candidate vertices, the smallest vertex using
the above comparison relation.
Mainline ordering
Given an `m.room.power_levels` event *P*, the *mainline of* *P* is the
list of events generated by starting with *P* and recursively taking the
`m.room.power_levels` events from the `auth_events`, ordered such that
*P* is last. Given another event *e*, the *closest mainline event to*
*e* is the first event encountered in the mainline when iteratively
descending through the `m.room.power_levels` events in the `auth_events`
starting at *e*. If no mainline event is encountered when iteratively
descending through the `m.room.power_levels` events, then the closest
mainline event to *e* can be considered to be a dummy event that is
before any other event in the mainline of *P* for the purposes of
condition 1 below.
The *mainline ordering based on* *P* of a set of events is the ordering,
from smallest to largest, using the following comparison relation on
events: for events *x* and *y*, *x* &lt; *y* if
1. the closest mainline event to *x* appears *before* the closest
mainline event to *y*; or
2. the closest mainline events are the same, but *x*'s
`origin_server_ts` is *less* than *y*'s `origin_server_ts`; or
3. the closest mainline events are the same and the events have the
same `origin_server_ts`, but *x*'s `event_id` is *less* than *y*'s
`event_id`.
Iterative auth checks
The *iterative auth checks algorithm* takes as input an initial room
state and a sorted list of state events, and constructs a new room state
by iterating through the event list and applying the state event to the
room state if the state event is allowed by the [authorization
rules](../server_server/%SERVER_RELEASE_LABEL%.html#authorization-rules).
If the state event is not allowed by the authorization rules, then the
event is ignored. If a `(event_type, state_key)` key that is required
for checking the authorization rules is not present in the state, then
the appropriate state event from the event's `auth_events` is used if
the auth event is not rejected.
#### Algorithm
The *resolution* of a set of states is obtained as follows:
1. Take all *power events* and any events in their auth chains,
recursively, that appear in the *full conflicted set* and order them
by the *reverse topological power ordering*.
2. Apply the *iterative auth checks algorithm*, starting from the
*unconflicted state map*, to the list of events from the previous
step to get a partially resolved state.
3. Take all remaining events that weren't picked in step 1 and order
them by the mainline ordering based on the power level in the
partially resolved state obtained in step 2.
4. Apply the *iterative auth checks algorithm* on the partial resolved
state and the list of events from the previous step.
5. Update the result by replacing any event with the event with the
same key from the *unconflicted state map*, if such an event exists,
to get the final resolved state.
#### Rejected events
Events that have been rejected due to failing auth based on the state at
the event (rather than based on their auth chain) are handled as usual
by the algorithm, unless otherwise specified.
Note that no events rejected due to failure to auth against their auth
chain should appear in the process, as they should not appear in state
(the algorithm only uses events that appear in either the state sets or
in the auth chain of the events in the state sets).
Rationale
This helps ensure that different servers' view of state is more likely
to converge, since rejection state of an event may be different. This
can happen if a third server gives an incorrect version of the state
when a server joins a room via it (either due to being faulty or
malicious). Convergence of state is a desirable property as it ensures
that all users in the room have a (mostly) consistent view of the state
of the room. If the view of the state on different servers diverges it
can lead to bifurcation of the room due to e.g. servers disagreeing on
who is in the room.
Intuitively, using rejected events feels dangerous, however:
1. Servers cannot arbitrarily make up state, since they still need to
pass the auth checks based on the event's auth chain (e.g. they
can't grant themselves power levels if they didn't have them
before).
2. For a previously rejected event to pass auth there must be a set of
state that allows said event. A malicious server could therefore
produce a fork where it claims the state is that particular set of
state, duplicate the rejected event to point to that fork, and send
the event. The duplicated event would then pass the auth checks.
Ignoring rejected events would therefore not eliminate any potential
attack vectors.
Rejected auth events are deliberately excluded from use in the iterative
auth checks, as auth events aren't re-authed (although non-auth events
are) during the iterative auth checks.

99
content/rooms/v3.md
View file

@ -3,3 +3,102 @@ title: Room Version 3
type: docs
weight: 30
---
# Room Version 3
This room version builds on [version 2](v2.html) with an improved event
format.
Table of Contents
## Client considerations
This room version changes the format for event IDs sent to clients.
Clients should be aware that these event IDs may contain slashes and
other potentially problematic characters. Clients should be treating
event IDs as opaque identifiers and should not be attempting to parse
them into a usable form, just like with other room versions.
Clients should expect to see event IDs changed from the format of
`$randomstring:example.org` to something like
`$acR1l0raoZnm60CBwAVgqbZqoO/mYU81xysh1u7XcJk` (note the lack of domain
and the potentially problematic slash).
## Server implementation components
Warning
The information contained in this section is strictly for server
implementors. Applications which use the Client-Server API are generally
unaffected by the intricacies contained here. The section above
regarding client considerations is the resource that Client-Server API
use cases should reference.
Room version 3 uses the state resolution algorithm defined in [room
version 2](v2.html), and the event format defined here.
### Event IDs
Rationale
In other room versions (namely version 1 and 2) the event ID is a
distinct field from the remainder of the event, which must be tracked as
such. This leads to complications where servers receive multiple events
with the same ID in either the same or different rooms where the server
cannot easily keep track of which event it should be using. By removing
the use of a dedicated event ID, servers are required to track the
hashes on an event to determine its ID.
The event ID is the [reference
hash](../server_server/%SERVER_RELEASE_LABEL%.html#reference-hashes) of
the event encoded using [Unpadded
Base64](../appendices.html#unpadded-base64), prefixed with `$`. A
resulting event ID using this approach should look similar to
`$CD66HAED5npg6074c6pDtLKalHjVfYb2q4Q3LZgrW6o`.
Event IDs should not be sent over federation to servers when the room
uses this room version. On the receiving end of an event, the server
should compute the relevant event ID for itself.
Additionally, the `auth_events` and `prev_events` have had a format
change compared to other room versions to make it easier to handle.
Instead of a tuple of values, they are now plain lists of events.
{{definition\_ss\_pdu\_v3}}
### Changes to APIs
Due to the event ID being removed from the event, some APIs need to
change. All APIs which currently accept an event ID must do so with the
new format. Servers must append the calculated event ID to all events
sent to clients where an event ID would normally be expected.
Because the format of events has changed, servers must be aware of the
room version where the event resides so that the server may parse and
handle the event. The federation API has taken this concern into
consideration by ensuring that servers are aware of (or can find) the
room version during a request.
### Authorization rules for events
The authorization rules for a given event have changed in this room
version due to the change in event format:
- The event no longer needs to be signed by the domain of the event ID
(as there is no domain in the event ID), but still needs to be
signed by the sender's domain.
- In past room versions, redactions were only permitted to enter the
DAG if the sender's domain matched the domain in the event ID being
redacted, or the sender had appropriate permissions per the power
levels. Due to servers now not being able to determine where an
event came from during event authorization, redaction events are
always accepted (provided the event is allowed by `events` and
`events_default` in the power levels). However, servers should not
apply or send redactions to clients until both the redaction event
and original event have been seen, and are valid. Servers should
only apply redactions to events where the sender's domains match, or
the sender of the redaction has the appropriate permissions per the
power levels.
The remaining rules are the same as [room version
1](v1.html#authorization-rules).

60
content/rooms/v4.md
View file

@ -3,3 +3,63 @@ title: Room Version 4
type: docs
weight: 40
---
# Room Version 4
This room version builds on [version 3](v3.html) using a different
encoding for event IDs.
Table of Contents
## Client considerations
This room version changes the format form event IDs sent to clients.
Clients should already be treating event IDs as opaque identifiers, and
should not be concerned with the format of them. Clients should still
encode the event ID when including it in a request path.
Clients should expect to see event IDs changed from the format of
`$randomstring:example.org` to something like
`$Rqnc-F-dvnEYJTyHq_iKxU2bZ1CI92-kuZq3a5lr5Zg` (note the lack of
domain).
## Server implementation components
Warning
The information contained in this section is strictly for server
implementors. Applications which use the Client-Server API are generally
unaffected by the intricacies contained here. The section above
regarding client considerations is the resource that Client-Server API
use cases should reference.
Room version 4 uses the same algorithms defined in [room version
3](v3.html), however using URL-safe base64 to generate the event ID.
### Event IDs
Rationale
Room version 3 generated event IDs that were difficult for client
implementations which were not encoding the event ID to function in
those rooms. It additionally raised concern due to the `/` character
being interpretted differently by some reverse proxy software, and
generally made administration harder.
The event ID is the [reference
hash](../server_server/%SERVER_RELEASE_LABEL%.html#reference-hashes) of
the event encoded using a variation of [Unpadded
Base64](../appendices.html#unpadded-base64) which replaces the 62nd and
63rd characters with `-` and `_` instead of using `+` and `/`. This
matches [RFC4648's definition of URL-safe
base64](https://tools.ietf.org/html/rfc4648#section-5). Event IDs are
still prefixed with `$` and may result in looking like
`$Rqnc-F-dvnEYJTyHq_iKxU2bZ1CI92-kuZq3a5lr5Zg`.
Just like in room version 3, event IDs should not be sent over
federation to servers when the room uses this room version. On the
receiving end of an event, the server should compute the relevant event
ID for itself. Room version 3 also changes the format of `auth_events`
and `prev_events` in a PDU.
{{definition\_ss\_pdu\_v4}}

44
content/rooms/v5.md
View file

@ -3,3 +3,47 @@ title: Room Version 5
type: docs
weight: 50
---
# Room Version 5
This room version builds on [version 4](v4.html) while enforcing signing
key validity periods for events.
Table of Contents
## Client considerations
There are no specific requirements for clients in this room version.
Clients should be aware of event ID changes in [room version
4](v4.html), however.
## Server implementation components
Warning
The information contained in this section is strictly for server
implementors. Applications which use the Client-Server API are generally
unaffected by the intricacies contained here. The section above
regarding client considerations is the resource that Client-Server API
use cases should reference.
Room version 5 uses the same algorithms defined in [room version
4](v4.html), ensuring that signing key validity is respected.
### Signing key validity period
When validating event signatures, servers MUST enforce the
`valid_until_ts` property from a key request is at least as large as the
`origin_server_ts` for the event being validated. Servers missing a copy
of the signing key MUST try to obtain one via the [GET
/\_matrix/key/v2/server](../server_server/%SERVER_RELEASE_LABEL%.html#get-matrix-key-v2-server-keyid)
or [POST
/\_matrix/key/v2/query](../server_server/%SERVER_RELEASE_LABEL%.html#post-matrix-key-v2-query)
APIs. When using the `/query` endpoint, servers MUST set the
`minimum_valid_until_ts` property to prompt the notary server to attempt
to refresh the key if appropriate.
Servers MUST use the lesser of `valid_until_ts` and 7 days into the
future when determining if a key is valid. This is to avoid a situation
where an attacker publishes a key which is valid for a significant
amount of time without a way for the homeserver owner to revoke it.

83
content/rooms/v6.md
View file

@ -3,3 +3,86 @@ title: Room Version 6
type: docs
weight: 60
---
# Room Version 6
This room version builds on [version 5](v5.html) while changing various
authorization rules performed on events.
Table of Contents
## Client considerations
The redaction algorithm has changed from [room version 1](v1.html) to
remove all rules against events of type `m.room.aliases`. Room versions
2, 3, 4, and 5 all use v1's redaction algorithm. The algorithm is
otherwise unchanged.
## Server implementation components
Warning
The information contained in this section is strictly for server
implementors. Applications which use the Client-Server API are generally
unaffected by the intricacies contained here. The section above
regarding client considerations is the resource that Client-Server API
use cases should reference.
Room version 6 makes the following alterations to algorithms described
in [room version 5](v5.html).
### Redactions
As mentioned in the client considerations portion of this specification,
all special meaning has been removed for events of type
`m.room.aliases`. The algorithm is otherwise unchanged.
### Authorization rules for events
Like redactions, all rules relating specifically to events of type
`m.room.aliases` are removed. They must still pass authorization checks
relating to state events.
Additionally, the authorization rules for events of type
`m.room.power_levels` now include the content key `notifications`. This
new rule takes the place of the rule which checks the `events` and
`users` keys.
For completeness, the changes to the auth rules can be represented as
follows:
...
-If type is `m.room.aliases`:
-
- a. If event has no `state_key`, reject.
- b. If sender's domain doesn't matches `state_key`, reject.
- c. Otherwise, allow.
...
If type is `m.room.power_levels`:
...
- * For each entry being added, changed or removed in both the `events` and `users` keys:
+ * For each entry being added, changed or removed in the `events`, `users`, and `notifications` keys:
i. If the current value is higher than the `sender`'s current power level, reject.
ii. If the new value is higher than the `sender`'s current power level, reject.
...
The remaining rules are the same as in [room version
3](v3.html#authorization-rules-for-events) (the last inherited room
version to specify the authorization rules).
### Canonical JSON
Servers MUST strictly enforce the JSON format specified in the
[appendices](../appendices.html#canonical-json). This translates to a
400 `M_BAD_JSON` error on most endpoints, or discarding of events over
federation. For example, the Federation API's `/send` endpoint would
discard the event whereas the Client Server API's `/send/{eventType}`
endpoint would return a `M_BAD_JSON` error.

1174
content/server-server-api.md
View file

File diff suppressed because it is too large Load diff