From ffe577371db7482a3df92a4f9dee6f984922119a Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Mon, 10 Sep 2018 16:29:30 -0600 Subject: [PATCH 01/14] Add a room version specification The "Room Specification" (or "Room Version Specification") is the specification that defines which room versions do what and are intended to be documents which speak the truth about how rooms operate under the hood. The approach taken here is a bit different than other specifications. For starters, the specification is versioned in this project instead of relying on the matrix.org repository to track compiled HTML. This is done for a couple reasons, the first being we're still developing the v1 specification while concurrently making a v2 spec and the second being trying to reduce the reliance on matrix.org's repository for specifications. Because the room spec is built into versions, some changes needed to be made. The `targets.yaml` now has a special syntax for indicating what version something is at, and the changelog generator can handle rendering different versions of the same changelog (as parsed from the RST). Some additional work has been put in to the changelog parsing to allow us to reference the v1 room spec as "v1" without having to sacrifice clarity in the changelog headings. Finally, this moves the state resolution algorithms into the versioned spec as a result of MSC1759 (https://github.com/matrix-org/matrix-doc/pull/1759). Note: this does not introduce the concept of versioned schemas (tabs) that I was previously working with. There's currently no use for them, so they are shelved elsewhere. --- changelogs/rooms.rst | 11 + changelogs/rooms/newsfragments/.gitignore | 1 + changelogs/rooms/pyproject.toml | 30 +++ meta/releasing-rooms-v2.md | 38 ++++ scripts/gendoc.py | 7 +- .../templating/matrix_templates/sections.py | 34 ++-- scripts/templating/matrix_templates/units.py | 56 +++++- .../appendices/identifier_grammar.rst | 42 +--- specification/rooms/intro.rst | 70 +++++++ specification/rooms/unstable.rst | 54 +++++ specification/rooms/v1.rst | 112 +++++++++++ specification/rooms/v2.rst | 181 +++++++++++++++++ specification/server_server_api.rst | 189 +----------------- specification/targets.yaml | 29 +++ 14 files changed, 609 insertions(+), 245 deletions(-) create mode 100644 changelogs/rooms.rst create mode 100644 changelogs/rooms/newsfragments/.gitignore create mode 100644 changelogs/rooms/pyproject.toml create mode 100644 meta/releasing-rooms-v2.md create mode 100644 specification/rooms/intro.rst create mode 100644 specification/rooms/unstable.rst create mode 100644 specification/rooms/v1.rst create mode 100644 specification/rooms/v2.rst diff --git a/changelogs/rooms.rst b/changelogs/rooms.rst new file mode 100644 index 00000000..55a16790 --- /dev/null +++ b/changelogs/rooms.rst @@ -0,0 +1,11 @@ +.. version: v2 + +.. Note: We set the version as "version 2" so that we can maintain a specific version +.. variable in the changelog. We already know the next version is going to be v2, so +.. this makes it easier to copy/paste unstable.rst to v2.rst + +Room version 1 +============== +.. version: v1 + +This is the first iteration of rooms in Matrix. diff --git a/changelogs/rooms/newsfragments/.gitignore b/changelogs/rooms/newsfragments/.gitignore new file mode 100644 index 00000000..b722e9e1 --- /dev/null +++ b/changelogs/rooms/newsfragments/.gitignore @@ -0,0 +1 @@ +!.gitignore \ No newline at end of file diff --git a/changelogs/rooms/pyproject.toml b/changelogs/rooms/pyproject.toml new file mode 100644 index 00000000..b56e19a9 --- /dev/null +++ b/changelogs/rooms/pyproject.toml @@ -0,0 +1,30 @@ +[tool.towncrier] + filename = "../rooms.rst" + directory = "newsfragments" + issue_format = "`#{issue} `_" + title_format = "{version}" + + [[tool.towncrier.type]] + directory = "breaking" + name = "Breaking Changes" + showcontent = true + + [[tool.towncrier.type]] + directory = "deprecation" + name = "Deprecations" + showcontent = true + + [[tool.towncrier.type]] + directory = "new" + name = "New Endpoints" + showcontent = true + + [[tool.towncrier.type]] + directory = "feature" + name = "Backwards Compatible Changes" + showcontent = true + + [[tool.towncrier.type]] + directory = "clarification" + name = "Spec Clarifications" + showcontent = true diff --git a/meta/releasing-rooms-v2.md b/meta/releasing-rooms-v2.md new file mode 100644 index 00000000..80277749 --- /dev/null +++ b/meta/releasing-rooms-v2.md @@ -0,0 +1,38 @@ +# How to release Room Version 2 + +Room versions are a bit special for the release given they have been +introduced at v2 rather than ever getting a v1 release. Additionally, +room versions have different release requirements due to the in-project +versioning of documents rather than relying on matrix.org to maintain +old generated output of specifications. + +As of writing, the project is structured to support 3 logical versions +for rooms: v1, v2, and unstable. Unstable is currently pointed at v2 +in order to aid development. After v2 is released, unstable may wish +to be left as an independent version or similarly be pointed at a "v3" +document. + +Due to room versions being versioned in-project, the generated output +from a release is not to be sent off to matrix-doc/matrix.org. Instead, +in `gendoc.py` the default value for `--room_version` should be set to +the current release (`v2`, for example) so the index renders the right +edition in the table. + +After editing `gendoc.py`, the changelog should be generated according +to the towncrier instructions. You may need to fix the `.. version: v2` +comment located in the `rooms.rst` changelog to be just underneath the +title instead of at the end of the section. + +The `targets.yaml` file needs to be set up to point unstable to the +right set of files. Ensure that `unstable.rst` is referenced instead +of `v2.rst`, and that `unstable.rst` has appropriate contents. + +Finally, in the `intro.rst` for room versions, re-add unstable to the +list of available versions. It is currently commented out to avoid +confusing people, so it should be possible to uncomment it and put it +back into the list. + +From there, the standard process of using a release branch, tagging it, +and announcing it to the world should be followed. If required, the +various other APIs should be updated to better support the new room +version. diff --git a/scripts/gendoc.py b/scripts/gendoc.py index 8310ad58..0455c90a 100755 --- a/scripts/gendoc.py +++ b/scripts/gendoc.py @@ -457,7 +457,7 @@ def main(targets, dest_dir, keep_intermediates, substitutions): rst_file = os.path.join(tmp_dir, "spec_%s.rst" % (target_name,)) if version_label: - d = os.path.join(dest_dir, target_name) + d = os.path.join(dest_dir, target_name.split('@')[0]) if not os.path.exists(d): os.mkdir(d) html_file = os.path.join(d, "%s.html" % version_label) @@ -529,6 +529,10 @@ if __name__ == '__main__': "--identity_release", "-i", action="store", default="unstable", help="The identity service release tag to generate, e.g. r1.2" ) + parser.add_argument( + "--room_version", "-r", action="store", default="unstable", + help="The current room version to advertise, e.g. v2" + ) parser.add_argument( "--list_targets", action="store_true", help="Do not update the specification. Instead print a list of targets.", @@ -555,6 +559,7 @@ if __name__ == '__main__': "%APPSERVICE_RELEASE_LABEL%": args.appservice_release, "%IDENTITY_RELEASE_LABEL%": args.identity_release, "%PUSH_GATEWAY_RELEASE_LABEL%": args.push_gateway_release, + "%CURRENT_ROOM_VERSION%": args.room_version, } exit (main(args.target or ["all"], args.dest, args.nodelete, substitutions)) diff --git a/scripts/templating/matrix_templates/sections.py b/scripts/templating/matrix_templates/sections.py index 185970c9..af497674 100644 --- a/scripts/templating/matrix_templates/sections.py +++ b/scripts/templating/matrix_templates/sections.py @@ -17,8 +17,11 @@ from batesian.sections import Sections import inspect import json import os +import logging +logger = logging.getLogger(__name__) + class MatrixSections(Sections): # pass through git ver so it'll be dropped in the input file @@ -28,26 +31,19 @@ class MatrixSections(Sections): def render_git_rev(self): return self.units.get("git_version")["revision"] - def render_client_server_changelog(self): + def render_changelogs(self): + rendered = {} changelogs = self.units.get("changelogs") - return changelogs["client_server"] - - # TODO: We should make this a generic variable instead of having to add functions all the time. - def render_push_gateway_changelog(self): - changelogs = self.units.get("changelogs") - return changelogs["push_gateway"] - - def render_identity_service_changelog(self): - changelogs = self.units.get("changelogs") - return changelogs["identity_service"] - - def render_server_server_changelog(self): - changelogs = self.units.get("changelogs") - return changelogs["server_server"] - - def render_application_service_changelog(self): - changelogs = self.units.get("changelogs") - return changelogs["application_service"] + for spec, versioned in changelogs.items(): + spec_var = "%s_changelog" % spec + logger.info("Rendering changelog for spec: %s" % spec) + for version, changelog in versioned.items(): + version_var = "%s_%s" % (spec_var, version) + logger.info("Rendering changelog for %s" % version_var) + rendered[version_var] = changelog + if version == "unstable": + rendered[spec_var] = changelog + return rendered def _render_events(self, filterFn, sortFn): template = self.env.get_template("events.tmpl") diff --git a/scripts/templating/matrix_templates/units.py b/scripts/templating/matrix_templates/units.py index 11a9d441..666434a5 100644 --- a/scripts/templating/matrix_templates/units.py +++ b/scripts/templating/matrix_templates/units.py @@ -757,6 +757,7 @@ class MatrixUnits(Units): is_ver = substitutions.get("%IDENTITY_RELEASE_LABEL%", "unstable") as_ver = substitutions.get("%APPSERVICE_RELEASE_LABEL%", "unstable") push_gw_ver = substitutions.get("%PUSH_GATEWAY_RELEASE_LABEL%", "unstable") + room_ver = substitutions.get("%CURRENT_ROOM_VERSION%", "unstable") # we abuse the typetable to return this info to the templates return TypeTable(rows=[ @@ -780,6 +781,10 @@ class MatrixUnits(Units): "`Push Gateway API `_", push_gw_ver, "Push notifications for Matrix events", + ), TypeTableRow( + "`Rooms `_", + room_ver, + "Specification for behaviour of rooms, such as event formats", ), ]) @@ -906,11 +911,26 @@ class MatrixUnits(Units): def load_changelogs(self): changelogs = {} + # Changelog generation is a bit complicated. We rely on towncrier to + # generate the unstable/current changelog, but otherwise use the RST + # edition to record historical changelogs. This is done by prepending + # the towncrier output to the RST in memory, then parsing the RST by + # hand. We parse the entire changelog to create a changelog for each + # version which may be of use in some APIs. + + # Map specific headers to specific keys that'll be used eventually + # in variables. Things not listed here will get lowercased and formatted + # such that characters not [a-z0-9] will be replaced with an underscore. + keyword_versions = { + "Unreleased Changes": "unstable" + } + + # Only generate changelogs for things that have an RST document for f in os.listdir(CHANGELOG_DIR): if not f.endswith(".rst"): continue path = os.path.join(CHANGELOG_DIR, f) - name = f[:-4] + name = f[:-4] # take off ".rst" # If there's a directory with the same name, we'll try to generate # a towncrier changelog and prepend it to the general changelog. @@ -959,15 +979,39 @@ class MatrixUnits(Units): prev_line = line else: # have title, get body (stop on next title or EOF) if re.match("^[=]{3,}$", line.strip()): - # we added the title in the previous iteration, pop it - # then bail out. - changelog_lines.pop() - break + # we hit another title, so pop the last line of + # the changelog and record the changelog + new_title = changelog_lines.pop() + if name not in changelogs: + changelogs[name] = {} + if title_part in keyword_versions: + title_part = keyword_versions[title_part] + title_part = title_part.strip().replace("^[a-zA-Z0-9]", "_").lower() + changelog = "".join(changelog_lines) + changelogs[name][title_part] = changelog + + # reset for the next version + changelog_lines = [] + title_part = new_title.strip() + continue # Don't generate subheadings (we'll keep the title though) if re.match("^[-]{3,}$", line.strip()): continue + if line.strip().startswith(".. version: "): + # The changelog is directing us to use a different title + # for the changelog. + title_part = line.strip()[len(".. version: "):] + continue + if line.strip().startswith(".. "): + continue # skip comments changelog_lines.append(" " + line + '\n') - changelogs[name] = "".join(changelog_lines) + if len(changelog_lines) > 0 and title_part is not None: + if name not in changelogs: + changelogs[name] = {} + if title_part in keyword_versions: + title_part = keyword_versions[title_part] + changelog = "".join(changelog_lines) + changelogs[name][title_part.replace("^[a-zA-Z0-9]", "_").lower()] = changelog return changelogs diff --git a/specification/appendices/identifier_grammar.rst b/specification/appendices/identifier_grammar.rst index bb5f9297..496aba31 100644 --- a/specification/appendices/identifier_grammar.rst +++ b/specification/appendices/identifier_grammar.rst @@ -16,6 +16,12 @@ Identifier Grammar ------------------ +Some identifiers are specific to given room versions, please see the +`room specification`_ for more information. + +.. _`room specification`: ../rooms/latest.html + + Server Name ~~~~~~~~~~~ @@ -78,38 +84,6 @@ Some recommendations for a choice of server name follow: * The length of the complete server name should not exceed 230 characters. * Server names should not use upper-case characters. - -Room Versions -~~~~~~~~~~~~~ - -Room versions are used to change properties of rooms that may not be compatible -with other servers. For example, changing the rules for event authorization would -cause older servers to potentially end up in a split-brain situation due to them -not understanding the new rules. - -A room version is defined as a string of characters which MUST NOT exceed 32 -codepoints in length. Room versions MUST NOT be empty and SHOULD contain only -the characters ``a-z``, ``0-9``, ``.``, and ``-``. - -Room versions are not intended to be parsed and should be treated as opaque -identifiers. Room versions consisting only of the characters ``0-9`` and ``.`` -are reserved for future versions of the Matrix protocol. - -The complete grammar for a legal room version is:: - - room_version = 1*room_version_char - room_version_char = DIGIT - / %x61-7A ; a-z - / "-" / "." - -Examples of valid room versions are: - -* ``1`` (would be reserved by the Matrix protocol) -* ``1.2`` (would be reserved by the Matrix protocol) -* ``1.2-beta`` -* ``com.example.version`` - - Common Identifier Format ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -327,7 +301,7 @@ matrix.to navigation .. NOTE:: This namespacing is in place pending a ``matrix://`` (or similar) URI scheme. - This is **not** meant to be interpreted as an available web service - see + This is **not** meant to be interpreted as an available web service - see below for more details. Rooms, users, aliases, and groups may be represented as a "matrix.to" URI. @@ -343,7 +317,7 @@ in RFC 3986: The identifier may be a room ID, room alias, user ID, or group ID. The extra parameter is only used in the case of permalinks where an event ID is referenced. The matrix.to URI, when referenced, must always start with ``https://matrix.to/#/`` -followed by the identifier. +followed by the identifier. Clients should not rely on matrix.to URIs falling back to a web server if accessed and instead should perform some sort of action within the client. For example, if diff --git a/specification/rooms/intro.rst b/specification/rooms/intro.rst new file mode 100644 index 00000000..6231d066 --- /dev/null +++ b/specification/rooms/intro.rst @@ -0,0 +1,70 @@ +.. Copyright 2018 New Vector Ltd +.. +.. Licensed under the Apache License, Version 2.0 (the "License"); +.. you may not use this file except in compliance with the License. +.. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, +.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +.. See the License for the specific language governing permissions and +.. limitations under the License. + +Room Specification +================== + +.. contents:: Table of Contents +.. sectnum:: + +Rooms are central to how Matrix operates, and have strict rules for what +is allowed to be contained within them. Rooms can also have various +algorithms that handle different tasks, such as what to do when two or +more events collide in the underlying DAG. To allow rooms to be improved +upon through new algorithms or rules, "room versions" are employed to +manage a set of expectations for each room. + + +Room version grammar +-------------------- + +Room versions are used to change properties of rooms that may not be compatible +with other servers. For example, changing the rules for event authorization would +cause older servers to potentially end up in a split-brain situation due to them +not understanding the new rules. + +A room version is defined as a string of characters which MUST NOT exceed 32 +codepoints in length. Room versions MUST NOT be empty and SHOULD contain only +the characters ``a-z``, ``0-9``, ``.``, and ``-``. + +Room versions are not intended to be parsed and should be treated as opaque +identifiers. Room versions consisting only of the characters ``0-9`` and ``.`` +are reserved for future versions of the Matrix protocol. + +The complete grammar for a legal room version is:: + + room_version = 1*room_version_char + room_version_char = DIGIT + / %x61-7A ; a-z + / "-" / "." + +Examples of valid room versions are: + +* ``1`` (would be reserved by the Matrix protocol) +* ``1.2`` (would be reserved by the Matrix protocol) +* ``1.2-beta`` +* ``com.example.version`` + + +Other room versions +------------------- + +The available room versions are: + +* `Version 1 `_ - The current version of most rooms. +* `Version 2 `_ - Currently in development. + +.. Note: the 'unstable' version is commented out pending a real release of rooms v2 +.. See meta/releasing-rooms-v2.md +.. * `Unstable `_ - The upcoming version of the room specification. diff --git a/specification/rooms/unstable.rst b/specification/rooms/unstable.rst new file mode 100644 index 00000000..44261814 --- /dev/null +++ b/specification/rooms/unstable.rst @@ -0,0 +1,54 @@ +.. Copyright 2018 New Vector Ltd +.. +.. Licensed under the Apache License, Version 2.0 (the "License"); +.. you may not use this file except in compliance with the License. +.. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, +.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +.. See the License for the specific language governing permissions and +.. limitations under the License. + + +.. DEV NOTE: This is stubbed as a template and not actually used anywhere. +.. See v2.rst for the "unstable" room version, which is currently under +.. development. +.. +.. See meta/releasing-rooms-v2.md + + +.. Note: This document appended to the end of the intro, so this next line +.. appears under "Other Room Versions". + +.. Warning:: + + This is the specification for unreleased changes to rooms. The stability + of rooms using this specification cannot be guaranteed. + + +Changelog +--------- + +.. topic:: unstable +{{rooms_changelog_unstable}} + +This version of the specification is generated from +`matrix-doc `_ as of Git commit +`{{git_version}} `_. + +For the full historical changelog, see +https://github.com/matrix-org/matrix-doc/blob/master/changelogs/rooms.rst + + +Some Module +----------- + +Lorem ipsum dolor sit amet, consectetur adipiscing elit. In sit amet +eros turpis. Quisque commodo diam vel massa ultrices, vel egestas eros +dignissim. Sed sit amet lacus eget metus auctor malesuada at ut odio. +In turpis leo, viverra et mi porttitor, condimentum bibendum dolor. + +.. {-{versioned_test_definition}-} diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst new file mode 100644 index 00000000..6c6795a4 --- /dev/null +++ b/specification/rooms/v1.rst @@ -0,0 +1,112 @@ +.. Copyright 2018 New Vector Ltd +.. +.. Licensed under the Apache License, Version 2.0 (the "License"); +.. you may not use this file except in compliance with the License. +.. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, +.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +.. See the License for the specific language governing permissions and +.. limitations under the License. + + +.. Note: This document appended to the end of the intro, so this next line +.. appears under "Other Room Versions". + +This is the specification for **room version 1** (``"1"``). + +Changelog +--------- + +.. topic:: Room version 1 +{{rooms_changelog_v1}} + +This version of the specification is generated from +`matrix-doc `_ as of Git commit +`{{git_version}} `_. + +For the full historical changelog, see +https://github.com/matrix-org/matrix-doc/blob/master/changelogs/rooms.rst + + +Server implementation components +-------------------------------- + +.. WARNING:: + The information contained in this section is strictly for server implementors. + Applications which use the Client-Server API are generally unaffected by the + details contained here, and can safely ignore their presence. + + +The algorithms defined here should only apply to version 1 rooms. Other algorithms +may be used by other room versions, and as such servers should be aware of which +version room they are dealing with prior to executing a given algorithm. + +.. WARNING:: + Although room version 1 is the most popular room version, it is known to have + undesirable effects. Servers implementing support for room version 1 should be + aware that restrictions should be generally relaxed and be aware that inconsistencies + may occur until room version 2 is ready and adopted. + +State resolution +~~~~~~~~~~~~~~~~ + +.. WARNING:: + This section documents the state resolution algorithm as implemented by + Synapse as of December 2017 (and therefore the de-facto Matrix protocol). + However, this algorithm is known to have some problems. + +The room state :math:`S'(E)` after an event :math:`E` is defined in terms of +the room state :math:`S(E)` before :math:`E`, and depends on whether +:math:`E` is a state event or a message event: + +* If :math:`E` is a message event, then :math:`S'(E) = S(E)`. + +* If :math:`E` is a state event, then :math:`S'(E)` is :math:`S(E)`, except + that its entry corresponding to :math:`E`'s ``event_type`` and ``state_key`` + is replaced by :math:`E`'s ``event_id``. + +The room state :math:`S(E)` before :math:`E` is the *resolution* of the set of +states :math:`\{ S'(E'), S'(E''), … \}` consisting of the states after each of +:math:`E`'s ``prev_event``\s :math:`\{ E', E'', … \}`. + +The *resolution* of a set of states is defined as follows. The resolved state +is built up in a number of passes; here we use :math:`R` to refer to the +results of the resolution so far. + +* Start by setting :math:`R` to the union of the states to be resolved, + excluding any *conflicting* events. + +* First we resolve conflicts between ``m.room.power_levels`` events. If there + is no conflict, this step is skipped, otherwise: + + * Assemble all the ``m.room.power_levels`` events from the states to + be resolved into a list. + + * Sort the list by ascending ``depth`` then descending ``sha1(event_id)``. + + * Add the first event in the list to :math:`R`. + + * For each subsequent event in the list, check that the event would be + allowed by the `authorization rules`_ for a room in state :math:`R`. If the + event would be allowed, then update :math:`R` with the event and continue + with the next event in the list. If it would not be allowed, stop and + continue below with ``m.room.join_rules`` events. + +* Repeat the above process for conflicts between ``m.room.join_rules`` events. + +* Repeat the above process for conflicts between ``m.room.member`` events. + +* No other events affect the authorization rules, so for all other conflicts, + just pick the event with the highest depth and lowest ``sha1(event_id)`` that + passes authentication in :math:`R` and add it to :math:`R`. + +A *conflict* occurs between states where those states have different +``event_ids`` for the same ``(state_type, state_key)``. The events thus +affected are said to be *conflicting* events. + + +.. _`authorization rules`: ../server_server/unstable.html#authorization-rules diff --git a/specification/rooms/v2.rst b/specification/rooms/v2.rst new file mode 100644 index 00000000..a000f056 --- /dev/null +++ b/specification/rooms/v2.rst @@ -0,0 +1,181 @@ +.. Copyright 2018 New Vector Ltd +.. +.. Licensed under the Apache License, Version 2.0 (the "License"); +.. you may not use this file except in compliance with the License. +.. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, +.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +.. See the License for the specific language governing permissions and +.. limitations under the License. + + +.. Note: This document appended to the end of the intro, so this next line +.. appears under "Other Room Versions". + +This is the specification for **room version 2** (``"2"``). + +.. Warning:: + + Room version 2 is under development and cannot be relied on in production + environments. + + +Changelog +--------- + +.. topic:: Room version 2 +{{rooms_changelog_v2}} + +This version of the specification is generated from +`matrix-doc `_ as of Git commit +`{{git_version}} `_. + +For the full historical changelog, see +https://github.com/matrix-org/matrix-doc/blob/master/changelogs/rooms.rst + + +Server implementation components +-------------------------------- + +.. WARNING:: + The information contained in this section is strictly for server implementors. + Applications which use the Client-Server API are generally unaffected by the + details contained here, and can safely ignore their presence. + + +The algorithms defined here should only apply to version 2 rooms. Other algorithms +may be used by other room versions, and as such servers should be aware of which +version room they are dealing with prior to executing a given algorithm. + + +State resolution +~~~~~~~~~~~~~~~~ + +The room state :math:`S'(E)` after an event :math:`E` is defined in terms of +the room state :math:`S(E)` before :math:`E`, and depends on whether +:math:`E` is a state event or a message event: + +* If :math:`E` is a message event, then :math:`S'(E) = S(E)`. + +* If :math:`E` is a state event, then :math:`S'(E)` is :math:`S(E)`, except + that its entry corresponding to :math:`E`'s ``event_type`` and ``state_key`` + is replaced by :math:`E`'s ``event_id``. + +The room state :math:`S(E)` before :math:`E` is the *resolution* of the set of +states :math:`\{ S'(E_1), S'(E_2), … \}` consisting of the states after each of +:math:`E`'s ``prev_event``\s :math:`\{ E_1, E_2, … \}`, where the resolution of +a set of states is given in the algorithm below. + +Definitions ++++++++++++ + +The state resolution algorithm for version 2 rooms uses the following +definitions, given the set of room states :math:`\{ S_1, S_2, \ldots \}`: + +Power events + A *power event* is a state event with type ``m.room.power_levels`` or + ``m.room.join_rules``, or a state event with type ``m.room.member`` where the + ``membership`` is ``leave`` or ``ban`` and the ``sender`` does not match the + ``state_key``. The idea behind this is that power events are events that have + may remove someone's ability to do something in the room. + +Unconflicted state map and conflicted state set + The *unconflicted state map* is the state where the value of each key exists + and is the same in each state :math:`S_i`. The *conflicted state set* is the + set of all other state events. Note that the unconflicted state map only has + one event per ``(event_type, state_key)``, whereas the conflicted state set + may have multiple events. + +Auth difference + The *auth difference* is calculated by first calculating the full auth chain + for each state :math:`S_i`, that is the union of the auth chains for each + event in :math:`S_i`, and then taking every event that doesn't appear in + every auth chain. If :math:`C_i` is the full auth chain of :math:`S_i`, then + the auth difference is :math:`\cup C_i - \cap C_i`. + +Full conflicted set + The *full conflicted set* is the union of the conflicted state set and the + auth difference. + +Reverse topological power ordering + The *reverse topological power ordering* of a set of events is the + lexicographically smallest topological ordering based on the DAG formed by + auth events. The reverse topological power ordering is ordered from earliest + event to latest. For comparing two topological orderings to determine which + is the lexicographically smallest, the following comparison relation on + events is used: for events :math:`x` and :math:`y`, :math:`x Date: Mon, 7 Jan 2019 13:09:21 -0700 Subject: [PATCH 02/14] Incorporate MSC1693 This is largely blatant copy/paste from the MSC with some formatting done to tidy it up a bit. --- specification/rooms/v2.rst | 43 +++++++++++++++++++++++++++++++++++++- 1 file changed, 42 insertions(+), 1 deletion(-) diff --git a/specification/rooms/v2.rst b/specification/rooms/v2.rst index a000f056..b04f94b4 100644 --- a/specification/rooms/v2.rst +++ b/specification/rooms/v2.rst @@ -155,7 +155,8 @@ Iterative auth checks state event is not allowed by the authorization rules, then the event is ignored. If a ``(event_type, state_key)`` key that is required for checking the authorization rules is not present in the state, then the appropriate - state event from the event's ``auth_events`` is used. + state event from the event's ``auth_events`` is used if the auth event is + not rejected. Algorithm +++++++++ @@ -179,3 +180,43 @@ The *resolution* of a set of states is obtained as follows: .. _`authorization rules`: ../server_server/unstable.html#authorization-rules + +Rejected events ++++++++++++++++ + +Events that have been rejected due to failing auth based on the state at the +event (rather than based on their auth chain) are handled as usual by the +algorithm, unless otherwise specified. + +Note that no events rejected due to failure to auth against their auth chain +should appear in the process, as they should not appear in state (the algorithm +only uses events that appear in either the state sets or in the auth chain of +the events in the state sets). + +.. admonition:: Rationale + + This helps ensure that different servers' view of state is more likely to + converge, since rejection state of an event may be different. This can happen if + a third server gives an incorrect version of the state when a server joins a + room via it (either due to being faulty or malicious). Convergence of state is a + desirable property as it ensures that all users in the room have a (mostly) + consistent view of the state of the room. If the view of the state on different + servers diverges it can lead to bifurcation of the room due to e.g. servers + disagreeing on who is in the room. + + Intuitively, using rejected events feels dangerous, however: + + 1. Servers cannot arbitrarily make up state, since they still need to pass the + auth checks based on the event's auth chain (e.g. they can't grant themselves + power levels if they didn't have them before). + 2. For a previously rejected event to pass auth there must be a set of state + that allows said event. A malicious server could therefore produce a + fork where it claims the state is that particular set of state, duplicate the + rejected event to point to that fork, and send the event. The + duplicated event would then pass the auth checks. Ignoring rejected events + would therefore not eliminate any potential attack vectors. + + +Rejected auth events are deliberately excluded from use in the iterative auth +checks, as auth events aren't re-authed (although non-auth events are) during +the iterative auth checks. From 71e6321f4d448d840bfb6634d3ee381876d35a97 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 16 Jan 2019 16:57:45 -0700 Subject: [PATCH 03/14] Rework how room versions are represented Versions are actually on a scale of recommendations, and are expected to be created as needed. The scale presented here (develop/beta/default/recommended/mandatory) is a more wordy version of what was previously discussed/intended for room versions - the labels aren't final and may be changed. --- changelogs/rooms/newsfragments/.gitignore | 1 - changelogs/rooms/pyproject.toml | 30 -------- meta/releasing-rooms-v2.md | 38 --------- scripts/gendoc.py | 5 -- scripts/templating/matrix_templates/units.py | 5 -- specification/index.rst | 81 ++++++++++++++++++++ specification/rooms/intro.rst | 70 ----------------- specification/rooms/unstable.rst | 54 ------------- specification/rooms/v1.rst | 24 ++---- specification/rooms/v2.rst | 30 ++------ specification/targets.yaml | 21 ----- 11 files changed, 91 insertions(+), 268 deletions(-) delete mode 100644 changelogs/rooms/newsfragments/.gitignore delete mode 100644 changelogs/rooms/pyproject.toml delete mode 100644 meta/releasing-rooms-v2.md delete mode 100644 specification/rooms/intro.rst delete mode 100644 specification/rooms/unstable.rst diff --git a/changelogs/rooms/newsfragments/.gitignore b/changelogs/rooms/newsfragments/.gitignore deleted file mode 100644 index b722e9e1..00000000 --- a/changelogs/rooms/newsfragments/.gitignore +++ /dev/null @@ -1 +0,0 @@ -!.gitignore \ No newline at end of file diff --git a/changelogs/rooms/pyproject.toml b/changelogs/rooms/pyproject.toml deleted file mode 100644 index b56e19a9..00000000 --- a/changelogs/rooms/pyproject.toml +++ /dev/null @@ -1,30 +0,0 @@ -[tool.towncrier] - filename = "../rooms.rst" - directory = "newsfragments" - issue_format = "`#{issue} `_" - title_format = "{version}" - - [[tool.towncrier.type]] - directory = "breaking" - name = "Breaking Changes" - showcontent = true - - [[tool.towncrier.type]] - directory = "deprecation" - name = "Deprecations" - showcontent = true - - [[tool.towncrier.type]] - directory = "new" - name = "New Endpoints" - showcontent = true - - [[tool.towncrier.type]] - directory = "feature" - name = "Backwards Compatible Changes" - showcontent = true - - [[tool.towncrier.type]] - directory = "clarification" - name = "Spec Clarifications" - showcontent = true diff --git a/meta/releasing-rooms-v2.md b/meta/releasing-rooms-v2.md deleted file mode 100644 index 80277749..00000000 --- a/meta/releasing-rooms-v2.md +++ /dev/null @@ -1,38 +0,0 @@ -# How to release Room Version 2 - -Room versions are a bit special for the release given they have been -introduced at v2 rather than ever getting a v1 release. Additionally, -room versions have different release requirements due to the in-project -versioning of documents rather than relying on matrix.org to maintain -old generated output of specifications. - -As of writing, the project is structured to support 3 logical versions -for rooms: v1, v2, and unstable. Unstable is currently pointed at v2 -in order to aid development. After v2 is released, unstable may wish -to be left as an independent version or similarly be pointed at a "v3" -document. - -Due to room versions being versioned in-project, the generated output -from a release is not to be sent off to matrix-doc/matrix.org. Instead, -in `gendoc.py` the default value for `--room_version` should be set to -the current release (`v2`, for example) so the index renders the right -edition in the table. - -After editing `gendoc.py`, the changelog should be generated according -to the towncrier instructions. You may need to fix the `.. version: v2` -comment located in the `rooms.rst` changelog to be just underneath the -title instead of at the end of the section. - -The `targets.yaml` file needs to be set up to point unstable to the -right set of files. Ensure that `unstable.rst` is referenced instead -of `v2.rst`, and that `unstable.rst` has appropriate contents. - -Finally, in the `intro.rst` for room versions, re-add unstable to the -list of available versions. It is currently commented out to avoid -confusing people, so it should be possible to uncomment it and put it -back into the list. - -From there, the standard process of using a release branch, tagging it, -and announcing it to the world should be followed. If required, the -various other APIs should be updated to better support the new room -version. diff --git a/scripts/gendoc.py b/scripts/gendoc.py index 0455c90a..72e3047c 100755 --- a/scripts/gendoc.py +++ b/scripts/gendoc.py @@ -529,10 +529,6 @@ if __name__ == '__main__': "--identity_release", "-i", action="store", default="unstable", help="The identity service release tag to generate, e.g. r1.2" ) - parser.add_argument( - "--room_version", "-r", action="store", default="unstable", - help="The current room version to advertise, e.g. v2" - ) parser.add_argument( "--list_targets", action="store_true", help="Do not update the specification. Instead print a list of targets.", @@ -559,7 +555,6 @@ if __name__ == '__main__': "%APPSERVICE_RELEASE_LABEL%": args.appservice_release, "%IDENTITY_RELEASE_LABEL%": args.identity_release, "%PUSH_GATEWAY_RELEASE_LABEL%": args.push_gateway_release, - "%CURRENT_ROOM_VERSION%": args.room_version, } exit (main(args.target or ["all"], args.dest, args.nodelete, substitutions)) diff --git a/scripts/templating/matrix_templates/units.py b/scripts/templating/matrix_templates/units.py index 666434a5..721501ff 100644 --- a/scripts/templating/matrix_templates/units.py +++ b/scripts/templating/matrix_templates/units.py @@ -757,7 +757,6 @@ class MatrixUnits(Units): is_ver = substitutions.get("%IDENTITY_RELEASE_LABEL%", "unstable") as_ver = substitutions.get("%APPSERVICE_RELEASE_LABEL%", "unstable") push_gw_ver = substitutions.get("%PUSH_GATEWAY_RELEASE_LABEL%", "unstable") - room_ver = substitutions.get("%CURRENT_ROOM_VERSION%", "unstable") # we abuse the typetable to return this info to the templates return TypeTable(rows=[ @@ -781,10 +780,6 @@ class MatrixUnits(Units): "`Push Gateway API `_", push_gw_ver, "Push notifications for Matrix events", - ), TypeTableRow( - "`Rooms `_", - room_ver, - "Specification for behaviour of rooms, such as event formats", ), ]) diff --git a/specification/index.rst b/specification/index.rst index bb918739..2f17ec29 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -418,6 +418,87 @@ dedicated API. The API is symmetrical to managing Profile data. Would it really be overengineered to use the same API for both profile & private user data, but with different ACLs? + +Room Versions +------------- + +Rooms are central to how Matrix operates, and have strict rules for what +is allowed to be contained within them. Rooms can also have various +algorithms that handle different tasks, such as what to do when two or +more events collide in the underlying DAG. To allow rooms to be improved +upon through new algorithms or rules, "room versions" are employed to +manage a set of expectations for each room. New room versions are assigned +as needed. + +Room version grammar +~~~~~~~~~~~~~~~~~~~~ + +Room versions are used to change properties of rooms that may not be compatible +with other servers. For example, changing the rules for event authorization would +cause older servers to potentially end up in a split-brain situation due to them +not understanding the new rules. + +A room version is defined as a string of characters which MUST NOT exceed 32 +codepoints in length. Room versions MUST NOT be empty and SHOULD contain only +the characters ``a-z``, ``0-9``, ``.``, and ``-``. + +Room versions are not intended to be parsed and should be treated as opaque +identifiers. Room versions consisting only of the characters ``0-9`` and ``.`` +are reserved for future versions of the Matrix protocol. + +The complete grammar for a legal room version is:: + + room_version = 1*room_version_char + room_version_char = DIGIT + / %x61-7A ; a-z + / "-" / "." + +Examples of valid room versions are: + +* ``1`` (would be reserved by the Matrix protocol) +* ``1.2`` (would be reserved by the Matrix protocol) +* ``1.2-beta`` +* ``com.example.version`` + +Recommended room versions +~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are varying degrees of what "recommended" means for a given room version. +Currently, this is split into 5 categories: + +* **Development**: This is the default state for all room versions. When in this + state, a room version is documented but not recommended for use outside of a + development environment. These versions are not production-ready. +* **Beta**: Versions in this state are not intended for wide-spread usage but + should be stable enough if a room requires the feature(s) introduced within. + Rooms will need to opt-in to these versions and should not be promoted to + upgrade. +* **Default**: Exactly 1 room version will be in this category. The version under + this category should be used when creating rooms (unless another version is + requested by the user). Servers may wish to promote rooms to opt-in to this + version. +* **Recommended**: Exactly 1 room version will be in this category as well. The + version here should be heavily promoted by servers for rooms to opt-in to using. + This version is often going to be the same as the Default version. +* **Mandatory**: Servers are required to implement versions in this category. When + a version is flagged as mandatory, additional rules may apply to rooms. For example, + servers may be required to stop supporting another room version and automatically + upgrade all affected rooms. + +With the above categories, the following applies: + +* Servers MUST have Room Version 1 as the Default room version. +* Servers MUST have Room Version 1 as the Recommended room version. +* Servers MUST implement Room Version 1 as a Mandatory room version. + +Complete list of room versions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The available room versions are: + +* `Version 1 `_ - The current version of most rooms. +* `Version 2 `_ - **Beta**. Implements State Resolution Version 2. + Specification Versions ---------------------- diff --git a/specification/rooms/intro.rst b/specification/rooms/intro.rst deleted file mode 100644 index 6231d066..00000000 --- a/specification/rooms/intro.rst +++ /dev/null @@ -1,70 +0,0 @@ -.. Copyright 2018 New Vector Ltd -.. -.. Licensed under the Apache License, Version 2.0 (the "License"); -.. you may not use this file except in compliance with the License. -.. You may obtain a copy of the License at -.. -.. http://www.apache.org/licenses/LICENSE-2.0 -.. -.. Unless required by applicable law or agreed to in writing, software -.. distributed under the License is distributed on an "AS IS" BASIS, -.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -.. See the License for the specific language governing permissions and -.. limitations under the License. - -Room Specification -================== - -.. contents:: Table of Contents -.. sectnum:: - -Rooms are central to how Matrix operates, and have strict rules for what -is allowed to be contained within them. Rooms can also have various -algorithms that handle different tasks, such as what to do when two or -more events collide in the underlying DAG. To allow rooms to be improved -upon through new algorithms or rules, "room versions" are employed to -manage a set of expectations for each room. - - -Room version grammar --------------------- - -Room versions are used to change properties of rooms that may not be compatible -with other servers. For example, changing the rules for event authorization would -cause older servers to potentially end up in a split-brain situation due to them -not understanding the new rules. - -A room version is defined as a string of characters which MUST NOT exceed 32 -codepoints in length. Room versions MUST NOT be empty and SHOULD contain only -the characters ``a-z``, ``0-9``, ``.``, and ``-``. - -Room versions are not intended to be parsed and should be treated as opaque -identifiers. Room versions consisting only of the characters ``0-9`` and ``.`` -are reserved for future versions of the Matrix protocol. - -The complete grammar for a legal room version is:: - - room_version = 1*room_version_char - room_version_char = DIGIT - / %x61-7A ; a-z - / "-" / "." - -Examples of valid room versions are: - -* ``1`` (would be reserved by the Matrix protocol) -* ``1.2`` (would be reserved by the Matrix protocol) -* ``1.2-beta`` -* ``com.example.version`` - - -Other room versions -------------------- - -The available room versions are: - -* `Version 1 `_ - The current version of most rooms. -* `Version 2 `_ - Currently in development. - -.. Note: the 'unstable' version is commented out pending a real release of rooms v2 -.. See meta/releasing-rooms-v2.md -.. * `Unstable `_ - The upcoming version of the room specification. diff --git a/specification/rooms/unstable.rst b/specification/rooms/unstable.rst deleted file mode 100644 index 44261814..00000000 --- a/specification/rooms/unstable.rst +++ /dev/null @@ -1,54 +0,0 @@ -.. Copyright 2018 New Vector Ltd -.. -.. Licensed under the Apache License, Version 2.0 (the "License"); -.. you may not use this file except in compliance with the License. -.. You may obtain a copy of the License at -.. -.. http://www.apache.org/licenses/LICENSE-2.0 -.. -.. Unless required by applicable law or agreed to in writing, software -.. distributed under the License is distributed on an "AS IS" BASIS, -.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -.. See the License for the specific language governing permissions and -.. limitations under the License. - - -.. DEV NOTE: This is stubbed as a template and not actually used anywhere. -.. See v2.rst for the "unstable" room version, which is currently under -.. development. -.. -.. See meta/releasing-rooms-v2.md - - -.. Note: This document appended to the end of the intro, so this next line -.. appears under "Other Room Versions". - -.. Warning:: - - This is the specification for unreleased changes to rooms. The stability - of rooms using this specification cannot be guaranteed. - - -Changelog ---------- - -.. topic:: unstable -{{rooms_changelog_unstable}} - -This version of the specification is generated from -`matrix-doc `_ as of Git commit -`{{git_version}} `_. - -For the full historical changelog, see -https://github.com/matrix-org/matrix-doc/blob/master/changelogs/rooms.rst - - -Some Module ------------ - -Lorem ipsum dolor sit amet, consectetur adipiscing elit. In sit amet -eros turpis. Quisque commodo diam vel massa ultrices, vel egestas eros -dignissim. Sed sit amet lacus eget metus auctor malesuada at ut odio. -In turpis leo, viverra et mi porttitor, condimentum bibendum dolor. - -.. {-{versioned_test_definition}-} diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst index 6c6795a4..c263a108 100644 --- a/specification/rooms/v1.rst +++ b/specification/rooms/v1.rst @@ -1,4 +1,4 @@ -.. Copyright 2018 New Vector Ltd +.. Copyright 2018-2019 New Vector Ltd .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. @@ -12,25 +12,11 @@ .. See the License for the specific language governing permissions and .. limitations under the License. +Room Version 1 +============== -.. Note: This document appended to the end of the intro, so this next line -.. appears under "Other Room Versions". - -This is the specification for **room version 1** (``"1"``). - -Changelog ---------- - -.. topic:: Room version 1 -{{rooms_changelog_v1}} - -This version of the specification is generated from -`matrix-doc `_ as of Git commit -`{{git_version}} `_. - -For the full historical changelog, see -https://github.com/matrix-org/matrix-doc/blob/master/changelogs/rooms.rst - +This room version is the first ever version for rooms, and contains the building +blocks for other room versions. Server implementation components -------------------------------- diff --git a/specification/rooms/v2.rst b/specification/rooms/v2.rst index b04f94b4..eef03497 100644 --- a/specification/rooms/v2.rst +++ b/specification/rooms/v2.rst @@ -1,4 +1,4 @@ -.. Copyright 2018 New Vector Ltd +.. Copyright 2018-2019 New Vector Ltd .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. @@ -12,31 +12,11 @@ .. See the License for the specific language governing permissions and .. limitations under the License. +Room Version 2 +============== -.. Note: This document appended to the end of the intro, so this next line -.. appears under "Other Room Versions". - -This is the specification for **room version 2** (``"2"``). - -.. Warning:: - - Room version 2 is under development and cannot be relied on in production - environments. - - -Changelog ---------- - -.. topic:: Room version 2 -{{rooms_changelog_v2}} - -This version of the specification is generated from -`matrix-doc `_ as of Git commit -`{{git_version}} `_. - -For the full historical changelog, see -https://github.com/matrix-org/matrix-doc/blob/master/changelogs/rooms.rst - +This room version builds off of `version 1 `_ with an improved state +resolution algorithm. Server implementation components -------------------------------- diff --git a/specification/targets.yaml b/specification/targets.yaml index b940148d..e966f66b 100644 --- a/specification/targets.yaml +++ b/specification/targets.yaml @@ -26,35 +26,14 @@ targets: files: - push_gateway.rst version_label: "%PUSH_GATEWAY_RELEASE_LABEL%" - rooms: - files: - - rooms/intro.rst - - # TODO: Switch this back to unstable.rst after releasing rooms v2 - # We temporarily do this so that "unstable" points to the in-dev - # version, however we may also want to hardlink to v2 in places and - # thus we maintain a v2 version of the same doc. - # - # See meta/releasing-rooms-v2.md - - #- rooms/unstable.rst - - rooms/v2.rst - version_label: unstable rooms@v1: # this is translated to be rooms/v1.html files: - - rooms/intro.rst - rooms/v1.rst version_label: v1 rooms@v2: # this is translated to be rooms/v2.html files: - - rooms/intro.rst - rooms/v2.rst version_label: v2 - rooms@latest: # this is translated to be rooms/latest.html - files: - - rooms/intro.rst - - rooms/v1.rst # TODO: Find a better way to link to the v2 spec - version_label: latest appendices: files: - appendices.rst From 3b47a5924b2a67f817aa14ae8ec03e797f09e819 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 16 Jan 2019 17:02:41 -0700 Subject: [PATCH 04/14] Remove extraneous changelog --- changelogs/rooms.rst | 11 ----------- 1 file changed, 11 deletions(-) delete mode 100644 changelogs/rooms.rst diff --git a/changelogs/rooms.rst b/changelogs/rooms.rst deleted file mode 100644 index 55a16790..00000000 --- a/changelogs/rooms.rst +++ /dev/null @@ -1,11 +0,0 @@ -.. version: v2 - -.. Note: We set the version as "version 2" so that we can maintain a specific version -.. variable in the changelog. We already know the next version is going to be v2, so -.. this makes it easier to copy/paste unstable.rst to v2.rst - -Room version 1 -============== -.. version: v1 - -This is the first iteration of rooms in Matrix. From 166d4ada8613a9e786c6d1b0b124b5bd6a02732f Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 16 Jan 2019 17:05:57 -0700 Subject: [PATCH 05/14] Fix room versions reference in appendices & s2s spec --- specification/appendices/identifier_grammar.rst | 6 +++--- specification/server_server_api.rst | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/specification/appendices/identifier_grammar.rst b/specification/appendices/identifier_grammar.rst index 496aba31..b4678988 100644 --- a/specification/appendices/identifier_grammar.rst +++ b/specification/appendices/identifier_grammar.rst @@ -16,10 +16,10 @@ Identifier Grammar ------------------ -Some identifiers are specific to given room versions, please see the -`room specification`_ for more information. +Some identifiers are specific to given room versions, please refer to the +`room versions specification`_ for more information. -.. _`room specification`: ../rooms/latest.html +.. _`room versions specification`: ../index.html#room-versions Server Name diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 05e6617b..8645888b 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -1318,4 +1318,4 @@ Example code .. _`Checking for a signature`: ../appendices.html#checking-for-a-signature .. _`Device Management module`: ../client_server/%CLIENT_RELEASE_LABEL%.html#device-management .. _`End-to-End Encryption module`: ../client_server/%CLIENT_RELEASE_LABEL%.html#end-to-end-encryption -.. _`room version specification`: ../rooms/latest.html +.. _`room version specification`: ../index.html#room-versions From a6f5d015867eff953e8f6690c78d7ce1c9a0a729 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 17 Jan 2019 14:39:06 -0700 Subject: [PATCH 06/14] Clarify that servers don't have to implement development/beta versions --- specification/index.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/specification/index.rst b/specification/index.rst index 2f17ec29..ff51f7c0 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -468,11 +468,12 @@ Currently, this is split into 5 categories: * **Development**: This is the default state for all room versions. When in this state, a room version is documented but not recommended for use outside of a - development environment. These versions are not production-ready. + development environment. These versions are not production-ready and servers + are not required to implement these versions. * **Beta**: Versions in this state are not intended for wide-spread usage but should be stable enough if a room requires the feature(s) introduced within. Rooms will need to opt-in to these versions and should not be promoted to - upgrade. + upgrade. Servers additionally do not have to implement these versions. * **Default**: Exactly 1 room version will be in this category. The version under this category should be used when creating rooms (unless another version is requested by the user). Servers may wish to promote rooms to opt-in to this From 96d754f42904b62da0eb6bba17263e563b346e51 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 17 Jan 2019 14:43:11 -0700 Subject: [PATCH 07/14] promote -> prompt --- specification/index.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specification/index.rst b/specification/index.rst index ff51f7c0..f4abf292 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -472,14 +472,14 @@ Currently, this is split into 5 categories: are not required to implement these versions. * **Beta**: Versions in this state are not intended for wide-spread usage but should be stable enough if a room requires the feature(s) introduced within. - Rooms will need to opt-in to these versions and should not be promoted to + Rooms will need to opt-in to these versions and should not be prompted to upgrade. Servers additionally do not have to implement these versions. * **Default**: Exactly 1 room version will be in this category. The version under this category should be used when creating rooms (unless another version is - requested by the user). Servers may wish to promote rooms to opt-in to this + requested by the user). Servers may wish to prompt rooms to opt-in to this version. -* **Recommended**: Exactly 1 room version will be in this category as well. The - version here should be heavily promoted by servers for rooms to opt-in to using. +* **Recommended**: Exactly 1 room version will be in this category as well. Servers + should prompt as strongly as possible for rooms to opt-in to upgrade to this version. This version is often going to be the same as the Default version. * **Mandatory**: Servers are required to implement versions in this category. When a version is flagged as mandatory, additional rules may apply to rooms. For example, From 0dde2489b666be2c6448a785e27acb15e13f8581 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 17 Jan 2019 14:48:42 -0700 Subject: [PATCH 08/14] Clarify what a Mandatory room version is --- specification/index.rst | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/specification/index.rst b/specification/index.rst index f4abf292..cdd56cb2 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -481,10 +481,13 @@ Currently, this is split into 5 categories: * **Recommended**: Exactly 1 room version will be in this category as well. Servers should prompt as strongly as possible for rooms to opt-in to upgrade to this version. This version is often going to be the same as the Default version. -* **Mandatory**: Servers are required to implement versions in this category. When - a version is flagged as mandatory, additional rules may apply to rooms. For example, - servers may be required to stop supporting another room version and automatically - upgrade all affected rooms. +* **Mandatory**: Servers are required to implement versions in this category, + regardless as to how they are otherwise categorized. This allows for a Beta room + version to be mandatorily implemented by all servers in extremely rare circumstances, + as an example. Being a Mandatory room version does not imply that it is Recommended + or a Default version, just that the server needs to support it. Additional rules + may apply to room versions which are Mandatory, such as forcing servers to upgrade + all known rooms to a particular version where possible. With the above categories, the following applies: From 19e94815f9bc65bacca00e9c3c078d983a941b91 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 17 Jan 2019 15:13:01 -0700 Subject: [PATCH 09/14] Try and improve the understanding of room versions --- specification/index.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/specification/index.rst b/specification/index.rst index cdd56cb2..ea4026f0 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -430,6 +430,13 @@ upon through new algorithms or rules, "room versions" are employed to manage a set of expectations for each room. New room versions are assigned as needed. +There is no implicit ordering or hierarchy to room versions, and their principles +are immutable once placed in the specification. Although there is a recommended +set of versions, some rooms may benefit from features introduced by other versions. +Rooms move between different versions by "upgrading" to the desired version. Due +to versions not being ordered or hierarchical, this means a room can "upgrade" to +version 1 from version 2, if it so desired. + Room version grammar ~~~~~~~~~~~~~~~~~~~~ From ba37f2d311aaba01dfa395195f14394e1c91f633 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 17 Jan 2019 16:19:25 -0700 Subject: [PATCH 10/14] prompt->advertise --- specification/index.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/specification/index.rst b/specification/index.rst index ea4026f0..67a0fa96 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -479,15 +479,15 @@ Currently, this is split into 5 categories: are not required to implement these versions. * **Beta**: Versions in this state are not intended for wide-spread usage but should be stable enough if a room requires the feature(s) introduced within. - Rooms will need to opt-in to these versions and should not be prompted to - upgrade. Servers additionally do not have to implement these versions. + Rooms may opt-in to these versions on their own, but should not be asked to + upgrade automatically. Servers do not have to implement these versions. * **Default**: Exactly 1 room version will be in this category. The version under this category should be used when creating rooms (unless another version is - requested by the user). Servers may wish to prompt rooms to opt-in to this + requested by the user). Servers may wish to advertise that rooms opt-in to this version. * **Recommended**: Exactly 1 room version will be in this category as well. Servers - should prompt as strongly as possible for rooms to opt-in to upgrade to this version. - This version is often going to be the same as the Default version. + should advertise as strongly as possible for rooms to opt-in to upgrade to this + version. This version is often going to be the same as the Default version. * **Mandatory**: Servers are required to implement versions in this category, regardless as to how they are otherwise categorized. This allows for a Beta room version to be mandatorily implemented by all servers in extremely rare circumstances, From ebe887d9314b3533b80956b0ef803e3b7e319be2 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Fri, 18 Jan 2019 09:56:04 -0700 Subject: [PATCH 11/14] Grammar Co-Authored-By: turt2live --- specification/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/index.rst b/specification/index.rst index 67a0fa96..0b024c9e 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -435,7 +435,7 @@ are immutable once placed in the specification. Although there is a recommended set of versions, some rooms may benefit from features introduced by other versions. Rooms move between different versions by "upgrading" to the desired version. Due to versions not being ordered or hierarchical, this means a room can "upgrade" to -version 1 from version 2, if it so desired. +version 1 from version 2, if it is so desired. Room version grammar ~~~~~~~~~~~~~~~~~~~~ From bd5e760a0d1a8fe3498fd77e27b948504762bef6 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 22 Jan 2019 18:02:21 -0700 Subject: [PATCH 12/14] Simplify the description for room versions Instead of trying to describe maturity, stability, and recommendedness in one list we should describe what is "safe" and "unsafe" to use. The default version is just something that servers should use, and is normally going to be stable. --- specification/index.rst | 49 ++++++++++------------------------------- 1 file changed, 12 insertions(+), 37 deletions(-) diff --git a/specification/index.rst b/specification/index.rst index 0b024c9e..e5f01abf 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -467,48 +467,23 @@ Examples of valid room versions are: * ``1.2-beta`` * ``com.example.version`` -Recommended room versions -~~~~~~~~~~~~~~~~~~~~~~~~~ - -There are varying degrees of what "recommended" means for a given room version. -Currently, this is split into 5 categories: - -* **Development**: This is the default state for all room versions. When in this - state, a room version is documented but not recommended for use outside of a - development environment. These versions are not production-ready and servers - are not required to implement these versions. -* **Beta**: Versions in this state are not intended for wide-spread usage but - should be stable enough if a room requires the feature(s) introduced within. - Rooms may opt-in to these versions on their own, but should not be asked to - upgrade automatically. Servers do not have to implement these versions. -* **Default**: Exactly 1 room version will be in this category. The version under - this category should be used when creating rooms (unless another version is - requested by the user). Servers may wish to advertise that rooms opt-in to this - version. -* **Recommended**: Exactly 1 room version will be in this category as well. Servers - should advertise as strongly as possible for rooms to opt-in to upgrade to this - version. This version is often going to be the same as the Default version. -* **Mandatory**: Servers are required to implement versions in this category, - regardless as to how they are otherwise categorized. This allows for a Beta room - version to be mandatorily implemented by all servers in extremely rare circumstances, - as an example. Being a Mandatory room version does not imply that it is Recommended - or a Default version, just that the server needs to support it. Additional rules - may apply to room versions which are Mandatory, such as forcing servers to upgrade - all known rooms to a particular version where possible. - -With the above categories, the following applies: - -* Servers MUST have Room Version 1 as the Default room version. -* Servers MUST have Room Version 1 as the Recommended room version. -* Servers MUST implement Room Version 1 as a Mandatory room version. - Complete list of room versions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Room versions are divided into two distinct groups: stable and unstable. Stable +room versions may be used by rooms safely. Unstable room versions are everything +else which is either not listed in the specification or flagged as unstable for +some other reason. Versions can switch between stable and unstable periodically +for a variety of reasons, including discovered security vulnerabilites and age. + +Clients should not ask room administrators to upgrade their rooms if the room is +running a stable version. Servers SHOULD use room version 1 as the default room +version when creating new rooms. + The available room versions are: -* `Version 1 `_ - The current version of most rooms. -* `Version 2 `_ - **Beta**. Implements State Resolution Version 2. +* `Version 1 `_ - **Stable**. The current version of most rooms. +* `Version 2 `_ - **Stable**. Implements State Resolution Version 2. Specification Versions ---------------------- From 5cafcd103fbdbce8122fbe8f67c0d65d08ac28ac Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 22 Jan 2019 21:43:32 -0700 Subject: [PATCH 13/14] Fix copyright > Since this is a copy-and-paste of old text, I think the copyright year should match when the original text was written, which according to git was 2017. Co-Authored-By: turt2live --- specification/rooms/v1.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst index c263a108..207fe8bc 100644 --- a/specification/rooms/v1.rst +++ b/specification/rooms/v1.rst @@ -1,4 +1,4 @@ -.. Copyright 2018-2019 New Vector Ltd +.. Copyright 2017,2019 New Vector Ltd .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. From 061f59547a393ab2d1d52eacf76368bc8c3800e6 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 23 Jan 2019 09:10:14 -0700 Subject: [PATCH 14/14] Minor wording changes --- specification/index.rst | 8 ++++---- specification/rooms/v1.rst | 4 ++-- specification/rooms/v2.rst | 4 ++-- specification/server_server_api.rst | 5 ++--- 4 files changed, 10 insertions(+), 11 deletions(-) diff --git a/specification/index.rst b/specification/index.rst index e5f01abf..0a2224d2 100644 --- a/specification/index.rst +++ b/specification/index.rst @@ -434,16 +434,16 @@ There is no implicit ordering or hierarchy to room versions, and their principle are immutable once placed in the specification. Although there is a recommended set of versions, some rooms may benefit from features introduced by other versions. Rooms move between different versions by "upgrading" to the desired version. Due -to versions not being ordered or hierarchical, this means a room can "upgrade" to -version 1 from version 2, if it is so desired. +to versions not being ordered or hierarchical, this means a room can "upgrade" +from version 2 to version 1, if it is so desired. Room version grammar ~~~~~~~~~~~~~~~~~~~~ Room versions are used to change properties of rooms that may not be compatible with other servers. For example, changing the rules for event authorization would -cause older servers to potentially end up in a split-brain situation due to them -not understanding the new rules. +cause older servers to potentially end up in a split-brain situation due to not +understanding the new rules. A room version is defined as a string of characters which MUST NOT exceed 32 codepoints in length. Room versions MUST NOT be empty and SHOULD contain only diff --git a/specification/rooms/v1.rst b/specification/rooms/v1.rst index 207fe8bc..37b1b7ce 100644 --- a/specification/rooms/v1.rst +++ b/specification/rooms/v1.rst @@ -34,8 +34,8 @@ version room they are dealing with prior to executing a given algorithm. .. WARNING:: Although room version 1 is the most popular room version, it is known to have undesirable effects. Servers implementing support for room version 1 should be - aware that restrictions should be generally relaxed and be aware that inconsistencies - may occur until room version 2 is ready and adopted. + aware that restrictions should be generally relaxed and that inconsistencies + may occur until room version 2 (or later) is ready and adopted. State resolution ~~~~~~~~~~~~~~~~ diff --git a/specification/rooms/v2.rst b/specification/rooms/v2.rst index eef03497..d39a7caa 100644 --- a/specification/rooms/v2.rst +++ b/specification/rooms/v2.rst @@ -60,8 +60,8 @@ Power events A *power event* is a state event with type ``m.room.power_levels`` or ``m.room.join_rules``, or a state event with type ``m.room.member`` where the ``membership`` is ``leave`` or ``ban`` and the ``sender`` does not match the - ``state_key``. The idea behind this is that power events are events that have - may remove someone's ability to do something in the room. + ``state_key``. The idea behind this is that power events are events that might + remove someone's ability to do something in the room. Unconflicted state map and conflicted state set The *unconflicted state map* is the state where the value of each key exists diff --git a/specification/server_server_api.rst b/specification/server_server_api.rst index 8645888b..09ad333f 100644 --- a/specification/server_server_api.rst +++ b/specification/server_server_api.rst @@ -752,9 +752,8 @@ is at the top):: Suppose E3 and E4 are both ``m.room.name`` events which set the name of the room. What should the name of the room be at E5? -Servers must use the appropriate recursively-defined algorithm as required -by the room version. For a description of each room version's algorithm, please -see the `room version specification`_ . +The algorithm to be used for state resolution depends on the room version. For +a description of each room version's algorithm, please see the `room version specification`_. Backfilling and retrieving missing events