Add a room version specification

The "Room Specification" (or "Room Version Specification") is the specification that defines which room versions do what and are intended to be documents which speak the truth about how rooms operate under the hood.

The approach taken here is a bit different than other specifications. For starters, the specification is versioned in this project instead of relying on the matrix.org repository to track compiled HTML. This is done for a couple reasons, the first being we're still developing the v1 specification while concurrently making a v2 spec and the second being trying to reduce the reliance on matrix.org's repository for specifications.

Because the room spec is built into versions, some changes needed to be made. The `targets.yaml` now has a special syntax for indicating what version something is at, and the changelog generator can handle rendering different versions of the same changelog (as parsed from the RST). Some additional work has been put in to the changelog parsing to allow us to reference the v1 room spec as "v1" without having to sacrifice clarity in the changelog headings.

Finally, this moves the state resolution algorithms into the versioned
spec as a result of MSC1759 (https://github.com/matrix-org/matrix-doc/pull/1759).

Note: this does not introduce the concept of versioned schemas (tabs) that I was previously working with. There's currently no use for them, so they are shelved elsewhere.

scripts/templating/matrix_templates/units.py

@@ -757,6 +757,7 @@ class MatrixUnits(Units):
         is_ver = substitutions.get("%IDENTITY_RELEASE_LABEL%", "unstable")
         as_ver = substitutions.get("%APPSERVICE_RELEASE_LABEL%", "unstable")
         push_gw_ver = substitutions.get("%PUSH_GATEWAY_RELEASE_LABEL%", "unstable")
+        room_ver = substitutions.get("%CURRENT_ROOM_VERSION%", "unstable")
 
         # we abuse the typetable to return this info to the templates
         return TypeTable(rows=[
@@ -780,6 +781,10 @@ class MatrixUnits(Units):
                 "`Push Gateway API <push_gateway/"+push_gw_ver+".html>`_",
                 push_gw_ver,
                 "Push notifications for Matrix events",
+            ), TypeTableRow(
+                "`Rooms <rooms/"+room_ver+".html>`_",
+                room_ver,
+                "Specification for behaviour of rooms, such as event formats",
             ),
         ])
 
@@ -906,11 +911,26 @@ class MatrixUnits(Units):
     def load_changelogs(self):
         changelogs = {}
 
+        # Changelog generation is a bit complicated. We rely on towncrier to
+        # generate the unstable/current changelog, but otherwise use the RST
+        # edition to record historical changelogs. This is done by prepending
+        # the towncrier output to the RST in memory, then parsing the RST by
+        # hand. We parse the entire changelog to create a changelog for each
+        # version which may be of use in some APIs.
+
+        # Map specific headers to specific keys that'll be used eventually
+        # in variables. Things not listed here will get lowercased and formatted
+        # such that characters not [a-z0-9] will be replaced with an underscore.
+        keyword_versions = {
+            "Unreleased Changes": "unstable"
+        }
+
+        # Only generate changelogs for things that have an RST document
         for f in os.listdir(CHANGELOG_DIR):
             if not f.endswith(".rst"):
                 continue
             path = os.path.join(CHANGELOG_DIR, f)
-            name = f[:-4]
+            name = f[:-4]  # take off ".rst"
 
             # If there's a directory with the same name, we'll try to generate
             # a towncrier changelog and prepend it to the general changelog.
@@ -959,15 +979,39 @@ class MatrixUnits(Units):
                     prev_line = line
                 else:  # have title, get body (stop on next title or EOF)
                     if re.match("^[=]{3,}$", line.strip()):
-                        # we added the title in the previous iteration, pop it
-                        # then bail out.
-                        changelog_lines.pop()
-                        break
+                        # we hit another title, so pop the last line of
+                        # the changelog and record the changelog
+                        new_title = changelog_lines.pop()
+                        if name not in changelogs:
+                            changelogs[name] = {}
+                        if title_part in keyword_versions:
+                            title_part = keyword_versions[title_part]
+                        title_part = title_part.strip().replace("^[a-zA-Z0-9]", "_").lower()
+                        changelog = "".join(changelog_lines)
+                        changelogs[name][title_part] = changelog
+
+                        # reset for the next version
+                        changelog_lines = []
+                        title_part = new_title.strip()
+                        continue
                     # Don't generate subheadings (we'll keep the title though)
                     if re.match("^[-]{3,}$", line.strip()):
                         continue
+                    if line.strip().startswith(".. version: "):
+                        # The changelog is directing us to use a different title
+                        # for the changelog.
+                        title_part = line.strip()[len(".. version: "):]
+                        continue
+                    if line.strip().startswith(".. "):
+                        continue  # skip comments
                     changelog_lines.append("    " + line + '\n')
-            changelogs[name] = "".join(changelog_lines)
+            if len(changelog_lines) > 0 and title_part is not None:
+                if name not in changelogs:
+                    changelogs[name] = {}
+                if title_part in keyword_versions:
+                    title_part = keyword_versions[title_part]
+                changelog = "".join(changelog_lines)
+                changelogs[name][title_part.replace("^[a-zA-Z0-9]", "_").lower()] = changelog
 
         return changelogs