Threads: The base (#1254)

* Spec MSC3440: Threading (just the base)

Other threading MSCs to follow

* Spec MSC3856: Threads list API

* Spec MSC3715:  Add`dir` to `/relations`

* changelog

* Apply suggestions from code review

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Update changelogs/client_server/newsfragments/1254.feature

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

changelogs/client_server/newsfragments/1254.feature

@@ -0,0 +1 @@
+Add threading via `m.thread` relations, as per [MSC3440](https://github.com/matrix-org/matrix-spec-proposals/pull/3440), [MSC3816](https://github.com/matrix-org/matrix-spec-proposals/pull/3816), [MSC3856](https://github.com/matrix-org/matrix-spec-proposals/pull/3856), and [MSC3715](https://github.com/matrix-org/matrix-spec-proposals/pull/3715).

content/client-server-api/_index.md

@@ -1972,6 +1972,7 @@ This specification describes the following relationship types:
 
 * [Rich replies](#rich-replies) (**Note**: does not use `rel_type`).
 * [Event replacements](#event-replacements).
+* [Threads](#threading).
 
 #### Aggregations
 
@@ -2056,6 +2057,7 @@ The endpoints where the server *should* include bundled aggregations are:
 * [`GET /sync`](#get_matrixclientv3sync) when the relevant section has a `limited` value
   of `true`.
 * [`POST /search`](#post_matrixclientv3search) for any matching events under `room_events`.
+* {{< added-in v="1.4" >}} [`GET /rooms/{roomId}/threads`](#get_matrixclientv1roomsroomidthreads)
 
 {{% boxes/note %}}
 The server is **not** required to return bundled aggregations on deprecated endpoints
@@ -2642,3 +2644,4 @@ systems.
 {{< cs-module name="moderation_policies" >}}
 {{< cs-module name="spaces" >}}
 {{< cs-module name="event_replacements" >}}
+{{< cs-module name="threading" >}}

content/client-server-api/modules/threading.md

@@ -0,0 +1,201 @@
+---
+type: module
+---
+
+### Threading
+
+{{% added-in v="1.4" %}}
+
+Threads allow users to visually branch their conversations in a room. Typically mostly used
+when a room is discussing multiple topics, threads provide more organisation of communication
+that traditional [rich replies](#rich-replies) can't always offer.
+
+Clients SHOULD render threads differently to regular messages or replies in the timeline, such
+as by providing some context to what is going on in the thread but keeping the full conversation
+history behind a disclosure.
+
+Threads are established using a `rel_type` of `m.thread` and reference the *thread root* (the
+first event in a thread). It is not possible to create a thread from an event with a `rel_type`,
+which includes not being able to nest threads. All conversation in a thread reference the thread
+root instead of the most recent message, unlike rich reply chains.
+
+As a worked example, the following represents a thread and how it'd be formed:
+
+```json
+{
+  // irrelevant fields excluded
+  "type": "m.room.message",
+  "event_id": "$alice_hello",
+  "sender": "@alice:example.org",
+  "content": {
+    "msgtype": "m.text",
+    "body": "Hello world! How are you?"
+  }
+}
+```
+
+```json
+{
+  // irrelevant fields excluded
+  "type": "m.room.message",
+  "event_id": "$bob_hello",
+  "sender": "@bob:example.org",
+  "content": {
+    "m.relates_to": {
+      "rel_type": "m.thread",
+      "event_id": "$alice_hello"
+    },
+    "msgtype": "m.text",
+    "body": "I'm doing okay, thank you! How about yourself?"
+  }
+}
+```
+
+```json
+{
+  // irrelevant fields excluded
+  "type": "m.room.message",
+  "event_id": "$alice_reply",
+  "sender": "@alice:example.org",
+  "content": {
+    "m.relates_to": {
+      "rel_type": "m.thread",
+      "event_id": "$alice_hello" // note: always references the *thread root*
+    },
+    "msgtype": "m.text",
+    "body": "I'm doing great! Thanks for asking."
+  }
+}
+```
+
+As shown, any event without a `rel_type` can become a thread root by simply referencing it
+using an `m.thread` relationship.
+
+#### Fallback for unthreaded clients
+
+Clients which understand how to work with threads should simply do so, however clients which
+might not be aware of threads (due to age or scope) might not be able to helpfully represent
+the conversation history to its users.
+
+To work around this, events sent by clients which understand threads include [rich reply](#rich-replies)
+metadata to attempt to form a reply chain representation of the conversation. This representation
+is not ideal for heavily threaded rooms, but allows for users to have context as to what is
+being discussed with respect to other messages in the room.
+
+This representation is achieved by merging the two relationships and setting a new `is_falling_back`
+flag to `true`.
+
+```json
+// within an event's content...
+"m.relates_to": {
+  // The m.thread relationship structure
+  "rel_type": "m.thread",
+  "event_id": "$root",
+
+  // The rich reply structure
+  "m.in_reply_to": {
+    // The most recent message known to the client in the thread.
+    // This should be something with a high chance of being rendered by the other client,
+    // such as an `m.room.message` event.
+    "event_id": "$target"
+  },
+
+  // A flag to denote that this is a thread with reply fallback
+  "is_falling_back": true
+}
+```
+
+For `m.room.message` events represented this way, no [reply fallback](#fallbacks-for-rich-replies)
+is specified. This allows thread-aware clients to discard the `m.in_reply_to` object entirely
+when `is_falling_back` is `true`.
+
+{{% boxes/note %}}
+Clients which are acutely aware of threads (they do not render threads, but are otherwise
+aware of the feature existing in the spec) can treat rich replies to an event with a `rel_type`
+of `m.thread` as a threaded reply, for conversation continuity on the threaded client's side.
+
+To do this, copy the `event_id` (thread root) from the event being replied to, add the
+`m.in_reply_to` metadata, and add `is_falling_back: true` to `m.relates_to`.
+{{% /boxes/note %}}
+
+#### Replies within threads
+
+In the [fallback for unthreaded clients](#fallback-for-unthreaded-clients) section, a new
+`is_falling_back` flag is added to `m.relates_to`. This flag defaults to `false` when not
+provided, which also allows a threaded message to contain a reply itself.
+
+Aside from `is_falling_back` being `false` (or not specified), the fallback for unthreaded
+clients is used to create a reply within a thread: clients should render the event accordingly.
+
+#### Server behaviour
+
+##### Validation of `m.thread` relationships
+
+Servers SHOULD reject client requests which attempt to start a thread off an event with a
+`rel_type`. If the client attempts to target an event which already has an `m.thread`,
+`m.reference`, or any other `rel_type` then it should receive a HTTP 400 error response
+with appropriate error message, as per the [standard error response](#standard-error-response)
+structure.
+
+{{% boxes/note %}}
+A specific error code is not currently available for this case: servers should use `M_UNKNOWN`
+alongside the HTTP 400 status code.
+{{% /boxes/note %}}
+
+##### Server-side aggregation of `m.thread` relationships
+
+Given threads always reference the thread root, an event can have multiple "child" events which
+then form the thread itself. These events should be [aggregated](#aggregations) by the server.
+
+The aggregation for threads includes some information about the user's participation in the thread,
+the approximate number of events in the thread (as known to the server), and the most recent event
+in the thread (topologically). This is then bundled into the event as `m.thread`:
+
+```json
+{
+  "event_id": "$root_event",
+  // irrelevant fields not shown
+  "unsigned": {
+    "m.relations": {
+      "m.thread": {
+        "latest_event": {
+          // A serialized copy of the latest event in the thread.
+          // Some fields are not shown here for brevity.
+          "event_id": "$message",
+          "sender": "@alice:example.org",
+          "room_id": "!room:example.org",
+          "type": "m.room.message",
+          "content": {
+            "msgtype": "m.text",
+            "body": "Woo! Threads!"
+          }
+        },
+        "count": 7,
+        "current_user_participated": true
+      }
+    }
+  }
+}
+```
+
+`latest_event` is the most recent event (topologically to the server) in the thread sent by an
+un-[ignored user](#ignoring-users).
+
+Note that any bundled aggregations on `latest_event` should also be present. The server should be
+careful to avoid loops, though loops are not currently possible due to `m.thread` not being possible
+to target an event with a `rel_type` already.
+
+`count` is simply the number of events using `m.thread` as a `rel_type` pointing to the target event.
+It does not include events sent by [ignored users](#ignoring-users).
+
+`current_user_participated` is `true` when the authenticated user is either:
+1. The `sender` of the event receiving the bundle (they sent the thread root).
+2. The `sender` of an event which references the thread root with a `rel_type` of `m.thread`.
+
+#### Querying threads in a room
+
+Clients looking to get all the events in a thread can use
+[`GET /relations/{threadRootId}/m.thread`](#get_matrixclientv1roomsroomidrelationseventidreltype),
+however getting all threads in a room is done through a dedicated API:
+
+{{% http-api spec="client-server" api="threads_list" %}}

data/api/client-server/definitions/m.relates_to.yaml

@@ -34,8 +34,8 @@ properties:
 
       The relationship type determines how clients should perceive the event, and in what
       context. Some relationship types are processed server-side for "bundling", though not
-      all relationships require such behaviour. For example, an `m.thread` relationship type
+      all relationships require such behaviour. For example, an [`m.thread` relationship type](/client-server-api/#threading)
-      might denote that the event is part of a "thread" of messages and should be rendered as
+      denotes that the event is part of a "thread" of messages and should be rendered as
       such.
   event_id:
     type: string

data/api/client-server/relations.yaml

@@ -63,7 +63,7 @@ paths:
             The pagination token to start returning results from. If not supplied, results
             start at the most recent topological event known to the server.
 
-            Can be a `next_batch` token from a previous call, or a returned
+            Can be a `next_batch` or `prev_batch` token from a previous call, or a returned
             `start` token from [`/messages`](/client-server-api/#get_matrixclientv3roomsroomidmessages),
             or a `next_batch` token from [`/sync`](/client-server-api/#get_matrixclientv3sync).
           required: false
@@ -89,6 +89,16 @@ paths:
             Similarly, the server should apply a default value when not supplied.
           required: false
           x-example: 20
+        - in: query
+          type: string
+          enum: ["b", "f"]
+          name: dir
+          x-addedInMatrixVersion: "1.4"
+          description: |-
+            Optional (default `b`) direction to return events from. If this is set to `f`, events
+            will be returned in chronological order starting at `from`. If it
+            is set to `b`, events will be returned in *reverse* chronological
+            order, again starting at `from`.
       responses:
         # note: this endpoint deliberately does not support rate limiting, therefore a
         # 429 error response is not included.
@@ -193,7 +203,7 @@ paths:
             The pagination token to start returning results from. If not supplied, results
             start at the most recent topological event known to the server.
 
-            Can be a `next_batch` token from a previous call, or a returned
+            Can be a `next_batch` or `prev_batch` token from a previous call, or a returned
             `start` token from [`/messages`](/client-server-api/#get_matrixclientv3roomsroomidmessages),
             or a `next_batch` token from [`/sync`](/client-server-api/#get_matrixclientv3sync).
           required: false
@@ -219,6 +229,16 @@ paths:
             Similarly, the server should apply a default value when not supplied.
           required: false
           x-example: 20
+        - in: query
+          type: string
+          enum: ["b", "f"]
+          name: dir
+          x-addedInMatrixVersion: "1.4"
+          description: |-
+            Optional (default `b`) direction to return events from. If this is set to `f`, events
+            will be returned in chronological order starting at `from`. If it
+            is set to `b`, events will be returned in *reverse* chronological
+            order, again starting at `from`.
       responses:
         # note: this endpoint deliberately does not support rate limiting, therefore a
         # 429 error response is not included.
@@ -335,7 +355,7 @@ paths:
             The pagination token to start returning results from. If not supplied, results
             start at the most recent topological event known to the server.
 
-            Can be a `next_batch` token from a previous call, or a returned
+            Can be a `next_batch` or `prev_batch` token from a previous call, or a returned
             `start` token from [`/messages`](/client-server-api/#get_matrixclientv3roomsroomidmessages),
             or a `next_batch` token from [`/sync`](/client-server-api/#get_matrixclientv3sync).
           required: false
@@ -361,6 +381,16 @@ paths:
             Similarly, the server should apply a default value when not supplied.
           required: false
           x-example: 20
+        - in: query
+          type: string
+          enum: ["b", "f"]
+          name: dir
+          x-addedInMatrixVersion: "1.4"
+          description: |-
+            Optional (default `b`) direction to return events from. If this is set to `f`, events
+            will be returned in chronological order starting at `from`. If it
+            is set to `b`, events will be returned in *reverse* chronological
+            order, again starting at `from`.
       responses:
         # note: this endpoint deliberately does not support rate limiting, therefore a
         # 429 error response is not included.

data/api/client-server/threads_list.yaml

@@ -0,0 +1,135 @@
+# Copyright 2022 The Matrix.org Foundation C.I.C.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+swagger: '2.0'
+info:
+  title: "Matrix Client-Server Threads List API"
+  version: "1.0.0"
+host: localhost:8008
+schemes:
+  - https
+  - http
+basePath: /_matrix/client/v1
+consumes:
+  - application/json
+produces:
+  - application/json
+securityDefinitions:
+  $ref: definitions/security.yaml
+paths:
+  "/rooms/{roomId}/threads":
+    get:
+      x-addedInMatrixVersion: "1.4"
+      summary: Retrieve a list of threads in a room, with optional filters.
+      description: |-
+        Paginates over the thread roots in a room, ordered by the `latest_event` of each thread root
+        in its bundle.
+      operationId: getThreadRoots
+      security:
+        - accessToken: []
+      parameters:
+        - in: path
+          type: string
+          name: roomId
+          description: The room ID where the thread roots are located.
+          required: true
+          x-example: "!room:example.org"
+        - in: query
+          type: string
+          name: include
+          enum: [all, participated]
+          description: |-
+            Optional (default `all`) flag to denote which thread roots are of interest to the caller.
+            When `all`, all thread roots found in the room are returned. When `participated`, only
+            thread roots for threads the user has [participated in](/client-server-api/#server-side-aggreagtion-of-mthread-relationships)
+            will be returned.
+          x-example: "all"
+        - in: query
+          type: integer
+          name: limit
+          description: |-
+            Optional limit for the maximum number of thread roots to include per response. Must be an integer
+            greater than zero.
+
+            Servers should apply a default value, and impose a maximum value to avoid resource exhaustion.
+          x-example: 20
+        - in: query
+          type: string
+          name: from
+          description: |-
+            A pagination token from a previous result. When not provided, the server starts paginating from
+            the most recent event visible to the user (as per history visibility rules; topologically).
+          x-example: "next_batch_token"
+      responses:
+        200:
+          description: |-
+            A portion of the available thread roots in the room, based on the filter criteria.
+          examples:
+            application/json: {
+              "chunk": [{ "$ref": "../../event-schemas/examples/m.room.message$m.text.yaml" }],
+              "next_batch": "next_batch_token"
+            }
+          schema:
+            type: object
+            properties:
+              chunk:
+                type: array
+                description: |-
+                  The thread roots, ordered by the `latest_event` in each event's aggregation bundle. All events
+                  returned include bundled [aggregations](/client-server-api/#aggregations).
+
+                  If the thread root event was sent by an [ignored user](/client-server-api/#ignoring-users), the
+                  event is returned redacted to the caller. This is to simulate the same behaviour of a client doing
+                  aggregation locally on the thread.
+                items:
+                  $ref: "definitions/client_event.yaml"
+              next_batch:
+                type: string
+                description: |-
+                  A token to supply to `from` to keep paginating the responses. Not present when there are
+                  no further results.
+            required: [chunk]
+        403:
+          description: |-
+            The user cannot view or peek on the room. A meaningful `errcode`
+            and description error text will be returned. Example reasons for rejection are:
+
+            - The room is not set up for peeking.
+            - The user has been banned from the room.
+            - The room does not exist.
+          examples:
+            application/json: {
+              "errcode": "M_FORBIDDEN",
+              "error": "You are not allowed to view this room."
+            }
+          schema:
+            "$ref": "definitions/errors/error.yaml"
+        400:
+          description: |-
+            The request was invalid in some way. A meaningful `errcode`
+            and description error text will be returned. Example reasons for rejection are:
+
+            - The `from` token is unknown to the server.
+          examples:
+            application/json: {
+              "errcode": "M_INVALID_PARAM",
+              "error": "Unknown pagination token"
+            }
+          schema:
+            "$ref": "definitions/errors/error.yaml"
+        429:
+          description: This request was rate-limited.
+          schema:
+            "$ref": "definitions/errors/rate_limited.yaml"
+      tags:
+        - Threads