Distinguish 'client' from 'federation' events (#3658)

Fixes #3305 
Fixes #3380
 
The idea here is to better distinguish between a 'raw' event (as we send over the wire), and the 
'serialised' format, as sent in responses to the C-S api and in `PUT /_matrix/app/v1/transactions/{txnId}`.

It's made more complicated by the fact that there are _two_ serialisation formats, one used by `/sync`
and `/notifications`, and one by everything else (the difference being whether `room_id` is included).

In an ideal world, we wouldn't repeat `SerialisedEvent` every time it's used, and instead just link to the
first reference, but that's a job for another day.

Another job for another day is to get rid of things like `sync_state_event.yaml` (which is now used
only in one place, so should be inlined.)

changelogs/application_service/newsfragments/3658.clarification

@@ -0,0 +1 @@
+Distinguish between "federation" event format as exchanged by the Federation API, and the "client" event formats as used in the client-server and AS APIs.

changelogs/client_server/newsfragments/3658.clarification

@@ -0,0 +1 @@
+Distinguish between "federation" event format as exchanged by the Federation API, and the "client" event formats as used in the client-server and AS APIs.

content/client-server-api/_index.md

@@ -1440,32 +1440,6 @@ any given point in time:
 
     [E0]->[E1]->[E2]->[E3]->[E4]->[E5]
 
-{{% boxes/warning %}}
-The format of events can change depending on room version. Check the
-[room version specification](/rooms) for specific
-details on what to expect for event formats. Examples contained within
-the client-server specification are expected to be compatible with all
-specified room versions, however some differences may still apply.
-
-For this version of the specification, clients only need to worry about
-the event ID format being different depending on room version. Clients
-should not be parsing the event ID, and instead be treating it as an
-opaque string. No changes should be required to support the currently
-available room versions.
-{{% /boxes/warning %}}
-
-{{% boxes/warning %}}
-Event bodies are considered untrusted data. This means that any application using
-Matrix must validate that the event body is of the expected shape/schema
-before using the contents verbatim.
-
-**It is not safe to assume that an event body will have all the expected
-fields of the expected types.**
-
-See [MSC2801](https://github.com/matrix-org/matrix-doc/pull/2801) for more
-detail on why this assumption is unsafe.
-{{% /boxes/warning %}}
-
 ### Types of room events
 
 Room events are split into two categories:
@@ -1496,25 +1470,31 @@ sent by clients and other clients would receive it through Matrix,
 assuming the client has access to the `com.example` namespace.
 {{% /boxes/note %}}
 
-Note that the structure of these events may be different than those in
+### Room event format
-the server-server API.
 
-#### Event fields
+The "federation" format of a room event, which is used internally by homeservers
+and between homeservers via the Server-Server API, depends on the ["room
+version"](/rooms) in use by the room. See, for example, the definitions
+in [room version 1](/rooms/v1#event-format) and [room version
+3](/rooms/v3#event-format).
 
-{{% event-fields event_type="event" %}}
+However, it is unusual that a Matrix client would encounter this event
+format. Instead, homeservers are responsible for converting events into the
+format shown below so that they can be easily parsed by clients.
 
-#### Room event fields
+{{% boxes/warning %}}
+Event bodies are considered untrusted data. This means that any application using
+Matrix must validate that the event body is of the expected shape/schema
+before using the contents verbatim.
 
-{{% event-fields event_type="room_event" %}}
+**It is not safe to assume that an event body will have all the expected
+fields of the expected types.**
 
-#### State event fields
+See [MSC2801](https://github.com/matrix-org/matrix-doc/pull/2801) for more
+detail on why this assumption is unsafe.
+{{% /boxes/warning %}}
 
-In addition to the fields of a Room Event, State Events have the
+{{% definition path="api/client-server/definitions/client_event" %}}
-following field:
-
-| Key          | Type         | Description                                                                                                  |
-|--------------|--------------|--------------------------------------------------------------------------------------------------------------|
-| state_key    | string       | **Required.** A unique key which defines the overwriting semantics for this piece of room state. This value is often a zero-length string. The presence of this key makes this event a State Event. State keys starting with an `@` are reserved for referencing user IDs, such as room members. With the exception of a few events, state events set with a given user's ID as the state key MUST only be set by that user.         |
 
 ### Stripped state
 
@@ -1580,8 +1560,7 @@ updates not being sent.
 ### Size limits
 
 The complete event MUST NOT be larger than 65536 bytes, when formatted
-as a [PDU for the Server-Server
+with the [federation event format](#room-event-format), including any
-protocol](/server-server-api/#pdus), including any
 signatures, and encoded as [Canonical
 JSON](/appendices#canonical-json).
 

data/api/application-service/transactions.yaml

@@ -65,8 +65,7 @@ paths:
                 description: |-
                   A list of events, formatted as per the Client-Server API.
                 items:
-                  type: object
+                  $ref: "../client-server/definitions/client_event.yaml"
-                  title: Event
             required: ["events"]
       responses:
         200:

data/api/client-server/definitions/client_event.yaml

@@ -0,0 +1,47 @@
+# Copyright 2022 The Matrix.org Foundation C.I.C.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+title: ClientEvent
+description: |-
+  The format used for events when they are returned from a homeserver to a client
+  via the Client-Server API, or sent to an Application Service via the Application Services API.
+type: object
+allOf:
+  - $ref: "client_event_without_room_id.yaml"
+  - properties:
+      room_id:
+        description: The ID of the room associated with this event.
+        type: string
+        example: '!jEsUZKDJdhlrceRyVU:example.org'
+
+      unsigned:
+        properties:
+            redacted_because:
+              title: ClientEvent
+              example: {
+                "type": "m.room.redaction",
+                "sender": "@moderator:example.org",
+                "content": {
+                  "reason": "spam"
+                },
+                "redacts": "$26RqwJMLw-yds1GAH_QxjHRC1Da9oasK0e5VLnck_45",
+                "event_id": "$Nhl3rsgHMjk-DjMJANawr9HHAhLg4GcoTYrSiYYGqEE",
+                "origin_server_ts": 1632491098485,
+                "room_id": '!jEsUZKDJdhlrceRyVU:example.org',
+                "unsigned": {
+                  "age": 1257,
+                }
+            }
+    required:
+      - room_id

data/api/client-server/definitions/client_event_without_room_id.yaml

@@ -0,0 +1,110 @@
+# Copyright 2022 The Matrix.org Foundation C.I.C.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+title: ClientEventWithoutRoomID
+description: |-
+  The format used for events when they are returned from
+  API endpoints such as `/sync`, where the `room_id` is implied elsewhere
+  in the response.
+type: object
+required:
+  - event_id
+  - type
+  - sender
+  - origin_server_ts
+  - content
+properties:
+  event_id:
+    description: The globally unique identifier for this event.
+    type: string
+    example: '$26RqwJMLw-yds1GAH_QxjHRC1Da9oasK0e5VLnck_45'
+  type:
+    description: The type of the event.
+    type: string
+    example: 'm.room.member'
+  state_key:
+    description: |-
+      Present if, and only if, this event is a *state* event. The key making
+      this piece of state unique in the room. Note that it is often an empty
+      string.
+    type: string
+    example: '@user:example.org'
+  sender:
+    description: Contains the fully-qualified ID of the user who sent this event.
+    type: string
+    example: "@example:example.org"
+  origin_server_ts:
+    description: |-
+      Timestamp (in milliseconds since the unix epoch) on originating homeserver
+      when this event was sent.
+    type: integer
+    format: int64
+    example: 1632489532305
+  content:
+    description: |-
+      The body of this event, as created by the client which sent it.
+    type: object
+    example: {
+      "membership": "join"
+    }
+  unsigned:
+    title: UnsignedData
+    type: object
+    description: Contains optional extra information about the event.
+    properties:
+      age:
+        description: The time in milliseconds that has elapsed since the event was
+          sent. This field is generated by the local homeserver, and may be incorrect
+          if the local time on at least one of the two servers is out of sync, which can
+          cause the age to either be negative or greater than it actually is.
+        type: integer
+        format: int64
+        example: 1567437
+      redacted_because:
+        description: The event that redacted this event, if any.
+        type: object
+        title: ClientEventWithoutRoomID
+        example: {
+          "type": "m.room.redaction",
+          "sender": "@moderator:example.org",
+          "content": {
+            "reason": "spam"
+          },
+          "redacts": "$26RqwJMLw-yds1GAH_QxjHRC1Da9oasK0e5VLnck_45",
+          "event_id": "$Nhl3rsgHMjk-DjMJANawr9HHAhLg4GcoTYrSiYYGqEE",
+          "origin_server_ts": 1632491098485,
+          "unsigned": {
+            "age": 1257,
+          }
+        }
+      transaction_id:
+        description: |
+          The client-supplied transaction ID, for example, provided via
+          `PUT /_matrix/client/v3/rooms/{roomId}/send/{eventType}/{txnId}`,
+          if the client being given the event is the same one which sent it.
+        type: string
+      prev_content:
+        description: |
+          The previous `content` for this event. This field is generated
+          by the local homeserver, and is only returned if the event is a state event,
+          and the client has permission to see the previous content.
+        x-changedInMatrixVersion:
+          1.2: |
+            Previously, this field was specified at the top level of returned
+            events rather than in `unsigned` (with the exception of the [`GET
+            .../notifications`](/client-server-api/#get_matrixclientv3notifications)
+            endpoint), though in practice no known server implementations honoured
+            this.
+        title: EventContent
+        type: object

data/api/client-server/definitions/room_event_batch.yaml

@@ -1,28 +0,0 @@
-# Copyright 2018 New Vector Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-properties:
-  events:
-    description: List of events.
-    items:
-      allOf:
-      - $ref: ../../../event-schemas/schema/core-event-schema/sync_room_event.yaml
-      type: object
-      required:
-      - event_id
-      #- room_id - Not in /sync
-      - sender
-      - origin_server_ts
-    type: array
-type: object
-title: RoomEventBatch

data/api/client-server/definitions/state_event_batch.yaml

@@ -15,15 +15,7 @@ properties:
   events:
     description: List of events.
     items:
-      allOf:
+      $ref: client_event_without_room_id.yaml
-      - $ref: ../../../event-schemas/schema/core-event-schema/sync_state_event.yaml
-      type: object
-      required:
-      - event_id
-      #- room_id - Not in /sync
-      - sender
-      - origin_server_ts
-      - state_key
     type: array
 type: object
 title: StateEventBatch

data/api/client-server/definitions/timeline_batch.yaml

@@ -12,8 +12,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-allOf:
-- $ref: room_event_batch.yaml
 properties:
   limited:
     description: True if the number of events returned was limited by the `limit`
@@ -27,5 +25,12 @@ properties:
       If no earlier events are available, this property may be omitted from
       the response.
     type: string
+  events:
+    description: List of events.
+    type: array
+    items:
+      $ref: "client_event_without_room_id.yaml"
 type: object
 title: TimelineBatch
+required:
+  - events

data/api/client-server/event_context.yaml

@@ -92,28 +92,25 @@ paths:
                   A list of room events that happened just before the
                   requested event, in reverse-chronological order.
                 items:
-                  allOf:
+                  $ref: "definitions/client_event.yaml"
-                    - "$ref": "../../event-schemas/schema/core-event-schema/room_event.yaml"
               event:
                 description: |-
                   Details of the requested event.
                 allOf:
-                  - "$ref": "../../event-schemas/schema/core-event-schema/room_event.yaml"
+                  - $ref: "definitions/client_event.yaml"
               events_after:
                 type: array
                 description: |-
                   A list of room events that happened just after the
                   requested event, in chronological order.
                 items:
-                  allOf:
+                  $ref: "definitions/client_event.yaml"
-                    - "$ref": "../../event-schemas/schema/core-event-schema/room_event.yaml"
               state:
                 type: array
                 description: |-
                   The state of the room at the last event returned.
                 items:
-                  allOf:
+                  $ref: "definitions/client_event.yaml"
-                    - "$ref": "../../event-schemas/schema/core-event-schema/state_event.yaml"
           examples:
             application/json: {
               "end": "t29-57_2_0_2",

data/api/client-server/message_pagination.yaml

@@ -125,7 +125,7 @@ paths:
                   are available. Clients should continue to paginate until no `end` property
                   is returned.
                 items:
-                  "$ref": "../../event-schemas/schema/core-event-schema/room_event.yaml"
+                  "$ref": "definitions/client_event.yaml"
               state:
                 type: array
                 description: |-
@@ -138,7 +138,7 @@ paths:
                   sent to the client in prior calls to this endpoint, assuming
                   the membership of those members has not changed.
                 items:
-                  $ref: "../../event-schemas/schema/core-event-schema/state_event.yaml"
+                  $ref: "definitions/client_event.yaml"
             required: [start, chunk]
           examples:
             application/json: {

data/api/client-server/notifications.yaml

@@ -112,8 +112,7 @@ paths:
                       type: object
                       title: Event
                       description: The Event object for the event that triggered the notification.
-                      allOf:
+                      "$ref": "definitions/client_event_without_room_id.yaml"
-                        - $ref: ../../event-schemas/schema/core-event-schema/sync_room_event.yaml
                     profile_tag:
                       type: string
                       description: The profile tag of the rule that matched this event.

data/api/client-server/old_sync.yaml

@@ -84,10 +84,7 @@ paths:
                 type: array
                 description: "An array of events."
                 items:
-                  type: object
+                  $ref: "definitions/client_event.yaml"
-                  title: Event
-                  allOf:
-                    - "$ref": "../../event-schemas/schema/core-event-schema/room_event.yaml"
         400:
           description: "Bad pagination `from` parameter."
       tags:
@@ -206,10 +203,7 @@ paths:
                 type: array
                 description: A list of presence events.
                 items:
-                  type: object
+                  $ref: "definitions/client_event.yaml"
-                  title: Event
-                  allOf:
-                    - "$ref": "../../event-schemas/schema/core-event-schema/event.yaml"
               rooms:
                 type: array
                 items:
@@ -227,8 +221,7 @@ paths:
                         type: object
                         title: "InviteEvent"
                         description: "The invite event if `membership` is `invite`"
-                        allOf:
+                        $ref: "definitions/client_event.yaml"
-                          - "$ref": "../../event-schemas/schema/m.room.member.yaml"
                     messages:
                       type: object
                       title: PaginationChunk
@@ -260,10 +253,7 @@ paths:
                             messages that preceded them leaving. This array
                             will consist of at most `limit` elements.
                           items:
-                            type: object
+                            $ref: "definitions/client_event.yaml"
-                            title: RoomEvent
-                            allOf:
-                              - "$ref": "../../event-schemas/schema/core-event-schema/room_event.yaml"
                       required: ["end", "chunk"]
                     state:
                       type: array
@@ -273,10 +263,7 @@ paths:
                         user has left the room this will be the state of the
                         room when they left it.
                       items:
-                        title: StateEvent
+                        $ref: "definitions/client_event.yaml"
-                        type: object
-                        allOf:
-                          - "$ref": "../../event-schemas/schema/core-event-schema/state_event.yaml"
                     visibility:
                       type: string
                       enum: ["private", "public"]
@@ -289,10 +276,7 @@ paths:
                         The private data that this user has attached to
                         this room.
                       items:
-                        title: Event
+                        $ref: "definitions/client_event.yaml"
-                        type: object
-                        allOf:
-                          - "$ref": "../../event-schemas/schema/core-event-schema/event.yaml"
                   required: ["room_id", "membership"]
               account_data:
                 type: array
@@ -336,8 +320,7 @@ paths:
           examples:
             application/json: {"$ref": "../../event-schemas/examples/m.room.message$m.text.yaml"}
           schema:
-            allOf:
+            $ref: "definitions/client_event.yaml"
-              - "$ref": "../../event-schemas/schema/core-event-schema/event.yaml"
         404:
           description: The event was not found or you do not have permission to read this event.
       tags:

data/api/client-server/peeking_events.yaml

@@ -102,7 +102,7 @@ paths:
                   type: object
                   title: Event
                   allOf:
-                    - "$ref": "../../event-schemas/schema/core-event-schema/room_event.yaml"
+                    - "$ref": "definitions/client_event.yaml"
         400:
           description: "Bad pagination `from` parameter."
       tags:

data/api/client-server/room_initial_sync.yaml

@@ -119,10 +119,7 @@ paths:
                       messages that preceded them leaving. This array
                       will consist of at most `limit` elements.
                     items:
-                      type: object
+                      "$ref": "definitions/client_event.yaml"
-                      title: RoomEvent
-                      allOf:
-                        - "$ref": "../../event-schemas/schema/core-event-schema/room_event.yaml"
                 required: ["end", "chunk"]
               state:
                 type: array
@@ -132,10 +129,7 @@ paths:
                   user has left the room this will be the state of the
                   room when they left it.
                 items:
-                  title: StateEvent
+                  "$ref": "definitions/client_event.yaml"
-                  type: object
-                  allOf:
-                    - "$ref": "../../event-schemas/schema/core-event-schema/state_event.yaml"
               visibility:
                 type: string
                 enum: ["private", "public"]

data/api/client-server/rooms.yaml

@@ -58,8 +58,7 @@ paths:
               "$ref": "../../event-schemas/examples/m.room.message$m.text.yaml"
             }
           schema:
-            allOf:
+            "$ref": "definitions/client_event.yaml"
-              - "$ref": "../../event-schemas/schema/core-event-schema/event.yaml"
         404:
           description: The event was not found or you do not have permission to read this event.
           examples:
@@ -165,10 +164,7 @@ paths:
               has left the room then this will be the state of the room
               when they left as a list of events.
             items:
-              title: StateEvent
+              $ref: "definitions/client_event.yaml"
-              type: object
-              allOf:
-                - "$ref": "../../event-schemas/schema/core-event-schema/state_event.yaml"
         403:
           description: >
             You aren't a member of the room and weren't previously a
@@ -251,10 +247,7 @@ paths:
               chunk:
                 type: array
                 items:
-                  title: MemberEvent
+                  $ref: "definitions/client_event.yaml"
-                  type: object
-                  allOf:
-                    - "$ref": "../../event-schemas/schema/m.room.member.yaml"
         403:
           description: >
             You aren't a member of the room and weren't previously a

data/api/client-server/search.yaml

@@ -204,7 +204,7 @@ paths:
                               type: object
                               title: Event
                               description: The event that matched.
-                              "$ref": "../../event-schemas/schema/core-event-schema/room_event.yaml"
+                              "$ref": "definitions/client_event.yaml"
                             context:
                               type: object
                               title: Event Context
@@ -247,7 +247,7 @@ paths:
                                   items:
                                     title: Event
                                     type: object
-                                    "$ref": "../../event-schemas/schema/core-event-schema/room_event.yaml"
+                                    "$ref": "definitions/client_event.yaml"
                                 events_after:
                                   type: array
                                   title: Events After
@@ -255,7 +255,7 @@ paths:
                                   items:
                                     title: Event
                                     type: object
-                                    "$ref": "../../event-schemas/schema/core-event-schema/room_event.yaml"
+                                    "$ref": "definitions/client_event.yaml"
                       state:
                         type: object
                         title: Current state
@@ -271,7 +271,7 @@ paths:
                           title: Room State
                           items:
                             type: object
-                            "$ref": "../../event-schemas/schema/core-event-schema/state_event.yaml"
+                            "$ref": "definitions/client_event.yaml"
                       groups:
                         type: object
                         title: Groups

data/event-schemas/schema/core-event-schema/unsigned_prop.yaml

@@ -18,31 +18,5 @@ description: Contains optional extra information about the event.
 properties:
   age:
     description: The time in milliseconds that has elapsed since the event was
-      sent. This field is generated by the local homeserver, and may be incorrect
+      sent.
-      if the local time on at least one of the two servers is out of sync, which can
-      cause the age to either be negative or greater than it actually is.
     type: integer
-  redacted_because:
-    description: The event that redacted this event, if any.
-    title: Event
-    type: object
-  transaction_id:
-    description: |
-      The client-supplied transaction ID, for example, provided via
-      `PUT /_matrix/client/v3/rooms/{roomId}/send/{eventType}/{txnId}`,
-      if the client being given the event is the same one which sent it.
-    type: string
-  prev_content:
-    description: |
-      The previous `content` for this event. This field is generated
-      by the local homeserver, and is only returned if the event is a state event,
-      and the client has permission to see the previous content.
-    x-changedInMatrixVersion:
-      1.2: |
-        Previously, this field was specified at the top level of returned
-        events rather than in `unsigned` (with the exception of the [`GET
-        .../notifications`](/client-server-api/#get_matrixclientv3notifications)
-        endpoint), though in practice no known server implementations honoured
-        this.
-    title: EventContent
-    type: object