shuffle the server-server chapters a bit, and split out security model and other miscellanei into a new appendices chapter. Add a vestigial application services section

specification/30_server_server_api.rst

@@ -1,12 +1,13 @@
 Federation API
-===============
+==============
 
-Matrix home servers use the Federation APIs to communicate with each other.
+Matrix home servers use the Federation APIs (also known as server-server APIs)
+to communicate with each other.
 Home servers use these APIs to push messages to each other in real-time, to
 request historic messages from each other, and to query profile and presence
 information about users on each other's servers.
 
-The API's are implemented using HTTPS GETs and PUTs between each of the
+The APIs are implemented using HTTPS GETs and PUTs between each of the
 servers. These HTTPS requests are strongly authenticated using public key
 signatures at the TLS transport layer and using public key signatures in
 HTTP Authorization headers at the HTTP layer.
@@ -101,7 +102,9 @@ is another list containing the EDUs. This key may be entirely absent if there
 are no EDUs to transfer.
 
 (* Normally the PDU list will be non-empty, but the server should cope with
-receiving an "empty" transaction.)
+receiving an "empty" transaction, as this is useful for informing peers of other
+transaction IDs they should be aware of. This effectively acts as a push
+mechanism to encourage peers to continue to replicate content.)
 
 PDUs
 ----
@@ -318,6 +321,21 @@ SRV Records
 .. TODO-doc
   - Why it is needed
   
+Server-Server Authentication
+----------------------------
+
+.. TODO-doc
+  - Why is this needed.
+  - High level overview of process.
+  - Transaction signing; Matrix Authentication headers etc
+
+Server-Server Authorization
+---------------------------
+
+.. TODO-doc
+  - PDU signing (see the Event signing section earlier)
+  - State conflict resolution (see below)
+
 State Conflict Resolution
 -------------------------
 .. NOTE::
@@ -327,6 +345,14 @@ State Conflict Resolution
   - How do conflicts arise (diagrams?)
   - How are they resolved (incl tie breaks)
   - How does this work with deleting current state
+  - How do we reject invalid federation traffic?
+
+[[TODO(paul): At this point we should probably have a long description of how
+State management works, with descriptions of clobbering rules, power levels, etc
+etc... But some of that detail is rather up-in-the-air, on the whiteboard, and
+so on. This part needs refining. And writing in its own document as the details
+relate to the server/system as a whole, not specifically to server-server
+federation.]]
 
 Presence
 --------
@@ -414,158 +440,6 @@ result field. If such is present, then the result should contain only a field
 of that name, with no others present. If not, the result should contain as much
 of the user's profile as the home server has available and can make public.
 
-Server-Server Authentication
-----------------------------
-
-.. TODO-doc
-  - Why is this needed.
-  - High level overview of process.
-  - Transaction/PDU signing
-  - How does this work with redactions? (eg hashing required keys only)
-
-
-
-Threat Model
-------------
-
-Denial of Service
-~~~~~~~~~~~~~~~~~
-
-The attacker could attempt to prevent delivery of messages to or from the
-victim in order to:
-
-* Disrupt service or marketing campaign of a commercial competitor.
-* Censor a discussion or censor a participant in a discussion.
-* Perform general vandalism.
-
-Threat: Resource Exhaustion
-+++++++++++++++++++++++++++
-
-An attacker could cause the victims server to exhaust a particular resource
-(e.g. open TCP connections, CPU, memory, disk storage)
-
-Threat: Unrecoverable Consistency Violations
-++++++++++++++++++++++++++++++++++++++++++++
-
-An attacker could send messages which created an unrecoverable "split-brain"
-state in the cluster such that the victim's servers could no longer dervive a
-consistent view of the chatroom state.
-
-Threat: Bad History
-+++++++++++++++++++
-
-An attacker could convince the victim to accept invalid messages which the
-victim would then include in their view of the chatroom history. Other servers
-in the chatroom would reject the invalid messages and potentially reject the
-victims messages as well since they depended on the invalid messages.
-
-.. TODO-spec
-  Track trustworthiness of HS or users based on if they try to pretend they
-  haven't seen recent events, and fake a splitbrain... --M
-
-Threat: Block Network Traffic
-+++++++++++++++++++++++++++++
-
-An attacker could try to firewall traffic between the victim's server and some
-or all of the other servers in the chatroom.
-
-Threat: High Volume of Messages
-+++++++++++++++++++++++++++++++
-
-An attacker could send large volumes of messages to a chatroom with the victim
-making the chatroom unusable.
-
-Threat: Banning users without necessary authorisation
-+++++++++++++++++++++++++++++++++++++++++++++++++++++
-
-An attacker could attempt to ban a user from a chatroom with the necessary
-authorisation.
-
-Spoofing
-~~~~~~~~
-
-An attacker could try to send a message claiming to be from the victim without
-the victim having sent the message in order to:
-
-* Impersonate the victim while performing illict activity.
-* Obtain privileges of the victim.
-
-Threat: Altering Message Contents
-+++++++++++++++++++++++++++++++++
-
-An attacker could try to alter the contents of an existing message from the
-victim.
-
-Threat: Fake Message "origin" Field
-+++++++++++++++++++++++++++++++++++
-
-An attacker could try to send a new message purporting to be from the victim
-with a phony "origin" field.
-
-Spamming
-~~~~~~~~
-
-The attacker could try to send a high volume of solicicted or unsolicted
-messages to the victim in order to:
-
-* Find victims for scams.
-* Market unwanted products.
-
-Threat: Unsoliticted Messages
-+++++++++++++++++++++++++++++
-
-An attacker could try to send messages to victims who do not wish to receive
-them.
-
-Threat: Abusive Messages
-++++++++++++++++++++++++
-
-An attacker could send abusive or threatening messages to the victim
-
-Spying
-~~~~~~
-
-The attacker could try to access message contents or metadata for messages sent
-by the victim or to the victim that were not intended to reach the attacker in
-order to:
-
-* Gain sensitive personal or commercial information.
-* Impersonate the victim using credentials contained in the messages.
-  (e.g. password reset messages)
-* Discover who the victim was talking to and when.
-
-Threat: Disclosure during Transmission
-++++++++++++++++++++++++++++++++++++++
-
-An attacker could try to expose the message contents or metadata during
-transmission between the servers.
-
-Threat: Disclosure to Servers Outside Chatroom
-++++++++++++++++++++++++++++++++++++++++++++++
-
-An attacker could try to convince servers within a chatroom to send messages to
-a server it controls that was not authorised to be within the chatroom.
-
-Threat: Disclosure to Servers Within Chatroom
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could take control of a server within a chatroom to expose message
-contents or metadata for messages in that room.
-
-
-Identity Servers
-================
-.. NOTE::
-  This section is a work in progress.
-
-.. TODO-doc Dave
-  - 3PIDs and identity server, functions
-
-Lawful Interception
--------------------
-
-Key Escrow Servers
-~~~~~~~~~~~~~~~~~~
 
 Policy Servers
 ==============

specification/40_application_service_api.rst

@@ -0,0 +1,4 @@
+Application Service API
+=======================
+
+The Matrix client-server API and server-server APIs provide the means to implement a consistent self-contained federated messaging fabric.  However, they provide limited means of implementing custom application-layer logic in Matrix services, such as...

specification/50_appendices.rst

@@ -0,0 +1,144 @@
+Appendices
+==========
+
+Security Threat Model
+----------------------
+
+Denial of Service
+~~~~~~~~~~~~~~~~~
+
+The attacker could attempt to prevent delivery of messages to or from the
+victim in order to:
+
+* Disrupt service or marketing campaign of a commercial competitor.
+* Censor a discussion or censor a participant in a discussion.
+* Perform general vandalism.
+
+Threat: Resource Exhaustion
++++++++++++++++++++++++++++
+
+An attacker could cause the victims server to exhaust a particular resource
+(e.g. open TCP connections, CPU, memory, disk storage)
+
+Threat: Unrecoverable Consistency Violations
+++++++++++++++++++++++++++++++++++++++++++++
+
+An attacker could send messages which created an unrecoverable "split-brain"
+state in the cluster such that the victim's servers could no longer dervive a
+consistent view of the chatroom state.
+
+Threat: Bad History
++++++++++++++++++++
+
+An attacker could convince the victim to accept invalid messages which the
+victim would then include in their view of the chatroom history. Other servers
+in the chatroom would reject the invalid messages and potentially reject the
+victims messages as well since they depended on the invalid messages.
+
+.. TODO-spec
+  Track trustworthiness of HS or users based on if they try to pretend they
+  haven't seen recent events, and fake a splitbrain... --M
+
+Threat: Block Network Traffic
++++++++++++++++++++++++++++++
+
+An attacker could try to firewall traffic between the victim's server and some
+or all of the other servers in the chatroom.
+
+Threat: High Volume of Messages
++++++++++++++++++++++++++++++++
+
+An attacker could send large volumes of messages to a chatroom with the victim
+making the chatroom unusable.
+
+Threat: Banning users without necessary authorisation
++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+An attacker could attempt to ban a user from a chatroom with the necessary
+authorisation.
+
+Spoofing
+~~~~~~~~
+
+An attacker could try to send a message claiming to be from the victim without
+the victim having sent the message in order to:
+
+* Impersonate the victim while performing illict activity.
+* Obtain privileges of the victim.
+
+Threat: Altering Message Contents
++++++++++++++++++++++++++++++++++
+
+An attacker could try to alter the contents of an existing message from the
+victim.
+
+Threat: Fake Message "origin" Field
++++++++++++++++++++++++++++++++++++
+
+An attacker could try to send a new message purporting to be from the victim
+with a phony "origin" field.
+
+Spamming
+~~~~~~~~
+
+The attacker could try to send a high volume of solicicted or unsolicted
+messages to the victim in order to:
+
+* Find victims for scams.
+* Market unwanted products.
+
+Threat: Unsolicited Messages
+++++++++++++++++++++++++++++
+
+An attacker could try to send messages to victims who do not wish to receive
+them.
+
+Threat: Abusive Messages
+++++++++++++++++++++++++
+
+An attacker could send abusive or threatening messages to the victim
+
+Spying
+~~~~~~
+
+The attacker could try to access message contents or metadata for messages sent
+by the victim or to the victim that were not intended to reach the attacker in
+order to:
+
+* Gain sensitive personal or commercial information.
+* Impersonate the victim using credentials contained in the messages.
+  (e.g. password reset messages)
+* Discover who the victim was talking to and when.
+
+Threat: Disclosure during Transmission
+++++++++++++++++++++++++++++++++++++++
+
+An attacker could try to expose the message contents or metadata during
+transmission between the servers.
+
+Threat: Disclosure to Servers Outside Chatroom
+++++++++++++++++++++++++++++++++++++++++++++++
+
+An attacker could try to convince servers within a chatroom to send messages to
+a server it controls that was not authorised to be within the chatroom.
+
+Threat: Disclosure to Servers Within Chatroom
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+An attacker could take control of a server within a chatroom to expose message
+contents or metadata for messages in that room.
+
+
+Identity Servers
+================
+.. NOTE::
+  This section is a work in progress.
+
+.. TODO-doc Dave
+  - 3PIDs and identity server, functions
+
+Lawful Interception
+-------------------
+
+Key Escrow Servers
+~~~~~~~~~~~~~~~~~~