wholetrans/docs-matrix-spec
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Move drafts to attic to reduce confusion

Browse source
This commit is contained in:
Travis Ralston 2021-04-05 10:49:45 -06:00 • committed by Richard van der Hoff
parent f0e0477a3e
commit aab72c3d14
19 changed files with 0 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

11
attic/drafts/ancient_federated_versioning_design_notes.rst Normal file
View file

@ -0,0 +1,11 @@
Versioning is, like, hard for backfilling backwards because of the number of homeservers involved.
The way we solve this is by doing versioning as an acyclic directed graph of PDUs. For backfilling purposes, this is done on a per context basis.
When we send a PDU we include all PDUs that have been received for that context that hasn't been subsequently listed in a later PDU. The trivial case is a simple list of PDUs, e.g. A <- B <- C. However, if two servers send out a PDU at the same to, both B and C would point at A - a later PDU would then list both B and C.
Problems with opaque version strings:
- How do you do clustering without mandating that a cluster can only have one transaction in flight to a given remote homeserver at a time.
If you have multiple transactions sent at once, then you might drop one transaction, receive another with a version that is later than the dropped transaction and which point ARGH WE LOST A TRANSACTION.
- How do you do backfilling? A version string defines a point in a stream w.r.t. a single homeserver, not a point in the context.
We only need to store the ends of the directed graph, we DO NOT need to do the whole one table of nodes and one of edges.

244
attic/drafts/application_services.rst Normal file
View file

@ -0,0 +1,244 @@
.. TODO
Sometimes application services need to create rooms (e.g. when lazy loading
from room aliases). Created rooms need to have a user that created them, so
federation works (as it relies on an entry existing in m.room.member). We should
be able to add metadata to m.room.member to state that this user is an application
service, a virtual user, etc.
Application Services
====================
Overview
========
Application services provide a way of implementing custom serverside functionality
on top of Matrix without the complexity of implementing the full federation API.
By acting as a trusted service logically located behind an existing homeserver,
Application services are decoupled from:
* Signing or validating federated traffic or conversation history
* Validating authorisation constraints on federated traffic
* Managing routing or retry schemes to the rest of the Matrix federation
As such, developers can focus entirely on implementing application logic rather
than being concerned with the details of managing Matrix federation.
Features available to application services include:
* Privileged subscription to any events available to the homeserver
* Synthesising virtual users
* Synthesising virtual rooms
* Injecting message history for virtual rooms
Features not provided by application services include:
* Intercepting and filtering/modifying message or behaviour within a room
(this is a job for a Policy Server, as it requires a single logical focal
point for messages in order to consistently apply the custom business logic)
Example use cases for application services include:
* Exposing existing communication services in Matrix
* Gateways to/from standards-based protocols (SIP, XMPP, IRC, RCS (MSRP), SIMPLE, Lync, etc)
* Gateways to/from closed services (e.g. WhatsApp)
* Gateways could be architected as:
* Act as a virtual client on the non-Matrix network
(e.g. connect as multiple virtual clients to an IRC or XMPP server)
* Act as a server on the non-Matrix network
(e.g. speak s2s XMPP federation, or IRC link protocol)
* Act as an application service on the non-Matrix network
(e.g. link up as IRC services, or an XMPP component)
* Exposing a non-Matrix client interface listener from the AS
(e.g. listen on port 6667 for IRC clients, or port 5222 for XMPP clients)
* Bridging existing APIs into Matrix
* e.g. SMS/MMS aggregator APIs
* Domain-specific APIs such as SABRE
* Integrating more exotic content into Matrix
* e.g. MIDI<->Matrix gateway/bridge
* 3D world <-> Matrix bridge
* Application services:
* Search engines (e.g. elasticsearch search indices)
* Notification systems (e.g. send custom pushes for various hooks)
* VoIP Conference services
* Text-to-speech and Speech-to-text services
* Signal processing
* IVR
* Server-machine translation
* Censorship service
* Multi-User Gaming (Dungeons etc)
* Other "constrained worlds" (e.g. 3D geometry representations)
* applying physics to a 3D world on the serverside
* (applying gravity and friction and air resistance... collision detection)
* domain-specific merge conflict resolution of events
* Payment style transactional usecases with transactional guarantees
Architecture Outline
====================
The application service registers with its host homeserver to offer its services.
In the registration process, the AS provides:
* Credentials to identify itself as an approved application service for that HS
* Details of the namespaces of users and rooms the AS is acting on behalf of and
"subscribing to"
* Namespaces are defined as a list of regexps against which to match room aliases,
room IDs, and user IDs. Regexps give the flexibility to say, sub-domain MSISDN
ranges per AS, whereas a blunt prefix string does not. These namespaces are further
configured by setting whether they are ``exclusive`` or not. An exclusive namespace
prevents entities other than the aforementioned AS from creating/editing/deleting
entries within that namespace. This does not affect the visibility/readability of
entries within that namespace (e.g. it doesn't prevent users joining exclusive
aliases, or ASes from listening to exclusive aliases, but does prevent both users
and ASes from creating/editing/deleting aliases within that namespace).
* There is overlap between selecting events via the csv2 Filter API and subscribing
to events here - perhaps subscription involves passing a filter token into the
registration API.
* A URL base for receiving requests from the HS (as the AS is a server,
implementers expect to receive data via inbound requests rather than
long-poll outbound requests)
On HS handling events to unknown users:
* If the HS receives an event for an unknown user who is in the namespace delegated to
the AS, then the HS queries the AS for the profile of that user. If the AS
confirms the existence of that user (from its perspective), then the HS
creates an account to represent the virtual user.
* The namespace of virtual user accounts should conform to a structure like
``@.irc.freenode.Arathorn:matrix.org``. This lets Matrix users communicate with
foreign users who are not yet mapped into Matrix via 3PID mappings or through
an existing non-virtual Matrix user by trying to talk to them via a gateway.
* The AS can alternatively preprovision virtual users using the existing CS API
rather than lazy-loading them in this manner.
* The AS may want to link the matrix ID of the sender through to their 3PID in
the remote ecosystem. E.g. a message sent from ``@matthew:matrix.org`` may wish
to originate from Arathorn on irc.freenode.net in the case of an IRC bridge.
It's left as an AS implementation detail as to how the user should authorise
the AS to act on its behalf.
On HS handling events to unknown rooms:
* If the HS receives an invite to an unknown room which is in the namespace
delegated to the AS, then the HS queries the AS for the existence of that room.
If the AS confirms its existence (from its perspective), then the HS creates
the room.
* The initial state of the room may be populated by the AS by querying an
initialSync API (probably a subset of the CS initialSync API, to reuse the
same pattern for the equivalent function). As messages have to be signed
from the point of ``m.room.create``, we will not be able to back-populate
arbitrary history for rooms which are lazy-created in this manner, and instead
have to chose the amount of history to be synchronised into the AS as a one-off.
* If exposing arbitrary history is required, then:
* either: the room history must be preemptively provisioned in the HS by the AS via
the CS API (TODO: meaning the CS API needs to support massaged
timestamps), resulting in conversation history being replicated between
the HS and the source store.
* or: the HS must delegate conversation storage entirely to the
AS using a Storage API (not defined here) which allows the existing
conversation store to back the HS, complete with all necessary Matrix
metadata (e.g. hashes, signatures, federation DAG, etc). This obviously
increases the burden of implementing an AS considerably, but is the only
option if the implementer wants to avoid duplicating conversation history
between the external data source and the HS.
On HS handling events to existing users and rooms:
* If the HS receives an event for a user or room that already exists (either
provisioned by the AS or by normal client interactions), then the message
is handled as normal.
* Events in the namespaces of rooms and users that the AS has subscribed to
are pushed to the AS using the same pattern as the federation API (without
any of the encryption or federation metadata). This serves precisely the
same purpose as the CS event stream and has the same data flow semantics
(and indeed an AS implementer could chose to use the CS event stream instead)
* Events are linearised to avoid the AS having to handle the complexity of
linearisation, and because if linearisation is good enough for CS, it
should be good enough for AS. Should the AS require non-linearised events
from Matrix, it should implement the federation API rather than the AS API
instead.
* HS->AS event pushes are retried for reliability with sequence numbers
(or logical timestamping?) to preserve the linearisation order and ensure
a reliable event stream.
* Clustered HSes must linearise just as they do for the CS API. Clustered
ASes must loadbalance the inbound stream across the cluster as required.
On AS relaying events from unknown-to-HS users:
* AS injects the event to the HS using the CS API, irrespective of whether the
target user or room is known to the HS or not. If the HS doesn't recognise
the target it goes through the same lazy-load provisioning as per above.
* The reason for not using a subset of the federation API here is because it
allows AS developers to reuse existing CS SDKs and benefit from the more
meaningful error handling of the CS API. The sending user ID must be
explicitly specified, as it cannot be inferred from the access_token, which
will be the same for all AS requests.
* TODO: or do we maintain a separate ``access_token`` mapping? It seems like
unnecessary overhead for the AS developer; easier to just use a single
privileged ``access_token`` and just track which ``user_id`` is emitting events?
* If the AS is spoofing the identity of a real (not virtual) matrix user,
we should actually let them log themselves in via OAuth2 to give permission
to the AS to act on their behalf.
* We can't auth gatewayed virtual users from 3rd party systems who are being
relayed into Matrix, as the relaying is happening whether the user likes it
or not. Therefore we do need to be able to spoof sender ID for virtual users.
On AS relaying events in unknown-to-HS rooms:
* See above.
On AS publishing aliases for virtual rooms:
* AS uses the normal alias management API to preemptively create/delete public
directory entries for aliases for virtual rooms provided by the AS.
* In order to create these aliases, the underlying room ID must also exist, so
at least the ``m.room.create`` of that room must also be prepopulated. It seems
sensible to prepopulate the required initial state and history of the room to
avoid a two-phase prepopulation process.
On unregistering the AS from the HS:
* An AS must tell the HS when it is going offline in order to stop receiving
requests from the HS. It does this by hitting an API on the HS.
AS Visibility:
* If an AS needs to sniff events in a room in order to operate on them (e.g.
to act as a search engine) but not inject traffic into the room, it should
do so by subscribing to the relevant events without actually joining the room.
* If the AS needs to participate in the room as a virtual user (e.g. an IVR
service, or a bot, or a gatewayed virtual user), it should join the room
normally.
* There are rare instances where an AS may wish to participate in a room
(including inserting messages), but be hidden from the room list - e.g. a
conferencing server focus bot may wish to join many rooms as the focus and
both listen to VoIP setups and inject its own VoIP answers, without ever
being physically seen in the room. In this scenario, the user should set
its presence to 'invisible', a state that HSes should only allow AS-authed
users to set.
E2E Encryption
* The AS obviously has no visibility to E2E encrypted messages, unless it is
explicitly added to an encrypted room and participates in the group chat
itself.
Extensions to CS API
====================
* Ability to assert the identity of the virtual user for all methods.
* Ability to massage timestamps when prepopulating historical state and
messages of virtual rooms (either by overriding ``origin_server_ts`` (preferred) or
adding an ``as_ts`` which we expect clients to honour)
* Ability to delete aliases (including from the directory) as well as create them.

222
attic/drafts/data_flows.rst Normal file
View file

@ -0,0 +1,222 @@
Data flows for use cases
========================
::
<- Data from server to client
-> Data from client to server
Instant Messaging
-----------------
Without storage
~~~~~~~~~~~~~~~
::
Home screen
Data required on load:
<- For each room the user is joined: Name, topic, # members, last message, room ID, aliases
Data required when new message arrives for a room:
<- Room ID, message content, sender (user ID, display name, avatar url)
Data required when someone invites you to a room:
<- Room ID, sender (user ID, display name, avatar url), Room Name, Room Topic
Data required when you leave a room on another device:
<- Room ID
Data required when you join a room on another device:
<- Name, topic, # members, last message, room ID, aliases
Data required when your profile info changes on another device:
<- new profile info e.g. avatar, display name, etc.
Creating a room
-> Invitee list of user IDs, public/private, name of room, alias of room, topic of room
<- Room ID
Joining a room (and dumped into chat screen on success)
-> Room ID / Room alias
<- Room ID, Room aliases (plural), Name, topic, member list (f.e. member: user ID,
avatar, presence, display name, power level, whether they are typing), enough
messages to fill screen (and whether there are more)
Chat Screen
Data required when member name changes:
<- new name, room ID, user ID, when in the context of the room did this occur
Data required when the room name changes:
<- new name, room ID, old room name?
Invite a user:
-> user ID, room ID
<- display name / avatar of user invited (if known)
Kick a user:
-> user ID, room ID
<- what message it came after
Leave a room:
-> room ID
<- what message it came after
Send a message
-> Message content, room ID, message sequencing (eg sending my 1st, 2nd, 3rd msg)
<- actual content sent (if server mods it), what message it comes after (to correctly
display the local echo)
Place a call (receive a call is just reverse)
<- turn servers
-> SDP offer
-> Ice candidates (1 by 1; trickling)
<- SDP answer
<- Ice candidates
Scrolling back (infinite scrolling)
-> Identifier for the earliest message, # requested messages
<- requested messages (f.e change in display name, what the old name was), whether
there are more.
With storage
~~~~~~~~~~~~
::
Home Screen
On Load
-> Identifier which tells the server the client's current state (which rooms it is aware
of, which messages it has, what display names for users, etc..)
<- A delta from the client's current state to the current state on the server (e.g. the
new rooms, the *latest* message if different, the changed display names, the new
invites, etc). f.e Room: Whether the cache of the room that you have has been replaced
with this new state.
Pre-load optimisation (not essential for this screen)
-> Number of desired messages f.e room to cache
<- f.e Room: the delta OR the entire state
Bug Tracking
------------
::
Landing Page
On Load
<- Issues assigned to me, Issues I'm watching, Recent activity on other issues includes
comments, list of projects
Search for an issue (assume text)
-> Search string
<- List of paginated issues
Request page 2:
-> Page number requested
<- Page of paginated issues
Issue Page
On Load
-> Issue ID and Project ID (equiv to Room)
<- Issue contents e.g. priority, resolution state, etc. All comments e.g. user ID,
comment text, timestamp. Entire issue history e.g. changes in priority
Post a comment
-> Issue ID, comment content, Project ID (equiv to Room)
<- actual content sent (if modded), what comment it comes after
Set issue priority
-> Issue ID, Project ID, desired priority
<- What action in the history it came after
Someone else sets issue priority
<- Issue ID, Project ID, new priority, where in the history
Mapping model use cases to matrix models (Room, Message, etc)
=============================================================
To think about:
- Do we want to support the idea of forking off new rooms from existing ones? This
and forums could benefit from it.
Bug tracking UI
---------------
::
Projects => Rooms
Issues => Message Events
Comments => Message Events (relates_to key)
Projects:
- Unlikely that there will be 100,000s of issues, so having to pull in all the issues for a project is okay.
- Permissions are usually per project and this Just Works.
- New issues come in automatically and Just Work.
- Can have read-only members
Issues:
- Don't really want 1 Room per Issue, else you can have thousands of Rooms PER PROJECT, hence choice for
Issues as Messages. Don't need to join a room for each issue.
- Idea of issue owner is clear (sender of the message)
- Updating issues requires an additional event similar to comments (with ``relates_to``)? Could possibly
be state events? Don't really want all the history if say the priority was changed 1000 times, just want
the current state of the key.
Comments:
- Additional event with ``relates_to`` key.
Forum
-----
::
Forum => Room (with pointers to Board Rooms)
Boards => Room (with pointers to Thread Rooms)
Threads => Room
Messages => Message Events
Forum:
- Contains 10s of Boards.
- Contains special Message Events which point to different rooms f.e Board.
Boards:
- Contains 100s of Threads.
- Contains special Message Events which point to different rooms f.e. Thread.
Threads:
- Contains 100s of Messages.
Can't do this nicely with the current Federation API because you have loads of
Rooms and what does posting a message look like? Creating a thread is done by..?
The user who is posting cannot create the thread because otherwise they would be
the room creator and have ultimate privileges. So it has to be created by a bot
of some kind which ties into auth (Application services?). To follow a board,
you need a bot to join the Board Room and then watch it for changes...
Fundamental problem with forums is that there is only 1 PDU graph per room and
you either have to pull in lots of graphs separately or one graph and filter it
separately to get to the desired sub set of data. You have to subscribe into a
lot of graphs if you subscribe to a board... If you have the entire board...
good luck scrollbacking a particular thread.
Google+ Community
-----------------
::
Community => Room (with pointers to Category Rooms)
Category => Room
Post => Message Events
Comment => Message Events (relates_to key)
Community:
- Contains 10s of categories.
- Contains special Message Events which point to different rooms f.e Category.
- Moderators of the community are mods in this room. They are in charge of making
new categories and the subsequent rooms. Can get a bit funky if a mod creates a
category room without the same permissions as the community room... but another
mod can always delete the pointer to the buggy category room and make a new one.
- Do we want to support the idea of forking off new rooms from existing ones? This
and forums could benefit from it.
Category:
- Contains 1000s of posts.
- Same permissions as the community room. How to enforce? Fork off the community
room?
Posts:
- Contains 10s of comments.
This is similar to forums but you can more reasonably say "screw it, pull in the
entire community of posts."

141
attic/drafts/erik-model.rst Normal file
View file

@ -0,0 +1,141 @@
This is a standalone description of the data architecture of Synapse. There is a
lot of overlap with the current specification, so it has been split out here for
posterity. Hopefully all the important bits have been merged into the relevant
places in the main spec.
Model
-----
Overview
~~~~~~~~
Matrix is used to reliably distribute data between sets of `users`.
Users are associated with one of many matrix `servers`. These distribute,
receive and store data on behalf of its registered users. Servers can be run on
any host accessible from the internet.
When a user wishes to send data to users on different servers the local server
will distribute the data to each remote server. These will in turn distribute
to their local users involved.
A user sends and receives data using one or more authenticated `clients`
connected to his server. Clients may persist data locally or request it when
required from the server.
Events
~~~~~~
An event is a collection of data (the `payload`) and metadata to be distributed
across servers and is the primary data unit in Matrix. Events are extensible
so that clients and servers can add extra arbitrary fields to both the payload
or metadata.
Events are distributed to interested servers upon creation. Historical events
may be requested from servers; servers are not required to produce all
or any events requested.
All events have a metadata `type` field that is used by client and servers to
determine how the payload should be processed and used. There are a number of
types reserved by the protocol for particular uses, but otherwise types may be
defined by applications, clients or servers for their own purposes.
.. TODO : Namespacing of new types.
Graph
+++++
Each event has a list of zero or more `parent` events. These relations form
directed acyclic graphs of events called `event graphs`. Every event graph has
a single root event, and each event graph forms the basis of the history of a
matrix room.
Event graphs give a partial ordering of events, i.e. given two events one may
be considered to have come before the other if one is an ancestor of the other.
Since two events may be on separate branches, not all events can be compared in
this manner.
Every event has a metadata `depth` field that is a positive integer that is
strictly greater than the depths of any of its parents. The root event should
have a depth of 1.
[Note: if one event is before another, then it must have a strictly smaller
depth]
Integrity
+++++++++
.. TODO: Specify the precise subset of essential fields
Portions of events will be signed by one or more servers or clients. The parent
relations, type, depth and payload (as well as other metadata fields that will
be specified) must be signed by the originating server. [Note: Thus, once an
event is distributed and referenced by later events, they effectively become
immutable].
The payload may also be encrypted by clients, except in the case where the
payload needs to be interpreted by the servers. A list of event types that
cannot have an encrypted payload are given later.
State
~~~~~
Event graphs may have meta information associated with them, called `state`.
State can be updated over time by servers or clients, subject to
authorisation.
The state of a graph is split into `sections` that can be atomically updated
independently of each other.
State is stored within the graph itself, and can be computed by looking at the
graph in its entirety. We define the state at a given event to be the state of
the sub graph of all events "before" and including that event.
Some sections of the state may determine behaviour of the protocol, including
authorisation and distribution. These sections must not be encrypted.
State Events
++++++++++++
`State events` are events that update a section of state data for a room. These
state events hold all the same properties of events, and are part of the event
graph. The payload of the event is the replacement value for the particular
section of state being updated.
State events must also include a `state_key` metadata field. The pair of fields
type and state_key uniquely defines the section of state that is to be updated.
State Resolution
++++++++++++++++
A given state section may have multiple state events associated with it in a
given graph. A consistent method of selecting which state event takes
precedence is therefore required.
This is done by taking the latest state events, i.e. the set of events that are
either incomparable or after every other event in the graph. A state resolution
algorithm is then applied to this set to select the single event that takes
precedence.
The state resolution algorithm must be transitive and not depend on server
state, as it must consistently select the same event irrespective of the server
or the order the events were received in.
State Dictionary
++++++++++++++++
The state dictionary is the mapping from sections of state to the state events
which set the section to its current value. The state dictionary, like the
state itself, depends on the events currently in the graph and so is updated
with each new event received.
Since the sections of the state are defined by the pair of strings from the
type and state_key of the events that update them, the state dictionary can be
defined as a mapping from the pair (type, state_key) to a state event with
those values in the graph.
Deleting State
++++++++++++++
State sections may also be deleted, i.e. removed from the state dictionary. The
state events will still be present in the event graph.
This is done by sending a special state event indicating that the given entry
should be removed from the dictionary. These events follow the same rules for
state resolution, with the added requirement that it loses all conflicts.
[Note: This is required to make the algorithm transitive.]

253
attic/drafts/erikj_federation.rst Normal file
View file

@ -0,0 +1,253 @@
Federation
==========
.. sectnum::
.. contents:: Table of Contents
Auth chain
~~~~~~~~~~
The *auth chain* for an event is the recursive list of auth events and the auth
chain for those auth events.
.. Note:: The auth chain for an event gives all the information a server needs
to accept an event. However, being given an auth chain for an event
that appears valid does not mean that the event might not later be
rejected. For example if we discover that the sender had been banned
between the join event listed in the auth events and the event being
authed.
**TODO**: Clean the above explanations up a bit.
Auth chain resolution
~~~~~~~~~~~~~~~~~~~~~
If an auth check fails, or if we get told something we accepted should have
been rejected, we need to try and determine who is right.
If two servers disagree about the validity of the auth events, both should
inform the other of what they think the current auth chain is. If either are
missing auth events that they know are valid (through authorization and state
resolution) they process the missing events as usual.
If either side notice that the other has accepted an auth events we think
should be rejected (for reasons *not* in their auth chain), that server should
inform the other with suitable proof.
The proofs can be:
- An *event chain* that shows an auth event is *not* an ancestor of the event.
This can be done by giving the full ancestor chains up to the depth of the
invalid auth event.
- Given an event (and event chain?) showing that authorization had been revoked.
If a server discovers it cannot prove the other side is wrong, then it accepts
that the other is correct; i.e. we always accept that the other side is correct
unless we can prove otherwise.
Constructing a new event
------------------------
**TODO**
When constructing a new event, the server should insert the following fields:
- ``prev_events``: The list of event ids of what the server believes are the
current leaf nodes of the event graph (i.e., nodes that have been received
but are yet to be referenced by another event).
- ``depth``: An integer one greater than the maximum depth of the event's
previous events.
- ``auth_events``: The list of event ids that authorizes this event. This
should be a subset of the current state.
- ``origin_server_ts``: The time the server created the event.
- ``origin``: The name of the server.
Signing and Hashes
~~~~~~~~~~~~~~~~~~
**TODO**
Validation
----------
**TODO**
Domain specific string
A string of the form ``<prefix><localpart>:<domain>``, where <prefix> is a
single character, ``<localpart>`` is an arbitrary string that does not
include a colon, and `<domain>` is a valid server name.
``room_id``
A domain specific string with prefix ``!`` that is static across all events
in a graph and uniquely identifies it. The ``domain`` should be that of the
homeserver that created the room (i.e., the server that generated the
first ``m.room.create`` event).
``sender``
The entity that logically sent the event. This is usually a user id, but
can also be a server name.
User Id
A domain specific string with prefix ``@`` representing a user account. The
``domain`` is the homeserver of the user and is the server used to contact
the user.
Joining a room
--------------
If a user requests to join a room that the server is already in (i.e. the a
user on that server has already joined the room) then the server can simply
generate a join event and send it as normal.
If the server is not already in the room it needs to will need to join via
another server that is already in the room. This is done as a two step process.
First, the local server requests from the remote server a skeleton of a join
event. The remote does this as the local server does not have the event graph
to use to fill out the ``prev_events`` key in the new event. Critically, the
remote server does not process the event it responded with.
Once the local server has this event, it fills it out with any extra data and
signs it. Once ready the local server sends this event to a remote server
(which could be the same or different from the first remote server), this
remote server then processes the event and distributes to all the other
participating servers in that room. The local server is told about the
current state and complete auth chain for the join event. The local server
can then process the join event itself.
.. Note::
Finding which server to use to join any particular room is not specified.
Inviting a user
---------------
To invite a remote user to a room we need their homeserver to sign the invite
event. This is done by sending the event to the remote server, which then signs
the event, before distributing the invite to other servers.
Handling incoming events
------------------------
When a server receives an event, it should:
#. Check if it knows about the room. If it doesn't, then it should get the
current state and auth events to determine whether the server *should* be in
the room. If so continue, if not drop or reject the event
#. If the server already knew about the room, check the prev events to see if
it is missing any events. If it is, request them. Servers should limit how
far back they will walk the event graph for missing events.
#. If the server does not have all the prev events, then it should request the
current state and auth events from a server.
Failures
--------
A server can notify a remote server about something it thinks it has done
wrong using the failures mechanism. For example, the remote accepted an event
the local think it shouldn't have.
A failure has a severity level depending on the action taken by the local
server. These levels are:
``FATAL``
The local server could not parse the event, for example due to a missing
required field.
``ERROR``
The local server *could* parse the event, but it was rejected. For example,
the event may have failed an authorization check.
``WARN``
The local server accepted the event, but something was unexpected about it.
For example, the event may have referenced another event the local server
thought should be rejected.
A failure also includes several other fields:
``code``
A numeric code (to be defined later) indicating a particular type of
failure.
``reason``
A short string indicating what was wrong, for diagnosis purposes on the
remote server.
``affected``
The event id of the event this failure is responding to. For example, if
an accepted event referenced a rejected event, this would point to the
accepted one.
``source``
The event id of the event that was the source of this unexpected behaviour.
For example, if an accepted event referenced a rejected event, this would
point to the rejected one.
Appendix
========
**TODO**
Example event:
.. code::
{
"auth_events": [
[
"$14187571482fLeia:localhost:8480",
{
"sha256": "kiZUclzzPetHfy0rVoYKnYXnIv5VxH8a4996zVl8xbw"
}
],
[
"$14187571480odWTd:localhost:8480",
{
"sha256": "GqtndjviW9yPGaZ6EJfzuqVCRg5Lhoyo4YYv1NFP7fw"
}
],
[
"$14205549830rrMar:localhost:8480",
{
"sha256": "gZmL23QdWjNOmghEZU6YjqgHHrf2fxarKO2z5ZTbkig"
}
]
],
"content": {
"body": "Test!",
"msgtype": "m.text"
},
"depth": 250,
"event_id": "$14207181140uTFlx:localhost:8480",
"hashes": {
"sha256": "k1nuafFdFvZXzhb5NeTE0Q2Jkqu3E8zkh3uH3mqwIxc"
},
"origin": "localhost:8480",
"origin_server_ts": 1420718114694,
"prev_events": [
[
"$142071809077XNNkP:localhost:8480",
{
"sha256": "xOnU1b+4LOVz5qih0dkNFrdMgUcf35fKx9sdl/gqhjY"
}
]
],
"room_id": "!dwZDafgDEFTtpPKpLy:localhost:8480",
"sender": "@bob:localhost:8480",
"signatures": {
"localhost:8480": {
"ed25519:auto": "Nzd3D+emFBJJ4LCTzQEZaKO0Sa3sSTR1fGpu8OWXYn+7XUqke9Q1jYUewrEfxb3lPxlYWm/GztVUJizLz1K5Aw"
}
},
"type": "m.room.message",
"unsigned": {
"age": 500
}
}

146
attic/drafts/flows_and_auth.rst Normal file
View file

@ -0,0 +1,146 @@
Failures
--------
A server may encouter an error when trying to process an event received from a
remote server. In these cases the server may send a `failure` to the remote.
A `failure` references both the event that it was trying to process and the
event that triggered the processing. For example, a failure may be emitted if
one of the parents of the received events was not authorized.
A failure also includes a `severity` field that indicates what action was taken
by the server. There are three valid values:
* `Fatal`: The server failed to parse the event. The event is dropped by the
server as well as all descendants.
* `Error`: The server rejected the event, for example due to authorization.
That event is dropped, but descendants may be accepted.
* `Warn`: The server accepted all events, but believes the remote did
something wrong. For example, references an event the local server believes
is unauthorized.
Data Flows
----------
Invite
++++++
To invite a remote user to an existing room a server distributes an invitiation
event signed by the remote invitee's server (allowing other servers in the room
to be sure that the invitee's server had seen the invite.)
To get the remote server's signature on the event it is sent in a special
request to the remote server, which then responds with the signed invite (if it
accepted it as valid.) The remote server may respond with an error if the user
does not exist.
Join
++++
If a server is already in the room it can simply emit a join event for the user
joining.
If the server is not currently in the room it needs to join via a remote server
in the room, therefore to join a room a server must have have both the room id
and a list of remote servers that may be in the room.
To join via a remote server the local server first requests a valid join event
for the room from the remote server. The local then fills it out, signs it, and
then sends the final join event to the remote server for distribution. The
remore responds to this second request with the current state of the room at
the join request and the auth chain for that state.
Authorization
-------------
The authorization for an event depends solely on the current state at that
event. If a policy server is configured for the room, then the authorization
for the event is the signature of the policy server.
The state events that affect whether an event is authorized are called
`protocol events`, and are:
* `m.room.create`
* `m.room.power_levels`
* `m.room.member`
* `m.room.join_rules`
All events *must* list all the protocol events that grant them their
authorization. All origin servers *must* serve up on request the full graph of
protocol events for all events it has sent. The graph of protocol events is the
connected directed acyclic graph with events as nodes and the list of protocol
events their edges.
Join
++++
A user may join a room if:
* The join rule is "public".
* The join rule is "invite" and the user has been invited by a user that has
already joined.
* The user is in the `may_join` list.
Invite
++++++
A user may invite another user if the join rule is either "public" or "invite"
and the user has joined the room.
Creation
++++++++
A `m.room.create` must be the first event in the room.
Ban, Kick and Redaction
+++++++++++++++++++++++
To ban or kick another user in the room, or to redact an event, then the user
must have a power level of at least that specificied in the
`m.room.power_level` event for kick, ban and redactions.
Other Events
++++++++++++
A user may send an event if all the following hold true:
* The user is in the room.
* If the type of the event is listed in the `m.room.power_levels`, then the
user must have at least that power level. Otherwise, the user must have a
power level of at least `events_default` or `state_default`, depending on
if the event is a message or state event respectively.
Unauthorized Events
-------------------
An unauthorized event should not be accepted into the event graph, i.e. new
events should not reference any unauthorized events. There are situations where
this can happen and so it is not considered an error to include an unauthorized
event in the event graph. It is an error for events to refer unauthorized
events in their `auth_events` section and will in turn be considered
unauthorized.
A server may choose to store only the redacted form of an unauthorized event if
it is included in the event graph.
A server may emit a warning to a remote server if it references an event it
considers unauthorized.
State and Authorization Querying
--------------------------------
A local server may become aware that it and a remote server's view of the
current state are inconsistent. The local server may then send its current
state to the remote, which responds with its view of the current state. Both
servers should then recompute the local state. If they are conforming
implementations then they will reach the same conclusions.

1186
attic/drafts/general_api.rst Normal file
View file

File diff suppressed because it is too large Load diff

133
attic/drafts/human-id-rules.rst Normal file
View file

@ -0,0 +1,133 @@
Abstract
========
This document outlines a proposed format for human-readable IDs within Matrix.
For status see https://github.com/matrix-org/matrix-doc/pull/3/files
Background
----------
UTF-8 is the dominant character encoding for Unicode on the web. However,
using Unicode as the character set for human-readable IDs is troublesome. There
are many different characters which appear identical to each other, but would
produce different IDs. In addition, there are non-printable characters which
cannot be rendered by the end-user. This creates an opportunity for
phishing/spoofing of IDs, commonly known as a homograph attack.
Web browsers encountered this problem when International Domain Names were
introduced. A variety of checks were put in place in order to protect users. If
an address failed the check, the raw punycode would be displayed to
disambiguate the address.
The human-readable IDs in Matrix are Room Aliases and User IDs.
Room aliases look like ``#localpart:domain``. These aliases point to opaque
non human-readable room IDs. These pointers can change to point at a different
room ID at any time. User IDs look like ``@localpart:domain``. These represent
actual end-users (there is no indirection).
Proposal
========
User IDs and Room Aliases MUST be Unicode as UTF-8. Checks are performed on
these IDs by homeservers to protect users from phishing/spoofing attacks.
These checks are:
User ID Localparts:
- MUST NOT contain a ``:`` or start with a ``@`` or ``.``
- MUST NOT contain one of the 107 blacklisted characters on this list:
http://kb.mozillazine.org/Network.IDN.blacklist_chars
- After stripping " 0-9, +, -, [, ], _, and the space character it MUST NOT
contain characters from >1 language, defined by the `exemplar characters`_
on http://cldr.unicode.org/
.. _exemplar characters: http://cldr.unicode.org/translation/characters#TOC-Exemplar-Characters
Room Alias Localparts:
- MUST NOT contain a ``:``
- MUST NOT contain one of the 107 blacklisted characters on this list:
http://kb.mozillazine.org/Network.IDN.blacklist_chars
- After stripping " 0-9, +, -, [, ], _, and the space character it MUST NOT
contain characters from >1 language, defined by the `exemplar characters`_
on http://cldr.unicode.org/
.. _exemplar characters: http://cldr.unicode.org/translation/characters#TOC-Exemplar-Characters
In the event of a failed user ID check, well behaved homeservers MUST:
- Rewrite user IDs in the offending events to be punycode with an additional ``@``
prefix **before** delivering them to clients. There are no guarantees for
consistency between homeserver ID checking implementations. As a result, user
IDs MUST be sent in their *original* form over federation. This can be done in
a stateless manner as the punycode form has no information loss.
In the event of a failed room alias check, well behaved homeservers MUST:
- Send an HTTP status code 400 with an ``errcode`` of ``M_FAILED_HUMAN_ID_CHECK``
to the client if the client is attempting to *create* this alias.
- Send an HTTP status code 400 with an ``errcode`` of ``M_FAILED_HUMAN_ID_CHECK``
to the client if the client is attempting to *join* a room via this alias.
Examples::
@ebаy:example.org (Cyrillic 'a', everything else English)
@@xn--eby-7cd:example.org (Punycode with additional '@')
Homeservers SHOULD NOT allow two user IDs that differ only by case. This
SHOULD be applied based on the capitalisation rules in the CLDR dataset:
http://cldr.unicode.org/
This check SHOULD be applied when the user ID is created, in order to prevent
registration with the same name and different capitalisations, e.g.
``@foo:bar`` vs ``@Foo:bar`` vs ``@FOO:bar``. Homeservers MAY canonicalise
the user ID to be completely lower-case if desired.
Rationale
=========
Each ID is split into segments (localpart/domain) around the ``:``. For
this reason, ``:`` is a reserved character and cannot be a localpart character.
The 107 blacklisted characters are used to prevent non-printable characters and
spaces from being used. The decision to ban characters from more than 1 language
matches the behaviour of `Google Chrome for IDN handling`_. This is to protect
against common homograph attacks such as ebаy.com (Cyrillic "a", rest is
English). This would always result in a failed check. Even with this though
there are limitations. For example, сахар is entirely Cyrillic, whereas caxap is
entirely Latin.
.. _Google Chrome for IDN handling: https://www.chromium.org/developers/design-documents/idn-in-google-chrome
User ID localparts cannot start with ``@`` so that a namespace of localparts
beginning with ``@`` can be created. This namespace is used for user IDs which
fail the ID checks. A failed ID could look like ``@@xn--c1yn36f:example.org``.
If a user ID fails the check, the user ID on the event is renamed. This doesn't
require extra work for clients, and users will see an odd user ID rather than a
spoofed name. Renaming is done in order to protect users of a given HS, so if a
malicious HS doesn't rename their IDs, it doesn't affect any other HS.
Room aliases cannot be rewritten as punycode and sent to the HS the alias is
referring to as the HS will not necessarily understand the rewritten alias.
Other rejected solutions for failed checks
------------------------------------------
- Additional key: Informational key on the event attached by HS to say "unsafe
ID". Problem: clients can just ignore it, and since it will appear only very
rarely, easy to forget when implementing clients.
- Require client handshake: Forces clients to implement
a check, else they cannot communicate with the misleading ID. However, this
is extra overhead in both client implementations and round-trips.
- Reject event: Outright rejection of the ID at the point of creation /
receiving event. Point of creation rejection is preferable to avoid the ID
entering the system in the first place. However, malicious HSes can just
allow the ID. Hence, other homeservers must reject them if they see them in
events. Client never sees the problem ID, provided the HS is correctly
implemented. However, it is difficult to ensure that ALL HSes will come to the
same conclusion (given the CLDR dataset does come out with new versions).
Outstanding Problems
====================
Capitalisation
--------------
The capitalisation rules outlined above are nice but do not fully resolve issues
where ``@alice:example.com`` tries to speak with ``@bob:example.org`` using
``@Bob:example.org``. It is up to ``example.org`` to map ``Bob`` to ``bob`` in
a sensible way.

149
attic/drafts/model/presence.rst Normal file
View file

@ -0,0 +1,149 @@
API Efficiency
==============
A simple implementation of presence messaging has the ability to cause a large
amount of Internet traffic relating to presence updates. In order to minimise
the impact of such a feature, the following observations can be made:
* There is no point in a homeserver polling status for peers in a user's
presence list if the user has no clients connected that care about it.
* It is highly likely that most presence subscriptions will be symmetric - a
given user watching another is likely to in turn be watched by that user.
* It is likely that most subscription pairings will be between users who share
at least one Room in common, and so their homeservers are actively
exchanging message PDUs or transactions relating to that Room.
* Presence update messages do not need realtime guarantees. It is acceptable to
delay delivery of updates for some small amount of time (10 seconds to a
minute).
The general model of presence information is that of a HS registering its
interest in receiving presence status updates from other HSes, which then
promise to send them when required. Rather than actively polling for the
current state all the time, HSes can rely on their relative stability to only
push updates when required.
A homeserver should not rely on the longterm validity of this presence
information, however, as this would not cover such cases as a user's server
crashing and thus failing to inform their peers that users it used to host are
no longer available online. Therefore, each promise of future updates should
carry with a timeout value (whether explicit in the message, or implicit as some
defined default in the protocol), after which the receiving HS should consider
the information potentially stale and request it again.
However, because of the likelihood that two homeservers are exchanging messages
relating to chat traffic in a room common to both of them, the ongoing receipt
of these messages can be taken by each server as an implicit notification that
the sending server is still up and running, and therefore that no status changes
have happened; because if they had the server would have sent them. A second,
larger timeout should be applied to this implicit inference however, to protect
against implementation bugs or other reasons that the presence state cache may
become invalid; eventually the HS should re-enquire the current state of users
and update them with its own.
The following workflows can therefore be used to handle presence updates:
1 When a user first appears online their HS sends a message to each other HS
containing at least one user to be watched; each message carrying both a
notification of the sender's new online status, and a request to obtain and
watch the target users' presence information. This message implicitly
promises the sending HS will now push updates to the target HSes.
2 The target HSes then respond a single message each, containing the current
status of the requested user(s). These messages too implicitly promise the
target HSes will themselves push updates to the sending HS.
As these messages arrive at the sending user's HS they can be pushed to the
user's client(s), possibly batched again to ensure not too many small
messages which add extra protocol overheads.
At this point, all the user's clients now have the current presence status
information for this moment in time, and have promised to send each other
updates in future.
3 The HS maintains two watchdog timers per peer HS it is exchanging presence
information with. The first timer should have a relatively small expiry
(perhaps 1 minute), and the second timer should have a much longer time
(perhaps 1 hour).
4 Any time any kind of message is received from a peer HS, the short-term
presence timer associated with it is reset.
5 Whenever either of these timers expires, an HS should push a status reminder
to the target HS whose timer has now expired, and request again from that
server the status of the subscribed users.
6 On receipt of one of these presence status reminders, an HS can reset both
of its presence watchdog timers.
To avoid bursts of traffic, implementations should attempt to stagger the expiry
of the longer-term watchdog timers for different peer HSes.
When individual users actively change their status (either by explicit requests
from clients, or inferred changes due to idle timers or client timeouts), the HS
should batch up any status changes for some reasonable amount of time (10
seconds to a minute). This allows for reduced protocol overheads in the case of
multiple messages needing to be sent to the same peer HS; as is the likely
scenario in many cases, such as a given human user having multiple user
accounts.
API Requirements
================
The data model presented here puts the following requirements on the APIs:
Client-Server
-------------
Requests that a client can make to its homeserver
* get/set current presence state
Basic enumeration + ability to set a custom piece of text
* report per-device idle time
After some (configurable?) idle time the device should send a single message
to set the idle duration. The HS can then infer a "start of idle" instant and
use that to keep the device idleness up to date. At some later point the
device can cancel this idleness.
* report per-device type
Inform the server that this device is a "mobile" device, or perhaps some
other to-be-defined category of reduced capability that could be presented to
other users.
* start/stop presence polling for my presence list
It is likely that these messages could be implicitly inferred by other
messages, though having explicit control is always useful.
* get my presence list
[implicit poll start?]
It is possible that the HS doesn't yet have current presence information when
the client requests this. There should be a "don't know" type too.
* add/remove a user to my presence list
Server-Server
-------------
Requests that homeservers make to others
* request permission to add a user to presence list
* allow/deny a request to add to a presence list
* perform a combined presence state push and subscription request
For each sending user ID, the message contains their new status.
For each receiving user ID, the message should contain an indication on
whether the sending server is also interested in receiving status from that
user; either as an immediate update response now, or as a promise to send
future updates.
Server to Client
----------------
[[TODO(paul): There also needs to be some way for a user's HS to push status
updates of the presence list to clients, but the general server-client event
model currently lacks a space to do that.]]

231
attic/drafts/model/profiles.rst Normal file
View file

@ -0,0 +1,231 @@
========
Profiles
========
A description of Synapse user profile metadata support.
Overview
========
Internally within Synapse users are referred to by an opaque ID, which consists
of some opaque localpart combined with the domain name of their homeserver.
Obviously this does not yield a very nice user experience; users would like to
see readable names for other users that are in some way meaningful to them.
Additionally, users like to be able to publish "profile" details to inform other
users of other information about them.
It is also conceivable that since we are attempting to provide a
worldwide-applicable messaging system, that users may wish to present different
subsets of information in their profile to different other people, from a
privacy and permissions perspective.
A Profile consists of a display name, an (optional?) avatar picture, and a set
of other metadata fields that the user may wish to publish (email address, phone
numbers, website URLs, etc...). We put no requirements on the display name other
than it being a valid Unicode string. Since it is likely that users will end up
having multiple accounts (perhaps by necessity of being hosted in multiple
places, perhaps by choice of wanting multiple distinct identifies), it would be
useful that a metadata field type exists that can refer to another Synapse User
ID, so that clients and HSes can make use of this information.
Metadata Fields
---------------
[[TODO(paul): Likely this list is incomplete; more fields can be defined as we
think of them. At the very least, any sort of supported ID for the 3rd Party ID
servers should be accounted for here.]]
* Synapse Directory Server username(s)
* Email address
* Phone number - classify "home"/"work"/"mobile"/custom?
* Twitter/Facebook/Google+/... social networks
* Location - keep this deliberately vague to allow people to choose how
granular it is
* "Bio" information - date of birth, etc...
* Synapse User ID of another account
* Web URL
* Freeform description text
Visibility Permissions
======================
A homeserver implementation could offer the ability to set permissions on
limited visibility of those fields. When another user requests access to the
target user's profile, their own identity should form part of that request. The
HS implementation can then decide which fields to make available to the
requestor.
A particular detail of implementation could allow the user to create one or more
ACLs; where each list is granted permission to see a given set of non-public
fields (compare to Google+ Circles) and contains a set of other people allowed
to use it. By giving these ACLs strong identities within the HS, they can be
referenced in communications with it, granting other users who encounter these
the "ACL Token" to use the details in that ACL.
If we further allow an ACL Token to be present on Room join requests or stored
by 3PID servers, then users of these ACLs gain the extra convenience of not
having to manually curate people in the access list; anyone in the room or with
knowledge of the 3rd Party ID is automatically granted access. Every HS and
client implementation would have to be aware of the existence of these ACL
Token, and include them in requests if present, but not every HS implementation
needs to actually provide the full permissions model. This can be used as a
distinguishing feature among competing implementations. However, servers MUST
NOT serve profile information from a cache if there is a chance that its limited
understanding could lead to information leakage.
Client Concerns of Multiple Accounts
====================================
Because a given person may want to have multiple Synapse User accounts, client
implementations should allow the use of multiple accounts simultaneously
(especially in the field of mobile phone clients, which generally don't support
running distinct instances of the same application). Where features like address
books, presence lists or rooms are presented, the client UI should remember to
make distinct with user account is in use for each.
Directory Servers
=================
Directory Servers can provide a forward mapping from human-readable names to
User IDs. These can provide a service similar to giving domain-namespaced names
for Rooms; in this case they can provide a way for a user to reference their
User ID in some external form (e.g. that can be printed on a business card).
The format for Synapse user name will consist of a localpart specific to the
directory server, and the domain name of that directory server:
@localname:some.domain.name
The localname is separated from the domain name using a colon, so as to ensure
the localname can still contain periods, as users may want this for similarity
to email addresses or the like, which typically can contain them. The format is
also visually quite distinct from email addresses, phone numbers, etc... so
hopefully reasonably "self-describing" when written on e.g. a business card
without surrounding context.
[[TODO(paul): we might have to think about this one - too close to email?
Twitter? Also it suggests a format scheme for room names of
#localname:domain.name, which I quite like]]
Directory server administrators should be able to make some kind of policy
decision on how these are allocated. Servers within some "closed" domain (such
as company-specific ones) may wish to verify the validity of a mapping using
their own internal mechanisms; "public" naming servers can operate on a FCFS
basis. There are overlapping concerns here with the idea of the 3rd party
identity servers as well, though in this specific case we are creating a new
namespace to allocate names into.
It would also be nice from a user experience perspective if the profile that a
given name links to can also declare that name as part of its metadata.
Furthermore as a security and consistency perspective it would be nice if each
end (the directory server and the user's homeserver) check the validity of the
mapping in some way. This needs investigation from a security perspective to
ensure against spoofing.
One such model may be that the user starts by declaring their intent to use a
given user name link to their homeserver, which then contacts the directory
service. At some point later (maybe immediately for "public open FCFS servers",
maybe after some kind of human intervention for verification) the DS decides to
honour this link, and includes it in its served output. It should also tell the
HS of this fact, so that the HS can present this as fact when requested for the
profile information. For efficiency, it may further wish to provide the HS with
a cryptographically-signed certificate as proof, so the HS serving the profile
can provide that too when asked, avoiding requesting HSes from constantly having
to contact the DS to verify this mapping. (Note: This is similar to the security
model often applied in DNS to verify PTR <-> A bidirectional mappings).
Identity Servers
================
The identity servers should support the concept of pointing a 3PID being able to
store an ACL Token as well as the main User ID. It is however, beyond scope to
do any kind of verification that any third-party IDs that the profile is
claiming match up to the 3PID mappings.
User Interface and Expectations Concerns
========================================
Given the weak "security" of some parts of this model as compared to what users
might expect, some care should be taken on how it is presented to users,
specifically in the naming or other wording of user interface components.
Most notably mere knowledge of an ACL Pointer is enough to read the information
stored in it. It is possible that Home or Identity Servers could leak this
information, allowing others to see it. This is a security-vs-convenience
balancing choice on behalf of the user who would choose, or not, to make use of
such a feature to publish their information.
Additionally, unless some form of strong end-to-end user-based encryption is
used, a user of ACLs for information privacy has to trust other homeservers not
to lie about the identify of the user requesting access to the Profile.
API Requirements
================
The data model presented here puts the following requirements on the APIs:
Client-Server
-------------
Requests that a client can make to its homeserver
* get/set my Display Name
This should return/take a simple "text/plain" field
* get/set my Avatar URL
The avatar image data itself is not stored by this API; we'll just store a
URL to let the clients fetch it. Optionally HSes could integrate this with
their generic content attacmhent storage service, allowing a user to set
upload their profile Avatar and update the URL to point to it.
* get/add/remove my metadata fields
Also we need to actually define types of metadata
* get another user's Display Name / Avatar / metadata fields
TODO(paul): At some later stage we should consider the API for:
* get/set ACL permissions on my metadata fields
* manage my ACL tokens
Server-Server
-------------
Requests that homeservers make to others
* get a user's Display Name / Avatar
* get a user's full profile - name/avatar + MD fields
This request must allow for specifying the User ID of the requesting user,
for permissions purposes. It also needs to take into account any ACL Tokens
the requestor has.
* push a change of Display Name to observers (overlaps with the presence API)
Room Event PDU Types
--------------------
Events that are pushed from homeservers to other homeservers or clients.
* user Display Name change
* user Avatar change
[[TODO(paul): should the avatar image itself be stored in all the room
histories? maybe this event should just be a hint to clients that they should
re-fetch the avatar image]]

68
attic/drafts/model/protocol_examples.rst Normal file
View file

@ -0,0 +1,68 @@
PUT /send/abc/ HTTP/1.1
Host: ...
Content-Length: ...
Content-Type: application/json
.. code :: javascript
{
"origin": "localhost:5000",
"pdus": [
{
"content": {},
"context": "tng",
"depth": 12,
"is_state": false,
"origin": "localhost:5000",
"pdu_id": 1404381396854,
"pdu_type": "feedback",
"prev_pdus": [
[
"1404381395883",
"localhost:6000"
]
],
"ts": 1404381427581
}
],
"prev_ids": [
"1404381396852"
],
"ts": 1404381427823
}
HTTP/1.1 200 OK
======================================
GET /pull/-1/ HTTP/1.1
Host: ...
Content-Length: 0
HTTP/1.1 200 OK
Content-Length: ...
Content-Type: application/json
.. code :: javascript
{
origin: ...,
prev_ids: ...,
data: [
{
data_id: ...,
prev_pdus: [...],
depth: ...,
ts: ...,
context: ...,
origin: ...,
content: {
...
}
},
...,
]
}

113
attic/drafts/model/room-join-workflow.rst Normal file
View file

@ -0,0 +1,113 @@
==================
Room Join Workflow
==================
An outline of the workflows required when a user joins a room.
Discovery
=========
To join a room, a user has to discover the room by some mechanism in order to
obtain the (opaque) Room ID and a candidate list of likely homeservers that
contain it.
Sending an Invitation
---------------------
The most direct way a user discovers the existence of a room is from a
invitation from some other user who is a member of that room.
The inviter's HS sets the membership status of the invitee to "invited" in the
"m.members" state key by sending a state update PDU. The HS then broadcasts this
PDU among the existing members in the usual way. An invitation message is also
sent to the invited user, containing the Room ID and the PDU ID of this
invitation state change and potentially a list of some other homeservers to use
to accept the invite. The user's client can then choose to display it in some
way to alert the user.
[[TODO(paul): At present, no API has been designed or described to actually send
that invite to the invited user. Likely it will be some facet of the larger
user-user API required for presence, profile management, etc...]]
Directory Service
-----------------
Alternatively, the user may discover the channel via a directory service; either
by performing a name lookup, or some kind of browse or search acitivty. However
this is performed, the end result is that the user's homeserver requests the
Room ID and candidate list from the directory service.
[[TODO(paul): At present, no API has been designed or described for this
directory service]]
Joining
=======
Once the ID and homeservers are obtained, the user can then actually join the
room.
Accepting an Invite
-------------------
If a user has received and accepted an invitation to join a room, the invitee's
homeserver can now send an invite acceptance message to a chosen candidate
server from the list given in the invitation, citing also the PDU ID of the
invitation as "proof" of their invite. (This is required as due to late message
propagation it could be the case that the acceptance is received before the
invite by some servers). If this message is allowed by the candidate server, it
generates a new PDU that updates the invitee's membership status to "joined",
referring back to the acceptance PDU, and broadcasts that as a state change in
the usual way. The newly-invited user is now a full member of the room, and
state propagation proceeds as usual.
Joining a Public Room
---------------------
If a user has discovered the existence of a room they wish to join but does not
have an active invitation, they can request to join it directly by sending a
join message to a candidate server on the list provided by the directory
service. As this list may be out of date, the HS should be prepared to retry
other candidates if the chosen one is no longer aware of the room, because it
has no users as members in it.
Once a candidate server that is aware of the room has been found, it can
broadcast an update PDU to add the member into the "m.members" key setting their
state directly to "joined" (i.e. bypassing the two-phase invite semantics),
remembering to include the new user's HS in that list.
Knocking on a Semi-Public Room
------------------------------
If a user requests to join a room but the join mode of the room is "knock", the
join is not immediately allowed. Instead, if the user wishes to proceed, they
can instead post a "knock" message, which informs other members of the room that
the would-be joiner wishes to become a member and sets their membership value to
"knocked". If any of them wish to accept this, they can then send an invitation
in the usual way described above. Knowing that the user has already knocked and
expressed an interest in joining, the invited user's homeserver should
immediately accept that invitation on the user's behalf, and go on to join the
room in the usual way.
[[NOTE(Erik): Though this may confuse users who expect 'X has joined' to
actually be a user initiated action, i.e. they may expect that 'X' is actually
looking at synapse right now?]]
[[NOTE(paul): Yes, a fair point maybe we should suggest HSes don't do that, and
just offer an invite to the user as normal]]
Private and Non-Existent Rooms
------------------------------
If a user requests to join a room but the room is either unknown by the home
server receiving the request, or is known by the join mode is "invite" and the
user has not been invited, the server must respond that the room does not exist.
This is to prevent leaking information about the existence and identity of
private rooms.
Outstanding Questions
=====================
* Do invitations or knocks time out and expire at some point? If so when? Time
is hard in distributed systems.

276
attic/drafts/model/rooms.rst Normal file
View file

@ -0,0 +1,276 @@
===========
Rooms Model
===========
A description of the general data model used to implement Rooms, and the
user-level visible effects and implications.
Overview
========
"Rooms" in Synapse are shared messaging channels over which all the participant
users can exchange messages. Rooms have an opaque persistent identify, a
globally-replicated set of state (consisting principly of a membership set of
users, and other management and miscellaneous metadata), and a message history.
Room Identity and Naming
========================
Rooms can be arbitrarily created by any user on any homeserver; at which point
the homeserver will sign the message that creates the channel, and the
fingerprint of this signature becomes the strong persistent identify of the
room. This now identifies the room to any homeserver in the network regardless
of its original origin. This allows the identify of the room to outlive any
particular server. Subject to appropriate permissions [to be discussed later],
any current member of a room can invite others to join it, can post messages
that become part of its history, and can change the persistent state of the room
(including its current set of permissions).
Homeservers can provide a directory service, allowing a lookup from a
convenient human-readable form of room label to a room ID. This mapping is
scoped to the particular homeserver domain and so simply represents that server
administrator's opinion of what room should take that label; it does not have to
be globally replicated and does not form part of the stored state of that room.
This room name takes the form
#localname:some.domain.name
for similarity and consistency with user names on directories.
To join a room (and therefore to be allowed to inspect past history, post new
messages to it, and read its state), a user must become aware of the room's
fingerprint ID. There are two mechanisms to allow this:
* An invite message from someone else in the room
* A referral from a room directory service
As room IDs are opaque and ephemeral, they can serve as a mechanism to create
"ad-hoc" rooms deliberately unnamed, for small group-chats or even private
one-to-one message exchange.
Stored State and Permissions
============================
Every room has a globally-replicated set of stored state. This state is a set of
key/value or key/subkey/value pairs. The value of every (sub)key is a
JSON-representable object. The main key of a piece of stored state establishes
its meaning; some keys store sub-keys to allow a sub-structure within them [more
detail below]. Some keys have special meaning to Synapse, as they relate to
management details of the room itself, storing such details as user membership,
and permissions of users to alter the state of the room itself. Other keys may
store information to present to users, which the system does not directly rely
on. The key space itself is namespaced, allowing 3rd party extensions, subject
to suitable permission.
Permission management is based on the concept of "power-levels". Every user
within a room has an integer assigned, being their "power-level" within that
room. Along with its actual data value, each key (or subkey) also stores the
minimum power-level a user must have in order to write to that key, the
power-level of the last user who actually did write to it, and the PDU ID of
that state change.
To be accepted as valid, a change must NOT:
* Be made by a user having a power-level lower than required to write to the
state key
* Alter the required power-level for that state key to a value higher than the
user has
* Increase that user's own power-level
* Grant any other user a power-level higher than the level of the user making
the change
[[TODO(paul): consider if relaxations should be allowed; e.g. is the current
outright-winner allowed to raise their own level, to allow for "inflation"?]]
Room State Keys
===============
[[TODO(paul): if this list gets too big it might become necessary to move it
into its own doc]]
The following keys have special semantics or meaning to Synapse itself:
m.member (has subkeys)
Stores a sub-key for every Synapse User ID which is currently a member of
this room. Its value gives the membership type ("knocked", "invited",
"joined").
m.power_levels
Stores a mapping from Synapse User IDs to their power-level in the room. If
they are not present in this mapping, the default applies.
The reason to store this as a single value rather than a value with subkeys
is that updates to it are atomic; allowing a number of colliding-edit
problems to be avoided.
m.default_level
Gives the default power-level for members of the room that do not have one
specified in their membership key.
m.invite_level
If set, gives the minimum power-level required for members to invite others
to join, or to accept knock requests from non-members requesting access. If
absent, then invites are not allowed. An invitation involves setting their
membership type to "invited", in addition to sending the invite message.
m.join_rules
Encodes the rules on how non-members can join the room. Has the following
possibilities:
- "public" - a non-member can join the room directly
- "knock" - a non-member cannot join the room, but can post a single "knock"
message requesting access, which existing members may approve or deny
- "invite" - non-members cannot join the room without an invite from an
existing member
- "private" - nobody who is not in the 'may_join' list or already a member
may join by any mechanism
In any of the first three modes, existing members with sufficient permission
can send invites to non-members if allowed by the "m.invite_level" key. A
"private" room is not allowed to have the "m.invite_level" set.
A client may use the value of this key to hint at the user interface
expectations to provide; in particular, a private chat with one other use
might warrant specific handling in the client.
m.may_join
A list of User IDs that are always allowed to join the room, regardless of any
of the prevailing join rules and invite levels. These apply even to private
rooms. These are stored in a single list with normal update-powerlevel
permissions applied; users cannot arbitrarily remove themselves from the list.
m.add_state_level
The power-level required for a user to be able to add new state keys.
m.public_history
If set and true, anyone can request the history of the room, without needing
to be a member of the room.
m.archive_servers
For "public" rooms with public history, gives a list of homeservers that
should be included in message distribution to the room, even if no users on
that server are present. These ensure that a public room can still persist
even if no users are currently members of it. This list should be consulted by
the dirctory servers as the candidate list they respond with.
The following keys are provided by Synapse for user benefit, but their value is
not otherwise used by Synapse.
m.name
Stores a short human-readable name for the room, such that clients can display
to a user to assist in identifying which room is which.
This name specifically is not the strong ID used by the message transport
system to refer to the room, because it may be changed from time to time.
m.topic
Stores the current human-readable topic
Room Creation Templates
=======================
A client (or maybe homeserver?) could offer a few templates for the creation of
new rooms. For example, for a simple private one-to-one chat the channel could
assign the creator a power-level of 1, requiring a level of 1 to invite, and
needing an invite before members can join. An invite is then sent to the other
party, and if accepted and the other user joins, the creator's power-level can
now be reduced to 0. This now leaves a room with two participants in it being
unable to add more.
Rooms that Continue History
===========================
An option that could be considered for room creation, is that when a new room is
created the creator could specify a PDU ID into an existing room, as the history
continuation point. This would be stored as an extra piece of meta-data on the
initial PDU of the room's creation. (It does not appear in the normal previous
PDU linkage).
This would allow users in rooms to "fork" a room, if it is considered that the
conversations in the room no longer fit its original purpose, and wish to
diverge. Existing permissions on the original room would continue to apply of
course, for viewing that history. If both rooms are considered "public" we might
also want to define a message to post into the original room to represent this
fork point, and give a reference to the new room.
User Direct Message Rooms
=========================
There is no need to build a mechanism for directly sending messages between
users, because a room can handle this ability. To allow direct user-to-user chat
messaging we simply need to be able to create rooms with specific set of
permissions to allow this direct messaging.
Between any given pair of user IDs that wish to exchange private messages, there
will exist a single shared Room, created lazily by either side. These rooms will
need a certain amount of special handling in both homeservers and display on
clients, but as much as possible should be treated by the lower layers of code
the same as other rooms.
Specially, a client would likely offer a special menu choice associated with
another user (in room member lists, presence list, etc..) as "direct chat". That
would perform all the necessary steps to create the private chat room. Receiving
clients should display these in a special way too as the room name is not
important; instead it should distinguish them on the Display Name of the other
party.
Homeservers will need a client-API option to request setting up a new user-user
chat room, which will then need special handling within the server. It will
create a new room with the following
m.member: the proposing user
m.join_rules: "private"
m.may_join: both users
m.power_levels: empty
m.default_level: 0
m.add_state_level: 0
m.public_history: False
Having created the room, it can send an invite message to the other user in the
normal way - the room permissions state that no users can be set to the invited
state, but because they're in the may_join list then they'd be allowed to join
anyway.
In this arrangement there is now a room with both users may join but neither has
the power to invite any others. Both users now have the confidence that (at
least within the messaging system itself) their messages remain private and
cannot later be provably leaked to a third party. They can freely set the topic
or name if they choose and add or edit any other state of the room. The update
powerlevel of each of these fixed properties should be 1, to lock out the users
from being able to alter them.
Anti-Glare
==========
There exists the possibility of a race condition if two users who have no chat
history with each other simultaneously create a room and invite the other to it.
This is called a "glare" situation. There are two possible ideas for how to
resolve this:
* Each homeserver should persist the mapping of (user ID pair) to room ID, so
that duplicate requests can be suppressed. On receipt of a room creation
request that the HS thinks there already exists a room for, the invitation to
join can be rejected if:
- a) the HS believes the sending user is already a member of the room (and
maybe their HS has forgotten this fact), or
- b) the proposed room has a lexicographically-higher ID than the existing
room (to resolve true race condition conflicts)
* The room ID for a private 1:1 chat has a special form, determined by
concatenting the User IDs of both members in a deterministic order, such that
it doesn't matter which side creates it first; the HSes can just ignore
(or merge?) received PDUs that create the room twice.

108
attic/drafts/model/third-party-id.rst Normal file
View file

@ -0,0 +1,108 @@
======================
Third Party Identities
======================
A description of how email addresses, mobile phone numbers and other third
party identifiers can be used to authenticate and discover users in Matrix.
Overview
========
New users need to authenticate their account. An email or SMS text message can
be a convenient form of authentication. Users already have email addresses
and phone numbers for contacts in their address book. They want to communicate
with those contacts in Matrix without manually exchanging a Matrix User ID with
them.
Third Party IDs
---------------
[[TODO(markjh): Describe the format of a 3PID]]
Third Party ID Associations
---------------------------
An Associaton is a binding between a Matrix User ID and a Third Party ID (3PID).
Each 3PID can be associated with one Matrix User ID at a time.
[[TODO(markjh): JSON format of the association.]]
Verification
------------
An Assocation must be verified by a trusted Verification Server. Email
addresses and phone numbers can be verified by sending a token to the address
which a client can supply to the verifier to confirm ownership.
An email Verification Server may be capable of verifying all email 3PIDs or may
be restricted to verifying addresses for a particular domain. A phone number
Verification Server may be capable of verifying all phone numbers or may be
restricted to verifying numbers for a given country or phone prefix.
Verification Servers fulfil a similar role to Certificate Authorities in PKI so
a similar level of vetting should be required before clients trust their
signatures.
A Verification Server may wish to check for existing Associations for a 3PID
before creating a new Association.
Discovery
---------
Users can discover Associations using a trusted Identity Server. Each
Association will be signed by the Identity Server. An Identity Server may store
the entire space of Associations or may delegate to other Identity Servers when
looking up Associations.
Each Association returned from an Identity Server must be signed by a
Verification Server. Clients should check these signatures.
Identity Servers fulfil a similar role to DNS servers.
Privacy
-------
A User may publish the association between their phone number and Matrix User ID
on the Identity Server without publishing the number in their Profile hosted on
their homeserver.
Identity Servers should refrain from publishing reverse mappings and should
take steps, such as rate limiting, to prevent attackers enumerating the space of
mappings.
Federation
==========
Delegation
----------
Verification Servers could delegate signing to another server by issuing
certificate to that server allowing it to verify and sign a subset of 3PID on
its behalf. It would be necessary to provide a language for describing which
subset of 3PIDs that server had authority to validate. Alternatively it could
delegate the verification step to another server but sign the resulting
association itself.
The 3PID space will have a heirachical structure like DNS so Identity Servers
can delegate lookups to other servers. An Identity Server should be prepared
to host or delegate any valid association within the subset of the 3PIDs it is
resonsible for.
Multiple Root Verification Servers
----------------------------------
There can be multiple root Verification Servers and an Association could be
signed by multiple servers if different clients trust different subsets of
the verification servers.
Multiple Root Identity Servers
------------------------------
There can be be multiple root Identity Servers. Clients will add each
Association to all root Identity Servers.
[[TODO(markjh): Describe how clients find the list of root Identity Servers]]

27
attic/drafts/object_model.rst Normal file
View file

@ -0,0 +1,27 @@
..TODO
What are the start & end tokens doing here?!
::
+---------------+
| Room |
| "Room-ID" |
| {State} | +----------------------+
| Name------|-------->| Event m.room.name |
| Topic | | "Name" |
| [Aliases] | +----------------------+ +-------------+
| [Members]-|---+ +----------------------+ <----| Start Token |
| [Messages] | | | Event m.room.member | +-------------+
| | | | +---->| "invite/join/ban" |
+---------------+ | "User-ID" |
| | +----------------------+
| | +----------------------+
| | Message | Event m.room.message |
| +-------------->| {content} |<--+
| +----------------------+ |
| Comment +----------------------+ |
+------------------>| Event m.room.message | |
| {content} | |
| "relates-to"-------|---+ +-------------+
+----------------------+ <----| End Token |
+-------------+

48
attic/drafts/pstn_gatewaying.txt Normal file
View file

@ -0,0 +1,48 @@
Gatewaying to the PSTN via Matrix Application Services
======================================================
Matrix Application Services (AS) provides a way for PSTN users to interact
with Matrix via an AS acting as a gateway. Each PSTN user is represented as a
virtual user on a specific homeserver maintained by the AS. Typically the AS
is provisioned on a well-known AS-supplier HS (e.g. matrix.openmarket.com) or
is a service provisioned on the user's local HS.
In either scenario, the AS maintains virtual users of form
@.tel.e164:homeserver. These are lazily created (as per the AS spec) when
matrix users try to contact a user id of form @.tel.*:homeserver, or when the
AS needs to inject traffic into the HS on behalf of the PSTN user. The reason
for these being a visible virtual user rather than an invisible user or an
invisible sniffing AS is because they do represent real physical 3rd party
endpoints in the PSTN, and need to be able to send return messages.
Communication with an actual PSTN user happens in a normal Matrix room, which
for 1:1 matrix<->pstn contact will typically store all conversation history
with that user. On first contact, the matrix user invites the virtual user
into the room (or vice versa). In the event of switching to another AS-enabled
HS, the matrix user would kick the old AS and invite the new one. In the event
of needing loadbalancing between two SMS gateways (for instance), the user
would set visibility flags (TODO: specify per-message ACLs, or use crypto to
only sign messages so they're visible to certain other users?) to adjust which
virtual AS users could see which messages in the room.
For group chat, one or more AS virtual users may be invited to a group chat,
where-upon they will relay all the traffic in that group chat through to their
PSTN counterpart (and vice versa). This behaviour requires no additional
functionality beyond that required to support 1:1 chat.
When contacting a user, Matrix clients should check whether a given E.164
number is already mapped to a real Matrix user by querying the identity
servers (or subscribing to identity updates for a given E.164 number. TODO: ID
server subscriptions). If the E.164 number has a validated mapping in the ID
server to a Matrix ID, then this target ID should be used instead of
contacting the virtual user.
It's likely that PSTN gateway ASes will need to charge the end-user for use of
the gateway. The AS must therefore track credit per matrix ID it interacts
with, and stop gatewaying as desired once credit is exhausted. The task of
extracting credit from the end-user and adding it to the AS is not covered by
the Matrix specification.
For SMS routing, options are:
* Terminate traffic only (from a shared shortcode originator)
* Two-way traffic via a VMN. To save allocating huge numbers of VMNs to Matrix users, the VMN can be allocated from a pool such that each {caller,callee} tuple is unique (but the caller number will only work from that specific callee).

76
attic/drafts/reputation_thoughts.rst Normal file
View file

@ -0,0 +1,76 @@
(a stream of incoherent consciousness from matthew which should go somewhere)
Invite-graph based reputation data:
* Users need a reputation score to issue invites or join public rooms.
* A user can have many reputation scores in different audiences (and perhaps a global average?)
* A room (degenerate case: user) can align itself with a given audience in order to consume the reputation data for that audience.
* The people that a user invites inherits a proportion of their reputation.
* If your reputation in an audience is ever reduced, it similarly reduces the reputation you have ever conveyed to anyone else (which propagates through the invite graph).
* Users increase reputation by:
* Inviting someone.
* Upvoting their messages in a room (i.e. for the suitability of that audience)
* Users decrease reputation by:
* Blocking them.
* Downvoting their messages in a room (i.e. for the suitability of that audience)
Need to ensure the accounts are of a decent quality - making it harder to create sockpuppet accounts and associating them with real people is more important than the actual reputation problem.
Build a war game simulation to test?
Problems:
* How are audiences defined? Just a given unique set of users? Which then makes inheriting reputation easy between audiences - if the overlap is significant, the chances are the reputation rules are the same.
* But is it possible to have the same set of users in two different rooms have different rules for reputation? Probably yes, as the potential audience may include future invitees or indeed the general public, so history visibility rules should probably also contribute to this. But given privacy rules can change over time, each room should effectively define its own audience. So in the end, an audience === a room.
* Create a large network of fake users, and go and have them all vote up each other's score for a given audience.
* This can be solved if the root inviter is penalised, which then destroys all the reputation they conveyed to their graph.
Could Reputation == Power Level (!?!?!)
Inheritence semantics for reputation between different audiences is hard.
* You should base the reputation of a stranger on their reputation in other communities that you or your communities have some overlap with.
* Do you consider 2nd hand reputation data at all from private rooms? Or do you look only at the public reputation data?
How do you do these calculations in a byzantine world?
How do you do these calculations whilst preserving privacy?
* Only consider reputation data from rooms you are actually in?
* Store reputation data in room state?
* Have a function (HS? client? AS? spider?) that aggregates reputation data (and proves that the aggregation is accurate, almost like blockchain mining?)?
* Or have a separate reputation global db seperate from room state that people contribute metrics into (which gathers the aggregate data into a single place, and makes it easier to query reputation data for strangers)
How do you avoid backstabbing? (People maliciously ganging up on someone to downvote them)?
How do you avoid a voting war? (Community fragments; different factions turn up and try to downvote the other)?
* This is effectively two different audiences emerging in a single room.
* Perhaps this means we should model audiences separately from rooms.
* Perhaps audiences are literally ACL groups? And eventually, one might change the ACLs of a room to eject one of the groups?
* Or do you just synthesise audiences based on cliques of people who support each other? The act of upvoting someone is effectively aligning yourself as being part of the same audience?
So:
* Gather all public upvote/downvotes/invites/blocks in a global DB.
* Partition this into audiences based on who votes on who. Stuff which is read and not complained about could provide a small implicit approval? Although this makes it easy to flood content to boost your reputation, so bad idea.
* Partitioning algorithm could be quite subtle.
* You could end up with lots of small audiences (including invalid ones), and it's fairly unclear how they get aggregated into a single view. How should you treat a stranger who you have no audience-overlap with at all? Treat them as effectively having zero reputation from your perspective?
Problem:
* If the douchebag who invites spammers never says anything, how do you go vote on their reputation? Should there be some kind of back-propagation? Or is there explicitly a "this person invited a douchebag" downvote? Or hang on - how can they ever get reputation in the first place to invite their sockpuppets if they don't say anything (beyond the initial invite)?
* What if users simply don't talk in public? Is it right that we prevent them issuing invites just because they stick to private rooms? What about inviting people into those private rooms? I guess the point is that if these are public invites, then they need to have some kind of public reputation, or rely on out-of-band private invitation to establish trust?
* Are we rewarding people who don't change their habits? There's no time component considered here, and we punish people's entire history of invites and rep if they misbehave. The only way to escape is to create a new identity atm. Is this a feature or a bug?
* How does this handle people's accounts getting 0wn3d and doing things which wipe out their reputation? => This is always a risk; ignore it.
* Do you need a particular level of reputation to be able to vote on people?
Summary?
* Partition the global population into multiple overlapping clusters called 'audiences' based on mutual(?) upvote/downvote relationships in public rooms.
* Clusters of the same people but in different rooms could be modelled as separate (but overlapping) clusters.
* Each audience builds up a reputation score for the global population, blending in damped scores from overlapping audiences.
* Anyone can upvote/downvote, but the votes will not contribute to your personal opinion unless the voter overlaps with your audience's scoresheet.
* A room could adopt a given audience (that of the moderators'?) for considering the reputation of who can join, invite people, etc.
* A user uses their own 'audience of one' scoresheet to put a threshold on filtering out contact from other users (invites, messages, etc).
* Their personal scoresheet is presumably a blend of all the audiences they are already in.
* The act of inviting someone gives them some reputation, within your audiences, proportional to your own. Similarly blocking reduces reputation.
* If you are downvoted, it retrospectively reduces the weight of all of your upvote/downvotes (at least for audiences that the downvoter's opinion contributes to). Similarly for upvoting.
* This penalisation process is transitive.
Do we even need the penalisation stuff if audience partitioning works?

317
attic/drafts/use_cases.rst Normal file
View file

@ -0,0 +1,317 @@
General UI/UX requirements:
===========================
- Live updates
- No flicker:
* Sending message (local echo)
* Receiving images (encoding w/h)
* Scrollback
* Resolving display names (from user ID)
- Fast startup times
- Fast "opening room" times (esp. when clicking through from a notification)
- Low latency file transfer.
Use cases
---------
- #1: Lightweight IM client (no perm storage) - e.g. Web client
- #2: Bug tracking software
- #3: Forum
- #4: Google + style communities
- #5: Email style threading
- #6: Multi-column threaded IM
- #7: Mobile IM client (perm storage)
- #8: MIDI client
- #9: Animatrix client
- #10: Unity object trees
- #11: Social Network ("Walls", PMs, groups)
- #12: Minecraft-clone
- #13: Global 'Like' widget, which links through to a room.
#1 Web client UI
================
Model::
Rooms ----< Messages
- name - type (call/image)
- topic
Home Screen
What's visible:
- Recent chats ordered by timestamp of latest event (with # users)
- Your own display name, user ID and avatar url
- A searchable list of public rooms (with # users and alias + room name + room topic)
What you can do:
- Create a room (public/private, with alias)
- Join a room from alias
- Message a user (with user ID)
- Leave a recent room
- Open a room
- Open a chat history link.
- Search for a public room.
Chat Screen
What's visible:
- Enough scrollback to fill a "screen full" of content.
- Each message: timestamp, user ID, display name at the time the message was
sent, avatar URL at the time the message was sent, whether it was a bing message
or not.
- User list: for each user: presence, current avatar url in the room, current
display name in the room, power level, ordered by when they were last speaking.
- Recents list: (same as Home Screen)
- Room name
- Room topic
- Typing notifications
- Desktop/Push Notifications for messages
What you can do:
- Invite a user
- Kick a user
- Ban/Unban a user
- Leave the room
- Send a message (image/text/emote)
- Change someone's power level
- Change your own display name
- Accept an incoming call
- Make an outgoing call
- Get older messages by scrolling up (scrollback)
- Redact a message
- Resend a message which was not sent
Message sending:
- Immediate local echo
- Queue up messages which haven't been sent yet
- Reordering local echo to where it actually happened
VoIP:
- One entry in your display for a call (which may contain duration, type, status)
- Glare resolution
Scrollback:
- Display in reverse chronological order by the originating server's timestamp
- Terminates at the start of the room (which then makes it impossible to request
more scrollback)
Local storage:
- Driven by desire for fast startup times and minimal network traffic
- Display messages from storage and from the network without any gaps in messages.
- Persist scrollback if possible: Scrollback from storage first then from the
network.
Notifications:
- Receive notifications for rooms you're interested in (explicitly or from a default)
- Maybe per device.
- Maybe depending on presence (e.g. idle)
- Maybe depending on message volume
- Maybe depending on room config options.
Message contents:
- images
- video
- rich text
- audio
- arbitrary files
- location
- vcards (potentially)
Chat History Screen
What's visible:
- The linked message and enough scrollback to fill a "screen full" of content.
- Each message: timestamp, user ID, display name at the time the message was
sent, avatar URL at the time the message was sent, whether it was a bing message
or not.
- The historical user list. *TODO: Is this taken at the linked message, or at
wherever the user has scrolled to?*
What you can do:
- Get older messages by scrolling up (scrollback)
- Get newer messages by scrolling down
Public Room Search Screen
What's visible:
- The current search text.
- The homeserver being searched (defaults to the HS the client is connected to).
- The results of the current search with enough results to fill the screen
with # users and alias + room name + room topic.
What you can do:
- Change what you are searching for.
- Change the server that's being searched.
- Scroll down to get more search results.
User screen
What's visible:
- Display name
- Avatar
- User ID
What you can do:
- Start a chat with the user
#2 Bug tracking UI
==================
Model::
Projects ----< Issues ---< Comments
- key - summary - user
- name - ID - message
SYN SYN-52 Fix it nooow!
Landing page
What's visible:
- Issues assigned to me
- Issues I'm watching
- Recent activity on other issues (not refined to me)
- List of projects
What you can do:
- View an issue
- Create an issue
- Sort issues
- View a user
- View a project
- Search for issues (by name, time, priority, description contents, reporter, etc...)
Issue page
What's visible:
- Summary of issue
- Issue key
- Project affected
- Description
- Comments
- Priority, labels, type, purpose, etc..
- Reporter/assignee
- Creation and last updated times
- History of issue changes
What you can do:
- Comment on issue
- Change issue info (labels, type, purpose, etc..)
- Open/Close/Resolve the issue
- Edit the issue
- Watch/Unwatch the issue
#3 Forum UI
===========
Model::
Forum ----< Boards ----< Threads ----< Messages
- Matrix - Dev - HALP! - please halp!
Main page
What's visible:
- Categories (containing boards)
- Boards (with names and # posts and tagline and latest post)
What you can do:
- View a board
- View the latest message on a board
Board page
What's visible:
- Threads (titles, OP, latest post date+author, # replies, # upvotes, whether
the OP contains an image or hyperlink (small icon on title))
- Whether the thread is answered (with link to the answer)
- Pagination for posts within a thread (1,2,3,4,5...10)
- Pagination for threads within a board
- List of threads in chronological order
- Stickied threads
What you can do:
- View a user
- View a thread on a particular page
- View the latest message on a thread
- View older threads (pagination)
- Search the board
Thread page
What's visible:
- Messages in chronological order
- For each message: author, timestamp, # posts by author, avatar, registration
date, status message, message contents, # views of message
What you can do:
- Upvote the message
- Flag the message for a mod
- Reply to the message
- Subscribe to thread or message's RSS feed
- Go to previous/next thread
#4 Google+ community
====================
Model::
Community -----< Categories ----< Posts ---< Comments
Kerbal SP Mods, Help Text Text
(no title!)
Communities page
What's visible:
- List of communities
- For each community: # users, # posts, group pic, title
What you can do:
- Join a community
- View a community
Community Page
What's visible:
- Title, pic
- List of categories
- List of members with avatars (+ total #)
- Most recent posts with comments (most recent comment if >1)
What you can do:
- Join the group
- Post a post (with voting and options)
- Report abuse
- View member
- Expand comments
- Infinite scrolling
- Add a comment to a post
- Share a post
- +1 a post
#5 Email style threading
========================
Chat Screen
What's visible:
- Enough scrollback to fill a "screen full" of content.
- Threads:
- Initially will only display the timestamp and user ID of the *first*
message. But can expand to show the entire tree.
- Tree of messages indicating which message is a reply to which.
- Ordered by the arbitrary field (timestamp of oldest message in thread;
newest message in thread; sender id; sender display name; etc)
- Each message: timestamp, user ID, display name at the time of the message
- Room name
- Room topic
- Typing notifications
- Desktop/Push Notifications for messages
What you can do:
- Send a message in reply to another message:
- Immediate local echo, may cause messages to re-order
- Messages that haven't reached the server are queued.
- Thread is displayed where it should be in the thread order once the
message is sent.
- Start a new thread by sending a message.
#6 Multi-threaded IM
====================
Chat Screen
What's visible:
- A multi-column grid of threads from a number of chatrooms
Each concurrent thread is displayed in a different column.
The columns start and end as threads split and rejoin the main conversation
The messages for each thread are ordered by how recent they are::
Room #1 Room # 2 Room # 2
+------------+ +----------------+ Side thread.
| * Message1 | | * Root | +--------------+
| * Message2 | | * A1 -> Root | | * B1 -> Root |
+------------+ | * A2 -> A1 | | * B2 -> B1 |
| * M -> A2, B2 | +--------------+
+----------------+
- Typing notifications. Displayed within the correct thread/column.
What you can do:
- Send a message into a particular thread/column.
- Move an *existing* message into a new thread creating a new column
- Move an existing message into an existing thread, causing the threads to
reconverge (i.e. provide a route from the sidebar back into the existing
thread). This does not imply terminating the thread, which can continue
independently of the merge.

288
attic/drafts/websockets.rst Normal file
View file

@ -0,0 +1,288 @@
WebSockets API
==============
Introduction
------------
This document is a proposal for a WebSockets-based client-server API. It is not
intended to replace the REST API, but rather to complement it and provide an
alternative interface for certain operations.
The primary goal is to offer a more efficient interface than the REST API: by
using a bidirectional protocol such as WebSockets we can avoid the overheads
involved in long-polling (SSL negotiation, HTTP headers, etc). In doing so we
will reduce the latency between server and client by allowing the server to
send events as soon as they arrive, rather than having to wait for a poll from
the client.
Note: This proposal got continued in a google document you can find here:
https://docs.google.com/document/d/104ClehFBgqLQbf4s-AKX2ijr8sOAxcizfcRs_atsB0g
Handshake
---------
1. Instead of calling ``/sync``, the client makes a websocket request to
``/_matrix/client/rN/stream``, passing the query parameters ``access_token``
and ``since``, and optionally ``filter`` - all of which have the same
meaning as for ``/sync``.
* The client sets the ``Sec-WebSocket-Protocol`` to ``m.json``. (Servers may
offer alternative encodings; at present only the JSON encoding is
specified but in future we will specify alternative encodings.)
#. The server returns the websocket handshake; the socket is then connected.
If the server does not return a valid websocket handshake, this indicates that
the server or an intermediate proxy does not support WebSockets. In this case,
the client should fall back to polling the ``/sync`` REST endpoint.
Example
~~~~~~~
Client request:
.. code:: http
GET /_matrix/client/v2_alpha/stream?access_token=123456&since=s72594_4483_1934 HTTP/1.1
Host: matrix.org
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: x3JJHMbDL1EzLkh9GBhXDw==
Sec-WebSocket-Protocol: m.json
Sec-WebSocket-Version: 13
Origin: https://matrix.org
Server response:
.. code:: http
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: HSmrc0sMlYUkAGmm5OPpG2HaGWk=
Sec-WebSocket-Protocol: m.json
Update Notifications
--------------------
Once the socket is connected, the server begins streaming updates over the
websocket. The server sends Update notifications about new messages or state
changes. To make it easy for clients to parse, Update notifications have the
same structure as the response to ``/sync``: an object with the following
members:
============= ========== ===================================================
Key Type Description
============= ========== ===================================================
next_batch string The batch token to supply in the ``since`` param of
the next /sync request. This is not required for
streaming of events over the WebSocket, but is
provided so that clients can reconnect if the
socket is disconnected.
presence Presence The updates to the presence status of other users.
rooms Rooms Updates to rooms.
============= ========== ===================================================
Example
~~~~~~~
Message from the server:
.. code:: json
{
"next_batch": "s72595_4483_1934",
"presence": {
"events": []
},
"rooms": {
"join": {},
"invite": {},
"leave": {}
}
}
Client-initiated operations
---------------------------
The client can perform certain operations by sending a websocket message to
the server. Such a "Request" message should be a JSON-encoded object with
the following members:
============= ========== ===================================================
Key Type Description
============= ========== ===================================================
id string A unique identifier for this request
method string Specifies the name of the operation to be
performed; see below for available operations
param object The parameters for the requested operation.
============= ========== ===================================================
The server responds to a client Request with a Response message. This is a
JSON-encoded object with the following members:
============= ========== ===================================================
Key Type Description
============= ========== ===================================================
id string The same as the value in the corresponding Request
object. The presence of the ``id`` field
distinguishes a Response message from an Update
notification.
result object On success, the results of the request.
error object On error, an object giving the resons for the
error. This has the same structure as the "standard
error response" for the Matrix API: an object with
the fields ``errcode`` and ``error``.
============= ========== ===================================================
Request methods
~~~~~~~~~~~~~~~
It is not intended that all operations which are available via the REST API
will be available via the WebSockets API, but a few simple, common operations
will be exposed. The initial operations will be as follows.
``ping``
^^^^^^^^
This is a no-op which clients may use to keep their connection alive.
The request ``params`` and the response ``result`` should be empty.
``send``
^^^^^^^^
Send a message event to a room. The parameters are as follows:
============= ========== ===================================================
Parameter Type Description
============= ========== ===================================================
room_id string **Required.** The room to send the event to
event_type string **Required.** The type of event to send.
content object **Required.** The content of the event.
============= ========== ===================================================
The result is as follows:
============= ========== ===================================================
Key Type Description
============= ========== ===================================================
event_id string A unique identifier for the event.
============= ========== ===================================================
The ``id`` from the Request message is used as the transaction ID by the
server.
``state``
^^^^^^^^^
Update the state on a room.
============= ========== ===================================================
Parameter Type Description
============= ========== ===================================================
room_id string **Required.** The room to set the state in
event_type string **Required.** The type of event to send.
state_key string **Required.** The state_key for the state to send.
content object **Required.** The content of the event.
============= ========== ===================================================
The result is as follows:
============= ========== ===================================================
Key Type Description
============= ========== ===================================================
event_id string A unique identifier for the event.
============= ========== ===================================================
Example
~~~~~~~
Client request:
.. code:: json
{
"id": "12345",
"method": "send",
"params": {
"room_id": "!d41d8cd:matrix.org",
"event_type": "m.room.message",
"content": {
"msgtype": "m.text",
"body": "hello"
}
}
}
Server response:
.. code:: json
{
"id": "12345",
"result": {
"event_id": "$66697273743031:matrix.org"
}
}
Alternative server response, in case of error:
.. code:: json
{
"id": "12345",
"error": {
"errcode": "M_MISSING_PARAM",
"error": "Missing parameter: event_type"
}
}
Rationale
---------
Alternatives to WebSockets include HTTP/2, CoAP, and simply rolling our own
protocol over raw TCP sockets. However, the need to implement browser-based
clients essentially reduces our choice to WebSockets. HTTP/2 streams will
probably provide an interesting alternative in the future, but current browsers
do not appear to give javascript applications low-level access to the protocol.
Concerning the continued use of the JSON encoding: we prefer to focus on the
transition to WebSockets initially. Replacing JSON with a compact
representation such as CBOR, MessagePack, or even just compressed JSON will be
a likely extension for the future. The support for negotiation of subprotocols
within WebSockets should make this a simple transition once time permits.
The number of methods available for client requests is deliberately limited, as
each method requires code to be written to map it onto the equivalent REST
implementation. Some REST methods - for instance, user registration and login -
would be pointless to expose via WebSockets. It is likely, however, that we
will increate the number of methods available via the WebSockets API as it
becomes clear which would be most useful.
Open questions
--------------
Throttling
~~~~~~~~~~
At least in v2 sync, clients are inherently self-throttling - if they do not
poll quickly enough, events will be dropped from the next result. This proposal
raises the possibility that events will be produced more quickly than they can
be sent to the client; backlogs will build up on the server and/or in the
intermediate network, which will not only lead to high latency on events being
delivered, but will lead to responses to client requests also being delayed.
We may need to implement some sort of throttling mechanism by which the server
can start to drop events. The difficulty is in knowing when to start dropping
events. A few ideas:
* Use websocket pings to measure the RTT; if it starts to increase, start
dropping events. But this requires knowledge of the base RTT, and a useful
model of what constitutes an excessive increase.
* Have the client acknowledge each batch of events, and use a window to ensure
the number of outstanding batches is limited. This is annoying as it requires
the client to have to acknowledge batches - and it's not clear what the right
window size is: we want a big window for long fat networks (think of mobile
clients), but a small one for one with lower latency.
* Start dropping events if the server's TCP buffer starts filling up. This has
the advantage of delegating the congestion-detection to TCP (which already
has a number of algorithms to deal with it, to greater or lesser
effectiveness), but relies on homeservers being hosted on OSes which use
sensible TCP congestion-avoidance algorithms, and more critically, an ability
to read the fill level of the TCP send buffer.