Merge branch 'master' into rav/rework_objects
Conflicts: templating/matrix_templates/units.py
This commit is contained in:
Richard van der Hoff
commit
34ac544290
9 changed files with 298 additions and 65 deletions
142
drafts/markjh_end_to_end.rst
Normal file
142
drafts/markjh_end_to_end.rst
Normal file
|
|
@ -0,0 +1,142 @@
|
|||
Goals of Key-Distribution in Matrix
|
||||
===================================
|
||||
|
||||
* No Central Authority: Users should not need to trust a central authority
|
||||
when determining the authenticity of keys.
|
||||
|
||||
* Easy to Add New Devices: It should be easy for a user to start using a
|
||||
new device.
|
||||
|
||||
* Possible to discover MITM: It should be possible for a user to determine if
|
||||
they are being MITM.
|
||||
|
||||
* Lost Devices: It should be possible for a user to recover if they lose all
|
||||
their devices.
|
||||
|
||||
* No Copying Keys: Keys should be per device and shouldn't leave the device
|
||||
they were created on.
|
||||
|
||||
A Possible Mechanism for Key Distribution
|
||||
=========================================
|
||||
|
||||
Basic API for setting up keys on a server:
|
||||
|
||||
https://github.com/matrix-org/matrix-doc/pull/24
|
||||
|
||||
Client shouldn't trust the keys unless they have been verified, e.g by
|
||||
comparing fingerprints.
|
||||
|
||||
If a user adds a new device it should some yet to be specified protocol
|
||||
communicate with an old device and obtain a cross-signature from the old
|
||||
device for its public key.
|
||||
|
||||
The new device can then present the cross-signed key to all the devices
|
||||
that the user is in conversations with. Those devices should then include
|
||||
the new device into those conversations.
|
||||
|
||||
If the user cannot cross-sign the new key, e.g. because their old device
|
||||
is lost or stolen. Then they will need to reauthenticate their conversations
|
||||
out of band, e.g by comparing fingerprints.
|
||||
|
||||
|
||||
Goals of End-to-end encryption in Matrix
|
||||
========================================
|
||||
|
||||
* Access to Chat History: Users should be able to see the history of a
|
||||
conversation on a new device. User should be able to control who can
|
||||
see their chat history and how much of the chat history they can see.
|
||||
|
||||
* Forward Secrecy of Discarded Chat History: Users should be able to discard
|
||||
history from their device, once they have discarded the history it should be
|
||||
impossible for an adversary to recover that history.
|
||||
|
||||
* Forward Secrecy of Future Messages: Users should be able to recover from
|
||||
disclosure of the chat history on their device.
|
||||
|
||||
* Deniablity of Chat History: It should not be possible to prove to a third
|
||||
party that a given user sent a message.
|
||||
|
||||
* Authenticity of Chat History: It should be possible to prove amoungst
|
||||
the members of a chat that a message sent by a user was authored by that
|
||||
user.
|
||||
|
||||
|
||||
Bonus Goals:
|
||||
|
||||
* Traffic Analysis: It would be nice if the protocol was resilient to traffic
|
||||
or metadata analysis. However it's not something we want to persue if it
|
||||
harms the usability of the protocol. It might be cool if there was a
|
||||
way for the user to could specify the trade off between performance and
|
||||
resilience to traffic analysis that they wanted.
|
||||
|
||||
|
||||
A Possible Design for Group Chat using Olm
|
||||
==========================================
|
||||
|
||||
Protecting the secrecy of history
|
||||
---------------------------------
|
||||
|
||||
Each message sent by a client has a 32-bit counter. This counter increments
|
||||
by one for each message sent by the client. This counter is used to advance a
|
||||
ratchet. The ratchet is split into a vector four 256-bit values,
|
||||
:math:`R_{n,j}` for :math:`j \in {0,1,2,3}`. The ratchet can be advanced as
|
||||
follows:
|
||||
|
||||
.. math::
|
||||
\begin{align}
|
||||
R_{2^24n,0} &= H_1\left(R_{2^24(i-1),0}\right) \\
|
||||
R_{2^24n,1} &= H_2\left(R_{2^24(i-1),0}\right) \\
|
||||
R_{2^16n,1} &= H_1\left(R_{2^16(i-1),1}\right) \\
|
||||
R_{2^16n,2} &= H_2\left(R_{2^16(i-1),1}\right) \\
|
||||
R_{2^8i,2} &= H_1\left(R_{2^8(i-1),2}\right) \\
|
||||
R_{2^8i,3} &= H_2\left(R_{2^8(i-1),2}\right) \\
|
||||
R_{i,3} &= H_1\left(R_{(i-1),3}\right)
|
||||
\end{align}
|
||||
|
||||
Where :math:`H_1` and :math:`H_2` are different hash functions. For example
|
||||
:math:`H_1` could be :math:`HMAC\left(X,\text{"\textbackslash x01"}\right)` and
|
||||
:math:`H_2` could be :math:`HMAC\left(X,\text{"\textbackslash x02"}\right)`.
|
||||
|
||||
So every :math:`2^24` iterations :math:`R_{n,1}` is reseeded from :math:`R_{n,0}`.
|
||||
Every :math:`2^16` iterations :math:`R_{n,2}` is reseeded from :math:`R_{n,1}`.
|
||||
Every :math:`2^8` iterations :math:`R_{n,3}` is reseeded from :math:`R_{n,2}`.
|
||||
|
||||
This scheme allows the ratchet to be advanced an arbitrary amount forwards
|
||||
while needing only 1024 hash computations.
|
||||
|
||||
This the value of the ratchet is hashed to generate the keys used to encrypt
|
||||
each mesage.
|
||||
|
||||
A client can decrypt chat history onwards from the earliest value of the
|
||||
ratchet it is aware of. But cannot decrypt history from before that point
|
||||
without reversing the hash function.
|
||||
|
||||
This allows a client to share its ability to decrypt chat history with another
|
||||
from a point in the conversation onwards by giving a copy of the ratchet at
|
||||
that point in the conversation.
|
||||
|
||||
A client can discard history by advancing a ratchet to beyond the last message
|
||||
they want to discard and then forgetting all previous values of the ratchet.
|
||||
|
||||
Proving and denying the authenticity of history
|
||||
-----------------------------------------------
|
||||
|
||||
Client sign the messages they send using a Ed25519 key generated per
|
||||
conversation. That key, along with the ratchet key, is distributed
|
||||
to other clients using 1:1 olm ratchets. Those 1:1 ratchets are started using
|
||||
Triple Diffie-Hellman which provides authenticity of the messages to the
|
||||
participants and deniability of the messages to third parties. Therefore
|
||||
any keys shared over those keys inherit the same levels of deniability and
|
||||
authenticity.
|
||||
|
||||
Protecting the secrecy of future messages
|
||||
-----------------------------------------
|
||||
|
||||
A client would need to generate new keys if it wanted to prevent access to
|
||||
messages beyond a given point in the conversation. It must generate new keys
|
||||
whenever someone leaves the room. It should generate new keys periodically
|
||||
anyway.
|
||||
|
||||
The frequency of key generation in a large room may need to be restricted to
|
||||
keep the frequency of messages broadcast over the individual 1:1 channels
|
||||
low.
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
{
|
||||
"type": "object",
|
||||
"title": "The current membership state of a user in the room.",
|
||||
"description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail. \n\nThe ``third_party_invite`` property will be set if this invite is an ``invite`` event and is the successor of an ``m.room.third_party_invite`` event, and absent otherwise.\n\nThis event also includes an ``invite_room_state`` key **outside the** ``content`` **key**. This contains an array of ``StrippedState`` Events. These events provide information on a few select state events such as the room name.",
|
||||
"description": "Adjusts the membership state for a user in a room. It is preferable to use the membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying to force this state change directly will fail. \n\nThe ``third_party_invite`` property will be set if this invite is an ``invite`` event and is the successor of an ``m.room.third_party_invite`` event, and absent otherwise.\n\nThis event may also include an ``invite_room_state`` key **outside the** ``content`` **key**. If present, this contains an array of ``StrippedState`` Events. These events provide information on a few select state events such as the room name.",
|
||||
"allOf": [{
|
||||
"$ref": "core-event-schema/state_event.json"
|
||||
}],
|
||||
|
|
|
|||
|
|
@ -94,6 +94,27 @@ def main(input_module, file_stream=None, out_dir=None, verbose=False):
|
|||
|
||||
return '\n\n'.join(output_lines)
|
||||
|
||||
def fieldwidths(input, keys, defaults=[], default_width=15):
|
||||
"""
|
||||
A template filter to help in the generation of tables.
|
||||
|
||||
Given a list of rows, returns a list giving the maximum length of the
|
||||
values in each column.
|
||||
|
||||
:param list[dict[str, str]] input: a list of rows. Each row should be a
|
||||
dict with the keys given in ``keys``.
|
||||
:param list[str] keys: the keys corresponding to the table columns
|
||||
:param list[int] defaults: for each column, the default column width.
|
||||
:param int default_width: if ``defaults`` is shorter than ``keys``, this
|
||||
will be used as a fallback
|
||||
"""
|
||||
def colwidth(key, default):
|
||||
return reduce(max, (len(row[key]) for row in input),
|
||||
default if default is not None else default_width)
|
||||
|
||||
results = map(colwidth, keys, defaults)
|
||||
return results
|
||||
|
||||
# make Jinja aware of the templates and filters
|
||||
env = Environment(
|
||||
loader=FileSystemLoader(in_mod.exports["templates"]),
|
||||
|
|
@ -103,6 +124,7 @@ def main(input_module, file_stream=None, out_dir=None, verbose=False):
|
|||
env.filters["indent"] = indent
|
||||
env.filters["indent_block"] = indent_block
|
||||
env.filters["wrap"] = wrap
|
||||
env.filters["fieldwidths"] = fieldwidths
|
||||
|
||||
# load up and parse the lowest single units possible: we don't know or care
|
||||
# which spec section will use it, we just need it there in memory for when
|
||||
|
|
|
|||
|
|
@ -1,17 +1,8 @@
|
|||
{% import 'tables.tmpl' as tables -%}
|
||||
|
||||
{{common_event.title}} Fields
|
||||
{{(7 + common_event.title | length) * title_kind}}
|
||||
|
||||
{{common_event.desc | wrap(80)}}
|
||||
|
||||
================== ================= ===========================================
|
||||
Key Type Description
|
||||
================== ================= ===========================================
|
||||
{% for row in common_event.rows -%}
|
||||
{# -#}
|
||||
{# Row type needs to prepend spaces to line up with the type column (19 ch) -#}
|
||||
{# Desc needs to prepend the required text (maybe) and prepend spaces too -#}
|
||||
{# It also needs to then wrap inside the desc col (43 ch width) -#}
|
||||
{# -#}
|
||||
{{row.key}}{{row.type|indent(19-row.key|length)}}{{row.desc | indent(18 - (row.type|length)) |wrap(43) |indent_block(37)}}
|
||||
{% endfor -%}
|
||||
================== ================= ===========================================
|
||||
{{ tables.paramtable(common_event.rows, ["Key", "Type", "Description"]) }}
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
{% import 'tables.tmpl' as tables -%}
|
||||
|
||||
``{{event.type}}``
|
||||
{{(4 + event.type | length) * title_kind}}
|
||||
*{{event.typeof}}*
|
||||
|
|
@ -7,18 +9,7 @@
|
|||
{% for table in event.content_fields -%}
|
||||
{{"``"+table.title+"``" if table.title else "" }}
|
||||
|
||||
======================= ================= ===========================================
|
||||
{{table.title or "Content"}} Key Type Description
|
||||
======================= ================= ===========================================
|
||||
{% for row in table.rows -%}
|
||||
{# -#}
|
||||
{# Row type needs to prepend spaces to line up with the type column (19 ch) -#}
|
||||
{# Desc needs to prepend the required text (maybe) and prepend spaces too -#}
|
||||
{# It also needs to then wrap inside the desc col (43 ch width) -#}
|
||||
{# -#}
|
||||
{{row.key}}{{row.type|indent(24-row.key|length)}}{{row.desc|wrap(43,row.req_str | indent(18 - (row.type|length))) |indent_block(42)}}
|
||||
{% endfor -%}
|
||||
======================= ================= ===========================================
|
||||
{{ tables.paramtable(table.rows, [(table.title or "Content") ~ " Key", "Type", "Description"]) }}
|
||||
|
||||
{% endfor %}
|
||||
Example:
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
{% import 'tables.tmpl' as tables -%}
|
||||
|
||||
``{{endpoint.method}} {{endpoint.path}}``
|
||||
{{(5 + (endpoint.path | length) + (endpoint.method | length)) * title_kind}}
|
||||
{% if "alias_for_path" in endpoint -%}
|
||||
|
|
@ -13,17 +15,7 @@
|
|||
|
||||
Request format:
|
||||
{% if (endpoint.req_param_by_loc | length) %}
|
||||
=========================================== ================= ===========================================
|
||||
Parameter Value Description
|
||||
=========================================== ================= ===========================================
|
||||
{% for loc in endpoint.req_param_by_loc -%}
|
||||
*{{loc}} parameters*
|
||||
---------------------------------------------------------------------------------------------------------
|
||||
{% for param in endpoint.req_param_by_loc[loc] -%}
|
||||
{{param.key}}{{param.type|indent(44-param.key|length)}}{{param.desc|indent(18-param.type|length)|wrap(43)|indent_block(62)}}
|
||||
{% endfor -%}
|
||||
{% endfor -%}
|
||||
=========================================== ================= ===========================================
|
||||
{{ tables.split_paramtable(endpoint.req_param_by_loc) }}
|
||||
{% else %}
|
||||
`No parameters`
|
||||
{% endif %}
|
||||
|
|
@ -34,18 +26,7 @@ Response format:
|
|||
{% for table in endpoint.res_tables -%}
|
||||
{{"``"+table.title+"``" if table.title else "" }}
|
||||
|
||||
======================= ========================= ==========================================
|
||||
Param Type Description
|
||||
======================= ========================= ==========================================
|
||||
{% for row in table.rows -%}
|
||||
{# -#}
|
||||
{# Row type needs to prepend spaces to line up with the type column (20 ch) -#}
|
||||
{# Desc needs to prepend the required text (maybe) and prepend spaces too -#}
|
||||
{# It also needs to then wrap inside the desc col (42 ch width) -#}
|
||||
{# -#}
|
||||
{{row.key}}{{row.type|indent(24-row.key|length)}}{{row.desc|wrap(42,row.req_str | indent(26 - (row.type|length))) |indent_block(50)}}
|
||||
{% endfor -%}
|
||||
======================= ========================= ==========================================
|
||||
{{ tables.paramtable(table.rows) }}
|
||||
|
||||
{% endfor %}
|
||||
{% endif -%}
|
||||
|
|
|
|||
|
|
@ -1,21 +1,12 @@
|
|||
{% import 'tables.tmpl' as tables -%}
|
||||
|
||||
``{{event.msgtype}}``
|
||||
{{(4 + event.msgtype | length) * title_kind}}
|
||||
{{event.desc | wrap(80)}}
|
||||
{% for table in event.content_fields -%}
|
||||
{{"``"+table.title+"``" if table.title else "" }}
|
||||
|
||||
================== ================= ===========================================
|
||||
{{table.title or "Content"}} Key Type Description
|
||||
================== ================= ===========================================
|
||||
{% for row in table.rows -%}
|
||||
{# -#}
|
||||
{# Row type needs to prepend spaces to line up with the type column (19 ch) -#}
|
||||
{# Desc needs to prepend the required text (maybe) and prepend spaces too -#}
|
||||
{# It also needs to then wrap inside the desc col (43 ch width) -#}
|
||||
{# -#}
|
||||
{{row.key}}{{row.type|indent(19-row.key|length)}}{{row.desc|wrap(43,row.req_str | indent(18 - (row.type|length))) |indent_block(37)}}
|
||||
{% endfor -%}
|
||||
================== ================= ===========================================
|
||||
{{ tables.paramtable(table.rows, [(table.title or "Content") ~ " Key", "Type", "Description"]) }}
|
||||
|
||||
{% endfor %}
|
||||
Example:
|
||||
|
|
|
|||
104
templating/matrix_templates/templates/tables.tmpl
Normal file
104
templating/matrix_templates/templates/tables.tmpl
Normal file
|
|
@ -0,0 +1,104 @@
|
|||
{#
|
||||
# A set of macros for generating RST tables
|
||||
#}
|
||||
|
||||
|
||||
{#
|
||||
# write a table for a list of parameters.
|
||||
#
|
||||
# 'rows' is the list of parameters. Each row should have the keys
|
||||
# 'key', 'type', and 'desc'.
|
||||
#}
|
||||
{% macro paramtable(rows, titles=["Parameter", "Type", "Description"]) -%}
|
||||
{{ split_paramtable({None: rows}, titles) }}
|
||||
{% endmacro %}
|
||||
|
||||
|
||||
{#
|
||||
# write a table for the request parameters, split by location.
|
||||
# 'rows_by_loc' is a map from location to a list of parameters.
|
||||
#
|
||||
# As a special case, if a key of 'rows_by_loc' is 'None', no title row is
|
||||
# written for that location. This is used by the standard 'paramtable' macro.
|
||||
#}
|
||||
{% macro split_paramtable(rows_by_loc,
|
||||
titles=["Parameter", "Type", "Description"]) -%}
|
||||
|
||||
{% set rowkeys = ['key', 'type', 'desc'] %}
|
||||
{% set titlerow = {'key': titles[0], 'type': titles[1], 'desc': titles[2]} %}
|
||||
|
||||
{# We need the rows flattened into a single list. Abuse the 'sum' filter to
|
||||
# join arrays instead of add numbers. -#}
|
||||
{% set flatrows = rows_by_loc.values()|sum(start=[]) -%}
|
||||
|
||||
{# Figure out the widths of the columns. The last column is always 50 characters
|
||||
# wide; the others default to 10, but stretch if there is wider text in the
|
||||
# column. -#}
|
||||
{% set fieldwidths = (([titlerow] + flatrows) |
|
||||
fieldwidths(rowkeys[0:-1], [10, 10])) + [50] -%}
|
||||
|
||||
{{ tableheader(fieldwidths) }}
|
||||
{{ tablerow(fieldwidths, titlerow, rowkeys) }}
|
||||
{{ tableheader(fieldwidths) }}
|
||||
{% for loc in rows_by_loc -%}
|
||||
|
||||
{% if loc != None -%}
|
||||
{{ tablespan(fieldwidths, "*" ~ loc ~ " parameters*") }}
|
||||
{% endif -%}
|
||||
|
||||
{% for row in rows_by_loc[loc] -%}
|
||||
{{ tablerow(fieldwidths, row, rowkeys) }}
|
||||
{% endfor -%}
|
||||
{% endfor -%}
|
||||
|
||||
{{ tableheader(fieldwidths) }}
|
||||
{% endmacro %}
|
||||
|
||||
|
||||
|
||||
{#
|
||||
# Write a table header row, for the given column widths
|
||||
#}
|
||||
{% macro tableheader(widths) -%}
|
||||
{% for arg in widths -%}
|
||||
{{"="*arg}} {% endfor -%}
|
||||
{% endmacro %}
|
||||
|
||||
|
||||
|
||||
{#
|
||||
# Write a normal table row. Each of 'widths' and 'keys' should be sequences
|
||||
# of the same length; 'widths' defines the column widths, and 'keys' the
|
||||
# attributes of 'row' to look up for values to put in the columns.
|
||||
#}
|
||||
{% macro tablerow(widths, row, keys) -%}
|
||||
{% for key in keys -%}
|
||||
{% set value=row[key] -%}
|
||||
{% if not loop.last -%}
|
||||
{# the first few columns need space after them -#}
|
||||
{{ value }}{{" "*(1+widths[loop.index0]-value|length) -}}
|
||||
{% else -%}
|
||||
{# the last column needs wrapping and indenting (by the sum of the widths of
|
||||
the preceding columns, plus the number of preceding columns (for the
|
||||
separators)) -#}
|
||||
{{ value | wrap(widths[loop.index0]) |
|
||||
indent_block(widths[0:-1]|sum + loop.index0) -}}
|
||||
{% endif -%}
|
||||
{% endfor -%}
|
||||
{% endmacro %}
|
||||
|
||||
|
||||
|
||||
|
||||
{#
|
||||
# write a tablespan row. This is a single value which spans the entire table.
|
||||
#}
|
||||
{% macro tablespan(widths, value) -%}
|
||||
{{value}}
|
||||
{# we write a trailing space to stop the separator being misinterpreted
|
||||
# as a header line. -#}
|
||||
{{"-"*(widths|sum + widths|length -1)}} {% endmacro %}
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
@ -171,6 +171,7 @@ def get_json_schema_object_fields(obj, enforce_title=False, include_parents=Fals
|
|||
include_parents=include_parents,
|
||||
)
|
||||
value_type = nested_objects[0]["title"]
|
||||
value_id = value_type
|
||||
|
||||
tables += [x for x in nested_objects if not x.get("no-table")]
|
||||
elif prop_type == "array":
|
||||
|
|
@ -181,12 +182,14 @@ def get_json_schema_object_fields(obj, enforce_title=False, include_parents=Fals
|
|||
enforce_title=True,
|
||||
include_parents=include_parents,
|
||||
)
|
||||
value_type = "[%s]" % nested_objects[0]["title"]
|
||||
value_id = nested_objects[0]["title"]
|
||||
value_type = "[%s]" % value_id
|
||||
tables += nested_objects
|
||||
else:
|
||||
value_type = props[key_name]["items"]["type"]
|
||||
if isinstance(value_type, list):
|
||||
value_type = " or ".join(value_type)
|
||||
value_id = value_type
|
||||
value_type = "[%s]" % value_type
|
||||
array_enums = props[key_name]["items"].get("enum")
|
||||
if array_enums:
|
||||
|
|
@ -201,6 +204,7 @@ def get_json_schema_object_fields(obj, enforce_title=False, include_parents=Fals
|
|||
)
|
||||
else:
|
||||
value_type = prop_type
|
||||
value_id = prop_type
|
||||
if props[key_name].get("enum"):
|
||||
if len(props[key_name].get("enum")) > 1:
|
||||
value_type = "enum"
|
||||
|
|
@ -221,6 +225,7 @@ def get_json_schema_object_fields(obj, enforce_title=False, include_parents=Fals
|
|||
fields["rows"].append({
|
||||
"key": key_name,
|
||||
"type": value_type,
|
||||
"id": value_id,
|
||||
"required": required,
|
||||
"desc": desc,
|
||||
"req_str": "**Required.** " if required else ""
|
||||
|
|
@ -365,10 +370,16 @@ class MatrixUnits(Units):
|
|||
|
||||
if req_tables > 1:
|
||||
for table in req_tables[1:]:
|
||||
nested_key_name = [
|
||||
s["key"] for s in req_tables[0]["rows"] if
|
||||
s["type"] == ("%s" % (table["title"],))
|
||||
][0]
|
||||
nested_key_name = {
|
||||
"key": s["key"]
|
||||
for rtable in req_tables
|
||||
for s in rtable["rows"]
|
||||
if s["id"] == table["title"]
|
||||
}.get("key", None)
|
||||
|
||||
if nested_key_name is None:
|
||||
raise Exception("Failed to find table for %r" % (table["title"],))
|
||||
|
||||
for row in table["rows"]:
|
||||
row["key"] = "%s.%s" % (nested_key_name, row["key"])
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue