From 6edb90a08e50c75536fa51f7075e42f4703db6e6 Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 14 Nov 2017 14:36:38 +0000 Subject: [PATCH 01/13] Document threepids Adds the /msisdn' 3pid type and generally fleshes out what a 3pid is and how they work. This merges most of the work from Max Dor in https://github.com/matrix-org/matrix-doc/pull/1039 with some tweaks and additions. --- api/identity/lookup.yaml | 8 +++-- specification/appendices/threepids.rst | 48 ++++++++++++++++++++++++++ specification/identity_service_api.rst | 5 +++ specification/targets.yaml | 1 + 4 files changed, 59 insertions(+), 3 deletions(-) create mode 100644 specification/appendices/threepids.rst diff --git a/api/identity/lookup.yaml b/api/identity/lookup.yaml index 18e5e77d..bd3635b4 100644 --- a/api/identity/lookup.yaml +++ b/api/identity/lookup.yaml @@ -1,4 +1,6 @@ # Copyright 2016 OpenMarket Ltd +# Copyright 2017 Kamax.io +# Copyright 2017 New Vector Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -32,13 +34,13 @@ paths: type: string name: medium required: true - description: The literal string "email". + description: The medium type of the 3pid. See `Appendices`_. x-example: "email" - in: query type: string name: address required: true - description: The email address being looked up. + description: The address of the 3pid being looked up. See `Appendices`_. x-example: "louise@bobs.burgers" responses: 200: @@ -82,4 +84,4 @@ paths: description: The unix timestamp at which the association was verified. signatures: type: object - description: The signatures of the verifying identity service which show that the association should be trusted, if you trust the verifying identity service. + description: The signatures of the verifying identity services which show that the association should be trusted, if you trust the verifying identity services. diff --git a/specification/appendices/threepids.rst b/specification/appendices/threepids.rst new file mode 100644 index 00000000..2f8147a8 --- /dev/null +++ b/specification/appendices/threepids.rst @@ -0,0 +1,48 @@ +.. Copyright 2017 Kamax.io +.. +.. Licensed under the Apache License, Version 2.0 (the "License"); +.. you may not use this file except in compliance with the License. +.. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, +.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +.. See the License for the specific language governing permissions and +.. limitations under the License. + +3PID Types +---------- +3PIDs represent identifiers on other namespaces that might be associated with a +particular person. They comprise a tuple of `medium` which is a string that +identifies the namespace in which the identifier exists and an `address`: a +string representing the identifier in that namespace. This must be a canonical +form of the identifier, ie. if multiple strings could represent the same +identifier, only one of these strings must be used in a 3PID address, in a +well-defined manner. + +For example, for e-mail, the `medium` is 'email' and the `address` would be the +email address, eg. the string 'bob@example.com'. Since domain resolution is +case-insensitive, the email address 'bob@Example.com' also has a 3PID address +of 'bob@example.com' and not 'bob@Example.com'. + +The namespaces defined by this specification are listed below. More namespaces +may be defined in future versions of this specification. + +E-Mail +~~~~~~ +Medium: `email` + +Represents E-Mail addresses. The `address` is the raw email address in +user@domain form with the domain in lowercase. It must not contain other text +such as real name, angle brackets or a mailto: prefix. + +PSTN Phone numbers +~~~~~~~~~~~~~~~~~~ +Medium: `msisdn` + +Represents telephone numbers on the public switched telephone network. The +`address` is the telephone number represented as a MSISDN (Mobile Station +International Subscriber Directory Number) as defined by the E.164 numbering +plan. Note that MSISDNs do not include a leading '+'. diff --git a/specification/identity_service_api.rst b/specification/identity_service_api.rst index fa03e162..90b20d23 100644 --- a/specification/identity_service_api.rst +++ b/specification/identity_service_api.rst @@ -1,4 +1,6 @@ .. Copyright 2016 OpenMarket Ltd +.. Copyright 2017 Kamax.io +.. Copyright 2017 New Vector Ltd .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. @@ -52,6 +54,8 @@ necessarily provide evidence that they have validated associations, but claim to have done so. Establishing the trustworthiness of an individual identity service is left as an exercise for the client. +3PID types are described in the `Appendices`_. + Privacy ------- @@ -291,3 +295,4 @@ It will look up ``token`` which was stored in a call to ``store-invite``, and fe } .. _`Unpadded Base64`: ../appendices.html#unpadded-base64 +.. _`Appendices`: ../appendices.html#threepids diff --git a/specification/targets.yaml b/specification/targets.yaml index fb68e13d..6a52fd84 100644 --- a/specification/targets.yaml +++ b/specification/targets.yaml @@ -33,6 +33,7 @@ targets: files: - appendices.rst - appendices/base64.rst + - appendices/threepids.rst - appendices/signing_json.rst - appendices/identifier_grammar.rst - appendices/threat_model.rst From 0f5a4efce6affc1dc002097d16028aa8598e0582 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Tue, 14 Nov 2017 07:52:51 -0700 Subject: [PATCH 02/13] Spec URL previews (#1064) * Spec URL previews * Require authentication on /upload and /preview_url --- api/client-server/content-repo.yaml | 78 +++++++++++++++++++++++++++++ changelogs/client_server.rst | 5 ++ 2 files changed, 83 insertions(+) diff --git a/api/client-server/content-repo.yaml b/api/client-server/content-repo.yaml index ebb63ba4..e8f23c12 100644 --- a/api/client-server/content-repo.yaml +++ b/api/client-server/content-repo.yaml @@ -18,15 +18,23 @@ info: host: localhost:8008 schemes: - https + - http basePath: /_matrix/media/%CLIENT_MAJOR_VERSION% +consumes: + - application/json + - "*/*" produces: - application/json - "*/*" +securityDefinitions: + $ref: definitions/security.yaml paths: "/upload": post: summary: Upload some content to the content repository. produces: ["application/json"] + security: + - accessToken: [] parameters: - in: header name: Content-Type @@ -60,6 +68,10 @@ paths: application/json: { "content_uri": "mxc://example.com/AQwafuaFswefuhsfAFAgsw" } + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/error.yaml" tags: - Media "/download/{serverName}/{mediaId}": @@ -93,6 +105,10 @@ paths: type: "string" schema: type: file + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/error.yaml" tags: - Media "/download/{serverName}/{mediaId}/{fileName}": @@ -133,6 +149,10 @@ paths: type: "string" schema: type: file + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/error.yaml" tags: - Media "/thumbnail/{serverName}/{mediaId}": @@ -184,5 +204,63 @@ paths: enum: ["image/jpeg", "image/png"] schema: type: file + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/error.yaml" + tags: + - Media + "/preview_url": + get: + summary: "Get information about a URL for a client" + produces: ["application/json"] + security: + - accessToken: [] + parameters: + - in: query + type: string + x-example: "https://matrix.org" + name: url + description: "The URL to get a preview of" + required: true + - in: query + type: number + x-example: 1510610716656 + name: ts + description: |- + The preferred point in time to return a preview for. The server may + return a newer version if it does not have the requested version + available. + responses: + 200: + description: |- + The OpenGraph data for the URL, which may be empty. Some values are + replaced with matrix equivalents if they are provided in the response. + The differences from the OpenGraph protocol are described here. + schema: + type: object + properties: + "matrix:image:size": + type: number + description: |- + The byte-size of the image. Omitted if there is no image attached. + "og:image": + type: string + description: |- + An MXC URI to the image. Ommitted if there is no image. + examples: + application/json: { + "og:title": "Matrix Blog Post", + "og:description": "This is a really cool blog post from matrix.org", + "og:image": "mxc://example.com/ascERGshawAWawugaAcauga", + "og:image:type": "image/png", + "og:image:height": 48, + "og:image:width": 48, + "matrix:image:size": 102400 + } + 429: + description: This request was rate-limited. + schema: + "$ref": "definitions/error.yaml" tags: - Media diff --git a/changelogs/client_server.rst b/changelogs/client_server.rst index 10387c56..e428605e 100644 --- a/changelogs/client_server.rst +++ b/changelogs/client_server.rst @@ -63,6 +63,9 @@ - ``GET /account/whoami`` (`#1063 `_). + - ``GET /media/{version}/preview_url`` + (`#1064 `_). + - Spec clarifications: - Add endpoints and logic for invites and third-party invites to the federation @@ -99,6 +102,8 @@ - Clarify the relationship between ``username`` and ``user_id`` in the ``/register`` API (`#1032 `_). + - Clarify rate limiting and security for content repository. + (`#1064 `_). r0.2.0 ====== From b789f615229b445d5399350c35bfc1314516002a Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 14 Nov 2017 14:58:30 +0000 Subject: [PATCH 03/13] Make link more specific --- specification/identity_service_api.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/identity_service_api.rst b/specification/identity_service_api.rst index 90b20d23..19275593 100644 --- a/specification/identity_service_api.rst +++ b/specification/identity_service_api.rst @@ -54,7 +54,7 @@ necessarily provide evidence that they have validated associations, but claim to have done so. Establishing the trustworthiness of an individual identity service is left as an exercise for the client. -3PID types are described in the `Appendices`_. +3PID types are described in `3PID Types`_ Appendix. Privacy ------- @@ -295,4 +295,4 @@ It will look up ``token`` which was stored in a call to ``store-invite``, and fe } .. _`Unpadded Base64`: ../appendices.html#unpadded-base64 -.. _`Appendices`: ../appendices.html#threepids +.. _`3PID Types`: ../appendices.html#pid-types From 493b9362096d7d7463570d120d06a4881b13b49c Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 14 Nov 2017 15:01:58 +0000 Subject: [PATCH 04/13] Make these links more specific too --- api/identity/lookup.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/api/identity/lookup.yaml b/api/identity/lookup.yaml index bd3635b4..ed16e4bf 100644 --- a/api/identity/lookup.yaml +++ b/api/identity/lookup.yaml @@ -34,13 +34,13 @@ paths: type: string name: medium required: true - description: The medium type of the 3pid. See `Appendices`_. + description: The medium type of the 3pid. See the `3PID Types`_ Appendix. x-example: "email" - in: query type: string name: address required: true - description: The address of the 3pid being looked up. See `Appendices`_. + description: The address of the 3pid being looked up. See the `3PID Types`_ Appendix. x-example: "louise@bobs.burgers" responses: 200: @@ -85,3 +85,5 @@ paths: signatures: type: object description: The signatures of the verifying identity services which show that the association should be trusted, if you trust the verifying identity services. + +.. _`3PID Types`: ../appendices.html#pid-types From e80675c9e9135050f3b7334a40c680dcf2485b00 Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 14 Nov 2017 15:03:51 +0000 Subject: [PATCH 05/13] Link in markdown, not swagger --- api/identity/lookup.yaml | 2 -- specification/client_server_api.rst | 1 + 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/api/identity/lookup.yaml b/api/identity/lookup.yaml index ed16e4bf..d039f2b5 100644 --- a/api/identity/lookup.yaml +++ b/api/identity/lookup.yaml @@ -85,5 +85,3 @@ paths: signatures: type: object description: The signatures of the verifying identity services which show that the association should be trusted, if you trust the verifying identity services. - -.. _`3PID Types`: ../appendices.html#pid-types diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 09f73bb9..7cd79479 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -1446,3 +1446,4 @@ have to wait in milliseconds before they can try again. .. _/user//account_data/: #put-matrix-client-%CLIENT_MAJOR_VERSION%-user-userid-account-data-type .. _`Unpadded Base64`: ../appendices.html#unpadded-base64 +.. _`3PID Types`: ../appendices.html#pid-types From 8d557ec9dc0a78352f379f1308daf78a7acee382 Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 14 Nov 2017 15:05:58 +0000 Subject: [PATCH 06/13] Actually this isn't necessary in this section --- specification/client_server_api.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/specification/client_server_api.rst b/specification/client_server_api.rst index 7cd79479..09f73bb9 100644 --- a/specification/client_server_api.rst +++ b/specification/client_server_api.rst @@ -1446,4 +1446,3 @@ have to wait in milliseconds before they can try again. .. _/user//account_data/: #put-matrix-client-%CLIENT_MAJOR_VERSION%-user-userid-account-data-type .. _`Unpadded Base64`: ../appendices.html#unpadded-base64 -.. _`3PID Types`: ../appendices.html#pid-types From ff85ec055ec51922ae94d1e8a4b8b938e5fe9a28 Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 14 Nov 2017 15:07:22 +0000 Subject: [PATCH 07/13] Move 3pid types down --- specification/targets.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/targets.yaml b/specification/targets.yaml index 6a52fd84..62799afe 100644 --- a/specification/targets.yaml +++ b/specification/targets.yaml @@ -33,9 +33,9 @@ targets: files: - appendices.rst - appendices/base64.rst - - appendices/threepids.rst - appendices/signing_json.rst - appendices/identifier_grammar.rst + - appendices/threepids.rst - appendices/threat_model.rst - appendices/test_vectors.rst groups: # reusable blobs of files when prefixed with 'group:' From 3487f5d75a20790a8c9eff2d3734a48891b4ae61 Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 14 Nov 2017 15:08:56 +0000 Subject: [PATCH 08/13] Spell out what we mean by a 3pid (and consequently rejig all the fixed-width formatting) --- specification/appendices/threepids.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/specification/appendices/threepids.rst b/specification/appendices/threepids.rst index 2f8147a8..e578a559 100644 --- a/specification/appendices/threepids.rst +++ b/specification/appendices/threepids.rst @@ -14,13 +14,13 @@ 3PID Types ---------- -3PIDs represent identifiers on other namespaces that might be associated with a -particular person. They comprise a tuple of `medium` which is a string that -identifies the namespace in which the identifier exists and an `address`: a -string representing the identifier in that namespace. This must be a canonical -form of the identifier, ie. if multiple strings could represent the same -identifier, only one of these strings must be used in a 3PID address, in a -well-defined manner. +Third Party Identifiers (3PIDs) represent identifiers on other namespaces that +might be associated with a particular person. They comprise a tuple of `medium` +which is a string that identifies the namespace in which the identifier exists +and an `address`: a string representing the identifier in that namespace. This +must be a canonical form of the identifier, ie. if multiple strings could +represent the same identifier, only one of these strings must be used in a 3PID +address, in a well-defined manner. For example, for e-mail, the `medium` is 'email' and the `address` would be the email address, eg. the string 'bob@example.com'. Since domain resolution is From 1e4cba6b5577fb59892fe63ea2442b436955b1ee Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 14 Nov 2017 15:10:11 +0000 Subject: [PATCH 09/13] Don't linkify email addresses in examples --- specification/appendices/threepids.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specification/appendices/threepids.rst b/specification/appendices/threepids.rst index e578a559..64dfa4b1 100644 --- a/specification/appendices/threepids.rst +++ b/specification/appendices/threepids.rst @@ -23,9 +23,9 @@ represent the same identifier, only one of these strings must be used in a 3PID address, in a well-defined manner. For example, for e-mail, the `medium` is 'email' and the `address` would be the -email address, eg. the string 'bob@example.com'. Since domain resolution is -case-insensitive, the email address 'bob@Example.com' also has a 3PID address -of 'bob@example.com' and not 'bob@Example.com'. +email address, eg. the string ``bob@example.com``. Since domain resolution is +case-insensitive, the email address ``bob@Example.com`` also has a 3PID address +of ``bob@example.com`` and not ``bob@Example.com``. The namespaces defined by this specification are listed below. More namespaces may be defined in future versions of this specification. From 4526ff2ea69d85bd13071baa8b6fd65bbe493c68 Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 14 Nov 2017 15:11:30 +0000 Subject: [PATCH 10/13] Double backticks in rst :/ --- specification/appendices/threepids.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/specification/appendices/threepids.rst b/specification/appendices/threepids.rst index 64dfa4b1..a275143f 100644 --- a/specification/appendices/threepids.rst +++ b/specification/appendices/threepids.rst @@ -15,14 +15,14 @@ 3PID Types ---------- Third Party Identifiers (3PIDs) represent identifiers on other namespaces that -might be associated with a particular person. They comprise a tuple of `medium` +might be associated with a particular person. They comprise a tuple of ``medium`` which is a string that identifies the namespace in which the identifier exists -and an `address`: a string representing the identifier in that namespace. This +and an ``address``: a string representing the identifier in that namespace. This must be a canonical form of the identifier, ie. if multiple strings could represent the same identifier, only one of these strings must be used in a 3PID address, in a well-defined manner. -For example, for e-mail, the `medium` is 'email' and the `address` would be the +For example, for e-mail, the ``medium`` is 'email' and the ``address`` would be the email address, eg. the string ``bob@example.com``. Since domain resolution is case-insensitive, the email address ``bob@Example.com`` also has a 3PID address of ``bob@example.com`` and not ``bob@Example.com``. @@ -32,17 +32,17 @@ may be defined in future versions of this specification. E-Mail ~~~~~~ -Medium: `email` +Medium: ``email`` -Represents E-Mail addresses. The `address` is the raw email address in +Represents E-Mail addresses. The ``address`` is the raw email address in user@domain form with the domain in lowercase. It must not contain other text such as real name, angle brackets or a mailto: prefix. PSTN Phone numbers ~~~~~~~~~~~~~~~~~~ -Medium: `msisdn` +Medium: ``msisdn`` Represents telephone numbers on the public switched telephone network. The -`address` is the telephone number represented as a MSISDN (Mobile Station +``address`` is the telephone number represented as a MSISDN (Mobile Station International Subscriber Directory Number) as defined by the E.164 numbering plan. Note that MSISDNs do not include a leading '+'. From e12c0b54ff8c93a77702faed74232abb7eec0ea8 Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 14 Nov 2017 15:12:30 +0000 Subject: [PATCH 11/13] Formatting & grammar --- specification/appendices/threepids.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/appendices/threepids.rst b/specification/appendices/threepids.rst index a275143f..cd312710 100644 --- a/specification/appendices/threepids.rst +++ b/specification/appendices/threepids.rst @@ -16,9 +16,9 @@ ---------- Third Party Identifiers (3PIDs) represent identifiers on other namespaces that might be associated with a particular person. They comprise a tuple of ``medium`` -which is a string that identifies the namespace in which the identifier exists +which is a string that identifies the namespace in which the identifier exists, and an ``address``: a string representing the identifier in that namespace. This -must be a canonical form of the identifier, ie. if multiple strings could +must be a canonical form of the identifier, *ie.* if multiple strings could represent the same identifier, only one of these strings must be used in a 3PID address, in a well-defined manner. From 9c5972ffe34039f965cb184ac24dfb64c2d6601d Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 14 Nov 2017 15:31:02 +0000 Subject: [PATCH 12/13] More formatting / grammar --- specification/appendices/threepids.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specification/appendices/threepids.rst b/specification/appendices/threepids.rst index cd312710..652ec4c7 100644 --- a/specification/appendices/threepids.rst +++ b/specification/appendices/threepids.rst @@ -18,12 +18,12 @@ Third Party Identifiers (3PIDs) represent identifiers on other namespaces that might be associated with a particular person. They comprise a tuple of ``medium`` which is a string that identifies the namespace in which the identifier exists, and an ``address``: a string representing the identifier in that namespace. This -must be a canonical form of the identifier, *ie.* if multiple strings could +must be a canonical form of the identifier, *i.e.* if multiple strings could represent the same identifier, only one of these strings must be used in a 3PID address, in a well-defined manner. For example, for e-mail, the ``medium`` is 'email' and the ``address`` would be the -email address, eg. the string ``bob@example.com``. Since domain resolution is +email address, *e.g.* the string ``bob@example.com``. Since domain resolution is case-insensitive, the email address ``bob@Example.com`` also has a 3PID address of ``bob@example.com`` and not ``bob@Example.com``. @@ -35,7 +35,7 @@ E-Mail Medium: ``email`` Represents E-Mail addresses. The ``address`` is the raw email address in -user@domain form with the domain in lowercase. It must not contain other text +``user@domain`` form with the domain in lowercase. It must not contain other text such as real name, angle brackets or a mailto: prefix. PSTN Phone numbers From 14df31458b86a4c1be022d842cf15cda63a28dde Mon Sep 17 00:00:00 2001 From: David Baker Date: Tue, 14 Nov 2017 15:39:25 +0000 Subject: [PATCH 13/13] Attempt to clarify example a little --- specification/appendices/threepids.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/appendices/threepids.rst b/specification/appendices/threepids.rst index 652ec4c7..84860740 100644 --- a/specification/appendices/threepids.rst +++ b/specification/appendices/threepids.rst @@ -24,8 +24,8 @@ address, in a well-defined manner. For example, for e-mail, the ``medium`` is 'email' and the ``address`` would be the email address, *e.g.* the string ``bob@example.com``. Since domain resolution is -case-insensitive, the email address ``bob@Example.com`` also has a 3PID address -of ``bob@example.com`` and not ``bob@Example.com``. +case-insensitive, the email address ``bob@Example.com`` is also has the 3PID address +of ``bob@example.com`` (without the capital 'e') rather than ``bob@Example.com``. The namespaces defined by this specification are listed below. More namespaces may be defined in future versions of this specification.