Privacy

A few years ago, back when Matrix was originally implementing end-to-end encryption, we asked Moxie (the project lead for Signal) whether he’d ever consider connecting Signal (then TextSecure) to Matrix. After all, one of Matrix’s goals is to be an interoperability layer between other communication silos, and one of the reasons for us using Signal’s Double Ratchet Algorithm for Matrix’s encryption was to increase our chances of one day connecting with other apps using the same algorithm (Signal, WhatsApp, Google Allo, Skype, etc). Moxie politely declined, and then a few months later wrote “The ecosystem is moving” to elaborate his thoughts on why he feels he “no longer believes that it is possible to build a competitive federated messenger at all.”

At the time we didn’t respond via a blog post; instead we ended up talking it through a few times in person to see how misaligned we really were. The conclusion was that we agreed to disagree and Moxie said he’d be happy to be proved wrong, and wished us good luck. However, the subject has come up again thanks to Moxie’s talk on the same subject at 36C3 last week, and we keep getting asked to write a formal response on the Matrix side. So, here’s an attempt to do so. (Moxie didn’t want the 36C3 talk recorded, and I haven’t watched it, so this is responding to the original blog post).

From my perspective, the main points proposed in ‘The ecosystem is moving’ boil down to:

• Decentralised systems are harder to design and build than centralised ones, as coordination is harder if you don’t have a single authority to trust.

• Decentralised systems are harder and slower to evolve than centralised ones, as you can’t force participants to rapidly roll out (or even agree on) new features.

• Users in federated systems tend to coalesce around the best/biggest server that the bulk of people use - which means that server typically gets to see a disproportionate amount of communication metadata (who’s talking to who, and when), and has disproportionate power over the network, which could bully others away from running their own deployments.

• If users don’t trust their app provider, they can always go switch apps, which gives them freedom.

• Open systems are less secure because you have no control over the quality of the implementations - if anyone can bring their own client or server to the table, all it takes is one bad implementation to compromise everyone in the vicinity.

Now, all of these points are valid to some extent.

It’s absolutely true that decentralised systems are harder than centralised ones. Prior to Matrix we built centralised comms systems - we literally can do a side-by-side comparison for the same team to see how easily and fast we built our centralised comms system relative to Matrix. Empirically It took us around 6 times longer to get to the same feature-set with Matrix.

It’s also true that decentralised systems are harder to evolve than centralised ones - you can’t just push out a given feature with a single app update, but you have to agree and publish a public spec, support incremental migration, and build governance processes and community dynamics which encourage everyone to implement and upgrade. This is hard, but not impossible: we’ve spent loads of time and money on Matrix’s governance model and spec process to get it right. It’s still not perfect, but we haven’t seen much fragmentation so far, and when we’re pushing out a feature empirically we can and do go just as fast as the centralised alternatives. (E2E by default is a bit of a special case because we’ve had to go and reimplement many features users take for granted today in an E2E-capable manner, but we’re sprinting to get it done in the coming weeks). A bigger problem is that there are hundreds of spec change proposals which folks would like to see in the protocol, and finding a way to manage expectations and parallelise spec progress is hard - something we’re looking to improve in 2020 (although still figuring out how!)

It’s also fair that in a multi-server federated model, users naturally tend to sign up on the most prominent server(s) (e.g. the matrix.org homeserver in the case of Matrix). In practice, the matrix.org homeserver currently makes up about 35% of the visible Matrix network by active users. It’s also true that Matrix servers currently store metadata about who’s talking to who, and when, as a side-effect of storing and relaying messages on behalf of their users. And without an adequate protocol governance system in place, a large server could start pushing around smaller ones in terms of protocol behaviour. In practice, we’re looking into solving metadata protection in Matrix by experimenting with hybrid P2P / Client Server models - letting users store their metadata purely clientside if they so desire, and potentially obfuscating who’s talking to who via mixnets of blinded store & forward servers (more about this coming up at FOSDEM). Combined with nomadic accounts, this would let us eventually turn off the matrix.org server entirely and eliminate the pseudo-centralisation effect - the default ‘server’ would be the one running on your client.

It’s true that if a user doesn’t trust (say) Telegram, they are free to go switch to Signal or WhatsApp or whatever instead… at the massive expense of having to persuade all their friends to install yet another app, and fragmenting their conversation history across multiple apps.

Finally, it’s also true that because anyone can develop a Matrix client or server and connect to the global network, there’s a risk of bad quality implementations in the wild. There are many forks of Riot on the app stores - we simply can’t vouch for whether they are secure. Similarly there are Matrix clients whose E2E encryption is partial, missing, or unreviewed. And there are a wide range of different Matrix servers run by different people with different agendas in different locations, which may be more or less trustworthy.

HOWEVER: all of this completely ignores one critical thing - the value of freedom. Freedom to select which server to use. Freedom to run your own server (perhaps invisibly in your app, in a P2P world). Freedom to pick which country your server runs in. Freedom to select how much metadata and history to keep. Freedom to choose which apps to use - while still having the freedom to talk to anyone you like (without them necessarily installing yet another app). Freedom to connect your own functionality - bots, bridges, integrations etc. Freedom to select which identifiers (if any) to use to register your account. Freedom to extend the protocol. Freedom to write your own client, or build whole new as-yet-unimagined systems on top.

It’s true that if you’re writing a messaging app optimised for privacy at any cost, Moxie’s approach is one way to do it. However, this ends up being a perversely closed world - a closed network, where unofficial clients are banned, with no platform to build on, no open standards, and you end up thoroughly putting all your eggs in one basket, trusting past, present & future Signal to retain its values, stay up and somehow dodge compromise & censorship… despite probably being the single highest value attack target on the ‘net.

Quite simply, that isn’t a world I want to live in.

We owe the entire success of the Internet (let alone the Web) to openness, interoperability and decentralisation. To declare that openness, interoperability and decentralisation is ‘too hard’ and not worth the effort when building a messaging solution is to throw away all the potential of the vibrancy, creativity and innovation that comes from an open network. Sure, you may end up with a super-private messaging app - but one that starts to smell alarmingly like a walled garden like Facebook’s Internet.org initiative, or an AOL keyword, or Google’s AMP.

So, we continue to gladly take up Moxie’s challenge to prove him wrong - to show that it’s both possible and imperative to create an open decentralised messaging platform which (if you use reputable apps and servers) can be as secure and metadata-protecting as Signal… and indeed more so, given you can run your server off the grid, and don’t need to register with a phone number, and in future may not even need a server at all.

--Matthew

(Comments over at HN)

Hi all,

Over the course of today we've been made aware of folks port-scanning the general internet to discover private Matrix servers, looking for publicly visible room directories, and then trying to join rooms listed in them.

If you are running a Matrix server that is intended to be private, you must correctly configure your server to not expose its public room list to the general public - and also ensure that any sensitive rooms are invite-only (especially if the server is federated with the public Matrix network).

In Synapse, this means ensuring that the following options are set correctly in your homeserver.yaml:

# If set to 'false', requires authentication to access the server's public rooms
# directory through the client API. Defaults to 'true'.
#
#allow_public_rooms_without_auth: false

# If set to 'false', forbids any other homeserver to fetch the server's public
# rooms directory via federation. Defaults to 'true'.
#
#allow_public_rooms_over_federation: false

For private servers, you will almost certainly want to explicitly set these to false, meaning that the server's "public" room directory is hidden from the general internet and wider Matrix network.

You can test whether your room directory is visible to arbitrary Matrix clients on the general internet by viewing a URL like https://sandbox.modular.im/_matrix/client/r0/publicRooms (but for your server). If it gives a "Missing access token" error, you are okay.

You can test whether your room directory is visible to arbitrary Matrix servers on the general internet by loading Riot (or similar) on another server, and entering the target server's domain name into the room directory's server selection box. If you can't see any rooms, then are okay.

Relatedly, please ensure that any sensitive rooms are set to be "invite only" and room history is not world visible - particularly if your server is federated, or if it has public registration enabled. This stops random members of the public peeking into them (let alone joining them).

Relying on security-by-obscurity is a very bad idea: all it takes is for someone to scan the whole internet for Matrix servers, and then trying to join (say) #finance on each discovered domain (either by signing up on that server or by trying to join over federation) to cause problems.

Finally, if you don't want the general public reading your room directory, please also remember to turn off public registration on your homeserver. Otherwise even with the changes above, if randoms can sign up on your server to view & join rooms then all bets are off.

We'll be rethinking the security model of room directories in future (e.g. whether to default them to being only visible to registered users on the local server, or whether to replace per-server directories with per-community directories with finer grained access control, etc) - but until this is sorted, please heed this advice.

If you have concerns about randoms having managed to discover or join rooms which should have been private, please contact [email protected].

Hi all,

Back in June we wrote about our plans to tighten up data privacy in Matrix after some areas for improvement were brought to our attention. To quickly recap: the primary concern was that the default config for Riot specifies identity servers and integration managers run by New Vector (the company which the original Matrix team set up to build Riot and fund Matrix dev) - and so folks using a standalone homeserver may end up using external services without realising it. There were some other legitimate issues raised too (e.g. contact information should be obfuscated when checking if your contacts are on Matrix; Riot defaulted to using Google for STUN (firewall detection) if no TURN server had been set up on their server; Synapse defaults to using matrix.org as a key notary server).

We’ve been working away at this fairly solidly over the last few months. Some of the simpler items shipped quickly (e.g. Riot/Web had a stupid bug where it kept incorrectly loading the integration manager; Riot/Android wasn’t clear enough about when contact discovery was happening; Riot/Web wasn’t clear enough about the fact device names are publicly visible; etc) - but other bits have turned out to be incredibly time-consuming to get right.

However, we’re in the process today of releasing Synapse 1.4.0 and Riot/Web 1.4.0 (it’s coincidence the version numbers have lined up!) which resolve the majority of the remaining issues. The main changes are as follows:

1. Riot no longer automatically uses identity servers by default.
   Identity servers are only useful when inviting users by email address, or when discovering whether your contacts are on Matrix. Therefore, we now wait until the user tries to perform one of these operations before explaining that they need an identity server to do so, and we prompt them to select one if they want to proceed. This makes it abundantly clear that the user is connecting to an independent service, and why.

2. Integration Managers and identity servers now have the ability to force users to accept terms of use before using them.
   This means they can explicitly spell out the data privacy & usage policy of the server as required by GDPR, and it should now be impossible for a user to use these services without realising it. This was particularly fun in the case of identity servers, which previously had no concept of users and so couldn’t track whether users had agreed to their terms & conditions or not… and because homeservers sometimes talk to the identity server on behalf of users rather than the user talking direct, the privacy policy flow gets even hairier. But it’s solved now, and a nice side-effect of this is that users can now explicitly select their Integration Manager in Riot, in case they want to use Dimension or similar rather than the default provided by Modular.

3. Synapse no longer uses identity servers for verifying registrations or verifying password reset.
   Originally, Synapse made use of the fact that the Identity Service contains email/msisdn verification logic to handle identity verification in general on behalf of the homeserver. However, in retrospect this was a mistake: why should the entity running your identity server have the right to verify password resets or registration details on your homeserver? So, we have moved this logic into Synapse. This means Synapse 1.4.0 requires new configuration for email/msisdn verification to work - please see the upgrade notes for full details.

4. Sydent now supports discovering contacts based on hashed identifiers.
   MSC2134 specifies entirely new IS APIs for discovering contacts using a hash of their identifier rather than directly exposing the raw identifiers being searched for. This is implemented in Riot/iOS and Riot/Android and should be in the next major release; Riot/Web 1.4.0 has it already.

5. Synapse now warns in its logs if you are using matrix.org as a default trusted key server, in case you wish to use a different server to help discover other servers’ keys.

6. Synapse now garbage collects redacted messages after N days (7 days by default). (It doesn’t yet garbage collect attachments referenced from redacted messages; we’re still working on that).

7. Synapse now deletes account access data (IP addresses and User Agent) after N days (28 days by default) of a device being deleted.

8. Riot warns before falling back to using STUN (and defaults to turn.matrix.org rather than stun.google.com) for firewall discovery (STUN) when placing VoIP calls, and makes it clear that this is an emergency fallback for misconfigured servers which are missing TURN support. (We originally deleted the fallback entirely, but this broke things for too many people, so we’ve kept it but warn instead).

All of this is implemented in Riot/Web 1.4.0 and Synapse 1.4.0. Riot/Web 1.4.0 shipped today (Fri Sept 27th) and we have a release candidate for Synapse 1.4 (1.4.0rc1) today which fully ship on Monday.

For full details please go check out the Riot 1.4.0 and Synapse 1.4.0 blog posts.

Riot/Mobile is following fast behind - most of the above has been implemented and everything should land in the next release. RiotX/Android doesn’t really have any changes to make given it hadn’t yet implemented Identity Service or Integration Manager APIs.

This has involved a surprisingly large amount of spec work; no fewer than 9 new Matrix Spec Changes (MSC) have been required as part of the project. In particular, this results in a massive update to the Identity Service API, which will be released very shortly with the new MSCs. You can see the upcoming changes on the unstable branch and compare with the previous 0.2.1 stable release, as well as checking the detailed MSCs as follows:

• MSC1961: Integration manager authentication APIs
• MSC2078: Sending Third-Party Request Tokens via the Homeserver
• MSC2134: Identity Hash Lookups
• MSC2140: Terms of Service for ISes and IMs
• MSC2229: Allowing 3PID Owners to Rebind
• MSC2230: Store Identity Server in Account Data
• MSC2263: Give homeservers the ability to handle their own 3PID registrations/password resets
• MSC2264: Add an unstable feature flag to MSC2140 for clients to detect support
• MSC2290: Separate Endpoints for Threepid Binding

This said, there is still some work remaining for us to do here. The main things which haven’t made it into this release are:

• Preferring to get server keys from the source server rather than the notary server by default (https://github.com/matrix-org/synapse/pull/6110). This almost made it in, but we need to test it more first - until then, your specified notary server will see roughly what servers your servers are trying to talk to. In future this will be mitigated properly by MSC1228 (removing mxids from events).
• Configurable data retention periods for rooms. We are tantalisingly close with this - https://github.com/matrix-org/synapse/pull/5815 is an implementation that the French Govt deployment is using; we need to port it into mainline Synapse.
• Authenticating access to the media repository - for now, we still rely on media IDs being almost impossible to guess to protect the data rather than authenticating the user.
• Deleting items from the media repository - we still need to hook up deletion APIs.
• Garbage collecting forgotten rooms. If everyone leaves & forgets a room, we should delete it from the DB.
• Communicating erasure requests over federation

We’ll continue to work on these as part of our ongoing maintenance backlog.

Separately to the data privacy concerns, we’ve had a separate wave of feedback regarding how we handle GDPR Data Subject Access Requests (DSARs). Particularly: whether DSAR responses should contain solely the info your have directly keyed by the requesting Matrix ID - or if we should provide all the data “visible” to that ID (i.e. the history of the conversations they’ve been part of). We went and got professional legal advice on this one, and the conclusion is that we should keep our responses to DSARs as tightly scoped as possible. We updated Matrix.org’s privacy policy and DSAR tools to reflect the new legal input.

Finally, it’s really worth calling out the amount of effort that went into this project. Huge huge thanks to everyone involved (given it’s cut across pretty much every project & subteam we have working on the core of Matrix) who have soldiered through the backlog. We’ve been tracking progress using our feature-dashboard tool which summarises Github issues based on labels & issue lifecycle, and for better or worse it’s ended up being the biggest project board we’ve ever had. You can see the live data here (warning, it takes tens of seconds to spider Github to gather the data) - or, for posterity and ease of reference, I’ve included the current issue list below. The issues which are completed have “done” after them; the ones still in progress say “in progress”, and ones which haven’t started yet have nothing. We split the project into 3 phases - phases 1 and 2 represent the items needed to fully solve the privacy concerns, phase 3 is right now a mix of "nice to have" polish and some more speculative items. At this point we’ve effectively finished phase 1 on Synapse & Riot/Web, and Riot/Mobile is following close behind. We're continuing to work on phase 2, and we’ll work through phase 3 (where appropriate) as part of our general maintenance backlog.

I hope this gives suitable visibility on how we’re considering privacy; after all, Matrix is useless as an open communication protocol if the openness comes at the expense of user privacy. We’ll give another update once the remaining straggling issues are closed out; and meanwhile, now the bulk of the privacy work is out of the way on Riot/Web, we can finally get back to implementing the UI E2E cross-signing verification and improving first time user experience.

Thanks for your patience and understanding while we’ve sorted this stuff out; and thanks once again for flying Matrix :)

In the absence of comments on the current blog, please feel free to discuss over at HN, or alternatively come ask stuff in our AMA over at /r/privacy (starting ~5pm GMT+1 (UK) on Friday Sept 27th).

🔗The Privacy Project Dashboard Of Doom

• phase:1
  • vector-im/riot-web
    • 5846 Riot duplicates calls to Scalar (done)
    • 5921 Ability to disable all identity server functionality via the config file (done)
    • 6802 riot shouldn't try to load integrations from an integration manager unless the user has accepted that manager's privacy policies (done)
    • 10088 Prompt to accept integration manager polices on use (done)
    • 10090 Make sure there are no ugly edge cases running Riot without an integrations manager (done)
    • 10091 Decouple setting an email for password reset from publishing your threepid to the identity server and support choice of identity server (on registration) (done)
    • 10092 Prompt to accept Identity Server policies on registration (done)
    • 10093 Prompt to accept identity server policies before inviting them to a room (done)
    • 10094 Store identity server in Account Data and support choosing identity server integration in User Settings (done)
    • 10159 Enable toggling a threepid's association with the current IS in User Settings (done)
    • 10173 VoIP: Stop falling back to Google for STUN (done)
    • 10216 We should make it clear in the UI that device names are publicly readable (done)
    • 10386 Terms modal prompt should appear without a flash of the integration manager portal first (done)
    • 10408 Identity server v2 API authentication (done)
    • 10423 Deploy a develop environment for t's and c's flows for IM and IS (done)
    • 10424 Remove the bind true flag from 3PID calls on registration (done)
    • 10425 Ability to disconnect from the identity server by pressing buttons in user settings. (done)
    • 10452 Test IS v2 access tokens for validity (done)
    • 10497 Integration manager terms hides terms that need signing (done)
    • 10510 Remove the bind true flag from 3PID adds in settings (done)
    • 10519 Update discovery and account 3PID sections when either changes (done)
    • 10522 Handle terms agreement in the Discovery section of settings (done)
    • 10525 IS interactions proxied through the HS also need terms agreement (done)
    • 10528 don't try & show threepids in discovery section if we don't have an ID server (done)
    • 10539 Inline terms agreement on change IS and IM (done)
    • 10540 When using Riot without an IS, identity requests are sent to server hosting Riot (done)
    • 10546 Prompt for fallback VOIP should only happen when the user tries to make a call (done)
    • 10550 warn when disconnecting ID server if there are any current bindings (done)
    • 10552 Allow email registration, password reset & adding 3pids when no IS (done)
    • 10553 Remove the ability to set an IS at login/registration (done)
    • 10556 Use the hashed v2 lookup API for 3PIDs (done)
    • 10561 Scalar should serve a .well-known file (done)
    • 10566 Use nicer APIs for toggling a 3PID's shared status (done)
    • 10571 Allow email registration when no IS (done)
    • 10572 Allow password reset when no IS (done)
    • 10573 Allow adding 3pids when no IS (done)
    • 10574 Placeholder issue for MSCs that should land prior to the privacy release (done)
    • 10593 Placeholder issue for privacy migration path (done)
    • 10597 Ensure IS in account data is read / written as specced (done)
    • 10599 Send IS for adding MSISDNs only when required by HS (done)
    • 10603 Getting a terms prompt when inviting an email address clears the address picker (done)
    • 10619 Handle the case of no IS in features that require IS to lookup (done)
    • 10620 Change auth flows so that you default to no IS (done)
    • 10634 Changing to vector.im IS in Settings fails because /terms 404s (done)
    • 10635 Inline terms agreement in Discovery should still show change and do not use (done)
    • 10636 The UX is a little confusing if you haven't accepted the terms of the default identity server. (done)
    • 10637 Unable to add email address to HS account without IS set (done)
    • 10660 Sydent implementation: Unauthed v1, clone all endpoints, auth v2 (done)
    • 10666 In login/register, riot freezes if you go to Advanced, remove the IS url and click next (done)
    • 10669 Checking email invite account is unclear when no IS set (done)
    • 10674 Email help text on registration should be updated without binding (done)
    • 10677 Change text about other users being able to invite you by email on registration page (done)
    • 10744 Only set terms URLs in the account when they change (done)
    • 10749 Changing from one IS to another does not trigger bound 3PID warning (done)
    • 10750 Bound 3PIDs warning should better explain what's being left behind (done)
    • 10754 Lowercase emails during IS lookup calls (done)
    • 10755 Terms account data is meant to be additive, but currently sets only new URLs (done)
    • 10756 After changing IS to termstest.matrix.org the text field isn't cleared and the button remains active (done)
    • 10757 After changing IS back to vector.im the text field isn't cleared and the button remains active (done)
    • 10758 Revoke may be too vague for unbinding a 3PID (done)
    • 10779 When publishing a threepid to an IS, if you click 'continue' before clicking the link in your email, we delete your threepid from the HS. (done)
    • 10800 Community invite should not talk about ISes at all. (done)
    • 10823 IS with unagreed terms will block you from adding even unshared 3PIDs to your HS (done)
    • 10839 Use the new backend APIs for adding-3pid-to-homeserver and binding-3pid-on-identity-server when they exist. (done)
    • 10878 UX regression when trying to invite by email if no IS is configured (done)
    • 10883 Riot should check for r0.6.0 server support alongside feature flags (done)
    • 10923 Send validation tokens to submit_url from HS (done)
    • 10933 Do not include ID server or access token when HS supports MSC2290 (done)
    • 10939 Registration with MSISDN needs to use submit_url (done)
    • 10941 threepid_creds for registration and password reset still include id_server (done)
    • 10959 Remove id_server from email threepid_creds (done)
    • 10604 QA: Re-test ALL of the identity server interactions (in progress)
    • 10958 Terms of Service nitpicks (in progress)
    • 10704 We treat an empty string identity server as not having one
  • vector-im/riot-ios
    • 2518 Make sure there are no ugly edge cases running Riot without an integrations manager (done)
    • 2521 Privacy: Improve wording on contact upload UI (done)
    • 2532 VoIP: Stop falling back to Google for STUN (done)
    • 2600 Prompt to accept integration manager polices on use (done)
    • 2601 Decouple setting an email for password reset from publishing your threepid to the identity server and support choice of identity server (on registration) (done)
    • 2602 Prompt to accept identity server policies before inviting them to a room (done)
    • 2603 Identity server v2 API authentication (done)
    • 2606 Settings: Add a Discovery section (done)
    • 2608 Registration: Update consents flow (done)
    • 2643 Ability to disable all identity server functionality via the config file (done)
    • 2646 VoIP: Fallback to matrix.org STUN server with a confirmation dialog (done)
    • 2647 MXRestClient: Move IS API to its own (done)
    • 2648 Remove the bind true flag from 3PID calls on registration (done)
    • 2650 Remove the bind true flag from 3PID adds in settings (done)
    • 2651 Store identity server in Account Data and support choosing identity server integration in User Settings (done)
    • 2652 Use the hashed v2 lookup API for 3PIDs (done)
    • 2653 Allow email registration, password reset & adding 3pids when no IS (done)
    • 2654 Use nicer APIs for toggling a 3PID's shared status (done)
    • 2657 Allow email registration when no IS (done)
    • 2658 Allow password reset when no IS (done)
    • 2659 Allow adding 3pids when no IS (done)
    • 2660 Handle terms agreement in the Discovery section of settings (done)
    • 2661 Remove the ability to set an IS at login/registration (done)
    • 2662 We should make it clear in the UI that device names are publicly readable (done)
    • 2664 Placeholder issue for privacy migration path (done)
    • 2665 Ensure IS in account data is read / written as specced (done)
    • 2672 Handle the case of no IS in features that require IS to lookup (done)
    • 2675 Email help text on registration should be updated without binding (done)
    • 2682 Send IS for adding MSISDNs only when required by HS (done)
    • 2686 Use wellknown to discover the IS of a custom HS (done)
    • 2696 Lowercase emails during IS lookup calls (done)
    • 2704 Use id_access_token in CS API when required (done)
    • 2604 Settings: Ability to change the identity server (in progress)
    • 2649 Ability to disconnect from the identity server by pressing buttons in user settings (in progress)
    • 2713 Use the new backend APIs for adding-3pid-to-homeserver and binding-3pid-on-identity-server when they exist. (in progress)
    • 2718 Riot should check for r0.6.0 server support alongside feature flags (in progress)
    • 2683 Checking email invite account is unclear when no IS set
    • 2731 Send validation tokens to submit_url from HS
    • 2732 Do not include ID server or access token when HS supports MSC2290
    • 2736 IS terms cannot be accepted anymore
  • vector-im/riot-android
    • 3223 VoIP: Stop falling back to Google for STUN (done)
    • 3224 Make sure there are no ugly edge cases running Riot without an integrations manager (done)
    • 3225 Prompt to accept integration manager polices on use(done)
    • 3226 Decouple setting an email for password reset from publishing your threepid to the identity server and support choice of identity server (on registration) (done)
    • 3227 Prompt to accept identity server policies before inviting them to a room (done)
    • 3228 Identity server v2 API authentication (done)
    • 3229 Settings: Ability to change the identity server (done)
    • 3230 Settings: Add a Discovery section (done)
    • 3231 Registration: Update consents flow (done)
    • 3252 Remove the bind true flag from 3PID calls on registration (done)
    • 3253 Ability to disconnect from the identity server by pressing buttons in user settings (done)
    • 3254 Remove the bind true flag from 3PID adds in settings (done)
    • 3256 Store identity server in Account Data and support choosing identity server integration in User Settings (done)
    • 3257 Use the hashed v2 lookup API for 3PIDs (done)
    • 3258 Allow email registration, password reset & adding 3pids when no IS (done)
    • 3260 Allow email registration when no IS (done)
    • 3261 Allow password reset when no IS (done)
    • 3262 Allow adding 3pids when no IS (done)
    • 3263 Handle terms agreement in the Discovery section of settings (done)
    • 3264 Remove the ability to set an IS at login/registration (done)
    • 3265 We should make it clear in the UI that device names are publicly readable (done)
    • 3268 Placeholder issue for privacy migration path (done)
    • 3269 Ensure IS in account data is read / written as specced (done)
    • 3277 Handle the case of no IS in features that require IS to lookup (done)
    • 3278 Email help text on registration should be updated without binding (done)
    • 3281 Send IS for adding MSISDNs only when required by HS (done)
    • 3283 Use wellknown to discover the IS of a custom HS (done)
    • 3289 Lowercase emails during IS lookup calls (done)
    • 3295 Use id_access_token in CS API when required (done)
    • 3300 Use the new backend APIs for adding-3pid-to-homeserver and binding-3pid-on-identity-server when they exist.(done)
    • 3312 Riot should check for r0.6.0 server support alongside feature flags (done)
    • 3316 Implement MSC2290 (done)
    • 3324 id_server param sent when registering an email on server that does not requires one (done)
    • 3326 Discovery Settings / infinite loading if terms not signed on existing IS (done)
    • 3327 vector.im is seen as having no terms, but it does (done)
    • 3251 VoIP: Fallback to matrix.org STUN server with a confirmation dialog (in progress)
    • 3248 Ability to disable all identity server functionality via the config file
    • 3318 Send validation tokens to submit_url from HS
    • 3322 Do not include ID server or access token when HS supports MSC2290
  • vector-im/riotX-android
    • 490 Allow email registration, password reset & adding 3pids when no IS (done)
    • 432 Prompt to accept integration manager polices on use (in progress)
    • 431 Make sure there are no ugly edge cases running Riot without an integrations manager
    • 433 Decouple setting an email for password reset from publishing your threepid to the identity server and support choice of identity server (on registration)
    • 434 Prompt to accept identity server policies before inviting them to a room
    • 435 Identity server v2 API authentication
    • 436 Settings: Ability to change the identity server
    • 438 Settings: Add a Discovery section
    • 439 Registration: Update consents flow
    • 489 Use the hashed v2 lookup API for 3PIDs
  • matrix-org/matrix-doc
    • 1961 MSC1961: Integration manager authentication APIs (done)
    • 2078 MSC2078: Sending Third-Party Request Tokens via the Homeserver (done)
    • 2130 Identity service should do lookups based on hashed 3PIDs, not plaintext ones. (done)
    • 2134 MSC2134: Identity Hash Lookups (done)
    • 2139 Method for ISes and Integration managers to provide terms of service (done)
    • 2140 MSC2140: Terms of Service for ISes and IMs (done)
    • 2229 MSC2229: Allowing 3PID Owners to Rebind (done)
    • 2230 MSC2230: Store Identity Server in Account Data (done)
    • 2263 MSC2263: Give homeservers the ability to handle their own 3PID registrations/password resets (done)
    • 2264 MSC2264: Add an unstable feature flag to MSC2140 for clients to detect support (done)
    • 2290 MSC2290: Separate Endpoints for Threepid Binding (in progress)
  • matrix-org/sydent
    • 178 Support for MSC2140 (done)
    • 179 Support testing with matrix-is-tester (done)
    • 180 Support for MSC2140 (done)
    • 181 Deny bindings to anything other than the mxid you're authed as (done)
    • 184 Hash 3PID lookups (done)
    • 186 Don't fail the unbind request if the binding doesn't exist (done)
    • 187 Add more tweaks to terms branch (done)
    • 189 Redact looked-up threepids from the application logs. (done)
    • 191 Add OpenID auth to 3PID hash lookup (done)
    • 197 Have 'mappings' contain medium as well as address (done)
    • 198 Throw exception on bad args & catch in wrapper (done)
    • 199 Return algorithm, lookup_pepper in M_INVALID_PEPPER response (done)
    • 200 Support MSC2140 register/terms endpoints (done)
    • 201 Port v1 endpoints to v2 (done)
    • 202 Deny bindings to anything other than the mxid you're authed as (done)
    • 204 Fix the signing servlet (done)
    • 205 Correct API v1 path check in get_args (done)
    • 206 Fix terms POST (done)
    • 207 Deploy policy file (done)
    • 208 Remove PII logging from log-level INFO (done)
    • 213 Delete threepid assocs on unbind and remove existing with migration (done)
    • 217 Allow non-matrix.org MXIDs via v2 only once terms are enabled (done)
    • 218 Unbind fails to remove the binding on matrix.org IS (done)
    • 220 Unbind fails to work for bindings where it was attempted with #213 (done)
    • 192 Sydent doesn't delete 3PIDs from DB on unbind (in progress)
  • matrix-org/synapse
    • 1287 We need to actually censor redactions from the DB (SYN-284) (done)
    • 5751 Make homeservers able to handle registration-with-email without depending on an Identity Server (done)
    • 5786 Support IS v2 API with authentication for requests proxied to the IS (done)
    • 5827 Add 3PID unbind API for MSC2140 (done)
    • 5835 [1/2] Allow homeservers to send registration emails | Sending the email (done)
    • 5861 Use the v2 lookup API for 3PID invites (done)
    • 5862 Allow account owners to rebind 3PIDs as in MSC2229 (done)
    • 5868 Add flag in /versions for whether clients should send id_server params (done)
    • 5874 Placeholder issue for privacy migration path (done)
    • 5875 Remove trusted_third_party_id_servers functionality (done)
    • 5876 Replace trust_identity_server_for_password_resets with account_threepid_delegate (done)
    • 5892 Switch to using v2 Identity Service APIs other than lookup (MSC 2140) (done)
    • 5897 Use the v2 lookup API for 3PID invites (done)
    • 5927 Add a m.id_access_token flag to unstable_features (done)
    • 5928 Split account_threepid_handler into a msisdn and email versions (done)
    • 5929 Use account_threepid_delegate for sending SMS (done)
    • 5930 Add m.id_access_token flag (done)
    • 5934 Censor redactions in DB after a month (done)
    • 5935 Use federation blacklist for requests to identity servers (done)
    • 5937 Revert "Use the v2 lookup API for 3PID invites" (done)
    • 5940 [2/2] Allow homeservers to send registration emails | Accepting the verification (done)
    • 5945 Revert "Add m.id_access_token flag" (done)
    • 5948 Support v2 Identity Server APIs (done)
    • 5964 Remove bind_email and bind_msisdn (done)
    • 5968 Set m.require_identity_server to always be False (done)
    • 5969 Change account_threepid_delegate to a dictionary (done)
    • 5972 Add m.require_identity_server to /versions unstable_flags (done)
    • 5974 Add m.id_access_token to /versions unstable_features (MSC2264) (done)
    • 5976 Use the v2 Identity Service API for lookups (MSC2134 + MSC2140) (done)
    • 5979 v2 3PID Invites (part of MSC2140) (done)
    • 5980 Add POST /_matrix/client/r0/account/3pid/unbind (MSC2140) (done)
    • 5987 Allow Synapse to send registration emails + choose Synapse or an external server to handle 3pid validation (done)
    • 5996 Allow users to rebind 3pids they own (MSC2229) (done)
    • 6000 Use the federation blacklist for requests to untrusted Identity Servers (done)
    • 6011 Use account_threepid_delegate for 3pid validation (done)
    • 6013 Fix existing v2 identity server calls (MSC2140) (done)
    • 6042 Allow HS to send emails when adding an email to the HS (done)
    • 6043 Implement MSC2290 (done)
    • 6044 Add an unstable feature flag for separate add/bind 3pid APIs (done)
    • 6062 Use unstable prefix for 3PID unbind API (done)
    • 6067 Drop support for bind param on POST /account/3pid (MSC2290) (done)
    • 6076 Synapse doesn't give clients a submit_url on requestToken breaking msisdn adding (done)
    • 6078 Add POST submit_token endpoint for MSISDN (done)
    • 6079 Add submit_url response parameter to msisdn /requestToken (done)
    • 6087 Remove hardcoded defaults of matrix.org and vector.im in configuration (done)
    • 6090 Explicitly log when a homeserver does not have a trusted key server configured(done)
    • 6096 Riot Web expects the email validation next_link to have the sid appended (done)
    • 6100 Remove email from registration flows if it’s unsupported (done)
    • 6103 _check_threepid in auth.py incorrect for MSISDN (done)
    • 6045 Support Client-Server API r0.6.0 (in progress)
    • 1263 When we redact events, any mxc content they refer to should be redacted too (SYN-216)
    • 4565 Metadata resistance.
    • 6086 Fetch signing-keys directly from servers before falling back to the trusted_key_servers
• phase:2
  • vector-im/riot-web
    • 4913 Option for homeservers to set a default integration manager (done)
    • 10161 Store Integration Manager preferences in account data and allow user to change them somewhere sensible (done)
    • 10967 Disconnect from identity server fails (done)
    • 10443 Remove v1 IS API fallbacks once servers are updated
    • 10557 Prompt users each time before sending data to an Identity Server that doesn't have a terms of service (unless you have actively set that IS in your account data).
    • 10696 Allow users to disconnect from an integration manager entirely in the same way that we support doing this for identity servers
    • 10909 Unable to disconnect from IS if the server is unavailable
    • 10936 Refine submit_url handling to only activate with separate add and bind
  • vector-im/riot-ios
    • 2711 Provide a better UX for users considering sharing their contact list with an IS to discover people they know already using Matrix
  • vector-im/riot-android
    • 3282 Checking email invite account is unclear when no IS set
    • 3323 Post MSC2290 bug / Clicking on the validation mail link fails registration
  • matrix-org/matrix-doc
    • 1957 MSC1957: Integration manager discovery (done)
  • matrix-org/sydent
    • 196 Remove the lookup_hash field from local_threepid_associations (in progress)
  • matrix-org/synapse
    • 5881 Remove trust_identity_server_for_password_resets and account_threepid_delegate options
• phase:3
  • vector-im/riot-web
    • 10563 Add better domain for turn.matrix.org use as STUN server (done)
    • 10087 Prompt to accept integration manager policies on registration
    • 10167 Present an aggregated terms of service dialogue at registration if possible
    • 10168 If a user has disabled the identity server integration on their account, we should make invite user handle this nicely
    • 10401 Invite new users to publish their threepids to the identity server
    • 10422 Use more contextual prompt for integration manager polices on use
    • 10453 Log out from IM on Riot log out and IM removal
    • 10455 Log out from IS on Riot log out and IS removal
    • 10487 Store the date of users having read and accepted the IM policies in the IM db
    • 10488 Store the date of users having read and accepted the IS policies in the IS db
    • 10498 Terms test scalar requires the legacy ?v query param on the new account route
    • 10575 We should show the 'must configure TURN' warning when a call fails, even after using the fallback turn.matrix.org
    • 10615 Change all IS access token APIs to use getIdentityAccessToken only
    • 10671 riot submits sign-ed25519 requests as POST requests with params in query string and empty body
    • 10746 /invite doesn't force terms with older homeservers
    • 10830 Show IS policy links in user settings somewhere.
    • 10950 Unhelpful 400 error when adding MSISDN and server doesn’t delegate
  • vector-im/riot-ios
    • 2710 Show IS policy links in user settings somewhere.
  • matrix-org/matrix-doc
    • 447 We need a way to be able to expire data from a HS. (SPEC-141)
  • matrix-org/synapse
    • 5830 pushers table contains user device names, which may include user real names

It was drawn to our attention this afternoon that there is a bug in our GDPR data portability tooling that resulted in the data dump including some events that should not have been included.

This tooling has recently been updated (here is the new code), and the bug only affects reports generated with the updated tool. So far we have generated one report using the updated tooling.

The bug affects events which:

• were sent in rooms in which, at the point at which the message was sent, the message visibility was set to 'shared' or 'world readable', and
• were pulled in over federation from another server after the data subject left the room

As a reminder, 'shared' message visibility means anyone in the room can view the message, from the point in time at which visibility was set to 'shared' and 'world readable' means anyone can read the messages without joining the room, from the point in time at which visibility was set to 'world readable'.

Events are pulled onto a homeserver over federation when a user on that homeserver tries to access events which, for whatever reason, their homeserver does not already have a local copy. This most often happens when their homeserver is offline for any period of time, but can also happen when a user is the first user from their homeserver to join a room with active participants on other homeservers.

We're still analysing the data but so far it looks like the bug resulted in only a small number of events that were not publicly-accessible being shared (there were also publicly-accessible events mistakenly included). At this stage we have identified 19 events from 4 users across 2 rooms (the dump contained ~3.5 million events). This is not to diminish the severity of the bug - just to reassure that the scale of its impact appears to be extremely limited.

It is also worth noting that any encrypted events erroneously included in the dump will not have been decryptable (since the data subject would not have had access to the keys).

🔗Update (2019-08-06)

In our original analysis we stated that 19 events were shared erroneously. On closer analysis we missed 5 other timeline events - the correct figure is 24 timeline events originating from 4 users over 2 rooms. However, this figure focused on timeline data and does not take into account all state events (such as user joins, parts, topic changes etc). When considering these too, a further 56 state events were erroneously shared, referencing 64 users across these 2 rooms (mainly detailing when users had joined/left the room after the requesting user themselves had left). These membership events contained avatar & display name details which may not have been public (but in practice, the vast majority appear to be public data).

Aside from the events referenced above, the full dump contained ~20,000 events that also ought not to have been included; however these events were already publicly accessible due to being part of publicly accessible rooms (eg Matrix HQ) and so we do not consider them a breach of data.

🔗What caused the bug?

Events that are pulled in over federation are assigned a negative 'stream ordering' ID. This is designed to avoid their being sent down the sync (where they would likely be out of sequence). In normal operation (accessing your homeserver via a Matrix client) these events would be appropriately filtered, but a bug in the data dump tooling caused them to be included.

The bug was introduced as a result of two factors:

• The event filtering code assumes that the user is currently in the room - this was not intuitive, and was not called out in the documentation
• When we fetched the events from the database, we tried to limit to events sent before the user left the room. On reflection, we used the wrong ordering mechanism (stream ordering instead of topological ordering), resulting in the inclusion of events that were fetched from a remote server after the data subject had left

We are working to fix the bug, and we'll update here when it is resolved. As a reminder, please do report security bugs responsibly as per the Security Disclosure Policy so we can validate the issue and mitigate abuse.

As is standard practice for any data breach, we have notified the ICO.

As a step towards implementing Terms of Service for Sydent Identity Servers (MSC2140), we're rolling out a couple of changes to the two Identity Servers run by New Vector (running at vector.im and matrix.org):

1. We have erased all of the data where there is any chance that the data subject didn't understand how, why or with whom their data was being shared.
2. We've made a change to Sydent so that it no longer persists new associations relating to users on homeservers not run by New Vector.

The impact of these changes is that users on homeservers not run by New Vector will no longer be discoverable by their email or telephone number via the Identity Servers running at vector.im and matrix.org. As we roll out the rest of the changes for Terms of Service for Identity Servers, this functionality will again be made available for users who make an informed choice to opt in.

🔗Registration with Email and Password Reset

In the short term, the New Vector Identity Servers will continue to support registration with email (signing up with an email address as well as a matrix username) and password reset. However, as we continue to improve Identity Server data hygiene practices, we will phase out their use in registration with email and password reset entirely. We have already made the change to Synapse to support password reset without relying on an Identity Server (though this can optionally be re-enabled).

Once Synapse can support registration with email without relying on an Identity Server we will announce a schedule for disabling registration with email and password reset in our Identity Servers entirely. After this point, homeserver administrators will have to make sure their homeservers are configured to send email to keep registration with email and password reset working. More details on this to follow - please watch this space.

Hi all,

A few weeks ago there was some discussion around the privacy of typical Matrix configurations, particularly how Riot's default config uses vector.im as an Identity Server (for discovering users on Matrix by their email address or phone number) and scalar.vector.im as an Integration Manager (i.e. the mechanism for adding hosted bots/bridges/widgets into rooms). This means that Riot, even if using a custom homeserver and running from a custom Riot deployment, will try to talk to *.vector.im (run by New Vector; the company formed by the core Matrix team in 2017) for some operations unless an alternative IS or IM has been specified in the config.

We haven't done as good a job at explaining this as we could have, and this blog post is a progress update on how we're fixing that and improving other privacy considerations in general.

Firstly, the reason Riot is configured like this is for the user's convenience: in general, we believe most users just want to discover other people on Matrix as easily as possible, and a logically-centralised server for looking up user matrix IDs by email/phone number (called third party IDs, or 3PIDs) is the only comprehensive way of doing so. Decentralising this data while protecting the privacy of the 3PIDs and their matrix IDs is a Hard Problem which we're unaware of anyone having solved yet. Alternatively, you could run a local identity server, but it will end up having to delegate to a centralised identity server anyway for IDs it has no other way to know about. Similarly, providing a default integration server that just works out of the box (rather than mandating the user configures their own) is a matter of trying to keep Riot's UX simple, especially when onboarding users, and especially given Riot's reputation for complexity at the best of times.

That said, the discussion highlighted some areas for improvement. Specifically:

1. When doing work on making Matrix GDPR compliant back in May 2018, we set up a single privacy policy for the services we run, and got users to agree to it by locking them out of the matrix.org homeserver until they did. However, we missed that users not on the matrix.org homeserver might still be using our Identity Service (IS) & Integration Manager (IM) without accepting the privacy policy. Over the last few weeks we've been working on addressing this - it turns out that it's a pain to fix, given the Identity Service doesn't have the concept of users, so tracking which users have agreed to the policy policy or not means some fairly major changes. The current proposal is to change the Identity Service to use a form of OpenID to authenticate users against their homeserver in order to check they've accepted the IS's terms of use - see MSC2140 for the gory details.

Meanwhile, Riot is being updated to prompt the user to accept the IS & IM terms of use (if different to the HS's), and thus make it crystal clear to the user that they are using an IS & IM and that they have the option not to if desired - see https://github.com/vector-im/riot-web/issues/10167 and associated issues. This includes also explicitly prompting the user as to whether they want 3PIDs they provide at registration to be discoverable, as per https://github.com/vector-im/riot-web/issues/10091.

2. Riot on iOS & Android gives the option of scanning your local addressbook to discover which of your contacts are on Matrix. The wording explaining this wasn't clear enough on Android - which we promptly fixed. Separately, the contact details sent to the server are currently not obfuscated. This is partially because we hadn't got to it, and partially because obfuscating them doesn't actually help much with privacy, given an attacker can just scan through possible obfuscated phone numbers and email addresses to deobfuscate them. However, we've been working through obfuscating the contact details anyway by hashing as per MSC2134, which has all the details. We're also adding an explicit lookup warning in Riot/Web, as per https://github.com/vector-im/riot-web/issues/10093.

3. There was a bug where Riot/Web was querying the Integration Manager every time you opened a room, even if that room had no integrations (actually, it did it 3 times in a row). This got fixed and released in Riot/Web 1.2.2 back on June 19th.

4. Matrix needs to authenticate whether events were actually sent by the server that claimed to send them. We do this by having servers sign their events when they create them, and publishing the public half of their signing keys for anyone to query. However, this poses problems if you receive an event which is signed by a server which isn't currently online. To solve this, we have the concept of trusted_key_servers (aka notary servers), which your server can query to see if they know about the missing server's keys. By default, matrix.org is configured as Synapse's trusted notary, but you can of course change this. If you choose an unreliable server as the notary (e.g. by not setting one at all) then there's a risk that you won't be able to look up signing keys, and a splitbrain will result where your server can't receive certain events, but other servers in the room can. This can then result in your server being unable to participate in the room entirely, if it's missing key events in the room's lifetime.

   Our plan here is to get rid of notaries entirely by changing how event signing works as per MSC1228, but this is going to take a while. Meanwhile we're going to check Synapse's code to ensure it doesn't talk to the notary server unnecessarily. (E.g. it should be caching the signing keys locally, and it should only use the notary server if the remote server is down.)

5. When doing VoIP in Matrix, clients need to use a TURN server to discover their network conditions and perform firewall traversal. The TURN server should be specified by your homeserver (and each homeserver deployment should ideally include a TURN server). However, for users who have not configured a TURN server, Riot (on all 3 platforms) defaulted to use Google's public STUN service (stun.l.google.com). STUN is a subset of TURN which provides firewall discovery, but not traffic relaying. This slightly increased the chances of calls working for users without a proper TURN server, but not by much - and rather than fall back to Google, we've decided to simply remove it from Riot (e.g. https://github.com/matrix-org/matrix-ios-sdk/commit/24832a2b14fb72ae6f051d5aba40262d11eef65d). This means that VoIP might get less reliable for users who were relying on this fallback, but you really should be running your own TURN server anyway if you want VoIP to work reliably on your homeserver.

6. We should make it clearer in Riot that device names are world-readable, and not just for the user's own personal reference. This is https://github.com/vector-im/riot-web/issues/10216

As you can see, much of the work on improving these issues is still in full swing, although some has already shipped. As should also be obvious, these issues are categorically not malicious: Matrix (and Riot) literally exists to give users full control and autonomy over their communication, and privacy is a key part of that. These are avoidable issues which can and will be solved. It's worth noting that we have to prioritise privacy issues alongside all the other development in Matrix however: there's no point in having excellent privacy if there are other bugs stopping the platform from being usable.

We'll do another blog post to confirm once most of the fixes here have landed - meanwhile, hopefully this post provides some useful visibility on how we're going about improving things.

Hi all,

As mentioned in our last blog post on GDPR, to make sure that everyone has read and understood the important details about how their personal data is processed by the matrix.org homeserver, users who haven't yet agreed to the privacy notice and terms and conditions will be blocked from sending new messages until they have.

Users will continue to be able to receive messages, so they won't miss out on any messages sent to them before they've agreed to the terms.

The System Alerts room has already sent every user their unique link to review and agree, and if anyone missed that message, the latest Riot.im web and mobile will display a helpful error message guiding users who are yet to agree through the agreement process.

If you have any questions or difficulties, please let us know at [email protected].

Thanks!

Tom

If you've connected to the matrix.org homeserver today, you'll have noticed some activity in support of GDPR compliance. The most obvious of these is an invite from System Alerts (aka @server:matrix.org):

We've rolled out the System Alerts feature to communicate important platform information to all of a homeserver's users. Today, we're using it to communicate the arrival of our new (and much-improved) Privacy Notice and Terms and Conditions to users on matrix.org.

The System Alerts service takes the form of an (unrejectable) invite to a room. We took this approach to support maximum compatibility with the myriad Matrix clients (since all Matrix clients can support conversations in a room ?).

When we first rolled out System Alerts, we didn't allow users leave the System Alerts room. Sorry! We got a bit overexcited - we've fixed that now (though please do provide your agreement before you leave).

🔗What do I need to do?

At some point today the System Alerts service will provide you with unique link, directing you to review the new terms and provide your agreement.

For us to process your personal data lawfully, it's really important that we know you understand and agree to our Privacy Notice and Terms and Conditions. For that reason, we will shortly be blocking any users who haven't indicated their acceptance, so please act quickly when you receive your link.

Once the block is enabled, users who haven't accepted the terms will see an error when they try and send a message, join a room, or send an invite. This message will also include the unique link to review and accept the terms, so users who haven't seen the message from System Alerts will know what to do.

Don't worry if you're reading this some time after May 25 - accepting the terms at any time will unblock message sending on your account, and you won't have missed any messages sent to you.

If you have any thoughts or suggestions on the legal documentation, you can provide comment via github.

Hi all,

As the May 25th deadline looms, we've had lots and lots of questions about how GDPR (the EU's new General Data Protection Regulation legislation) applies to Matrix and to folks running Matrix servers - and so we've written this blog post to try to spell out what we're doing as part of maintaining the Matrix.org server (and bridges and hosted integrations etc), in case it helps folks running their own servers.

The main controversial point is how to handle Article 17 of the GDPR: 'Right to Erasure' (aka Right to be Forgotten). The question is particularly interesting for Matrix, because as a relatively new protocol with somewhat distinctive semantics it's not always clear how the rules apply - and there's no case law to seek inspiration from.

The key question boils down to whether Matrix should be considered more like email (where people would be horrified if senders could erase their messages from your mail spool), or should it be considered more like Facebook (where people would be horrified if their posts were visible anywhere after they avail themselves of their right to erasure).

Solving this requires making a judgement call, which we've approached from two directions: firstly, considering what the spirit of the GDPR is actually trying to achieve (in terms of empowering users to control their data and have the right to be forgotten if they regret saying something in a public setting) - and secondly, considering the concrete legal obligations which exist.  

The conclusion we've ended up with is to (obviously) prioritise that Matrix can support all the core concrete legal obligations that GDPR imposes on it - whilst also having a detailed plan for the full 'spirit of the GDPR' where the legal obligations are ambiguous.  The idea is to get as much of the longer term plan into place as soon as possible, but ensure that the core stuff is in place for May 25th.

Please note that we are still talking to GDPR lawyers, and we'd also very much appreciate feedback from the wider Matrix community - i.e. this plan is very much subject to change.  We're sharing it now to ensure everyone sees where our understanding stands today.

The current todo list breaks down into the following categories. Most of these issues have matching github IDs, which we'll track in a progress dashboard.

🔗Right to Erasure

We're opting to follow the email model, where the act of sending an event (i.e. message) into a room shares a copy of that message to everyone who is currently in that room.  This means that in the privacy policy (see Consent below) users will have to consent to agreeing that a copy of their messages will be transferred to whoever they are addressing.  This is also the model followed by IM systems such as WhatsApp, Twitter DMs or (almost) Facebook Messenger.

This means that if a user invokes their right to erasure, we will need to ensure that their events will only ever be visible to users who already have a copy - and must never be served to new users or the general public. Meanwhile, data which is no longer accessible by any user must of course be deleted entirely.

In the email analogy: this is like saying that you cannot erase emails that you have sent other people; you cannot try to rewrite history as witnessed by others... but you can erase your emails from a public mail archive or search engine and stop them from being visible to anyone else.

It is important to note that GDPR Erasure is completely separate from the existing Matrix functionality of "redactions" which let users remove events from the room. A "redaction" today represents a request for the human-facing details of an event (message, join/leave, avatar change etc) to be removed.  Technically, there is no way to enforce a redaction over federation, but there is a "gentlemen's agreement" that this request will be honoured.

The alternative to the 'email-analogue' approach would have been to facilitate users' automatically applying the existing redact function to all of the events they have ever submitted to a public room. The problem here is that defining a 'public room' is subtle, especially to uninformed users: for instance, if a message was sent in a private room (and so didn't get erased), what happens if that room is later made public? Conversely, if right-to-erasure removed messages from all rooms, it will end up destroying the history integrity of 1:1 conversations, which pretty much everyone agrees is abhorrent. Hence our conclusion to protect erased users from being visible to the general public (or anyone who comes snooping around after the fact) - but preserving their history from the perspective of the people they were talking to at the time.

In practice, our core to-do list for Right to Erasure is:

• As a first cut, provide Article 17 right-to-erasure at a per-account granularity. The simplest UX for this will be an option when calling the account deactivation API to request erasure as well as deactivation. There will be a 30 day grace period, and (ideally) a 2FA confirmation (if available) to avoid the feature being abused.
• Homeservers must delete events that nobody has access to any more (i.e. if all the users in a room have GDPR-erased themselves). If users have deactivated their accounts without GDPR-erasure, then the data will persist in case they reactivate in future.
• Homeservers must delete media that nobody has access to any more. This is hard, as media is referenced by mxc:// URLs which may be shared across multiple events (e.g. stickers or forwarded events, including E2E encrypted events), and moreover mxc:// URLs aren't currently authorized.  As a first cut, we track which user uploaded the mxc:// content, and if they erase themselves then the content will also be erased.
• Homeservers must not serve up unredacted events over federation to users who were not in the room at the time. This poses some interesting problems in terms of the privacy implications of sharing MXIDs of erased users over federation - see "GDPR erasure of MXIDs" below.
• Matrix must specify a way of informing both servers and clients (especially bots and bridges) of GDPR erasures (as distinct from redactions), so that they can apply the appropriate erasure semantics.

🔗GDPR erasure of Matrix IDs

One interesting edge case that comes out of GDPR erasure is that we need a way to stop GDPR-erased events from leaking out over federation - when in practice they are cryptographically signed into the event Directed Acyclic Graph (DAG) of a given room. Today, we can remove the message contents (and preserve the integrity of the room's DAG) via redaction - but this still leaves personally identifying information in the form of the Matrix IDs (MXIDs) of the sender of these events.

In practice, this could be quite serious: imagine that you join a public chatroom for some sensitive subject (e.g. #hiv:example.com) and then later on decide that you want to erase yourself from the room. It would be very undesirable if any new homeserver joining that room received a copy of the DAG showing that your MXID had sent thousands of events into the room - especially if your MXID was clearly identifying (i.e. your real name).

Mitigating this is a hard problem, as MXIDs are baked into the DAG for a room in many places - not least to identify which servers are participating in a room. The problem is made even worse by the fact that in Matrix, server hostnames themselves are often personally identifying (for one-person homeservers sitting on a personal domain).

We've spent quite a lot time reasoning through how to fix this situation, and a full technical spec proposal for removing MXIDs from events can be found at https://docs.google.com/document/d/1ni4LnC_vafX4h4K4sYNpmccS7QeHEFpAcYcbLS-J21Q. The high level proposal is to switch to giving each user a different ID in the form of a cryptographic public key for every room it participates in, and maintaining a mapping of today's MXIDs to these per-user-per-room keys.  In the event of a GDPR erasure, these mappings can be discarded, pseudonymising the user and avoiding correlation across different rooms. We'd also switch to using cryptographic public keys as the identifiers for Rooms, Events and Users (for cross-room APIs like presence).

This is obviously a significant protocol change, and we're not going to do it lightly - we're still waiting for legal confirmation on whether we need it for May 25th (it may be covered as an intrinsic technical limitation of the system).  However, the good news is that it paves the way towards many other desirable features: the ability to migrate accounts between homeservers; the ability to solve the problem of how to handle domain names being reused (or hijacked); the ability to decouple homeservers from DNS so that they can run clientside (for p2p matrix); etc.  The chances are high that this proposal will land in the relatively near future (especially if mandated by GDPR), so input is very appreciated at this point!

🔗Consent

GDPR describes six lawful bases for processing personal data. For those running Matrix servers, it seems the best route to compliance is the most explicit and active one: consent.

Consent requires that our users are fully informed as to exactly how their data will be used, where it will be stored, and (in our case) the specific caveats associated with a decentralised, federated communication system. They are then asked to provide their explicit approval before using (or continuing to use) the service.

In order to gather consent in a way that doesn't break all of the assorted Matrix clients connecting to matrix.org today, we have identified both an immediate- and a long-term approach.

The (immediate-term) todo list for gathering consent is:

• Modify Synapse to serve up a simple 'consent tool' static webapp to display the privacy notice/terms and conditions and gather consent to this API.
  • Add a 'consent API' to the CS API which lets a server track whether a given user has consented to the server's privacy policy or not.
• Send emails and push notifications to advise users of the upcoming change (and link through to the consent tool)
• Develop a bot that automatically connects to all users (new and existing), posting a link to the consent tool. This bot can also be used in the future as a general 'server notice channel' for letting server admins inform users of privacy policy changes; planned downtime; security notices etc.
• Modify synapse to reject message send requests for all users who have not yet provided consent
  • return a useful error message which contains a link to the consent tool
• Making our anonymised user analytics for Riot.im 'opt in' rather than 'opt out' - this isn't a requirement of GDPR (since our analytics are fully anonymised) but reflects our commitment to user data sovereignty

Long-term:

• Add a User Interactive Auth flow for the /register API to gather consent at register
• As an alternative to the bot:
  • Fix user authentication in general to distinguish between 'need to reauthorize without destroying user data' and 'destroy user data and login again', so we can use the re-authorize API to gather consent via /login without destroying user data on the client.
  • port the /login API to use User Interactive Auth and also use it to gather consent for existing users when logging in

🔗Deactivation

Account deactivation (the ability to terminate your account on your homeserver) intersects with GDPR in a number of places.

Todo list for account deactivation:

• Remove deactivated users from all rooms - this finally solves the problem where deactivated users leave zombie users around on bridged networks.
• Remove deactivated users from the homeserver's user directory
• Remove all 3PID bindings associated with a deactivated user from the identity servers
• Improve the account deactivation UX to make sure users understand the full consequences of account deactivation

🔗Portability

GDPR states that users have a right to extract their data in a structured, commonly used and machine-readable format.

In the medium term we would like to develop this as a core feature of Matrix (i.e. an API for exporting your logs and other data, or for that matter account portability between Matrix servers), but in the immediate term we'll be meeting our obligations by providing a manual service.

The immediate todo list for data portability is:

• Expose a simple interface for people to request their data
• Implement the necessary tooling to provide full message logs (as a csv) upon request. As a first cut this would be the result of manually running something like select * from events where user=?.

🔗Other

GDPR mandates rules for all the personal data stored by a business, so there are some broader areas to bear in mind which aren't really Matrix specific, including:

• Making a clear statement as to how data is processed if you apply for a job
• Ensuring you are seeking appropriate consent for cookies
• Making sure all the appropriate documentation, processes and training materials are in place to meet GDPR obligations.

🔗Conclusion

So, there you have it. We'll be tracking progress in github issues and an associated dashboard over the coming weeks; for now https://github.com/matrix-org/synapse/issues/1941 (for Right to Erasure) or https://github.com/vector-im/riot-meta/issues/149 (GDPR in general) is as good as place as any to gather feedback. Alternatively, feel free to comment on the original text of this blog post: https://docs.google.com/document/d/1JTEI6RENnOlnCwcU2hwpg3P6LmTWuNS9S-ZYDdjqgzA.

It's worth noting that we feel that GDPR is an excellent piece of legislation from the perspective of forcing us to think more seriously about our privacy - it has forced us to re-prioritise all sorts of long-term deficiencies in Matrix (e.g. dependence on DNS; improving User Interactive authentication; improving logout semantics etc). There's obviously a lot of work to be done here, but hopefully it should all be worth it!