Massive news for the Matrix ecosystem today: New Vector (the startup which the Matrix core team formed to fund development in 2017) has raised an additional $8.5M of funding in order to speed up Riot/Matrix development and expand Matrix hosting via Modular.im!
The new funding comes in the form of a Series-A equity investment in New Vector from three of the top venture capital funds in London. The round is led by Notion - a fund set up by the founders of MessageLabs, who many will know as one of the leaders in secure hosted email services. Notion's long history with email means they immediately clocked the potential of Matrix's mission to build a new open global communication network - after all, Matrix aims to provide a worthy replacement to email (and the phone network, for that matter!). Joining Notion in the round is First Minute - a fund set up by the founders of Lastminute.com (arguably the UK's most famous original dotcom), and Dawn - one of the largest SaaS tech specialist funds in Europe (famous for backing iZettle, Mimecast, Neo4J and many more).
The last funding round in Jan 2017 from Status was instrumental in stabilising the big 1.0 release of Matrix and exiting beta in June; creating the Matrix.org Foundation as a neutral custodian for the standard; stabilising and optimising Synapse; redesigning Riot’s user interface; bringing in a full-time professional UI/UX designer to the team; supporting the huge amount of encryption work required to turn on E2EE by default (cross-signing, key backups, device verification, e2e search, the pantalaimon e2e daemon etc); creating RiotX/Android; and launching the Modular.im hosting platform.
With today’s new funding, the priorities for Matrix will be:
Turning on end-to-end encryption by default for DMs
Much better support for grouping rooms into Communities
More anti-abuse/anti-spam mechanisms
Shrinking Synapse (and/or finishing Dendrite)
Canonical DMs (having one DM per user, and have them feel clearly distinct from ‘rooms’)
...and furthering development on P2P Matrix, so users can have full control of their communications without having to run or trust a server.
On the New Vector side, this funding will support:
A whole new wave of UX improvements to Riot (particularly around onboarding and first time user experience).
Making Modular hosting as polished and powerful as possible.
Creating a whole new set of next-generation Modular integrations.
While New Vector’s contributions to the Matrix ecosystem can’t be ignored, it’s important to remember that the Matrix protocol and specification itself is governed and controlled by the independent and neutral Matrix.org Foundation and its extensive governance processes. We set up the Foundation very deliberately to enforce the protocol's neutrality, formalise the project's mission, goals and values and hold true to them no matter what - specifically to protect the project from conflicts of interest with commercial Matrix endeavours, including New Vector.
That said, New Vector would not be taking money from any investors if they did not believe their goals are aligned with Matrix's. To clarify:
It turns out that these goals are not incompatible if one understands that the potential of the Matrix ecosystem is directly linked to its openness and size (hint: funding sources who didn’t understand this self-selected out ;). By funding Matrix development and helping the open ecosystem and public network grow, New Vector can go provide more Matrix hosting via Modular.im and more Government & Enterprise deployments via Vector.im. Critically, other companies can and do build on top of Matrix too - and frankly the more players there are, the more valuable the network, and the more value to be shared for everyone (including New Vector). This model worked relatively well for the Web, and we believe it'll work for Matrix too.
Update: the best way to gauge the investment in New Vector is to hear it first hand from the investors. Jos from Notion is leading the round, and has a fascinating blog post (written with zero input from Matrix or New Vector folks) to explain where the investors are coming from.
Update 2: More excellend first-hand analysis from Dan at Dawn Capital, who does a really deep dive into how they see Matrix and New Vector. Another must read.
In this case, all of New Vector's new investors have a background as respected tech entrepreneurs, and everyone involved categorically understands that Matrix itself is a neutral open source project, and the mission is to help build up the whole network to be as successful as possible rather than sabotage it by constraining it in any way.
All in all, it’s great news for the ecosystem: Matrix is 5 years old now, and while the project is growing faster than ever (over 300% more active users in the last year!) - it's fair to say that we haven't moved as fast as our mainstream competition - for instance, Slack is only a year older, and Discord is a year younger(!) Obviously much of this is due to Matrix being a completely different proposition: we've been creating an open spec; multiple client codebases; multiple server codebases; the bridges; a fault tolerant decentralised network - not to mention the complexities of decentralised E2E encryption. Based on comparing with our endeavours prior to Matrix, we estimate building this stuff in an open and decentralised manner takes roughly 6 times longer.
But the project is now in a position where the foundations are solid: the protocol is out of beta, reference servers and clients are production ready, and it’s more than time to make all of this mainstream. We have to redouble our focus on user experience and ensure that we compare favourably to today’s established alternatives while staying true to Matrix’s principles. Making sure there are Matrix apps out there which provide a credible alternative to with the likes of Slack and WhatsApp (until they eventually join Matrix, of course) is what will make the difference between Matrix being a cliquey FOSS curiosity versus really being the natural successor to today’s instant messaging, email and phone networks.
In the end; Matrix needs full-time contributors in order to continue to grow, and keeping New Vector funded is a very good way to achieve that (New Vector is hiring!). (That said, if any philanthropic billionaires are reading this, the Matrix.org Foundation is actively soliciting donations to improve Matrix independently of New Vector's efforts - particularly around the areas of countering online abuse and disinformation).
In the meantime, huge thanks to Jos at Notion for believing in Matrix and leading this funding round in New Vector - and huge thanks to the other investors who saw the potential! And most of all, thank you to all those supporting Matrix, whether by donating to the Foundation, promoting and using the protocol, or contributing code to the ecosystem. You are the ones keeping the dream alive :)
You can read things from the NV angle over at https://blog.vector.im/8-5m-to-accelerate-matrix/. We hope you’re as excited as we are to open a whole new chapter as Matrix picks up yet more momentum :D
-- Matthew, Amandine, and the whole Matrix team
Hello Matrix enthusiasts! Yesterday we released matrix-appservice-slack 1.0. This marks a major milestone in bridge development for the Matrix.org team, being our first bridge to ever reach 1.0. The decision to release this version came after we decided that the bridge had gained enough features and reached a point of stability where it could be deployed in the wild with minimal risk.
For those not in the know, the Slack bridge is Node.JS based, and bridges slack channels & users into Matrix seamlessly. And for those wondering, yes it works with Mattermost too (since their API is compatible with Slack)! In previous versions only a limited subset of features were supported, making heavy usage of Slack’s webhook API. As of 1.0, the bridge now makes use of the newer Slack Events/RTM API which gives us all we need for a richer bridging experience. Everything from edits and reactions to typing notifications is supported in the 1.0 release.
Finally for those who are self hosting, we are pleased to offer the ability to "puppet" your Slack account using the bridge. Puppeting is the process in which the bridge will send messages as if you were sending them from the Slack client directly, when you talk using your Matrix account. This opens the door to seamless bridging and direct messaging support.
Threading & Reactions!
The bridge has undergone some pretty serious code surgery as well. The whole codebase has been rewritten in TypeScript to take advantage of type checking and generic types. The bridge is currently based upon the matrix-appservice-bridge library. The datastore interface now supports PostgreSQL, wich allows for administrators to inspect and edit the database while the bridge is running, as well as offering helpful performance boost over the NeDB datastore format that was used previously. Finally, the codebase has proper Unit and Integration Tests to ensure new changes will not cause any regressions in behaviour. In short, now is an excellent time to get involved and hack on the bridge. There is already a crafted list of easy issues for new and experienced bridge devs.
Memory usage of the bridge comparison
CPU usage of the bridge comparison
In terms of how many users matrix.org is currently serving at the moment, we present to you some figures:
Of course, our work doesn’t stop at 1.0. The plan for the immediate future of the bridge is to continue adding support for other event types coming from Matrix and Slack to create an ever richer experience. Obvious features are things like topic changes, and syncing membership across the bridge. In the long term future we would also like to add community support to the bridge, so whole Slack workspaces can be bridged across with a single click.
That’s all from me, and I would like to say a massive thank you to Cadair and Ben for both their code and review work on the project and as always, thank you to the community for using the bridge and reporting issues. 🙂
Wheyhey as I live and breathe 1.4.0 is here!
1.4.0 should be considered ‘The Privacy Release’ and is the cumulation of a huge body of work spanning multiple projects to improve data privacy in the matrix ecosystem. You can read more about the full project here.
While we consider 1.4.0 to be a huge leap forward in terms of data privacy, it is really important to note that it contains breaking changes.
If either apply to you failure to act will break your installation. Full details can be found in the upgrade notes.
It is possible for a user to associate an email address or phone number with their account, for a number of reasons:
Before an email address or phone number can be added to a user's account, or before such an address is used to carry out a password-reset, Synapse must confirm the operation with the owner of the email address or phone number. It does this by sending an email or text giving the user a link or token to confirm receipt. This process is known as '3pid verification'. ('3pid', or 'threepid', stands for third-party identifier, and we use it to refer to external identifiers such as email addresses and phone numbers.)
Previous versions of Synapse delegated the task of 3pid verification to an identity server by default. In most cases this server is vector.im or matrix.org.
In Synapse 1.4.0, for security and privacy reasons, the homeserver will no longer delegate this task to an identity server by default. Instead, the server administrator will need to explicitly decide how they would like the verification messages to be sent.
In the medium term, the vector.im and matrix.org identity servers will disable support for delegated 3pid verification entirely. However, in order to ease the transition, they will retain the capability for a limited period. Delegated email verification will be disabled on Monday 2nd December 2019 (giving roughly 2 months notice). Disabling delegated SMS verification will follow some time after that once SMS verification support lands in Synapse.
Once delegated 3pid verification support has been disabled in the vector.im and matrix.org identity servers, all Synapse versions that depend on those instances will be unable to verify email and phone numbers through them. There are no imminent plans to remove delegated 3pid verification from Sydent generally. (Sydent is the identity server project that backs the vector.im and matrix.org instances).
Prior to 1.4.0, the identity server was providing two related-but-separate functions:
a directory for users to publish their contact details and to look up their contacts by their email addresses and phone numbers. a "trusted third party communications network delegate", providing SMS- and email-sending functionality to all homeservers for registration and password reset.
The intention behind the identity server's providing 2. was one of convenience: to save homeserver admins the burden of configuring their homeservers to send email, as well as sparing us the effort of writing generic SMS-gateway integration that would allow any homeserver admin to configure their homeserver to send SMS via their chosen SMS aggregator.
By exposing this 'trusted email/sms delegate' functionality, however, and by including references to the New Vector identity servers in the default configuration for both Synapse and Riot, we created a situation in which users on non-New Vector run homeservers (who had never seen our privacy notice) could easily end up sharing their email addresses and phone numbers with a New Vector identity server.
Using identity servers for registration and password reset introduced yet further complexity - since password reset is a sensitive operation, it was important that the homeserver only use 'trusted' identity servers for this purpose (with the trust being configured by the homeserver admin in the homeserver config). But this created ambiguity over who was ultimately responsible for the choice of identity server and when they could make that choice - the client could present a default identity server at login/registration/password reset time, which the user could choose to override before logging in, but unless the identity server ultimately selected were on the homeserver admin's 'trusted identity servers' list, any identity server operations that were proxied via the homeserver would have been blocked by the homeserver. However, not all identity server operations initiated by the client were proxied via the homeserver - some were sent directly from the client to the identity server, and would not have been filtered.
The best way to solve this problem is to have individual homeservers take ownership for their own email and SMS needs. In the case of email this means configuring details of an appropriate SMTP server or continuing to delegate through an identity server that will allow it to do so. If a homeserver admin makes the active choice to use New Vector's identity servers for delegation (up until 1st December), they should make their own users aware through their privacy notice.
This delegation is entirely separate from the user's choice of identity server for user directory services. As of right now the user is free to choose and trust whichever identity server they wish, or to choose not to use an identity server at all.
Yes, 1.4.0 now automatically garbage collects redacted messages (defaults to 7 days) and removes unused IP and user agent information stored in the user_ips table (defaults to 30 days). Finally, Synapse now warns in its logs if you are using matrix.org as a trusted key server, in case you wish to use a different server to help discover other servers’ keys.
Aside from privacy, we’ve expanded our OpenTracing support and fixed a host of bugs. However the thing that is most exciting is switching on our solution for mitigating forward extremities build up’ by default.
In some cases rooms can accumulate ‘forward extremities’, which are simply an artefact of attempting to resolve the room state over multiple servers. Forward extremities are necessary to ensure that each server can independently arrive at the same view of the room eventually, however processing these extremities can be computationally expensive and degrade server performance overall.
Originally it was an experimental config option but we now feel confident to turn it on by default for all instances - it should make a big difference for the CPU of servers in fragmented rooms.
So that’s it folks, thanks for making it this far. As ever, you can get the new update here or any of the sources mentioned at https://github.com/matrix-org/synapse. Also, check out our Synapse installation guide page
The changelog since 1.3.1 follows:
client_secretin server logs. (#6158)
devicestable, and improve its performance on Postgres. (#6135)
Note that this release includes significant changes around 3pid verification. Administrators are reminded to review the upgrade notes.
trust_identity_server_for_password_resetsconfig option with
account_threepid_delegates, and make the
id_serverparameteter optional on
*/requestTokenendpoints, as per MSC2263. (#5876, #5969, #6028)
/lookupAPI where available, with fallback to v1. (Implements MSC2134 plus
id_access_token authenticationfor v2 Identity Service APIs from MSC2140). (#5897)
/registerala MSC2140. (#5964)
/versionsas per MSC2264. (#5974)
POST /_matrix/client/unstable/account/3pid/unbindendpoint from MSC2140 for unbinding a 3PID from an identity server without removing it from the homeserver user account. (#5980, #6062)
account_threepid_delegate.msisdnfor validating threepid sessions. (#6011)
/account/3pid/bindas per MSC2290. (#6043)
bindparameter from Client Server POST
/accountendpoint as per MSC2290. (#6067)
POST /add_threepid/msisdn/submit_tokenendpoint for proxying submitToken on an
submit_urlresponse parameter to
m.require_identity_serverflag to /version's unstable_features. (#5972)
synapse_federation_known_servers: represents the total number of servers your server knows about (i.e. is in rooms with), including itself. Enable by setting
metrics_flags.known_serversto True in the configuration.(#5981)
synapse_build_info: exposes the Python version, OS version, and Synapse version of the running server. (#6005)
report_stats_endpointoption to configure where stats are reported to, if enabled. Contributed by @Sorunome. (#6012)
/register/availableif registration has been disabled. (#6082)
trusted_key_serversconfig field configured. (#6090)
power_level_content_override.usersdoes not contain
password_reset_success_template, when they are actually
public_baseurl. Thanks to @aaronraimist for the fix! (#5909)
falseand the thumbnail was dynamically generated. Fix reported by rkfg. (#5915)
sidis not provided to
federation_certificate_verification_whitelistnow will not cause
TypeErrorsto be raised (a regression in 1.3). Additionally, it now supports internationalised domain names in their non-canonical representation. (#5996)
submit_tokenendpoint until we implement
users_set_deactivated_flagbackground update. (#6092)
next_linkfor email validation. (#6097)
UID/GIDif they are already correct. (#5970)
SYNAPSE_WORKERenvvar to specify python module. (#6058)
INSTALL.mdto say that Python 2 is no longer supported. (#5953)
/_matrix/client/r0/registerendpoint. Contributed by Awesome Technologies Innovationslabor GmbH. (#5877)
users_in_public_roomsto improve the performance of directory queries. (#5894)
failure_tscolumn to the
destinationsdatabase table. (#6016, #6072)
Back in June we wrote about our plans to tighten up data privacy in Matrix after some areas for improvement were brought to our attention. To quickly recap: the primary concern was that the default config for Riot specifies identity servers and integration managers run by New Vector (the company which the original Matrix team set up to build Riot and fund Matrix dev) - and so folks using a standalone homeserver may end up using external services without realising it. There were some other legitimate issues raised too (e.g. contact information should be obfuscated when checking if your contacts are on Matrix; Riot defaulted to using Google for STUN (firewall detection) if no TURN server had been set up on their server; Synapse defaults to using matrix.org as a key notary server).
We’ve been working away at this fairly solidly over the last few months. Some of the simpler items shipped quickly (e.g. Riot/Web had a stupid bug where it kept incorrectly loading the integration manager; Riot/Android wasn’t clear enough about when contact discovery was happening; Riot/Web wasn’t clear enough about the fact device names are publicly visible; etc) - but other bits have turned out to be incredibly time-consuming to get right.
However, we’re in the process today of releasing Synapse 1.4.0 and Riot/Web 1.4.0 (it’s coincidence the version numbers have lined up!) which resolve the majority of the remaining issues. The main changes are as follows:
Riot no longer automatically uses identity servers by default. Identity servers are only useful when inviting users by email address, or when discovering whether your contacts are on Matrix. Therefore, we now wait until the user tries to perform one of these operations before explaining that they need an identity server to do so, and we prompt them to select one if they want to proceed. This makes it abundantly clear that the user is connecting to an independent service, and why.
Synapse no longer uses identity servers for verifying registrations or verifying password reset. Originally, Synapse made use of the fact that the Identity Service contains email/msisdn verification logic to handle identity verification in general on behalf of the homeserver. However, in retrospect this was a mistake: why should the entity running your identity server have the right to verify password resets or registration details on your homeserver? So, we have moved this logic into Synapse. This means Synapse 1.4.0 requires new configuration for email/msisdn verification to work - please see the upgrade notes for full details.
Sydent now supports discovering contacts based on hashed identifiers. MSC2134 specifies entirely new IS APIs for discovering contacts using a hash of their identifier rather than directly exposing the raw identifiers being searched for. This is implemented in Riot/iOS and Riot/Android and should be in the next major release; Riot/Web 1.4.0 has it already.
Synapse now warns in its logs if you are using matrix.org as a default trusted key server, in case you wish to use a different server to help discover other servers’ keys.
Synapse now garbage collects redacted messages after N days (7 days by default). (It doesn’t yet garbage collect attachments referenced from redacted messages; we’re still working on that).
Synapse now deletes account access data (IP addresses and User Agent) after N days (28 days by default) of a device being deleted.
Riot warns before falling back to using STUN (and defaults to turn.matrix.org rather than stun.google.com) for firewall discovery (STUN) when placing VoIP calls, and makes it clear that this is an emergency fallback for misconfigured servers which are missing TURN support. (We originally deleted the fallback entirely, but this broke things for too many people, so we’ve kept it but warn instead).
All of this is implemented in Riot/Web 1.4.0 and Synapse 1.4.0. Riot/Web 1.4.0 shipped today (Fri Sept 27th) and we have a release candidate for Synapse 1.4 (1.4.0rc1) today which fully ship on Monday.
Riot/Mobile is following fast behind - most of the above has been implemented and everything should land in the next release. RiotX/Android doesn’t really have any changes to make given it hadn’t yet implemented Identity Service or Integration Manager APIs.
This has involved a surprisingly large amount of spec work; no fewer than 9 new Matrix Spec Changes (MSC) have been required as part of the project. In particular, this results in a massive update to the Identity Service API, which will be released very shortly with the new MSCs. You can see the upcoming changes on the unstable branch and compare with the previous 0.2.1 stable release, as well as checking the detailed MSCs as follows:
This said, there is still some work remaining for us to do here. The main things which haven’t made it into this release are:
We’ll continue to work on these as part of our ongoing maintenance backlog.
Finally, it’s really worth calling out the amount of effort that went into this project. Huge huge thanks to everyone involved (given it’s cut across pretty much every project & subteam we have working on the core of Matrix) who have soldiered through the backlog. We’ve been tracking progress using our feature-dashboard tool which summarises Github issues based on labels & issue lifecycle, and for better or worse it’s ended up being the biggest project board we’ve ever had. You can see the live data here (warning, it takes tens of seconds to spider Github to gather the data) - or, for posterity and ease of reference, I’ve included the current issue list below. The issues which are completed have “done” after them; the ones still in progress say “in progress”, and ones which haven’t started yet have nothing. We split the project into 3 phases - phases 1 and 2 represent the items needed to fully solve the privacy concerns, phase 3 is right now a mix of "nice to have" polish and some more speculative items. At this point we’ve effectively finished phase 1 on Synapse & Riot/Web, and Riot/Mobile is following close behind. We're continuing to work on phase 2, and we’ll work through phase 3 (where appropriate) as part of our general maintenance backlog.
I hope this gives suitable visibility on how we’re considering privacy; after all, Matrix is useless as an open communication protocol if the openness comes at the expense of user privacy. We’ll give another update once the remaining straggling issues are closed out; and meanwhile, now the bulk of the privacy work is out of the way on Riot/Web, we can finally get back to implementing the UI E2E cross-signing verification and improving first time user experience.
Thanks for your patience and understanding while we’ve sorted this stuff out; and thanks once again for flying Matrix :)
id_access_tokenin CS API when required (done)
id_access_tokenin CS API when required (done)
Some of you will have been bitten by a bug that prevented Synapse 1.3.0 from starting up correctly. If this is you, please upgrade.
Additionally we have taken the opportunity to fix a dependency bug for our intrepid packagers.
Apologies for the inconvenience.
The changelog since 1.3.0 follows:
sdnotifypython package. (#5871)
Well now, Synapse 1.3.0 is here.
The main thing to know about 1.3.0 is that is contains performance improvements to reduce disk I/O and reduce RAM usage. We’ve been running it on matrix.org for a week or so and are really pleased with the results.
Check out our message send heat map.
Other than that there are a bunch of bug fixes and tweaks to generally make things run more smoothly.
The changelog since 1.2.1 follows:
publicRoomswhen the public room list was cached. (#5851)
M_UNKNOWNfor errcode when a deactivated user attempts to login. (#5686)
enable_media_repois set to False in the configuration. If a media repo worker is used, the admin APIs relating to the media repo will be served from it instead. (#5754, #5848)
/syncstream could get wedged in rare circumstances. (#5825)
--log-configcommand line flags, and removes the deprecated
log_fileconfiguration file options. Users of these options should migrate their options into the dedicated log configuration. (#5678, #5729)
window.openerin default welcome page. (#5695)
/versionfederation requests. (#5730)
.well-knownrequests by sharing the SSL options between requests. (#5794)