Google Summer of Code acceptees were announced, and we're really excited to have FIVE Matrix-related projects to look forward to!
Three of the students will be mentored by matrix.org folk, and two more will work on Fractal under the GNOME org.
Fractal Hackfest 2018 will take place next month in Strasbourg, hosted by Epitech. They'll be planning a roadmap for the year, and working on their current focus areas.
After the blog post on matrix.org yesterday, lots of attention developed around the news. Take a look at this Hacker News thread with lots of discussion, which has stayed on the front page for nearly 24 hours, and a megathread on Reddit/r/linux: nearly 1000 upvotes and growing.
Staying in QMatrix-land, there is a pre-release of Quaternion available. Take a look at v0.0.9, improvements include "redactions, file downloading, room creation and settings editing, better timeline visualisation and much more"!
f0x has been beavering away on neo, and has announced that Alpha 0.01 is available now. neo is a webclient in React, it feels really light and fast. I asked f0x about the uptick in work, and he mentioned he got started with React recently, but likes it a lot.
Says TravisR: "I've whipped together a simple Trello notification bot for matrix: matrix-trello-bot. Future plans include the ability to create/update/delete cards and more granular control of what is notified about. Currently it acts very similar to how the Github notifications bot works."
MTRNord is working on a "matrix DSL", basically a config file language that generates a project Template in multiple languages. The project is just now at the discussion stage, so join #matrix_dsl:matrix.ffslfl.net to check out what's going on. (DSL = Domain Specific Language.)
Says Coffee: "urllib-requests-adapter has been updated to work with matrix-python-sdk 0.2.0. urllib-requests-adapter is a lightweight replacement for requests and its dependencies, currently standing at 126 SLOC."
Missed this one last week, but matrix-stfu from xwiki is a message filtering tool released recently, it can "mass remove everything which was said by a particular user in a particular room". XWiki use matrix as their internal chat system for the ~30 people at the company.
There is a update out on try-matrix-now, matrix.org's central listing of projects. Try filtering and see whether benpa mangled the metadata for your project. Submit any changes as markdown PRs on the repo.
benpa is currently working on a Proposals page for the Spec to properly stack spec proposal status, at last!
uhoreg pointed to some charming coverage over on https://itsfoss.com: Riot: A Distributed Way of Having IRC and VOIP Client and Home Server, by Shirish. The article covers some details about riot-web and the open source ethos of Matrix, but my favourite quote by far:
"Without Matrix, Riot would be like a body without a soul."
Last year we had two students working on Matrix-projects over the summer - you can read the retrospective here - and now we are again offering students to work on Matrix as part of GSoC! Currently we are in the stage where students can propose interesting project ideas to any of the open source organisations picked by Google. Of course, we encourage students to get in touch with us and discuss their ideas before writing their application - please come say hi in the #gsoc:matrix.org room!
We are very eager to see what ideas students come up with. We have added our own ideas here, but students are expected to do some research and come up with their own ideas for projects. We have also written down some general tips on what to include in the application.
Applications can be submitted from March 20th, so there's still plenty of time to have a play with Matrix and come up with a cool project idea!
Another season has passed; the leaves are dropping from the trees in the northern hemisphere (actually, in the time it's taken us to finish this post, most of them have dropped :-/) and once again the Matrix team has been hacking away too furiously to properly update the blog. So without further delay here's an update on all things Matrix!
You can tell when it's finished based on whether it's stopped logging about the 'background_deduplicate_state' task. There was a bug in 0.18.0 that meant this process was very slow (weeks) on sqlite DBs and chewed 100% CPU; this was fixed in 0.18.1, and subsequently we've also had 0.18.2 (various perf and bug fixes, and a new modular internal API for authentication) and the current release: 0.18.3 to address a major vulnerability on deployments using LDAP with obsolete versions (0.9.x) of the python ldap3 library - e.g. Debian Stable. Folks using the Debian Stable packages must upgrade immediately.
Other big changes in Synapse 0.18 were:
To find out more and get upgraded if you haven't already, please check out the full changelog.
The current scaling approach for this is called 'Workers' - where we've split out a whole bunch of different endpoints from the main Synapse process into child 'worker' processes which replicate their state from the master Synapse process. These workers are designed to scale horizontally, adding as many as you like to handle the traffic load. It's not full active/active horizontal scalability in that you're still limited by the performance of the master process and the database master you're writing to, but it's a great way to escape Python's global interpreter lock limiting processes effectively to a single core, and in practice it's a huge improvement and works pretty well as of Synapse 0.18.
You can read more about the architecture and how to run your Synapse in worker-mode over at https://github.com/matrix-org/synapse/blob/master/docs/workers.rst.
Riot (now Element) itself is a fairly thin layer on top of the official client Matrix SDKs, and so 95% of the work for Riot (now Element) took the form of updates to matrix-js-sdk, matrix-react-sdk, matrix-ios-sdk, matrix-ios-kit, matrix-android-sdk and synapse itself. There's been a tonne of changes here since June, but the main highlights are:
When E2E first landed on Riot/Web (now Element) in September we were missing mobile support, encrypted attachments, encrypted VoIP signalling, and the ability to retrieve encrypted history on new devices - as well as a formal audit of the underlying Olm and Megolm libraries. Since then things have progressed enormously with most of the core team working since September on filling in the gaps, as well as getting audited and fixing all the weird and wonderful edge cases that the audit showed up. All the missing stuff has been landing on the develop branches over the last few weeks, with encrypted attachments landing on web on Nov 10; encrypted VoIP landing on Nov 11; etc. Watch this space for news on the upcoming cross-platform public beta!
The Bot integrations are all provided by go-neb - a complete rewrite in Golang and general reimagining of the old python NEB bot which old-timers will recall as the very first bot written for the Matrix ecosystem. Go-neb has effectively now become a general purpose golang bot/integration framework for Matrix, with the various different services implemented as plugins for Github, JIRA, Giphy, Guggy etc. Critically it supports authenticating Matrix users through to the remote service, letting normal Matrix users interact with Github and friends using their actual Github identity rather than via a bot user - this is a huge huge improvement over the original naive python NEB.
If you like Go and you like Matrix, we'd strongly suggest having a go (hah) at adding new services into go-neb: anything implemented against go-neb will also magically be hosted and available as part of the "Manage Integrations" interface in Riot (now Element), as well as being available to anyone else running their own go-nebs. For full details of the architecture and how to implement new plugins, go check out the full README.
If Matrix is to provide a good FOSS alternative to systems like Slack it's critical to have a large array of available integrations, so we really hope that the community will help us grow the list!
In short, this means that any user can go and take an existing Matrix room and link it through to Slack, IRC, Gitter, and more.
You can read the full changelogs for 0.5.0 and 0.6.0 at https://github.com/matrix-org/matrix-appservice-irc/blob/master/CHANGELOG.md, but the main highlights are:
matrix-ircd really is an incredibly promising way of getting folks onto Matrix, as it exposes the entirety of Matrix as a virtual IRC network. This means that IRC addicts can jack straight into Matrix, talking native IRC from their existing IRC clients - but interacting directly with Matrix rooms as if they were IRC channesls without going through a bridge. Obviously you lose all of the features and semantics whith Matrix provides beyond IRC, but this is still a great way to get started.
The project is currently alpha but provides a good functioning base to extend, and Erik's explicitly asking for help from the Rust and Matrix community to fill in all the missing features. If you're interested in helping, please come talk on #matrix-ircd:matrix.org!.
This obviously has some major advantages (your actions on Gitter use your real Gitter account rather than a bot), and some disadvantages too (you can't use Matrix features when interacting with other Matrix users in the same room, and the Gitter channel itself is not decentralised into Matrix). However, it's a really cool example of how the other model can work - and within the core team, we've been arguing back and forth for ages now on whether normal bridges or "sidecar" bridges like this one are a more preferable architecture. Thanks to Remram's work we can try both side by side! Go check it out at https://github.com/remram44/matrix-appservice-gitter-twisted.
Aviral's work has been widespread throughout Riot (now Element): adding consistent Emoji support throughout the app via Emoji One, implementing the beta Rich Text Editor (RTE) and all-new autocompletion UI, as well as a bunch of spec proposals for rich message semantics and an initial Slack Webhooks application service. You can read his wrap up at http://www.aviraldg.com/p/gsoc-2016-wrapup and use the code in Riot/Web (now Element) today. We're currently working on fixing the final issues on RTE and auto-complete and hope to enable them by default real soon now!
Meanwhile, Half-Shot's work ended up focusing on bridging through to Twitter and working on the Threading spec proposal for Matrix. You can find out all about the Twitter bridge at https://half-shot.github.io/matrix-appservice-twitter; it works incredibly well (arguably too well, given the amount of traffic it can bridge into Matrix! :S) - and we're currently working on hosting a version of it on matrix.org for all your tweeting needs. You can see Half-Shot's wrapup blog post over at https://half-shot.uk/gsoc16_evaulation.
Finally, as a bit of a wildcard, we discovered the other day that there was also another GSoC project using Matrix by Waqee Khalid, supported by the Berkman Center for Internet and Society at Harvard to switch Apache Wave (formerly Google Wave) over to using Matrix rather than XMPP for federation! The implementation looks a little curious here, as Wave used XMPP as a blunt pubsub layer for synchronising protobuf deltas - and it looks like this implementation uses Matrix similarly, thus killing any interop with the rest of Matrix, which is a bit of a shame. If anyone knows more about the project we'd love to hear though!
Either way, it's been a pleasure to work with the GSoC community and we owe Aviral and Half-Shot (and Waqee!) a huge debt of gratitude for spending their summers (and more!) hacking away improving Matrix. So, thanks Google for making GSoC possible and thanks to the GSoCers for all their contributions, effort & enthusiasm! Watch this space for updates on RTE, new-autocomplete and the twitter bridge going live...
More recently, we were lucky enough to get an indepth video interview with Bryan Lunduke as part of his 'Linux & Whatnot' series, and also a write-up in NetworkWorld alongside Signal & Wire as part of Bryan's journeys in the land of encrypted messaging.
Huge thanks to everyone who's been nice enough to spread the word of Matrix!
Then in October we spoke about scaling Python/Twisted for Matrix at PyCon France in Rennes - this was really fun, albeit slightly embarrassing to be the only talk about Python/Twisted in a track otherwise entirely about Python 3 and asyncio :D That said, the talk seemed to be well received and it was fantastic to meet some of the enthusiastic French Python community and see folks in the audience who were already up and running on Matrix!
As you can see, things are accelerating quite nicely - and this is ignoring all the traffic in the rest of the Matrix ecosystem that happens not to be federated onto the matrix.org HS, not to mention the huge amounts of traffic due to bridging.
Our plans over the next few months are going to involve:
Now more than ever before we believe that it is absolutely critical to have a healthy and secure decentralised communications ecosystem on the 'net (whether that's Matrix, XMPP, Tox or whatever) - so thank you again for participating in our one :) And if you don't already run your server, please grab a Synapse and have fun!
Hi folks - another few months have gone by and once again the core Matrix team has ended up too busy hacking away on the final missing pieces of the Matrix jigsaw puzzle to have been properly updating the blog; sorry about this. The end is in sight for the current crunch however, and we expect to return to regular blog updates shortly! Meanwhile, rather than letting news stack up any further, here's a quick(?) attempt to summarise all the things which have been going on!
There's also been a huge amount of work going on behind the scenes on horizontal scalability for Synapse. We haven't drawn much attention to this yet (or documented it) as it's still quite experimental and in flux, but the main change is to add the concept of application-layer replication to Synapse - letting you split the codebase into multiple endpoints which can then be run in parallel, each replicating their state off the master synapse process. For instance, right now the Matrix.org homeserver is actually running off three different processes: the main synapse; another specific to calculating push notifications and another specific to serving up the /sync endpoint. These three are then abstracted behind the dendron layer (which also implements the /login endpoint). The idea is that one can then run multiple instances of the /sync and pusher (and other future) endpoints to horizontally scale. For now, they share a single database writer, but in practice this has improved our scalability and performance on the Matrix.org HS radically.
In future we'll actually document how to run these, as well as making it easy to spin up multiple concurrent instances - in the interim if you find you're hitting performance limits running high-traffic synapses come talk to us about it on #matrix-dev:matrix.org. And the longer term plan continues to be to switch out these python endpoint implementations in future for more efficient implementations. For instance, there's a golang implementation of the media repository currently in development which could run as another endpoint cluster.
Future work on Vector is focused on showcasing end-to-end encryption, providing a one-click interface for adding bots/integrations & bridges to a room, and generally enormously improving the UX and polish. Meanwhile, there's an F-Droid release for Android landing any day now.
If you haven't checked it out recently, it's really worth a look :)
Also, Ruma just released some truly excellent documentation as a high-level introduction to Matrix (thanks to Leah and Jimmy) - much better than anything we have on the official Matrix.org site. Go check it out if you haven't already!
This deserves a post in its own right, but the key thing to know is that Olm is almost ready - and indeed the work-in-progress E2E UX is even available on the develop branch of vector if you enable E2E in the new 'Labs' section in User Settings. Olm itself is usable only for 'burn after reading' strictly PFS messages, but Megolm integration with Vector & Synapse will follow shortly afterwards which will finally provide the E2E nirvana we've all been waiting for :)
We strongly recommend checking out the videos which have all been published at Decentralised Web Summit, including lightning talks introducing both Matrix and Vector, and digging into as many of the projects advertised as possible. It was particularly interesting for us to get to know Tim Berners-Lee's latest project at MIT: Solid - which shares quite a lot of the same goals as Matrix, and subsequently seeing Tim pop up on Matrix via Vector. We're really looking forward to working out how Matrix & Solid can complement each other in future.
PRs very welcome at https://github.com/matrix-org/matrix.to.
For instance, the IRC bridge has had a complete overhaul to add in huge numbers of new features and finally deployed for Freenode a few weeks ago:
!nickwill now be preserved across bridge restarts.
!nickcommands are issued).
bodyif the HTML content contains any unrecognised tags. This makes passing Markdown from Matrix to IRC much nicer.
matrixClients.displayNameis now implemented.
-are temporarily filtered out from being bridged.
MemberListSyncerto avoid HOL blocking.
/initialSyncrequest to avoid hitting this endpoint more than once, as it may be very slow.
user_ideven if they had a display name set.
Finally, NEB - the Matrix.org bot framework is currently being ported from Python to Golang to act as a general Go SDK for rapidly implementing new bot capabilities.
There's little point in all of the effort going into bridges and bots if it's too hard for normal users to deploy them, so on the Vector side of things there's an ongoing project to build a commercial-grade bot/bridge hosted service offering for Matrix which should make it much easier for non-sysadmins to quickly add their own bots and bridges into their rooms. There's nothing to see yet, but we'll be yelling about it when it's ready!
Thanks, once again, to everyone who's been supporting and using Matrix - whether it's by hanging out in the public chatrooms, running your own server, writing your own clients, bots, or servers, or just telling your friends about the project. The end of the beginning is in sight: thanks for believing in us, and thank you for flying Matrix.
Matthew, Amandine & the Matrix Team.
use_frozen_eventsoff by default (PR #877)
/publicRoomendpoints when possible (PR #856)
/thumbnailwhen can't connect to remote server (PR #862)
/publicRoomsperformance (PR #859)
url_preview_ip_range_whitelistconfig param (PR #760)
NULLpassword hashes from the database (PR #775)
get_users_with_read_receipts_in_roomcache (PR #809)
get_joined_hosts_for_roomuse get_users_in_room (PR #828)
/syncrequests (PR #668)
Congratulations to Aviral Dasgupta and Will "Half-Shot" Hunt who will be working with Matrix for their Google Summer of Code projects!
As mentioned, picking two projects out of all our proposals was no easy task. However, we now look forward to getting started, and we are sure Aviral and Half-Shot will help make Matrix even better over the next few months!
Aviral will be developing a flexible plugin system to facilitate integrating various services such as github/trello/duckduckgo with Matrix. Meanwhile, Half-Shot will be looking at adding features on top of Matrix - infact, he's already built a MPD DJ bot and started working on a .NET SDK. Aviral too, has been committing various enhancements already.
According to Google's GSoC timeline we are currently in the "Community Bonding" phase, which lasts till May 22, 2016 - which is when the projects formally kick off.
We're looking forward to seeing what awesome things Aviral and Half-Shot come up with!
As previously announced, Matrix is participating in Google Summer of Code (GSoC) 2016. We have had a lot of interest: lots of people joining Matrix to talk to us about their project ideas and a total of 38 project proposals. We have even had some code contributions to our various projects from people who discovered Matrix via GSoC!
It's our first year as a GSoC mentoring organisation and we were only allocated two project slots. This means that we had the tough decision of choosing between some really good projects - and that means a lot of you who applied will unfortunately be left feeling disappointed. Selecting our two projects was very difficult, and we talked it over until we all agreed. Please remember that not being picked does not mean that your proposal was bad.
If you missed out on a GSoC slot this year, that doesn't have to stop you from contributing, either by hacking on your own project or contributing to an existing Matrix project. It's a great way to hone your programming skills and we'll be more than happy to help out and support you - find us in #matrix:matrix.org and #matrix-dev:matrix.org.
All the best from the Matrix team and good luck to everyone in their summer projects, whether GSoC or not!