Two weeks ago we got together in Berlin for
another
(Un)boiling The Ocean
event
(slight name change because Mastodon does not deal well with metaphors). This time it was laser-focused on
local-first sync
, i.e. software that can move seamlessly between real-time collaboration when there’s a network connection, and working offline when there is no connection.
The New p2panda
This event was the next step in our ongoing collaboration with the
p2panda project
. p2panda provides building blocks for local-first software and is spearheaded by
Andreas Dzialocha
and
Sam Andreae
. Since our initial discussions in late 2023 they made a number of structural changes to p2panda, making it more modular and easier to use for cases like ours, i.e. native GNOME apps.
Sam and Andreas introducing the new p2panda release.
This
new version of p2panda
shipped a few weeks ago, in the form of a dozen separate Rust crates, along with a new website and new documentation.
On Saturday night we had a little Xmas-themed release party for the new p2panda version, with food, Glühwein, and two talks from
Eileen Wagner
(on
peer-to-peer UX patterns
) and
Sarah Grant
(on radio communication).
The Hackfest
Earlier on Saturday and then all day Sunday we had a very focused and productive hackfest to finally put all the pieces together and build our long-planned prototype codenamed “Aardvark”, a local-first collaborative text editor using the p2panda stack.
Simplified diagram of the overall architecture, with the GTK frontend, Automerge for CRDTs, and p2panda for networking.
Our goal was to put together a simple Rust GTK starter project with a TextView, read/write the TextView’s content in and out of an
Automerge
CRDT, and sync it with other local peers via p2panda running in a separate thread. Long story short: we pulled it off! By the end of the hackfest we had basic collaborative editing working on the local network (modulo some bugs across the stack). It’s of course still a long road from there to an actual releasable app, but it was a great start.
The reason why we went with a text editor is not because it’s the easiest thing to do — freeform text is actually one of the more difficult
types of CRDT
. However, we felt that in order to get momentum around this project it needs to be something that we ourselves will actually use every day. Hence, the concrete use case we wanted to target was replacing Hedgedoc for taking notes at meetings (particularly useful when having meetings at
offline
, where there’s no internet).
The current state of Aardvark: Half of the UI isn’t hooked up to anything yet, and it only sort of works on the local network :)
While the Berlin gang was hacking on the text editor, we also had
Ada
, a remote participant, looking into what it would take to do collaborative sketching in
Rnote
. This work is still in the investigation stage, but we’re hopeful that it will get somewhere as a second experiment with this stack.
Thanks to everyone who attended the hackfest, in particular Andreas for doing most of the organizing, and Sam Andreae and Sebastian Wick, who came to Berlin specifically for the event! Thanks also to
Weise7
for hosting us, and
offline
for hosting the release party.
The Long Game
Since it’s early days for all of this stuff, we feel that it’s currently best to experiment with this technology in the context of a specific vertically integrated app. This makes it easy to iterate across the entire stack while learning how best to fit together various pieces.
However, we’re hoping that eventually we’ll settle on a standard architecture that will work for many types of apps, at which point parts of this could be split out into a system service of some kind. We could then perhaps also have standard APIs for signaling servers (sometimes needed for peers to find each other) and “dumb pipe” sync/caching servers that only move around encrypted packets (needed in case none of the other peers are online). With this there could be many different interchangeable sync server providers, making app development fully independent of any specific provider.
Martin Kleppmann’s
talk at Local-First Conf 2024
outlines his vision for an ecosystem of local-first apps which all use the same standard sync protocol and can thus share sync services, or sync peer-to-peer.
This is all still pretty far out, but we imagine a world where as an app developer the only thing you need to do to build real-time collaboration is to integrate a CRDT for your data, and use the standard system API for the sync service to find peers and send/receive data.
With this in place it should be (almost) as easy to build apps with seamless local-first collaboration as it is to build apps using only the local file system.
Next Steps
It’s still early days for Aardvark, but so far everyone’s very excited about it and development
has been going strong
since the hackfest. We’re hoping to keep this momentum going into next year, and build the app into a more full-fledged Hedgedoc replacement as part of p2panda’s NGI project by next summer.
That said, we see the main value of this project not in the app itself, but rather the possibility for our community to experiment with local-first patterns, in order to create capacity to do this in more apps across our ecosystem. As part of that effort we’re also interested in working with other app developers on integration in their apps, making bindings for other languages, and working on shared UI patterns for common local-first user flows such as adding peers, showing network status, etc.
If you’d like to get involved, e.g. by contributing to Aardvark, or trying local-first sync in your own app using this stack feel free to reach out on Matrix (aardvark:gnome.org), or the
Aardvark repo on Github
.
Happy hacking!
chevron_right
