Bowtie delivers secure private network access, without Bowtie-the-company sitting between you and your private resources. We have already explored the trade-offs that companies make when they let someone else run this infrastructure, and now we will explore some of the product and engineering challenges in delivering a SaaS-like experience without intermediating the network flow.
This is the first of several posts about how the Bowtie control plane works, covering our use of Automerge, the “Local First” conflict free replicated data type implementation.
The Promise
We use Automerge because we need a resilient, horizontally scalable, and fully partition-tolerant data store. Every Bowtie controller can operate independently, or can cluster to scale capacity, introduce fault tolerance, and connect multiple sites together to form the network tapestry common to many businesses. This makes our product easy to think about and deploy. Most web applications still work on an “N Tier” architecture: cache up front, data processing in the middle, and data storage at the back. We needed an architecture that would sip resources and be totally self-sufficient, even down to one node.
Automerge folks like to say their database is like git for a JSON document, with no merge conflicts. Quite importantly, in both very good and kind of painful ways, they do not mean “it is like git with GitHub….” Git certainly existed before GitHub, but it was not used as much. Now, almost all software is managed in git, and almost always with GitHub or some GitHub-like central repository for collaboration.
It is important to remember though that git does not need a central repository. I can have a git repository, and you can have a git repository, and we can keep track of our changes together over email, or SSH, or by passing a thumb drive full of changes around. You can see this happening on the Linux kernel distribution list.
Automerge being like git without merge conflicts means approximately that if you and I both have an Automerge document, and we both make different changes, even to the same records, when we join back together, it will still be an Automerge document with the same structure, with a consistent state, and with the full history of both sets of changes. WIth this, you can make an application.
The Caveats
Building a new application architecture on a new database has fairly predictable drawbacks. Most significant is that it is not a well-trodden path, meaning that you must establish many of the patterns you will need on your own.
Caveat 1: Sync and Store
Automerge, like git, has a few different suggested mechanisms for passing changes around.
You can pass whole documents and compare them, or you can leverage their Byzantine Fault Tolerant Sync Protocol implementation from Martin Klepmann’s 2022 paper. Either way, you need to bring your own connection, persistence, and security model.
We spent a few months with each of these items, ultimately deciding to sponsor Ink & Switch to develop a canonical open source reference Rust implementation as a better, generically applicable starting point.
Importantly, the network, storage, and security mechanisms utilize a plug-in architecture. That means that Automerge can do what it does best (writing and syncing state) and we can do what we do best: building an authenticated, encrypted network of related private applications and networks.
We added a local storage mechanism, so every controller stores its own copy of state, and added our network policy engine to it. With every controller authenticated and networked together, they can securely gossip changes as they happen, knowing that any changes are coming from other trusted controllers. No data ever leaves your network and every controller runs an independent, synchronized copy of global state.
Caveat 2: Querying and Manipulating
If you are familiar with other document stores, or with dealing with piles of JSON documents, then querying and manipulating data in Automerge will feel similar. We made a handful of attempts at various approaches and ended with two libraries. The first is another Ink and Switch project: Autosurgeon. Autosurgeon is an elegant library that binds Rust’s incredible type and trait system into Automerge by “hydrating” and “reconciling” objects back and forth from the document into local objects.
Autosurgeon needed a little more help hydrating and reconciling partial objects instead of the whole document, so we also wrote a library we called “Automerge_orm” for finding or upserting smaller objects into a larger document. We have broken that out of our main code base into a public open source project.
You might have noticed that I said “upserting”. There is one more big caveat with this architecture: you never know if the controller has seen an object yet. In practice, this means that we do a lot of “update or insert” operations. If you got an object from one controller, and you put it on another controller during a partition, we want those two controllers to result in the same state when they sync again. This is most easily done by using UUIDs everywhere and not relying on ACID like guarantees about data consistency. You have to be a little defensive at both insertion and retrieval to make sure that you have everything you need, but we have seen many instances where Rust’s strong typing and exhaustive error handling made this fairly straightforward.
You can also use some fairly clever computer science to make sure that, even in a complicated distributed system, you can achieve exactly-once operations, which is pretty cool.
The Payoff
With all the pieces in place, we see data convergence on the order of tens-of-milliseconds across a cluster sized for organizations with several thousand concurrent users. This enables us to deliver our product in the same way you interact with a SaaS solution, without single points of failure and without requiring you to rely on our infrastructure. All this happens while delivering zero trust access at line-like speeds across the overlay network.
If you have a background in distributed systems and want to help us push forward decentralized architectures, please get in touch!