Over the past decade, computing resources that were previously available only to large organizations became available to almost anyone. Using cloud-scale development platforms like Amazon Web Services, developers can write software that runs on hundreds or even thousands of servers, and do so relatively cheaply.
But it is still difficult to write software that makes efficient use of this abundant computing. For some projects, like creating websites, there are well-known software architectures that work reasonably well. In other areas, there's been progress building generalized tools (for example, Hadoop in data processing). For the most part, however, developers need to solve the parallelization problem over and over again for each application they develop. New tools that help them do this are sorely needed.
Today, I am excited to announce that a16z is investing $20M in Improbable, a London-based company that was founded by a group of computer scientists from the University of Cambridge. Improbable’s technology solves the parallelization problem for an important class of problems: anything that can be defined as a set of entities that interact in space. This basically means any problem where you want to build a simulated world. Developers who use Improbable can write code as if it will run on only one machine (using whatever simulation software they prefer, including popular gaming/physics engines like Unity and Unreal), without having to think about parallelization. Improbable automatically distributes their code across hundreds or even thousands of machines, which then work together to create a seamlessly integrated, simulated world.
The Improbable team had to solve multiple hard problems to make this work. Think of their tech as a “spatial operating system”: for every object in the world — a person, a car, a microbe —the system assigns “ownership” of different parts of that entity to various worker programs. As entities move around (according to whatever controls them -- code, humans, real-world sensors) they interact with other entities. Often these interactions happen across machines, so Improbable needs to handle inter-machine messaging. Sometimes entities need to be reassigned to new hardware to load balance. When hardware fails or network conditions degrade, Improbable automatically reassigns the workload and adjusts the network flow. Getting the system to work at scale under real-world conditions is a very hard problem that took the Improbable team years of R&D.
One initial application for the Improbable technology is in gaming. Game developers have been trying to build virtual worlds for decades, but until now those worlds have been relatively small, usually running on only a handful of servers and relying on hacks to create the illusion of scale. With Improbable, developers can now create games with millions of persistent, complex, interacting entities. In addition, they can spend their time inventing game features instead of building back-end systems.
Beyond gaming, Improbable is useful in any field that models complex systems — biology, economics, defense, urban planning, transportation, disease prevention, etc. Think of simulations as the flip side to “big data.” Data science is useful when you already have large data sets. Simulations are useful when you know how parts of the system work and want to generate data about the system as a whole. Simulations are especially well suited for asking hypothetical questions: what would happen to the world if we changed X and Y? How could we change X and Y to get the outcome we want?
Improbable was started three years ago at Cambridge by Herman Narula and Rob Whitehead. They have since built an outstanding team of engineers and computer scientists from companies like Google and top UK computer science programs. They’ve done all of this on a small seed financing, supplemented by customer revenue and research grants. We are thrilled to partner with Improbable on their mission to develop and popularize simulated worlds.