No more DSLs: Implement and deploy a distributed system with a single program

submited by

Style Pass

2021-07-09 19:30:12

You can write a distributed system as a single program in a general-purpose language, and run that program to directly run the full system for production or for testing the system against its users. I call this a "single-program system". In this article, I describe the approach used to do this at $DAYJOB, and show some working example programs. Although $DAYJOB's single-program systems are proprietary, they're built with open source libraries which can be used today by anyone. And although these examples are in Python, this approach is in no way tied to Python and this article should be comprehensible and useful to anyone interested in distributed systems.

There are distributed languages and other models which also aim at the goal of implementing a distributed system as a single holistic program. Such languages assume the presence of a great deal of distributed systems infrastructure that is maintained outside the language. A program written in such a language is then only one component in a larger distributed system. The single-program system approach, instead, allows you to write a program which manages every aspect of the distributed system above the hardware level, such that running the program deploys the distributed system, and the program and the distributed system really can be said to be the same thing.

At $DAYJOB, each team maintains libraries for their components; these libraries are used to write the single-program system. Some components are in Python, but others are written in other languages (mostly C/C++/Java) and run in a separate process. In this way, performance-sensitive components of a distributed system are written in faster languages than Python, in the same way that Fortran libraries are used to speed up math in Python programs. All the component libraries are maintained in a single repository to reduce compatibility issues.