DI is a simple, declarative language to configure, boot and connect containers within your local infrastructure and across public clouds. DI enables you to create a complex distributed system of containers by working only with a simplified abstract view of your system and not worrying about the low-level implementation.
If you have ever managed a distributed system, you know that as the system grows, it becomes increasingly difficult to build, configure, and maintain. For a distributed system to function well, it needs consistency among its individual components, as well as flexible and reliable composition to combine its individual components into a whole. Additionally, the system requires comprehensive configuration to control things like data flow, placement and access control.
Ensuring all of the above requirements is difficult, and in recognition of this fact, several companies and open source projects have made good contributions towards simpler system administration. However, even with these tools at hand, it still requires carefully executed API calls and detailed understanding of a system's underlying infrastructure to sucessfully build and maintain it.
With DI, you can leave these low-level details aside and instead focus on the high-level structure and configuration of your system. For instance, for a simple web deployment, we might want a database, some number of web servers connected to the public internet, and some batch processing servers connected to the web servers and the database. Even though this overall structure seems simple, the actual implementation quickly becomes complicated.
DI enables you to bypass these complications and build your distributed system from the top down, maintaining only an abstract view of your infrastructure and a set of explicit policies it follows. You never have to worry about firewall configurations, server placement or other implicit security enforcements; your high-level view and explicit policies make it easy to monitor and verify the state of your system. Regardless of whether your application is hosted on a single or multiple cloud providers, DI ensures that your system is correctly booted and that it always follows your configuration.
As described above, DI allows you to forget about low-level specifics and detailed API calls. With DI, you simply describe the desired state of your system, and DI then handles the rest. To describe your configuration, you will use two main building blocks: atoms and labels.
An atom is the smallest unit of the DI language. Each atom is either an administrative user, a public hostname or IP address, a container, or a virtual machine. Simply creating an atom in your DI config file is enough to boot and register the entity in your system. Similarly, if you want to shut down or unregister a component of your system, you just remove the corresponding atom in your config file.
By default, atoms in DI cannot communicate with each other due to an implicit "deny all" firewall. Communication between atoms must therefore be explicitly permitted by the connect keyword. This setup makes it easy to monitor and manage data access in your system.
labels are logical groups of atoms and/or other labels. With labels you can create named logical units within your system that you can then connect and manage. For instance, after allowing connections from label1 to label2, any container in label1 can connect to atoms in label2 by opening a socket to the hostname label2.di. If multiple atoms implement label2, DI will automatically load balance new connections across all available atoms.
Before you start DI, you must specify the structure and requirements for your distributed system in the config file. This file is written in the declarative DI domain-specific language (dsl) that allows you to describe your system in an clear and abstract manner. You might for instance specify the cloud provider(s), the number of VMs, how many containers you want, what kind of containers, logical groups (labels), as well as connections between labels and/or the public internet.
As an example, to boot 3 docker containers with the latest Ubuntu image and a postgres database, you put the following commands in your config file.
(label "containers" (makeList 3 (docker "ubuntu")))
(label "database" (docker "postgres"))
After this you will have a simple network:
We can easily expand our network to a basic deployment structure with the above database, a batch processing system and 5 Apache containers. As noted above, none of the containers can communicate by default, so we will furthermore open a few meaningful connections.
// Create 5 Apache containers, and label them "webTier"
(label "webTier" (makeList 5 (docker "httpd")))
// Create 2 Spark containers, and label them "batch"
(label "batch" (makeList 2 (docker "spark")))
// A deployment consists of a database, a webTier, and a batch processing
(label "deployment" (list "database" "webTier" "batch"))
// Allow the public internet to connect to the webTier over port 80
(connect 80 "public" "webTier")
// Allow the webTier to connect to the database on port 1433
(connect 1433 "webTier" "database")
// Allow the batch processer to connect to the database on and the webTier via SSH
(connect 22 "batch" (list "webTier" "database"))
// Allow all containers in the webTier to connect to each other on any port
(connect (list 0 65535) "webTier" "webTier")
After the above commands, our network looks a lot more interesting:
With the config file in place, DI will now boot your system. If you modify the configuration after the system booted, DI will make the corresponding changes to your system in the least distruptive way possible.
If you are interested in contributing to DI, check out dev.md for development instructions, details about the code structure, and more.