The Problem
There are quite a few websites out there that let you complete coding exercises, provide browser-based editors, and run the code for you once you’re done. The main difference between these and what I’m trying to create is that a challenge can be more than just a codebase and a testsuite.
Users should be able to create a challenge from a Docker container - an example would be a CTF (Capture the Flag) cybersecurity exercise.
There are some important considerations when implementing such a system:
- Virtual machines need to be isolated from other VMs as well as the host system.
- VMs shouldn’t be able to communicate with anything on the Internet that they don’t have reason to.
- The main reason for this is to eliminate the possibility of VMs being used for malicious purposes - e.g. as part of a botnet or similar. If the VM is limited as much as possible when it comes to networking this further reduces attack surface.
- VMs should be ready straight away! I think it’s important that once a user chooses to start an exercise that uses a VM, they get to interact with it straight away without having to wait for one to be provisioned. Obviously this isn’t realistic if we’re building an image every time, or provisioning a VM through a cloud provider for every user, so these are out of the window (and terribly inefficient/costly at scale).
In addition to that last point: as well as being quick to boot, VMs should be quick to destroy and cleanup.
Cleanup should also be reliable and repeatable with the same results every time - we wouldn’t want hundreds of dangling VMs staying around when they’re not being used.
The Options
My first thought was to use Docker containers. It made the most sense - users would submit VM-based challenges as a Docker image in the first place, so surely this was a no-brainer!
It turns out, though, that the level of isolation provided by containers isn’t as strong as a VM, due to the fact that there’s not as much separation between the running process and the host operating system as there would be in a virtual machine.
So how do we deploy full-blown virtual machines instantaneously while still retaining all the functionality we need, all from a Docker image?
Luckily, there’s a project developed by AWS that allows us to do exactly that - it’s called Firecracker.
Firecracker manages so-called microVMs using the Linux KVM, making optimisations to improve security and improve boot times. It’s used at scale in the real world for AWS Lambda and Fargate, so I think that at least warrants taking it into consideration for production use.
You can find the Firecracker repository here, I’d recommend having a look over the README as how it works is quite interesting.
Ignition
Right, so we know that it’s possible to start up virtual machines quickly and securely - but how do we actually make that happen from a Docker image?
Unfortunately, Firecracker uses kernel and filesystem (ext4) images, and although it is possible to create bootable disk images from a Docker container, this seems much too complicated and unreliable to use in a production environment.
Another open-source creation to the rescue, Weaveworks Ignite! Ignite is a layer on top of Firecracker that enables you to create microVMs from Docker images, as well as abstracting away a lot of the complicated network configuration required for setting up a microVM by default.
In the next part I’ll go over how I actually went about implementing my initial proof-of-concept system, so stay tuned for that.
Thanks for reading!
