VMine
VMine (in Italian, "tiny VMs") is a Virtual Machine Manager like libvirt, Xen, Ganeti etc, except a lot simpler and dumber. It is intended for a specific use case: managing ephemeral, short-lived virtual machines for use in systems integration testing, generally in the context of a Continuous Integration environment. It uses QEmu/KVM under the hood.
The service offers an HTTP API which allows you to create a group of VMs attached to a shared private network (so they can talk to each other), which is routed to the Internet (to download packages etc). The network parameters are selected, randomly, by the caller. Creating a VM group returns a unique ID that can be used to shut it down when you're done. That's it, that is the totality of the API.
You can choose a small number of runtime parameters for the virtual machines: RAM, CPU, and which base image to use. Images are defined in an image registry, which is just a YAML file with a list of named rootfs / kernel / initrd images. Since these are ephemeral VMs, the rootfs image will be mounted read-only, with changes going to a temporary file which is then discarded.
Running the vmine daemon
The vmine daemon needs to run as root (or at the very least it needs the CAP_NET_ADMIN, and possibly other, capabilities), as it will make changes to the network configuration of the host system, creating bridge and TAP interfaces and firewall rules. It does however make an effort to do this properly, with matched setup / teardown steps, so it will leave your network configuration as it found it, on graceful termination. The intention is for vmine to leak no resources on the long term, even in case of errors.
The only requirement is that the host system should have IPv4 forwarding enabled:
$ echo 1 > /proc/sys/net/ipv4/ip_forward
In order to start the daemon, you have to build it first. Clone this repository locally, then build the code (set GO111MODULE=on if you have a Go version older than 1.15):
$ go build ./cmd/vmine
The only required configuration is the image registry file, see the "VM Images" section below for a description of its format. Once you have that, start the daemon with:
$ sudo ./vmine --images=my-images.yml
This will start the daemon, with the HTTP API endpoint listening on port 4949.
VM lifecycle
All virtual machines in a group share the same fate: if one terminates, the entire group is turned down. They also all share the fate of the vmine daemon: when it is stopped or restarted, all groups are shut down.
All groups have a TTL (which by default is 1 hour if not explicitly specified), after which they are shut down automatically. This is to avoid resource leaks when clients "forget" to call the shutdown API.
The boot strategy implemented should allow VMs to become available within a few seconds after they've been started.
VM Images
VMine can't run on arbitrary VM images because it needs some cooperation from the VM in order to configure the network and other host parameters. The changes required are actually very simple and can be examined by looking at the aux/post-install.sh script. Fortunately it's pretty easy to build your own base VM images, see the "Building an image" section below.
Also, since we don't use the BIOS to boot, every image needs to be accompanied by its kernel / initrd image pair, stored separately from the image itself.
The image registry is just a YAML file containing pointers to the set of rootfs / kernel / initrd files for all available VM images. This is stored as a dictionary of name: paths pairs, along with an explicit indication of the default image to use when clients don't explicitly specify one.
A valid image registry file could look like this:
images:
buster:
path: "/images/buster/rootfs.img"
kernel_path: "/images/buster/vmlinuz-4.19.0-14-amd64"
initrd_path: "/images/buster/initrd.img-4.19.0-14-amd64"
bullseye:
path: "/images/bullseye/rootfs.img"
kernel_path: "/images/bullseye/vmlinuz-5.10.0-1-amd64"
initrd_path: "/images/bullseye/initrd.img-5.10.0-1-amd64"
default: "buster"
which defines two images, buster and bullseye, and sets the default to buster.
Download pre-built images
For convenience, we build Debian stable / nextstable images weekly with this project's CI. You can download these images from the project's Package Registry.
Building an image
VMine can only work with QCow2 rootfs images.
Simple steps for Debian-based distributions, using vmdb2:
$ IMG=/path/to/rootfs.img
$ sudo apt install vmdb2 lsof libguestfs-tools
$ sudo env DIST=bookworm ./sysimages/build.sh root.tar
The above will create a root.tar containing the rootfs and the kernel image, suitable to be used by vmine once decompressed.
If you want to use a caching proxy for the installation phase, use the http_proxy env var, with a http:// prefix (just host:port won't work), e.g.:
$ export http_proxy=http://127.0.0.1:3142
Note that this will not result in having a proxy configured in the resulting image, it will only be used to speed up debootstrap. You should use the AUTOPKGTEST_APT_PROXY env var for that (check out the autopkgtest-build-qemu manpage).
Image setup
The VM images generated with the above procedure are bare-bones Debian installations with no customization beyond network configuration.
There is no console access, you're expected to connect over SSH using a SSH key that you provide at the time of VM creation. The SSH public key for the root user can be set by specifying the ssh_key attribute in the create-group request.
The firewall configuration set up by vmine does not allow direct access from the public Internet to the virtual machines, as a consequence of using well-known authorization credentials. To connect to the virtual machines, you're expected to jump through the VM host.
Named groups
In certain circumstances it can be useful to have a way to consistently refer to the same VM group: for example, when integrating with CI systems that manage their own environment lifecycle such as Gitlab CI with the Environments feature.
In these cases, it's possible to use a name for a group (any non-empty string) when creating or deleting it. It will be silently hashed to a unique ID internally.
HTTP API
The HTTP API accepts POST requests with a JSON-encoded payload.
/api/create-group
Creates a new group of VMs, which will be assigned a unique ID.
Sample request:
{
"network": "10.12.13.0/24",
"ssh_key": "...",
"ttl": "1h",
"hosts": [
{
"name": "host1",
"ip": "10.12.13.100",
"image": "buster",
"ram": 4096,
"cpu": 1
},
...
]
}
Most parameters except for the network configuration can be omitted, and will use defaults. A configuration that fails to validate (because, for instance, a host's IP address is not contained in the specified network, or the network is already used by another VM group, etc.) will result in a 400 error.
To create a named group, specify the name attribute in the request.
A successful response will contain the unique ID of the newly created group, e.g.:
{
"group_id": "0fb1"
}
The ID should be treated as an opaque token.
/api/stop-group
Stops a running VM group, given its unique ID.
Sample request:
{
"group_id": "0fb1"
}
For a named group, it is possible to use the name attribute:
{
"name": "my-group-1"
}
An empty response is returned.
Quick Testing
Clone this repository locally, then build the code (set GO111MODULE=on if you have a Go version older than 1.15):
$ go build ./cmd/vmine
Build a VM image, for example a Debian stable (Buster) one, following the instructions in the "Building an image" section above. Then create a copy of images.yml and modify it by pointing it at the full paths for the image, the kernel, and the initrd you just generated:
$ cp images.yml my-images.yml
$ vi my-images.yml
Run the server as root:
$ sudo ./vmine --images=my-images.yml
To start a few VMs you can then make a HTTP request to the API using curl:
$ curl -X POST -d '{"network": "10.12.32.0/24", "hosts": [{"ip": "10.12.32.10"}, {"ip": "10.12.32.11"}]}' \
http://localhost:4949/api/create-group
Ctrl-C will shut down the server, stop any running VMs, and clear up any changes to the network and iptables configuration.
Implementation notes
All VMs are children of the main vmine process and are stopped when it exits. Centralizing process control allows us to clean up neatly all associated resources (network devices, firewall rules) when a VM exits.
Vmine allows a limited amount of run-time configuration, mostly limited to network configuration. We do this via kernel boot parameters, which are then parsed by a script that performs runtime setup at boot. This is one of the reasons (speed being the other) for booting the image with an external kernel, rather than via BIOS and GRUB (in that scenario we would need DHCP and some sort of network metadata service, AWS-style -- this is a lot simpler).