# Run Firecracker on DigitalOcean Apparently, DigitalOcean supports nested virtualization on their instances, so it’s worth checking how many Firecracker instances I can run. I’m starting with the cheapest option $5/mo running Ubuntu 20.04 LTS. Setup an instance: ```bash $ # copy vmlinux and rootfs.ext4 that we built in the last post to the instance $ apt-get update $ apt install golang make $ git clone https://github.com/firecracker-microvm/firectl $ cd firectl $ make $ cp firectl /usr/local/bin $ cd ../ $ mkdir firecracker $ cd firecraker $ wget https://github.com/firecracker-microvm/firecracker/releases/download/v0.24.2/firecracker-v0.24.2-x86_64.tgz $ tar -zxf firecracker-v0.24.2-x86_64.tgz $ cp firecracker-v0.24.2-x86_64 /usr/local/bin/firecracker $ cp jailer-v0.24.2-x86_64 /usr/local/bin/jailer $ firectl --kernel=./vmlinux --root-drive=./rootfs.ext4 --kernel-opts="console=ttyS0 noapic reboot=k panic=1 pci=off nomodules rw" ``` The first thing that I noticed is it takes between 2.4-2.9sec to start a Firecracker instance. Each instance takes about 60MiB RAM, so I should be able to launch around 13 virtual machines on the 5$/mo instance. Everything seems stable enough. I managed to run a simple Python application on each instance without any noticeable side-effects. I’m kinda torn that between Kubernetes cluster (which doesn’t work as expected in DigitalOcean and bare-metal offerings are quite expensive) and creating a home-grown basic orchestrator; it is likely going to be a rabbit’s hole, I need to research whether there’re any options I can use as an alternative. DigitalOcean provides premium instances that run on the second generation Intel Xeon processor, so it’s interesting to see if there’re any benefits of using them. KVM is not enabled (but it’s supported) on these instances by default, so I followed instructions from https://help.ubuntu.com/community/KVM/Installation to enable it. ```bash $ apt-get install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils $ adduser `id -un` libvirt $ adduser `id -un` kvm $ virsh list --all Id Name State -------------------- $ rmmod kvm $ modprobe -a kvm $ mkdir /dev/kvm $ chown root:libvirt /dev/kvm $ apt install cpu-checker $ kvm-ok INFO: /dev/kvm exists KVM acceleration can be used $ lsmod | grep -i kvm kvm 663552 0 $ ls /lib/modules/$(uname -r)/kernel/arch/x86/kvm/* /lib/modules/5.4.0-51-generic/kernel/arch/x86/kvm/kvm-amd.ko /lib/modules/5.4.0-51-generic/kernel/arch/x86/kvm/kvm.ko /lib/modules/5.4.0-51-generic/kernel/arch/x86/kvm/kvm-intel.ko $ modprobe kvm-intel modprobe: ERROR: could not insert 'kvm_intel': Input/output error ``` Ouch, `kvm_intel` is not enabled and cannot be loaded. Is it that nested virtualization is not enabled on these instances or is it a defect in the kernel it’s running? I’ll try another distro. Ubuntu 18.04 has an old go package, so before compiling `firectl` we need to install at least 1.16: ```bash $ add-apt-repository ppa:longsleep/golang-backports $ apt update $ apt install golang-go ``` Also, `kvm_intel` is loaded, and it looks promising: ```bash $ lsmod | grep -i kvm kvm_intel 217088 0 kvm 614400 1 kvm_intel irqbypass 16384 1 kvm ``` After I followed firecracker setup instructions, I was able to start an instance. I didn’t notice any visible performance improvements. I didn’t run an extensive test, so it could be that the devil is in the detail and CPU-wise, it could be faster. However, for now, my concern is boot time. As the last experiment for today, I tried another $60/mo general-purpose instance, which improved boot time to ~2sec. Perhaps, I need some ansible scripts to automate this process to validate other offerings.