My face as a computer, also a handy button to go back to the site root

Boot an ephemeral Linux VM in macOS with qemu

○ 9 min read

I work on macOS, but sometimes it's very helpful to tinker with a full-fat Linux environment. Booting up a whole virtual machine is going to be more heavyweight than running something like docker run --rm -it fedora , but as I host most of my personal projects on a boring server it can be useful to have a quick(ish) feedback loop for testing things like systemd and cron.

Many Linux distributions provide cloud images, which are generally comparable to their server versions but designed to boot without a manual install process. These types of images pair really well with qemu, an open-source system emulator that can use hardware virtualisation (macOS has the Hypervisor framework) to run disposable guest operating systems.

Let's make sure we have everything we need:

$ brew install qemu curl gnupg xorriso

We can then grab the current cloud image for Fedora, but you can replace this with basically any distro you want:

# I'm an old man using an x86 machine, amend to aarch64 if you're fancy
$ curl -O https://download.fedoraproject.org/pub/fedora/linux/releases/38/Cloud/x86_64/images/Fedora-Cloud-Base-38-1.6.x86_64.qcow2
$ curl -O https://download.fedoraproject.org/pub/fedora/linux/releases/38/Cloud/x86_64/images/Fedora-Cloud-38-1.6-x86_64-CHECKSUM
$ curl -O https://fedoraproject.org/fedora.gpg
$ gpgv --keyring ./fedora.gpg Fedora-Cloud-38-1.6-x86_64-CHECKSUM
$ sha256sum -c Fedora-Cloud-38-1.6-x86_64-CHECKSUM

And then run our shiny new Fedora-Cloud-38-1.6-x86_64.img using qemu:

$ qemu-system-x86_64 \
	-accel hvf \
	-cpu host \
	-m 1G \
	-drive if=virtio,file=Fedora-Cloud-Base-38-1.6.x86_64.qcow2 \
	-nic user,model=virtio-net-pci,hostfwd=tcp:127.0.0.1:2222-:22 \
	-snapshot \
	-nographic

Woah. That's a lot. What's going on here?

Your applications, for the most part, want to spend their time communicating with all the fabulous hardware you have in your computer - CPU, memory, network card, GPU, storage, etc. However, for many eminently sensible reasons, your software isn't allowed to speak directly to this hardware, and instead sends and receives a bunch of system calls through an operating system acting as a happy intermediary.

Virtualisation add another layer of indirection to all this, allowing you to run 'guest' operating systems which think they're talking directly with computer hardware but are instead talking to another operating system, known as the host. Many CPUs feature technology to allow this to all happen with maximum performance and minimum overhead, and contemporary machines can offer performance almost as good on many virtualisation workloads as running a single operating system communicating straight with the hardware.

The magic software that makes this work is called a hypervisor, and historically there's been chat about hypervisors that operate in place of a main operating system (known as Type 1) and hypervisors that operate on top of an existing operating system (or Type 2). We won't go into that here - instead, we'll just enjoy a broad, top-level visualisation of how a guest operating system loosely functions:

A hypervisor sits between a guest operating system and the host operating system

Qemu is quite low-level compared to other software in this space, and is perhaps more commonly used via higher-level abstractions like libvirt. Running it directly exposes you to many, many, many many many command line switches, which you will continually forget time and time again. But I think there's a big advantage here, for my relatively straightforward use case at least, in that it can be housed very nicely in a little stateless, portable shell script that you check-in to version control. Also, I've never had too much luck with libvirt on macOS. And, most wonderfully, it saves you having to learn another configuration format on top.

Here's what each of those flags means:

-accel hvf

Use the macOS hypervisor framework. Computer go fast!

-cpu host

This uses CPU host passthrough, which means we're not emulating a CPU in our guest environment. This should give us better performance than if we, say, wanted our VM to think our x86 processor was actually an aarch64 one.

-m 1G

This sets the amount of available memory to the guest OS. So while my machine has 16GB of memory, our Linux VM won't be able to use more than 1GB. I normally set this to as close to the production server as possible.

-drive if=virtio,file=Fedora-Cloud-Base-38-1.6.x86-64.qcow2

This mounts the Fedora image file we've downloaded as the filesystem for the VM. To do this we use VirtIO, another emulation framework but for IO devices, to mount the Fedora image. macOS takes care of all of this for us.

-nic user,model=virtio-net-pci,hostfwd=tcp:127.0.0.1:2222-:22

The -nic flag enables networking, once again using VirtIO. User networking in qemu defaults to a 10.0.2.0/24 address space, with the second IP in the network reserved as the address of the host: 10.0.2.2. We also use the hostfwd option to forward port 2222 on the host to port 22 on the guest, for SSH.

-snapshot

Writes to temporary files instead of the disk image, meaning everything gets nuked when the guest OS is stopped. This saves us having to make another qcow2 file.

-nographic

This disables graphical support, and instead pipes the machine's serial port to the terminal. The main gotcha here is that you have to press Ctrl + A, then X in order to quit. You will definitely forget this.

This is all working great, and now we get a login prompt for our machine-in-a-machine. Wait, a login? Uh oh:

Fedora Linux 38 (Cloud Edition)
Kernel 6.2.9-300.fc38.x86_64 on an x86_64 (ttyS0)

eth0: 10.0.2.15 fec0::7b27:4767:382e:dafc
localhost login:

This can be a tricky moment. How are you supposed to login? What's the password going to be? You can try fedora here, or maybe root - but unfortunately you're not getting past this screen (for now).

Instead of Googling the answer, we can be indulgent and poke around a bit first. Our image file is in a qcow2 format, which is the default for qemu. I don't know how to work directly with that on macOS, but luckily we also have qemu-img which can convert to a more common img format:

$ qemu-img convert -f qcow2 -O raw Fedora-Cloud-Base-38-1.6.x86_64.qcow2 fedora.img

The fedora.img file contains multiple partitions, one of which contains a standard Linux directory layout. If we mount that partition (aside: getting macOS to recognise ext4 and btrfs filesystems is a bit of a faff that I won't cover here) we can check out the filesystem:

# /etc/shadow contains information about a system's users
$ cat /etc/shadow | grep root
root:!locked::0:99999:7:::

As expected, the root user is locked (thanks to the !) and so we can't login with a password.

The next bit requires a bit of background/circular knowledge (you need to know where to check for cloud-init, which you wouldn't know to do unless you knew about cloud-init) but we can also check that there's a file at /etc/cloud/cloud.cfg:

$ cat /etc/cloud/cloud.cfg | wc -l
108

This suggests our Fedora Cloud distribution is indeed using a tool called cloud-init to handle its configuration, including users and groups. This isn't a surprise: it's pretty much the de facto standard for cloud images. The documentation for cloud-init suggests that each distribution sets up a default user, which we can check:

$ cat /etc/cloud/cloud.cfg | grep 'default_user:' -A 6
   default_user:
     name: fedora
     lock_passwd: True
     gecos: fedora Cloud User
     groups: [wheel, adm, systemd-journal]
     sudo: ["ALL=(ALL) NOPASSWD:ALL"]
     shell: /bin/bash

So, by the time we run this image in a VM, cloud-init has done its magic and we do have a fedora user - but it doesn't have a password specified by default. So we can't login (yet). We need to further configure our VM instance using cloud-init. The documentation has a pretty good tutorial to get up and running, which I'll mostly be recapping here.

In short: we need to feed it some YAML. Of course you do, because you can't configure something intended for the cloud without at least some YAML.

We'll setup a directory and the relevant files:

$ mkdir cloud-init && touch cloud-init/{user,meta,vendor}-data

And then populate our user-data file:

$ cat << EOF > ./cloud-init/user-data
#cloud-config
password: password
chpasswd:
  expire: False

EOF

I don't worry about the meta-data and vendor-data files, but having them exist (even if empty) will speed up the boot time of the VM.

The next step is to get the operating system aware of these files. With cloud-init's NoCloud data source, that's done by mounting a CIDATA volume or using a webserver. The latter is how the real cloud servers will operate, but that means having a webserver running in another tab and that feels like making more faff when it comes to scripting all this.

Instead, we can make a CD image with xorriso:

$ xorriso -outdev ./nocloud.iso \
  -joliet on \
  -blank as_needed \
  -map ./cloud-init/ / \
  -volid CIDATA

And return to qemu-system-x86_64, mounting our nocloud.iso as another drive:

$ qemu-system-x86_64 \
  -accel hvf \
  -cpu host \
  -m 1G \
  -drive if=virtio,file="Fedora-Cloud-Base-38-1.6.x86_64.qcow2" \
  -drive if=virtio,file="nocloud.iso",index=2,media=cdrom \
  -nic user,model=virtio-net-pci,hostfwd=tcp:127.0.0.1:2222-:22 \
  -snapshot \
  -nographic

Whew! That's a lot. But, once that machine boots, we can login as fedora and use our brand new password of password:

Fedora Linux 38 (Cloud Edition)
Kernel 6.2.9-300.fc38.x86_64 on an x86_64 (ttyS0)

eth0: 10.0.2.15 fec0::5962:c9cc:4f1f:45fb
localhost login: fedora
Password:
[fedora@localhost ~]$ hostnamectl
     Static hostname: localhost
           Icon name: computer-vm
             Chassis: vm 🖴
          Machine ID: d6416aa014fa49268ab214be6ee827d9
             Boot ID: e6b3da91a607484d8927910580cd9a6c
      Virtualization: qemu
    Operating System: Fedora Linux 38 (Cloud Edition)
         CPE OS Name: cpe:/o:fedoraproject:fedora:38
      OS Support End: Tue 2024-05-14
OS Support Remaining: 10month 3w 1d
              Kernel: Linux 6.2.9-300.fc38.x86_64
        Architecture: x86-64
     Hardware Vendor: QEMU
      Hardware Model: Standard PC _i440FX + PIIX, 1996_
    Firmware Version: rel-1.16.2-0-gea1b7a073390-prebuilt.qemu.org
       Firmware Date: Tue 2014-04-01

Perfect! From here there's a lot you can do that your regular containerised environment just can't reach, from exploring systemd, playing around with cgroups or poking around with some complicated iptables commands. It's also probably worth properly setting up SSH and customising a proper user - the documentation will set you off down that road.

Also: it's just cool to be able to spin up a Linux instance every now and then.