building LFS in a qemu virtual machine

Dec 28, 2022 linux

Table Of Contents

Happy valentines day. Linux From Scratch (LFS) is a series of projects showing you how to build your own linux system. You start with a machine running a regular linux distribution you can use (the “host”) and at the end you reboot that machine into your custom linux install instead of whatever you started with. This post goes over how and why you might want to use a virtual machine as a host.

why LFS?

So right from the start, should you bother with LFS at all? This project is not “Design your SaaS onboarding funnel for better conversions in 30 days.” The artifact you’ll have at the end will only be useful to you, and the only goal should be to learn. More explicitly:

The linux platform is ubiquitous if your career or side projects are tech or tech-adjacent. If you are a software engineer, you likely develop on and deploy to some iteration of a linux system. And even if you don’t take away anything you should do differently, I firmly believe it’s never a disadvantage to know more about the system your code runs on.

Also it’s pretty fun to mess around with it. Ultimately, if it’s still exciting to you, don’t listen to what anyone says – try it. If it’s not, don’t. Life is too short for homework.

why use a VM (or QEMU specifically)

I don’t have a compelling reason for QEMU over any other virtualization platform, except that it’s simple and has all the functionality we need right out of the default installation.

Using a virtual machine is advantageous for three reasons:

  1. easy backups
  2. safer for your actual laptop
  3. avoid cluttering your laptop’s file system

Going forward I’ll refer to the actual machine you’re typing on as your “laptop”, because to be more precise I’d need to say something like your “hosts-host” and that’s unwieldy. The virtual machine “guest” is going to be our LFS “host.”

backups

Before we get into the LFS process, understand that your virtual machine will be created using a large file which QEMU treats as its file system and disk image. It’s just a file.

# create a new 4G image
qemu-img create -f qcow2 base.img 4G

# if you want to create a raw file, you can pass -f raw instead
# or use dd
dd if=/dev/zero of=/path/to/base.raw bs=1 count=0 seek=4G

# or even better, use fallocate
fallocate -l 4G base.raw

You can use the cp command for easy backups. But we can do even better. QEMU enables “overlay” images, which allow you to create your base image and then save mutations to a totally separate file. If you need to revert back, simply start your VM with the base image instead.

And proceed to try your potentially machine-borking experiment using the overlay image.

Here’s how you could create a chain of images to use as virtual “checkpoints” to revert to.

# create an experimental image based on base.img
qemu-img create -o backing_file=base.img,backing_fmt=qcow2 -f qcow experiment1.cow

# create an extension of your first experiment based
# on the one you just created
qemu-img create -o backing_file=experiment1.cow,backing_fmt=qcow2 -f qcow experiment1-a.cow

I’d recommend keeping this workflow fairly linear, but if you ever got confused about the chain of base images you’ve created, qemu will tell you with the --backing-chain command:

qemu-img info --backing-chain experiment1-a.cow

I think the most complicated chain I ended up with looked like this.

B a s e i m a g e ( u b u n t u . i m g ) C h e c k p o i n t # 1 ( c h r o o t . i m g ) B A B A o t o t o t o t t e t e l m l m o p o p a t a t d d e # e # r 1 r 2

which represents a workflow of making my base ubuntu image, setting up a working chroot and saving that overlay, and then trying the risky step of overwriting my bootloader. Attempt #1 was discarded and work resumed on the overlay created for Attempt #2.

safety

It goes without saying, but the laptop you’re planning to use for LFS is unlikely to be one you’d be comfortable bricking, but that’s exactly the sort of thing that can happen when you go overwriting your bootloader, or any other damn stupid thing. (There are many bolded warnings in the book before executing any dangerous steps.) If you’re using a virtual image, you can be reasonably sure the worst that can happen is you’ll need to start over from scratch. Which in this case means using your working, functional laptop to create another QEMU image and using the “back” button in your browser to rewind your progress through the book.

clutter

I personally don’t like to mess with my filesystems too much after I install linux on a machine. LFS requires creating a separate partition, which you will format, mount, and develop on. And this is how you start “from scratch”. Since disk space is cheap these days, I find it cleaner to create the virtual filesystem in the QEMU image and partition that. It uses all the same tooling.

how

I chose to use Ubuntu as my host image, since the install process if fairly simple and the tooling has historically been stable “out of the box.” My laptop itself is running arch linux, and that’s my own choice to live with, but using the stable Ubuntu distro as a starting point for LFS allows me to write this post from a starting point everyone else can reproduce. (Arch is rolling release which means there’s no “stable” distribution for us all to start on.)

install ubuntu

We’re going to download an Ubuntu desktop image, create the QEMU disk image for our host, and then boot the machine in such a way that QEMU lets us pretend a CD containing that image is in the cd-rom drive. You don’t need to actually put it on a CD, I don’t even have a cd-rom drive.

Download the image from ubuntu.com/download/desktop.

# create the host image
qemu-img create -f qcow2 ubuntu.img 50G

# boot it
qemu-system-x86_64 -hda ubuntu.img \
  -boot d \
  -cdrom /home/chris/Downloads/ubuntu-22.04.1-desktop-amd64.iso \
  -m 8G \    # give it 8G of memory, more than enough
  -accel kvm # makes everything faster if your laptop supports it

Now you should see an interactive window with the iso boot screen on it. Follow the instructions to install Ubuntu on your image file. You can basically use all the defaults.

Tip: the QEMU window will steal your mouse focus once you’re mousing around inside the Ubuntu desktop environment. Use ctrl+alt+G to get it back.

Once you have an installed system, I like to also install ssh and set that up, so that I don’t need to use the terminal inside the desktop environment to interact with it. Personal preference, but here’s how to do that:

sudo apt-get install openssh-server
sudo systemctl restart openssh

run the vm

Now at any time you can run your VM using the following command:

 qemu-system-x86_64 -hda qemu/ubuntu.img -boot d -m 8G -accel kvm -nic user,hostfwd=tcp::60022-:22

Notice that I forwarded port 60022 to port 60022 on my laptop to port 22 on the VM. That way we don’t need to worry about what IP gets assigned to the host, we can access the machine with:

ssh root@127.0.0.1 -p 60022

and as a bonus you can use whatever terminal you like on your laptop instead of installing and configuring one on your host.

Really only made this its own section because it’s a handy command to have around. I’d put it in a file called run-vm.sh, and use that from now on.

LFS has you run a version check script first to check for prerequisite software. Here’s what I had to install for all the compilers and libraries etc.

sudo apt-get install build-essentials bison gawk texi2html texinfo

resize your partition

If you followed all the default settings, Ubuntu should be taking up basically your entire 50 Gigabyte image. This is great for normal usage, but installing LFS is abnormal. We need to create another partition for LFS, and to do that we need to resize the existing one.

You can’t resize a partition which is mounted, so boot the image up again like before with the Ubuntu ISO (in section install ubuntu), and select the option to “Try Ubuntu,” without installing it. From there you can use the disk utility to resize the existing partition and create a new one. I made my new one 20GB.

You could also just expand the image size you created originally. If you opt with this route, make the initial one <50G. That’s way too much space. Then you can run

qemu-img resize disk_image +20G

with the same result.

installing linux from scratch

This last section will just be notes I had from going through the book. By now you’re well equipped to begin installing LFS using the QEMU virtual host.

I installed the (current) stable Linux From Scratch v11.2.

the process

At this point, you can continue with the book. In general, the process breaks down into three major parts:

Install basic toolchain on your new partition: you’re logged into your host, with the new partition formatted and mounted on /mnt/lfs, and you’re using tools on your host to compile an isolated toolchain without dependencies on the host itself. This is also known as “cross-compilation.” You make use of the --prefix flag to install the tools into /mnt/lfs/tools (instead of where you’d normally find these utilities), and the --with-sysroot flag to tell the compiler and linkers to look at your (currently empty) /mnt/lfs directory for its dependencies. You also use the --target flag to designate the machine’s target triplet of the form machine-vendor-os or cpu-vendor-kernel-os.

Compile basic utilities with the cross-toolchain you just built: you’re compiling basic utilities and installing them into their final locations. The compiler/linkers you’re using in this step are the ones you built in the last step, but the system isn’t ready for even basic use yet, and you still need to operate within the environment of your host system.

Thoughout these two sections (each is comprised of several chapters) you’ll use a $LFS environment variable to avoid typing out /mnt/lfs every time you need it, and if you set it wrong you run the risk of overwriting critical system tools and bricking your box. But we don’t!

Enter the chroot: you can now use your built system as it if it was the actual root of its own installation. Or you can change the root, in other words. You’ll use the chroot (change root) command to enter this environment. The command will look something like this (run with your host’s root user):

chroot "$LFS" /usr/bin/env -i \
  HOME=/root \
  TERM="$TERM" \
  PS1='(lfs chroot) \u:\w\$ ' \
  PATH=/usr/bin:/usr/sbin \
  /bin/bash --login

From within the chroot you can build out the rest of the system. The last steps involve configuring these tools, overwriting your bootloader, and managing services.

rebooting your host

At some point you might reboot your host (or reboot your laptop, which will terminate all processes including QEMU running your host). Once you’ve run your ./run-vm.sh script to restart the host, you’ll need to ssh into the host and follow these steps to get back to work on LFS:

mount the lfs partition

For me this is the following command, but make a note of how it might differ for you. For example, maybe you used another format, or the partition wasn’t on sda6.

sudo mount -v -t ext4 /dev/sda6 $LFS

mount the virtual filesystems

We use bind mounts to persist the filesystem structure within LFS. This step is explained in chapter 7.3, but helpful to write it down somewhere:

sudo su # be root
mount -v --bind /dev $LFS/dev
mount -v --bind /dev/pts $LFS/dev/pts
mount -vt proc proc $LFS/proc
mount -vt sysfs sysfs $LFS/sys
mount -vt tmpfs tmpfs $LFS/run

Now you can enter the chroot like normal.

gcc first-pass fix

In LFS Chapter 5.3.1 we build our first pass of the GNU Compiler Collection v12.2.0. It includes instructions for a number of dependencies you’ll need to build first, and I found that I needed one more.

I also needed ISL, which can be downloaded from here: https://libisl.sourceforge.io/isl-0.24.tar.gz. Extract it into the GCC directory just like the other dependencies, and install it in the same manner, and then you should be able to proceed.

gcc final pass testing

In LFS Chapter 8.26.1 we install GCC from the chroot. When installing the final version of these crucial system utilities it’s nice to run the tests, and the book guides you through this. I ran into an error for this one:

su tester -c "PATH=$PATH make -k check"
su: Cannot drop the controlling terminal

Running /bin/bash gives us some hints as to what’s wrong:

su tester -s /bin/bash
Cannot execute /bin/bash: Permission denied

The permissions of bash are correct (755):

ls -latr /bin/bash
-rwxr-xr-x 1 root root 948624 Nov 24 2022 /bin/bash

But the permissions for the root directory were off:

ls -ld /
drwx------ 20 1001 1000 4096 Nov 24 16:44 /

The fix is to set the permissions of the root directory to the same:

chmod 755 /

configuring the network

In LFS Chapter 9.5.1 we are configuring the network settings for our new installation. So the context here is that you’ve rebooted into your new installation of LFS, and even though we’ve had flawless network access through our host Ubuntu install, LFS knows nothing about what interfaces are available on the machine and what IPs they’re listening on.

At first I was configuring inside the file /etc/sysconfig/ifconfig.eth0. Running

dmesg | grep eth0

reveals that our network interface has been renamed to enp0s3. So rename the config file:

mv /etc/sysconfig/ifconfig.eth0 /etc/sysconfig/ifconfig.enp0s3

According to the QEMU networking docs, our IP address and default gateway can be configured as follows:

ONBOOT=yes
IFACE=enp0s3
SERVICE=ipv4-static
IP=10.0.2.15
GATEWAY=10.0.2.2
PREFIX=24
BROADCAST=10.0.2.255