tengel/papyri
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
papyri/md/grub2_with_bls.md
tengel 8a1f38d024 update grub2_with_bls.md
2024-03-20 11:40:22 -05:00

16 KiB
Raw Permalink Blame History

GRUB2 with BLS

BLS Overview

Starting with EL8/OL8 (RHEL, CentOS, OracleLinux) the GRUB2 boot design is using BootLoaderSpec (BLS):

This article goes over the design and usage of the BLS environment as delivered by Red Hat with RHEL 8. Generally speaking, the non-UEFI and UEFI configurations are now the same content, just located in different directories for UEFI boot needs. Previously with EL7 (non-BLS) the GRUB2 config files were different content as well as locations on UEFI devices.

Design Overview

The EL8 BootLoaderSpec design takes the traditional single-file GRUB2 approach and extracts each kernel's configuration to it's own unique file per kernel, located in /boot/loader/entries.

  • The master file /etc/default/grub is still used to change the kernel parameters via GRUB_CMDLINE_LINUX
  • The /etc/default/grub file introduces a new setting GRUB_ENABLE_BLSCFG=true (default)
  • Running grub2-mkconfig will now update /boot/grub2/grubenv even if your output is temporary file! See the new --no-grubenv-update option to grub2-mkconfig
  • The operational problem with grubby and grub2-mkconfig generating different text strings for the kernels should be gone

Unlike EL7 where grubby use was disassociated with using /etc/default/grub and grub2-mkconfig output, the new design with BLS configuration is deeply embedded and using grub2-mkconfig is now the preferred usage after editing /etc/default/grub such that it will update the environment boot block with the new settings. This is a new change to use the environment boot block /boot/grub2/grubenv for critical needs; previously it was simply used to store the name of the kernel to boot, it now stores that plus the kernel options and a few other assorted key=value settings.

Changing Kernel Parameters

Red Hat has published a new document outlining the various methods used to update the kernel configurations while BLS is active (the default with EL8) which should be considered the authoritative document:

This article will present deeper operational use using the tooling provided; the GRUB2 packages provide a large number of commandline tools not covered in the above document explicitly. In general, on a RHEL type system (CentOS, OL, etc.) simply typing grub2-<tab><tab> will reveal the bevy of tools present.

If any manual / custom kernel option edits were made to the grubenv boot block file using grub2-editenv, they will get overwritten by the values listed in /etc/default/grub when grub2-mkconfig is next run. Ensure that any manual usage of grub2-editenv is followed up with an edit to /etc/default/grub as well.

Operational Use

The most common use is to adjust - add, remove, or update - a kernel parameter to boot the system.

  • Update /etc/default/grub with new edits to GRUB_CMDLINE_LINUX
  • Run grub2-mkconfig -o /boot/grub2/grub.cfg (non-UEFI) or grub2-mkconfig -o /boot/efi/EFi/redhat/grub.cfg (UEFI device)
    • "redhat" will be "centos" or "oracle" (the vendor name) of the RHEL-clone on UEFI devices
  • Verify changes with grub2-editenv list

Example: we will add "nomodeset" to the existing configuration to a non-UEFI server; a UEFI server simply names a different config file, the work is identical.

### Before without nomodeset ###

# grep ^GRUB_CMDLINE_LINUX /etc/default/grub
GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/rhel_rh8rc2--100-swap rd.lvm.lv=rhel_rh8rc2-100/root rd.lvm.lv=rhel_rh8rc2-100/swap rdblacklist=bfa,lpfc"

# grub2-editenv list | grep ^kernelopts
kernelopts=root=/dev/mapper/rhel_rh8rc2--100-root ro crashkernel=auto resume=/dev/mapper/rhel_rh8rc2--100-swap rd.lvm.lv=rhel_rh8rc2-100/root rd.lvm.lv=rhel_rh8rc2-100/swap rdblacklist=bfa,lpfc

### Changing to add nomodeset ###

# grep ^GRUB_CMDLINE_LINUX /etc/default/grub
GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/rhel_rh8rc2--100-swap rd.lvm.lv=rhel_rh8rc2-100/root rd.lvm.lv=rhel_rh8rc2-100/swap rdblacklist=bfa,lpfc nomodeset"

# cp -a /boot/grub2/grub.cfg{,.bak}
# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
done
# diff -uN /boot/grub2/grub.cfg.bak /boot/grub2/grub.cfg
(no output)

# grub2-editenv list | grep ^kernelopts
kernelopts=root=/dev/mapper/rhel_rh8rc2--100-root ro crashkernel=auto resume=/dev/mapper/rhel_rh8rc2--100-swap rd.lvm.lv=rhel_rh8rc2-100/root rd.lvm.lv=rhel_rh8rc2-100/swap rdblacklist=bfa,lpfc nomodeset

### Grepping the raw binary boot block file ###

# grep ^kernelopts /boot/grub2/grubenv
kernelopts=root=/dev/mapper/rhel_rh8rc2--100-root ro crashkernel=auto resume=/dev/mapper/rhel_rh8rc2--100-swap rd.lvm.lv=rhel_rh8rc2-100/root rd.lvm.lv=rhel_rh8rc2-100/swap rdblacklist=bfa,lpfc nomodeset

You will notice in the above example the old and new configuration files are identical (the diff shows no output from the backup). As the unique kernel configurations are now stored in separate files, the grub2-mkconfig command has no changes to this master file, however it is instrumental in updating the /boot/grub2/grubenv environment boot block binary file with the new options.

Boot Time Editing

The config files on disk have a variable embedded ($kernelopts) which GRUB2 reads in from the environment boot block during it's boot initialization. When at the boot time GRUB2 menu, choosing (E)dit to bring up the kernel options is expanded into it's final form and ready to edit as usual, no extra work is required. Due to the new design, the options are all left-aligned on screen and generally easier to read and alter, but otherwise the user experience is identical to previous releases.

BLS Kernel Configuration File

Each kernel gets a unique file in /boot/loader/entries named (/etc/machine-id)-(uname -r).conf - the file contains variable references to data which will be constructed dynamically at boot time by reading these values out of the /boot/grub2/grubenv environment boot block.

# cat /boot/loader/entries/$(cat /etc/machine-id)-$(uname -r).conf

title Red Hat Enterprise Linux (4.18.0-80.el8.x86_64) 8.0 (Ootpa)
version 4.18.0-80.el8.x86_64
linux /vmlinuz-4.18.0-80.el8.x86_64
initrd /initramfs-4.18.0-80.el8.x86_64.img $tuned_initrd
options $kernelopts $tuned_params
id rhel-20190313123447-4.18.0-80.el8.x86_64
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel

As you may notice, the options line is where the variable substitutions will take place. It is possible to create unique boot options for this kernel by editing this file as needed, which is generally equivalent in the older EL7 design of editing the menuentry block in /boot/grub2/grub.cfg (as grubby does by design).

File Locations

The location and purpose of various files within the GRUB2 BLS design. For UEFI based machines, "EFI/redhat" will be "EFI/centos" or "EFI/oracle" instead.

File Purpose and Usage
/bin/kernel-install The script called at the end of a kernel-core RPM install/upgrade to add the new kernel
/boot/efi/EFI/redhat/grub.cfg The UEFI specific version of the GRUB2 configuration file
/boot/efi/EFI/redhat/grubenv The UEFI specific GRUB Environment Boot Block, a 1024 byte binary file, storing the kernel parameters
/boot/grub2/grub.cfg The generic non-UEFI GRUB2 file which reads in the BLS configs dynamically at boot
/boot/grub2/grubenv The generic non-UEFI GRUB Environment Boot Block, a 1024 byte binary file, storing the kernel parameters. On a UEFI system, this should be a symlink to the UEFI specific version listed below
/boot/loader/entries/ The directory where each kernel BLS config file lives. The format of the filenames are (machine-id)-(uname -r).conf
/etc/default/grub The kernel commandline parameters are configured here for all kernels
/etc/machine-id The unique ID of the server - much like a UUID - which is used for the BLS filename for each kernel

The name of each kernel's unique BLS config file under /boot/loader/entries/ is using the above defined format, example:

# ls /boot/loader/entries/$(cat /etc/machine-id)-$(uname -r).conf
/boot/loader/entries/91cb597eca364d6b9fcdccd29b2cb938-4.18.0-80.el8.x86_64.conf

The Dracut Rescue image has it's own config file without a kernel version, (/etc/machine-id)-0-rescue.conf.

Plugin Directories

The BLS plugin directories using the natural sort ordering of number-prefixed names (20-grub.install e.g.):

Directory Purpose and Usage
/usr/lib/kernel/install.d/ System level drop-in directory for packages to place scripts to run; pre-populated by scripts from the grub2-common, tuned and systemd-udev packages amongst others.
/etc/kernel/install.d/ Userspace overrides for the system level drop-ins, and location of additional BLS scripts to run.

The scripts are coalesced and sorted before being run; Red Hat delivers 2x 0-byte override files as part of the grub2-common package; the system level scripts have builtin checks which should prevent them from running, however it appears Red Hat is ensuring they do not get run by simply placing dummy files with the same name in /etc/ to make sure.

# ls -l /etc/kernel/install.d/
-rw-r--r--. 1 root root 0 Dec 19 19:00 20-grubby.install
-rw-r--r--. 1 root root 0 Dec 19 19:00 90-loaderentry.install

# rpm -qf /etc/kernel/install.d/*.install | sort -u
grub2-common-2.02-66.el8.noarch

Vendor Provided Scripts

The default installation provides a handful of drop-in scripts; logistically, these are the core makeup of the work being done. They are collected from /etc/ and /usr/ (and a local ".install" subdirectory, placed into an array then sorted and de-duped by the parent script called at the end of the RPM install/upgrade. Each script is then run with a set of arguments in a standard order - new kernel version, working directory, and path to the compressed kernel image (vmlinuz).

The compressed kernel image has been moved! It is no longer in /boot/vmlinuz-(uname -r) direct from the RPM - the kernel image is now located in the same directory as the kernel modules, /lib/modules/(uname -r)/vmlinuz . The 20-grub.install plugin script below copies the kernel image to /boot/ as part of it's process work when called to add the new kernel by /bin/kernel-install.

A general list of the critical files and their use as drop-in scripts from a default installation:

File Package Purpose and Usage
/bin/kernel-install systemd-udev Called at the end of a kernel RPM install/upgrade, like so: /bin/kernel-install add 4.18.0-80.el8.x86_64 /lib/modules/4.18.0-80.el8.x86_64/vmlinuz Internally, after gathering the scripts available it runs each script below in order, like so: "$f" add "$KERNEL_VERSION" "$BOOT_DIR_ABS" "$KERNEL_IMAGE" ...where BOOT_DIR_ABS is an internal working directory it has calculated dynamically. This script has a built in special behaviour - if any one of the below scripts exits with the magic exit code 77, it will abort running any other scripts.
/etc/grub.d/10_linux grub2-tools The traditional menu builder script has been updated to trigger an internal GRUB2 module for reading BLS, instead of building the kernel menus itself. The script now writes insmod blscfg; blscfg ...to /boot/grub2/grub.conf where the kernel menu entries used to reside.

Path: /usr/lib/kernel/install.d/

File Package Purpose and Usage
20-grubby.install systemd-udev The script as used by previous GRUB2 work and the /sbin/new-kernel-pkg script from older versions of EL7. This script self short-circuits on it's own due to the lack of /sbin/new-kernel-pkg being present on disk. Additionally circumvented by Red Hat design (see below).
20-grub.install grub2-common This is the core script which builds the new configuration file in /boot/loader/entries/ for the new kernel being installed. It then copies a handful of the kernel files (vmlinuz, config, System.map, etc.) out of the RPM packaged /lib/modules/(uname -r)/ directory over to /boot/. Finally, it runs the grub2-editenv command to set saved_entry in the GRUB environment boot block (grubenv) to the new kernel to boot.
50-depmod.install systemd-udev Builds the standard /lib/modules/(uname -r)/modules.dep file (kernel module cross dependencies, "depmod -a") as with any kernel module updates.
50-dracut.install dracut This builds the initramfs / initrd file for the newly installed kernel. Of note, if the (non default, missing by default) files /etc/kernel/cmdline and /usr/lib/kernel/cmdline exist they will be read into scope. It will then read in /proc/cmdline from the current boot if those are missing - notice it is not reading the /etc/default/grub values, it's using what has been booted at that moment.
51-dracut-rescue.install dracut-config-rescue Sets up the standard rescue image and initrd in /boot/, generally ignored or disabled as desired as this kernel is mostly out of date as time-since-install progresses.
90-loaderentry.install systemd-udev This is the script which is designed to build the BLS kernel configurations in /boot/loader/entries/ as designed by the systemd-udev package. Red Hat's design nullifies this configuration plugin by shipping a 0-byte override (see below), and using the 20-grub.install plugin as described above to achieve the same goal.
92-tuned.install tuned Updates the values of any custom tuned-provided values to the BLS entries; examining the BLS configuration you will notice the variable $tuned_params on the "options" line, right after the variable for the $kernelopts. This is not stored in the GRUB boot block! (grubenv) – it's an inline sed of the BLS config file directly.
99-grub-mkconfig.install grub2-common This script is only used on PP64 based hardware, it's purpose is generally that of 20-grub.install but trivially designed for non-BLS configurations. It will short-circuit itself if not running on PP64 based hardware and exit.

Path: /etc/kernel/install.d/

File Package Purpose and Usage
20-grubby.install grub2-common 0-byte file designed to cicumvent the default script provided by systemd-udev
90-loaderentry.install grub2-common 0-byte file designed to cicumvent the default script provided by systemd-udev

References