Use the aarch64 templates if the efi architecture was set to AA64.
NB: since we only support efi booting for aarch64 this should be
good enough for aarch64 selection.
The code of the `org.osbuild.kickstart` stage already supported
adding the `--remote` option for `ostreesetup` via the `remote`
option but it was not included in the schema.
Add an optional `remote` to the deployment of the ostree so that
the it is tied to the specified remote. This is needed later for
updating the commit from that remote.
Greenboot is the idea of automatically rolling back bad updates,
i.e. updates that do not boot successfully. The implementation
is split between the boot loader and a user space component.
The latter sets two variables `boot_counter`, which indicates
the maximum number of boot attempts and `boot_success` which
tells the boot laoder if a previous boot was successful. The
bootloader on the other hand will decrement the counter variable
and reset the success indicator one.
An implementation of the user space component for rpm-ostree is
called `greenboot`.
The `refs` variable is used as parameter to indicate which commits
to checkout, but it also was used as result variable to store the
ids of the commits that actually got checked out; naturally, it
was initialized to the empty array. This of course meant nothing
was ever actually checked out. Doh.
Fedora 34 and thus RHEL 9 have adopted the unified grub config
scheme[1], where the main config is always placed in the same
location across all platforms, i.e. `boot/grub2`, and a stub
config that redirects to the main config is placed into the ESP.
osbuild has always done that in the case of hybrid boot, but not
for pure EFI systems. The new `uefi.unified` config option can
be used to select that new unified scheme even for the case of
pure EFI systems (aarch64 or non-hybrid boot).
Add a simple test for the grub stage.
[1] https://fedoraproject.org/wiki/Changes/UnifyGrubConfig
Previously:
We used images built from pre-mass-rebuild composes but installed packages
from post-mass-rebuild composes. This caused weird stuff like sshd crashing
when installing non-related packages via dnf.
Now:
Both the image and repositories are post-mass-rebuild ones. This should solve
these weird issues.
This stage takes /usr/lib/passwd and /usr/etc/passwd from an OSTree
checkout, merges them into one file, and store it as /etc/passwd in the
buildroot.
It does the same for /etc/group.
The reason for doing this is that there is an issue with unstable UIDs
and GIDs when creating OSTree commits from scratch. When there is a
package that creates a system user or a system group, it can change the
UID and GID of users and groups that are created later.
This is not a problem in traditional deployments because already created
users and groups never change their UIDs and GIDs, but with OSTree we
recreate the files from scratch and then replace the previous one so it
can actually change.
By copying the files to the build root before doing any other
operations, we can make sure that the UIDs and GIDs of already existing
users and groups won't change.
Co-author: Christian Kellner <christian@kellner.me>
New input type that takes a ostree repo and checks out any number
of commits inside that repo to a temporary directory. Each commit
will be checked out to a separate sub-directory. The name of the
dir is the commit id of the corresponding commit.
This input can thus be used to access files and directories of
commits in stages.
Include the packit tool to be able to do updates from the container.
Additionally, include python-boto3 so that `make test-src` now does
not complain about missing `boto3` imports.
Also include ipython3 for quick python code snippet testing and
lsof to debug processes that keep files open.
NB: for packit to work you have to have the right tokens in your
local packit configration file (see the upstream doc). Additionally,
`packit propose-downstream`, to create PRs for a new downstream
release, needs kerberos auth to work correctly.
Instead of using the version specific, pre-depsolved f34 build manifest,
use the new version agnostic build manifest (fedora-build.mpp). NB: this
is included directly as mpp so that its variables get defined by the
including manifest. This should make it even easier to update manifests
to new fedora releases.
Include a build manifest that is itself not have tied to a specified
version and thus is meant to be included with the following vars
pre-defined as .mpp file:
- arch architecture (x86_64)
- releasever release version (f34)
- snapshot rpmrepo snapshot (20210326)
That ironically fix the underlying bug that flush_buf is trying to
fix too, since now an exception is thrown and we are back to auto
clear. The file fd is then closed when the process is terminated.
Anyway, the right fix is to call the correct function.
Manually clear the buffer cache of the loop device, which seems to
be required in order to make sure that data written via the loop
device is actually landing in the file:
Since commit c1379f6 the file descriptor of the loop device is
explicitly cleared. This broke manifests that involved creating a
FAT filesystem. Said file system could later not be mounted. The
breaking change was identified to indeed be commit c1379f6. Using
`biosnoop` we saw that some write operations were missing when
clearing the file descriptor that were present when using the
auto-clearing feature of the loop device (see below). Reading the
corresponding kernel source (v5.13.8), the current theory is that
when using the auto clear feature, once the last handle on the
loop device is closed, the code path in the kernel is:
blkdev_close (fs/block_dev.c)
blkdev_put (fs/block_dev.c)
__blkdev_put (fs/block_dev.c)
sync_blockdev (fs/block_dev.c)
On the other hand when manually clearing the file descriptor, the
code path seems to be:
loop_clr_fd (fs/loop.c)
__loop_clr_fd (fs/loop.c)
The latter first removes the backing file and then calls `bdput`,
and thus no call to sync_blockdev is made.
Luckily, sync_blockdev can be called via an ioctl, `BLKFLSBUF`,
which we no do, via the new helper function `lo.flush_buf`. This
fixes the observed issue and leads to the same biosnoop trace as
observed when using the auto clear feature without explicitly
clearing the fd.
NB: we considered reverting the commit c1379f6, but we want to make
sure that we control to point when the backing file is cleared from
the fd, since sub-sequent osbuild stages will re-use the file and
we want to ensure no loop device still has the file open and that
all the data in is in the file.
-- biosnoop trace --
4.115946 mkfs.fat 731297 loop1 R 0 4096 0.08
4.116096 mkfs.fat 731297 loop1 R 8 4096 0.02
4.116176 mkfs.fat 731297 loop1 R 16 4096 0.02
[...]
4.120632 mkfs.fat 731297 loop1 R 400 4096 0.02
4.200354 org.osbuild.lo 731281 vda W 4182432 32768 0.64
4.200429 org.osbuild.lo 731281 vda W 6279584 32768 0.70
4.200657 ? 0 R 0 0 0.19
4.200946 org.osbuild.lo 731281 vda W 3328128 4096 0.20
4.201109 ? 0 R 0 0 0.13
[the following entires were missing with manual flushing:]
4.201601 org.osbuild.lo 731281 loop1 W 0 4096 0.24
4.201634 org.osbuild.lo 731281 loop1 W 8 4096 0.26
4.201645 org.osbuild.lo 731281 loop1 W 16 4096 0.27
[...]
4.203118 org.osbuild.lo 731281 loop1 W 432 4096 0.25
Reported-by: Achilleas Koutsou <achilleas@koutsou.net>
Reported-by: Tomas Hozza <thozza@redhat.com>
Add a simple check that data written through the loop device is
actually ending up in the file. NB: this this will _fail_ if the
fd is cleared via `clear_fd` without the use of `flush_buf`. It
seems that the kernel (as of 5.13.8) will indeed not clear the
buffer cache of the loop device if the backing file is detached
via `LOOP_CLR_FD`. On the other hand, if the autoclear flag is,
i.e. the backing file cleared when the last file descriptor of
the loop device is closed, the buffer cached will be cleared as
part of the `release` operation of the block device.
Add a helper method to call `ioctl(fd, BLK_IOC_FLUSH_BUFFER, 0)`
from python. NB: the ioctl number 0x1261 is wrong on at least
alpha and sparc. A later test will use this call so we should
catch the usage of it on those platforms.
Properly label the build root for the f34 build root manifest v2.
Also label the cp and tar binaries with `install_exec_t` so they
can read and copy labels unknown to the host.
Add a new stage that allows the modification of LVM2 metadata,
most importantly it allows for renaming of the volume group.
It internally uses the new `utils.lvm2` module.
This module provides a `Disk` class that can be used
to read in LVM images and explore and manipulate its
metadata directly, i.e. it reads and writes the data
and headers directly. This allows one to rename an
volume group without having to involve the kernel,
which does not like to have two active LVM volume
groups with the same name.
Normally `DeviceService.close` would be called when the connection
was closed on the other end. Expose that close method via RPC so
that clients can call it explicitly. This should be mostly useful
in testing.
Implement a new `lock` option (default: False), which will lock
the device by passing `lock=True` to `LoopControl.loop_for_fd`.
The main purpose for this is to block udev from probing the device
while the stage is run.
NB: some tools might also try to lock the device and fail.
Explicitly clear the fd. There might be a race with udev or some
other process that still as a reference to the loop device so we
might not be able to immediately clear it. Thus wait for it with
a timeout of 30 seconds which should hopefully be enough. If the
device does not clear then any consecutive action that involves
it might not be safe to execute so we let the timeout exception
be reported to osbuild.
Add support for locking the loopback block device via `flock(2)`.
The main use case for this is to prevent systemd-udevd from
proben the device while any modification is done to it. See the
systemd page, https://www.freedesktop.org/software/systemd, for
more details.
Add the corresponding tests to it.
