Instead of keeping the loop device of the base image and then opening
each partition as a loop device, remove the original loop device of the
base image and then create a loop device for each partition from the
file itself using the partition offsets.
The open_image() function is renamed to convert_image() and now only
handles converting qcow2 files to raw files if necessary.
The loop_open() context is done in analyse_image() instead, so that the
base loop device can be closed without removing the converted image.
This fixes the following issue with LVM partitions:
When the same lvm partition UUID is on two devices (e.g., /dev/loop0p4
and /dev/loop1), the 'vgchange -ay' command fails with the following
error:
Cannot activate LVs in VG rootvg while PVs appear on duplicate
devices.
This happens when we open the LVM partition as a separate loop device,
which we do for all partitions that we want to inspect.
NB: It's possible to restrict the vgchange command to a specific device
with --devices, but this isn't available in older versions of lvm2 (it
was introduced in 2.03.11).
Add support for reporting the install container images in an image.
NB: this does not use `podman` but reads the overlay storage
directly and therefore does currently not take additional image
locations or different storage drivers into account. For now this
is not a problem since we don't support any of that.
Check for the rpm database in $tree/usr/share/rpm and explicitly
also in $tree/var/lib/rpm and if the respective location exists
pass it as argument to rpm. This should fix the situation where
the default database on the host is in a different location than
in the tree. Fedora < 36 and RHEL have the location in /var but
Fedora starting 36 and rpm-ostree have it in /usr.
When encountering an LVM2 layout, activate all its logical volumes
so that they can be mounted.
NB: we need to pass "norecovery" to the mount options because LVM
does not setup the device mapper tables read-only even though the
underlying loopback device is and then xfs will try to write to
its journal and the kernel will panic. Attempts to reload the DM
tables as readonly didn't work.
NB: this will not work if we are trying to inspect an image that
has a volume group name that is also present on the host. We
could open the image file read-write and modify its vg name, but
that would mean modifying the image file and thus we would need
to copy it first.
Pass `-c /dev/null` to `blkid` to force it not to use its cache.
When iterating over partitions, only record the ones that have a file-
system and save them in a filesystem to device map. Then use that for
mounting. This also prepares the way for LVM and LUKS where there is
not a 1:1 mapping between partition and filesystem.
Image info currently cannot handle LVM and will abort with something
like:
mount: /tmp/tmpzwlch91r: unknown filesystem type 'LVM2_member'.
Detect LVM setup and just exit for now.
Extend the information gathered by `image-info`. This is needed to
properly inspect GCE images.
Enhancements:
- read all ssh client configuration files
- read all sshd configuration files
- read all YUM / DNF repos
- read DNF Automatic configuration
- fix reading of DNF configuration
Signed-off-by: Tomas Hozza <thozza@redhat.com>
In some cases, e.g. when analysing an extracted `tar` image, it could
happen that the tools used by `image-info` to analyse the image could
modify its content (e.g. create new files which were originally not on
the image). This is especially an issue with `rpm`, which DB backend
seemed to create files in `/var/lib/rpm/` when run.
Ensure that the analysed directory can not be modified by bind-mounting
it as a read only, before the analysis.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
One branch of the analyse_directory() function's if statement used
undefined variable `repo`. Copy its existing definition already used in
the function.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Use the `subprocess_check_output()` wrapper function in the
`read_selinux_ctx_mismatch()` parser function, instead of
directly calling `subprocess.check_output()`.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Extend the report to list also all documentation files, which are
normally part of the installed RPM packages, but were not installed
on the system. This can happen e.g. when '--excludedocs' option
is used when installing packages using rpm.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Read uncommented lined from /etc/resolv.conf and add them as a list to
the image-info report. The list of lines is present in the report even
if it is empty, so that the report is explicit about the file content
and presence.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Extend image-info to read systemd service unit drop-ins from
multiple paths:
- /etc/systemd/system/*.service.d
- /usr/lib/systemd/system/*.service.d
Signed-off-by: Tomas Hozza <thozza@redhat.com>
image-info's code which analysed image with multiple partitions was not
correctly working with more than two partitions, which had to be the
root '/' and EFI partition '/boot/efi'. The consequence was that SELinux
labels on paths which were mounted incorrectly could have been reported
as incorrect.
Modify `append_partitions()` to first read the fstab entries and then
mount all partitions using their UUID in the correct order. Only then
analyze the image filesystem tree.
Regenerate affected image test cases.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
image-info could produce a weird fstab error with an empty list as a
member, when analysing images not built using osbuild. Ensure that any
lines in fstab with only whitespace characters are skipped during image
analysis.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Some images with ESP, e.g. the `rhel-ec2-aarch64`, have the `/boot` on
a separate partition. `image-info` currently produces traceback on such
images, e.g.:
Traceback (most recent call last):
File "/home/thozza/devel/osbuild-composer/./tools/image-info", line 1997, in <module>
main()
File "/home/thozza/devel/osbuild-composer/./tools/image-info", line 1991, in main
report = analyse_image(target)
File "/home/thozza/devel/osbuild-composer/./tools/image-info", line 1863, in analyse_image
append_partitions(report, device, loctl)
File "/home/thozza/devel/osbuild-composer/./tools/image-info", line 1849, in append_partitions
append_filesystem(report, tree)
File "/home/thozza/devel/osbuild-composer/./tools/image-info", line 1809, in append_filesystem
with open(f"{tree}/grub2/grubenv") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp3i__6m1w/grub2/grubenv'
The reason is that `grub2/grubenv` on the `/boot` partition is a symlink
to `../efi/EFI/redhat/grubenv`. However the `efi` directory on the
`/boot` partition is empty and the ESP must be mounted to it for the
expected path to exist.
Modify `image-info` to mount the ESP to `efi` directory if it exists on
the inspected partition.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
When `image-info` inspects ostree images, the `/usr/etc` is bind-mounted
to `/etc`. This results in conflicting SELinux policy specification for
these files and makes the outcome dependent on the `setfiles` build.
All the files in `/etc` have mismatch in the expected vs. actual SELinux
context.
Exclude `/etc` from the check of SELinux ctx mismatches in case the
analysed tree is from an ostree-based image.
Sort the list returned `read_selinux_ctx_mismatch()` based on the item's
`filename` key, to make the result consistent across runs.
`image-info` can not read SELinux labels from the images, which are not
known to the host. This makes the report content depend on the host
environment. As a temporary workaround, relabel the image-info script with
osbuild_exec_t label to allow it to read unknown SELinux labels.
Modify documentation in `test/README.md` to explain the issue with
`image-info` and unknown SELinux labels.
Modify the `generate-all-test-cases` to relabel `image-info` before
generating test cases.
Modify the `image_tests.sh` to relabel `image-info` before running image
test cases.
Add 'tar' image for 'rhel-8' on 's390x' back to the matrix of generated
test cases, as it was removed by mistake. Regenerate the image test
case. Remove 'tar' image from 'rhel-84' on 's390x' from the matrix of
generated test cases, as it is not supported.
Regenerate all affected image test cases.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
One occurrance of read_partition() call have not been modified as part of
previous commit [1], which is a bug.
[1] 598c2b6939
Signed-off-by: Tomas Hozza <thozza@redhat.com>
This is a preparation for defining EC2 images for RHEL-8.5.0. These
extensions to image-info tool represent modifications done to the
official EC2 images currently produced as RHEL release. It is important
to be able to analyse these aspects of images, before we define them in
osbuild-composer, to ensure that the resulting images will be consistent
with the current state.
- Read non-empty lines from /etc/hosts file and add them to the report.
- Read content of /etc/machine-id and add it to the report.
- Read uncommented key/values from /etc/systemd/logind.conf and add them
to the report.
- Read all ifcfg-* files from /etc/sysconfig/network-scripts/ and add
their values to the report.
- Read content of /etc/locale.conf and add it to the report.
- Read SELinux configuration from /etc/selinux/config and add it to the
report.
- Inspect the filesystem tree for SELinux context mismatches and add
them to the report.
- Read configuration files from /etc/modprobe.d/ and for now report only
all blacklisted kernel modules.
- Read RHSM configuration from /etc/rhsm/rhsm.conf and add it to the
report.
- Read cloud-init configuration from /etc/cloud/cloud.conf and add it to
the report.
- Read all *.conf files from /etc/dracut.conf.d/ and add their content to
the report.
- Read VC and X11 keyboard configuration and add it to the report.
- Read specific configuration directives from Chrony configuration and
add them to the report. Specifically 'server', 'pool', 'peer' and
'leapsectz'.
- Read drop-in configurations for *.service unit files from
/etc/systemd/system/ and add them to the report.
- Read all configuration files from /etc/tmpfiles.d/ and add them to the
report.
- Read all configuration files from /etc/sysctl.d/ and add them to the
report.
- Read the Tuned active profile and profile mode and add them to the
report.
- Read all configuration files from /etc/security/limits.d and add them
to the report.
- Read sudoers configuration from /etc/sudoers and files in
/etc/sudoers.d/ and add uncommented lines to the report. No
sophisticated parsing is done, because the configuration format
grammar is too complicated for the purpose of image-info.
- Read udev rules configuration files from /etc/udev/rules.d/ and add
them to the report.
- Read DNF configuration and defined vars and add them to the report.
- Read profile ID and enabled features used by authselect.
- Enable SELinux, extended attributes and POSIX ACLs support when
unpacking 'tar' image type to prevent potential mismatches
Regenerate all image test cases to reflect changes in the image-info
output. Modify the distro-arch-imagetype-map.json to cover all
combinations currently covered by existing image test cases.
Add doc strings to all read_* functions.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
After change to image-info [1], which shows the format version for qcow2
images, the `image-format` changed from string to a dictionary. However
the `open_image()` function still compares it with string. This causes
`raw` images to be converted by the script again to `raw` format. This
change fixes the issue, so that `raw` images are not converted, but used
as they are.
[1] 5937b9adca
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Change the "image-format" from a string to a dict, with a "type":
$value entry, where $value contains the previous plain string
data.
Additionally, include the qcow2 format version, if the given
image is indeed a qcow2.
Adapt all manifest test accordingly (partly done by Ondřej)
Python 3 script used for conversion of manifest tests:
import os
import json
for name in os.listdir(os.getcwd()):
if not name.endswith(".json"):
continue
print(name)
with open(name, "r") as old:
data = json.load(old)
info = data.get("image-info", {})
format = info.get("image-format")
if not format:
continue
info["image-format"] = {
"type": format
}
if format != "qcow2":
continue
info["image-format"]["compat"] = "1.1"
with open(name + ".new", "w") as new:
json.dump(data, new, indent=2)
new.write("\n")
new.flush()
os.rename(name+".new", name)
test: use the new image-info format in all test manifests
The previous commit converted only qcow2 and openstack manifests but this change
is actually needed for all manifests produced by the qemu assembler.
Co-Developed-by: Ondřej Budai <ondrej@budai.cz>
Modify RHEL 8.3 and 8.4 KVM guest images definition to produce osbuild
manifest with `org.osbuild.rhsm` stage to disable both RHSM DNF plugins
(`product-id` and `subscription-manager`).
Update `/docs/news/unreleased/osbuild-rhsm-stage.md` to note that RHEL
8.3 and 8.4 qcow2 image definitions are updated to disable RHSM DNF
plugins by default.
Enhance `tools/image-info` tool to add RHSM-specific section to its
output in case RHSM DNF plugins configurations exist in the tree.
Regenerate all RHEL image test cases affected by the patch set.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The org.osbuild.sysconfig stage is now supported. Config updates can be
made to the kernel and network files. Currently, the same values are
used for all image types in rhel84. The image-info script is updated to
allow testing the sysconfig info.
The algorithm from calculating changed in osbuild 24, thus breaking the
testsuite. As the the inputhash is merely a implementation detail, there's
no need to test it in the image tests. Nevertheless, the inputhash is also
tested in the osbuild's testsuite.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
`image-info` tools parses output of `systemctl list-unit-files` run on a
different tree (with `--root` option), to determine the list of enabled
and disabled services on the inspected image. However since Fedora 33
(and presumably since systemd v246), the output of `systemctl
list-unit-files` changed. Some units previously reported as "enabled" or
"disabled" are now reported as "alias", which means, that they are just
a symlink to a different unit.
There is no systemd command, that would take an "alias" unit and would
report its state as "enabled" or "disabled" and could run on a different
tree (with "--root" option).
To make the list of reported services in the given state consistent on
systems with older and new (v246+) systemd version, check all "alias"
units and append them to the list of services with a specific status,
if their target is also listed in in the list.
Example of the `systemctl list-unit-files` output change:
~]# rpm -q systemd
systemd-246.6-3.fc33.x86_64
~]# systemctl list-unit-files ctrl-alt-del.target
UNIT FILE STATE VENDOR PRESET
ctrl-alt-del.target alias -
~]# rpm -q systemd
systemd-245.8-2.fc32.x86_64
~]# systemctl list-unit-files ctrl-alt-del.target
UNIT FILE STATE VENDOR PRESET
ctrl-alt-del.target enabled disabled
This change makes it possible to produce consistent output for an
inspected image, regardless if the `image-info` tool is run on Fedora
32, Fedora 33 or RHEL-8.
Also regenerate all Fedora 33 test cases, since this commit changes the
content of produced list of enabled / disabled services since Fedora 33.
The list is now consistent with what would be produced by `image-info`
for an image on older Fedora (e.g. 32) or RHEL-8.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
An image only had a systemd stage added if its blueprint contained
services or if its image type contained enabled services. The systemd
stage is now also added if the image type contains disabled services or
a default target.
The RHEL 8.4 qcow2 image type now specifies the multi-user default target.
In order to test this the image-info tool now includes the default
target in its output. Image test manifests are updated to include this
change.
The conversion with `qemu-img convert` often fails on aarch64 systems
with LOTS of CPUs. This is fixed in RHEL 8 for aarch64, but not in
Fedora.
Set the maximum coroutines to 1 to avoid this issue until the bug is
fixed.
Bug: https://bugs.launchpad.net/qemu/+bug/1805256
Signed-off-by: Major Hayden <major@redhat.com>
Add support for partitions that can 'blkid' can not handle, like
the bios boot partition. For such partitions, 'blkid' will fail;
in that case all additional information will just be 'None'.
Convert any image that is not a raw image, e.g. a qcow2, to a raw
image and open the partitions via loop devices. This replaces the
usage of nbd, which was racy and flaky.
Instead on relying on the kernel for the partition parsing, this
is now done manually via loop devices and start + offset taken
from sfdisk. As a result the read_partition function has been
adapted to be called at later time, after the partitions were
opened via loop devices.
Not using nbd also means that the partition table is not scanned
by the kernel anymore and udev is not triggered. As a result the
'PARTUUID' property is not present for dos/mbr partition layouts,
since it is auto-generated by udev/blkid. Relevant blkid files
and functions are:
blkid_partition_gen_uuid(par)
called from probe_dos_pt()
in file libblkid/src/partitions/dos.c line 295
defined in libblkid/src/partitions/partitions.c line 1374
which generates the uuid via snprintf using the format:
'"%.33s-%02x", par->tab->id, par->partno'
Based on https://github.com/karelzak/util-linux at ce8985cc7
NB: the loop device code is imported from osbuild, making this
tool depend on osbuild's private library.
NB: As of the image conversion, more disk space is required to
examine non-raw images.
If a compressed file is detected (xz, bzip2, gzip), uncompress it
before analyzing it. NB: this has to happen after the is_tarball
check, because a tarball would also count as compressed.
Directly iterate over the partition list entries instead if using
indices. Optimize visual flow for the happy case by immediately
continuing if a partition has no file-system.
