image-info's code which analysed image with multiple partitions was not
correctly working with more than two partitions, which had to be the
root '/' and EFI partition '/boot/efi'. The consequence was that SELinux
labels on paths which were mounted incorrectly could have been reported
as incorrect.
Modify `append_partitions()` to first read the fstab entries and then
mount all partitions using their UUID in the correct order. Only then
analyze the image filesystem tree.
Regenerate affected image test cases.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
image-info could produce a weird fstab error with an empty list as a
member, when analysing images not built using osbuild. Ensure that any
lines in fstab with only whitespace characters are skipped during image
analysis.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Some images with ESP, e.g. the `rhel-ec2-aarch64`, have the `/boot` on
a separate partition. `image-info` currently produces traceback on such
images, e.g.:
Traceback (most recent call last):
File "/home/thozza/devel/osbuild-composer/./tools/image-info", line 1997, in <module>
main()
File "/home/thozza/devel/osbuild-composer/./tools/image-info", line 1991, in main
report = analyse_image(target)
File "/home/thozza/devel/osbuild-composer/./tools/image-info", line 1863, in analyse_image
append_partitions(report, device, loctl)
File "/home/thozza/devel/osbuild-composer/./tools/image-info", line 1849, in append_partitions
append_filesystem(report, tree)
File "/home/thozza/devel/osbuild-composer/./tools/image-info", line 1809, in append_filesystem
with open(f"{tree}/grub2/grubenv") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp3i__6m1w/grub2/grubenv'
The reason is that `grub2/grubenv` on the `/boot` partition is a symlink
to `../efi/EFI/redhat/grubenv`. However the `efi` directory on the
`/boot` partition is empty and the ESP must be mounted to it for the
expected path to exist.
Modify `image-info` to mount the ESP to `efi` directory if it exists on
the inspected partition.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
When `image-info` inspects ostree images, the `/usr/etc` is bind-mounted
to `/etc`. This results in conflicting SELinux policy specification for
these files and makes the outcome dependent on the `setfiles` build.
All the files in `/etc` have mismatch in the expected vs. actual SELinux
context.
Exclude `/etc` from the check of SELinux ctx mismatches in case the
analysed tree is from an ostree-based image.
Sort the list returned `read_selinux_ctx_mismatch()` based on the item's
`filename` key, to make the result consistent across runs.
`image-info` can not read SELinux labels from the images, which are not
known to the host. This makes the report content depend on the host
environment. As a temporary workaround, relabel the image-info script with
osbuild_exec_t label to allow it to read unknown SELinux labels.
Modify documentation in `test/README.md` to explain the issue with
`image-info` and unknown SELinux labels.
Modify the `generate-all-test-cases` to relabel `image-info` before
generating test cases.
Modify the `image_tests.sh` to relabel `image-info` before running image
test cases.
Add 'tar' image for 'rhel-8' on 's390x' back to the matrix of generated
test cases, as it was removed by mistake. Regenerate the image test
case. Remove 'tar' image from 'rhel-84' on 's390x' from the matrix of
generated test cases, as it is not supported.
Regenerate all affected image test cases.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
One occurrance of read_partition() call have not been modified as part of
previous commit [1], which is a bug.
[1] 598c2b6939
Signed-off-by: Tomas Hozza <thozza@redhat.com>
This is a preparation for defining EC2 images for RHEL-8.5.0. These
extensions to image-info tool represent modifications done to the
official EC2 images currently produced as RHEL release. It is important
to be able to analyse these aspects of images, before we define them in
osbuild-composer, to ensure that the resulting images will be consistent
with the current state.
- Read non-empty lines from /etc/hosts file and add them to the report.
- Read content of /etc/machine-id and add it to the report.
- Read uncommented key/values from /etc/systemd/logind.conf and add them
to the report.
- Read all ifcfg-* files from /etc/sysconfig/network-scripts/ and add
their values to the report.
- Read content of /etc/locale.conf and add it to the report.
- Read SELinux configuration from /etc/selinux/config and add it to the
report.
- Inspect the filesystem tree for SELinux context mismatches and add
them to the report.
- Read configuration files from /etc/modprobe.d/ and for now report only
all blacklisted kernel modules.
- Read RHSM configuration from /etc/rhsm/rhsm.conf and add it to the
report.
- Read cloud-init configuration from /etc/cloud/cloud.conf and add it to
the report.
- Read all *.conf files from /etc/dracut.conf.d/ and add their content to
the report.
- Read VC and X11 keyboard configuration and add it to the report.
- Read specific configuration directives from Chrony configuration and
add them to the report. Specifically 'server', 'pool', 'peer' and
'leapsectz'.
- Read drop-in configurations for *.service unit files from
/etc/systemd/system/ and add them to the report.
- Read all configuration files from /etc/tmpfiles.d/ and add them to the
report.
- Read all configuration files from /etc/sysctl.d/ and add them to the
report.
- Read the Tuned active profile and profile mode and add them to the
report.
- Read all configuration files from /etc/security/limits.d and add them
to the report.
- Read sudoers configuration from /etc/sudoers and files in
/etc/sudoers.d/ and add uncommented lines to the report. No
sophisticated parsing is done, because the configuration format
grammar is too complicated for the purpose of image-info.
- Read udev rules configuration files from /etc/udev/rules.d/ and add
them to the report.
- Read DNF configuration and defined vars and add them to the report.
- Read profile ID and enabled features used by authselect.
- Enable SELinux, extended attributes and POSIX ACLs support when
unpacking 'tar' image type to prevent potential mismatches
Regenerate all image test cases to reflect changes in the image-info
output. Modify the distro-arch-imagetype-map.json to cover all
combinations currently covered by existing image test cases.
Add doc strings to all read_* functions.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
After change to image-info [1], which shows the format version for qcow2
images, the `image-format` changed from string to a dictionary. However
the `open_image()` function still compares it with string. This causes
`raw` images to be converted by the script again to `raw` format. This
change fixes the issue, so that `raw` images are not converted, but used
as they are.
[1] 5937b9adca
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Change the "image-format" from a string to a dict, with a "type":
$value entry, where $value contains the previous plain string
data.
Additionally, include the qcow2 format version, if the given
image is indeed a qcow2.
Adapt all manifest test accordingly (partly done by Ondřej)
Python 3 script used for conversion of manifest tests:
import os
import json
for name in os.listdir(os.getcwd()):
if not name.endswith(".json"):
continue
print(name)
with open(name, "r") as old:
data = json.load(old)
info = data.get("image-info", {})
format = info.get("image-format")
if not format:
continue
info["image-format"] = {
"type": format
}
if format != "qcow2":
continue
info["image-format"]["compat"] = "1.1"
with open(name + ".new", "w") as new:
json.dump(data, new, indent=2)
new.write("\n")
new.flush()
os.rename(name+".new", name)
test: use the new image-info format in all test manifests
The previous commit converted only qcow2 and openstack manifests but this change
is actually needed for all manifests produced by the qemu assembler.
Co-Developed-by: Ondřej Budai <ondrej@budai.cz>
Modify RHEL 8.3 and 8.4 KVM guest images definition to produce osbuild
manifest with `org.osbuild.rhsm` stage to disable both RHSM DNF plugins
(`product-id` and `subscription-manager`).
Update `/docs/news/unreleased/osbuild-rhsm-stage.md` to note that RHEL
8.3 and 8.4 qcow2 image definitions are updated to disable RHSM DNF
plugins by default.
Enhance `tools/image-info` tool to add RHSM-specific section to its
output in case RHSM DNF plugins configurations exist in the tree.
Regenerate all RHEL image test cases affected by the patch set.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The org.osbuild.sysconfig stage is now supported. Config updates can be
made to the kernel and network files. Currently, the same values are
used for all image types in rhel84. The image-info script is updated to
allow testing the sysconfig info.
The algorithm from calculating changed in osbuild 24, thus breaking the
testsuite. As the the inputhash is merely a implementation detail, there's
no need to test it in the image tests. Nevertheless, the inputhash is also
tested in the osbuild's testsuite.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
`image-info` tools parses output of `systemctl list-unit-files` run on a
different tree (with `--root` option), to determine the list of enabled
and disabled services on the inspected image. However since Fedora 33
(and presumably since systemd v246), the output of `systemctl
list-unit-files` changed. Some units previously reported as "enabled" or
"disabled" are now reported as "alias", which means, that they are just
a symlink to a different unit.
There is no systemd command, that would take an "alias" unit and would
report its state as "enabled" or "disabled" and could run on a different
tree (with "--root" option).
To make the list of reported services in the given state consistent on
systems with older and new (v246+) systemd version, check all "alias"
units and append them to the list of services with a specific status,
if their target is also listed in in the list.
Example of the `systemctl list-unit-files` output change:
~]# rpm -q systemd
systemd-246.6-3.fc33.x86_64
~]# systemctl list-unit-files ctrl-alt-del.target
UNIT FILE STATE VENDOR PRESET
ctrl-alt-del.target alias -
~]# rpm -q systemd
systemd-245.8-2.fc32.x86_64
~]# systemctl list-unit-files ctrl-alt-del.target
UNIT FILE STATE VENDOR PRESET
ctrl-alt-del.target enabled disabled
This change makes it possible to produce consistent output for an
inspected image, regardless if the `image-info` tool is run on Fedora
32, Fedora 33 or RHEL-8.
Also regenerate all Fedora 33 test cases, since this commit changes the
content of produced list of enabled / disabled services since Fedora 33.
The list is now consistent with what would be produced by `image-info`
for an image on older Fedora (e.g. 32) or RHEL-8.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
An image only had a systemd stage added if its blueprint contained
services or if its image type contained enabled services. The systemd
stage is now also added if the image type contains disabled services or
a default target.
The RHEL 8.4 qcow2 image type now specifies the multi-user default target.
In order to test this the image-info tool now includes the default
target in its output. Image test manifests are updated to include this
change.
The conversion with `qemu-img convert` often fails on aarch64 systems
with LOTS of CPUs. This is fixed in RHEL 8 for aarch64, but not in
Fedora.
Set the maximum coroutines to 1 to avoid this issue until the bug is
fixed.
Bug: https://bugs.launchpad.net/qemu/+bug/1805256
Signed-off-by: Major Hayden <major@redhat.com>
Add support for partitions that can 'blkid' can not handle, like
the bios boot partition. For such partitions, 'blkid' will fail;
in that case all additional information will just be 'None'.
Convert any image that is not a raw image, e.g. a qcow2, to a raw
image and open the partitions via loop devices. This replaces the
usage of nbd, which was racy and flaky.
Instead on relying on the kernel for the partition parsing, this
is now done manually via loop devices and start + offset taken
from sfdisk. As a result the read_partition function has been
adapted to be called at later time, after the partitions were
opened via loop devices.
Not using nbd also means that the partition table is not scanned
by the kernel anymore and udev is not triggered. As a result the
'PARTUUID' property is not present for dos/mbr partition layouts,
since it is auto-generated by udev/blkid. Relevant blkid files
and functions are:
blkid_partition_gen_uuid(par)
called from probe_dos_pt()
in file libblkid/src/partitions/dos.c line 295
defined in libblkid/src/partitions/partitions.c line 1374
which generates the uuid via snprintf using the format:
'"%.33s-%02x", par->tab->id, par->partno'
Based on https://github.com/karelzak/util-linux at ce8985cc7
NB: the loop device code is imported from osbuild, making this
tool depend on osbuild's private library.
NB: As of the image conversion, more disk space is required to
examine non-raw images.
If a compressed file is detected (xz, bzip2, gzip), uncompress it
before analyzing it. NB: this has to happen after the is_tarball
check, because a tarball would also count as compressed.
Directly iterate over the partition list entries instead if using
indices. Optimize visual flow for the happy case by immediately
continuing if a partition has no file-system.
Read the image information before opening the image. This prepares
for a change where we will use information about the image to
convert it to raw instead of using nbd.
When extracting the commit tarball as well as doing the ostree
commit checkout, use `/var/tmp` instead of `/tmp`, because the
latter might be backed by a tmpfs and thus by RAM; this should
only be used for small files.
Support analyzing ostree repositories or directory that contain an
ostree commit as created by osbuild. Will return the mode of the
repository along with the references and their commits. For the
first references, the commit is resolved and checked-out to a temp
directory. This directory in turn is then analyzed via the existing
append_filesystem function. The latter has gained some small ostree
specific tweaks.
NB: for the ostree checkout /usr/etc/ is bind mounted to /etc in
order to make append_filesystem happy. The rpm verification step
is NOT run, because that is not really compatible with ostree.
Move the code of the current main into is own method and call that
from main. This prepares support for analyzing other types than
images. Additionally, add argument parsing via argparse to get a
help texts.
systemctl list-unit-files doesn't produce machine readable output.
parse_unit_files isn't very good at reading it and can produce duplicate
records.
This commit fixes it by deduplicating and sorting the units. This is a bit
hacky solution, but should work just alright. In the future we might
dump list-unit-files and do the job ourself, but let's not recreate the
systemd logic for now.
If we detect a ESP (via its type UUID) remember it and mount it at
any filesystem that has a /boot/efi (there should in theory be
only one). This is needed so grubenv can be read, which is likely
a link from /boot/grub2/grubenv to ../efi/EFI/$vendor/grubenv.
Additionally this will make rpm verify not report that all the efi
binaries are missing, e.g.:
"missing": [
"/boot/efi/EFI",
"/boot/efi/EFI/BOOT",
"/boot/efi/EFI/BOOT/BOOTIA32.EFI",
"/boot/efi/EFI/BOOT/BOOTX64.EFI",
"/boot/efi/EFI/BOOT/fbia32.efi",
"/boot/efi/EFI/BOOT/fbx64.efi",
"/boot/efi/EFI/fedora",
"/boot/efi/EFI/fedora",
...
This shows the changes an image has relative to what its rpm database
thinks is installed. Output is:
"rpm-verify": {
"missing": [ <missing files> ],
"changed": { <map from filename to rpm attribute octet > }
}
Alas, this makes running image-info slower.
