Explicitly specify the workdir when running manifest_tests, make
potential debugging of the test case on CI runner easier (because
otherwise the workdir would get removed after failing test).
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
In case the workdir is not provided to the script explicitly as an
argument, the script will use a temporary directory under /var/tmp as
its workdir. In such case, the workdir will be deleted on exit. This
should mitigate potentially confusing behavior when executing the script
multiple times with different arguments, while never specifying the
workdir.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Extract the opening of LVM LV devices from `discover_lvm()` to
`OSBuildDeviceManager` class as `open_lvm_lv()` method.
`open_lvm_lv()` returns the path to the opened device in the devpath set
in the underlying `DeviceManager`. The `org.osbuild.lvm2.lv`
implementation takes the responsibility for creating and managing
device nodes. This means that we don't need to be creating any device
nodes directly in `osbuild-image-info`, especially in the current
working directory. This was previously causing issues when inspecting
two images with different LVM layout in a sequence.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a new class `OSBuildDeviceManager`, which wraps
`devices.DeviceManager`, so that we can consolidate all code that is
opening devices using osbuild, in it. As the fist step, move the
`loop_open()` function to the class.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Rename the function for naming consistency and always include the actual
error from `pvdisplay` when raising RuntimeError.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Extract the code that discovers LVM LV names for a given VG, from
`discover_lvm()` into a separate function `lvm_lvs_for_vg()`. This
improves the readability of the code. In addition, some values returned
by the `lvdisplay` invocation were never used. Don't request them and
simplify the code. Rename variables that hold LV names to clearly
express that.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Per discussion in the team, we see little value in rebuilding RHEL-8.10
images on RHEL-8.10 for the purpose of manifest testing in osbuild. So
let's not do that anymore.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a simple script and an action to update images ref in Schutzfile on
schedule.
Both, the script and action are based on those in the osbuild/images
repository and the credit for those goes to Achilleas Koutsou.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Switch to the new manifest_tests based on the osbuild/images CI cache,
instead of using the manifest-db.
For now, run tests only for a subset of manifest configurations. This
can be changed in the future as needed.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a script to setup the environment for manifest tests to run. This
includes enabling specific repositories on RHEL, installing required
packages and installing AWSCLI for accessing S3.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add new implementation of the manifest tests, which goal is to ensure
that the osbuild behavior didn't change. This is ensured by comparing
image-info report produced for image artifact built using older
(known-to-be-good) osbuild version and the latest osbuild version
(potentially from a PR).
Previously, we used the osbuild/manifest-db repository, which contained
pre-generated manifests with their corresponding image-info report.
Unfortunately, this setup prooved to be cumbersome to maintain and keep
updated.
We are already building images for known manifests in the osbuild/images
repository. These are then uploaded to AWS S3 cache. The images are
built with a pinned osbuild version, which will be always older than the
one that we would be using for image build in osbuild PR.
So the intention of this new script is to take advantage of the
osbuild/images S3 cache. As part of the test case (for a specific distro
/ arch / image_type / config):
- download the manifest from S3
- download the image artifact, built from the manifest, from S3
- generate image-info report for the downloaded image
- rebuild the downloaded manifest using current version of osbuild
- generate image-info report for the rebuilt image
- compare the two image-info reports. If there is no difference, the
test case PASS, otherwise it will FAIL.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This commit fixes a race/threading issue with the way the monitor
works. The osbuild monitor can be called from multiple threads,
e.g. in buildroot.py:run() monitor.log() is called but also
in host.py:_stdout_ready(). This can lead to out-of-order writes
when many messages need to be processed.
We did not notice this so far because we were lucky and also
log was just used for information. But now it is used to transmit
the jsonseq data which means out-of-order communication results
in broken json.
Closes: https://github.com/osbuild/image-builder-cli/issues/110
This commit adds a small test that ensures that we notice when
the solver API adds new top-level keys. When this happens the
images library breaks and we need to increase the
`Provides: osbuild-dnf-json-api` version in the `osbuild.spec`.
See e.g. https://github.com/osbuild/osbuild/pull/1992
Due to the `modules` field that was added to the depsolve result the
depsolve json response is now incompatible with previous versions. This
requires a bump to the version.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
The NEVRA field accidentally made it into the public API. It shouldn't
be as it is not used downstream and in fact breaks downstream.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
The modularity changes introduced a new field in the DNF JSON responses.
This bump ensures that current deployments don't break.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
Add two unit tests for the read_default_target() function:
1. When default target should be found.
2. When there should be no default target.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Remove the requirement for `:` in the name which would have been
selected by the fronted, instead asking dnf "is this a module?".
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
Instead of passing in the path we name the file according to the module
name. Path can be reintroduced later if absolutely necessary.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
Add a new `enabled_repos` field on the testcases which explicitly lists
which repositories are passed into a certain testcase. This allows us to
pass appstream only to the module testcase.
Re-adjust the package lists again since we're now not using appstream in
all depsolve tests.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
Instead of returning the string contents of the module configuration
file let's return some more structured data so the resulting file can be
written with a stage.
Note that we don't do the same for the fallback file as we are only
provided this as a YAML blob and unparsing it to then immediately
reparse it again is counterproductive.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
With the enabling of AppStream some more (perhaps optional) packages are
included.
This also adds a test case which installs a module and verifies that
that module is returned.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
The CentOS Stream 9 repository metadata contains modules; these are
necessary for testing modularity depsolving.
Note that the filelists metadata is kept empty to keep repository size
down.
Co-authored-by: Michael Vogt <michael.vogt@gmail.com>
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
This changes the depsolver to expect modules inside the normal
`package-specs` instead of having a separate `module-install-specs`.
Modules passed to a "normal" `dnf install` must start with an `@` and
contain a `:`. This is up to the user to pass correctly.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
The commit implements modularity for the dnf4 dependency solver.
The dependency solver now takes two new keys per transaction:
`module-install-specs` and `module-enable-specs`, each of which is a
list of module specifications.
When modules are used and packages for them are installed a new
dictionary is returned in the response where each key is the name of an
enabled module and the value are the contents of the module
configuration and the module failsafe file that DNF needs in the
resulting system.
These values can be used by consumers of the dependency solver, such as
`images`, to create the appropriate inline sources and copy stages.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
The name 'cache' is more recognisable and conveys the purpose of the
option just as well (if not better) than 'store'.
Also, the option that controls the size of this directory is called
'cache-max-size', so we're already referring to it as a cache.
This helps to clear a risk concern for
automotive.
Add optional flag to org.osbuild.mkfs.ext4 stage
to enable/disable both lazy_itable_init and
lazy_journal_init extended options. Both mke2fs
options are controled by the same flag, namely
`lazy_init`.
Signed-off-by: Albert Esteve <aesteve@redhat.com>
Add mpp files to create manifests that will embed the container (that
contains /dev/null) in an OS tree and compose an ostree commit from it.
This used to cause the rpm-ostree compose command to fail with:
error: Writing commit: While writing rootfs to mtree: Not a regular file or symlink: null
This commit tweaks build() to be mypy clean without the need to
call assert. This drops the map() and instead we use the existing
dict-like access of the manifest to get the pipeline. In practise
this should not happen but lets be prepared.
Note that a small tweak for the error is needed to make it clear
what is happening.
Thanks to Simon for raising this.
This commit makes the returns of the of {Build,Download}Result.as_dict()
explicit. Ideally this would just be a dataclass and dataclass.asdict()
but because we need to support python3.6 this is not possible today.
This commit adds error reporting from source download errors
to the monitor. It reuses the `BuildResult` for symmetry but
we probably want to refactor this a bit to make source handling
a bit more similar to stages.
