Parameterize `test_sources` via `pytest.mark.parameterize`, so that
now the product of source and test cases for that source is visible
to pytest and thus also the caller.
Instead of testing the legacy `SourcesServer` and `sources.get`,
test the `Source.download` method, which is the new and exclusive
way how sources are used in osbuild. For this, the complete info
for a specific source is now included the specific test case.
For the `org.osbuild.curl` source this means that the respective
information is moved from `sources.json`, which is then unused
and thus removed. The test case that checks for an unknown
checksum is also removed because `Source.download` just fetches
everything instead of a subset.
Use `pytest`, which is a dependency of osbuild since a few versions,
instead of `unittest` for the source tests. The main reason is to be
able to use `pytest.mark.parameterize` in the near future to easily
generate a test matrix.
Add a new option `qcow2_comapt` which can be used explicitly
select the compatibility level of the qcow2 file format. Qemu
version 1.1 introduced extensions to the format that became
the default with 1.7, which are not readable by qemu < 1.1.
Thus if the resulting qcow2 should be read by such older qemu
versions, the compatibility level needs to be set to 0.10.
According to the OCI Image Format Specification[1] history entries
for layers in the container are optional; but when trying to push
a container quay.io via skopeo (copy oci-archive:… docker://quay)
it will fail with "Cannot convert an image with 0 history entries".
This seems to come from the containers/image[2] library when the
container is converted back from the docker distribution format
to oci-archive on quay.io. Thus it seems that when skopeo converts
the image to the docker format for the distribution it does not
fill any the history entries, which are then assumed and required
to be there when converting back.
To fix this, insert history entries for each layer that is created.
[1] https://github.com/opencontainers/image-spec/blob/master/config.md
[2] https://github.com/containers/image/
Only include a very specific set of extended attributes:
- user.*: user specified extended attributes
- security.ima: Integrity Measurement Architecture (IMA)
- security.capability: Linux capabilities(7)
This follows what containers/storage[1] and containers/buildah[2]
are doing. It is important to note that we DO NOT want selinux
related extended attributes (`security.selinux`) in there, which
seems to be pulled in by some versions of `tar` even if that was
seemingly excluded via `--no-selinux`. Therefore we also exclude
selinux and xattrs explicitly from the wrapping container to
make sure they are never included.
[1] 35ebda8ae2/pkg/archive/archive.go (L399)
[2] 214e4c9335/copier/xattrs.go (L19)
This reverts commit 59184b23a2.
This change breaks current testing and is not critical.
We will reintroduce it later when there is time to adapt the tests.
instead of duplicating the same script here!
Specifies needed cloud credentials.
NOTE: don't start osbuild-composer in deploy.sh because this is
now done by /usr/libexec/osbuild-composer-test/provision.sh. Otherwise
leads to errors because the socket is already taken.
Instead of deleting and re-creating /etc/machine-id, just truncate
it to an empty file. This should let the mode be 0444, which is
the mode that systemd also creates it with.
Use `ostree-ouput` instead of `files-output` as prefix for the
temporary directory prefix.
Also fix the description of the "origin" to reflect that
pipeline and source origins are supported.
Additionally remove some dead code.
We explicitly pinned the F32 CI images in the past due to update issues
in F33. However, those have been resolved and we should switch back to
the most recent Fedora CI images.
This commits switches all instances of the osbuild-ci image back to the
latest stream, snapshot taken on 2021-02-19 13:11 (latest-202102191311).
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
In order to have a more stable package metadata representation,
sort the generated metadata by name. Adapt the tests' metadata
file to reflect that change.
When the expected metadata does not match the produced metadata,
write the latter to `/tmp` for closer inspection; additionally
this should help update the metadata file in case the changes
are expected.
Since `/home` will not end up in the commit¹ move the home
directories to `/var/home`. This is done after the new root
file system has been initialized, and only if `/home` is not
empty.
¹ it is neither copied back in the preptree stage itself, nor
would it be picked up by rpm-ostree compose tree postprocess
were it copied back.
Instead of using stderr for the ostree subprocess command
capture its output so that in the case of an error we get
properly return the error output. With the old behavior
all the `ostree` command output would land in the journal
of the worker.
Source, for compatability reasons, have two modes: download only
and download and export. The difference is the arguments that
are passed to the source: For download only, the `output` param
is empty. In this case also `checksums` *can* be empty and if so
it means everything, i.e. the commits, should be fetched. The
latter was not properly handled so far. Adjust the logic, which
now closely mimics that of the `org.osbuild.curl` source to fix
this case.
Also catch exceptions invoking `ostree` and properly return them
via the json error messaging.
This is a partial revert of commit d584a1e, which converted the
dynamically generate stage tests to sub tests. The problem with
the latter is that they can't be individually run, which makes
testing changes to specific stages cumbersome to develop.
Therefore switch back to a model where the stage tests are
dynamically generated via a class decorator.
In the output test, check that for a successful pipeline run all
the elements are present: main result, assembler result, stage
result.
NB: Build result is hard to test because we would need to actually
build a valid build root.
If a pipeline has an assembler and that assembler failed, the
overall status of the build also needs to be marked as failed.
This used to be the case, but a bug got introduced when the
format abstraction code was added.
This is, like the stage with the same name, an assembler that
will exit with an error code (default 255, but can be specified
via the assembler options). It is mostly useful for testing.
Set the "GRUB_CMDLINE_LINUX" variable in /etc/default/grub to the
kernel command line options. This is used by `grub2-mkconfig` to
assemble the full kernel command line when generating the menu
entires. NB: `GRUB_CMDLINE_LINUX` does NOT include the root fs
bits (`root=...`), since that is generated by grub2-mkconfig
itself.
Do the check if there is a /etc/machine-id before moving /etc to
/usr/etc, because otherwise /etc/machine-id will obviously not
exist and thus the detection is broken.
Instead of including SELinux labels for the content layers via the
`--selinux` tar option, make sure selinux labels are not included by
using the `--no-selinux` option.
The inclusion of the labels was a mistake, since they should be
determined by the target system because selinux labels are not
namespaced. On RHEL/Fedora the SELinux label used is something like
`system_u:object_r:container_ro_file_t:s0` for all the files in the
container.
Including the label was leading to permission problems because
the files had a different label on the host and programs inside
the container get `EACCES`, i.e. Permission denied, errors when
accessing files with the different label.
Interestingly this does not happen on Fedora 33 but only on RHEL.
One possibility is that the overlayfs kernel driver in RHEL is
behaving differently on RHEL than on Fedora.
The pylorax implementation of the template running code supports
globbing, as well as `--exclude` and `--optional` commands. These
are handled independently for each `installpkg` command, so that
requesting the installation of firmware packages in one command
together with an exclude of `*alsa` does indeed only exclude any
alsa firmware packages but not any other alsa packages. The
previous version of this script would just build an global list
for excludes which has a drastically different result because
a global exclude of "*alsa" would result in a global exclusion of
all matching packages and probably a dependency error.
Therefore, add support for dnf based filtering of packages on a
per `installpkg` command bases, very much like pylorax does.
Add a simple helper that is meant to gather the list of packages
to be installed via a lorax template that uses the "installpkg"
directives. A prominent example is the 'runtime-install.tmpl'
script from lorax-templates-generic, used to create boot isos.
This builds osbuild in F33aarch64 as well as RHEL8.4 and CentOS8. No tests are currently run.
With the mockbuilds in place we will be able to run composer CI against osbuild master, when necessary.
