DNF has more elaborate locking system and can wait for other instances of
itself when installing packages. Using rpm directly to install local
package is causing failures in CI due to it not being able to acquire
lock on `/var/lib/rpm/.rpm.lock`.
Using DNF should improve the situation, although there is no good
documentation to link and support this claim for sure.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This is how it is done also in osbuild-composer. The downside of doing
this in `deploy.sh` is that team SSH keys are not set as authorized for
mock-build jobs, which make it impossible to log into the machine and
debug anything.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
`bandit` is a fast Python vulnerability scanner, this provides a default
configuration file for it for those that want to run it.
`bandit --ini .bandit -r osbuild/` will get you results.
Do not specify the default value for 'expected_size' argument in
assertImageFile() function declaration. Previously, it was set to
`None`, which was never taken into account. Moreover, all callers of the
function always provide an explicit value, so the default was never
really used.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a new optional pytest CLI argument `--unsupported-fs` allowing to
specify file-systems which should be treated as unsupported in the
platform where running tests. Any test cases dependent on such
file-system support will be sipped.
This will allow to run unit tests and selectively skipping test cases
for unsupported file-systems.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Port assembler tests from unittest to pytest. In addition, use
parametrized tests when testing various filesystems and various
combinations.
This is important to be able to selectively skip the test for if a
specific filesystem is not supported by the kernel (e.g. btrfs is not
supported on RHEL). Skipping a unittest subtest is not possible, which
is the motivation to move away from it and use only pytest.
Test output is now also much nicer for parametrized test cases.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The code uses x86_64 as a default, see:
basearch = options.get("basearch", "x86_64")
Let's declare that explicitly in the schema to prevent any confusion.
The test case was testing a subset of the functionality, which is
already tested by test case in `tests.yml`. Delete it, since it does not
add any value.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Fix the following errors:
```
osbuild/util/lvm2.py:117: error: Only instance methods can be decorated with @property
osbuild/api.py:50: error: Only instance methods can be decorated with @property
osbuild/sources.py:85: error: Only instance methods can be decorated with @property
```
Chaining of `@classmethod` and `@property` has been deprecated since
Python 3.11 with a note that chaining didn't work correctly in some
cases.
Relevant links:
https://github.com/python/mypy/issues/13746https://docs.python.org/3.11/whatsnew/3.11.html#language-builtins
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Previously, the unit test depended on osbuild modules being installed on
the system. As a result, this made the test not work in CI where we do
not install osbuild when running unit tests. In addition, the stage
executed by the unit test would use different version of osbuild
internals than the version that is being tests, which could result in
issues or not testing the intended code.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The directory does not exist when the unit test is run in CI. Handle
this case by ensuring that parent directories are created as needed.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
In CI, we list specific tests from `test/run` to run them in parallel.
This is different than what we do with tests in `test/mod` and
`test/src`.
It seems that as a result, we did not run tests from the following
files in CI:
- `test_devices.py`
- `test_mount.py`
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The right way to enable services is to use a preset file instead of
writing directly into /etc. This adds a new stage called
`org.osbuild.systemd.preset` to do so.
Added another skopeo stage to skopeo/a.mpp.json with a skopeo source for
a container hosted on the osbuild-composer gitlab registry. The name
points to a manifest list, which refers to two containers (amd64 and
arm64) that contain a single text file (README.md). The `index` field
is enabled to include the manifest-list as an extra input to the stage.
The diff is updated with the new expected file list.
The containers were created with buildah:
amd=$(buildah from --arch=amd64 scratch)
arm=$(buildah from --arch=arm64 scratch)
buildah config --created-by "Achilleas Koutsou" "${amd}"
buildah config --created-by "Achilleas Koutsou" "${arm}"
buildah copy "${amd}" README.md
buildah copy "${arm}" README.md
amdid=$(buildah commit --format=docker --rm "${amd}")
armid=$(buildah commit --format=docker --rm "${arm}")
name="registry.gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/manifest-list-test"
buildah manifest create "${name}" "${amdid}" "${armid}"
podman manifest push --all "${name}" dir:container
Add support for resolving manifest lists in osbuild-mpp.
Adds an `index` boolean field to the container image struct for
mpp-resolve-images. When enabled, the preprocessor will also store the
manifest-list digest as a separate skopeo-index source and add it to the
skopeo stage under the `manifest-lists` input.
When a manifest list is matched with a container image, the skopeo
stage will merge the specified manifest into the container image dir
before copying it to the registry in the OS tree.
If there is no manifest to merge, we maintain the old behaviour of
symlinking the source to work around the ":" in filename issue.
Otherwise, we copy the container directory so that we can merge the
manifest in the new location.
Add an extra optional input type to the skopeo stage called
`manifest-lists`. This is a list of file-type inputs that must be a
list of manifest lists, downloaded by the skopeo-index source.
The manifests are parsed and automatically associated with an image from
the required `images` inputs. If any manifest list is specified and not
used, this is an error.
Adding manifest-lists currently has no effect.
A new source module that can download a multi-image manifest list from a
container registry. This module is very similar to the skopeo source,
but instead downloads a manifest list with `--multi-arch=index-only`.
The checksum of the source object must be the digest of the manifest
list that will be stored and the manifest that is downloaded must be a
manifest-list.
Change the local storage format for containers to the `dir` format.
The `dir` format will be used to retain signatures and manifests.
The remove-signatures option is removed since the storage format now
supports them.
The final move (os.rename()) at the end of the fetch_one() method now
creates the checksum directory if it doesn't exist and moves the child
archive into it, adding to any existing archives that might exist in
other formats (from a previous version downloading a `docker-archive`).
Dropped the .tar suffix from the symlink in the skopeo stage since it's
not necessary and the target of the link might be a directory now.
The parent class exists() method checks if there is a *file* in the
sources cache that matches the checksum. For containers, this used to
be a file called container-image.tar under a directory that matches the
checksum, so for containers it always returned False. Added an override
for the skopeo source that checks for the new directory archive.
The format so far was assumed to be `docker-archive` if the container
was coming from a source and `oci-archive` if it was coming from a
pipeline. The source format will now be changed to `dir` instead of
`docker-archive`. The pipeline format remains `oci-archive`.
With the new archive format being `dir`, the source can't be linked into
the build root and is bind mounted instead with the use of a MountGuard
created with the instance of the service, and torn down when the service
is stopped.
The _data field is removed from the map functions. It was unused and
these functions aren't part of the abstract class so they don't need to
have consistent signatures.
Update the skopeo stage with support for the newly supported `dir`
format.
Manifest-db is finally unblocked and we can update the reference commit.
Done manually this time to speed up the process.
I'm also updating the set of distributions we are testing on, upgrading
it to what's is generated on manifest db.
As of today the functionality is broken if somebody passes more than one
karg as the join would simply unify the strings but
ignition_network_kcmdline is unquoted and results in just the very first
karg to be consumed by the set command.
Signed-off-by: Antonio Murdaca <antoniomurdaca@gmail.com>
Extend the copy stage to optionally allow removing the destination
before copying. This allows one to not follow symlinks if the
destination is a symlink to a file. By default, `cp` would change
the file pointed to by the destination if it is symlink.
Extend the stage doc text to cover the behavior with regard to
destination being a symlink.
Add unit tests for the copy stage to also test the newly added option.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a top level property "files" to the schema and move the rest of the
existing schema one level down. This way we can support adding global
properties in the future if we ever need to expand the scope of the
stage.
Pattern for valid environment variable names as defined in
The Open Group Base Specifications Issue 7, 2018 edition
IEEE Std 1003.1-2017 (Revision of IEEE Std 1003.1-2008)
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html
Updated tests to match UPPERCASE ONLY var names.
