osbuild-mpp supports yaml files. The advantage of yaml is that it's less
verbose and allows comments. This way, we can start documenting all the
test manifests in this repository.
This is the first step: This commit just renames the files, so we keep the
history intact. Remember: YAML is a super-set(*) of JSON, so this is fine.
There is no functional change in this commit.
(*) There are some caveats, but we don't need to care in this case.
`tox` is a standard testing tool for Python projects, this allows you to
test locally with all your installed Python version with the following
command:
`tox -m test -p all`
To run the tests in parallel for all supported Python versions.
To run linters or type analysis:
```
tox -m lint -p all
tox -m type -p all
```
This commit *also* disables the `import-error` warning from `pylint`,
not all Python versions have the system-installed Python libraries
available and they can't be fetched from PyPI.
Some linters have been added and the general order linters run in has
been changed. This allows for quicker test failure when running
`tox -m lint`. As a consequence the `test_pylint` test has been removed
as it's role can now be fulfilled by `tox`.
Other assorted linter fixes due to newer versions:
- use a str.join method (`consider-using-join`)
- fix various (newer) mypy and pylint issues
- comments starting with `#` and no space due to `autopep8`
This also changes our CI to use the new `tox` setup and on top of that
pins the versions of linters used. This might move into separate
requirements.txt files later on to allow for easier updating of those
dependencies.
Add a new stage to generate an OpenSCAP tailoring file. The stage
overrides a base OpenSCAP profile by enabling and disabling user
selected rules and creates a new profile name which can be used for
OpenSCAP scanning and remediation.
This is a straightforward port of the UKI from Fedora to CentOS Stream.
Changes:
- Packages that does not exist in CS9 were removed:
- shim-ia32
- grub2-efi-ia32-cdboot
- btrfs-progs
- Squashfs compression was changed from lz4 to gzip, because lz4 for squashfs
isn't enabled in RHEL 9 kernel
Refactor unit test implementation for `parted`, `sfdisk` and `sgdisk`
stages by extracting the common parts into a helper function. Each stage
now implements only its own function for filtering `sfdisk --json`
output and calls the common helper function.
In addition, flip the oder when comparing the expected and actual output
from `sfdisk --json`, to make it more easier to comprehend.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The `sfdisk` tool on RHEL-8 does not include the `sectorsize` in its
output when we are testing the `sgdisk` stage. As a result, the test
case fails, because the expected and actual output differs. Modify the
test to delete the `sectorsize` key from the expected output if it is
not present in the actual output.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The `sfdisk` tool on RHEL-8 does not include the `sectorsize` in its
output when we are testing the `parted` stage. As a result, the test
case fails, because the expected and actual output differs. Modify the
test to delete the `sectorsize` key from the expected output if it is
not present in the actual output.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The partition name in the sgdisk stage was previously quoted when
passed to sgdisk as an argument. I think that this was done because
the sgdisk man page states that:
```
If you want to set a name that includes a space, enclose it in
quotation marks, as in sgdisk -c 1:"Sample Name" /dev/sdb.
```
However, this should apply only when sgdisk is run in a shell, so that
the argument is not split by shell into multiple arguments and passes
as a single string.
The stage is executing sgdisk using Python `subprocess` module, which
does not need strings with spaces to be quoted, because they are passed
to the command as separate items which are not split in any way.
The previous behavior of the stage was that these quotes became part of
the actual partition name in the partition table.
After a discussion within the team, we determined that this is a bug.
However, fixing it would result in osbuild producing a different
artifact for the same manifest, compared to osbuild version without such
fix. This is undesired.
For backward compatibility, a new `quote_partition_name` property is
added to the stage options, which can be used to make the stage not
quote the partition name when passed to `sgdisk`. As a result, the
partition name won't be quoted in the partition table.
The default stage behavior us kept.
Modify unit tests to use this option by default.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
LOOP_CONFIGURE allows to atomically configure the decive when opening
it. This avoid the possibility of a race condition where between set_fd
and set_status some operations are already accepted by the loopback
device. See https://lwn.net/Articles/820408/
This feature was included in the linux kernel 5.8 however it is safe to
not include any kind of fallback to the previous method as @obudai
points out that:
LOOP_CONFIGURE was backported into RHEL 8 kernel in RHEL 8.4 as a part
of https://bugzilla.redhat.com/show_bug.cgi?id=1881760 (block layer:
update to upstream v5.8).
Since RHEL 8.4 is currently the oldest supported release that we support
running osbuild on, it might be just fine implementing this without the
fallback.
From a centos stream 8 container:
kernel-4.18.0-448.el8.x86_64
- loop: Fix missing discard support when using LOOP_CONFIGURE (Ming Lei) [1997338]
- [block] loop: Set correct device size when using LOOP_CONFIGURE (Ming Lei) [1881760]
- [block] loop: unset GENHD_FL_NO_PART_SCAN on LOOP_CONFIGURE (Ming Lei) [1881760]
- [block] loop: Add LOOP_CONFIGURE ioctl (Ming Lei) [1881760]
Add conditional skip to some tests that depend on rpm-ostree
availability, but were not checking for its presence. These tests would
previously fail if rpm-ostree is not available. They will be skipped
now.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The `directory` argument has been added only since Python 3.7, which
breaks the unit test on Python 3.6.
Reimplement the intended behavior by overriding the `translate_path()`
method, which takes the `directory` value into account on newer Python
versions.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Do not specify the default value for 'expected_size' argument in
assertImageFile() function declaration. Previously, it was set to
`None`, which was never taken into account. Moreover, all callers of the
function always provide an explicit value, so the default was never
really used.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a new optional pytest CLI argument `--unsupported-fs` allowing to
specify file-systems which should be treated as unsupported in the
platform where running tests. Any test cases dependent on such
file-system support will be sipped.
This will allow to run unit tests and selectively skipping test cases
for unsupported file-systems.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Port assembler tests from unittest to pytest. In addition, use
parametrized tests when testing various filesystems and various
combinations.
This is important to be able to selectively skip the test for if a
specific filesystem is not supported by the kernel (e.g. btrfs is not
supported on RHEL). Skipping a unittest subtest is not possible, which
is the motivation to move away from it and use only pytest.
Test output is now also much nicer for parametrized test cases.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Previously, the unit test depended on osbuild modules being installed on
the system. As a result, this made the test not work in CI where we do
not install osbuild when running unit tests. In addition, the stage
executed by the unit test would use different version of osbuild
internals than the version that is being tests, which could result in
issues or not testing the intended code.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The directory does not exist when the unit test is run in CI. Handle
this case by ensuring that parent directories are created as needed.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The right way to enable services is to use a preset file instead of
writing directly into /etc. This adds a new stage called
`org.osbuild.systemd.preset` to do so.
Added another skopeo stage to skopeo/a.mpp.json with a skopeo source for
a container hosted on the osbuild-composer gitlab registry. The name
points to a manifest list, which refers to two containers (amd64 and
arm64) that contain a single text file (README.md). The `index` field
is enabled to include the manifest-list as an extra input to the stage.
The diff is updated with the new expected file list.
The containers were created with buildah:
amd=$(buildah from --arch=amd64 scratch)
arm=$(buildah from --arch=arm64 scratch)
buildah config --created-by "Achilleas Koutsou" "${amd}"
buildah config --created-by "Achilleas Koutsou" "${arm}"
buildah copy "${amd}" README.md
buildah copy "${arm}" README.md
amdid=$(buildah commit --format=docker --rm "${amd}")
armid=$(buildah commit --format=docker --rm "${arm}")
name="registry.gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/manifest-list-test"
buildah manifest create "${name}" "${amdid}" "${armid}"
podman manifest push --all "${name}" dir:container
Extend the copy stage to optionally allow removing the destination
before copying. This allows one to not follow symlinks if the
destination is a symlink to a file. By default, `cp` would change
the file pointed to by the destination if it is symlink.
Extend the stage doc text to cover the behavior with regard to
destination being a symlink.
Add unit tests for the copy stage to also test the newly added option.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add a top level property "files" to the schema and move the rest of the
existing schema one level down. This way we can support adding global
properties in the future if we ever need to expand the scope of the
stage.
Pattern for valid environment variable names as defined in
The Open Group Base Specifications Issue 7, 2018 edition
IEEE Std 1003.1-2017 (Revision of IEEE Std 1003.1-2008)
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html
Updated tests to match UPPERCASE ONLY var names.
Add a new `org.osbuild.chown` stage for setting user and group ownershop
of files. The stage runs the `chown` from the image using `chroot` to
enable it to use users and groups that exist only in the (image) tree.
Add unit test testing the stage in various scenarios.
Co-authored-by: Janine Olear <pninak@web.de>
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Before we could only ask OSBuild to mount a device as readonly. But
devices can have more mount options than this. Supporting more options
is necessary for the new version of image-info that is using OSBuild's
internals in order to mount the image it wants to work on. Otherwise,
for instance, some umasks aren't applied properly and we can get
differences in rpm-verify results, thus corrupting the DB.
Mount is now accepting:
* readonly
* uid
* gid
* umask
* shortname
This reverts commit a988aacf99.
After some discussion, the original behavior was intentional. With the
added support for gracefully handling the existence of directories, the
stage would originally not set the mode of an existing directory, while
now it will. Additional issue is that `mkdir` applies the provided mode
- umask, which was intentional. Setting the same mode without taking
umask value into account is not desired.
Add a new optional stage option to not fail if the specified directory
already exists. This will make it easier to support creation of custom
repositories via customizations in osbuild-composer. The reason is that
if a specified directory exists in an image, because it was created by
an RPM, then creating it would fail. However, the user may have
specified different mode for the directory, than it already has. Since
there is no way to know for sure if the directory already exists on the
image, without building the image itself, it is desired to handle this
case gracefully as valid in specific use cases.
The default behavior stays the same - specifying an existing directory
path will lead to an error.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Instead of using `Path.stat` use `os.stat` since the former only
gained the `follow_symlinks` argument in 3.10 but we still need
to support Python 3.6 for RHEL 7 and 8.
Additionally, reduce the precision by converting timestamps to an
integer to avoid false negatives due to floating point arithmetic.
The cachedir-tag specification defines how to mark directories as
cache-directories. This allows tools like `tar` to ignore those
directories if desired (e.g., see `tar --ignore-caches`). This is very
useful to avoid huge cache-directories in backups and remote
synchronizations.
The spec simply defines a file called `CACHEDIR.TAG` with the first 43
bytes to be: "Signature: 8a477f597d28d172789f06886806bc55" (which
happens to be the MD5-checksum of ".IsCacheDirectory". Further content
is to be ignored. Any such files marks the directory in question as a
cache-directory.
The cachedir-tag has been successfully deployed in tools like `cargo`
and `VLC`, and is currently discussed to be implemented in Firefox. More
information is available here: https://bford.info/cachedir/
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
Add an extension to the FsCache tests which verifies cache coherency and
atomicity of the FsCache implementation. Additionally, if available, it
utilizes a cache on NFS storage to test network-support.
Unfortunately, the stress-tests keep triggering kernel-oopses in the NFS
client driver, so they are disabled for now. However, once investigated,
we can re-enable them.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
Add a helper that copies an entire directory tree including all metadata
into the cache. Use it in the ObjectStore to commit entries.
Unlike FsCache.store() this does not require entering the context from
the call-site. Instead, all data is directly passed to the cache and the
operation is under full control of the cache.
The ObjectStore is adjusted to make use of this. This requires exposing
the root-path (rather than the tree-path) to be accessible for
individual objects, hence a `path`-@property is added alongside the
`tree`-@property. Note that `__fspath__` still refers to the tree-path,
since this is the only path really required for outside access other
than from the object-manager itself.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
Add a new `source_epoch` attribute that if set, will lead to all
mtimes that are newer or equal to the creation date being clamped
to the specified `source_epoch` time when the object is finalized.
New utility function to clamp all mtimes of a given path to a
certain timestamp. Clamp here means that any timestamp later
than the specified upper bound will be set to the upper bound.
Add a new field to the cache-information called `version`, which is a
simple integer that is incremented on any backward-incompatible change.
The cache-implementation is modified to avoid any access to the cache
except for `<cache>/staging/`. This means, changes to the staging area
must be backwards compatible at all cost. Furthermore, it means we can
always successfully run osbuild even on possibly incompatible caches,
because we can always just ignore the cache and fully rely on the
staging area being accessible.
The `load()` method will always return cache-misses. The `store()`
method simply discards the entry instead of storing it. Note that
`store()` needs to provide a context to the caller, hence this
implementation simply creates another staging-context to provide to the
caller and then discard. This is non-optimal, but keeps the API simple
and avoids raising an exception to the caller (but this can be changed
if it turns out to be problematic or unwanted).
Lastly, the `cache.info` field behaves as usual, since this is also the
field used to read the cache-version. However, this file is never
written to improve resiliency and allow blacklisting buggy versions from
the past.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
Port the existing object store tests from `unittest` to `pytest`.
Allow all tests that can run without root privileges to do so. No
functional change of the test itself.
Integrate the recently added file system cache `FsCache` into our
object store `ObjectStore`. NB: This changes the semantics of it:
previously a call to `ObjectStore.commit` resulted in the object
being in the cache (i/o errors aside). But `FsCache.store`, which
is now the backing store for objects, will only commit objects if
there is enough space left. Thus we cannot rely that objects are
present for reading after a call to `FsCache.store`. To cope with
this we now always copy the object into the cache, even for cases
where we previously moved it: for the case where commit is called
with `object_id` matching `Object.id`, which is the case for when
`commit` is called for last stage in the pipeline. We could keep
this optimization but then we would have to special case it and
not call `commit` for these cases but only after we exported all
objects; or in other words, after we are sure we will never read
from any committed object again. The extra complexity seems not
worth it for the little gain of the optimization.
Convert all the tests for the new semantic and also remove a lot
of them that make no sense under this new paradigm.
Add a new command line option `--cache-max-size` which will set
the maximum size of the cache, if specified.
Code is based on `common.DataSizeToUint64` in Composer, with a
modification to allow `unlimited` so that the result is compatible
with `fscache.MaximumSizeType`.
[1] f4aed3e6e2/internal/common/helpers.go (L46)
In the `store_server` test, pass the store to `enter_context`,
instead of the `stack`; the latter is an interesting form of
recursion, and totally not what we want.
