When the urls' secrets field is not set, an error is thrown when trying
to get the name of the secrets. The secrets now have a default value of
{} when they are checked for the name.
We are now building RPMs via mock in Jenkins, so we don't need these RPM
build jobs in GitHub Workflows.
Signed-off-by: Major Hayden <major@redhat.com>
Build the RPMs in a mock using a simple script so that ansible-osbuild
can focus fully on deployment rather than compiling RPMs.
Signed-off-by: Major Hayden <major@redhat.com>
Work around the quay.io issues by using the standard Fedora containers.
Also, make most of the dnf operations a little quieter to make it easier
to find problems.
Signed-off-by: Major Hayden <major@redhat.com>
Major changes are the new assembler, support for passing client
certificates to access Red Hat content in the files source and
the shiny new ManifestPreProcessor.
Add a new assembler that takes a tree and creates a Open Container
Initiative[2] image according to the OCI image format[2]. The final
result is a tarball, aka a "orci-archive", that can be pulled into
podman with `podman pull oci-archive:<archive>`. Currently the only
required options are `filename` and `architecture`.
[1] https://www.opencontainers.org/
[2 ]https://github.com/opencontainers/image-spec/
We want to get rid of `tree_id` and `output_id` because the they
are now considered internals of the store and clients should not
use them directly. NB: they are still there indirectly as the id
of the last stage and the assembler.
Also, the `output_id` was never correct here, because it was the
`tree_id` as well. Ups.
This adds F32 manifests in ./test/data/. To avoid magically deducing the
package list out of the void, this adds a ManifestPreProcessor (MPP)
called `./tools/mpp-depsolve.py`. What this does is it takes a manifest
on stdin, modifies it, and produces a manifest on stdout.
The `mpp-depsolve.py` preprocessor takes a manifest and modifies all the
`org.osbuild.rpm` stages. It parses a new option to that stage called
`mpp-depsolve`, which contains a package-list, a repo-list, and dnf
metadata. It then drops this `mpp-depsolve` option (since it would be an
invalid manifest otherwise), depsolves the packages, inserts a proper
"packages" option as well as appends the correct paths to the sources
entry.
With this in place, this adds `mpp-f32-base.json` and
`mpp-f32-build.json` in ./test/data/manifests/. These will then be used
as base F32 manifests for our test-suite.
Lastly, this adds `./test/data/README.md` as a place to document the
files we place in `./test/data/`, since most of the files do not allow
for comments.
When osbuild is given a manifest, the sources' urls can contain fields
for both a url path and a secret for that url. If the secret is
org.osbuild.rhsm the system's rhsm certificates are retrieved. These
certs are included when the files are curled.
Introduce a new `tar` option, which when given together with the
required `tar.filename` option, will result in the output of the
assembler being a tarball that contains the repo and the compose
information (`compose.json`).
Requires the `tar` command to be present in the build root. Modify
the sample to use that option and include the tar for the build
pipeline.
The `tree-diff` tool currently requires access to our local checkout,
since we do not install the tool. Provide accessors in `TestBase` so we
do not hard-code the path everywhere.
Add a new OSBuild class to `./test/test.py`. This class is an extension
of `./test/osbuildtest.py`, but no longer requires the `output_id` and
`tree_id` identifiers of osbuild.
Furthermore, this new executor uses context-managers to make sure any
temporary object is only accessed for a contained time-frame.
Make sure to verify that the pipeline actually produced any output
before attempting to copy it out. This fixes osbuild running with
`--output-directory` but without assembler.
The idea is that source can themselves spawn other modules, esp.
new secrets modules. For this they need to know the library dir,
aka 'libdir' throughout the osbuild source. Therefore change the
SourceServer to directly get the library directory instead of
just the sub-directory to the sources. Then pass the library
directory to via the JSON API to the source.
Adjust all usage of the SourceServer, including the tests.
Add the artifact preservation bits from the osbuild-composer jenkinsfile
and use the shared `run_tests()` function to avoid repeating ourselves.
Signed-off-by: Major Hayden <major@redhat.com>
The NEWS file currently uses non-markdown indentation, but uses the
markdown file-format. Since both GitHub-Repository-Viewer and the
Release Pages can display proper markdown, lets just re-indent the file
according to the markdown specs.
While at it, also switch out the list-indicator in front of the
timestamp to a proper hyphen, to avoid it being rendered as list.
When Jenkins clones a repository for testing, it does the base clone
first and then merges the code from the PR afterwards. This ensures that
the code merges properly and is tested properly, but it also makes a SHA
that only exists inside Jenkins. 😢
Tell ansible-osbuild to use the repository that Jenkins made so that the
SHA is valid.
Signed-off-by: Major Hayden <major@redhat.com>
Tell ansible-osbuild to build osbuild/osbuild-composer RPMs in a mock
chroot and install them from there instead of building them with
Ansible.
This brings us closer to how koji and brew build RPMs.
Signed-off-by: Major Hayden <major@redhat.com>
Move the `test_boot` test from Travis over to the Github-Actions based
CI. This is the last test on Travis, and the Travis CI can now be
disabled, if we so wish.
This test leaves a valid `travis.yml` file around, since Travis will
still be enabled on the repository. We should first disable Travis and
then drop this file, if we want to get rid of it.
Use the images provided by `osbuild/containers` tagged as GHCI (GitHub
CI). These images are fully under our control, cached on the GitHub
infrastructure, and prepared to run `systemd-nspawn` and friends in a
docker container.
The GHCI infrastructure is versioned. New updates to the CI
infrastructure are not automatically picked up. Instead, the `v1` tag
has to be explicitly redirected to new image builds to deploy them. If
a new deployment causes CI failures, we can simply redirect the `v1` tag
back to the previous image builds and get the previous behavior back.
The `osbuild/containers` repository contains the required
infrastructure for this logic. If new dependencies are required in the
CI environment, the respective Dockerfiles must be updated. As a
temporary workaround (e.g., as part of a PR that introduces this), you
can simply add `dnf install -y <package>` to the required entries in
`.github/workflows/*`.
This changes the `modprobe nbd` invocation to be non-fatal on failure,
since it might very well fail on reasonable setups. `modprobe` fails if
it cannot find a module in `/lib/modules`, even if it could reasonably
well figure out whether a module is already loaded. The reason is that
it needs the metadata from the module file to find the required modules
parameters.
If you have `nbd` already loaded but not mapped in `/lib/modules`, the
current call will cause test failures, even though the test would run
smoothly.
Fix this by never requiring `modprobe nbd` to succeed, but instead rely
on the tests failing if accessing `nbd` fails.
This drops the directory './test/testing-rpms'. The directory was
introduced in:
commit d975effc42
Author: Martin Sehnoutka <sehnoutka.martin@gmail.com>
Date: Thu Jul 25 11:12:27 2019 +0200
improve vagrant test and its documentation
It used to be the automatic target directory to store rpms created via
`make copy-rpms-to-test`. This target no longer exists. It was dropped
in:
commit 59b7b545b2
Author: Lars Karlitski <lars@karlitski.net>
Date: Fri Mar 6 11:07:52 2020 +0100
Makefile: remove vagrant rules
Introduce a third test-group called `src` alongside `mod` and `run.
This will contain tests that run against the source code of osbuild.
This initial commit introduces `test/src/test_pylint.py` which will run
the python linter against all our sources.
Use the new `locate_test_data()` helper to get access to test-data.
Guard the test with `have_test_data()` to skip it in case test-data
access is not available.
Use the `can_modify_immutable()` helper from the TestBase parent class
so we do not duplicate the code in multiple places. Similarly, make use
of the `have_rpm_ostree()` helper.
Add a new base class called `TestBase` to our test-suite. This allows
sharing common code between our tests without requiring them to import
each other. Furthermore, it paves the way towards executing all our
tests as part of the `unittest` framework, including pylint and others.
For now, this adds the following features to `TestBase`:
* Common test-guards that are shared between our tests, like
`can_modify_immutable()` or `have_rpm_ostree()`.
* Accessors to the test-checkout. This is `have_test_checkout()` to
check whether the running test has a repository checkout, and
`locate_test_checkout()` to get a path to the repository checkout.
This will allow us to put pylint and friends into the unittest
framework, guard them properly, and still allow running the tests
from a global install which might not have access to a checkout.
For now, we always assume we run from a checkout.
* Accessors to test-data. If we start installing tests as a module
into the system, we cannot bundle test-data together with code.
Therefore, two accessors `have_test_data()` and `locate_test_data()`
are implemented to guard access to test data. If a checkout is
available, it will be used to locate test-data.
In the future, we want to be able to pass a separate path to the
test-data, thus allowing us to install tests into a system.
Drop the `kwargs` forwarding from buildroot.run() to subprocess.run().
We do not use it other than for `stdin=subprocess.DEVNULL`. Set that
option directly instead.
Doing the kwargs forwarding mixes the argument namespaces and is very
hard to read. It is not clear from the call-site which argument goes to
buildroot.run() and which to subprocess.run().
Lastly, it requires us to manually fetch `check` just to make pylint
happy. Lets just drop this dance and make the API explicit.
We currently don't seem to use anything that requires us to use
the draft 7 of the specification. The minimum version that we
need is draft 4, which is also supported by the python-jsonschema
version in RHEL 8.2 (which is 2.6.0).
Attempt osbuild testing on the internal Jenkins deployment with
nodes that are destroyed after each use. The internal Jenkins looks for
a Jenkinsfile inside the `schutzbot` directory.
Let's not remove the `jenkins` directory (used by jenkins.osbuild.org)
yet until we know the internal Jenkins is stable and performs well.
Signed-off-by: Major Hayden <major@redhat.com>
Extract the `suppress_oserror()` function from the ObjectManager and
make it available as utility for other code as well.
This also adds a bunch of tests that verify it works as expected.
This changes `-chardev stdio` to `-chardev file` and uses a temporary
file to communicate with QEMU.
This fixes an issue where `-chardev stdio` hangs if `STDIN` is not a
TTY. I could not figure out how to make it work without a TTY, and it
does not print any meaningful diagnostics. Problem is, in CI and other
automated runners, we do not necessarily have a TTY as STDIN.
This just switches to a temporary file, which seems to work under all
circumstances.
Drop the --build-env command-line argument. It is not used by anything.
Furthermore, our manifests now allow embedding build-environments, so
there is little reason to continue supporting this.
Now that as a result of commit 4d2f15f all symlinks have been
dropped from the individual module paths, the search for module
contents can be simplified again.
In case `--libdir` is not specified on the command line, and thus
`args.libdir` is `None`, pass the standard `/usr/lib/osbuild` path
to the meta.Index constructor. Otherwise no schema information can
be found.
By using a small Jenkins pipeline in the repository, we can define
almost all of our testing parameters in the repo itself and not inside
Jenkins. 🥳
This also allows us to use the GitHub Branch Source plugin and
auto-discover new repositories without `ok to test` bombs in
pull requests.
Signed-off-by: Major Hayden <major@redhat.com>
