All EXIT traps are cleared on line 280 so the cleanup trap is never run
and VMs are waiting for 4 hours to get cleaned by
scheduled-cloud-cleaner. Run the cleanup at the end and rely on
scheduled-cloud-cleaner only in case of failures before that.
instead of manually updating CIV version every once in a while. Get
always the latest version.
In CIV CI, this test runs before any change can be introduced into the
container image, so no unexpedted errors should come from the CIV side.
When the user specifies any of the distro, arch, or image type values to
filter generation, invalid combinations would cause a panic, which made
it hard to filter requests based just on an image type.
Instead of failing, print an error message to inform the user, but
continue with the rest of the jobs.
This way, a user is informed that a certain combination is invalid if
they make a mistake, but can also filter on a single image type and only
get valid manifests out of the run.
Version 5.22 introduced a new option to /etc/containers/policy.json called
keyPaths, see
https://github.com/containers/image/pull/1609
EL9 immediately took advantage of this new feature and started using it, see
04645c4a84
This quickly became an issue in our code: The go library (containers/image)
parses the configuration file very strictly and refuses to create a client
when policy.json with an unknown key is present on the filesystem. As we
used 5.21.1 that doesn't know the new key, our unit tests started to
failing when containers-common was present.
Reproducer:
podman run --pull=always --rm -it centos:stream9
dnf install -y dnf-plugins-core
dnf config-manager --set-enabled crb
dnf install -y gpgme-devel libassuan-devel krb5-devel golang git-core
git clone https://github.com/osbuild/osbuild-composer
cd osbuild-composer
# install the new containers-common and run the test
dnf install -y https://kojihub.stream.centos.org/kojifiles/packages/containers-common/1/44.el9/x86_64/containers-common-1-44.el9.x86_64.rpm
go test -count 1 ./...
# this returns:
--- FAIL: TestClientResolve (0.00s)
client_test.go:31:
Error Trace: client_test.go:31
Error: Received unexpected error:
Unknown key "keyPaths"
invalid policy in "/etc/containers/policy.json"
github.com/containers/image/v5/signature.NewPolicyFromFile
/osbuild-composer/vendor/github.com/containers/image/v5/signature/policy_config.go:88
github.com/osbuild/osbuild-composer/internal/container.NewClient
/osbuild-composer/internal/container/client.go:123
github.com/osbuild/osbuild-composer/internal/container_test.TestClientResolve
/osbuild-composer/internal/container/client_test.go:29
testing.tRunner
/usr/lib/golang/src/testing/testing.go:1439
runtime.goexit
/usr/lib/golang/src/runtime/asm_amd64.s:1571
Test: TestClientResolve
client_test.go:32:
Error Trace: client_test.go:32
Error: Expected value not to be nil.
Test: TestClientResolve
When run with an older containers-common, it succeeds:
dnf install -y https://kojihub.stream.centos.org/kojifiles/packages/containers-common/1/40.el9/x86_64/containers-common-1-40.el9.x86_64.rpm
go test -count 1 ./...
PASS
To sum it up, I had to upgrade github.com/containers/image/v5 to v5.22.0.
Unfortunately, this wasn't so simple, see
go get github.com/containers/image/v5@latest
go: github.com/containers/image/v5@v5.22.0 requires
github.com/letsencrypt/boulder@v0.0.0-20220331220046-b23ab962616e requires
github.com/honeycombio/beeline-go@v1.1.1 requires
github.com/gobuffalo/pop/v5@v5.3.1 requires
github.com/mattn/go-sqlite3@v2.0.3+incompatible: reading github.com/mattn/go-sqlite3/go.mod at revision v2.0.3: unknown revision v2.0.3
It turns out that github.com/mattn/go-sqlite3@v2.0.3+incompatible has been
recently retracted https://github.com/mattn/go-sqlite3/pull/998 and this
broke a ton of packages depending on it. I was able to fix it by adding
exclude github.com/mattn/go-sqlite3 v2.0.3+incompatible
to our go.mod, see
https://github.com/mattn/go-sqlite3/issues/975#issuecomment-955661657
After adding it,
go get github.com/containers/image/v5@latest
succeeded and tools/prepare-source.sh took care of the rest.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
See https://github.com/BurntSushi/toml/issues/360
A recent change in BurntSushi/toml made encoding fail (later changed to
error) if a struct is marked as omitempty and is comparable. Go docs about
equality: https://go.dev/doc/go1#equality. Basically: A struct is comparable
if all of its fields are comparable. Slices are not comparable.
Customizations are marked as omitempty but they contain a lot of slices,
thus they are not comparable. The new version of BurntSushi/toml therefore
panics when we encode them.
The solution is to remove the omitempty tag from Customizations.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
When running an osbuild job, we read `/etc/redhat-release` to get the
host OS name to attach as metadata to the job result.
Only Fedora and RHEL ship this file, which makes the osbuild job always
fail on other distributions.
The main reason to report host OS back to the worker server is due to
Koji composes and the koji-finalize job, which pushes it to Koji. The
motivation is to have enough information to potentially re-instantiate
/ identify the original builder host OS. There are no specific
requirements on the string.
Modify the code to use `/etc/os-release` to determine the host OS. Fall
back to using `linux` as the host OS, in case reading `os-release`
fails, log the error and continue with the job. The `linux` fallback is
suggested by the `os-release` spec [1]
[1] https://www.freedesktop.org/software/systemd/man/os-release.html#ID=
Co-authored-by: Achilleas Koutsou <achilleas@koutsou.net>
some scripts skip the test if it's not supported for that
distro-version. Disable them in gitlab-ci.yml so we don't waste CI
resources.
To disable them, we are using the `rules` on each job with a regex
pattern. Using `=~` (pattern matches) as a WHITELIST and `!~` (pattern
does not match) as a BLACKLIST.
The search response from mocks/dnfjson is a map of responses indexed by the
comma-separated list of packages and globs being requested. Add support
for this.
The search command is more complicated than depsolve and dump. It needs
to return results based on the requested package names and globs.
Add a number of mock responses for the new search command, including
search results, all packages, and error responses that are triggered by
using special package names: nonexistingpkg, badpackage1, baddepsolve.
Instead of fetching all available packages from dnf-json and then
searching the results this uses SearchMetadata when a package name or
glob is passed to the API. It only uses FetchMetadata when fetching
the full list of packages.
This also fixes a bug where the error response to a projects/info
request used the id of 'ModulesError'. It now uses 'ProjectsError'.
Use the DNF query API
(https://dnf.readthedocs.io/en/latest/api_queries.html) to quickly
return results matching a glob pattern. Multiple package glob results
are combined into a single response.
This adds a search dict to the arguments. 'packages' is a list of package
names or globs to search for.
An optional 'latest' boolean will return only the latest NEVRA instead
of all matching builds in the metadata.
eg.
"search": {
"latest": false,
"packages": ["tmux", "vim*", "*ssh*"]
},
Since the oscap remediation stage in osbuild runs
the oscap package in `chroot`, it is necessary to
install the `openscap-scanner` package to the image
itself rather than the build root.
The module is not present in official RHEL-9.1 ISO image and it is
causing boot issues when used with newer content. HTTP boot is
not affected by this change and works as expected.
this is no longer needed b/c the nightly CI jobs are now using the same
json definitions as the regular CI jobs, just changing baseurl. See
previous commit.
Having the GPG check enabled for Google repos in `gce*` images will make
DNF try to import the relevant keys when upgrading, downgrading or
installing any packages from the repo. However due to Google still using
SHA-1 for GPG keys used to sign their RPMs, importing it will make any
transaction that includes such RPM to fail.
Disabling the GPG check will ensure that DNF won't attempt to import
Google GPG keys.
Related to https://issuetracker.google.com/issues/223626963
The repo is not needed any more, because the Google Cloud SDK is not
installed in the images by default. If anyone wants to install the SDK,
they can add the appropriate repo definition.
The repo is not needed any more, because the Google Cloud SDK is not
installed in the images by default. If anyone wants to install the SDK,
they can add the appropriate repo definition.
The Google SDK ships pre-compiled binaries. It is undesirable to install
it by default in `gce` and `gce-rhui` in its current shape. Also not
installing it does not anyhow affect the RHEL integration as the guest
OS in GCP.
The Google SDK ships pre-compiled binaries. It is undesirable to install
it by default in `gce` and `gce-rhui` in its current shape. Also not
installing it does not anyhow affect the RHEL integration as the guest
OS in GCP.
