Commit graph

4522 commits

Author SHA1 Message Date
Sanne Raymaekers
fa18206918 osbuild-service-maintenance: Rely on the db's timezone in tests
Avoid calculating the new expire date on the test host, because
differences in timezones can yield unexpected failures.
2022-08-01 10:59:05 +02:00
Sanne Raymaekers
69d4429e8f osbuild-service-maintenance: Honor dryrun in db cleanup 2022-08-01 10:59:05 +02:00
Sanne Raymaekers
2eaad3701d osbuild-service-maintenance: Reword log messages and query name
The query is deleting multiple jobs, and it's deleting the entire job,
not just the results.
2022-08-01 10:59:05 +02:00
Tomas Hozza
322ae02f73 internal/container: delete leftover dead code
This issue was found by Coverity:

Error: DEADCODE (CWE-561): [#def1]
osbuild-composer-58/_build/src/github.com/osbuild/osbuild-composer/internal/container/client.go:386: cond_null: Condition "err != nil", taking false branch. Now the value of "err" is "nil".
osbuild-composer-58/_build/src/github.com/osbuild/osbuild-composer/internal/container/client.go:390: null: At condition "err != nil", the value of "err" must be "nil".
osbuild-composer-58/_build/src/github.com/osbuild/osbuild-composer/internal/container/client.go:390: dead_error_condition: The condition "err != nil" cannot be true.
osbuild-composer-58/_build/src/github.com/osbuild/osbuild-composer/internal/container/client.go:391: dead_error_line: Execution cannot reach this statement: "<temporary>.$0 = container....".
  389|
  390|   	if err != nil {
  391|-> 		return resolvedIds{}, nil
  392|   	}
  393|
2022-07-29 23:20:09 +02:00
Christian Kellner
e3de1d03c0 distro/rhel7: blacklist skx_edac,intel_cstate kernel modules on azure
This was done in RHEL 8, 9 but RHEL 7 was not yet merged so this was
missed there. See e.g. 5c1530ee53
2022-07-28 17:00:11 +01:00
Sanne Raymaekers
c78c69b273 osbuild-service-maintenance: Run vacuum analyze after each delete 2022-07-28 16:53:55 +02:00
Sanne Raymaekers
a221de5db7 templates/composer: Remove non-existent secret
The secret not existing causes the deployment to fail during a
validation stage.

```
[ERROR] [openshift_base.py:_validate_resources_used_exist] - [Deployment/composer] Secret db does not exist
```
2022-07-28 11:24:25 +02:00
Achilleas Koutsou
234e16f35e test/diff-manifests: fix error handling
The change made in 7f563a6db1 would
require the shell option `-e` to not be set, so that we could capture
the exit code after the command fails.
Fix the error handling by putting the commands that we want to handle in
the test part of an `if` clause.

In addition, error messages are now printed in red.
2022-07-28 10:12:17 +02:00
Achilleas Koutsou
dc95382ba3 rhel90: add kernel modules for iso
These modules are on the official RHEL 9.0 ISO.  Adding them for feature
parity.
2022-07-28 10:12:17 +02:00
Achilleas Koutsou
02bb7a0b4f rhel90: add dracut modules for iso
The nvdimm module is required for booting the image via UEFI HTTP.

The rest are added for feature parity with the official RHEL 9.0 ISO.

Fixes rhbz#2030730
2022-07-28 10:12:17 +02:00
Ondřej Budai
9def545570 dbjobqueue: fix bad errors.As usages
errors.As is meant to check whether err (or other error in its chain) can
be assigned to the value that target is pointing at.

Let's consider this example:

errors.As(err, &pgx.ErrNoRows)

pgx.ErrNoRows (and pgx.ErrTxClosed) is typed as error, thus in all
errors.As calls, the target is typed as *error. Err is always an error.
So this call is basically asking whether error can be assigned to error.
If err != nil, this is always true, thus this check doesn't make any sense
over a plain err != nil.

Go 1.19 now checks this issue and if it's found, it refuses to compile the
code, see:

https://go-review.googlesource.com/c/tools/+/339889

This commit changes usages of errors.As() to errors.Is(). The Is() method
doesn't check assignability but equality (the only different between Is()
and a plain old == operator is that Is() also inspects the whole error chain).

This fixes the check because now, we are basically checking if err (or
any other error in its chain) == pgx.ErrTxClosed which is exactly what we
want.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-07-27 18:29:59 +02:00
Achilleas Koutsou
93592dabc1 test: regenerate manifests 2022-07-27 18:29:59 +02:00
Achilleas Koutsou
c20e1e53c4 osbuild: use path as secondary sort key for fstab
Most filesystems entries in fstab don't have a PassNo, which makes the
order of those entries dependent on the sorting algorithm.  Changes in
the algorithm can introduce changes in the sort order, which we don't
like.

Add a secondary sorting key, the Path, which is guaranteed unique, to
guarantee stable ordering.
2022-07-27 18:29:59 +02:00
Gianluca Zuccarelli
e5d9d2d045 worker/server: rename JobStatus() to JobInfo()
Since the `jobStatus` functions return a `JobInfo`
struct that contains the `JobStatus`, it makes sense
to rename the function names for the sake of consistency.
2022-07-27 13:37:14 +02:00
Gianluca Zuccarelli
95c8657f9e metrics: remove arch from osbuild type
The osbuild jobtype currently contains the
architecture as a suffix. Since the arch
is now being supplied as a label, the
`arch` suffix can be removed.
2022-07-27 13:37:14 +02:00
Gianluca Zuccarelli
967ac1c35e worker/server: job status struct
The number of return values from the `jobStatus`
function was growing and getting out of hand. Not
all return values were being used in all cases
and so returning a single struct with the information
and status of a job makes more sense. Then in each case
the resulting fields can be used as needed.
2022-07-27 13:37:14 +02:00
Gianluca Zuccarelli
9f4e765657 metrics: build jobs arch label
Add the architecture label to build jobs
which will enable filtering and monitoring
build jobs by architecture. Build job results
contain the `arch` field in the results struct,
this is then used to pass to the metrics, where
there is a value, otherwise it is set to an
empty string.
2022-07-27 13:37:14 +02:00
Gianluca Zuccarelli
8b4aff3857 worker/server: remove duplicate metrics
Remove a duplicate call to the `DequeueJobMetrics`
function in the worker server. This duplicate call
resulted in negative numbers for pending jobs in
the prometheus metrics.
2022-07-27 13:37:14 +02:00
schutzbot
ac522a58ee Post release version bump
[skip ci]
2022-07-27 08:31:06 +00:00
Achilleas Koutsou
2b6faea320 Remove image info from all test manifests
We never generated it for the newest distro versions and we don't test
it on any of the old ones.

It's better this way.
2022-07-27 10:01:16 +02:00
Jakub Rusz
74bf3aa3df test/diff-manifests: checkout repo to $head on exit
During the diff-manifests.sh test the source repository checkout is
changed to generate manifests from current main branch for comparion. We
want to checkout back to $head after the script is done or in case of
any unexpected exit.
2022-07-26 17:08:32 +02:00
Jakub Rusz
cf562dbb57 CI: drop /tmp/artifacts upload to Gitlab
Contents of this folder are uploaded to S3 with shutzbot/upload_artifacts.sh
Also fixing epel installation in the mentioned script.
2022-07-26 17:08:32 +02:00
Xiaofeng Wang
c71ae8f455 test: Update tests to support embeded container image feature
Container image can only be embeded on commit Use fedora image
instead of ubi8 and only run embeded image checking on  RHEL
8.7, 9.1, CS8 and CS9.
2022-07-26 17:07:31 +02:00
Christian Kellner
89c2bb0d3e spec: require osbuild 62
We need the `org.osbuild.containers.storage.conf` stage and its
fixes that only landed in osbuild 62.
2022-07-26 17:07:31 +02:00
Christian Kellner
14931f1ebe ci: pin osbuild to include pytoml fix
osbuild commit 9956f54 includes a fix for the `containers.storage.conf`
to work with RHEL 8 by trying to include `pytoml` if including `toml`
fails. We need that for the RHEL 8 based container embedding on OSTree
tests.
2022-07-26 17:07:31 +02:00
Christian Kellner
34c94ab92b container: rework GetDefaultAuthFile and don't cache its result
Instead of using a cached result `GetDefaultAuthFile`, always
do call the function when a new `Client` is created, since at
least `/run/containers` can get created as a side-effect by
one of the container. Now that we check eagerly and often the
path check function was reworked to only return paths that do
exist and are accessible.

Also check if `REGISTRY_AUTH_FILE` is set and if so, and it
is accessible use that.

To check accessability, use `unix.Access` instead of `os.Stat`,
since On Fedora/RHEL 9 `os.Stat` is implemented via `statx` and
will indeed return `EACCES` for inaccessible paths. But on RHEL
8 `lstat` is used and that will return `ENOENT` but then later
when trying to open the file we will get `EPERM`.
2022-07-26 17:07:31 +02:00
Christian Kellner
62406070b6 distro/rhel9: support embedding containers in ostree commits
Add support for embedding containers in OSTree commits by
storing them in `/usr/share/containers/storage`. The storage
engine is configured accordingly so that this extra location
is automatically taken into account by e.g. `podman`.
2022-07-26 17:07:31 +02:00
Christian Kellner
d7f1ed8ba7 distro/rhel8: support embedding containers in ostree commits
Add support for embedding containers in OSTree commits by
storing them in `/usr/share/containers/storage`. The storage
engine is configured accordingly so that this extra location
is automatically taken into account by e.g. `podman`.
2022-07-26 17:07:31 +02:00
Christian Kellner
e290502a1d container: add a check to ensure auth file path is always set
We never want an empty path but always force a specific auth
file location, even if the location does not actually exist,
due to the peculiarities mentioned in the comment of the
`container.GetDefaultAuthFile` function.
2022-07-26 17:07:31 +02:00
Christian Kellner
e38e7c717d container: add a auth file path getter for Client
This is so we can check and verify what is set.
2022-07-26 17:07:31 +02:00
Christian Kellner
8a06b9ddf3 osbuild: add bindings for org.osbuild.containers.storage.conf stage
Add support for the `containers.storage.conf` stage with helper
constructors that should make it easy to use.
Add a small test for it.
2022-07-26 17:07:31 +02:00
Achilleas Koutsou
f3c166c3fd CI: fix artifact collection for diff-manifests 2022-07-25 21:23:02 +02:00
Achilleas Koutsou
46b127c898 diff-manifests: never use main explicitly
The base branch name is read from the pull request metadata.  Let's use
it everywhere and never assume `main`.
2022-07-25 21:23:02 +02:00
Achilleas Koutsou
b5c47dbe8a test/diff-manifests: warn if generation fails on merge-base
If the manifest generation fails on the merge-base with main (the target
branch of the PR), add a note to the review comment.
2022-07-25 21:23:02 +02:00
Achilleas Koutsou
404d1130bd test/diff-manifests: add link to job artifacts in review comment 2022-07-25 21:23:02 +02:00
Achilleas Koutsou
4c7fc8fd4d test/diff-manifests: fail test if manifest generation fails on PR
Fail the test if manifest generation fails on the PR HEAD, but don't
fail if the generation on main fails.
This can happen if something breaks in main (the generator, a
repository, an image definition, etc) and the PR is meant to fix it.
2022-07-25 21:23:02 +02:00
Achilleas Koutsou
7f563a6db1 test/diff-manifests: minor change in error handling 2022-07-25 21:23:02 +02:00
Achilleas Koutsou
68227ed18b test/diff-manifests: print gen-manifests output
Don't silence the output.  It should be much less noisy now that the
progress isn't printed on a timer but only when something changes.
2022-07-25 21:23:02 +02:00
Achilleas Koutsou
2afc8fd1dd gen-manifests: minor output changes
Don't print "Worker started" message.
Print "done" after submitting all jobs to the queue.
2022-07-25 21:23:02 +02:00
Achilleas Koutsou
b20605d9f1 gen-manifests: append message to failed jobs on finish
When the 'Finished' message is printed, append [failed] to the message
if the job encountered an error.
2022-07-25 21:23:02 +02:00
Achilleas Koutsou
20b6d3866a gen-manifests: support arch, distro, image type selection
Multiple values can be specified by separating with commas.
2022-07-25 21:23:02 +02:00
Achilleas Koutsou
a018c69f90 gen-manifests: simplify flag definitions
Remove the reliance on flag variable pointers and read the argument
values directly into the target variables.
2022-07-25 21:23:02 +02:00
Achilleas Koutsou
d61b553045 gen-manifests: don't print progress when nothing changed
Update the progress line only when another line was received, which in
this case means a job has started or finished.
No need to keep reprinting the progress.
2022-07-25 21:23:02 +02:00
Achilleas Koutsou
6e4a55cf9e gen-manifests: print errors on stderr 2022-07-25 21:23:02 +02:00
Christian Kellner
78a96a5414 worker: support for configuring the containers auth file
Add a new configuration section for containers and an option
to configure the authfile in it.
2022-07-25 21:21:44 +02:00
Christian Kellner
ec4598f81f weldr: use worker job to resolve container
The main reason is that there should be only one place where the
container resolution is happening, which is the worker, so that
we only have one central place to configure aspects of it, like
container credentials.
2022-07-25 21:21:44 +02:00
Christian Kellner
50e630a76f worker: add new container resolve job type
This is a new job that can be used to resolve containers. It uses
the existing `container.Resolver` class to do the actual work.
2022-07-25 21:21:44 +02:00
Christian Kellner
c2f3f76d96 container: ability to set AuthFilePath for Resolver
Add a new field `AuthFilePath` and if it is not the empty string,
it will be forwaded that information to `Client`.
2022-07-25 21:21:44 +02:00
Christian Kellner
2c0594629f osbuild: add extraEnv argument to RunOSBuild
This adds the ability to supply extra environment variables to
the osbuild process.
2022-07-25 21:21:44 +02:00
Tomas Hozza
31072c1189 Cloud API: extend a code comment with more information 2022-07-24 08:40:58 +02:00