Commit graph

4538 commits

Author SHA1 Message Date
Sanne Raymaekers
1ab3138afe cloudapi: Add test for osbuild job failure due to target errors
The target errors are attached to the details of the osbuild job error.
2022-08-03 13:51:52 +02:00
Sanne Raymaekers
24fab1d5b9 cloudapi: Omit details in compose status job error if nil 2022-08-03 13:51:52 +02:00
Sanne Raymaekers
7b73cf395f cloudapi: Correctly test dependency errors
Dependency errors are not set by the workers, they're not set directly
in the job result. They are added by the worker server in case the job
error indicates it's a dependency error.
2022-08-03 13:51:52 +02:00
Sanne Raymaekers
111feda1f5 worker: Remove ellipsis operator from clienterrors.Error
The ellipsis operator was used as a hack to not need to pass any details
as an argument, but it makes what the end object will actually look like
less obvious. It also makes it impossible to pass an array to details
without getting a nested array.

Fixes #2874
2022-08-03 13:51:52 +02:00
Gianluca Zuccarelli
1fb6a574cb templates: filter worker dashboard on arch
Add the ability to filter the build job
types by architecture using the `arch`
dropdown.
2022-08-03 13:38:52 +02:00
Xiaofeng Wang
0fe36359ae test: Remove BIOS installation test because edge-installer supports
UEFI only

Due to Edge security reason, edge-simplified-installer and
edge-installer support UEFI only
2022-08-02 09:22:24 +02:00
Ygal Blum
ca32d7b729 cloudapi: Make container name and tag optional
If the user does not pass a name, use the distribution as a name
A provided tag is used only if name is provided. It
The tag's default is a generated using UUID to avoid collisions
2022-08-01 21:50:03 +01:00
Ygal Blum
1847a6ae05 tests/api: add container related test
Add a new cloud API test that will build an edge-container,
upload it to the gitlab CI registry, fetch it from there,
run it and compare that the OSTree commit contained in it
is indeed the one we expect.

Co-Developed-By: Christian Kellner <christian@kellner.me>
2022-08-01 21:50:03 +01:00
Christian Kellner
c845a11881 job/osbuild: better logging for container uploads
Add some more logging to the container uploads, which is esp.
helpful in the case something goes wrong.
2022-08-01 21:50:03 +01:00
Ygal Blum
9f8df4313c job/osbuild: set AuthFilePath for container upload if provided
If a `AuthFilePath` was configured, which should contain secrets
to access container registries, we set this on the `Client` so
that the secrets can be used during registry access.
2022-08-01 21:50:03 +01:00
Ygal Blum
64a3aac895 job/osbuild: move utility the container package
Extract a helper method that parses the reference and applies
defaults, i.e. registry and path, and move it to the contaier
package.
2022-08-01 21:50:03 +01:00
Ygal Blum
3231aabbc0 cloudapi: add support for uploading to a container registry
Worker
------
Add configuration for the default container registry.
Use the default container registry if not provided as part
of the image name.
When using the default registry use the configured values
Return the image url as part of the result.

Composer Worker API
-------------------
Add `ContainerTargetResultOptions` to return the image url

Composer API
------------
Add UploadOptions to allow setting of the image name and tag
Add UploadStatus to return the url of the uploaded image

Co-Developed-By: Christian Kellner <christian@kellner.me>
2022-08-01 21:50:03 +01:00
Sanne Raymaekers
14208d872b templates/dashboards: Add brew tenants
Also:
- Gives tenants a nice display name.
- Makes "All" the default
2022-08-01 21:45:06 +01:00
Sanne Raymaekers
9347a30775 templates/dashboards: Drop arch from osbuild jobtype
This changed in #2845, and the dashboards stopped working properly as
they were looking for `osbuild+:arch`.

Keep the glob however, to also capture older metrics. The glob can be
removed after 1 month, as that's how long metrics are stored.
2022-08-01 13:37:28 +02:00
Jakub Rusz
e0d092538d distro/rhel8+9: exclude qemu-guest-agent package in ec2 and gce images
Related RHBZ#2075815
2022-08-01 13:16:05 +02:00
Jakub Rusz
3b4ace9987 distro/rhel90: exclude dracut-config-rescue package in ec2 images
Related RHBZ#2064087
2022-08-01 13:16:05 +02:00
Sanne Raymaekers
fa18206918 osbuild-service-maintenance: Rely on the db's timezone in tests
Avoid calculating the new expire date on the test host, because
differences in timezones can yield unexpected failures.
2022-08-01 10:59:05 +02:00
Sanne Raymaekers
69d4429e8f osbuild-service-maintenance: Honor dryrun in db cleanup 2022-08-01 10:59:05 +02:00
Sanne Raymaekers
2eaad3701d osbuild-service-maintenance: Reword log messages and query name
The query is deleting multiple jobs, and it's deleting the entire job,
not just the results.
2022-08-01 10:59:05 +02:00
Tomas Hozza
322ae02f73 internal/container: delete leftover dead code
This issue was found by Coverity:

Error: DEADCODE (CWE-561): [#def1]
osbuild-composer-58/_build/src/github.com/osbuild/osbuild-composer/internal/container/client.go:386: cond_null: Condition "err != nil", taking false branch. Now the value of "err" is "nil".
osbuild-composer-58/_build/src/github.com/osbuild/osbuild-composer/internal/container/client.go:390: null: At condition "err != nil", the value of "err" must be "nil".
osbuild-composer-58/_build/src/github.com/osbuild/osbuild-composer/internal/container/client.go:390: dead_error_condition: The condition "err != nil" cannot be true.
osbuild-composer-58/_build/src/github.com/osbuild/osbuild-composer/internal/container/client.go:391: dead_error_line: Execution cannot reach this statement: "<temporary>.$0 = container....".
  389|
  390|   	if err != nil {
  391|-> 		return resolvedIds{}, nil
  392|   	}
  393|
2022-07-29 23:20:09 +02:00
Christian Kellner
e3de1d03c0 distro/rhel7: blacklist skx_edac,intel_cstate kernel modules on azure
This was done in RHEL 8, 9 but RHEL 7 was not yet merged so this was
missed there. See e.g. 5c1530ee53
2022-07-28 17:00:11 +01:00
Sanne Raymaekers
c78c69b273 osbuild-service-maintenance: Run vacuum analyze after each delete 2022-07-28 16:53:55 +02:00
Sanne Raymaekers
a221de5db7 templates/composer: Remove non-existent secret
The secret not existing causes the deployment to fail during a
validation stage.

```
[ERROR] [openshift_base.py:_validate_resources_used_exist] - [Deployment/composer] Secret db does not exist
```
2022-07-28 11:24:25 +02:00
Achilleas Koutsou
234e16f35e test/diff-manifests: fix error handling
The change made in 7f563a6db1 would
require the shell option `-e` to not be set, so that we could capture
the exit code after the command fails.
Fix the error handling by putting the commands that we want to handle in
the test part of an `if` clause.

In addition, error messages are now printed in red.
2022-07-28 10:12:17 +02:00
Achilleas Koutsou
dc95382ba3 rhel90: add kernel modules for iso
These modules are on the official RHEL 9.0 ISO.  Adding them for feature
parity.
2022-07-28 10:12:17 +02:00
Achilleas Koutsou
02bb7a0b4f rhel90: add dracut modules for iso
The nvdimm module is required for booting the image via UEFI HTTP.

The rest are added for feature parity with the official RHEL 9.0 ISO.

Fixes rhbz#2030730
2022-07-28 10:12:17 +02:00
Ondřej Budai
9def545570 dbjobqueue: fix bad errors.As usages
errors.As is meant to check whether err (or other error in its chain) can
be assigned to the value that target is pointing at.

Let's consider this example:

errors.As(err, &pgx.ErrNoRows)

pgx.ErrNoRows (and pgx.ErrTxClosed) is typed as error, thus in all
errors.As calls, the target is typed as *error. Err is always an error.
So this call is basically asking whether error can be assigned to error.
If err != nil, this is always true, thus this check doesn't make any sense
over a plain err != nil.

Go 1.19 now checks this issue and if it's found, it refuses to compile the
code, see:

https://go-review.googlesource.com/c/tools/+/339889

This commit changes usages of errors.As() to errors.Is(). The Is() method
doesn't check assignability but equality (the only different between Is()
and a plain old == operator is that Is() also inspects the whole error chain).

This fixes the check because now, we are basically checking if err (or
any other error in its chain) == pgx.ErrTxClosed which is exactly what we
want.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-07-27 18:29:59 +02:00
Achilleas Koutsou
93592dabc1 test: regenerate manifests 2022-07-27 18:29:59 +02:00
Achilleas Koutsou
c20e1e53c4 osbuild: use path as secondary sort key for fstab
Most filesystems entries in fstab don't have a PassNo, which makes the
order of those entries dependent on the sorting algorithm.  Changes in
the algorithm can introduce changes in the sort order, which we don't
like.

Add a secondary sorting key, the Path, which is guaranteed unique, to
guarantee stable ordering.
2022-07-27 18:29:59 +02:00
Gianluca Zuccarelli
e5d9d2d045 worker/server: rename JobStatus() to JobInfo()
Since the `jobStatus` functions return a `JobInfo`
struct that contains the `JobStatus`, it makes sense
to rename the function names for the sake of consistency.
2022-07-27 13:37:14 +02:00
Gianluca Zuccarelli
95c8657f9e metrics: remove arch from osbuild type
The osbuild jobtype currently contains the
architecture as a suffix. Since the arch
is now being supplied as a label, the
`arch` suffix can be removed.
2022-07-27 13:37:14 +02:00
Gianluca Zuccarelli
967ac1c35e worker/server: job status struct
The number of return values from the `jobStatus`
function was growing and getting out of hand. Not
all return values were being used in all cases
and so returning a single struct with the information
and status of a job makes more sense. Then in each case
the resulting fields can be used as needed.
2022-07-27 13:37:14 +02:00
Gianluca Zuccarelli
9f4e765657 metrics: build jobs arch label
Add the architecture label to build jobs
which will enable filtering and monitoring
build jobs by architecture. Build job results
contain the `arch` field in the results struct,
this is then used to pass to the metrics, where
there is a value, otherwise it is set to an
empty string.
2022-07-27 13:37:14 +02:00
Gianluca Zuccarelli
8b4aff3857 worker/server: remove duplicate metrics
Remove a duplicate call to the `DequeueJobMetrics`
function in the worker server. This duplicate call
resulted in negative numbers for pending jobs in
the prometheus metrics.
2022-07-27 13:37:14 +02:00
schutzbot
ac522a58ee Post release version bump
[skip ci]
2022-07-27 08:31:06 +00:00
Achilleas Koutsou
2b6faea320 Remove image info from all test manifests
We never generated it for the newest distro versions and we don't test
it on any of the old ones.

It's better this way.
2022-07-27 10:01:16 +02:00
Jakub Rusz
74bf3aa3df test/diff-manifests: checkout repo to $head on exit
During the diff-manifests.sh test the source repository checkout is
changed to generate manifests from current main branch for comparion. We
want to checkout back to $head after the script is done or in case of
any unexpected exit.
2022-07-26 17:08:32 +02:00
Jakub Rusz
cf562dbb57 CI: drop /tmp/artifacts upload to Gitlab
Contents of this folder are uploaded to S3 with shutzbot/upload_artifacts.sh
Also fixing epel installation in the mentioned script.
2022-07-26 17:08:32 +02:00
Xiaofeng Wang
c71ae8f455 test: Update tests to support embeded container image feature
Container image can only be embeded on commit Use fedora image
instead of ubi8 and only run embeded image checking on  RHEL
8.7, 9.1, CS8 and CS9.
2022-07-26 17:07:31 +02:00
Christian Kellner
89c2bb0d3e spec: require osbuild 62
We need the `org.osbuild.containers.storage.conf` stage and its
fixes that only landed in osbuild 62.
2022-07-26 17:07:31 +02:00
Christian Kellner
14931f1ebe ci: pin osbuild to include pytoml fix
osbuild commit 9956f54 includes a fix for the `containers.storage.conf`
to work with RHEL 8 by trying to include `pytoml` if including `toml`
fails. We need that for the RHEL 8 based container embedding on OSTree
tests.
2022-07-26 17:07:31 +02:00
Christian Kellner
34c94ab92b container: rework GetDefaultAuthFile and don't cache its result
Instead of using a cached result `GetDefaultAuthFile`, always
do call the function when a new `Client` is created, since at
least `/run/containers` can get created as a side-effect by
one of the container. Now that we check eagerly and often the
path check function was reworked to only return paths that do
exist and are accessible.

Also check if `REGISTRY_AUTH_FILE` is set and if so, and it
is accessible use that.

To check accessability, use `unix.Access` instead of `os.Stat`,
since On Fedora/RHEL 9 `os.Stat` is implemented via `statx` and
will indeed return `EACCES` for inaccessible paths. But on RHEL
8 `lstat` is used and that will return `ENOENT` but then later
when trying to open the file we will get `EPERM`.
2022-07-26 17:07:31 +02:00
Christian Kellner
62406070b6 distro/rhel9: support embedding containers in ostree commits
Add support for embedding containers in OSTree commits by
storing them in `/usr/share/containers/storage`. The storage
engine is configured accordingly so that this extra location
is automatically taken into account by e.g. `podman`.
2022-07-26 17:07:31 +02:00
Christian Kellner
d7f1ed8ba7 distro/rhel8: support embedding containers in ostree commits
Add support for embedding containers in OSTree commits by
storing them in `/usr/share/containers/storage`. The storage
engine is configured accordingly so that this extra location
is automatically taken into account by e.g. `podman`.
2022-07-26 17:07:31 +02:00
Christian Kellner
e290502a1d container: add a check to ensure auth file path is always set
We never want an empty path but always force a specific auth
file location, even if the location does not actually exist,
due to the peculiarities mentioned in the comment of the
`container.GetDefaultAuthFile` function.
2022-07-26 17:07:31 +02:00
Christian Kellner
e38e7c717d container: add a auth file path getter for Client
This is so we can check and verify what is set.
2022-07-26 17:07:31 +02:00
Christian Kellner
8a06b9ddf3 osbuild: add bindings for org.osbuild.containers.storage.conf stage
Add support for the `containers.storage.conf` stage with helper
constructors that should make it easy to use.
Add a small test for it.
2022-07-26 17:07:31 +02:00
Achilleas Koutsou
f3c166c3fd CI: fix artifact collection for diff-manifests 2022-07-25 21:23:02 +02:00
Achilleas Koutsou
46b127c898 diff-manifests: never use main explicitly
The base branch name is read from the pull request metadata.  Let's use
it everywhere and never assume `main`.
2022-07-25 21:23:02 +02:00
Achilleas Koutsou
b5c47dbe8a test/diff-manifests: warn if generation fails on merge-base
If the manifest generation fails on the merge-base with main (the target
branch of the PR), add a note to the review comment.
2022-07-25 21:23:02 +02:00