Commit graph

6730 commits

Author SHA1 Message Date
Christian Kellner
4b67e12958 ci: install gpgme-devel for db test
This prepares the usage of the `internal/container` from composer
directly, as opposed to the existing use in the worker. Said pkg
uses the `containers/image/v5`, which uses `proglottis/gpgme` and
the latter needs the gpgme C library. We therefore install it and
its dependencies.
2022-07-21 13:32:07 +02:00
Christian Kellner
4c55063a55 Dockerfile-ubi: install gpgme devel package in builder
The go package `proglottis/gpgme` a dependency of `containers/image/v5`
package uses `libgpgme`. In the near future `internal/container`, which
depends on `containers/image/v5`, will be used directly in composer and
thus we need to install the `gpgme` devel package and its build deps.
2022-07-21 13:32:07 +02:00
Christian Kellner
f76b1545ca test/diff-manifests.sh: install build dependencies via spec file
This test is compiling `gen-manifests` via `go run` and thus needs
to pick up build requirements for the source. Instead of manually
installing the go toolchain use the `dnf build-dep` command on the
spec file so we pick up current and future build dependencies.
2022-07-21 13:32:07 +02:00
Jakub Rusz
f93b38c1a8 CI: fix ci_details artifact placement
The output of the ci_details.sh needs to be placed in the /tmp/artifacts
in order to be uploaded with the rest of the artifacts.
2022-07-21 13:17:45 +02:00
Jakub Rusz
151a9f5c86 tests/filesystems: Fix error checking
Checking the results of each scenario separately and making it more
clear what has actually failed.
2022-07-21 13:17:45 +02:00
Alexander Todorov
6e6f595c1a ci: Disable some Fedora-36 OStree testing due to #2702 2022-07-21 12:19:38 +03:00
Alexander Todorov
0cbac05d42 tests: Temporarily disable cloud-image-val util CLOUDX-54 is resolved 2022-07-21 12:19:38 +03:00
Alexander Todorov
9960021a56 Update version of cloud-image-val b/c of failure on Fedora-36
the test scenario which failed appears to be a noop on Fedora-36
and has been tagged appropriately in cloud-image-val.
2022-07-21 12:19:38 +03:00
Alexander Todorov
63a4ae7626 Update fedora-36 manifests 2022-07-21 12:19:38 +03:00
Alexander Todorov
9cf8054d21 Add fedora-36.json repo definitions 2022-07-21 12:19:38 +03:00
Alexander Todorov
d36e005b37 Update tests with config for Fedora 36 2022-07-21 12:19:38 +03:00
Alexander Todorov
4dab7b74cf COMPOSER-1623: Enable Fedora 36 testing 2022-07-21 12:19:38 +03:00
Ondřej Budai
0adbce3606 test/koji: remove the old code for testing koji API
Koji API is no more, let's drop the test and rename koji-compose-v2.py to
koji-compose.py.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-07-19 16:00:52 +02:00
Ondřej Budai
e779562f3c worker: remove osbuild-koji job
Koji API removed by the previous commit was the last user of osbuild-koji job.
Let's remove it since nothing uses it. This also removes all of the
compatibility code in Cloud API, see concerns below:

Compatibility concerns:
- the internal deployment was moved to a completely different composer
  instance, thus there are no old jobs
- Fedora deployment is still unused in prod, thus we don't care about keeping
  backward compatibility of the old jobs

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-07-19 16:00:52 +02:00
Ondřej Budai
74eb3860df internal: remove kojiapi
We no longer use it, let's remove it. If you are wondering what to use instead,
use Cloud API. It supports everything that Koji API supported and more.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-07-19 16:00:52 +02:00
Ondřej Budai
058edd3d76 cloudapi: remove a confusing test
This test tested two things:

1) Invalid route - this is already covered by TestUnknownRoute
2) Invalid UUID in the compose status route - this is now covered by
   TestComposeStatusInvalidUUID

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-07-19 16:00:52 +02:00
Ondřej Budai
3e25f5ef76 remove all traces of fedora 34
Fedora 34 is EOL, let's remove all traces of it, including:

- distro definition
- repositories (and test one)
- test manifests
- special package set rules
- hacks from the spec file

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-07-18 20:38:33 +02:00
Chloe Kaubisch
86971ca312 templates: update dashboards to include tenant
Add a tenant variable to the composer dashboard, with the option
to select multiple tenants. Add tenant filter to queries accordingly.

link to dashboard: https://grafana.stage.devshift.net/d/image-builder-worker-with-tenant/image-builder-worker?orgId=1
2022-07-18 18:55:13 +02:00
Ondřej Budai
30a1f6a68f github: fix the coverity scan action
It was failing on:

vendor/github.com/proglottis/gpgme/data.go:4:11: fatal error: gpgme.h: No such file or directory

Let's install this package before running the check, I verified that this
fix works locally.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-07-18 11:19:21 +02:00
Jakub Rusz
efbd5ebd7b ci/tests: Change the way artifacts are collected
We want to be able to safely gather any artifacts without worrying about
any possible secrets leaking. Every artifacts that we want to upload
will now have to be placed in /tmp/artifacts which will then be uploaded
to S3 by the executor and link to the artifacts will be provided in the
logs. Only people with access to our AWS account can see them.
2022-07-18 11:33:57 +03:00
Brian C. Lane
d3af314e58 users_stage: Don't allow empty passwords
Make sure empty passwords are set to nil so they result in a locked
account.

Also add a test for the password hashing in NewUserStageOptions()
2022-07-16 22:54:26 +01:00
Brian C. Lane
6adf3f5b7b blueprint: Don't allow empty password
If the password is set to "" it will get hashed, allowing access to the
account in some circumstances. Console and ssh login don't appear to
work in practice, but su to the account from another user account is
possible.

This sets the empty password to nil which makes sure that it ends up as
a locked account.
2022-07-16 22:54:26 +01:00
Brian C. Lane
20bf0c4836 blueprint: Hash all user passwords
This commit changes blueprint behavior to always store the hash of the
password for the 'customizations.user' accounts. Note that missing or
blank passwords are not hashed and should be dealt with at a lower
layer.

Resolves: rhbz#2107358
2022-07-16 22:54:26 +01:00
schutzbot
963c53ce09 schutzfile: Update snapshots to 20220715 2022-07-16 19:58:26 +02:00
dependabot[bot]
2285a0a03b build(deps): bump actions/setup-go from 2 to 3
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 2 to 3.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-07-16 19:15:06 +02:00
Juan Abia
b8f6ea4570 tag created vmare VMs
tag VMs with 'gitlba-ci-test' images so they can get removed with
cloud cleaner
2022-07-16 19:13:09 +02:00
Achilleas Koutsou
79fa253d69 repositories: remove centos-8 repos
The CentOS Linux 8 packages have been removed from the mirrors.
CentOS 8 is replaced by CentOS Stream 8. [0]

Keep the centos-8.json symlinked to centos-stream-8.json because
composer's host distro detection picks up CS8 as centos-8.

[0] https://www.centos.org/news-and-events/convert-to-stream-8/
2022-07-16 17:52:42 +01:00
Xiaofeng Wang
9a66ffa78b test: Install package sssd in all edge images for BZ#2088459 2022-07-16 16:14:56 +02:00
Xiaofeng Wang
a5a1611335 test: Update test for push container image to registry
Pushing container image to registry feature is ready by PR #2462,
test should be update to cover this feature
2022-07-15 13:12:46 +02:00
Alexander Todorov
62a681870e Tests: Use unified diff format - easier to read 2022-07-15 10:53:27 +02:00
Gianluca Zuccarelli
00f4f1690f Dockerfile: update worker builder
Additional packages are required to build the
docker worker. This fix updates the builder
container to install the required libraries
and then create the worker binary.
2022-07-15 03:06:43 +02:00
Achilleas Koutsou
9d4a351ca6 Rename osbuild2 package to osbuild 2022-07-14 16:54:00 +02:00
Achilleas Koutsou
01d87b4e60 osbuild1: DELETED 2022-07-14 16:54:00 +02:00
Achilleas Koutsou
dcef56c75a osbuild2: move v1 stage metadata parsing to convertStageResults
Make the v1StageResult.Metadata a simple json.RawMessage and perform the
RawMessage to StageMetadata conversion in the convertStage functions.
This lets us get rid of the custom v1StageResult Unmarshaller and the
v1RawStageResult, and makes the whole conversion process easier to
trace.
2022-07-14 16:54:00 +02:00
Achilleas Koutsou
d68a95c533 osbuild2: copy Result types from osbuild1
Types, parsing functions and helpers copied from osbuild1 to
osbuild2/v1result.go.
The metadata handling is simplified: osbuild1 stage metadata for the RPM
and ostree-commit stages is identical to the osbuild2 counterparts.
2022-07-14 16:54:00 +02:00
Achilleas Koutsou
c1956ba6e1 Use osbuild2 Manifest in all tests
The test_distro Manifest, which is used in tests across multiple
packages, was using the old structure.  Updated to the v2 structure and
adapted all tests.
2022-07-14 16:54:00 +02:00
Achilleas Koutsou
8eb12018c0 osbuild2: move v1 result handling code to separate file 2022-07-14 16:54:00 +02:00
Achilleas Koutsou
1c218bc633 osbuild2: fix package docstring 2022-07-14 16:54:00 +02:00
Jakub Rusz
b297ee8d40 CI: temporarily siwtch RHOS-01 to non-ssd instances
There appears to be a problem with nested virtualization on newly added
hypervisors with ssd. I believe the issue is not preset on regular
instances, so switching to those until the issues can be resolved.
2022-07-14 12:00:50 +03:00
Jakub Rusz
e500bfbd6f test/manifests: renegerate fedora-35 manifests 2022-07-14 12:00:50 +03:00
Simon de Vlieger
78ae275c61 jobqueue: store an expiry date
This introduces an expiry date (default: 14 days from insert date) and
adjust the service-maintenance script to delete jobs that are older than
the expiration date.
2022-07-13 17:26:04 +02:00
schutzbot
37fc807bfa Post release version bump
[skip ci]
2022-07-13 08:30:26 +00:00
imagebuilder-bot
60bcfbb186 packit: Enable Bodhi update feature
While this feature is 'not mature yet' according to Packit developers,
we can enable it because there's no harm done. If Packit fails to
publish the Bodhi update then fedora-bot will take care of it.
2022-07-13 10:08:23 +02:00
Achilleas Koutsou
2eb9833370 osbuild-image-tests: ignore LVM UUID
Unconditionally ignore the UUID of the LVM container.  We don't control
this value so it isn't stable and makes our image tests fail.
2022-07-12 13:28:36 +02:00
Tom Gundersen
e844453c85 manifest: make filename optional and generalise Tree
We have three kinds of operating system trees, until we unify them to one,
hide them behind one interface. Use this to read the architecture from the
Tree rather than pass it in as a string to parent pipelines.

Also, make the filename parameter optional in a few places, there should be no
reason to set this rather than introspect it (except for backwards
compatibility).

Lastly, add another playground example sample to build a raw image.
2022-07-12 08:19:57 +01:00
Tom Gundersen
d00b98c134 manifest: don't stutter
Replace `pipeline.OSPipeline` with `pipeline.OS`, etc.

Also rename `LiveImg` to `RawImage`.
2022-07-12 08:19:57 +01:00
Tom Gundersen
529bc803db runner: introduce runner abstraction
For now all it does is represent the name of the runner and what requirements
it has of the build pipeline.

Move some package definitions from the runner package set to where it belongs.
2022-07-12 08:19:57 +01:00
Tom Gundersen
33fe2da25c osbuild-playground: rework slightly
Invoke osbuild, rather than output the manifest. Make it easier to include
several image types.
2022-07-12 08:19:57 +01:00
Tom Gundersen
1b924ae30c osbuild-worker/osbuild: move out execution helper
Move the execution helper for osbuild into the osbuild2 package so it can be
reused. In the process, generalise it slightly.
2022-07-12 08:19:57 +01:00
Tom Gundersen
6ae65f1951 test/data/manifests: regenerate to drop build packages
With the more fine-grained build package set, different images will use
different build pipelines and each of them will be smaller.

We don't currently cache build pipelines so there is no downside to this. Even
when we start caching the difference between having one build pipeline per
image and one shared one is minimal at scale. This will still benefit users
doing one-off builds on-prem.

Most importantly, this tracks things correctly, making pipelines more
composable without having to maintain a global list of dependencies.
2022-07-12 08:19:57 +01:00