Commit graph

5513 commits

Author SHA1 Message Date
Achilleas Koutsou
dfae9fbefb osbuild/skopeo: reverse the constructor args
Put the path (which becomes an option) first and the inputs second.
This is more in line with other stage constructors.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
069021093c osbuild: add skopeo-index source
New osbuild source that can download a manifest-list from a container
registry, using the `--multi-arch=index-only` option of skopeo copy.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
ad2a5bff7c osbuild: name in containers input is not optional
Remove the `omitempty` from the name field in the containers input.  It is
required.
2023-04-17 18:30:41 +02:00
Ondřej Budai
7edbaf6b43 dbjobqueue: put all SQL queries in dequeueMaybe into a transaction
This is needed to ensure atomicity of the whole dequeue operation.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-14 16:37:04 +02:00
Ondřej Budai
c3f6baad7f dbjobqueue: put all SQL queries into dequeueMaybe
Let's move all SQL queries together. In the following commit, we will actually
put all of them into a transaction in order to ensure atomicity.

This isn't a functional change, just code shuffling.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-14 16:37:04 +02:00
Ondřej Budai
464ce568b2 dbjobqueue: put all DequeueByID queries into a transaction
If inserting a heartbeat or querying dependencies fail, we don't want to
actually dequeue the job from the database.

The failures may be:
- context timeout/cancellation
- network issues

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-14 16:37:04 +02:00
Ondřej Budai
571b959cc1 dbjobqueue: make jobDependencies and dependants accept transactions
We will need this in following commits in order to make dequeuing atomic.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-14 16:37:04 +02:00
Sanne Raymaekers
9dc0881247 internal/worker: log dequeue failures 2023-04-14 12:12:41 +02:00
Alexander Todorov
0330e9c3e1 More robust filtering for unrecognized distros
in order to preserve the value `rhel-93` as unrecognized when
running on 9.2 nightly CI
2023-04-14 10:26:09 +03:00
Alexander Todorov
5cd73ad946 Don't fail cross-distro.sh on nightly CI pipeline
when it discovers "missing distros" b/c this check isn't suitable for
downstream RHEL anyway. See comment in script for more details!

Related: COMPOSER-1943
2023-04-14 10:26:09 +03:00
Sanne Raymaekers
14a9b2395d cloudapi/v2: expose repo metadata verification 2023-04-13 22:08:17 +02:00
yih
b90f51ff33 fix cs9 edge-simplified-installer test failure 2023-04-13 21:39:32 +02:00
Gianluca Zuccarelli
da2e80f99f test/cases: update old worker regression version
Update the version of the worker in the old worker/new composer
regression test. The test, up until now, was using a pinned version
of osbuild-worker (v51) which made use of dnf-json as a socket.
Testing a new composer version with such an outdated worker instance
is very unrealistic since the workers and composer are normally
updated in lock step, or in some cases the worker may be a version
behind.

This commit takes the current worker version and then downgrades it
to two versions ago (out of caution) to perform the tests. This
provides a more realistic scenario.
2023-04-12 15:48:03 +02:00
Simon de Vlieger
d94cea9150 fedora/disk: put a filesystem in EFI partition 2023-04-11 17:05:33 +02:00
Alexander Todorov
4890884a0e Parse modules info JSON according to weldr-client version 2023-04-06 09:32:10 +03:00
Alexander Todorov
240bf7e6fe Enable regression-insecure-repo test for nightly CI pipeline
Related: RHBZ#2177699
2023-04-06 09:32:10 +03:00
schutzbot
f5948ad5fd Post release version bump
[skip ci]
2023-04-05 08:24:00 +00:00
Diaa Sami
b4cf032239 containers/osbuild-composer: wait for fluentd in entrypoint
to avoid composer crashing and restarting when fluentd is not up
HMS-1496
2023-04-04 13:00:19 +02:00
Ondřej Budai
943ead790e upload/azure: skip uploading empty pages
The size of the page blob is defined on creation and the blob is
zero-initialized. Therefore, we can just skip all the pages that contain
only zeros. This should save a lot of bandwidth if used on sparse files as
e.g. operating system images. (:
2023-04-04 09:09:43 +02:00
Ondřej Budai
abe6ccfb50 upload/azure: migrate from azure-storage-blob-go to azure-sdk-for-go
https://github.com/Azure/azure-storage-blob-go/ is deprecated, the main SDK
should be now used instead. Let's migrate the code. There should be no
functional changes.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-04 09:09:43 +02:00
Ondřej Budai
9beddf626f upload/azure: remove the MD5 sum check
It doesn't actually make any sense. For Page Blobs, Azure doesn't compute any
hashes. The MD5 sum is basically just a property, which we set by one call and
get by the other call.

See
https://stackoverflow.com/questions/42229153/how-to-check-azure-storage-blob-file-uploaded-correctly/69319211#69319211

for more info.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-04 09:09:43 +02:00
Alexander Todorov
5e6d47e9d0 Sign RPM package during testing and verify GPG signatures
Related: RHBZ#2177699
2023-04-03 17:36:42 +02:00
Alexander Todorov
58cb12eeaa Fix typo in custom repository field 2023-04-03 17:36:42 +02:00
Irene Diez
47bb682221 cmd: handle generated warnings during Manifest initialization
Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Irene Diez
20a5b25320 distro/fedora: handle generated warnings during Manifest initialization
Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Irene Diez
2771af2530 distro/rhel7: handle generated warnings during Manifest initialization
Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Irene Diez
64a5946dc3 distro/rhel8: handle generated warnings during Manifest initialization
Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Irene Diez
d6b9b3a5bf distro/rhel9: handle generated warnings during Manifest initialization
Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Irene Diez
5fb989110a weldr: allow to send warnings on ComposeReply
This adds a new field `Warnings` to the `ComposeReply`
struct, allowing to send back any warnings (e.g. deprecation
notices) generated during the `checkOptions` step of the
manifest initialization.

See also https://github.com/osbuild/weldr-client/pull/99 which
handles the weldr-client side of things.

Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Irene Diez
8022c227ba distro: allow to return warnings in the Manifest function
This changes the `Manifest` function of the `ImageType`
interface so that any warnings detected during the
`checkOptions` step of the manifest initialization can
be propagated back to the Weldr-API (see next commit).

Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Sanne Raymaekers
d952e41161 internal/boot: boot VMWare VMs with EFI and SCSI
Since we use streamOptimized images everywhere, we no longer need to use
IDE and boot with bios. Let's test a more realistic scenario.
2023-03-31 09:24:42 +02:00
dependabot[bot]
1f070687f9 build(deps): bump github.com/opencontainers/runc from 1.1.3 to 1.1.5
Bumps [github.com/opencontainers/runc](https://github.com/opencontainers/runc) from 1.1.3 to 1.1.5.
- [Release notes](https://github.com/opencontainers/runc/releases)
- [Changelog](https://github.com/opencontainers/runc/blob/v1.1.5/CHANGELOG.md)
- [Commits](https://github.com/opencontainers/runc/compare/v1.1.3...v1.1.5)

---
updated-dependencies:
- dependency-name: github.com/opencontainers/runc
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-31 09:24:19 +02:00
Achilleas Koutsou
2a87a8b7f3 cloudapi: enable azure-eap7-rhui image type 2023-03-30 18:40:12 +02:00
Achilleas Koutsou
90cc2b18a8 distro/rhel8: assert error instead of skipping
We sometimes skip testing customizations for image types that don't
support them, other times, we check for a specific error.  If we don't
care about the specific error message that's received, running the test
and checking for error is better than skipping, because it ensures that
we're skipping the test for the right reasons and not accidentally
skipping a test for an image type that should support the
customizations.
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
203dd5275e distro/rhel8: exclude azure-eap7-rhui from customization tests
Skip customizations tests for the EAP image.

Create a map of unsupported image types for each test to make adding new
ones easier.
Use a `nil` struct pointer for the pipeline names test instead of an
empty struct.  Non-nil customizations can't be compared.
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
a0698ec359 test: generate RHEL 8 Azure EAP 7 test cases
Azure RHUI test cases updated with new repo image type tags.
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
4e3d94f9a7 test-case-generators: add request for Azure EAP 7 image type 2023-03-30 18:40:12 +02:00
Achilleas Koutsou
520e455088 test: add eap7 repo mirror to RHEL 8.6+ tests 2023-03-30 18:40:12 +02:00
Achilleas Koutsou
81d6f526d0 config: don't show azure-eap7-rhui in weldr
The azure-eap7-rhui image type should only be available for internal
builds (for now).
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
21d13a2ae3 distro/rhel8: specify ports for azure EAP
See COMPOSER-1859
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
fb5accc227 distro/rhel8: add env vars to azure EAP
Add the required environment variables to Azure EAP.
See COMPOSER-1860
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
3f299c7d43 distro: add ShellVars field to ImageConfig 2023-03-30 18:40:12 +02:00
Achilleas Koutsou
816d23988c manifest: add support for shell vars in OSCustomizations
ShellVars are added as a nested map of the form
filename->key->value.  A stage is generated and added if any are
defined.
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
92bbf5fb8b osbuild: add generator function for shell.init stage
Generates a stage from a basic nested map.
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
a024b923a3 shell: new package for representing shell init configs internally
New types for the internal representation of the shell init configs,
which currently consists of files and key-value pairs for environment
global variables.
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
2fcf3582b5 osbuild: add shell.init stage
Add support for the org.osbuild.shell.init stage and test validator.
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
636f3ff237 distro/rhel8: add azure EAP 7 image type
Initialise the image type as an Azure RHUI type with the name
"azure-eap7-rhui" and a workload attached that provides the extra
packages required by the EAP workload.
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
5b4e77797c distro/rhel8: remove outdated comment 2023-03-30 18:40:12 +02:00
Achilleas Koutsou
18ef5ae32d distro/rhel8: distro workload implementation and EAP
Add a local, private implementation of the Workload interface for RHEL
8.  The type should be used for concrete workloads that provide package
package lists for specific image types.

The eapWorkload() function creates a workload that provides the packages
required for EAP image type variants.
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
77b8328999 distro: add workload to image type
Support specifying workload on an image type.
This isn't used yet, but when set, the workload will define packages and
services and not allow any blueprint customizations.

In the future, there will be multiple workloads with each image type
specifying one (but multiple image types can have the same default
workload).  The custom workload will allow all customizations that are
currently supported.
2023-03-30 18:40:12 +02:00