Commit graph

5580 commits

Author SHA1 Message Date
Simon de Vlieger
258f450021 iot: change the copy stages
Split this into two separate copystages so that we are not overriding
types.
2023-04-18 21:21:09 +02:00
Simon de Vlieger
b0fce3bfd1 platform: add the aarch64_iot platform
This platform copies more files into `/boot` which are necessary to be
able to boot IoT on some single board computers.

We also immediately set this on the `Aarch64_IoT` platform which needs
u-boot to be placed in the `/boot`.

This closes #3312.
2023-04-18 21:21:09 +02:00
Simon de Vlieger
8041563a36 manifest: image-tree -> ostree-deployment
Describes much better what the pipeline actually does.
2023-04-18 21:21:09 +02:00
Simon de Vlieger
66345cf54c manifest: commit_deployment -> ostree_deployment 2023-04-18 21:21:09 +02:00
Jiri Popelka
10c46999b1 packit: remove upstream_package_name & downstream_package_name
they default to the GitHub repo name.

https://packit.dev/docs/configuration/#upstream_package_name
https://packit.dev/docs/configuration/#downstream_package_name
2023-04-18 11:36:56 +02:00
Jiri Popelka
5fda42e20c packit: metadata key can be removed
It's been deprecated for some time.
Its keys can be used directly.
2023-04-18 11:36:56 +02:00
Achilleas Koutsou
a56c70d505 spec: depend on osbuild v83
Set the dependency to osbuild v83 which contains the new features for
preserving manifest lists for containers.

Added in https://github.com/osbuild/osbuild/pull/1252
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
60a191bc78 Schutzfile: osbuild commit
Pin osbuild to current main for manifest-list feature.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
d936a8722d test/container-embedding: add container with manifest-list
Add a second container to the container-embedding test.  The digest
refers to a manifest-list to test the new feature.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
5a455b4406 test: add container source with manifest-list to manifests
Add a second container to the ostree-commit test manifests that refers
to a manifest list on the osbuild-composer registry on gitlab.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
ab9e612c99 containers: update tests
Update tests to include the list digests of the test images.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
d41a74475e osbuild/skopeo: helper function for files input
Add a helper function that collects all the manifest list digests from a
list of container specs and returns a FilesInput to be used with the
stage.

Use the function in the OS pipeline when adding containers.  The
manifests input to the stage constructor will be empty if there are no
manifest lists in the container specs.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
753bcaed78 osbuild: create skopeo-index source in GenSources()
When generating sources in GenSources(), add a skopeo-index source
reference for each list-digest found in the container specs.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
e5aaf3e514 container: save manifest list digest when resolving
While resolving a manifest list digest, store the list digest to return
with the resolvedIds.

This is done for both types of manifest list:
  application/vnd.docker.distribution.manifest.list.v2+json
and
  application/vnd.oci.image.index.v1+json
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
f38eb8406a container: include TLSVerify and ListDigest in spec ctor 2023-04-17 18:30:41 +02:00
Achilleas Koutsou
a7dd8ff94f container: add ListDigest to the spec
Add the ListDigest to the container Spec struct and all its copies so we
can store list digests when they are available and pass them on to the
appropriate osbuild stages, sources, and inputs.

Copy the value whenever a spec is moved to a different representation.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
c95072d686 osbuild: add manifest-lists input to skopeo stage
The skopeo stage in osbuild supports an second optional set of inputs
called `manifest-lists`.  This is an array of files, i.e.,
`org.osbuild.files` type input.

To support this we need a new type for the skopeo stage inputs that can
encompass both input types, images and manifest-lists.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
dfae9fbefb osbuild/skopeo: reverse the constructor args
Put the path (which becomes an option) first and the inputs second.
This is more in line with other stage constructors.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
069021093c osbuild: add skopeo-index source
New osbuild source that can download a manifest-list from a container
registry, using the `--multi-arch=index-only` option of skopeo copy.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
ad2a5bff7c osbuild: name in containers input is not optional
Remove the `omitempty` from the name field in the containers input.  It is
required.
2023-04-17 18:30:41 +02:00
Ondřej Budai
7edbaf6b43 dbjobqueue: put all SQL queries in dequeueMaybe into a transaction
This is needed to ensure atomicity of the whole dequeue operation.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-14 16:37:04 +02:00
Ondřej Budai
c3f6baad7f dbjobqueue: put all SQL queries into dequeueMaybe
Let's move all SQL queries together. In the following commit, we will actually
put all of them into a transaction in order to ensure atomicity.

This isn't a functional change, just code shuffling.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-14 16:37:04 +02:00
Ondřej Budai
464ce568b2 dbjobqueue: put all DequeueByID queries into a transaction
If inserting a heartbeat or querying dependencies fail, we don't want to
actually dequeue the job from the database.

The failures may be:
- context timeout/cancellation
- network issues

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-14 16:37:04 +02:00
Ondřej Budai
571b959cc1 dbjobqueue: make jobDependencies and dependants accept transactions
We will need this in following commits in order to make dequeuing atomic.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-14 16:37:04 +02:00
Sanne Raymaekers
9dc0881247 internal/worker: log dequeue failures 2023-04-14 12:12:41 +02:00
Alexander Todorov
0330e9c3e1 More robust filtering for unrecognized distros
in order to preserve the value `rhel-93` as unrecognized when
running on 9.2 nightly CI
2023-04-14 10:26:09 +03:00
Alexander Todorov
5cd73ad946 Don't fail cross-distro.sh on nightly CI pipeline
when it discovers "missing distros" b/c this check isn't suitable for
downstream RHEL anyway. See comment in script for more details!

Related: COMPOSER-1943
2023-04-14 10:26:09 +03:00
Sanne Raymaekers
14a9b2395d cloudapi/v2: expose repo metadata verification 2023-04-13 22:08:17 +02:00
yih
b90f51ff33 fix cs9 edge-simplified-installer test failure 2023-04-13 21:39:32 +02:00
Gianluca Zuccarelli
da2e80f99f test/cases: update old worker regression version
Update the version of the worker in the old worker/new composer
regression test. The test, up until now, was using a pinned version
of osbuild-worker (v51) which made use of dnf-json as a socket.
Testing a new composer version with such an outdated worker instance
is very unrealistic since the workers and composer are normally
updated in lock step, or in some cases the worker may be a version
behind.

This commit takes the current worker version and then downgrades it
to two versions ago (out of caution) to perform the tests. This
provides a more realistic scenario.
2023-04-12 15:48:03 +02:00
Simon de Vlieger
d94cea9150 fedora/disk: put a filesystem in EFI partition 2023-04-11 17:05:33 +02:00
Alexander Todorov
4890884a0e Parse modules info JSON according to weldr-client version 2023-04-06 09:32:10 +03:00
Alexander Todorov
240bf7e6fe Enable regression-insecure-repo test for nightly CI pipeline
Related: RHBZ#2177699
2023-04-06 09:32:10 +03:00
schutzbot
f5948ad5fd Post release version bump
[skip ci]
2023-04-05 08:24:00 +00:00
Diaa Sami
b4cf032239 containers/osbuild-composer: wait for fluentd in entrypoint
to avoid composer crashing and restarting when fluentd is not up
HMS-1496
2023-04-04 13:00:19 +02:00
Ondřej Budai
943ead790e upload/azure: skip uploading empty pages
The size of the page blob is defined on creation and the blob is
zero-initialized. Therefore, we can just skip all the pages that contain
only zeros. This should save a lot of bandwidth if used on sparse files as
e.g. operating system images. (:
2023-04-04 09:09:43 +02:00
Ondřej Budai
abe6ccfb50 upload/azure: migrate from azure-storage-blob-go to azure-sdk-for-go
https://github.com/Azure/azure-storage-blob-go/ is deprecated, the main SDK
should be now used instead. Let's migrate the code. There should be no
functional changes.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-04 09:09:43 +02:00
Ondřej Budai
9beddf626f upload/azure: remove the MD5 sum check
It doesn't actually make any sense. For Page Blobs, Azure doesn't compute any
hashes. The MD5 sum is basically just a property, which we set by one call and
get by the other call.

See
https://stackoverflow.com/questions/42229153/how-to-check-azure-storage-blob-file-uploaded-correctly/69319211#69319211

for more info.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-04 09:09:43 +02:00
Alexander Todorov
5e6d47e9d0 Sign RPM package during testing and verify GPG signatures
Related: RHBZ#2177699
2023-04-03 17:36:42 +02:00
Alexander Todorov
58cb12eeaa Fix typo in custom repository field 2023-04-03 17:36:42 +02:00
Irene Diez
47bb682221 cmd: handle generated warnings during Manifest initialization
Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Irene Diez
20a5b25320 distro/fedora: handle generated warnings during Manifest initialization
Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Irene Diez
2771af2530 distro/rhel7: handle generated warnings during Manifest initialization
Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Irene Diez
64a5946dc3 distro/rhel8: handle generated warnings during Manifest initialization
Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Irene Diez
d6b9b3a5bf distro/rhel9: handle generated warnings during Manifest initialization
Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Irene Diez
5fb989110a weldr: allow to send warnings on ComposeReply
This adds a new field `Warnings` to the `ComposeReply`
struct, allowing to send back any warnings (e.g. deprecation
notices) generated during the `checkOptions` step of the
manifest initialization.

See also https://github.com/osbuild/weldr-client/pull/99 which
handles the weldr-client side of things.

Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Irene Diez
8022c227ba distro: allow to return warnings in the Manifest function
This changes the `Manifest` function of the `ImageType`
interface so that any warnings detected during the
`checkOptions` step of the manifest initialization can
be propagated back to the Weldr-API (see next commit).

Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Sanne Raymaekers
d952e41161 internal/boot: boot VMWare VMs with EFI and SCSI
Since we use streamOptimized images everywhere, we no longer need to use
IDE and boot with bios. Let's test a more realistic scenario.
2023-03-31 09:24:42 +02:00
dependabot[bot]
1f070687f9 build(deps): bump github.com/opencontainers/runc from 1.1.3 to 1.1.5
Bumps [github.com/opencontainers/runc](https://github.com/opencontainers/runc) from 1.1.3 to 1.1.5.
- [Release notes](https://github.com/opencontainers/runc/releases)
- [Changelog](https://github.com/opencontainers/runc/blob/v1.1.5/CHANGELOG.md)
- [Commits](https://github.com/opencontainers/runc/compare/v1.1.3...v1.1.5)

---
updated-dependencies:
- dependency-name: github.com/opencontainers/runc
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-31 09:24:19 +02:00
Achilleas Koutsou
2a87a8b7f3 cloudapi: enable azure-eap7-rhui image type 2023-03-30 18:40:12 +02:00