Commit graph

5541 commits

Author SHA1 Message Date
dependabot[bot]
05eb1a95b6 build(deps): bump github.com/golang-jwt/jwt/v4 from 4.4.2 to 4.5.0
Bumps [github.com/golang-jwt/jwt/v4](https://github.com/golang-jwt/jwt) from 4.4.2 to 4.5.0.
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md)
- [Commits](https://github.com/golang-jwt/jwt/compare/v4.4.2...v4.5.0)

---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v4
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-04-20 11:20:05 +02:00
Achilleas Koutsou
03b093acb0 test/container-embedding: add checks for container names
Add a local name for the fedora minimal image which includes the tag
`v1`.

Check the image info for the expected names:
1. For the fedora-minimal image, the name as it appears in the blueprint
   should be included in the names list.
2. For the manifest-list-test image, the full source reference should be
   included in the names list since no name was specified in the
   blueprint.
2023-04-19 20:07:40 +02:00
Achilleas Koutsou
140571bf48 test: add "name" to test manifests with container
Add name field to the manifest-list-test container in the test
request.  The value is the same as the source but with a `v1` tag
added.

In the manifests, the name field for the manifest-list-test is added to
the skopeo stage.  The `name` option of the fedora-minimal container in
the skopeo stage is also changed to reflect the full source reference
including the `latest` tag.
2023-04-19 20:07:40 +02:00
Achilleas Koutsou
3ff1b3beb0 container: update unit tests to match expected behaviour 2023-04-19 20:07:40 +02:00
Achilleas Koutsou
f9e3d8659d container: explicitly set LocalName
Set the LocalName for the spec using a separate argument in the
NewSpec() constructor instead of reusing the `source` arg.
The name is already available in the calling scope in the client's
Resolve() method.

If the LocalName is an empty string, default to the remote (source)
reference.  This is a change from the previous behaviour which only used
the base source.Name().  The full source corresponds to the
user-provided source value, which includes any specified tag or digest.

The `name` argument which is used in the `Resolve()` function should
always correspond to the user-provided container name.
2023-04-19 20:07:40 +02:00
Achilleas Koutsou
8b1375bee0 test: regenerate manifests
The PR which added manifest lists and the format-request-map was changed
to include two containers for the `with-container` compose requests [1]
was rebased after the PR which added RHEL 8.9 and RHEL 9.3 was merged
[2].  The test manifests were not updated after the rebase, so they were
never created with the new request.

Updating them now.

[1] https://github.com/osbuild/osbuild-composer/pull/3336
[2] https://github.com/osbuild/osbuild-composer/pull/3350
2023-04-19 20:07:40 +02:00
Tomáš Hozza
c28f8077c1 Revert "distro/rhel8: don't install missing MSFT key into azure-sap-rhui"
This reverts commit 2b1facb44d.

The GPG key is now present in the RHUI client RPM, so there is no need
to not import it during the image build.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2023-04-19 15:04:42 +02:00
Tomáš Hozza
7a439ba59d Test/manifests: update Azure RHUI client repo snapshots
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2023-04-19 15:04:42 +02:00
Ondřej Budai
d8342862d5 customizations: create all groups unconditionally
We had this weird condition in code that prevented composer to create groups
with the same name as a user has. This unfortunately means that you are not
able to create a user with a primary group with a certain GID that has the
same name as the user. There's the gid field in the user customization,
but it requires that the group already exists.

In order to allow that, we need to remove the condition. From now on, it's
possible to create groups with the same name as a user has, which can be used
to create primary groups with a custom gid.

Note that the lorax compatibility behaviour was actually wrong. When lorax was
given a custom gid for a user, it didn't require the gid to exist. When it
didn't, the group was just created. Thus, we still don't have full backward
compatibility, but at least we now have support for this.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-19 11:50:49 +02:00
schutzbot
3d0254b436 Post release version bump
[skip ci]
2023-04-19 08:22:07 +00:00
dependabot[bot]
f3f647310d build(deps): bump github.com/docker/docker
Bumps [github.com/docker/docker](https://github.com/docker/docker) from 20.10.17+incompatible to 20.10.24+incompatible.
- [Release notes](https://github.com/docker/docker/releases)
- [Commits](https://github.com/docker/docker/compare/v20.10.17...v20.10.24)

---
updated-dependencies:
- dependency-name: github.com/docker/docker
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-04-19 08:52:30 +02:00
Simon de Vlieger
258f450021 iot: change the copy stages
Split this into two separate copystages so that we are not overriding
types.
2023-04-18 21:21:09 +02:00
Simon de Vlieger
b0fce3bfd1 platform: add the aarch64_iot platform
This platform copies more files into `/boot` which are necessary to be
able to boot IoT on some single board computers.

We also immediately set this on the `Aarch64_IoT` platform which needs
u-boot to be placed in the `/boot`.

This closes #3312.
2023-04-18 21:21:09 +02:00
Simon de Vlieger
8041563a36 manifest: image-tree -> ostree-deployment
Describes much better what the pipeline actually does.
2023-04-18 21:21:09 +02:00
Simon de Vlieger
66345cf54c manifest: commit_deployment -> ostree_deployment 2023-04-18 21:21:09 +02:00
Jiri Popelka
10c46999b1 packit: remove upstream_package_name & downstream_package_name
they default to the GitHub repo name.

https://packit.dev/docs/configuration/#upstream_package_name
https://packit.dev/docs/configuration/#downstream_package_name
2023-04-18 11:36:56 +02:00
Jiri Popelka
5fda42e20c packit: metadata key can be removed
It's been deprecated for some time.
Its keys can be used directly.
2023-04-18 11:36:56 +02:00
Achilleas Koutsou
a56c70d505 spec: depend on osbuild v83
Set the dependency to osbuild v83 which contains the new features for
preserving manifest lists for containers.

Added in https://github.com/osbuild/osbuild/pull/1252
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
60a191bc78 Schutzfile: osbuild commit
Pin osbuild to current main for manifest-list feature.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
d936a8722d test/container-embedding: add container with manifest-list
Add a second container to the container-embedding test.  The digest
refers to a manifest-list to test the new feature.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
5a455b4406 test: add container source with manifest-list to manifests
Add a second container to the ostree-commit test manifests that refers
to a manifest list on the osbuild-composer registry on gitlab.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
ab9e612c99 containers: update tests
Update tests to include the list digests of the test images.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
d41a74475e osbuild/skopeo: helper function for files input
Add a helper function that collects all the manifest list digests from a
list of container specs and returns a FilesInput to be used with the
stage.

Use the function in the OS pipeline when adding containers.  The
manifests input to the stage constructor will be empty if there are no
manifest lists in the container specs.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
753bcaed78 osbuild: create skopeo-index source in GenSources()
When generating sources in GenSources(), add a skopeo-index source
reference for each list-digest found in the container specs.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
e5aaf3e514 container: save manifest list digest when resolving
While resolving a manifest list digest, store the list digest to return
with the resolvedIds.

This is done for both types of manifest list:
  application/vnd.docker.distribution.manifest.list.v2+json
and
  application/vnd.oci.image.index.v1+json
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
f38eb8406a container: include TLSVerify and ListDigest in spec ctor 2023-04-17 18:30:41 +02:00
Achilleas Koutsou
a7dd8ff94f container: add ListDigest to the spec
Add the ListDigest to the container Spec struct and all its copies so we
can store list digests when they are available and pass them on to the
appropriate osbuild stages, sources, and inputs.

Copy the value whenever a spec is moved to a different representation.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
c95072d686 osbuild: add manifest-lists input to skopeo stage
The skopeo stage in osbuild supports an second optional set of inputs
called `manifest-lists`.  This is an array of files, i.e.,
`org.osbuild.files` type input.

To support this we need a new type for the skopeo stage inputs that can
encompass both input types, images and manifest-lists.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
dfae9fbefb osbuild/skopeo: reverse the constructor args
Put the path (which becomes an option) first and the inputs second.
This is more in line with other stage constructors.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
069021093c osbuild: add skopeo-index source
New osbuild source that can download a manifest-list from a container
registry, using the `--multi-arch=index-only` option of skopeo copy.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
ad2a5bff7c osbuild: name in containers input is not optional
Remove the `omitempty` from the name field in the containers input.  It is
required.
2023-04-17 18:30:41 +02:00
Ondřej Budai
7edbaf6b43 dbjobqueue: put all SQL queries in dequeueMaybe into a transaction
This is needed to ensure atomicity of the whole dequeue operation.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-14 16:37:04 +02:00
Ondřej Budai
c3f6baad7f dbjobqueue: put all SQL queries into dequeueMaybe
Let's move all SQL queries together. In the following commit, we will actually
put all of them into a transaction in order to ensure atomicity.

This isn't a functional change, just code shuffling.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-14 16:37:04 +02:00
Ondřej Budai
464ce568b2 dbjobqueue: put all DequeueByID queries into a transaction
If inserting a heartbeat or querying dependencies fail, we don't want to
actually dequeue the job from the database.

The failures may be:
- context timeout/cancellation
- network issues

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-14 16:37:04 +02:00
Ondřej Budai
571b959cc1 dbjobqueue: make jobDependencies and dependants accept transactions
We will need this in following commits in order to make dequeuing atomic.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-14 16:37:04 +02:00
Sanne Raymaekers
9dc0881247 internal/worker: log dequeue failures 2023-04-14 12:12:41 +02:00
Alexander Todorov
0330e9c3e1 More robust filtering for unrecognized distros
in order to preserve the value `rhel-93` as unrecognized when
running on 9.2 nightly CI
2023-04-14 10:26:09 +03:00
Alexander Todorov
5cd73ad946 Don't fail cross-distro.sh on nightly CI pipeline
when it discovers "missing distros" b/c this check isn't suitable for
downstream RHEL anyway. See comment in script for more details!

Related: COMPOSER-1943
2023-04-14 10:26:09 +03:00
Sanne Raymaekers
14a9b2395d cloudapi/v2: expose repo metadata verification 2023-04-13 22:08:17 +02:00
yih
b90f51ff33 fix cs9 edge-simplified-installer test failure 2023-04-13 21:39:32 +02:00
Gianluca Zuccarelli
da2e80f99f test/cases: update old worker regression version
Update the version of the worker in the old worker/new composer
regression test. The test, up until now, was using a pinned version
of osbuild-worker (v51) which made use of dnf-json as a socket.
Testing a new composer version with such an outdated worker instance
is very unrealistic since the workers and composer are normally
updated in lock step, or in some cases the worker may be a version
behind.

This commit takes the current worker version and then downgrades it
to two versions ago (out of caution) to perform the tests. This
provides a more realistic scenario.
2023-04-12 15:48:03 +02:00
Simon de Vlieger
d94cea9150 fedora/disk: put a filesystem in EFI partition 2023-04-11 17:05:33 +02:00
Alexander Todorov
4890884a0e Parse modules info JSON according to weldr-client version 2023-04-06 09:32:10 +03:00
Alexander Todorov
240bf7e6fe Enable regression-insecure-repo test for nightly CI pipeline
Related: RHBZ#2177699
2023-04-06 09:32:10 +03:00
schutzbot
f5948ad5fd Post release version bump
[skip ci]
2023-04-05 08:24:00 +00:00
Diaa Sami
b4cf032239 containers/osbuild-composer: wait for fluentd in entrypoint
to avoid composer crashing and restarting when fluentd is not up
HMS-1496
2023-04-04 13:00:19 +02:00
Ondřej Budai
943ead790e upload/azure: skip uploading empty pages
The size of the page blob is defined on creation and the blob is
zero-initialized. Therefore, we can just skip all the pages that contain
only zeros. This should save a lot of bandwidth if used on sparse files as
e.g. operating system images. (:
2023-04-04 09:09:43 +02:00
Ondřej Budai
abe6ccfb50 upload/azure: migrate from azure-storage-blob-go to azure-sdk-for-go
https://github.com/Azure/azure-storage-blob-go/ is deprecated, the main SDK
should be now used instead. Let's migrate the code. There should be no
functional changes.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-04 09:09:43 +02:00
Ondřej Budai
9beddf626f upload/azure: remove the MD5 sum check
It doesn't actually make any sense. For Page Blobs, Azure doesn't compute any
hashes. The MD5 sum is basically just a property, which we set by one call and
get by the other call.

See
https://stackoverflow.com/questions/42229153/how-to-check-azure-storage-blob-file-uploaded-correctly/69319211#69319211

for more info.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-04 09:09:43 +02:00
Alexander Todorov
5e6d47e9d0 Sign RPM package during testing and verify GPG signatures
Related: RHBZ#2177699
2023-04-03 17:36:42 +02:00