Commit graph

2874 commits

Author SHA1 Message Date
Gianluca Zuccarelli
08c10b51a0 internal/distro: fedora repo customizations
Wire up the repository customizations for
fedora images.
2023-04-21 17:40:00 +02:00
Gianluca Zuccarelli
f29e3e9b0e osbuild: yum repo stage from RepoConfig
Create an osbuild yum repository from
`rpmmd.RepoConfig`. Additionally, remove
pointers from the `YumRepository` struct,
since this will add values for fields that
weren't explicitly set by the user in the
repo customizations.
2023-04-21 17:40:00 +02:00
Gianluca Zuccarelli
29643c2e06 customizations: custom repo utility functions
Create some utility functions that will be used for implementing
custom repo configuration files. This commit adds these functions:
- a helper to get the filename of a custom repo, or the
  `<repo-id>.repo` if the filename is empty
- a function to convert the custom repos to a map of `RepoConfig`.
  This function also creates an `fsnode.File` for each inline gpg
  key set in the customizations and swaps the inline key for the
  file path. The function returns the map of `RepoConfig` and a list
  of `fsnode.File` containing the inline gpg keys.
2023-04-21 17:40:00 +02:00
Gianluca Zuccarelli
3b6fddb14a internal/rpmmd: add missing fields to RepoConfig
Further align the RepoConfig object to the dnf
spec and add missing fields.
2023-04-21 17:40:00 +02:00
Gianluca Zuccarelli
d44703cdc8 rpmmd/repository: repoconfig pointers
Convert some of the fields in the `RepoConfig` struct
to pointers. Since `RepoConfig` will be used to convert
custom repositories to an array of `osbuild.YumRepository`,
we need to ensure that fields that are not set explicitly
are not saved to the `/etc/yum.repos.d` repository files.
2023-04-21 17:40:00 +02:00
Gianluca Zuccarelli
75e2138878 customizations: add custom repositories
Define blueprint custom repositories. These
definitions have been based on the upstream
definitions of a DNF repo[1].

[1] See dnf.conf
2023-04-21 17:40:00 +02:00
Gianluca Zuccarelli
4d42808b6a internal/rpmmd: RepoConfig baseurl change
Update the internal RepoConfig object to
accept a slice of baseurls rather than a
single field. This change was needed to
align RepoConfig with the dnf spec [1].

Additionally, this change adds custom json
marshal and unmarshal functions to ensure
backwards compatibility with older workers.
Add json tags to the internal rpmmd config
since this is serialized in dnfjson.
Add unit tests to check the serialization
is okay.

[1] See dnf.config
2023-04-21 17:40:00 +02:00
Gianluca Zuccarelli
17d730593c internal/cloudapi: fix test
Minor fix to a test to correct the
expected vs actual fields.
2023-04-21 17:40:00 +02:00
Achilleas Koutsou
3ff1b3beb0 container: update unit tests to match expected behaviour 2023-04-19 20:07:40 +02:00
Achilleas Koutsou
f9e3d8659d container: explicitly set LocalName
Set the LocalName for the spec using a separate argument in the
NewSpec() constructor instead of reusing the `source` arg.
The name is already available in the calling scope in the client's
Resolve() method.

If the LocalName is an empty string, default to the remote (source)
reference.  This is a change from the previous behaviour which only used
the base source.Name().  The full source corresponds to the
user-provided source value, which includes any specified tag or digest.

The `name` argument which is used in the `Resolve()` function should
always correspond to the user-provided container name.
2023-04-19 20:07:40 +02:00
Tomáš Hozza
c28f8077c1 Revert "distro/rhel8: don't install missing MSFT key into azure-sap-rhui"
This reverts commit 2b1facb44d.

The GPG key is now present in the RHUI client RPM, so there is no need
to not import it during the image build.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2023-04-19 15:04:42 +02:00
Ondřej Budai
d8342862d5 customizations: create all groups unconditionally
We had this weird condition in code that prevented composer to create groups
with the same name as a user has. This unfortunately means that you are not
able to create a user with a primary group with a certain GID that has the
same name as the user. There's the gid field in the user customization,
but it requires that the group already exists.

In order to allow that, we need to remove the condition. From now on, it's
possible to create groups with the same name as a user has, which can be used
to create primary groups with a custom gid.

Note that the lorax compatibility behaviour was actually wrong. When lorax was
given a custom gid for a user, it didn't require the gid to exist. When it
didn't, the group was just created. Thus, we still don't have full backward
compatibility, but at least we now have support for this.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-19 11:50:49 +02:00
Simon de Vlieger
258f450021 iot: change the copy stages
Split this into two separate copystages so that we are not overriding
types.
2023-04-18 21:21:09 +02:00
Simon de Vlieger
b0fce3bfd1 platform: add the aarch64_iot platform
This platform copies more files into `/boot` which are necessary to be
able to boot IoT on some single board computers.

We also immediately set this on the `Aarch64_IoT` platform which needs
u-boot to be placed in the `/boot`.

This closes #3312.
2023-04-18 21:21:09 +02:00
Simon de Vlieger
8041563a36 manifest: image-tree -> ostree-deployment
Describes much better what the pipeline actually does.
2023-04-18 21:21:09 +02:00
Simon de Vlieger
66345cf54c manifest: commit_deployment -> ostree_deployment 2023-04-18 21:21:09 +02:00
Achilleas Koutsou
ab9e612c99 containers: update tests
Update tests to include the list digests of the test images.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
d41a74475e osbuild/skopeo: helper function for files input
Add a helper function that collects all the manifest list digests from a
list of container specs and returns a FilesInput to be used with the
stage.

Use the function in the OS pipeline when adding containers.  The
manifests input to the stage constructor will be empty if there are no
manifest lists in the container specs.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
753bcaed78 osbuild: create skopeo-index source in GenSources()
When generating sources in GenSources(), add a skopeo-index source
reference for each list-digest found in the container specs.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
e5aaf3e514 container: save manifest list digest when resolving
While resolving a manifest list digest, store the list digest to return
with the resolvedIds.

This is done for both types of manifest list:
  application/vnd.docker.distribution.manifest.list.v2+json
and
  application/vnd.oci.image.index.v1+json
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
f38eb8406a container: include TLSVerify and ListDigest in spec ctor 2023-04-17 18:30:41 +02:00
Achilleas Koutsou
a7dd8ff94f container: add ListDigest to the spec
Add the ListDigest to the container Spec struct and all its copies so we
can store list digests when they are available and pass them on to the
appropriate osbuild stages, sources, and inputs.

Copy the value whenever a spec is moved to a different representation.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
c95072d686 osbuild: add manifest-lists input to skopeo stage
The skopeo stage in osbuild supports an second optional set of inputs
called `manifest-lists`.  This is an array of files, i.e.,
`org.osbuild.files` type input.

To support this we need a new type for the skopeo stage inputs that can
encompass both input types, images and manifest-lists.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
dfae9fbefb osbuild/skopeo: reverse the constructor args
Put the path (which becomes an option) first and the inputs second.
This is more in line with other stage constructors.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
069021093c osbuild: add skopeo-index source
New osbuild source that can download a manifest-list from a container
registry, using the `--multi-arch=index-only` option of skopeo copy.
2023-04-17 18:30:41 +02:00
Achilleas Koutsou
ad2a5bff7c osbuild: name in containers input is not optional
Remove the `omitempty` from the name field in the containers input.  It is
required.
2023-04-17 18:30:41 +02:00
Sanne Raymaekers
9dc0881247 internal/worker: log dequeue failures 2023-04-14 12:12:41 +02:00
Sanne Raymaekers
14a9b2395d cloudapi/v2: expose repo metadata verification 2023-04-13 22:08:17 +02:00
Simon de Vlieger
d94cea9150 fedora/disk: put a filesystem in EFI partition 2023-04-11 17:05:33 +02:00
Ondřej Budai
943ead790e upload/azure: skip uploading empty pages
The size of the page blob is defined on creation and the blob is
zero-initialized. Therefore, we can just skip all the pages that contain
only zeros. This should save a lot of bandwidth if used on sparse files as
e.g. operating system images. (:
2023-04-04 09:09:43 +02:00
Ondřej Budai
abe6ccfb50 upload/azure: migrate from azure-storage-blob-go to azure-sdk-for-go
https://github.com/Azure/azure-storage-blob-go/ is deprecated, the main SDK
should be now used instead. Let's migrate the code. There should be no
functional changes.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-04 09:09:43 +02:00
Ondřej Budai
9beddf626f upload/azure: remove the MD5 sum check
It doesn't actually make any sense. For Page Blobs, Azure doesn't compute any
hashes. The MD5 sum is basically just a property, which we set by one call and
get by the other call.

See
https://stackoverflow.com/questions/42229153/how-to-check-azure-storage-blob-file-uploaded-correctly/69319211#69319211

for more info.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-04-04 09:09:43 +02:00
Irene Diez
20a5b25320 distro/fedora: handle generated warnings during Manifest initialization
Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Irene Diez
2771af2530 distro/rhel7: handle generated warnings during Manifest initialization
Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Irene Diez
64a5946dc3 distro/rhel8: handle generated warnings during Manifest initialization
Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Irene Diez
d6b9b3a5bf distro/rhel9: handle generated warnings during Manifest initialization
Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Irene Diez
5fb989110a weldr: allow to send warnings on ComposeReply
This adds a new field `Warnings` to the `ComposeReply`
struct, allowing to send back any warnings (e.g. deprecation
notices) generated during the `checkOptions` step of the
manifest initialization.

See also https://github.com/osbuild/weldr-client/pull/99 which
handles the weldr-client side of things.

Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Irene Diez
8022c227ba distro: allow to return warnings in the Manifest function
This changes the `Manifest` function of the `ImageType`
interface so that any warnings detected during the
`checkOptions` step of the manifest initialization can
be propagated back to the Weldr-API (see next commit).

Signed-off-by: Irene Diez <idiez@redhat.com>
2023-03-31 14:14:57 +02:00
Sanne Raymaekers
d952e41161 internal/boot: boot VMWare VMs with EFI and SCSI
Since we use streamOptimized images everywhere, we no longer need to use
IDE and boot with bios. Let's test a more realistic scenario.
2023-03-31 09:24:42 +02:00
Achilleas Koutsou
2a87a8b7f3 cloudapi: enable azure-eap7-rhui image type 2023-03-30 18:40:12 +02:00
Achilleas Koutsou
90cc2b18a8 distro/rhel8: assert error instead of skipping
We sometimes skip testing customizations for image types that don't
support them, other times, we check for a specific error.  If we don't
care about the specific error message that's received, running the test
and checking for error is better than skipping, because it ensures that
we're skipping the test for the right reasons and not accidentally
skipping a test for an image type that should support the
customizations.
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
203dd5275e distro/rhel8: exclude azure-eap7-rhui from customization tests
Skip customizations tests for the EAP image.

Create a map of unsupported image types for each test to make adding new
ones easier.
Use a `nil` struct pointer for the pipeline names test instead of an
empty struct.  Non-nil customizations can't be compared.
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
21d13a2ae3 distro/rhel8: specify ports for azure EAP
See COMPOSER-1859
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
fb5accc227 distro/rhel8: add env vars to azure EAP
Add the required environment variables to Azure EAP.
See COMPOSER-1860
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
3f299c7d43 distro: add ShellVars field to ImageConfig 2023-03-30 18:40:12 +02:00
Achilleas Koutsou
816d23988c manifest: add support for shell vars in OSCustomizations
ShellVars are added as a nested map of the form
filename->key->value.  A stage is generated and added if any are
defined.
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
92bbf5fb8b osbuild: add generator function for shell.init stage
Generates a stage from a basic nested map.
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
a024b923a3 shell: new package for representing shell init configs internally
New types for the internal representation of the shell init configs,
which currently consists of files and key-value pairs for environment
global variables.
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
2fcf3582b5 osbuild: add shell.init stage
Add support for the org.osbuild.shell.init stage and test validator.
2023-03-30 18:40:12 +02:00
Achilleas Koutsou
636f3ff237 distro/rhel8: add azure EAP 7 image type
Initialise the image type as an Azure RHUI type with the name
"azure-eap7-rhui" and a workload attached that provides the extra
packages required by the EAP workload.
2023-03-30 18:40:12 +02:00