There are currently some context mismatches in the image because of the new
files from the sysconfig stage. Let's move the selinux stage to the end so
we fix them.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
There's currently the Package Deletion Process going on in CentOS Stream 9.
When a maintainer wants to get rid of a package, several steps must be taken
before the package is completely removed from all places. We want to include
osbuild-composer in the process so packages that are used here are not deleted
without us noticing it. Thus, maintainers going through the process should be
able to easily see all packages that osbuild-composer includes in images.
This commit moves all package lists from rhel90/distro to rhel90/packages
to allow CentOS Stream maintainers to easily packages required by composer.
Not a functional change.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Fix for v1 results: collect RPMs for main stages, not build.
Support for v2 results: filter out build stages from stage results,
which include stages from all pipelines (flattened).
Metadata from stages in the output from osbuild in the v2 schema are
stored at the top level of the result structure.
Adding the same Metadata types from v1:
- org.osbuild.rpm stage
- org.osbuild.ostree.commit stage
- Raw metadata (for every other stage type)
When the assembler (v1) or any stage (v2) has type ostree.commit,
retrieve the commit ID (hash) from the osbuild stage metadata and return
it with the rest of the metadata (package NEVRAs).
Currently, metadata from osbuild is discarded for all stages except RPM.
Adding explicit support for the ostree.commit stage/assembler and
storing the metadata in a known type.
For all other stages, store the metadata directly without parsing.
The rawAssemblerResult is removed. Assembler results are treated as
stage results.
Move the OSBuildStagesToRPMs function, associated test, and RPM type
from the worker into the rpmmd subpackge. We will use this function in
the cloud API to compile the NEVRAs for the new metadata endpoint.
If a user uses a temporary access key for login, a session token is also
needed.
This commit adds support for it to the internal aws library and also
to the osbuild-upload-aws helper. Note that this doesn't affect the main
osbuild-composer executable nor the worker. Everything here should work
as before and session tokens are not supported. Something for a follow up
if anyone needs it.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Koji image request handling now reads the exports defined by each image
type. All APIs now support reading the exports defined by each image
type. The worker still falls back to "assembler" in case the call comes
from an older version of composer.
Uploads an artifact to an S£ bucket and returns a presigned URL to allow
the user to download the file.
Although it uses a lot of common code with the AWS AMI upload target,
it's treated as a completely separate target.
Move OSTree option handling outside of the weldr API to make it usable
by other packages. New subpackage at internal/ostree.
Add support for ostree options ("Ref" and "URL") in the Cloud API.
Validate OSTree options and resolve the parent reference the same way as
in the Weldr API.
Unlike the Weldr API, the Cloud API doesn't support specifying the
Parent reference directly.
The exports list is included in the job information on the queue.
composer doesn't support f34 or f35, but it should be possible to build
these even with the f33 distro definition. Introduce f34 and f35 repos
and aliases for f33.
cloud-init was enabled explicitly in the image-factory kickstart and thus we
need to explicitly enable it too.
Fixes: rhbz#1960309
Fixes: COMPOSER-920
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Modify composer to use RepoRegistry, instead of loading the host
repositories, when initializing WeldrAPI.
Modify WeldrAPI to use RepoRegistry, instead of a map of repository
definitions. Make sure that the RepoRegistry method specific to image
type is used in Welder where appropriate. Specifically when depsolving a
Blueprint, which is used to build a specific image type. Update Weldr
API unit tests to reflect the change.
Add a new method to RepoRegistry, allowing to get list of repositories,
which should be used for building an image for a given architecture,
without specifying the exact image type. Add relevant unit tests.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Composer does not have 1:1 mapping of what can be the Host Distro name
and the names of supported distributions held in the Distroregistry.
The fact that the host distro `Name()` method as passed to the Weldr API
does not return the same name as what is used as distro name for
repository definitions. This makes it hard to use `distro.Distro` and
`distro.Arch` directly and rely on the values returned by them as their
name.
Add `New*HostDistro()` to all distro definitions, accepting the name
that should be returned by the distro's `Name()` method. This is useful
mainly if the host distro is Beta or Stream variant of the distro.
Change the distroregistry.Registry to contain host distro as a separate
value set when creating it using `New()` function. This value is
returned by `Registry.FromHost()` method. Determining the host distro is
handled by the `NewDefault()` function. Move the distro name mangling to
distroregistry package. Add relevant unit tests.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Extend the RepoConfig structure to contain new field ImageTypeTags.
Extend also other structures and functions as needed, to support loading
repository definitions, which use this new field. The idea is that a
repository should be used for building all image types, unless it has
some ImageTypeTags defined. In such case, it should be used only for
building the specific image types, which names are specified in the new
field.
Add RepoRegistry as a higher-level API to load and manage repository
definitions for each distribution. Currently it provides one method,
which returns a set of repositories needed to build a given image
type. The RepoRegistry uses the new ImageTypeTags field in the RepoConfig
structure and returns all the needed repositories for the image type.
Modify rpmmd unit tests and add unit tests for RepoRegistry.
Add News entry describing the change done to RepoConfig and its JSON
representation.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Separate the loading of repo definitions from JSON file from
`LoadRepositories()` to a standalone function
`loadRepositoriesFromFile()`, to make it easy to reuse it in the future.
Add unit tests for `LoadRepositories()` function.
Exclude github.com/osbuild/osbuild-composer/internal/rpmmd/test package
from test coverage. Package with just tests and no other code makes `go
test` to fail. This should be fixed in go 1.17.
See https://github.com/golang/go/issues/27333
Signed-off-by: Tomas Hozza <thozza@redhat.com>
fedoratest was yet another dummy distribution used by unit tests. After
the rework of test_distro, there is no reason to not use it as the only
distro implementation for testing purposes.
Remove fedoratest distro and replace it with test_distro in all affected
tests.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Extend the "Test Distro" implementation and definition to contain two
architectures and make the second architecture contain two image types.
Add New2() function returning another "Test Distro".
Modify the `internal/store` unit tests to reflect changes done to the
"Test Distro".
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The parent commit would be added to the sources unconditionally. This
is only necessary for the edge-installer image type.
This doesn't technically change the build behaviour of an existing
distro and image type. It simply avoids unnecessarily downloading an
ostree commit when only the ref is needed.
It does change the 'sources' section of the manifest however.
The single `pipelines()` function is now replaced by multiple functions
for different purposes:
- `edgeCorePipelines()` defines the pipelines that create an edge
commit. This is used by both the `edge-commit` and `edge-container`
images.
- `edgeCommitPipelines()` and `edgeContainerPipelines()` define the
pipelines for `edge-commit` and `edge-container` respectively. They
share the core pipelines but differ in their final pipeline for
assembling the image into either a tarball or a container.
- `edgeInstallerPipeline()` shares almost no common parts with other
pipelines (only the `buildPipeline()`).
The `pipelines` function for each image type is set during creation of
the instance.
Individual pipeline functions are no longer methods of the image type.
Separate function for installer-specific pipelines.
The installer image type is very different compared to the other edge
types (and even more so compared to the more general images), so
separating out the pipelines that are specific to the installer Manifest
makes the whole method a bit more readable.
This function is not "wired" to the main pipelines generation, but will
be when the image type is defined.
Added a helper function to the bootiso stage for setting the BCJ option
for xz compression.
The FSCompression struct is changed to use a pointer for the Options
substruct so it can be omitted when nil (omitempty).
Based on rhel84 with minor changes:
- moved options and customizations checking to its own method on
imageType.
- added global `osVersion = "8.5"` for use in various labels and
metadata files.
- pipelines() function is empty and returns with "not implemented" error
since no image types are defined.
When a users wants to install a package that itself is excluded or its
dependency is excluded, it fails the build. There is no known workaround
for this shorcoming of our current design.
Therefore, remove a package from the list of excluded if it is
explicitly mentioned in a blueprint. This will not solve the issue with
dependencies, but it will create a possibility of a workaround.
Also, introduce regression test to verify the bug fix and hook it into
CentOS CI (this issue was reported against RHEL, but CentOS runs on AWS
so it is better to verify the fix there).
