Commit graph

1066 commits

Author SHA1 Message Date
Tomas Hozza
f06e66b94b osbuild2: support for servers option in org.osbuild.chrony
Add support for the newly added `servers` option in the
`org.osbuild.chrony` osbuild stage [1]. The option allows one to
specify timeservers to be used by chrony, including a subset of
lower-level configuration options per each server.

Implement a custom JSON marshaller method for `ChronyStageOptions` to
ensure that exactly one of 'timeservers' or 'servers' is specified, as
mandated by the stage schema.

Optional values in `ChronyConfigServer` are declared as pointers to
distinguish the case when the value was explicitly set by the user from
the default value when the structure instance is created. All of these
options should be omitted from the JSON, but only when not explicitly
set, not when their value us "0" for int or "false" for bool. Downside
of this approach is that one can not easily use pointer to a basic
type literal in the struct literal. Passing the basic type literal has
to be workarounded using an intermediate variable, which address is
used in the struct literal.

Add unit test cases for the added functionality.

[1] https://github.com/osbuild/osbuild/pull/692

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-07-08 19:37:09 +02:00
Tomas Hozza
8367542f25 osbuild2: support for subscription-manager option in org.osbuild.rhsm
Add support for the newly added `subscription-manager` option in the
`org.osbuild.rhsm` osbuild stage [1]. The option allows one to
create configure subset of subscription-manager options in its
configuration file `/etc/rhsm/rhsm.conf`.

Add unit test cases for the added functionality.

[1] https://github.com/osbuild/osbuild/pull/659

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-07-08 19:37:09 +02:00
Tomas Hozza
29ade764ce osbuild2: support for unit-dropins option in org.osbuild.systemd
Add support for the newly added `unit-dropins` option in the
`org.osbuild.systemd` osbuild stage [1]. The stage allows one to
create `.service` unit files drop-in configuration files under
`/usr/lib/systemd/system/`.

Add unit test cases for the added functionality.

[1] https://github.com/osbuild/osbuild/pull/664

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-07-08 19:37:09 +02:00
Tomas Hozza
20e1cfeba4 osbuild2: add support for org.osbuild.cloud-init stage
Add support for the `org.osbuild.cloud-init` osbuild stage [1],
which allows one to configure cloud-init by creating
configuration files under `/etc/cloud/cloud.cfg.d/`.

Add unit test cases for the newly added stage.

[1] https://github.com/osbuild/osbuild/pull/689

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-07-08 19:37:09 +02:00
Tomas Hozza
9957f378ca osbuild2: support for network-scripts option in org.osbuild.sysconfig
Add support for the newly added `network-scripts` option in the
`org.osbuild.sysconfig` osbuild stage [1]. The stage allows one to
create `ifcfg` configuration files under
`/etc/sysconfig/network-scripts`.

Add unit test cases for the added functionality as well as for other
options of the stage, which were missing.

[1] https://github.com/osbuild/osbuild/pull/663

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-07-08 19:37:09 +02:00
Tomas Hozza
bef4628a08 osbuild2: add support for org.osbuild.systemd-logind stage
Add support for the `org.osbuild.systemd-logind` osbuild stage [1],
which allows one to configure systemd-logind by creating
configuration drop-ins.

Add unit test cases for the newly added stage.

[1] https://github.com/osbuild/osbuild/pull/668

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-07-08 19:37:09 +02:00
Tomas Hozza
5e97dcf2b3 osbuild2: add missing JSON unmarshaling test for org.osbuild.dracut stage
The `org.osbuild.dracut` stage was previously not tested for JSON
unmarshalling. Add the missing test case and extend Stage's
`UnmarshalJSON()` method.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-07-08 19:37:09 +02:00
Tomas Hozza
92719e05a2 osbuild2: add support for org.osbuild.dracut.conf stage
Add support for the `org.osbuild.dracut.conf` osbuild stage [1], which
allows one to configure dracut by creating configuration files.

Add unit test cases for the newly added stage.

[1] https://github.com/osbuild/osbuild/pull/688

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-07-08 19:37:09 +02:00
Tomas Hozza
c0e8d4bdc4 osbuild2: add support for org.osbuild.modprobe stage
Add support for the `org.osbuild.modprobe` osbuild stage [1], which
allows one to configure modprobe by creating configuration files.

Add unit test cases for the newly added stage.

[1] https://github.com/osbuild/osbuild/pull/670

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-07-08 19:37:09 +02:00
Tomas Hozza
9386a02984 internal/common: add helper functions to get pointer to a basic type value
Add two new helper functions `IntToPtr()` and `BoolToPtr()` to the
`internal/common` package, which can be used to conveniently set
basic type literals in a struct literal, in which pointer to the basic
type is expected.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-07-08 19:37:09 +02:00
Ondřej Budai
385648223d spec: drop hacks for Fedora 32
There are not needed anymore, yay!

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-07-05 11:16:08 +02:00
Ondřej Budai
5ae6203d65 distro: drop Fedora 32
It's EOL.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-07-05 11:16:08 +02:00
Gianluca Zuccarelli
ea5b7e113a fix: small typo in code comments 2021-07-02 10:38:04 +02:00
Gianluca Zuccarelli
9aa8ebdfff fix: unused function arg 2021-07-02 10:38:04 +02:00
Achilleas Koutsou
5f53b3865f rhel85: append "boot" package sets to "packages"
If an image type is marked "bootable", append the package set with  name
"boot".
2021-07-01 12:48:32 +02:00
Achilleas Koutsou
e8dd5b3248 rhel85: move package sets to separate file
Package sets shared between multiple image types are statically defined
in functions.
2021-07-01 12:48:32 +02:00
Achilleas Koutsou
49df4a70af rhel85: add base build package set to distribution 2021-07-01 12:48:32 +02:00
Achilleas Koutsou
f1e805658c rhel85: distro and arch package sets
Add package sets to the top level distribution and the
architectures. Merge package sets when requested through
`imageType.PackageSets()`.
This allows us to have "cascading" package sets:
    distro > arch > image type

Previously this was only supported for the build packages. The idea is
generalised based on the idea of arbitrary package sets.

Certain package sets have special meaning (for the distro):
- packages: blueprint packages and special option packages are merged
  into this one.
- build: is not expected to be defined on the image type so it is merged
  from the distro and arch explicitly.
2021-07-01 12:48:32 +02:00
Achilleas Koutsou
38f5b23e87 rhel85: tar image type
Traditional tar image type.
Pipelines generation function for tar image type uses the same pipelines
and stages as the tar installer, but exports the OS image directly.
2021-07-01 12:48:32 +02:00
Achilleas Koutsou
ec381fefb9 rhel85: generalise and simplify tar stage creation 2021-07-01 12:48:32 +02:00
Achilleas Koutsou
7e1ab365c8 rhel85: change kickstart file location
Global const for defining kickstart path for all installer types.
Kernel options added to point to path.

Kickstart file renamed to osbuild.ks for all types and placed in the
bootiso root.

Fixes #1363
2021-07-01 12:48:32 +02:00
Achilleas Koutsou
32c69b3cf9 rhel85: add subscription stage to core pipeline 2021-07-01 12:48:32 +02:00
Achilleas Koutsou
242f6d3c44 rhel85: tar installer main pipeline
Define a core set of packages and stages shared by images.
2021-07-01 12:48:32 +02:00
Achilleas Koutsou
19d15af788 rhel85: new image type tar-installer 2021-07-01 12:48:32 +02:00
Ondřej Budai
e1e4a1f845 distro/rhel90: add s390utils-core to the build pipeline for s390x
A recent update to crypto-policies changed its requirements from:

Recommends: grubby

to:

Recommends: (grubby if kernel)

We don't install kernel in the build pipeline, thus grubby was now no longer
installed. This caused also s390utils-core to not be installed on s390x.
s390utils-core has to be in the build pipeline though because we use
/usr/sbin/zipl from it to install the bootloader to s390x images.

Long story short, images for s390x currently cannot be build because
/usr/sbin/zipl is no longer in the build pipeline. This commit fixes that
by explicitly adding s390utils-core to the package list for s390x.

I verified this fix on an actual s390x machine. The test manifest was
regenerated but it's actually still the same because we use an older
snapshot that doesn't contain the newest crypto-policies change.

Kudos to Tomáš Hozza for helping me out!

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-06-30 14:48:57 +02:00
Achilleas Koutsou
779103cb96 rhel85: add tests 2021-06-29 17:26:43 +01:00
Achilleas Koutsou
e388706712 store: add new image type mappings and fix typos 2021-06-29 17:26:43 +01:00
Achilleas Koutsou
139c8eb3c4 Revert "rhel85: nolint package for unused private functions"
Pipeline generation functions are now used by the new image definitions.

This reverts commit cca084d5ce.
2021-06-29 17:26:43 +01:00
Achilleas Koutsou
f789f785a4 rhel85: rhel-edge image definitions
Image definitions added for all three edge types:
edge-commit
edge-container
edge-installer

Manifest pipelines generation methods attached to image types.
2021-06-29 17:26:43 +01:00
Ondřej Budai
9bcd219528 distro/rhel90: make selinux stage the last one
There are currently some context mismatches in the image because of the new
files from the sysconfig stage. Let's move the selinux stage to the end so
we fix them.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-06-29 13:56:26 +02:00
Ondřej Budai
4c16b2f77e distro/rhel90: include qemu-guest-agent in qcow2
Fixes COMPOSER-990

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-06-29 13:56:26 +02:00
Ondřej Budai
c37705e7d5 distro/rhel90: move all package lists to a separate file
There's currently the Package Deletion Process going on in CentOS Stream 9.
When a maintainer wants to get rid of a package, several steps must be taken
before the package is completely removed from all places. We want to include
osbuild-composer in the process so packages that are used here are not deleted
without us noticing it. Thus, maintainers going through the process should be
able to easily see all packages that osbuild-composer includes in images.

This commit moves all package lists from rhel90/distro to rhel90/packages
to allow CentOS Stream maintainers to easily packages required by composer.

Not a functional change.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-06-29 13:56:26 +02:00
Achilleas Koutsou
cfb244ab34 cloudapi: check if metadata is nil before converting 2021-06-29 09:33:05 +01:00
Achilleas Koutsou
51503cc000 cloudapi: collect RPMs from v2 result stages
Fix for v1 results: collect RPMs for main stages, not build.
Support for v2 results: filter out build stages from stage results,
which include stages from all pipelines (flattened).
2021-06-29 09:33:05 +01:00
Achilleas Koutsou
b46addea5c osbuild1: convert stage metadata from v2
While unmarshalling osbuild output and converting from v1 to v2, convert
the metadata as well.

Added checks to successful parsing and conversion test.
2021-06-29 09:33:05 +01:00
Achilleas Koutsou
1dcaa26966 osbuild2: add support for reading stage metadata
Metadata from stages in the output from osbuild in the v2 schema are
stored at the top level of the result structure.

Adding the same Metadata types from v1:
- org.osbuild.rpm stage
- org.osbuild.ostree.commit stage
- Raw metadata (for every other stage type)
2021-06-29 09:33:05 +01:00
Achilleas Koutsou
dce3e1de9e distro: test manifest version detection 2021-06-29 09:33:05 +01:00
Achilleas Koutsou
39e646ae68 cloudapi: return ostree commit ID with metadata
When the assembler (v1) or any stage (v2) has type ostree.commit,
retrieve the commit ID (hash) from the osbuild stage metadata and return
it with the rest of the metadata (package NEVRAs).
2021-06-29 09:33:05 +01:00
Achilleas Koutsou
e87394fea1 distro: method for detecting manifest version
Small helper method for distro.Manifest that can detect the schema
version of the Manifest.
2021-06-29 09:33:05 +01:00
Achilleas Koutsou
72f0f7ffed osbuild1: store all stage result metadata
Currently, metadata from osbuild is discarded for all stages except RPM.
Adding explicit support for the ostree.commit stage/assembler and
storing the metadata in a known type.
For all other stages, store the metadata directly without parsing.

The rawAssemblerResult is removed. Assembler results are treated as
stage results.
2021-06-29 09:33:05 +01:00
Achilleas Koutsou
df5921403d cloudapi: add compose/id/metadata endpoint
Returns NEVRAs, compiled from the job manifest.
Will also return the ostree commit ID for edge commit types.
2021-06-29 09:33:05 +01:00
Achilleas Koutsou
6b3920783f rpmmd: move RPM metadata tooling to internal pkg
Move the OSBuildStagesToRPMs function, associated test, and RPM type
from the worker into the rpmmd subpackge. We will use this function in
the cloud API to compile the NEVRAs for the new metadata endpoint.
2021-06-29 09:33:05 +01:00
Ondřej Budai
579a5df698 upload/aws: add support for session tokens
If a user uses a temporary access key for login, a session token is also
needed.

This commit adds support for it to the internal aws library and also
to the osbuild-upload-aws helper. Note that this doesn't affect the main
osbuild-composer executable nor the worker. Everything here should work
as before and session tokens are not supported. Something for a follow up
if anyone needs it.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-06-28 13:14:19 +03:00
sanne
4f86b4fd45 worker: Use http.PostForm to post data
Avoid having to encode the data ourselves.
2021-06-23 10:33:22 +02:00
Colin Walters
a6ccde879b rpm_ostree_stage: Minor fixes to comments
Just a drive by commit while I was reading the code.
2021-06-21 12:11:09 +02:00
Achilleas Koutsou
1a3447ed38 kojiapi: include image type exports in Koji job args
Koji image request handling now reads the exports defined by each image
type. All APIs now support reading the exports defined by each image
type. The worker still falls back to "assembler" in case the call comes
from an older version of composer.
2021-06-18 14:02:09 +01:00
Achilleas Koutsou
14a74cf173 ostree: add test for VerifyRef() 2021-06-18 14:02:09 +01:00
Achilleas Koutsou
e5b28c0bb3 New upload target: AWS S3
Uploads an artifact to an S£ bucket and returns a presigned URL to allow
the user to download the file.

Although it uses a lot of common code with the AWS AMI upload target,
it's treated as a completely separate target.
2021-06-18 14:02:09 +01:00
Achilleas Koutsou
49a8d98c28 Move ResolveRef test to new package 2021-06-18 14:02:09 +01:00
Achilleas Koutsou
b2f5e1cd72 cloudapi: support ostree options
Move OSTree option handling outside of the weldr API to make it usable
by other packages. New subpackage at internal/ostree.

Add support for ostree options ("Ref" and "URL") in the Cloud API.
Validate OSTree options and resolve the parent reference the same way as
in the Weldr API.

Unlike the Weldr API, the Cloud API doesn't support specifying the
Parent reference directly.

The exports list is included in the job information on the queue.
2021-06-18 14:02:09 +01:00