Commit graph

6730 commits

Author SHA1 Message Date
Ondřej Budai
5c9af03785 osbuild2/grub2.inst: always include location in stage options
Location property is actually required. If it's omitted from the stage
options, the stage just fails.

Also see: https://github.com/osbuild/osbuild/pull/759

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-08-05 13:43:17 +02:00
Tomas Hozza
e37d5e5b17 rhel85: include dracut-config-generic in x86 UEFI boot package set
Fix the split of x86_64 boot package set into the Legacy and UEFI set.
The package `dracut-config-generic` is needed for Legacy, UEFI and
Hybrid boot mode, therefore it must be included in both package sets.

In the case of the Hybrid mode, the duplicated package is not expected
to cause any issues, because it should not appear as duplicated after
depsolving the package set using DNF.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-05 09:34:07 +02:00
Tomas Hozza
a79e541cec osbuild2: fix typo in files_input.go
Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-05 09:34:07 +02:00
Tomas Hozza
a590dd7591 docs/news: reflect EC2 image changes in the RHEL-8.5 images note
The `ami` image type was redefined based on the official RHEL EC2
images. In addition, two new image types `ec2` and `ec2-ha` were
defined. These new image types are by default not exposed via Weldr
API.

Update the docs/news/unreleased entry related to RHEL-8.5 main images to
reflect these changes.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-05 09:34:07 +02:00
Tomas Hozza
4e92b65721 composer: don't expose ec2 and ec2-ha RHEL images via WeldrAPI
The `ec2` and `ec2-ha` images include RHUI client packages, which are
not publicly available. For this reason, building of such images in the
on-premise use case via WeldrAPI would always fail, unless the system
would be inside the Red Hat internal network or VPN.

Mark the `ec2` and `ec2-ha` image types for `rhel-*` distribution as
denied in WeldrAPI by default.

Extend and modify affected unit tests.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-05 09:34:07 +02:00
Tomas Hozza
aed3bf785c image-info: mount partitions in correct order when analysing image
image-info's code which analysed image with multiple partitions was not
correctly working with more than two partitions, which had to be the
root '/' and EFI partition '/boot/efi'. The consequence was that SELinux
labels on paths which were mounted incorrectly could have been reported
as incorrect.

Modify `append_partitions()` to first read the fstab entries and then
mount all partitions using their UUID in the correct order. Only then
analyze the image filesystem tree.

Regenerate affected image test cases.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-05 09:34:07 +02:00
Tomas Hozza
972515ad84 image-info: ignore lines with only whitespace characters in fstab
image-info could produce a weird fstab error with an empty list as a
member, when analysing images not built using osbuild. Ensure that any
lines in fstab with only whitespace characters are skipped during image
analysis.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-05 09:34:07 +02:00
Chloe Kaubisch
53109945ef cloudapi: create binder
Add an echo.Binder() to simplify parsing the compose request body.
2021-08-04 11:49:50 +02:00
sanne
57555991e0 test/api: Use relevant distributions testing package conflicts 2021-08-03 15:14:22 +02:00
Chloe Kaubisch
29af662bf8 cloudapi: replace chi with echo
standardize middleware by replacing chi with echo
2021-08-03 13:09:13 +02:00
Tomas Hozza
c97420e879 rhel85: add definitions of RHEL ec2 and ec2-ha images
Add new image type definitions `ec2` and `ec2-ha` representing the
official RHEL ec2 image types.

Add a `xzArchivePipeline()`, which returns a pipeline producing a XZ
archive from a file produced by a different pipeline.

Add rpmrepo snapshots for `rhui` and `ha` repositories used to generate
image test cases. `rhui` is used by the `ec2` image and it is available
on x86_64 and aarch64 architectures. `ha` is used by the `ec2-ha` image
and it is available only for x86_64.

The new image type definitions are currently not used by any
API test case.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-02 19:05:09 +02:00
Tomas Hozza
49ed70a565 rhel85: introduce BootType and use it for arch and image definitions
Previously, the support of UEFI has been captured only on the level or
architecture definition as a binary boolean value. In reality some of
the architectures are able to support legacy, UEFI or hybrid boot.

Introduce a new BootType value, defined on the architecture level, which
can be set to one of the three boot types mentioned above. The value set
on the architecture level can be overridden on the image type level in
the image type definition.

Add two unexported helper methods to the `imageType`, specifically
`getBootType()` which returns the boot type that  should be used for the
image type and architecture combination. The values set explicitly in
the image type or architecture definition should not be used directly.
Second added method is `supportsUEFI()`, which returns boolean value
representing the fact if the image type supports UEFI boot.

Split and define the boot package sets separately for the legacy and
UEFI boot. The `PackageSets()` method of the imageType structure is
modified to take the boot type into consideration and append appropriate
package sets to the "os" package set.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-02 19:05:09 +02:00
Tomas Hozza
02ff0fc606 rhel85: redefine the ami image based on RHEL ec2 images
Redefine the `ami` image type in RHEL-8.5 to be based on RHEL
ec2 images. The pipeline has different default settings, therefore the
common "os" pipeline is not used. The RHEL ec2 images have a different
default size than the original `ami` image definition. The RHEL ec2
images use a different default partitioning scheme. Their configuration
is slightly different for each architecture and the x86_64 version
of the image does not support UEFI.

Update rpmrepo snapshots used to generate RHEL-8.5 x86_64 and aarch64
image test cases.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-02 19:05:09 +02:00
Tomas Hozza
a2ecf46e04 osbuild2: extend chrony stage with leapsectz property
Add support for specifying the `leapsectz` property of
`org.osbuild.chrony` stage. Modify stage tests to test setting of the
value.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-02 19:05:09 +02:00
Tomas Hozza
6c32ff048a internal/common: add function to get a pointer to string literal
Add a new helper function to get a pointer to a string literal.

Add unit test for the new function.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-02 19:05:09 +02:00
Tomas Hozza
52ccf1d6ef osbuild2: add xz stage implementation
Add support for osbuild `org.osbuild.xz` stage. The stage accepts Files
inputs.

Add unit tests for the added functionality.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-02 19:05:09 +02:00
Tomas Hozza
8271910051 osbuild2: refactor files inputs
osbuild stage inputs were originally implemented in composer as
stage-specific inputs, while in reality, they are defined as individual
inputs, usually accepted by multiple stages. Therefore a single stage
input can be passed to any stage, as long as the stage accepts it.

Files inputs type was previously defined, but not used by any stage.
Creation of proper inputs type structures is currently handled in
`internal/distro/rhel85/stage_inputs.go` instead.

Refactor files inputs type to be usable directly as an input type
structure for stages, which accept it. For now, implement only the
`org.osbuild.pipeline` origin and related input reference.

Add unit tests for the `FilesInputs`.

Define input origin names as string constants, so that they can be used
by inputs implementations, instead of using string literals.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-02 19:05:09 +02:00
Tomas Hozza
241c5cc9d6 osbuild2: fix typo in systemd-logind stage
Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-02 19:05:09 +02:00
Tomas Hozza
0a71054d86 Weldr API: allow globing in distro image type deny list
Allow globing patterns in distro-specific image type deny list of Weldr
API configuration. Extend unit tests to verify simple globing patterns.

Update NEWS entry.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-02 18:51:03 +02:00
Tomas Hozza
a9b676e43e Weldr API: return the actual getImageType() error in composeHandler()
Refactor the `composeHandler()` method to send the actual error
returned by `getImageType()` as an API response.

Modify tests to handle the changed error message in API calls.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-02 18:51:03 +02:00
Tomas Hozza
dc8a93b3f6 Weldr API: rename checkImageTypeDenylist() and return bool
Rename the `checkImageTypeDenylist()` method to `isImageTypeAllowed()`
and return boolean value instead of error.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-02 18:51:03 +02:00
Tomas Hozza
b150d57c18 Weldr API: make Image Type denylist distribution-specific
Change the Image Type denylist in Weldr API from being applied to all
distributions to being distribution-specific. A special name `*`
can be used in the configuration to match any distribution
or any image type.

Modify NEWS entry and unit tests to reflect this change.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-02 18:51:03 +02:00
Tomas Hozza
076bbc5456 Weldr API: introduce Image Type denylist for filtering exposed images
Extend Weldr API to accept a list of denied image types, which should
not be exposed via API for any supported distribution. This
functionality will be needed to not expose image types which can't be
successfully built outside of Red Hat VPN. Example of such images are
the official RHEL EC2 images, which include RHUI client packages not
available publicly.

Image Types are filters when listing available compose types and
creating a new compose using Weldr API.

Extend osbuild-composer configuration to allow specifying the list of
denied Image Types for Weldr API.

Add unit tests for implemented changes.

Add NEWS entry describing the newly introduced functionality.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-08-02 18:51:03 +02:00
Alexander Todorov
c70e437272 Enable additional aarch64 test runners
Signed-off-by: Alexander Todorov <atodorov@redhat.com>
2021-07-30 22:03:00 +02:00
Achilleas Koutsou
248632eaba test/data: update manifests
The previous manifests were generated inside a RHEL VM which means that
the selinux context mismatch check didn't run properly
(see https://bugzilla.redhat.com/show_bug.cgi?id=1973754).

Regenerating them on F34 shows some mismatches for certain image types.

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-07-29 16:45:20 +02:00
Achilleas Koutsou
51cb4ae043 distro/rhel85: move selinux stage to last
In the main OS pipeline created by osPipeline(), there is often a set of
stages that are added after the common ones. The SELinux stage must
always be last in the pipeline, so it is added externally by the calling
function.

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-07-29 16:45:20 +02:00
Achilleas Koutsou
daae2bbfca distro/rhel85: set build root for main os pipeline
OS image tree was being built on the host instead.

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-07-29 16:45:20 +02:00
Juan Abia
c4fa4e4bb7 test: Replace hard coded ssh keys on some test
Generate a new ssh key each time. Only implemented on bash test scripts,
go tests pending.
2021-07-29 09:38:03 +03:00
Lars Karlitski
9c2c92f729 jobqueue: Introduce jobqueue backed by a postgres database
Co-authored-by: sanne <sanne.raymaekers@gmail.com>
2021-07-28 21:52:31 +01:00
Lars Karlitski
871c6e9cbb fsjobqueue: make canceling a finished job an error
This mirrors FinishJob(), which also errors when the job is already
finished.
2021-07-28 21:52:31 +01:00
Lars Karlitski
30492bfc60 jobqueue: move fsjobqueue's generic tests into new package
fsjobqueue_test contained tests that are generically testing the
JobQueue interface. Split those out into its own package `jobqueuetest`.

These tests will be useful when implementing a new package that conforms
to the JobQueue interface.
2021-07-28 21:52:31 +01:00
Gianluca Zuccarelli
bf62f3868e weldr: add filesystem configuration support 2021-07-28 21:44:03 +01:00
Gianluca Zuccarelli
fa15541ea5 blueprint: test minsize and sector-size validation 2021-07-28 21:44:03 +01:00
Gianluca Zuccarelli
90733e79f3 blueprint: add filesystem minsize function 2021-07-28 21:44:03 +01:00
Gianluca Zuccarelli
7408be580c client: test supported and unsupported mountpoints 2021-07-28 21:44:03 +01:00
Gianluca Zuccarelli
c671a0dab1 distro: test filesystem errors 2021-07-28 21:44:03 +01:00
Gianluca Zuccarelli
f4ee0bfabf distro: add root filesystem support 2021-07-28 21:44:03 +01:00
Gianluca Zuccarelli
d0e4558b2b blueprint: add filesystem customisations support 2021-07-28 21:44:03 +01:00
Martin Sehnoutka
700afb5af9 test/data: remove rhel-8-beta repository override
It contains beta repositories for RHEL 8.3, which is far outdated and
the file is not used anywhere any more.
2021-07-28 13:31:44 +02:00
Thomas Lavocat
4729990ac0
cloudapi: Add User customization
Cloud api now exposes user customization that let a customer able to add
a new user with a set of groups and a ssh key.

Testing:
* adds 2 users to the AWS image, accessible with a temp ssh key.
* the first one is in the group wheel, the other is not

Fixes #1574
2021-07-28 10:40:25 +02:00
Thomas Lavocat
5e127de303 Worker: report stages status after RunOSBuild
To help along with debugging, this commit makes the worker able to print
the status of the different stages with a oneliner for each successfull
stages and a detailed message for failed ones.

Sample output:
Jul 23[..]: Build stages results:
Jul 23[..]: org.osbuild.rpm  success
Jul 23[..]: org.osbuild.selinux  success
Jul 23[..]: Stages results:
Jul 23[..]: org.osbuild.rpm  success
Jul 23[..]: org.osbuild.fix-bls  success
Jul 23[..]: org.osbuild.fstab  success
Jul 23[..]: org.osbuild.grub2  success
Jul 23[..]: org.osbuild.locale  success
Jul 23[..]: org.osbuild.timezone  success
Jul 23[..]: org.osbuild.users failure:
Jul 23[..]:  [/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping
Jul 23[..]: All rules containing unresolvable specifiers will be skipped.
Jul 23[..]: Failed to create file /sys/fs/selinux/checkreqprot: Read-only file system
Jul 23[..]: useradd: group 'toto' does not exist

Fixes #1584
2021-07-27 18:13:10 +02:00
Achilleas Koutsou
df1eddbf07 test/data: update aarch64 manifests
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-07-27 16:53:24 +01:00
Achilleas Koutsou
b113cf0f8a test/data: update x86-64 manifests
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-07-27 16:53:24 +01:00
Achilleas Koutsou
9bb2ae09bd distro/rhel85: fix pkg set for grub2 stage
The package set is needed by the GRUB 2 stage options function to find
the Kernel package in order to add the saved entry option. The Kernel
package is defined in the blueprints package set, not the OS package
set.

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-07-27 16:53:24 +01:00
Martin Sehnoutka
b950d6e062 weldr: accept rhsm parameter in sources
The system sources allow specification of the rhsm parameter, but it
isn't available in the sources configured over the Weldr API. This patch
implements support for it.
2021-07-27 15:39:36 +02:00
Ondřej Budai
05ffdc70aa ci: run only koji.sh for rhel_90 distro
Previously, all sorts of tests that provided no value were run for RHEL 9.0.
This commit limits its testing to the only test that makes sense: koji.sh.
See #1461 for more details.

Fixes #1461

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-07-27 10:18:05 +03:00
Tomas Hozza
928131003c osbuild2: fix incorrect function name in dracut.conf stage
Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-07-26 22:01:33 +02:00
Tomas Hozza
cdc65f82da osbuild2: extract unit drop-ins from systemd -> systemd.unit
The part creating Systemd unit drop-ins was extracted from
`org.osbuild.systemd` stage to `org.osbuild.systemd.unit`,
before the osbuild v30 release. Update the composer implementation
to reflect the change.

Related to https://github.com/osbuild/osbuild/pull/739.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-07-26 22:01:33 +02:00
Tomas Hozza
49aebf1c68 osbuild2: change systemd-logind stage to match osbuild v30
The `org.osbuild.systemd-logind` stage was reworked before the osbuild
v30 release. Update the composer implementation to match the expected
schema.

Related to https://github.com/osbuild/osbuild/pull/739.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-07-26 22:01:33 +02:00
Tomas Hozza
6cd626d462 osbuild2: change dracut.conf stage to match osbuild v30
The `org.osbuild.dracut.conf` stage was reworked before the osbuild v30
release. Update the composer implementation to match the expected
schema.

Related to https://github.com/osbuild/osbuild/pull/739.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-07-26 22:01:33 +02:00