Commit graph

4389 commits

Author SHA1 Message Date
schutzbot
37fc807bfa Post release version bump
[skip ci]
2022-07-13 08:30:26 +00:00
imagebuilder-bot
60bcfbb186 packit: Enable Bodhi update feature
While this feature is 'not mature yet' according to Packit developers,
we can enable it because there's no harm done. If Packit fails to
publish the Bodhi update then fedora-bot will take care of it.
2022-07-13 10:08:23 +02:00
Achilleas Koutsou
2eb9833370 osbuild-image-tests: ignore LVM UUID
Unconditionally ignore the UUID of the LVM container.  We don't control
this value so it isn't stable and makes our image tests fail.
2022-07-12 13:28:36 +02:00
Tom Gundersen
e844453c85 manifest: make filename optional and generalise Tree
We have three kinds of operating system trees, until we unify them to one,
hide them behind one interface. Use this to read the architecture from the
Tree rather than pass it in as a string to parent pipelines.

Also, make the filename parameter optional in a few places, there should be no
reason to set this rather than introspect it (except for backwards
compatibility).

Lastly, add another playground example sample to build a raw image.
2022-07-12 08:19:57 +01:00
Tom Gundersen
d00b98c134 manifest: don't stutter
Replace `pipeline.OSPipeline` with `pipeline.OS`, etc.

Also rename `LiveImg` to `RawImage`.
2022-07-12 08:19:57 +01:00
Tom Gundersen
529bc803db runner: introduce runner abstraction
For now all it does is represent the name of the runner and what requirements
it has of the build pipeline.

Move some package definitions from the runner package set to where it belongs.
2022-07-12 08:19:57 +01:00
Tom Gundersen
33fe2da25c osbuild-playground: rework slightly
Invoke osbuild, rather than output the manifest. Make it easier to include
several image types.
2022-07-12 08:19:57 +01:00
Tom Gundersen
1b924ae30c osbuild-worker/osbuild: move out execution helper
Move the execution helper for osbuild into the osbuild2 package so it can be
reused. In the process, generalise it slightly.
2022-07-12 08:19:57 +01:00
Tom Gundersen
6ae65f1951 test/data/manifests: regenerate to drop build packages
With the more fine-grained build package set, different images will use
different build pipelines and each of them will be smaller.

We don't currently cache build pipelines so there is no downside to this. Even
when we start caching the difference between having one build pipeline per
image and one shared one is minimal at scale. This will still benefit users
doing one-off builds on-prem.

Most importantly, this tracks things correctly, making pipelines more
composable without having to maintain a global list of dependencies.
2022-07-12 08:19:57 +01:00
Tom Gundersen
fa7d2f2647 manifest/build: explicitly pull in glibc
This should have no practical effect, but ldconfig is used from
runners, so it is strictly speaking a requirement.

At the same time document the remaining TODO's in the build
pipeline.
2022-07-12 08:19:57 +01:00
Tom Gundersen
b405ce4d89 manifest/build: replace explicit dnf with implicit rpm
The pipelines don't use dnf, only rpm. Let the ones that need it pull it in.
2022-07-12 08:19:57 +01:00
Tom Gundersen
b714dcb780 manifest/build: pull in tar implicitly
The tar pipeline requires tar, let only that pull it in.
2022-07-12 08:19:57 +01:00
Tom Gundersen
fed620b861 disk/partition_table: expose GetBuildPackages() function
This describes the packages necessary to create the partition table.

Use this in the pipelines and drop the explicit mentions from the build
pipeline.
2022-07-12 08:19:57 +01:00
Tom Gundersen
0d3d35e154 manifest/build: pull in selinux packages implicitly
The build pipeline requires the selinux packages only if we are going to be
labelling the files in the target OS. Otherwise, skip it.

manifest/build: pull in selinux-targeted unconditionally

This is unconditionally used by the build pipeline itself, until we make that
conditional, it needs to be installed.
2022-07-12 08:19:57 +01:00
Tom Gundersen
8df99d2b31 manifest/build: we depend on /bin/cp, always pull in coreutils 2022-07-12 08:19:57 +01:00
Tom Gundersen
9160319152 manifest/build: move out qemu-img
Let each pipeline that requires it pull in qemu-img.
2022-07-12 08:19:57 +01:00
Achilleas Koutsou
2c2d4812d4 CI: bring back tests on RHEL 8.4
We stopped testing on RHEL 8.4 because it wasn't changing, but now it
will be (or might) since it lives inside the common rhel8 package.
Testing the distro ensures we don't break it.  RHEL 8.4 is still
supported as EUS.

We will soon change the distro definition to specifically build 8.4 EUS.

Pin osbuild version for RHEL 8.4.

Change the ostree test to support 8.4 (and not 8.5).
2022-07-08 09:30:34 +01:00
Achilleas Koutsou
fdc6790472 distro: remove rhel84 package 2022-07-08 09:30:34 +01:00
Achilleas Koutsou
6c7c55825d diff-manifests: save diff as artifact
Manifest diffs can sometimes get large and putting them in the log makes
life harder for everyone.
Save them in a single file in the job artifacts instead.

Update the comment left by Schutzbot on the PR to mention the artifacts.
2022-07-08 09:30:34 +01:00
Achilleas Koutsou
f176326b71 test: update RHEL 8.4 manifests and image info
Regenerated all RHEL 8.4 manifests and image info with new definitions.
Edge image types renamed from `rhel-edge-*` to `edge-*`.
2022-07-08 09:30:34 +01:00
Achilleas Koutsou
577ea9eb95 format-request-map: minor fixes and updates
- azure-rhui filename fix (.xz)
- azure-rhui doesn't need distro override
- minor formatting fixes
2022-07-08 09:30:34 +01:00
Achilleas Koutsou
3289f9b07b distro/rhel8: explicitly enable google agent services for 8.4
The google-guest-agent services don't get enabled on 8.4.  The reason
for this is currently unknown.  Enabling them explicitly works.
2022-07-08 09:30:34 +01:00
Achilleas Koutsou
8a25113104 distro/rhel8: explicitly enable greenboot services for 8.4 edge
The greenboot packages on 8.4 are not enabled by presets, so we need to
explicitly enable them when building.
2022-07-08 09:30:34 +01:00
Achilleas Koutsou
0bf09ba60c distro/rhel8: separate minor-version-specific image types
edge-raw and edge-simplified-installer: only on 8.6+
ec2 and ec2-ha: available on all RHEL 8
ec2-sap: available on 8.4 and 8.6+ (no 8.5)

The ec2-sap image requires ansible, which in 8.4 is called `ansible` and
was replaced by `ansible-core` in 8.6.
2022-07-08 09:30:34 +01:00
Achilleas Koutsou
0adb742ede test-case-generators/repos: new repos for RHEL 8.4
Added unversioned (el8, no minor version) repositories for RHEL 8.4
that provide packages for building ec2 and azure-rhui image types.

Added new repo snapshots to RHEL 8.4: ha, sap, and saphana
2022-07-08 09:30:34 +01:00
Achilleas Koutsou
6a4e46e51a test-case-generators/repos: rhel-edge-commit -> edge-commit
With the merging of 8.4 into the main rhel8 package, the name
'rhel-edge-commit' is no longer the primary name for the image type.
More generally, the 'rhel-' prefix doesn't appear in the main name for
any image type anymore.
2022-07-08 09:30:34 +01:00
Achilleas Koutsou
ca1b559e30 Move RHEL 8.4 definition to the common rhel8 package 2022-07-08 09:30:34 +01:00
Achilleas Koutsou
a8b0a5f0bd test: convert all RHEL 8.4 v1 manifests to v2
Converted by loading them through the manifest parser in osbuild and
formatting them through v2 before dumping.
These are not the "real" manifests that osbuild-composer would generate,
but they will make it a tiny bit easier to compare and detect changes in
the distro definition when it's moved to the common rhel8 package.
2022-07-08 09:30:34 +01:00
schutzbot
0060048b81 schutzfile: Update snapshots to 20220701 2022-07-07 16:06:37 +02:00
Ondřej Budai
c0df932fc6 Dockerfile: bump the shutdown period to 15 seconds
5 seconds didn't fully help with the issue, let's try 15 and see what happens.

See https://github.com/osbuild/osbuild-composer/pull/2797#issuecomment-1172862359
for more details.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-07-07 15:33:32 +02:00
fkolwa
14fd1fe279 Update RHEL90-91 repo config and and 9.1 test cases 2022-07-07 14:28:59 +02:00
fkolwa
832c555a21 internal/distro: Add GCE RHUI image type+package set 2022-07-07 14:28:59 +02:00
dependabot[bot]
0d1dbd9780 build(deps): bump gopkg.in/ini.v1 from 1.66.4 to 1.66.6
Bumps [gopkg.in/ini.v1](https://github.com/go-ini/ini) from 1.66.4 to 1.66.6.
- [Release notes](https://github.com/go-ini/ini/releases)
- [Commits](https://github.com/go-ini/ini/compare/v1.66.4...v1.66.6)

---
updated-dependencies:
- dependency-name: gopkg.in/ini.v1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-07-07 13:19:10 +02:00
Tom Gundersen
ab4a3c0885 manifest/commit_server: pull in nginx
nginx is a requirement for this pipeline to work.
2022-07-07 12:00:56 +01:00
Tom Gundersen
452cb2dae9 environment: encapsulate the environment images are deployed to
This represents how our systems should integrate into their environment, typically using
some sort of agent, or commonly cloud-init.

In the future we could imagine this representing network configuration or any other kind
of configuration necessary to reach the environment as well.

For now EC2 and Azure is supported, and stub environments are
added to show the idea, but these are not implemented/used
yet.
2022-07-07 12:00:56 +01:00
Tom Gundersen
7a534d4d1e workload: introduce abstraction to encapsulate image workloads
The workload encapsulates what the user wants to run on top of the image. Everything
else we do abstracts away the OS, the hardware, the environment, and what is left is what
matters: the workload.

For now only the `Custom` payload is implemented which requires the user to name the
packages they want installed, the repositories to pull them from and what systemd
services to enable.

A few other stub workloads are added to show the idea, but these are not used.

The ideal is for the workload to have only the minimal number of configuration options.
2022-07-07 12:00:56 +01:00
Tom Gundersen
39c3d6ec35 manifest/os: move fs tool selection into the pipeline
Always include the tools for all the filesystem types in the partition table. There may be
usecases for having additional ones, for instance if the partition table is not known, but
this gives us a minimal baseline.

This includes dosfstools in images that have a vfat partition but did not include the tools.
2022-07-07 12:00:56 +01:00
Tom Gundersen
d791138b27 manifest/os: implicitly include selinux policy package 2022-07-07 12:00:56 +01:00
Tom Gundersen
1e03627447 manifest/os: only implicitly add to base packages
Any package that is added by the pipeline definition should be in the base package set
and the user package set should only be for packages explicitly added by the user.

Any combination of implicitly added packages should depsolve, or it is a bug. However,
user provided packages can have conflicts which must be handled gracefully.

This change is not breaking, as that would be a bug (per the above) and it makes our
behaviour more predictable as any conflicts are caused by explicitly added packages.

Note that this changes the logic from the kernel package being depsolved twice to only
being depsolved in the base package set.
2022-07-07 12:00:56 +01:00
Tom Gundersen
c296b666cf distro/fedora: drop usage of MakePackageChainSet
Pass PackageSets when initialising the Manifest, and read the chains back out.

This also fixes a bug where all repos were always used, rather than filtering per
package set.

Finally, this moves the 'chrony' inclusion from distro.go to the OSPipeline where
it belongs. In doing so the logic is changed slightly, where chrony is now
installed if NTP servers are configured (regardless of source), whereas in the
past it was included if the timezone was set in the blueprint (which made no sense).
2022-07-07 12:00:56 +01:00
Tom Gundersen
e4b5048d98 manifest/os: include kernel in package set
If the kernel name is set, then the packaegSpecs must include the kernel package, ensure
this by including the kernel in the package list.

We currently include the kernel both in the userPackages and the base packages.
Including it only in the user packages does not work as the base could end up pulling it
in as well. However, it would be semantically more correct to include it only in the base
set, so if possible we should do that and drop it from the user packages (in a follow-up).
2022-07-07 12:00:56 +01:00
Tom Gundersen
1bccf72b6b manifest/os: detect if the lvm2 package should be included
If the partition table includes logical volumes, the lvm2 package should be installed on
the target system.

Drop the corresponding logic from fedora/distro.go.
2022-07-07 12:00:56 +01:00
Tom Gundersen
ec8cc01f95 platform: introduce hardware platform abstraction
These objects describes the hardware an image runs on. Including
 - architecture
 - bootloader
 - required firmware

Use the platform abstraction to move firmware packages out of the package set
definitions.
2022-07-07 12:00:56 +01:00
Gianluca Zuccarelli
682481d4d7 distro: move isMountpointAllowed
Move the `isMountpointAllowed` function to the common
distro package since this function is shared used in
multiple distro packages.
2022-07-05 23:13:43 +01:00
Tom Gundersen
4e2cea61bb manifest/os: move packages needed for early boot into the pipeline definition
This simplifies the package set definitions in the fedora distro, and unifies the handling
of IoT and non-IoT images.
2022-07-05 22:39:40 +01:00
Tom Gundersen
03fc2d5bfd distro/fedora: install kernel and bootloader in ostree commits
This is a partial revert of 006c5b26, where kernel settings and bootloaders were only
installed on bootable systems.

However, ostree-based systems need the ability to pick up kernels and bootloaders from
the commit to install on the instance, so make this conditional on being bootable or an
ostree commit. This is probably an indication that we need a different abstraction.
2022-07-05 22:39:40 +01:00
Tom Gundersen
a45caf5756 distro/fedora: image type should be more specific than arch
Currently an image type could override the boot loader in the architecture, but we should
not allow an image type to select a bootloader type the architecture does not support. So
only in case the architecture supports hybrid boot do we allow the image type to select
one of the two types.

This enables UEFI on AMIs for aarch64.
2022-07-05 22:39:40 +01:00
Juan Abia
c2c43594b1 test/cases: add new mountpoints in filesystem.sh
There are a list of mountpoints in the documentation, but only
some of this mountpoints are tested in the test filesystem.sh. Add the
ones missing.
2022-07-05 14:19:01 +02:00
Tom Gundersen
907b4a8686 distro/fedora: pipelines.go -> manifests.go
Simplify the manifest generation by doing it in terms of manifests rather than slices
of pipelines.
2022-07-05 12:15:50 +02:00
Tom Gundersen
ba31887ff2 fedora/pipelines/os: minor refactoring
Collect partition tables / boot loaders / kernels together and
make them conditional on the system being bootable.

As a side-effect, we no longer install the grub2 modules in ostree commits.
2022-07-05 12:15:50 +02:00