Commit graph

2181 commits

Author SHA1 Message Date
Christian Kellner
865a899f70 container/client: rework tls settings
Instead of having an extra field, `TlsVerify`, on the `Client` and
then later setting the corresponding `SystemContext` options, use
the existing `SystemContext` field of `Client`. The corresponding
field is a tri-state: unset, true, false, which is represented as
a pointer to boolean in the `Client`'s new getter and setter. This
also inverts the boolean logic from verify TLS to skip TLS which
aligns very well with the corresponding fields in the upload target
struct.
In addition we properly capitalize some existing variables.
2022-07-21 13:32:07 +02:00
Ondřej Budai
e779562f3c worker: remove osbuild-koji job
Koji API removed by the previous commit was the last user of osbuild-koji job.
Let's remove it since nothing uses it. This also removes all of the
compatibility code in Cloud API, see concerns below:

Compatibility concerns:
- the internal deployment was moved to a completely different composer
  instance, thus there are no old jobs
- Fedora deployment is still unused in prod, thus we don't care about keeping
  backward compatibility of the old jobs

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-07-19 16:00:52 +02:00
Ondřej Budai
74eb3860df internal: remove kojiapi
We no longer use it, let's remove it. If you are wondering what to use instead,
use Cloud API. It supports everything that Koji API supported and more.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-07-19 16:00:52 +02:00
Ondřej Budai
058edd3d76 cloudapi: remove a confusing test
This test tested two things:

1) Invalid route - this is already covered by TestUnknownRoute
2) Invalid UUID in the compose status route - this is now covered by
   TestComposeStatusInvalidUUID

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-07-19 16:00:52 +02:00
Ondřej Budai
3e25f5ef76 remove all traces of fedora 34
Fedora 34 is EOL, let's remove all traces of it, including:

- distro definition
- repositories (and test one)
- test manifests
- special package set rules
- hacks from the spec file

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-07-18 20:38:33 +02:00
Brian C. Lane
d3af314e58 users_stage: Don't allow empty passwords
Make sure empty passwords are set to nil so they result in a locked
account.

Also add a test for the password hashing in NewUserStageOptions()
2022-07-16 22:54:26 +01:00
Brian C. Lane
6adf3f5b7b blueprint: Don't allow empty password
If the password is set to "" it will get hashed, allowing access to the
account in some circumstances. Console and ssh login don't appear to
work in practice, but su to the account from another user account is
possible.

This sets the empty password to nil which makes sure that it ends up as
a locked account.
2022-07-16 22:54:26 +01:00
Brian C. Lane
20bf0c4836 blueprint: Hash all user passwords
This commit changes blueprint behavior to always store the hash of the
password for the 'customizations.user' accounts. Note that missing or
blank passwords are not hashed and should be dealt with at a lower
layer.

Resolves: rhbz#2107358
2022-07-16 22:54:26 +01:00
Achilleas Koutsou
9d4a351ca6 Rename osbuild2 package to osbuild 2022-07-14 16:54:00 +02:00
Achilleas Koutsou
01d87b4e60 osbuild1: DELETED 2022-07-14 16:54:00 +02:00
Achilleas Koutsou
dcef56c75a osbuild2: move v1 stage metadata parsing to convertStageResults
Make the v1StageResult.Metadata a simple json.RawMessage and perform the
RawMessage to StageMetadata conversion in the convertStage functions.
This lets us get rid of the custom v1StageResult Unmarshaller and the
v1RawStageResult, and makes the whole conversion process easier to
trace.
2022-07-14 16:54:00 +02:00
Achilleas Koutsou
d68a95c533 osbuild2: copy Result types from osbuild1
Types, parsing functions and helpers copied from osbuild1 to
osbuild2/v1result.go.
The metadata handling is simplified: osbuild1 stage metadata for the RPM
and ostree-commit stages is identical to the osbuild2 counterparts.
2022-07-14 16:54:00 +02:00
Achilleas Koutsou
c1956ba6e1 Use osbuild2 Manifest in all tests
The test_distro Manifest, which is used in tests across multiple
packages, was using the old structure.  Updated to the v2 structure and
adapted all tests.
2022-07-14 16:54:00 +02:00
Achilleas Koutsou
8eb12018c0 osbuild2: move v1 result handling code to separate file 2022-07-14 16:54:00 +02:00
Achilleas Koutsou
1c218bc633 osbuild2: fix package docstring 2022-07-14 16:54:00 +02:00
Simon de Vlieger
78ae275c61 jobqueue: store an expiry date
This introduces an expiry date (default: 14 days from insert date) and
adjust the service-maintenance script to delete jobs that are older than
the expiration date.
2022-07-13 17:26:04 +02:00
Tom Gundersen
e844453c85 manifest: make filename optional and generalise Tree
We have three kinds of operating system trees, until we unify them to one,
hide them behind one interface. Use this to read the architecture from the
Tree rather than pass it in as a string to parent pipelines.

Also, make the filename parameter optional in a few places, there should be no
reason to set this rather than introspect it (except for backwards
compatibility).

Lastly, add another playground example sample to build a raw image.
2022-07-12 08:19:57 +01:00
Tom Gundersen
d00b98c134 manifest: don't stutter
Replace `pipeline.OSPipeline` with `pipeline.OS`, etc.

Also rename `LiveImg` to `RawImage`.
2022-07-12 08:19:57 +01:00
Tom Gundersen
529bc803db runner: introduce runner abstraction
For now all it does is represent the name of the runner and what requirements
it has of the build pipeline.

Move some package definitions from the runner package set to where it belongs.
2022-07-12 08:19:57 +01:00
Tom Gundersen
1b924ae30c osbuild-worker/osbuild: move out execution helper
Move the execution helper for osbuild into the osbuild2 package so it can be
reused. In the process, generalise it slightly.
2022-07-12 08:19:57 +01:00
Tom Gundersen
fa7d2f2647 manifest/build: explicitly pull in glibc
This should have no practical effect, but ldconfig is used from
runners, so it is strictly speaking a requirement.

At the same time document the remaining TODO's in the build
pipeline.
2022-07-12 08:19:57 +01:00
Tom Gundersen
b405ce4d89 manifest/build: replace explicit dnf with implicit rpm
The pipelines don't use dnf, only rpm. Let the ones that need it pull it in.
2022-07-12 08:19:57 +01:00
Tom Gundersen
b714dcb780 manifest/build: pull in tar implicitly
The tar pipeline requires tar, let only that pull it in.
2022-07-12 08:19:57 +01:00
Tom Gundersen
fed620b861 disk/partition_table: expose GetBuildPackages() function
This describes the packages necessary to create the partition table.

Use this in the pipelines and drop the explicit mentions from the build
pipeline.
2022-07-12 08:19:57 +01:00
Tom Gundersen
0d3d35e154 manifest/build: pull in selinux packages implicitly
The build pipeline requires the selinux packages only if we are going to be
labelling the files in the target OS. Otherwise, skip it.

manifest/build: pull in selinux-targeted unconditionally

This is unconditionally used by the build pipeline itself, until we make that
conditional, it needs to be installed.
2022-07-12 08:19:57 +01:00
Tom Gundersen
8df99d2b31 manifest/build: we depend on /bin/cp, always pull in coreutils 2022-07-12 08:19:57 +01:00
Tom Gundersen
9160319152 manifest/build: move out qemu-img
Let each pipeline that requires it pull in qemu-img.
2022-07-12 08:19:57 +01:00
Achilleas Koutsou
fdc6790472 distro: remove rhel84 package 2022-07-08 09:30:34 +01:00
Achilleas Koutsou
3289f9b07b distro/rhel8: explicitly enable google agent services for 8.4
The google-guest-agent services don't get enabled on 8.4.  The reason
for this is currently unknown.  Enabling them explicitly works.
2022-07-08 09:30:34 +01:00
Achilleas Koutsou
8a25113104 distro/rhel8: explicitly enable greenboot services for 8.4 edge
The greenboot packages on 8.4 are not enabled by presets, so we need to
explicitly enable them when building.
2022-07-08 09:30:34 +01:00
Achilleas Koutsou
0bf09ba60c distro/rhel8: separate minor-version-specific image types
edge-raw and edge-simplified-installer: only on 8.6+
ec2 and ec2-ha: available on all RHEL 8
ec2-sap: available on 8.4 and 8.6+ (no 8.5)

The ec2-sap image requires ansible, which in 8.4 is called `ansible` and
was replaced by `ansible-core` in 8.6.
2022-07-08 09:30:34 +01:00
Achilleas Koutsou
ca1b559e30 Move RHEL 8.4 definition to the common rhel8 package 2022-07-08 09:30:34 +01:00
fkolwa
832c555a21 internal/distro: Add GCE RHUI image type+package set 2022-07-07 14:28:59 +02:00
Tom Gundersen
ab4a3c0885 manifest/commit_server: pull in nginx
nginx is a requirement for this pipeline to work.
2022-07-07 12:00:56 +01:00
Tom Gundersen
452cb2dae9 environment: encapsulate the environment images are deployed to
This represents how our systems should integrate into their environment, typically using
some sort of agent, or commonly cloud-init.

In the future we could imagine this representing network configuration or any other kind
of configuration necessary to reach the environment as well.

For now EC2 and Azure is supported, and stub environments are
added to show the idea, but these are not implemented/used
yet.
2022-07-07 12:00:56 +01:00
Tom Gundersen
7a534d4d1e workload: introduce abstraction to encapsulate image workloads
The workload encapsulates what the user wants to run on top of the image. Everything
else we do abstracts away the OS, the hardware, the environment, and what is left is what
matters: the workload.

For now only the `Custom` payload is implemented which requires the user to name the
packages they want installed, the repositories to pull them from and what systemd
services to enable.

A few other stub workloads are added to show the idea, but these are not used.

The ideal is for the workload to have only the minimal number of configuration options.
2022-07-07 12:00:56 +01:00
Tom Gundersen
39c3d6ec35 manifest/os: move fs tool selection into the pipeline
Always include the tools for all the filesystem types in the partition table. There may be
usecases for having additional ones, for instance if the partition table is not known, but
this gives us a minimal baseline.

This includes dosfstools in images that have a vfat partition but did not include the tools.
2022-07-07 12:00:56 +01:00
Tom Gundersen
d791138b27 manifest/os: implicitly include selinux policy package 2022-07-07 12:00:56 +01:00
Tom Gundersen
1e03627447 manifest/os: only implicitly add to base packages
Any package that is added by the pipeline definition should be in the base package set
and the user package set should only be for packages explicitly added by the user.

Any combination of implicitly added packages should depsolve, or it is a bug. However,
user provided packages can have conflicts which must be handled gracefully.

This change is not breaking, as that would be a bug (per the above) and it makes our
behaviour more predictable as any conflicts are caused by explicitly added packages.

Note that this changes the logic from the kernel package being depsolved twice to only
being depsolved in the base package set.
2022-07-07 12:00:56 +01:00
Tom Gundersen
c296b666cf distro/fedora: drop usage of MakePackageChainSet
Pass PackageSets when initialising the Manifest, and read the chains back out.

This also fixes a bug where all repos were always used, rather than filtering per
package set.

Finally, this moves the 'chrony' inclusion from distro.go to the OSPipeline where
it belongs. In doing so the logic is changed slightly, where chrony is now
installed if NTP servers are configured (regardless of source), whereas in the
past it was included if the timezone was set in the blueprint (which made no sense).
2022-07-07 12:00:56 +01:00
Tom Gundersen
e4b5048d98 manifest/os: include kernel in package set
If the kernel name is set, then the packaegSpecs must include the kernel package, ensure
this by including the kernel in the package list.

We currently include the kernel both in the userPackages and the base packages.
Including it only in the user packages does not work as the base could end up pulling it
in as well. However, it would be semantically more correct to include it only in the base
set, so if possible we should do that and drop it from the user packages (in a follow-up).
2022-07-07 12:00:56 +01:00
Tom Gundersen
1bccf72b6b manifest/os: detect if the lvm2 package should be included
If the partition table includes logical volumes, the lvm2 package should be installed on
the target system.

Drop the corresponding logic from fedora/distro.go.
2022-07-07 12:00:56 +01:00
Tom Gundersen
ec8cc01f95 platform: introduce hardware platform abstraction
These objects describes the hardware an image runs on. Including
 - architecture
 - bootloader
 - required firmware

Use the platform abstraction to move firmware packages out of the package set
definitions.
2022-07-07 12:00:56 +01:00
Gianluca Zuccarelli
682481d4d7 distro: move isMountpointAllowed
Move the `isMountpointAllowed` function to the common
distro package since this function is shared used in
multiple distro packages.
2022-07-05 23:13:43 +01:00
Tom Gundersen
4e2cea61bb manifest/os: move packages needed for early boot into the pipeline definition
This simplifies the package set definitions in the fedora distro, and unifies the handling
of IoT and non-IoT images.
2022-07-05 22:39:40 +01:00
Tom Gundersen
03fc2d5bfd distro/fedora: install kernel and bootloader in ostree commits
This is a partial revert of 006c5b26, where kernel settings and bootloaders were only
installed on bootable systems.

However, ostree-based systems need the ability to pick up kernels and bootloaders from
the commit to install on the instance, so make this conditional on being bootable or an
ostree commit. This is probably an indication that we need a different abstraction.
2022-07-05 22:39:40 +01:00
Tom Gundersen
a45caf5756 distro/fedora: image type should be more specific than arch
Currently an image type could override the boot loader in the architecture, but we should
not allow an image type to select a bootloader type the architecture does not support. So
only in case the architecture supports hybrid boot do we allow the image type to select
one of the two types.

This enables UEFI on AMIs for aarch64.
2022-07-05 22:39:40 +01:00
Tom Gundersen
907b4a8686 distro/fedora: pipelines.go -> manifests.go
Simplify the manifest generation by doing it in terms of manifests rather than slices
of pipelines.
2022-07-05 12:15:50 +02:00
Tom Gundersen
ba31887ff2 fedora/pipelines/os: minor refactoring
Collect partition tables / boot loaders / kernels together and
make them conditional on the system being bootable.

As a side-effect, we no longer install the grub2 modules in ostree commits.
2022-07-05 12:15:50 +02:00
Tom Gundersen
de6c628069 manifest/os: drop kernelName and bootLoader from New()
The kernel name is optional and can be set later.

The bootloader we skip entirely. Instead, set the architecture, which now becomes
mandatory. Use it to deduce the bootloader, and in the future other pipelines can read
this property from the OS Pipeline, rather than having it passed in.
2022-07-05 12:15:50 +02:00