Commit graph

2320 commits

Author SHA1 Message Date
Achilleas Koutsou
0c77e4be60 manifest: rpm stage options for ostree commits
When building a fedora ostree commit, set the new ostree_booted flag in
the rpm stage and the dbpath to /usr/share/rpm
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
0b3bfd0d02 osbuild: update rpm stage with new options
- db path: set alternative rpm database path [1]
- ostree-booted: create the /run/ostree-booted marker which marks an
  install as ostree.  Install-time scripts in packages use this marker
  to treat ostree installations differently.

[1] https://github.com/osbuild/osbuild/pull/666
[2] https://github.com/osbuild/osbuild/pull/1085
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
ccbf15878b manifest: add grub config for OSTreeDeployment
These are based on the official Fedora IoT Raw Image but are added to
the base OSTreeDeployment manifest.  For now, let's add them to all
ostree deployments that will use these new types and we can extract the
values to make them configurable when we need to.
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
a505defad3 manifest: lock the root password for OSTreeDeployment 2022-09-13 16:06:19 +01:00
Achilleas Koutsou
fbe8c1cb39 manifest: use new ostree mount in fstab stage for commit deployment
Use the new stage mount instead of the old stage-specific options.
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
1de55c73e6 ostree: define type for handling ostree remote config
A new struct in ostree can be used to define configuration options for
the ostree remote of an image.  So far remotes were always set up with
the remote URL used to pull the commit.  Now we support setting a
different remote with extra configuration options.

This is used by the fedora-iot-raw-image to set up the remote
configuration of the final image, separately from the source of the
commit.

Test manifests updated.
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
0386d68db4 manifest: add ostree mount to stages in OSTreeDeployment
Stages that modify files in the ostree deployment need the sysroot
bindmount for any changes to take effect.
Adding the mount to the keymap and locale stages since they need to
modify files in /etc.
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
0750049739 osbuild: update org.osbuild.ostree.remotes stage
New options added to stage in osbuild [0].

[0] https://github.com/osbuild/osbuild/pull/1097
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
0ff1c3241c osbuild: add mount ostree helper method to Stage
A convenience method that adds an ostree deployment mount to any Stage.
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
4a68df45c7 osbuild: add ostree.deployment mount type
New mount type added: org.osbuild.ostree.deployment.

Adding this to a stage will setup all needed bind mounts so that a given
`tree` will look like an active OSTree deployment, very much as OSTree
does during early boot.

This is often necessary when making changes to files in /etc for an
ostree image.
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
7246f8fdd5 osbuild: fix base Mount struct properties
Source and Target are not required -> omitempty
Options is of type MountOptions which is an interface, so we don't need
to make it a pointer.
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
76031d4104 manifest: add OSTreeDeployment config options
Adding support for config options to OSTreeDeployment that are required
by the IoT raw image:
- Kernel command line options
- Keyboard layout
- Locale

Test manifests updated.
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
ce7b251b54 platform: fix error message for invalid image format 2022-09-13 16:06:19 +01:00
Achilleas Koutsou
21d2c78f20 manifest: remove s390x case from OSTreeDeployment
Not supported.  No need to switch.
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
901b3a5da9 fedora: add platform packages to RawOSTreeImage build root
Include the platform packages when getting the build packages for the
RawOSTreeImage.
rpm-ostree is explicitly added for this image type.
dracut-config-generic and efibootmgr are temporarily added here, but we
should define a platform that includes them instead (some cleanup
required in general).
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
b85fe8109c manifest: add the platform configuration to the RawOSTreeImage 2022-09-13 16:06:19 +01:00
Achilleas Koutsou
c38c7ad126 disk: add packages for LUKS containers
Add the packages required for LUKS containers to the build root when
getting the list of packages from the PartitionTable.
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
f32e653315 store: trim compat string map
The map is meant to maintain compatibility with changes in image type
names.  Most entries however map to themselves.  It's redundant to keep
updating this map with every new image type name and map it to itself.
Instead, check the map for a compat string and if one doesn't exist,
return the original.

Conversely, when the inverse lookup is performed, first check if the
compat string is a valid image type name and only iterate the map if
it's not.

Now we don't need to update the map every time we introduce a new image
type, unless we need to map an old name to a new one.
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
c6843db6b4 fedora: add the iot-raw-image to x86 and aarch64
Add the new image type to the list in each architecture and update
tests.

Ignore ostree raw images in Kernel count test in distro_test_common:

Edge and IoT raw images don't need a kernel specified in their OS
pipeline.  The kernel (and the OS in general, including all packages)
come from the commit that is pulled and deployed in the image.

This test passes on RHEL (for edge-raw-image types) because the
blueprint defaults to returning the main kernel, but this isn't
necessary and is likely to change in the near future.

Co-Authored-By: Ondřej Budai <ondrej@budai.cz>
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
74a196e427 image: complete OSTreeRawImage.InstantiateManifest()
Create all the necessary pipelines for the OSTreeRawImage and return the
artifact from the XZ pipeline.

Co-Authored-By: Ondřej Budai <ondrej@budai.cz>
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
f39b3bf6d3 manifest: add xz pipeline for compressing ostree images
The XZ pipeline is added explicitly for ostree raw images for now.
It should be changed to support compressing general raw images as well.

Co-Authored-By: Ondřej Budai <ondrej@budai.cz>
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
2683474519 manifest: add RawOSTreeImage
The RawOSTreeImage pipeline is the OSTree-equivalent to the RawImage.
The serialization is very similar between the two (almost identical),
but the two Pipelines depend on different Tree implementations.

Co-Authored-By: Ondřej Budai <ondrej@budai.cz>
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
a8a2191670 manifest: implement OSTreeDeployment pipeline
New pipeline for creating an ostree commit deployment.
Heavily based on the ostreeDeployPipeline from RHEL 9.x.

Co-Authored-By: Ondřej Budai <ondrej@budai.cz>
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
526ea0541c distro/fedora: return OSTreeRawImage from iotRawImage
Some basic information and ostree options copied to image.

Co-Authored-By: Ondřej Budai <ondrej@budai.cz>
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
5d61ba5d39 image: add OSTreeRawImage ImageKind
InstantiateManifest is a stub.  No implementation yet.

Co-Authored-By: Ondřej Budai <ondrej@budai.cz>
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
280d27a1c8 distro/fedora: add iot partition tables
Based on the official images.
Converted to GPT (from DOS).
2022-09-13 16:06:19 +01:00
Achilleas Koutsou
660444fed1 distro/fedora: introduce fedora-iot-raw-image
Image definition is a stub.  No implementation yet and not added to
distro image type list.

Co-Authored-By: Ondřej Budai <ondrej@budai.cz>
2022-09-13 16:06:19 +01:00
Ondřej Budai
a8fe0d2620 distro/fedora: add support for Fedora 37 and 38
The most interesting change is the removal of smc-meera-fonts in 37. As
suggested, rit-meera-new-fonts is used instead.

Existing F35 and F36 manifests updated with package changes.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-09-13 16:06:19 +01:00
Ondřej Budai
ff8af88a24 distro/fedora: deduplicate Fedora distribution structs
These are just super-simple to construct using a small helper.

It would be great if we can make `distro.go` totally version-agnostic but
that's something for the future.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-09-13 16:06:19 +01:00
Lukas Zapletal
a8afca4634 Introduce logging adapter for jobqueue 2022-09-09 16:27:38 +02:00
Tomas Hozza
351bb69d2b distro: use reflection in ImageConfig.InheritFrom()
As it turned out, people make mistakes and forget to write some parts of
code, unless a unit test screams at them. This is true for the
`InheritFrom()` method, which is not handling all members of the
`ImageConfig` structure.

Use reflection, instead of inheriting from each specific hard-coded
structure member. This will make the implementation future-proof in case
the `ImageConfig` structure is extended with additional members.
2022-09-06 14:43:24 +02:00
Tomas Hozza
c8382f1654 distro: use pointers for basic types in ImageConfig
Using basic types as values in the `ImageConfig` structure makes it
impossible to distinguish if the empty value for the type was set
intentionally or if it is just the value the variable was initialized
to. This is very bad especially for `bool` type.

While working on unifying `vhd` and `azure-rhui` image types I found
out, that some newly added variables in the `ImageConfig` structure
were forgotten in the `InheritFrom()` method. This makes it impossible
to inherit their values from a parent configuration. This is however
required for the unification of `vhd` and `azure-rhui` image types. As
described above, it would be impossible to decide whether a `bool` value
should be inherited from the parent configuration or not. The only
solution is to use a pointer to the type. For consistency, use pointer
for all basic types.

Adjust distro implementations accordingly.
2022-09-06 14:43:24 +02:00
Achilleas Koutsou
aed7808b03 rhel9: explicitly add containernetworking-plugins to edge
In podman v4.0.0 the default network backend was switched from cni to
netavark.  However, podman will choose cni if there are already
containers, images, or cni networks preset on a system [1].

Starting with podman v4.2.0, containernetworking-plugins is no longer a
hard requirement for podman.  So when an edge commit is built with an
embedded container, podman v4.2.0+ will choose the cni network and fail
with an error because the plugin isn't installed.

Adding the package explicitly alongside podman to avoid this issue with
future RHEL 9.1 edge builds when they include containers.

This change does not affect test manifests.  The package is already
included in manifests as a dependency of podman < v4.2.0.

See rhbz#2123210

[1] a083f790ab/pkg/config/containers.conf (L275-L278)
2022-09-02 10:00:28 +02:00
Brian C. Lane
c32f94d6f2 weldr: Preload metadata at startup
For each of the supported distros start a goroutine to depsolve
'filesystem' which will preload the metadata making subsequent responses
faster.

This is safe to do without limits because we only supposed a limited
number of distros, and without additional locking because this is the
the same as hitting the API with multiple depsolve requests at the same
time.
2022-08-31 12:18:51 +01:00
fkolwa
1b3f509318 internal/distro: Add GCPGuestAgentConf to ImageConfig 2022-08-30 22:21:29 +02:00
fkolwa
93b57652ba internal/distro/rhel90: Add GCPGuestAgentConf
Add NewGcpGuestAgentConfStage to osPipeline
2022-08-30 22:21:29 +02:00
fkolwa
2d3997967f internal/distro/rhel8: Add GCPGuestAgentConf
Add NewGcpGuestAgentConfStage to osPipeline
2022-08-30 22:21:29 +02:00
fkolwa
1fbdb21f6b internal/osbuild: Add GCP guest-agent conf stage
Add GCP guest-agent conf stage test
2022-08-30 22:21:29 +02:00
Sanne Raymaekers
d13347e1ca cloudapi: Add endpoints to clone aws images across regions
Support for creating multiple amis from a single compose. It uses the
AWSEC2* jobs to push images to new regions, and share them with new
accounts.

The compose it depends upon has to have succeeded.
2022-08-30 16:14:52 +02:00
Sanne Raymaekers
599829a3b8 worker: Return dependent jobs in OsbuildJobStatus 2022-08-30 16:14:52 +02:00
Sanne Raymaekers
0fe3f1b2ae jobqueue: Query job dependents 2022-08-30 16:14:52 +02:00
Sanne Raymaekers
099b34b301 worker: Define new jobs to handle copying and resharing of images
The copy job copies from one region to another. It does not preserve the
sharing on the ami and it's snapshot, that needs to be queued
separately.
2022-08-30 16:14:52 +02:00
Sanne Raymaekers
5f29dc312a distro: Only add rhsm facts stage for RHEL 2022-08-30 15:32:50 +02:00
Sanne Raymaekers
d7b8bb122c cloudapi/v2: Add rhsm facts for non-koji cloudapi composes 2022-08-30 15:32:50 +02:00
Ondřej Budai
fa514c5326 blueprint: remove omitempty from Customizations
See https://github.com/BurntSushi/toml/issues/360

A recent change in BurntSushi/toml made encoding fail (later changed to
error) if a struct is marked as omitempty and is comparable. Go docs about
equality: https://go.dev/doc/go1#equality. Basically: A struct is comparable
if all of its fields are comparable. Slices are not comparable.

Customizations are marked as omitempty but they contain a lot of slices,
thus they are not comparable. The new version of BurntSushi/toml therefore
panics when we encode them.

The solution is to remove the omitempty tag from Customizations.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-08-29 10:25:38 +02:00
Sanne Raymaekers
774cc9e638 cloudapi/v2: Don't add rhsm facts
There are workers everywhere which can't deal with this change just
yet. Let's wait until workers can deal with this before adding facts via
the api.
2022-08-26 18:29:26 +01:00
Simon de Vlieger
c5f335bceb Add the rhsm.facts stage.
We initially provide a Facts file that stores the `ApiType`. This is the
API that was used to request the compose.
2022-08-25 18:56:35 +02:00
Achilleas Koutsou
18abe661a7 distro/fedora: enable hybrid boot for iot-installer 2022-08-25 14:54:56 +01:00
Achilleas Koutsou
b9c636763e distro/rhel8: enable hybrid boot for edge-installer
Regression introduced when unifying distro versions.
2022-08-25 14:54:56 +01:00
Achilleas Koutsou
37e70329a7 distro/rhel9: enable hybrid boot for edge-installer
Regression introduced when unifying distro versions.
2022-08-25 14:54:56 +01:00