The VMDK image must be in stream-optimized format in order to be
imported to VSphere. osbuild-composer does not produce VMDK by default
as stream-optimized. Instead, it is converted on the fly when the image
build job has been submitted via Weldr API.
Since we are aiming mainly for the VSphere use case with the VMDK image
in the service, the image should be ready for importing to VSphere.
Implement a temporary workaround for the Cloud API and AWS S3 target to
upload stream-optimized VMDK image.
Adjust the `api.sh` test case to not convert the VMDK image downloaded
form S3, before importing it to VSphere.
Install `cloud-init` by default on the VMDK image on RHEL-85/86/90 and
as a result also CentOS Stream 8/9.
Regenerate image test cases.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
test cases fixup
We should honour `pkg.CheckGPG` when creating the file inputs for the
rpm stage. This was lost in the transition from v1 to v2 manifests.
Regenerate image test manifests.
Co-authored-by: Tomas Hozza <thozza@redhat.com>
Signed-off-by: Tomas Hozza <thozza@redhat.com>
If a kernel is specified we already set the `SavedEntry`. To preserve
that the saved entry setting will survive grub2 config regeneration
we also need to set `Config.Default` to "saved". Do so. All older
pipelines have been prepared to reset this value to preserve the old
behavior. All new pipelines have been prepared to not overwrite this
setting when `ImageConfig.Grub2Config` is applied.
The `grub2.Config.Default` key is owned by `NewGrub2StageOptionsUnified`
and thus needs to be preserved in case there is a `Grub2Config` setting
present in the image configuration (`ImageConfig`).
The `grub2.Config.Default` key is owned by `NewGrub2StageOptions`
and thus needs to be preserved in case there is a `Grub2Config`
setting present in the image configuration (`ImageConfig`).
A future change to `NewGrub2StageOptions` will result in a change of
`Config.Default` to `saved` if a kernel is set. Preserve the current
behavior by manually resetting it to the empty string.
A future change to `NewGrub2StageOptions` will result in a change of
`Config.Default` to `saved` if a kernel is set. Preserve the current
behavior by manually resetting it to the empty string.
If a home directory has a trailing slash, the `useradd` command fails to
set the correct selinux contexts for the home directory on creation.
This can lead to various issues, but the one that we came across was
that the ~/.ssh directory and authorized_keys file cannot be read by
sshd and we couldn't log in to the system.
This only manifests if the user is created through the kickstart file
because:
1. `useradd` does not set the selinux contexts when creating the
directory
2. Anaconda runs `restorecon` on the home directory and authorized_keys
file when it creates them, but uses the install-time mount path
`/mnt/sysroot/...` for which selinux does not have contexts.
In most cases we get around this bug because we run `setfiles` on the
tree at the end of our pipelines.
For the ostree case, the relabeling in Anaconda is done correctly.
Enable the user module unconditionally for the image-installer:
- If users are specified for the kickstart file, the module is required
to set up the users.
- If no users are specified, the module can be used at install time to
create users.
Updated relevant test cases (manifests).
Use single NewAnacondaStageOptions() from osbuild2 instead of
implementing in each distro.
The new function conditionally adds the user module when there are users
that need to be created at install time (image- and edge-installers).
Use single NewGroupsStageOptions() from osbuild2 instead of implementing
in each distro.
The new function does not set the Group.Name field anymore. The field
does not exist in the osbuild schema and was silently ignored.
The field in the stage has been marked 'omitempty' and the relevant
manifests have been updated.
If a user requires that packages from a certain repository are checked using
a GPG key, they should specify it. Now, this is enforced to catch this issue
earlier than in osbuild.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Oh no, we made a mistake here: Both our json repositories and repo files in
/etc/yum.repos.d have the GPG key in a field named `gpgkey`. Unfortunately,
cloudapi uses a field named `gpg_key`. One consequence of this issue is that
our api.sh test is meant to pass GPG keys in the compose request but since
it's using a bad field name (`gpgkey`), the key is actually not used.
I've decided to fix this in cloudapi: The `gpg_key` field is now renamed to
`gpgkey`. This is a breaking change but no one is using this API anyway so
we think it's better to do this now than introducing weird backward
compatible hacks.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
We have to do a small hack to enable edge-commit on Fedora because its name
is different. We can also change this in the image definition but I want to
iterate quickly on the Fedora Integration MVP and don't want to run in
any conflicts with
https://github.com/osbuild/osbuild-composer/pull/2461
This commit also enables a test for Fedora IoT built through the API.
While enabling the test, I also simplified our decision logic for SSH_USER
and DISTRO.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Use the Extended Bootloader Partition GUID for `/boot`, instead of the
Linux filesystem data GUID. This is useful for autodetection of a
partition purpose based on its GUID without reading the `/etc/fstab`
first.
Ensure that when creating mountpoints, e.g. when converting the
partitions layout to LVM, the `/boot` partition get the proper GUID
assigned.
Regenerate RHEL-90 and centos-9 image test cases.
Related to https://bugzilla.redhat.com/show_bug.cgi?id=2057231
Transactions are tied to a connection so this is actually not a functional
change. Nevertheless, I think it's nice to explicitly state that we are
using a transaction.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Otherwise, there might be an already waiting dequeuer and if something is
enqueued before `sqlListen` is called, we will lost this notification.
Also, a small log message was added when shutting down the listener.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Call `Shutdown()` on all http servers. This means we will finish processing
any pending requests (including depsolving), but we will not listen to new
ones.
In particular, we will not answer to the readiness probe, so no new traffic
will be routed to this container.
Once all pending requests have been handled composer will shut down
gracefully and the liveness probe will return failure.
Note that in order for this to work correctly no requests should ever take longer
than the shutdown timeout (by default 30s).
This will allow us to use the service accounts which work against
identity.api.openshift.com. These are much easier to manage, especially
with the new multi-tenancy, as there's a single page to create/expire
them across an account.
They also have the added benefit of not expiring automatically when
they're not used like offline tokens, and immediate expiration when
desired.
Using the simplified installer we were experiencing slow system boots.
Turns out we're incurring into https://bugzilla.redhat.com/show_bug.cgi?id=1839923
This patch just drops the console kargs - to be aligned with the
anaconda installer that doesn't experience this slow down.
The slow down doesn't happen on virtual machines as there's always a
ttyS0 there
Signed-off-by: Antonio Murdaca <runcom@linux.com>
The repo name is already part of the `rpmmd.RepoConfig` structure. Do
not ignore when calling `dnf-json` and and pass it the value.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Previously, all dequeuers (goroutines waiting for a job to be dequeued) were
listening for new messages on postgres channel jobs (LISTEN jobs). This didn't
scale well as each dequeuer required to have its own DB connection and the
number of DB connections is hard-limited in the pool's config.
I changed the logic to work somewhat differently: dbjobqueue.New() now spawns
a goroutine that listens on the postgres channel. If there's a new message,
the goroutine just wakes up all dequeuers using a standard go channel.
Go channels are cheap so this should scale much better.
A test was added that confirms that 100 dequeuers are not a big deal now. This
test failed when I tried to run on it on the previous commit. I tried even 1000
locally and it was still fine.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Removing queued_at and started_at is pretty straightforward, it wasn't needed.
Removing token might seem concerning but basically we were just pulling
the same value from DB as we were pushing there. I think there's no value in
doing that.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The `API.arch` member was (mostly) used to read the name of the
architecture.
The only non-name use was for the purposes of reading RPM repositories
from the configuration, in `reporegistry.ReposByArch()`, a thin wrapper
around `reporegistry.ReposByArchName()`.
Removing the `arch` member from the API and using the new `archName`
that is set up in the API constructor lets us control the arch name that
is set without relying on a valid `distro.Arch` object being available
(which would depend on having a valid `distro.Distro` object).
Replaced all calls to `ReposByArch()` with `ReposByArchName()` which
depends on the arch and distro name strings instead of a full
`distro.Arch`.
When the host distribution is not known or supported, instead of failing
with an error, print a warning to the log and initialise the API with
the architecture name and distro name.
This enables running the weldr API on unsupported distros for
cross-distro building.
Guards against a nil arch member when initialising the store.
Add an error object to the ComposeStatus.ImageStatus.
The error object contains a human-readable error reason
and optional details in the case of an error.
This commit adds a very in-depth test for multi-tenancy. It queues several
composes and then runs all jobs belonging to them while checking that
they are run by the correct tenant.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Yeah, we have TestRoute. It has one issue though: It doesn't have support
for passing a custom context. One option is to extend the method with yet
argument but since it already has 9 (!!!), this seems like a huge mess.
Therefore, I decided to invent a new small library for writing API tests.
It uses structs heavily which means that adding features to it doesn't
mean changing 100 lines of code (like adding another arg to TestRoute does).
I hope that we can start using this library more in our tests as it was
designed to be very flexible and powerfule.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
