Write an openapi spec for the worker API and use `deepmap/oapi-codegen`
to generate scaffolding for the server-side using the `labstack/echo`
server.
Incidentally, echo by default returns the errors in the same format that
worker API always has:
{ "message": "..." }
The API itself is unchanged to make this change easier to understand. It
will be changed to better suit our needs in future commits.
vCenter requires images to be uploaded as vmdk StreamOptimized. Lorax
always produced images on this format, so we should make sure to do the
same for our VMWare images.
Allow LocalTarget to request the images produced by osbuild be converted
to be streamOptimized before saving in composer, and hook the weldr API
up to enable this option for vmdk images.
Ideally this should simply be an option in osbuild, but that would
require some more work, which we will not manage in time for RHEL8.3.
Therefore do this minimal fix.
Note that that means the images produced by our manifests (including in
our image-test test cases) are not on the format that the weldr API
returns, so the tests we run on them would also, for now, need to
convert before uploading to vCenter.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The osbuild-composer-rcm package was never finished, not in use and will be replaced by osbulid-composer-koji.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The subpackages are not really meant to be used with a different version of
the osbuild-composer package. This commit enforces.the usage of a matching
version in the spec file.
20c40b59 added a dependency to osbuild-ostree. However, it wasn't added to the
old specfile. This commit fixes it.
The old specfile is used only on Fedora 31. which might never get the support
for building ostree images, but I prefer the two specfiles to be as close to
each other as possible.
RHEL 8 doesn't have golang in the i686 buildroot anymore. Fedora doesn't have
i686 images and kernel anymore. Also, osbuild-composer doesn't have support
for building i686.
Let's exclude i686 for these reasons, it's dead.
The new Fedora IoT image type uses org.osbuild.ostree.commit assembler with
tar option that was introduced in osbuild 15, therefore the dependency version
must be bumped.
This also bumps the submodule to 9cbedc04. That's basically version 15 of
osbuild with one fix related to secrets.
Currently, removing osbuild-composer produces these warnings:
Failed to stop osbuild-worker@.service: Unit name
osbuild-worker@.service is missing the instance name.
See system logs and 'systemctl status osbuild-worker@.service' for details.
Failed to stop osbuild-remote-worker@.service: Unit name
osbuild-remote-worker@.service is missing the instance name.
See system logs and 'systemctl status osbuild-remote-worker@.service' for
details.
%systemd_preun calls on a package upgrade this:
systemctl --no-reload disable --now [ARGS...]
I tried using a wildcard:
systemctl --no-reload disable --now "osbuild-worker@*.service"
But this gives:
Invalid unit name "osbuild-worker@*.service" was escaped as
"osbuild-worker@\x2a.service" (maybe you should use systemd-escape?)
Failed to disable unit: Unit file osbuild-worker@\x2a.service does not exist.
In the end I decided to use two commands - disable and stop. Disable works
without a wildcard, stop requires one. This solution should mimic systemctl
disable --now pretty well.
Currently, upgrading osbuild-composer produces these warnings:
Failed to try-restart osbuild-worker@.service: Unit name
osbuild-worker@.service is missing the instance name.
See system logs and 'systemctl status osbuild-worker@.service' for details.
Failed to try-restart osbuild-remote-worker@.service: Unit name
osbuild-remote-worker@.service is missing the instance name.
See system logs and 'systemctl status osbuild-remote-worker@.service'
for details.
%systemd_postun_with_restart calls on a package upgrade this:
systemctl try-restart [ARGS...]
However, try-restart requires using a wildcard when restarting all
the services created from a template unit. This commit therefore adds the
wildcards.
Previously, vhd images were tested using QEMU. This commit changes that to
boot them in the actual Azure infrastructure.
Azure VMs have quite a lot of dependencies - a network interface, a virtual
network, a network security group, a public ip address and a disk. Azure CLI
and Azure Portal handle the creation of all these resources internally.
However, when using the API, the caller is responsible to create all these
resources before creating an actual VM.
To handle the creation of all the resources in the right order, a deployment
is used. A deployment is a set of resources defined in a JSON document.
It can optionally take parameters to customize each deployment. After the
deployment is finished, the VM is up and ready to be tested using SSH.
Sadly, the deployments are a bit hard to clean-up. One would expect that
deleting a deployment removes all the deployed resources. However, it doesn't
work this way and therefore it's needed to clean up all resources "manually".
For this reason, our deployment sets a unique tag on all the resources created
by the deployment. After this test is finished, the API is queried for all
the resources with the tag and then, they're deleted in the right order.
We will soon need the support for vhdx format, which is supported since
osbuild 12.
This commit bumps the dependency in the spec file and also updates the
submodule.
The newest osbuild is needed for support for aarch64. The aarch64 images
are using GPT which requires stable partuuid which was included in the
latest osbuild release (11). This will be used to produce stable
image-info tests.
On Fedora, the `%gobuild` macro turns off go modules and sets gopath to
installed libraries (in `/usr/share/gocode`).
When building the test binaries, we used `go test -c` directly, which
downloaded the libraries into GOPATH. That's wasteful and doesn't work
in mock (without internet). Replicate what `%gobuild` does by setting
`GO111MODULE=off` and `GOPATH` to the correct value.
RHEL uses modules as intended. No change is necessary.
With this change the integration tests can now also be run as unit tests
against the mocked server. The way it works is this:
internal/client/unit_test.go sets up the mock server and is built
when the `integration` build tag is *not* included.
internal/client/integration_test.go sets up the connection to an
existing server and is built when the `integration` build tag *is*
included.
The test code is built and run for both cases.
Currently they all pass for the integration test run. The unit test
cases need some work because the mocked server isn't a real server with
real depsolving and package lists. A future commit will fix this.
In the new spec file, the tests-related code is always at the bottom of the
containing block to make the conditional code easier to read. Do the same
in the old spec file.
Also, in the new spec file, some whitespaces are different (rpmlint reported
mix used of tabs and spaces), let's do the same changes in the old spec file.
We're currently rewriting all the integration tests to use the Go
testing framework. This commit does the switch for the image tests.
I decided not to use the testing framework in functions which are
not directly tight to testing (booting images, running osbuild). I think
it's reasonable to use classic error handling there and propagate the errors
to places directly tight to testing and use the testing library.
This enables us to reuse the code in different part of projects if needed.
Convert weldrcheck to use the standard go testing framework along with
the github.com/stretchr/testify/require assert package.
This also removes the cmd/osbuild-weldr-tests and builds the test binary
directly from the weldrcheck package. This makes it easier to organize
the code instead of putting it all into a single main_test.go file.
This commit makes the osbuild-image-tests binary doing the same set of tests
like the old test/run script.
Changes from test/run:
- qemu/nspawn are now killed gracefully. Firstly, SIGTERM is sent.
If the process doesn't exit till the timeout, SIGKILL is sent.
I changed this because nspawn leaves some artifacts behind when killed
by SIGKILL.
- the unsharing of network namespace now works differently because of
systemd issue #15079
This allows us to take advantage of the `testing` package. It also gives
the resulting test binary common command line arguments (same as `go
test`).
Tests need to be compiled with `go test -c`, which injects a `Main()`
that calls the Test* functions.
This is not supported by the golang rpm macros. Thus, build this binary
by calling `go test -c` directly, but taking care to pass the same
linker flags as the `%gobuild` macro.
Mark the test binary with the `integration` build constraint, so that
`go test ./...` doesn't pick them up. That's only for unit tests.
The idea is to move all other test binaries to this scheme as well.
Spec file changes by Lars Karlitski <lars@karlitski.net>
Since the recent commit (04db4fa) we are no longer provider
of lorax-composer. This was needed because when installing cockpit-composer,
which depends on lorax-composer, dnf chose us as the lorax-composer provider
instead of the original lorax-composer package. As we are not yet ready to
fully replace lorax-composer, this broke the cockpit-composer tests.
This commit is the first part of long term fix. Osbuild-composer is now
a provider of weldr. Soon, lorax-composer will also be a provider of
weldr. After that, cockpit-composer will be switched to be depending
on weldr. Also, it will suggest lorax-composer, which will force dnf
to use lorax-composer as the default weldr backend. If someone wants
to experiment with osbuild-composer, it will be possible, because
it will also be a provider of weldr and there will be no need to
install lorax-composer to satisfy cockpit-composer dependencies.
We're not fully compatible with lorax-composer's API yet, and we don't
provide lorax-composer's systemd unit files.
As a result, cockpit-composer's integration tests fail, because `dnf
install lorax-composer` on Fedora can result in installing
osbuild-composer in some cases.
This runs tests against a running API server, either lorax-composer or
osbuild-composer, and reports the results to stdout. It uses the
/run/weldr/api.socket to communicate with the server.
Prior this commit installing the worker sub-packages shows the following
warning:
Failed to preset unit: Unit file osbuild-worker@.service does not exist.
Moving the unit file to the sub-package fixes it.
./test/run test suite has served us well over the last months. However,
there is currently a major effort to run the better defined integration
test suite on a CI. Nonetheless, two very important parts are still missing
from the integration test suite: inspecting the image with image-info
and booting the image. This commit begins the work on this matter by porting
a part of ./test/run suite to Go. Currently, only image-info tests work, the
rest will come in the following commits.
This changes osbuild-composer's behavior to match lorax-composer when
encountering invalid versions. Instead of leaving them as-is it will
return a BlueprintError explaining the problem. eg.
"errors": [
{
"id": "BlueprintsError",
"msg": "Invalid 'version', must use Semantic Versioning: is not in dotted-tri format"
}
]
This is enforced on new blueprints (including the workspace). If a
previously stored blueprint has an invalid version and a new one is
pushed it will use the new version number instead of trying to bump the
invalid one.
This also moves the version bump logic into blueprint instead of store,
and adds an Initialize function that will make sure that the blueprint
has sane default values for any missing fields.
This includes tests for the Initialize and BumpVersion functions.
