Currently, removing osbuild-composer produces these warnings:
Failed to stop osbuild-worker@.service: Unit name
osbuild-worker@.service is missing the instance name.
See system logs and 'systemctl status osbuild-worker@.service' for details.
Failed to stop osbuild-remote-worker@.service: Unit name
osbuild-remote-worker@.service is missing the instance name.
See system logs and 'systemctl status osbuild-remote-worker@.service' for
details.
%systemd_preun calls on a package upgrade this:
systemctl --no-reload disable --now [ARGS...]
I tried using a wildcard:
systemctl --no-reload disable --now "osbuild-worker@*.service"
But this gives:
Invalid unit name "osbuild-worker@*.service" was escaped as
"osbuild-worker@\x2a.service" (maybe you should use systemd-escape?)
Failed to disable unit: Unit file osbuild-worker@\x2a.service does not exist.
In the end I decided to use two commands - disable and stop. Disable works
without a wildcard, stop requires one. This solution should mimic systemctl
disable --now pretty well.
Currently, upgrading osbuild-composer produces these warnings:
Failed to try-restart osbuild-worker@.service: Unit name
osbuild-worker@.service is missing the instance name.
See system logs and 'systemctl status osbuild-worker@.service' for details.
Failed to try-restart osbuild-remote-worker@.service: Unit name
osbuild-remote-worker@.service is missing the instance name.
See system logs and 'systemctl status osbuild-remote-worker@.service'
for details.
%systemd_postun_with_restart calls on a package upgrade this:
systemctl try-restart [ARGS...]
However, try-restart requires using a wildcard when restarting all
the services created from a template unit. This commit therefore adds the
wildcards.
Previously, vhd images were tested using QEMU. This commit changes that to
boot them in the actual Azure infrastructure.
Azure VMs have quite a lot of dependencies - a network interface, a virtual
network, a network security group, a public ip address and a disk. Azure CLI
and Azure Portal handle the creation of all these resources internally.
However, when using the API, the caller is responsible to create all these
resources before creating an actual VM.
To handle the creation of all the resources in the right order, a deployment
is used. A deployment is a set of resources defined in a JSON document.
It can optionally take parameters to customize each deployment. After the
deployment is finished, the VM is up and ready to be tested using SSH.
Sadly, the deployments are a bit hard to clean-up. One would expect that
deleting a deployment removes all the deployed resources. However, it doesn't
work this way and therefore it's needed to clean up all resources "manually".
For this reason, our deployment sets a unique tag on all the resources created
by the deployment. After this test is finished, the API is queried for all
the resources with the tag and then, they're deleted in the right order.
We will soon need the support for vhdx format, which is supported since
osbuild 12.
This commit bumps the dependency in the spec file and also updates the
submodule.
The newest osbuild is needed for support for aarch64. The aarch64 images
are using GPT which requires stable partuuid which was included in the
latest osbuild release (11). This will be used to produce stable
image-info tests.
On Fedora, the `%gobuild` macro turns off go modules and sets gopath to
installed libraries (in `/usr/share/gocode`).
When building the test binaries, we used `go test -c` directly, which
downloaded the libraries into GOPATH. That's wasteful and doesn't work
in mock (without internet). Replicate what `%gobuild` does by setting
`GO111MODULE=off` and `GOPATH` to the correct value.
RHEL uses modules as intended. No change is necessary.
With this change the integration tests can now also be run as unit tests
against the mocked server. The way it works is this:
internal/client/unit_test.go sets up the mock server and is built
when the `integration` build tag is *not* included.
internal/client/integration_test.go sets up the connection to an
existing server and is built when the `integration` build tag *is*
included.
The test code is built and run for both cases.
Currently they all pass for the integration test run. The unit test
cases need some work because the mocked server isn't a real server with
real depsolving and package lists. A future commit will fix this.
In the new spec file, the tests-related code is always at the bottom of the
containing block to make the conditional code easier to read. Do the same
in the old spec file.
Also, in the new spec file, some whitespaces are different (rpmlint reported
mix used of tabs and spaces), let's do the same changes in the old spec file.
We're currently rewriting all the integration tests to use the Go
testing framework. This commit does the switch for the image tests.
I decided not to use the testing framework in functions which are
not directly tight to testing (booting images, running osbuild). I think
it's reasonable to use classic error handling there and propagate the errors
to places directly tight to testing and use the testing library.
This enables us to reuse the code in different part of projects if needed.
Convert weldrcheck to use the standard go testing framework along with
the github.com/stretchr/testify/require assert package.
This also removes the cmd/osbuild-weldr-tests and builds the test binary
directly from the weldrcheck package. This makes it easier to organize
the code instead of putting it all into a single main_test.go file.
This commit makes the osbuild-image-tests binary doing the same set of tests
like the old test/run script.
Changes from test/run:
- qemu/nspawn are now killed gracefully. Firstly, SIGTERM is sent.
If the process doesn't exit till the timeout, SIGKILL is sent.
I changed this because nspawn leaves some artifacts behind when killed
by SIGKILL.
- the unsharing of network namespace now works differently because of
systemd issue #15079
This allows us to take advantage of the `testing` package. It also gives
the resulting test binary common command line arguments (same as `go
test`).
Tests need to be compiled with `go test -c`, which injects a `Main()`
that calls the Test* functions.
This is not supported by the golang rpm macros. Thus, build this binary
by calling `go test -c` directly, but taking care to pass the same
linker flags as the `%gobuild` macro.
Mark the test binary with the `integration` build constraint, so that
`go test ./...` doesn't pick them up. That's only for unit tests.
The idea is to move all other test binaries to this scheme as well.
Spec file changes by Lars Karlitski <lars@karlitski.net>
Since the recent commit (04db4fa) we are no longer provider
of lorax-composer. This was needed because when installing cockpit-composer,
which depends on lorax-composer, dnf chose us as the lorax-composer provider
instead of the original lorax-composer package. As we are not yet ready to
fully replace lorax-composer, this broke the cockpit-composer tests.
This commit is the first part of long term fix. Osbuild-composer is now
a provider of weldr. Soon, lorax-composer will also be a provider of
weldr. After that, cockpit-composer will be switched to be depending
on weldr. Also, it will suggest lorax-composer, which will force dnf
to use lorax-composer as the default weldr backend. If someone wants
to experiment with osbuild-composer, it will be possible, because
it will also be a provider of weldr and there will be no need to
install lorax-composer to satisfy cockpit-composer dependencies.
We're not fully compatible with lorax-composer's API yet, and we don't
provide lorax-composer's systemd unit files.
As a result, cockpit-composer's integration tests fail, because `dnf
install lorax-composer` on Fedora can result in installing
osbuild-composer in some cases.
This runs tests against a running API server, either lorax-composer or
osbuild-composer, and reports the results to stdout. It uses the
/run/weldr/api.socket to communicate with the server.
Prior this commit installing the worker sub-packages shows the following
warning:
Failed to preset unit: Unit file osbuild-worker@.service does not exist.
Moving the unit file to the sub-package fixes it.
./test/run test suite has served us well over the last months. However,
there is currently a major effort to run the better defined integration
test suite on a CI. Nonetheless, two very important parts are still missing
from the integration test suite: inspecting the image with image-info
and booting the image. This commit begins the work on this matter by porting
a part of ./test/run suite to Go. Currently, only image-info tests work, the
rest will come in the following commits.
This changes osbuild-composer's behavior to match lorax-composer when
encountering invalid versions. Instead of leaving them as-is it will
return a BlueprintError explaining the problem. eg.
"errors": [
{
"id": "BlueprintsError",
"msg": "Invalid 'version', must use Semantic Versioning: is not in dotted-tri format"
}
]
This is enforced on new blueprints (including the workspace). If a
previously stored blueprint has an invalid version and a new one is
pushed it will use the new version number instead of trying to bump the
invalid one.
This also moves the version bump logic into blueprint instead of store,
and adds an Initialize function that will make sure that the blueprint
has sane default values for any missing fields.
This includes tests for the Initialize and BumpVersion functions.
As the worker can now be running on a different machine than the composer
it makes sense to install only worker binary on some machines. This commit
does exactly that - worker is now its own subpackage with the beautiful name
of golang-github-osbuild-composer-worker.
The main osbuild-composer package requires the worker subpackage, therefore
there will always be worker installed with composer. When composer is started
one local worker process will be spawned. If you don't want the default
worker process you need to mask its unit file:
systemctl mask osbuild-worker@1.service
Introduce a new osbuild-tests command and ship it in the -tests
sub-package.
The intended usecase is to install the -tests subpackage into an
otherwise pristine VM, and call the osbuild-tests binary over ssh
from the outside of the booted VM. If the binary exits with a return
code of 0, the tests passed, otherwise they failed. The VM should
not be reused after running the tests.
Signed-off-by: Tom Gundersen <teg@jklm.no>
During development of a new distro, we need to test composer against
nightly or beta repositories, but we cannot ship composer itself
with the nightly repository information hardcoded in. At the same
time, we want to distinguish between the system repositories of the
host and the repositories we use to generate images (the host may not
use the same distro/version/architecture as the target, and it may
include custom repositories that the target should not).
We therefore ship per distro repository information that can be
overriden (typically in testing) by dropping files in /etc.
For now use the latest nightlies for RHEL-8.2, we may want to
replace these with the official mirrors for GA eventually.
Signed-off-by: Tom Gundersen <teg@jklm.no>
In RHEL golang dependencies must be vendored, whereas on Fedora they
must be packaged separately. Add conditionals accordingly.
Some macros are not yet available in RHEL, so fall back to older
versions. We may want to just use macros avilable everywhere
unconditionally, but I left that for a follow-up.
Signed-off-by: Tom Gundersen <teg@jklm.no>
this is a small helper to enable easy switch between backends
in the existing test suite where we reference everything by name.
The current working variant is:
yum install $BACKEND
systemctl enable $BACKEND.socket
where BACKEND is either lorax-composer or osbuild-composer!
This commandline tools uploads a file to S3, as a proof of concept.
All options are mandatory. Credentials are only read from the
commandline and not from the environment or configuration files.
The next step is to add support for importing from S3 to EC2,
currently the images we produce cannot be imported as-is, so this
requires more research.
To try this out: create an S3 bucket, get your credentials and
call the tool, passing any value as `key`. Note that if the key
already exists, it will be overwritten.
Signed-off-by: Tom Gundersen <teg@jklm.no>
It uses Azure SDK to connect to Azure storage, creates a container there
and uploads the image. Unfortunately the API for page blobs does not
include some thread pool for upload so I implemented one myself. The
performance can be tweaked using the upload chunk size and number of
parallel threads.
The package is prepared to be refactored into common module within
internals package as soon as we agree on the of these common packages
for image upload.
Add azure-blob-storage rpm package as a dependency
It didn't work for me using the `golang(package)` syntax. Using the
package name explicitly works.
