Don't give out job ids to workers, but `tokens`, which serve as an
indirection. This way, restarting composer won't confuse it when a stray
worker returns a result for a job that was still running. Also,
artifacts are only moved to the final location once a job finishes.
This change breaks backwards compatibility, but we're not yet promising
a stable worker API to anyone.
This drops the transition tests in server_test.go. These don't make much
sense anymore, because there's only one allowed transition, from running
to finished. They heavily relied on job slot ids, which are not easily
accessible with the `TestRoute` API. Overall, adjusting this seemed like
too much work for their benefit.
The code generator uses the `operationID` field to generate server
handlers, client functions, and types. Use simpler names to make the
generated code easier to read.
This kind of common base path is better set in the top-level
`server.url` field, so that it can be adjusted.
For now, drop it completely, as we already broke the consistency when
introducing the `/status` route.
This change breaks backwards compatibility, but we're not yet promising
a stable worker API to anyone.
Write an openapi spec for the worker API and use `deepmap/oapi-codegen`
to generate scaffolding for the server-side using the `labstack/echo`
server.
Incidentally, echo by default returns the errors in the same format that
worker API always has:
{ "message": "..." }
The API itself is unchanged to make this change easier to understand. It
will be changed to better suit our needs in future commits.
Notes:
ATM will not run any actual tests b/c we want to make sure the
pipeline configuration is correct.
run_tests() will call the deploy.sh script and then do nothing
b/c of the "dummy-" prefix which doesn't match any actual tests!
We are storing some data in the user's home directory, so let's print
the username so we know what that is.
In particular, this would tell us which user has been authorized to log
in via ssh.
Signed-off-by: Tom Gundersen <teg@jklm.no>
More specifically only those that are needed in
/cmd/osbuild-image/tests.
This patch can be merged with the previous one if we want to make sure
every commit can be built, but I'm going to keep it like this for now so
that we can easily see the changes.
We need this for greenboot-status, in the RHEL for Edge images. This
updates the generator for x86_64 and aarch64 and updates the test cases
for rhel-edge-commit.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We now have greenboot available so update the packages and services
also add exclude sections as subscription-manager is for some reason
getting pulled in which brings dnf and other deps we explicitly don't
want.
Signed-off-by: Peter Robinson <pbrobinson@gmail.com>
[teg: minor fixes and squash several fixup commits]
explicitly specify the cluster and the default resource pool
when importing b/c the import process creates a temporary VM,
which requires a ResourcePool to provision. Same thing when
provisioning a VM.
Now that we've reduced how much of our PSI quota we are using so the
OpenStack boot tests will work, we need to use AWS for jobs more often.
This should allow test runs to complete a little sooner by freeing up
PSI resources for the jobs that are only able to run there.
Signed-off-by: Major Hayden <major@redhat.com>
Prior this commit we only had support for username/password authentication
in the koji integration. This wasn't particularly useful because this
auth type isn't used in any production instance.
This commit adds the support for GSSAPI/Kerberos authentication.
The implementation uses kerby library which is very lightweight wrapper
around C gssapi library.
Also, the koji unit test and the run-koji-container script were modified
so the GSSAPI auth is fully tested.
In the near future, we will need to communicate with Koji using HTTPS.
This will surely bring the need for ignoring bad certificates/providing
our own self-signed ones. Thus, this commit prepares the Koji integration
by adding a way to accept a custom http transport which can be used to
customize the TLS settings.
Previously, Koji instance could be both logged-in and not logged-in.
This change disallows it: Now, the Koji instance is created by calling
koji.Login, so it must be always logged-in. This change should lead to more
robust code.
Services in Github Actions are cool but have some drawbacks:
1) We want to be able to use the container setup locally, therefore there's
run-koji-container script which does exactly the same setup as it's defined
in Github Actions. We don't want duplicities though.
2) In the near future, we will need more complicated setup - generating
certificates before a container is started. This is not possible with
the current Github Actions capabilities.
This commit removes the container setup from Github Actions and just reuses
the run-koji-container script in the GH Actions environment. This way we
have only one setup which is also more flexible.
run-koji-container has now two actions: start and stop:
- ./run-koji-container.sh start
- ./run-koji-container.sh stop
The start action starts all containers. When it exits, all containers are
started and running in the background. To stop and removethem, use the stop
action.
This change is needed so we're able to easily use this script also in the CI
environment.
The setup should be container engine agnostic. This changes allows this script
to be run on systems which prefer docker over podman (e.g. Github Actions).
- inside RunJob() there is a deferred function which will remove
the entire temporary directory in which images are created, including
the streamOptimized file
- inside testBootUsingVMware(), which wants to use this function,
there is already a deferred function which removes the converted
image
osbuild support returning metadata about each of the stages/assembler
runs. Parse the results from the rpm stage, which contains the header
fields from the installed RPMs, in particular the MD5 sum of the RPMs in
question. This information is needed to be passed as metadata to koji
when uploading images.
Signed-off-by: Tom Gundersen <teg@jklm.no>
In the same way `osbuild.Manifest` is the input to the osbuild API,
`osbuild.Result` is the output. Move it to the `osbuild` package where
it belongs.
This is not a functional change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
#932 bumped the minimal Go version to 1.13. Prepare-source script had to be
rerun because of some changes in vendoring.
Unfortunately, while this PR was waiting to merged, #797 also touching
vendoring was merged. This "commit race condition" led to a failing check
on master branch because of bad vendoring.
This commit reruns the prepare-source script to fix the vendoring once again.
All our downstream platforms now support Go 1.13:
RHEL 8.2: golang-1.13.4
Fedora 31: golang-1.13.14
There's no reason anymore to stay on 1.12, therefore this commit bumps
the minimal required Go version to 1.13
This test is not run anywhere because it was surpassed by image tests with
azure boot type which perform more than just uploading a randomly generated
file to Azure. Let's delete dead code.
The tests are no longer run on Travis, therefore we don't need the special
setup to run them there.
This change should also fix#929 that is probably caused due to osbuild
executed in a weird way.
Fixes#929
Applying a `Name` tag to the snapshot makes the name of the image appear
in the snapshot listing in EC2's console. It also makes it easier to
remove a snapshot after deregistering the AMI.
Signed-off-by: Major Hayden <major@redhat.com>
Although the obuild-remote-worker@.service unit can be started, it can't
be enabled at boot time since the `Install` section is missing in the
unit file.
Add a small `[Install]` section with the same `WantedBy` as
osbuild-composr.service.
Fixes#924.
Signed-off-by: Major Hayden <major@redhat.com>
For reasons unknown, golangci-lint's default 1m0s timeout is *slightly*
too short for CI runs occasionally. Extend it to 5 minutes to ensure the
job always has enough time to run.
Signed-off-by: Major Hayden <major@redhat.com>
acf91a4 enabled fastestmirror but also calls `base.init_plugins()` to
initialize dnf plugins. This is not necessary and not what we want
conceptually.
Not necessary, because `fastestmirror` is a dnf built-in (it was a
plugin during yum-times [1]). The same patch sets the `fastestmirror`
option as well. Thus, this patch does not revert functionality.
Not what we want, because we're using dnf more as a library, explicitly
passing all options. Plugins depend on additional host configuration,
which we'd like to avoid pulling in. In particular, the
subscription-manager plugin tries reading certificates in `/etc/pki`,
which are not readable by the `osbuild-composer` user. This leads to
these errors in the journal:
[ERROR] dnf-json:54297:MainThread @logutil.py:194 -
[Errno 13] Permission denied: '/var/log/rhsm/rhsm.log' -
Further logging output will be written to stderr
[ERROR] dnf-json:54297:MainThread @identity.py:156 -
Reload of consumer identity cert /etc/pki/consumer/cert.pem
raised an exception with msg:
[Errno 13] Permission denied: '/etc/pki/consumer/key.pem'
These errors are not fatal, but could confuse people when inspecting
logs to find unrelated problems. This patch makes them disappear.
[1] https://fedoraproject.org/wiki/Yum_to_DNF_Cheatsheet
The vmdk images are meant to be imported into VMWare vCenter, not to be
booted directly.
Our image-info tests verifies that this work, they should be converted
into integration tests similar to the aws.sh script in the future.
Signed-off-by: Tom Gundersen <teg@jklm.no>
