2 configurations for the listeners are now possible:
- enableJWT=false with client ssl auth
- enableJWT=true with https
Actual verification of the tokens is handled by
https://github.com/openshift-online/ocm-sdk-go.
An authentication handler is run as the top level handler, before any
routing is done. Routes which do not require authentication should be
listed as exceptions.
Authentication can be restricted using an ACL file which allows
filtering based on JWT claims. For more information see the inline
comments in ocm-sdk/authentication.
As an added quirk the `-v` flag for the osbuild-composer executable was
changed to `-verbose` to avoid flag collision with glog which declares
the `-v` flag in the package `init()` function. The ocm-sdk depends on
glog and pulls it in.
Prepending the definition to the spec file feels brittle. Stop doing
that – pass --define to rpmbuild instead.
Keep copying the spec file from git though, so that it's always using
the one from the last commit, and not the current checkout.
rpmlint doesn't like SRPMs built from specfiles with a filename different
than what it's in the Name field inside the spec. This commit removes
the renaming. Now, the specfile is always named osbuild-composer.spec.
The Koji test in Github actions was always a bit quick and dirty solution.
I think it's much nicer solution to run it on Schutzbot.
Therefore, this commit moves the koji_test.go to a new osbuild-koji-tests
executable. This new test isn't run in the base test suite as one would
anticipate but inside the koji.sh test. This is needed because
osbuild-koji-tests requires a running koji instance. This might change
in the future but I think it works for now.
This removes the osbuild-composer-cloud package, binary, systemd units,
the (unused) test binary, and the (only-run-on-RHEL) test in aws.sh.
Instead, move the cloud API into the main package, using the same
socket as the koji API, osbuild-composer-api.socket. Expose it next to
the koji API on route `/api/composer/v1`.
This is a backwards incompatible change, but only of the -cloud parts,
which have been marked as subject to change.
Currently, we have osbuild-image-tests binary committed in the master branch.
IMHO the root cause is that we don't have it in .gitignore. Actually, I think
that it is pretty hard keep .gitignore in sync with the build target.
This commit solves the situation by putting all the built binaries into bin
directory and adding this directory into .gitignore. This way, it's much
harder to actually commit a new Go binary into repository.
This commit doesn't remove the binary as #1017 already does that.
These had the generic name osbulid-tests, rename the binary to more
closely describe what is under test: weldr API as composed by
composer-cli.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This commit introduces a new test binary responsible for testing TLS
authentication.
Currently, it covers both remote worker API and Koji API. It tests that
the server refuses certificates issued by an untrusted CA or self-signed ones.
Also, it tests that the certificate is issued for an allowed domain.
TODO: certs with subject alternative name are currently not used in tests.
They should work just right, but a proper testing requires more tinkering with
OpenSSL than I'm willing to accept at this time
1) generating a private key and signing request was merged into one command
2) -sha256 was dropped, let openssl decide which digest should be used
3) signing request is deleted after the it's signed
Add a systemd socket for Koji API. If enabled when osbuild-composer.service
is started, the service will also listen on the socket and serve Koji API
there.
Note that Koji API doesn't upload to Koji yet, this still needs to be hooked
up.
Based on a patch from Tom Gundersen, thanks!
The tests are no longer run on Travis, therefore we don't need the special
setup to run them there.
This change should also fix#929 that is probably caused due to osbuild
executed in a weird way.
Fixes#929
The osbuild-composer-rcm package was never finished, not in use and will be replaced by osbulid-composer-koji.
Signed-off-by: Tom Gundersen <teg@jklm.no>
c886e1be changed the indentation of the NEWS file. This commit changes also
the indentation in the news template produced by `make release`.
Also - in front of the Location was changed to — to please David.
The .* before the digit group is greedy and it catches the first version
digit when the version >= 10. This commit changes the .* to [^[:digit:]]*
so it doesn't catch the first digit.
Import a new `NEWS.md` file with the changelog for each release. This is
targetted at distributors, so it should contain information on new
features, modified features, and anything important to packagers. This
is not a replacement for the git-changelog, but rather a human-readable
instruction to packagers what needs to be changed when using the new
release.
Additionally, copy `make release` from *osbuild* to help in creating new
release entries in NEWS.md.
This adds `make man` with SRCDIR and BUILDDIR support. This is copied
from *osbuild*. Since `make man` will be the official way to build the
man-pages, lets also import `make help` and some documentation for the
Makefile.
With this change the integration tests can now also be run as unit tests
against the mocked server. The way it works is this:
internal/client/unit_test.go sets up the mock server and is built
when the `integration` build tag is *not* included.
internal/client/integration_test.go sets up the connection to an
existing server and is built when the `integration` build tag *is*
included.
The test code is built and run for both cases.
Currently they all pass for the integration test run. The unit test
cases need some work because the mocked server isn't a real server with
real depsolving and package lists. A future commit will fix this.
This commit switches the rpm and srpm targets to use the new spec file.
Also, it introduces old-rpm and old-srpm targets to build an rpm from
the old spec file.
Note that the rpm target builds also the tests subpackages (they are not
built by default).
We're currently rewriting all the integration tests to use the Go
testing framework. This commit does the switch for the image tests.
I decided not to use the testing framework in functions which are
not directly tight to testing (booting images, running osbuild). I think
it's reasonable to use classic error handling there and propagate the errors
to places directly tight to testing and use the testing library.
This enables us to reuse the code in different part of projects if needed.
Convert weldrcheck to use the standard go testing framework along with
the github.com/stretchr/testify/require assert package.
This also removes the cmd/osbuild-weldr-tests and builds the test binary
directly from the weldrcheck package. This makes it easier to organize
the code instead of putting it all into a single main_test.go file.
This allows us to take advantage of the `testing` package. It also gives
the resulting test binary common command line arguments (same as `go
test`).
Tests need to be compiled with `go test -c`, which injects a `Main()`
that calls the Test* functions.
This is not supported by the golang rpm macros. Thus, build this binary
by calling `go test -c` directly, but taking care to pass the same
linker flags as the `%gobuild` macro.
Mark the test binary with the `integration` build constraint, so that
`go test ./...` doesn't pick them up. That's only for unit tests.
The idea is to move all other test binaries to this scheme as well.
Spec file changes by Lars Karlitski <lars@karlitski.net>
It is an equivalent to what we already have for Weldr API but this one
is for the RCM API. It should test the expected use cases:
* submit a compose
* get a status
The spec file in the current working directory might have changes. When
building rpms with the commit hash in the version, they ought to be
built with the spec file from that hash as well.
Prior to this patch, `make rpm` would produce rpms that have the latest
tag as their versions. This was confusing, because one could never know
which contents are in a locally built rpm.
Change this so that the is version always based on the commit hash of
HEAD. This is easy: the golang macros read a `%commit` macro when it
exists and do this for us.
To simplify more, only define `%_topdir` to ./rpmbuild and use
rpmbuild's known directory structure (SPEC, SOURCES, RPMS, ...)
otherwise, to make it easier to find build results.
Build the specfile, tarball, source rpms, and rpms with `make rpm`,
without separate sub-targets. We can reintroduce them if they're needed
somewhere.
Also remove the `check-working-directory` target. It should be clear
from the output that only the currently-committed files are included,
because the resulting tarball and rpms contain the commit hash. Without
the check, one can work on the Makefile without having to commit all the
time, for example ;)
This runs tests against a running API server, either lorax-composer or
osbuild-composer, and reports the results to stdout. It uses the
/run/weldr/api.socket to communicate with the server.
There's a usecase for running workers at a different machine than
the composer. For example when there's need for making images for
architecture different then the composer is running at. Although osbuild has
some kind of support for cross-architecture builds, we still consider it
as experimental, not-yet-production-ready feature.
This commit adds a support to composer and worker to communicate using TCP.
To ensure safe communication through the wild worlds of Internet, TLS is not
only supported but even required when using TCP. Both server and client
TLS authentication are required. This means both sides must have their own
private key/certificate pair and both certificates must be signed using one
certificate authority. Examples how to generate all this fancy crypto stuff
can be found in Makefile.
Changes on the composer side:
When osbuild-remote-worker.socket is started before osbuild-composer.service,
osbuild-composer also serves jobqueue API on this socket. The unix domain
socket is not affected by this changes - it is enabled at all times
independently on the remote one. The osbuild-remote-worker.socket listens
by default on TCP port 8700.
When running the composer with remote worker socket enabled, the following
files are required:
- /etc/osbuild-composer/ca-crt.pem (CA certificate)
- /etc/osbuild-composer/composer-key.pem (composer private key)
- /etc/osbuild-composer/composer-crt.pem (composer certificate)
Changes on the worker side:
osbuild-worker has now --remote argument taking the address to a composer
instance. When present, the worker will try to establish TLS secured TCP
connection with the composer. When not present, the worker will use
the unix domain socket method. The unit template file osbuild-remote-worker
was added to simplify the spawning of workers. For example
systemctl start osbuild-remote-worker@example.com
starts a worker which will attempt to connect to the composer instance
running on the address example.com.
When running the worker with --remote argument, the following files are
required:
- /etc/osbuild-composer/ca-crt.pem (CA certificate)
- /etc/osbuild-composer/worker-key.pem (worker private key)
- /etc/osbuild-composer/worker-crt.pem (worker certificate)
By default osbuild-composer.service will always spawn one local worker.
If you don't want it you need to mask the default worker unit by:
systemctl mask osbuild-worker@1.service
Closing remarks:
Remember that both composer and worker certificate must be signed by
the same CA!
During development of a new distro, we need to test composer against
nightly or beta repositories, but we cannot ship composer itself
with the nightly repository information hardcoded in. At the same
time, we want to distinguish between the system repositories of the
host and the repositories we use to generate images (the host may not
use the same distro/version/architecture as the target, and it may
include custom repositories that the target should not).
We therefore ship per distro repository information that can be
overriden (typically in testing) by dropping files in /etc.
For now use the latest nightlies for RHEL-8.2, we may want to
replace these with the official mirrors for GA eventually.
Signed-off-by: Tom Gundersen <teg@jklm.no>
It uses Azure SDK to connect to Azure storage, creates a container there
and uploads the image. Unfortunately the API for page blobs does not
include some thread pool for upload so I implemented one myself. The
performance can be tweaked using the upload chunk size and number of
parallel threads.
The package is prepared to be refactored into common module within
internals package as soon as we agree on the of these common packages
for image upload.
Add azure-blob-storage rpm package as a dependency
It didn't work for me using the `golang(package)` syntax. Using the
package name explicitly works.
This makes no difference, so let's just put them where the Fedora
guidelines say they should be.
Also, make sure to own the containing directory.
Signed-off-by: Tom Gundersen <teg@jklm.no>
