Commit graph

1372 commits

Author SHA1 Message Date
Tom Gundersen
c7d993d537 scutzbot: deploy pre-generated SSL certs for testing 2020-10-06 13:08:26 +02:00
Tom Gundersen
9d18c3bfa2 tests: rename composer-cli tests
These had the generic name osbulid-tests, rename the binary to more
closely describe what is under test: weldr API as composed by
composer-cli.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-10-06 13:08:26 +02:00
Jacob Kozol
678de9d1ef distro: add fedora 33 support
Fedora 33 images can now be built and test cases are added for the new
images. The fedora 33 qcow2 and vmdk images are based off of the
official images and their kickstarters found here:
https://pagure.io/fedora-kickstarts. The fedora 33 iot image is based
off of the the config found here: https://pagure.io/fedora-iot/ostree.
The openstack, azure, and amazon image types have changes made to them
based off of the changes made to the qcow2. The changes between fedora
32 and fedora 33 are as follows:

Grub now loads its kernel command line options from
etc/kernel/cmdline, /usr/lib/kernel/cmdline, and /proc/cmdline instead
of from grub env. This is addressed by adding kernelCmdlineStageOptions
to use osbuild's kernel-cmdline stage to set these options. Alongside
`ro biosdevname=0 net.ifnames=0`, we also set `no_timer_check
console=tty1 console=ttyS0,115200n8` per what is set in the official
qcow2. For azure and amazon, the kernelOptions are still set as they
were in fedora 32.

The timezone is now set to UTC if a user does not set a timezone in the
blueprint customizations. Also, the hostname is set to
localhost.localdomain if the hostname isn't set in the blueprint.

Finally, the following packages have been removed:

polkit
geolite2-city
geolite2-country
zram-generator-defaults
2020-10-01 10:11:03 +02:00
Lars Karlitski
40d8440660 kojiapi: prefix all routes with /api/composer-koji/v1
The cloud API will be moved to `/api/composer/v1` in the future.

Mention this in the `servers` section of the openapi.yml (relative URLs
are allowed) too, even though our generator does not consider it.
2020-09-24 21:08:56 +01:00
Ondřej Budai
5b57814664 api/worker, koji: change CA logic for client certificates
Prior this commit, /etc/osbuild-composer/ca-crt.pem certificate was
used as an authority to validate client certificates.

After this commit, the host's trusted certificates are used to do
the validation. Ability to override this behaviour is also introduced:

In osbuild-composer config file, under koji and worker sections, a new CA
option is now available. If set, osbuild-composer uses it as a path
to certificate used to validate client certificates instead of the
default ones.

With this feature, it's possible to restore the validation behaviour
used before this change. Just put following lines in
/etc/osbuild-composer/osbuild-composer.toml:

[koji]
ca = "/etc/osbuild-composer/ca-crt.pem"

[worker]
ca = "/etc/osbuild-composer/ca-crt.pem"
2020-09-23 11:08:21 +01:00
Ondřej Budai
68be242850 tests: introduce auth tests
This commit introduces a new test binary responsible for testing TLS
authentication.

Currently, it covers both remote worker API and Koji API. It tests that
the server refuses certificates issued by an untrusted CA or self-signed ones.
Also, it tests that the certificate is issued for an allowed domain.

TODO: certs with subject alternative name are currently not used in tests.
They should work just right, but a proper testing requires more tinkering with
OpenSSL than I'm willing to accept at this time
2020-09-23 11:08:21 +01:00
Ondřej Budai
28aff2259a schutzbot: start also the remote socket
The following commits will add testing of authentication on this socket.

This commit also sets up the config for composer other it won't start
with remote socket enabled.
2020-09-23 11:08:21 +01:00
Ondřej Budai
7fc3b47348 kojiapi: add domain allowlist
This commit adds a domain allowlist which works the same way as the one
for remote workers.

To accept just w1.osbuild.org and w2.osbuild.org, use:

[koji]
domain_allowlist = [ "w1.osbuild.org", "w2.osbuild.org" ]
2020-09-23 11:08:21 +01:00
Ondřej Budai
c9abb66637 osbuild-composer/config: rework the structure
Prior this change, the structure was following:

[koji.localhost.kerberos]

This change modifies it to:

[koji.servers.localhost.kerberos]

This allows us to put more config options under the koji section. See
following commits, they use this new possibility.
2020-09-23 11:08:21 +01:00
Ondřej Budai
9a26d077db worker/api: add domain allowlist
There's need for control which certificates to accept. This commit introduces
the domain allowlist. The basic idea is that composer accepts only
certificates issued to domain names specified in osbuild-composer config file.
It allows multiple domains to be specified.

To accept just w1.osbuild.org and w2.osbuild.org, use:
domain_allowlist = [ "w1.osbuild.org", "w2.osbuild.org" ]
2020-09-23 11:08:21 +01:00
sanne raymaekers
d5938d52b2 cloudapi: Add rhsm parameter to repository 2020-09-22 11:38:28 +02:00
sanne raymaekers
9404b0640a test: Regenerate the rhel-8 cases to add subscriptions 2020-09-22 11:38:28 +02:00
sanne raymaekers
9ca50ae3ac osbuild-composer-cloud: introduce the cloud-specific service 2020-09-22 11:38:28 +02:00
Martin Sehnoutka
d9f254f9d1 *.sh: apply fixes suggested by shellcheck
also:
tweak some automatic fixes to be more readable
fix automatically generated bash fixes for SSH options
2020-09-19 15:10:39 +02:00
Martin Sehnoutka
b06e6dd916 *.sh: apply automatically all suggestions from shellcheck
This commit was automatically generated using:
```
$ fd --exclude vendor sh | xargs shellcheck -f diff | git apply
```
2020-09-19 15:10:39 +02:00
Tom Gundersen
4f39a33d34 osbuild-composer-koji.socket: use standard https port
Rather than using the arbitrary port 8701, use the standard 443. The
worker API will remain on a separate port, and as long as the two APIs
are exposed by the same binary that will have to remain separate at
8700.

Move the test instance of koji on localhost from 443 to 4343, to avoid a
conflict.

In a follow-up we should also give this API a prefix, so the cloud API
can share the same port with it.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-09-17 17:34:57 +02:00
Tom Gundersen
c6cf9de85d koji: add config files to configure kerberos settings
Kerberos keytabs and principals are configured per koji server both in
composer and in the worker.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-09-16 00:15:02 +01:00
Tom Gundersen
9666be2891 schutzbot: add koji integration tests
This sets up containers running koji and supporting infrastructure, and
calls the osbuild-composer-koji API to build and image and push it into
our testing instance.

koji-compose.py and various fixes by Christian Kellner.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-09-16 00:15:02 +01:00
Xiaofeng Wang
bb94c76818 test: Add rhel-8-beta.json to override default one
OSTree testing needs redhat-release-8.3-0.4.el8.x86_64 to
fix BZ#1848453
2020-09-03 22:51:39 +01:00
Xiaofeng Wang
c1b0d348d6 test: Add ostree image installation and upgrade test and add a
new parallel stage for ostree test
2020-09-03 22:51:39 +01:00
Tom Gundersen
7e0711b805 test/generation/rhel8: move to today's nightly
We need this for greenboot-status, in the RHEL for Edge images. This
updates the generator for x86_64 and aarch64 and updates the test cases
for rhel-edge-commit.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-09-03 11:02:44 +01:00
Tom Gundersen
496f921eff schutzbot: don't boot-test vmdk images in libvirt
The vmdk images are meant to be imported into VMWare vCenter, not to be
booted directly.

Our image-info tests verifies that this work, they should be converted
into integration tests similar to the aws.sh script in the future.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-08-23 14:45:27 +02:00
Alexander Todorov
dcee05d8b6 Read vCenter credentials for Schutzbot & document them 2020-08-20 21:59:54 +02:00
Alexander Todorov
9cce43d384 tests: upload & test in vCenter. Closes #338 2020-08-20 21:59:54 +02:00
Tom Gundersen
dc5434d627 schutzbot/deploy: do not override repos when deploying on 8.2
We will automatically detect whether to use GA or Beta content, so we
can drop the override as 8.2 is GA so will get GA content.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-08-19 15:06:21 +02:00
Alexander Todorov
adc384002d Document which ENV variables are needed for testing in OpenStack 2020-08-07 19:58:37 +02:00
Alexander Todorov
0469be7a22 tests: Change boot typo to openstack. Refs #876
this will cause the images to be uploaded and booted in our
OpenStack cluster.
2020-08-07 19:58:37 +02:00
Martin Sehnoutka
55c86afaa6 test cases: fix f32 iot commit inputhash
The inputhash represents pipeline id, which is calculated from the
pipeline dictionary defined in the test case. The definition has been
modified without correction of the inputhash (because we did not run it
in our CI :-) ).

This patch fixed the inputhash to reflect the changes in the pipeline.
2020-08-03 17:46:49 +02:00
Major Hayden
3d538a6663 Add Lorax TOML workaround
Copy the TOML workaround from the aws script to the qemu script.

Signed-off-by: Major Hayden <major@redhat.com>
2020-07-27 10:27:15 -05:00
Major Hayden
574c4f622c Update test cases
Signed-off-by: Major Hayden <major@redhat.com>
2020-07-27 10:27:15 -05:00
Major Hayden
f326760971 Remove nvram when undefining VM on aarch64 2020-07-27 10:27:15 -05:00
Major Hayden
7f78502046 s390x: wait longer for smoke test check 2020-07-27 10:27:15 -05:00
Major Hayden
0d56d284d9 Work around ppc64le virt issues 2020-07-27 10:27:15 -05:00
Martin Sehnoutka
3fc03503a5 distro/f32: manually relabel cp in buildroot with install_t
By labeling `cp` with `system_u:object_r:install_exec_t:s0` we allow it
to copy labels unknown to the host.

See also corresponding commit in osbuild:
e80130a830
2020-07-22 10:47:46 +01:00
Martin Sehnoutka
636b761b6b distro/f32: add SELinux RPM to build pipeline
This patch adds SELinux stage to build pipeline for f32 just like
we do for RHEL8. This helps with issues like this:
https://github.com/osbuild/osbuild-composer/issues/798
2020-07-22 10:47:46 +01:00
chloenayon
ae3fb5e1c5 distro: enable cloud-init services for openstack and qcow2 images for fedora
Previously, cloud-init was installed on qcow and openstack images but was not
enabled, preventing a user from logging in to these images. Cloud-init, cloud-final,
cloud-init-local and cloud-config services are enabled to make cloud-init work
and to maintain consistency with official fedora 31 and fedora 32 images.

The image tests have been updated to reflect these changes.

Fixes #698
2020-07-18 11:57:05 +02:00
Major Hayden
4b0409df5b Update Fedora 31/32 aarch test cases
Add Fedora 32 test cases with the base OS + updates (to work around
aarch64 bugs in the original release). Add updates for Fedora 31 for
consistency.

Fixes #861.

Signed-off-by: Major Hayden <major@redhat.com>
2020-07-15 09:46:06 +02:00
Major Hayden
41e255bc8c Make qemu integration tests more reliable
Set a static network configuration using cloud-init so that we don't
have to wait a long time for DHCP to work. Also, since we know the IP
and it is set earlier, this reduces the delay in testing on systems
without nested virt (like AWS).

Fixes #844.

Signed-off-by: Major Hayden <major@redhat.com>
2020-07-13 11:19:07 -05:00
Major Hayden
ef0826599d Fix valk > talk typo 2020-07-09 08:53:23 -05:00
Major Hayden
3113ba02d8 Install awscli version 2 for AWS tests
Signed-off-by: Major Hayden <major@redhat.com>
2020-07-09 08:53:23 -05:00
Major Hayden
e88b9c4bd7 🛑 Stop using internal repositories
Remove the F31/F32 internal repositories since they are causing dnf
download errors from time to time.

The 8.2 repository override should have never been there in the first
place since we're using the CDN.

The 8.3 repository is no longer needed since we can register the RHEL
8.3 instances against the staging CDN and get beta content from there.

Fixes #822.

Signed-off-by: Major Hayden <major@redhat.com>
2020-07-09 08:53:23 -05:00
Ondřej Budai
843f7701cf test/azure: don't tag the resources
The tag is not used anymore, see the previous commits.
2020-07-09 12:50:06 +02:00
Ondřej Budai
57b1788c9d test/azure: specify resource names as parameters
Prior this commit the resource names were generated in the deployment
template, so the Go code actually didn't know them. This commit generates
all names in the Go code, so they can be used in the future commits.
2020-07-09 12:50:06 +02:00
Major Hayden
0f32f35756 📦 Use raw image format for AWS
Although the vhdx format is smaller than raw, AWS often has some issues
booting instances from an imported vhdx snapshot. The raw format seems
to boot consistently at AWS.

Fixes #754.

Signed-off-by: Major Hayden <major@redhat.com>
2020-07-02 13:11:11 -05:00
Tom Gundersen
fe9f2c55b8 repositories/rhel-8: default to beta repos
The next release we are targeting is RHEL 8.3 Beta, switch the
repositories over.

In the future, we should distinguish between beta and non-beta based on
/etc/os-release so we don't have to hard-code this upstream.

The beta repositories are not actually available yet, so in our tests,
we have to override this again to point at the GA ones (currently 8.2).

Needless to say, this situation leaves much to be desired.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-29 16:04:38 +02:00
Tom Gundersen
b32f0feeae test/cases: add test-cases to verify customizations
Add one test-case per distro that tries to cover all the customizations
we support.

For now omit firewall customizations, as there are open questions about
how to support that on all image types.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-29 16:04:38 +02:00
Ondřej Budai
0b74ebd2ff test: bump 8.3 repo
The previously used snapshot was deleted. This commit changes the override to
use Internal Snapshot 2.0, which should not be deleted in the near future.
2020-06-29 12:29:20 +02:00
Tom Gundersen
18b17c87fa test/cases/rhel/edge: update with dropped fstab
The previous commit removes fstab, update the test-cases accordingly:

`sudo ./tools/test-case-generators/generate-test-cases --distro rhel-8
--arch x86_64 --image-types rhel-edge-commit --output test/cases/
--store /mnt/osbuild`

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-21 16:16:14 +02:00
Major Hayden
cf2392681a Remove 8.2 test case symlinks
These symlinks are no longer needed now that the testing scripts have
been adjusted.

Signed-off-by: Major Hayden <major@redhat.com>
2020-06-17 11:47:50 +03:00
Major Hayden
5107115e8b Set consistent ssh-keyscan timeout
Fedora's timeout for ssh-keyscan is short by default, but RHEL's is much
longer. Set the timeout to 5 seconds for consistency.

Signed-off-by: Major Hayden <major@redhat.com>
2020-06-16 11:48:45 +03:00