Commit graph

290 commits

Author SHA1 Message Date
Lars Karlitski
b60bb1d011 distro: pass compression argument to the tar assembler 2019-12-13 12:21:47 +01:00
Lars Karlitski
57aea4160d weldr: return version as string from api/status
This is what lorax-composer does and composer-cli expects.
2019-12-13 00:14:40 +01:00
Lars Karlitski
42f46a5992 distro/rhel82: make openstack's package list constistent with lorax 2019-12-13 00:14:07 +01:00
Tom Gundersen
cce58cae8f submodule: update to current master
This contains new API for getting multi-arch to work. We expect more additions
before the next release, but this way we can get things working in parallel.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-12 18:16:01 +01:00
Martin Sehnoutka
b7b60363dc Modify RHEL8.2 VHD image to boot in Azure 2019-12-12 13:52:21 +01:00
Tom Gundersen
d074395474 distro/rhel-8.2: update VHD images to be closer to lorax
The current one was taken from fedora and that failed both due to an
SELinux bug in RHEL8.2, but also because the list of packages was
wrong for the distro.

We don't enable WAAgent, which we probably should. And there are a few
other config tweaks to cloud-init and WAAgent that we skip too.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-12 13:52:21 +01:00
Lars Karlitski
305b2dda5c rpmmd: add IgnoreSSL key to RepoConfig
This is the opposite of dnf.conf's `sslverify`, because go's default for
booleans is always false. This is error prone: we'd like to default to
true.
2019-12-12 01:27:04 +00:00
Lars Karlitski
2b42612336 distro: insert custom sources into pipeline
This is an additional argument on `distro.Pipeline` right now. In the
future, we probably want this per-blueprint.
2019-12-11 21:19:32 +00:00
Lars Karlitski
d9ad5d7062 weldr: add toml output to projects/source/info 2019-12-11 18:53:10 +00:00
Lars Karlitski
72e1da47ab weldr: make projects/source/new accept text/x-toml
composer-cli sends sources as toml if the original format is toml.
2019-12-11 18:53:10 +00:00
Tom Gundersen
7036c157fb travis: disable PPC tests
These seemed not to get triggered (no available workers?), so disable
for now. We do want PPC support, but it is currently our last
priority.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-11 15:23:24 +01:00
Tom Gundersen
04012641ba spec: don't ship devel tools
Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-11 15:23:24 +01:00
Tom Gundersen
d55abdfd75 osbuild-worker: disable the osbuild cache
We still need to figure out the garbage collection, so for now don't
enable the cache.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-11 15:23:24 +01:00
Tom Gundersen
22d57ff1a6 service: don't pull down the socket when the service goes down
The intention is for the sockets always to stay around, even when
the service is restarted. This way the worker can always connect.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-11 15:23:24 +01:00
Tom Gundersen
df73a9bfad service: make sure the worker pulls in the socket
The worker cannot be started unless the socket is available, so make
this a hard requirement.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-11 15:23:24 +01:00
Tom Gundersen
ef6fbe2a25 service: pull in the worker at runtime
Rather than enabling the worker when composer is enabled, make sure
it is started whenever composer is.

This means that one worker instance is always available, and more can
be enabled if required. They are then also started whenever composer
is.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-11 15:23:24 +01:00
Tom Gundersen
118b185fdd osbuild-{composer/worker}: exit cleanly
Only panic on compile-time errors (e.g., built for unsupported
architecture). Otherwise, use log.Fatalf(), which is equivalent to
printing and exiting with return code 1. Only ever do this from
main(), in all other cases pass on the error object.

This is mostly relevant when the server disconects, in which case
we'll get EOF, and will now restart cleanly instead of panicing.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-11 15:23:24 +01:00
Tom Gundersen
42dfbf7da6 travis: also run unittests on all arches
There should be no difference, but do this as a sanity check.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-11 15:23:24 +01:00
Tom Gundersen
d33fc5f010 composer: add provisional multi-arch support
The pipeline generation now takes the architecture as an argument.
Currently only x86_64 is supported. The architecture is detected
at start-up, and passed down to each pipeline translation.

For osbuild-pipeline we now requrie the architecture to be passed
in.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-11 15:23:24 +01:00
Lars Karlitski
dcc9cdedee distro: rename output of qcow2 image to disk.qcow2
Integration tests currently expect that and they need to be able to run
against lorax-composer (which outputs disk.qcow2) and osbuild-composer.
2019-12-10 20:21:47 +00:00
Lars Karlitski
167e09fc54 osbuild-pipeline: display blueprint parse errors 2019-12-10 20:21:47 +00:00
Lars Karlitski
ea70d9dfc5 blueprint: don't set omitempty for toml groups and modules
The integration tests expect the empty arrays to be there.
2019-12-10 20:21:47 +00:00
Lars Karlitski
0718766fbc weldr: fix reply on compose/info route
composer-cli requires `dependencies.packages` to exist. We can still
leave it empty for now, because it doesn't look at the contents.
2019-12-10 20:21:47 +00:00
Lars Karlitski
b7dac20c74 weldr: add test for blueprint/depsolve route 2019-12-10 20:38:22 +01:00
Lars Karlitski
280a7c3950 distro/rhel82: use latest nightly
Now that osbuild-composer is inserting repository checksums dynamically,
we can use a URL that points to a changing repository.
2019-12-10 20:38:22 +01:00
Lars Karlitski
d3a0b788a2 distro: set the repository checksum dynamically
Instead of having a static repository checksum, set it dynamically from
the metadata that osbuild-composer last saw. This is implemented in
dnf-json, which returns the checksums for each repository on every call.

This enables the use of repositories that change over time, such as
fedora-updates. Note that the osbuild pipeline will break when such a
repository changes. This is intentional: pipelines have to be
reproducible.
2019-12-10 20:38:22 +01:00
Lars Karlitski
75218ad2d9 distro/fedora30: don't include variables in repo URL
Replace those variables with their contents everywhere the repository
URLs are used is error-prone.
2019-12-10 20:38:22 +01:00
Tom Gundersen
0d091decb5 test: add provisional multi-arch support
Currently we still only build for x86_64, but now the test suite is
prepared for hooking up other architectures.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-10 02:47:35 +01:00
Tom Gundersen
9bedc0c393 test/run: allow filter on distro
This way we can test the distros on their respective CI, as not
all distros can be built in all environment. In particular RHEL
needs to be on a subscribed host.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-10 02:47:35 +01:00
Tom Gundersen
34a4f24141 test/run: print a bit more useful info on test-runs
Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-10 02:47:35 +01:00
Tom Gundersen
8c68018d62 test/run: make relative OSBUILD_TEST_STORE consistent
Make it relative to the directory invoking ./run, rather than passing it along.
2019-12-10 02:47:35 +01:00
Tom Gundersen
3adcac0123 test/run: merge boot and non-boot test handling
We can now select specific cases, but whether or not to check image-info
or boot the image is determined purely by the contents of the json test
case.

We still run the tests as two travis workers just to avoid the timeout,
this should clearly be reworked.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-10 02:47:35 +01:00
Tom Gundersen
c090ab3812 test/run: drop support for non-pipeline test cases
We were supporting downloading an image and checking its image info. We don't
want to rely on external resources, and we should not test images made by others.

Drop it.
2019-12-10 02:47:35 +01:00
Tom Gundersen
cc04f92997 test/cases: rename json sections
Rename to fit more closely with their use before we add too many more
of these.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-10 02:47:35 +01:00
Tom Gundersen
6c44d231ee test/run/nspawn: name the instances differently
We were failing on name reuse. We should be able to not name these
at all, but nspawn is not happy with that.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-10 02:47:35 +01:00
Tom Gundersen
82c156ef1b test/cases: boot test the empty blueprint ami image
There should be no need to create a custom blueprint, even the empty one
should boot.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-10 02:47:35 +01:00
Tom Gundersen
a7203db963 test/run/qemu: support compressed images
Uncompress, strip the extension and pass to qemu as usual.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-10 02:47:35 +01:00
Tom Gundersen
404ad2023d test/run: wait on is-system-running
Once we are connected, wait for the command to return a definitive
result, rather than retry.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-10 02:47:35 +01:00
Tom Gundersen
ad7c5bc735 test/run: improve output on ssh retry
We expect to have to retry the ssh connection, so let's print something
less scary when that happens.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-10 02:47:35 +01:00
Tom Gundersen
4ec7ac1ecd test/cases: start to unify boot tests with other image tests
We now have three top-level maps, that can be combined in any way:

boot-test: information about how to boot the image
compose: information about how to generate the pipeline
pipeline: the pipeline to generate the image
expected: the expected image-info

This creates compose entries for all the boot tests, but the blueprints
are named 'blueprint-draft', as we are not yet verifynig that the pipeline
is correct.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-10 02:47:35 +01:00
Tom Gundersen
49387604e4 test/run: pass cloud-init data to qemu
This passes the redhat user with ssh key as an ISO image to our
qemu instances, making sure images relying on cloud-init rather than
hardcoded user credentials can be used in our tests.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-10 02:47:35 +01:00
Tom Gundersen
59ab975fea test: use the standard port for sshd
For the qemu tests this makes no difference as we are anyway forwarding
the ports. But the nspawn tests share the same network namespace between
the image and the ssh client running the test without any forwarding. In
order for that to work we had to modify the image to use a non-standard
port.

We don't want this for two reasons: we want to make sure we test our images
unmodified, and also this meant that when we changed our pipeline generation
we were not verifying that the boot test cases were updated accordingly. As
a result they have now drifted.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-10 02:47:35 +01:00
Tom Gundersen
79b868d423 test/run: run each test in its own network namespace
This makes sure we can use any port to connect to sshd, without
worrynig about clashes with parallel tests, or an sshd instance
running on the host.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-10 02:47:35 +01:00
Ondřej Budai
6bbc89d5f3 api: implement /compose/delete route 2019-12-10 01:14:36 +01:00
Ondřej Budai
6d15833e4e store: introduce state directory
Prior to this commit outputs directory used by local target was owned by root.
This made impossible for osbuild-composer to delete images. (osbuild-composer
doesn't run as root).

This commit introduces state directory in which osbuild-composer creates
outputs directory. Because this directory is owned by osbuild-composer, it's
able to delete files inside.
2019-12-10 01:14:36 +01:00
Christian Kellner
0cf8a20744 image-info: mount ESP at /boot/efi if found
If we detect a ESP (via its type UUID) remember it and mount it at
any filesystem that has a /boot/efi (there should in theory be
only one). This is needed so grubenv can be read, which is likely
a link from /boot/grub2/grubenv to ../efi/EFI/$vendor/grubenv.
Additionally this will make rpm verify not report that all the efi
binaries are missing, e.g.:
"missing": [
      "/boot/efi/EFI",
      "/boot/efi/EFI/BOOT",
      "/boot/efi/EFI/BOOT/BOOTIA32.EFI",
      "/boot/efi/EFI/BOOT/BOOTX64.EFI",
      "/boot/efi/EFI/BOOT/fbia32.efi",
      "/boot/efi/EFI/BOOT/fbx64.efi",
      "/boot/efi/EFI/fedora",
      "/boot/efi/EFI/fedora",
      ...
2019-12-09 18:08:52 +01:00
Ondřej Budai
4e62f181fa api: implement /compose/logs route
The implementation is just a stub returning always the same tar archive.
The ability to return actual logs will be implemented in the future - osbuild
isn't currently returning any logs.
2019-12-08 00:21:30 +01:00
Ondřej Budai
d7f81b36e6 api: implement /compose/info route
Not everything is implemented in this route but enough to pass the tests.
2019-12-06 10:42:19 +01:00
Ondřej Budai
ad7650384e api: use committed blueprint in POST /compose
lorax-composer apparently doesn't use blueprint from workspace to create a new
compose. This commit implements the same behaviour.
2019-12-05 13:08:50 +01:00
Ondřej Budai
24faab8c18 store: clean up blueprint API
Prior to this commit blueprint getters looked like C-style API with output
parameters. This commit refactors them to more conventional multiple return
values API.
2019-12-05 13:08:50 +01:00