Commit graph

1209 commits

Author SHA1 Message Date
Tom Gundersen
50d469fe45 distro: replace BasePackages() with Packages()
Rather than getting a set of base packages from the ImageType, and then
appending the requested packages from the blueprint, pass the blueprint
into the new Packages() function, and return the full set of packages to
be depsolved.

This allows us to also append packages based on other customizations
too, and use that to append chrony when the timezone is set. This
matches the behavior anaconda had, and there was a TODO item to do this,
which had been overlooked.

Fixes #787.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-29 16:04:38 +02:00
Tom Gundersen
d31e3ebb65 distro: add groups before users
If a user is added to a group, then the group must be added first.
2020-06-29 16:04:38 +02:00
Ondřej Budai
0b74ebd2ff test: bump 8.3 repo
The previously used snapshot was deleted. This commit changes the override to
use Internal Snapshot 2.0, which should not be deleted in the near future.
2020-06-29 12:29:20 +02:00
Ondřej Budai
fc2788340f worker: set the osbuild success to false even on non-osbuild errors
composer uses the success field to decide whether a build succeeded or failed.
This is bad.

Unfortunately, fixing this requires kinda big code changes. This commit
changes the worker's behaviour to set the osbuild success flag to false
even on errors which weren't caused by osbuild (e.g. an upload error).
This is certainly hacky but I think it's still essential to tell the user
that an error occurred.

Fixes #789
2020-06-29 10:21:24 +02:00
Ondřej Budai
297dbe2fc7 worker: move nil result check to more appropriate place
Result can be nil only when there's an error. Move the code to a place where
it makes more sense.
2020-06-29 10:21:24 +02:00
Alexander Todorov
66b80e0dac tests: Add reproducer for #524
see this comment:
https://github.com/osbuild/osbuild-composer/issues/524#issuecomment-632017769

depending on the answer we should probably add more tests for
dnf-json too.
2020-06-26 22:02:46 +02:00
Martin Sehnoutka
607b4ed935 dnf-json: change confusing error message associated with dnf errors
The issue was introduced in 0d3c8329c0.
The patch correctly changed the base exception class, but it didn't
change the unfortunate use of hardcoded type name. This patch uses
Python's internal `__name__` attribute to get the type (exception) name.
2020-06-26 20:36:35 +02:00
Major Hayden
067726e91d Print success log line after job is done
We print messages to the log when the build fails, but we don't print
one when the build is successful.

I really want to celebrate our successes more often, so let's print a
success message when osbuild completes successfully. Since success
doesn't feel like success without an emoji, let's add one of those, too. 🎉

Signed-off-by: Major Hayden <major@redhat.com>
2020-06-26 20:30:01 +02:00
Christian Kellner
c3c78c2d0c image-info: use /var/tmp for bigger files/dirs
When extracting the commit tarball as well as doing the ostree
commit checkout, use `/var/tmp` instead of `/tmp`, because the
latter might be backed by a tmpfs and thus by RAM; this should
only be used for small files.
2020-06-26 20:25:21 +02:00
Alexander Todorov
6d7181e832 osbuild-composer-tests now Requires osbuild-composer-rcm 2020-06-24 15:14:47 +02:00
Alexander Todorov
5bf3af559f Shutzbot: install only osbuild-composer-tests
in order to catch missing dependencies
2020-06-24 15:14:47 +02:00
Alexander Todorov
e2619221cf CI: print the list of RPMs on the system
this will make it easier to figure out what was there before
installing SUT and help in debugging issues related to dependencies.
2020-06-24 15:13:51 +02:00
Major Hayden
bfad57bf7b CI: Retry osbuild-composer-tests installation
Reduce CI failures by trying to install osbuild-composer-tests five
times.

Fixes #784

Signed-off-by: Major Hayden <major@redhat.com>
2020-06-23 16:19:20 +02:00
Alexander Todorov
f7c4dca5d5 tests: upload & boot image in OpenStack. Closes #339 2020-06-23 16:17:30 +02:00
Tom Gundersen
18b17c87fa test/cases/rhel/edge: update with dropped fstab
The previous commit removes fstab, update the test-cases accordingly:

`sudo ./tools/test-case-generators/generate-test-cases --distro rhel-8
--arch x86_64 --image-types rhel-edge-commit --output test/cases/
--store /mnt/osbuild`

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-21 16:16:14 +02:00
Christian Kellner
408a704669 distro/{rhel8, fedora32}: no fstab for OSTree
How exactly the final file system layout looks like is determined
by the installer (anaconda) and thus can not be known at commit
creation time. Thus creating an /etc/fstab file is unnecessary and
the information in it probably wrong. The file wont be used though
because it will be overwritten during the installation process.
2020-06-21 16:16:14 +02:00
Martin Sehnoutka
7be8cefab9 tools: merge generate-test-case{,s} scripts
The fact that we have two scripts and one captures output of the other
makes it hard to use in scenarios where the test case generation often
fails. This output capturing results in subtle bugs like described in
issue: https://github.com/osbuild/osbuild-composer/issues/769

This patch merge these two scripts and use class to organize it
instead of files.
2020-06-19 16:36:16 +02:00
Major Hayden
4da9c3389b Pretty print osbuild JSON on failure
When osbuild fails, osbuild-composer's image tests print the output as
ugly JSON and that makes it difficult to determine why osbuild failed.

Instead, print the json in pretty format. 🦋

Fixes #782.

Signed-off-by: Major Hayden <major@redhat.com>
2020-06-18 08:46:19 -05:00
Martin Sehnoutka
282763a977 image-types: link the Azure requirements
This patch makes it easier to find the right document describing image
requirements for Linux VM images. It does not link the RHEL page
specifically because it is mainly for RHEL 6 and 7 whereas we build RHEL
8. It instead links documentation for generic Linux distribution.
2020-06-17 20:42:17 +02:00
Major Hayden
cf2392681a Remove 8.2 test case symlinks
These symlinks are no longer needed now that the testing scripts have
been adjusted.

Signed-off-by: Major Hayden <major@redhat.com>
2020-06-17 11:47:50 +03:00
Major Hayden
5107115e8b Set consistent ssh-keyscan timeout
Fedora's timeout for ssh-keyscan is short by default, but RHEL's is much
longer. Set the timeout to 5 seconds for consistency.

Signed-off-by: Major Hayden <major@redhat.com>
2020-06-16 11:48:45 +03:00
Major Hayden
989e15e2fb CI: Fix copy/paste errors in log gathering
Perhaps my ability to copy and paste code should be taken away. 🤦🏻‍♂️

Signed-off-by: Major Hayden <major@redhat.com>
2020-06-15 15:50:40 -05:00
Ondřej Budai
5991666b8a spec: make the subpackages require a matching version of composer
The subpackages are not really meant to be used with a different version of
the osbuild-composer package. This commit enforces.the usage of a matching
version in the spec file.
2020-06-15 07:18:31 -05:00
Ondřej Budai
6bc200df14 15
Release osbuild-composer version 15
2020-06-12 14:15:49 +02:00
Ondřej Budai
ab0a8057bf worker: ensure that the reported result is always non-nil
When osbuild crashes (e.g. when cp fails because of the machine running
out of disk space), it doesn't produce a machine-readable result. Due to
our suboptimal handling of the result struct (this is my fault), this can
lead to result == nil. However, composer expects that result != nil in all
cases because it uses the Success flag to assess the compose state. If
result == nil, it just crashes terribly.
2020-06-12 12:47:31 +02:00
Brian C. Lane
4f3f09fec5 worker: Check OSBuildOutput for nil before using it
It is possible for it to end up as nil so it needs to be checked
everywhere it is used.
2020-06-12 12:45:46 +02:00
Ondřej Budai
4958d15413 tests: build qcow2 instead of tar
Tar is only available on RHEL, switch to qcow2, that's everywhere.
2020-06-12 10:00:50 +02:00
Lars Karlitski
40b65144d1 osbuild-tests: add compose cancel test 2020-06-12 10:00:50 +02:00
Lars Karlitski
aa0c037bb2 osbuild-worker: support canceling jobs
Exit the whole worker process when a job was canceled, because osbuild
does not clean up all child processes when receiving SIGKILL.

Change the service to restart osbuild-worker also on success, and
decrease the restart timeout.
2020-06-12 10:00:50 +02:00
Lars Karlitski
be10efeb1e weldr: implement cancel route 2020-06-12 10:00:50 +02:00
Lars Karlitski
27acd03b68 worker: add API support for canceling jobs 2020-06-12 10:00:50 +02:00
Ondřej Budai
b759f4e0be Fix RHEL 8.3 repo override
BaseOS -> AppStream, see the diff
2020-06-12 09:59:25 +02:00
Major Hayden
4ca535c673 Add RHEL 8.3 repository override
Override the repository for RHEL 8.3 since they can't be registered yet.

Signed-off-by: Major Hayden <major@redhat.com>
2020-06-12 09:59:25 +02:00
Alexander Todorov
6d904f062c tests: composer-cli blueprints undo. Closes #615
the remaining test for composer-cli compose cancel is part of #757
2020-06-12 06:30:54 +02:00
Ondřej Budai
37bc80b31d image-tests: allow disabling of local booting
On some environments (like RHEL gating) there's no virtualization available.
This commit adds -disable-local-boot argument to osbuild-image-tests. When
this argument is present, the local booting is skipped. This doesn't affect
the cloud booting, the test binary still tries to do that. If no credentials
are available, the fall back to local booting will be skipped if
-disable-local-boot is given.
2020-06-12 06:29:32 +02:00
Lars Karlitski
33a4c55a6f osbuild-worker: don't use /var/cache for temporary directories
When osbuild-composer crashed, it left temporary directories in
`/var/cache`. Use `/var/tmp` for these output directories, because
systemd will clean these up (we set PrivateTmp=true).

Also, put the store into `/var/cache/osbuild-store`. The worker does not
checkpoint anything. The store is only used as a cache for rpms. That
can be shared between multiple workers and successive runs of a single
worker.
2020-06-12 00:13:37 +02:00
Major Hayden
89117ebf3a Deploy quickly in CI without Ansible
Speed up the deployment during CI runs and avoid the need for lots of
extra packages to run Ansible. This gets us closer to running bare
images for each distro that better simulates a customer environment.

Signed-off-by: Major Hayden <major@redhat.com>
2020-06-11 15:58:30 -05:00
Major Hayden
d607365663 Fix image test VERSION_ID
On RHEL, `VERSION_ID` shows up as `8.2` or `8.3`, but we need it to be
just `8`.

Remove the period and everything after it in `$VERSION_ID`.

Signed-off-by: Major Hayden <major@redhat.com>
2020-06-11 15:27:06 -05:00
Martin Sehnoutka
1e5c31a7ad distro/rhel8: introduce s390x support
This patch introduces support for s390x architecture. It has some
specific requirements like using kernel-cmdline stage before the RPM
stage because RPM scriptlets require these values to be present in the
system already. It also uses zipl stage and bootloader as opposed to all
the other architectures.
2020-06-11 21:50:09 +02:00
Martin Sehnoutka
d20682fcae osbuild: introduce kernel-cmdline stage
This stage can be used to set kernel boot parameters.
2020-06-11 21:50:09 +02:00
Tom Gundersen
fcd5ff76ac tools/image-info/ostree: don't record the commit sha
This is not stable between builds, instead, record the inputhash, which
is.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-11 20:44:17 +02:00
Tom Gundersen
2dfa0260a6 schutzbot: run image tests on RHEL for Edge
Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-11 20:44:17 +02:00
Tom Gundersen
3925b3f6eb test/cases/rhel8: add RHEL for Edge image test case
We currently don't have a boot-test for this, it must be performed
manually.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-11 20:44:17 +02:00
Tom Gundersen
4fe8bb6baf test/cases/rhel8/tar: add image info
image-info now supports tar-balls, so add this to the test-case.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-11 20:44:17 +02:00
Tom Gundersen
dec644dec4 tools/test-case-generation: don't skip image info for tar
This is now supported, so default to include tar image info.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-11 20:44:17 +02:00
Tom Gundersen
be3541335e tools/image-info: drop a stray print()
This resulted in invalid JSON on stdout.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-11 20:44:17 +02:00
Tom Gundersen
654eb9d280 tools/test-case-generator: add iot and edge test-case generation 2020-06-11 20:44:17 +02:00
Tom Gundersen
c51de68301 tools/test-case-generator: make image-types a required argument
Also allow a list of them to be passed. Generating 'all' image types
makes less sense now that different distros/arches support different
ones.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-11 20:44:17 +02:00
Peter Robinson
f32770fed6 distro/rhel8: add RHEL for Edge
Initial definition for RHEL for edge package set on x86.
This is based marginally on the upstream Fedora IoT package set
but slimmed down and adjusted for RHEL package naming.

Based on Tom's patch.

Signed-off-by: Tom Gundersen <teg@jklm.no>
Signed-off-by: Peter Robinson <pbrobinson@gmail.com>
2020-06-11 20:44:17 +02:00
Major Hayden
d9c7c2dd0b Convert image tests to bash
Simplify the image tests with bash and allow the output to appear live
in the Jenkins interface.

Signed-off-by: Major Hayden <major@redhat.com>
2020-06-11 13:34:36 -05:00