Previously, Koji instance could be both logged-in and not logged-in.
This change disallows it: Now, the Koji instance is created by calling
koji.Login, so it must be always logged-in. This change should lead to more
robust code.
In the same way `osbuild.Manifest` is the input to the osbuild API,
`osbuild.Result` is the output. Move it to the `osbuild` package where
it belongs.
This is not a functional change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The tests are no longer run on Travis, therefore we don't need the special
setup to run them there.
This change should also fix#929 that is probably caused due to osbuild
executed in a weird way.
Fixes#929
vCenter requires images to be uploaded as vmdk StreamOptimized. Lorax
always produced images on this format, so we should make sure to do the
same for our VMWare images.
Allow LocalTarget to request the images produced by osbuild be converted
to be streamOptimized before saving in composer, and hook the weldr API
up to enable this option for vmdk images.
Ideally this should simply be an option in osbuild, but that would
require some more work, which we will not manage in time for RHEL8.3.
Therefore do this minimal fix.
Note that that means the images produced by our manifests (including in
our image-test test cases) are not on the format that the weldr API
returns, so the tests we run on them would also, for now, need to
convert before uploading to vCenter.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is not shipped in RHEL, so use the library directly to query the IP
address. This is a massive hack, but let us revisit this after the next
release.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We need the same RPMs to work equally well on a host running a beta
release (pulling beta content) as on a machine running GA (pulling GA
content). Detect this at run-time and point at the right repository.
Testing this is a bit hairy as we are building 8.3 images, but obviously
there is currently no 8.3 content at the GA URLs.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The osbuild-composer-rcm package was never finished, not in use and will be replaced by osbulid-composer-koji.
Signed-off-by: Tom Gundersen <teg@jklm.no>
`systemd-tmpfiles` will helpfully delete "old" files in /var/tmp at regular
intervals. The files installed from rpm has the timestamps from when they
were packaged, which causes some to be cleaned up when the timer triggers.
The first timer triggers 15 minutes after boot, so we were sometimes hit
by this when our CI was under load.
Fixes#839 and #862.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Create a GetImageInfoCommand analogous to GetOsbuildCommand that
will adjust the PYTHONPATH for image-info on travis so that the
osbuild python module is accessible.
Previously, all resouces were created with a certain tag. When the cleanup
phase came, the Resources - List route[1] was used to get all resources with
the tag. Then, they were deleted in the right order.
Sadly, the Resources - List API has issues with listing disks. Sometimes,
it returns the virtual disks 15 minutes after they were created. As the
result, the disks have been left behind quite often and our bill was higher
than necessary.
This commit uses a different method - the Go code now knows all resource names
(see the previous commit), so it can delete all resources without listing them
using the "broken" API route.
[1]: https://docs.microsoft.com/en-us/rest/api/resources/resources/list
Prior this commit the resource names were generated in the deployment
template, so the Go code actually didn't know them. This commit generates
all names in the Go code, so they can be used in the future commits.
Rather than getting a set of base packages from the ImageType, and then
appending the requested packages from the blueprint, pass the blueprint
into the new Packages() function, and return the full set of packages to
be depsolved.
This allows us to also append packages based on other customizations
too, and use that to append chrony when the timezone is set. This
matches the behavior anaconda had, and there was a TODO item to do this,
which had been overlooked.
Fixes#787.
Signed-off-by: Tom Gundersen <teg@jklm.no>
composer uses the success field to decide whether a build succeeded or failed.
This is bad.
Unfortunately, fixing this requires kinda big code changes. This commit
changes the worker's behaviour to set the osbuild success flag to false
even on errors which weren't caused by osbuild (e.g. an upload error).
This is certainly hacky but I think it's still essential to tell the user
that an error occurred.
Fixes#789
We print messages to the log when the build fails, but we don't print
one when the build is successful.
I really want to celebrate our successes more often, so let's print a
success message when osbuild completes successfully. Since success
doesn't feel like success without an emoji, let's add one of those, too. 🎉
Signed-off-by: Major Hayden <major@redhat.com>
When osbuild fails, osbuild-composer's image tests print the output as
ugly JSON and that makes it difficult to determine why osbuild failed.
Instead, print the json in pretty format. 🦋Fixes#782.
Signed-off-by: Major Hayden <major@redhat.com>
When osbuild crashes (e.g. when cp fails because of the machine running
out of disk space), it doesn't produce a machine-readable result. Due to
our suboptimal handling of the result struct (this is my fault), this can
lead to result == nil. However, composer expects that result != nil in all
cases because it uses the Success flag to assess the compose state. If
result == nil, it just crashes terribly.
Exit the whole worker process when a job was canceled, because osbuild
does not clean up all child processes when receiving SIGKILL.
Change the service to restart osbuild-worker also on success, and
decrease the restart timeout.
On some environments (like RHEL gating) there's no virtualization available.
This commit adds -disable-local-boot argument to osbuild-image-tests. When
this argument is present, the local booting is skipped. This doesn't affect
the cloud booting, the test binary still tries to do that. If no credentials
are available, the fall back to local booting will be skipped if
-disable-local-boot is given.
When osbuild-composer crashed, it left temporary directories in
`/var/cache`. Use `/var/tmp` for these output directories, because
systemd will clean these up (we set PrivateTmp=true).
Also, put the store into `/var/cache/osbuild-store`. The worker does not
checkpoint anything. The store is only used as a cache for rpms. That
can be shared between multiple workers and successive runs of a single
worker.
This makes it easier to use the test binary with the `-run` argument.
Instead of the full path:
-test.run TestImages//usr/share/tests/osbuild-composer/cases/rhel_8.2-x86_64-openstack-boot.json
this only requires the actual name:
-test.run TestImages/rhel_8.2-x86_64-openstack-boot.json
When edd7b37ea added `--output-directory` to the invocation of osbuild,
it also removed `--store`.
This was a mistake: osbuild's default store is `.osbuild`, which is not
what we want. Restore the old behavior of passing a temporary directory,
but use the same for each test run.
Treating stdout and stderr separately makes it hard to match what
happened when. It's also easy to miss when `-v` is passed to the test
binary.
Print the output to stdout when osbuild fails, because the test
framework we're using does not print errors if they're too large.
Also, don't special-case exec.ExitError. Output might be useful in any
case.
Previously, all the osbuild-composer tools must be run from a directory with
dnf-json. This was often confusing, especially with the dnf-json-tests. This
commit changes the path to be absolute, so this is no longer an issue.
RPMMD had hardcoded path to dnf-json helper. This required all executables
using RPMMD to be run in the directory where dnf-json was located. This commit
makes RPMMD take the path to dnf-json as an argument. This allows its
consumers to specify whichever path they want.
Not a functional change
qcow2, unlike ami, is guaranteed to exist on all the architectures we
support.
Also, now that we preserve the cache, make two composes, rather than
just one. This verifies that nothing breaks horribly just because the
cache not being cold. The fact that we have a cache at all should make
this tolerably fast.
This should fix#693.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Schutzbot currently runs all the same tests an all arches/distro
combinations. This will not work as we introduce image types only on
some distro/arches.
In the future we should make the image-test binary more clever, so
Schutzbot won't have to tell it which cases to run at all. For now,
simply don't fail if the specified test-case does not exist.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is not a functional change, just helps to make the output more
readable to indicate which sub-command is being tested.
Signed-off-by: Tom Gundersen <teg@jklm.no>
most invocations of composer-cli do not need to parse JSON so use
the canonical invocation instead. I've left existing functions
which assert on the returned results intact.
'quiet' parameter has been removed b/c it was never used
Until osbuild-14, the images were unconditionally kept in the cache,
meaning the cache could grow very large. Now only the downloaded RPMs
are saved, which greatly limits how big it can grow.
Having the RPMs cached should speed up all but the first image build a
lot, so we should take advantage of that by not flushing the cache
between each build.
The cache is still flushed when the worker is stopped / restarted.
This moves the cache from /var/tmp/osbulid-worker* to
/var/cache/osbulid-worker/osbulid-worker-*. This means that each worker
gets a dedicated cache, in case there are several on one machine. In the
future we may want to combine them and only ever have one cache, but for
that we need improvements in parallel access and cache-cleanup.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This commit changes the store.PushSource function to take the key as
well as the SourceConfig so that it can be used for v0 or v1.
It adds helper functions for decoding the toml/json into a new
SourceConfig interface type which lets the core source/new code be
shared between the versions.
It also adds tests for the new API behavior.
The new tool osbuild-store-dump saves store.json to the current working
directory, with more or less arbitrary data in it.
This has been executed on osubild-composer-{12,13} (mutatis mutandis),
and the results are saved in `internal/store/test`. A new test is added
which loads these stores and does very basic verification on them having
been loaded correctly.
This is mostly meant to catch regressions that means old stores are able
to make composer crash, or lose all its data. It would not catch minor
errors that leave the stores syntactically correct.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Rather than Manifest() returning an osbuild.Manifest object, introduce a
new distro.Manifest object which represents it as an opaque, JSON
serializable object. This new type has the following properties:
1) its serialization is compatible with the input to osbuild,
2) any valid osbuild input can be deserialized into it, and
3) marshalling and unmarshaling to and from JSON is lossless.
This means that even as we change the subset of valid osbulid manifests
that we support, we can still load any previous state from disk, and it
will continue to work just as before, even though we can no longer
deserialize it into our internal notion of osbuild.Manifest.
This fixes the underlying problem of which #685 was a symptom.
Signed-off-by: Tom Gundersen <teg@jklm.no>
