Rather than getting a set of base packages from the ImageType, and then
appending the requested packages from the blueprint, pass the blueprint
into the new Packages() function, and return the full set of packages to
be depsolved.
This allows us to also append packages based on other customizations
too, and use that to append chrony when the timezone is set. This
matches the behavior anaconda had, and there was a TODO item to do this,
which had been overlooked.
Fixes#787.
Signed-off-by: Tom Gundersen <teg@jklm.no>
How exactly the final file system layout looks like is determined
by the installer (anaconda) and thus can not be known at commit
creation time. Thus creating an /etc/fstab file is unnecessary and
the information in it probably wrong. The file wont be used though
because it will be overwritten during the installation process.
This patch introduces support for s390x architecture. It has some
specific requirements like using kernel-cmdline stage before the RPM
stage because RPM scriptlets require these values to be present in the
system already. It also uses zipl stage and bootloader as opposed to all
the other architectures.
Initial definition for RHEL for edge package set on x86.
This is based marginally on the upstream Fedora IoT package set
but slimmed down and adjusted for RHEL package naming.
Based on Tom's patch.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Signed-off-by: Peter Robinson <pbrobinson@gmail.com>
Allow individual test-cases or sub-sets of test-cases to be generated
more easily.
We allow explicit skipping of image-info generation (and hence the
osbuild run), and also individual image types to be specified.
Also drop distros and image types that are no longer supported.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We require the right SELinux labels to be set on some of the tools we
use, make sure the policy is installed and the labels applied in the
buildroot, in the same way as in the target image.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Rather than passing an array of checksums, pass an array of objects
with package properties. One is the `checksum`, which is required, and
in addition to that we now pass the boolean check_gpg, which indicates
that that RPM should have its GPG signature verified.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Currently each repo is annotated with this information, use the mapping
we have from packages to their originating repos to annotate each
PackageSpec with the same information.
This information is not serialized, and not yet used. It may be used to
expose this information to osbuild's org.osbuild.rpm stage.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Use the new dnfPackageSpec only for communicating with dnf-json. This
allows us to drop the RepoID from the public type, as that is only meant
to be used internally in `Depsolve()`.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The sources weldr API already supports this, so hook it up to be
represented on disk and in our internal state tracking too.
This does not yet hook this up to be respected by osbuild, which
currently takes this to be unconditionally set to true.
Signed-off-by: Tom Gundersen <teg@jklm.no>
When gsl with version * was specified in the blueprint,
composer depsolved both x86_64 and i686 version of gsl.
This test case should prevent this from happening.
gsl is used because it has x86_64 and i686 versions on both RHEL and Fedora.
Also, gsl-devel package exists, which is not dependant on gsl and shouldn't
be depsolved.
blueprint.GetPackages() method was used to depsolve blueprints prior the
dnf-to-rpm switch. However, it got dropped during the switch. This commit
makes weldr use it again.
The nice effect of this change is that we can drop getPkgNameGlob function and
have only one function for getting package name-versions from a blueprint.
Also, blueprint.GetPackages() works better with * version. Previously, we had
issues with composer depsolving bash of version * to both x86_64 and i686
versions of bash package. GetPackages() converts the package to name-version
of just bash, which dnf-json correctly depsolves to just one architecture. On
the contrary, the previous method converted bash to name-version bash-*.*.*,
which confused dnf-json.
Note that conversion to bash-* is also wrong because this will cause dnf-json
to install all packages with prefix "bash-*".
RHEL doesn't have python3 in PATH by default. Instead it has /usr/libexec/
platform-python. When building the RPM, RHEL correctly mangles dnf-json's
shebang to the platform python. This commit switches osbuild-composer
to use this shebang because it should always respect target platform's
default python binary.
Fixes#745
RPMMD had hardcoded path to dnf-json helper. This required all executables
using RPMMD to be run in the directory where dnf-json was located. This commit
makes RPMMD take the path to dnf-json as an argument. This allows its
consumers to specify whichever path they want.
Not a functional change
This is a very minimal image type, which is not expected to do anything
with an empty blueprint. It is documented as a way to install RHEL for
Satellite.
This fixes#720.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The input options of qemu assembler do not require filesystem for every
partition. This patch makes it optional and omits the object in case it
is empty.
osbuild schema definition:
162797f206/assemblers/org.osbuild.qemu (L63)
Partition type, as defined in osbuild qemu assembler, is of type string
because it is either UUID or identifier. See here for details:
162797f206/assemblers/org.osbuild.qemu (L88)
Change type of the "Type" struct member to match the definition in
osbuild. This patch is needed for ppc64le support.
This is needed for lorax parity. When multiple blueprints are being frozen in
toml mode, the API returns an error. This is the same behaviour as in the
/blueprints/info route.
Fixes#667
The AWS API always uses pointers to strings everywhere, rather than
plain strings. The log messages got this wrong and printed the pointers
rather than dereferencing them.
Don't require users to do a memory dump to get at the AMIs, but instead
print the actual strings. Also change to %s from %v to let the linter
warn us in the future. %s accepts only strings, whereas %v prints
whatever you throw at it.
Signed-off-by: Tom Gundersen <teg@jklm.no>
When ImportSnapshot fails, it goes into the deleted state. However, the code
didn't consider this state as a failure. This commit fixes that.
Fixes#706
It was passing it through to the non-system delete function
and not returning an error. This checks for system repos first and
returns a 400, SystemSource error response if it is in the system list.
This changes store.DeleteSource to DeleteSourceByName for v0 use and
DeleteSourceByID for v1 usage.
It includes a new client function DeleteSourceV1, adds a new test, and
converts the tests for the previous Source V1 API commits to use
DeleteSourceV1.
This commit changes the store.GetAllSources to distinguish between
getting the source by the Name field, or by the ID (the key to the map)
using GetAllSourcesByName and ...ByID.
SourceConfig.RepoConfig() now takes an id parameter because SourceConfig
only stores the Name, not the ID.
In weldr I split the sourceInfoHandler into 2 separate functions for v0
and v1 behavior, with the core of the old function refactored as
getSourceConfigs and used by both of them.
This also adds new structs for the SourceResponseV0 and SourceResponseV1
as well as helper functions for converting to/from store.SourceConfig
This commit changes the store.PushSource function to take the key as
well as the SourceConfig so that it can be used for v0 or v1.
It adds helper functions for decoding the toml/json into a new
SourceConfig interface type which lets the core source/new code be
shared between the versions.
It also adds tests for the new API behavior.
This is the first patch in a series to add APIv1 support to the
/projects/source routes. The change involves using the store.Sources key
in a different way (as an id instead of as a duplicate of the struct's
Name field) but does not actually involve changing the Sources json in
the store.
In the V0 API the name of the source was used as the identifier, and
there was no short id. In V1 the source is identified by the API using
a short id, and the Name is just a field in the struct to describe the
source. This will become more obvious with the /projects/source/info
response.
This commit changes the following:
Changes store.ListSources to ListSourcesByName and explicitly pulls the
name from the source struct instead of the key. v0 will use this
function call.
Adds store.ListSourcesById which returns the source key as the
identifier. This is used by v1.
Adds a new weldr.SourcesListV1 response type, even though it is exactly
the same as the V1 response in this specific case. I thought it would be
better to have one called V1 than to reuse the V0 struct and possibly
confuse people.
The /projects/source/list API now lists the sources by name for v0 and id for v1.
A test has been added. You will notice it still uses v0 to push and
delete the sources. These will be updated when the new version of the
functions are added in subsequent commits.
The AWS upload process takes time and has multiple steps that can fail
or take an extended time to run.
Add some basic logging to various parts of the AWS upload process to
assist with troubleshooting in CI and production.
Signed-off-by: Major Hayden <major@redhat.com>
Workers don't report status for the osbuild run and the upload targets
separately. Before the move to the jobqueue, we explicitly set the
status of all targets when a compose finished. When I removed that,
the image status broke.
Set the status from what's returned by api.getComposeStatus() to restore
the original behavior.
Fixes#702
The new tool osbuild-store-dump saves store.json to the current working
directory, with more or less arbitrary data in it.
This has been executed on osubild-composer-{12,13} (mutatis mutandis),
and the results are saved in `internal/store/test`. A new test is added
which loads these stores and does very basic verification on them having
been loaded correctly.
This is mostly meant to catch regressions that means old stores are able
to make composer crash, or lose all its data. It would not catch minor
errors that leave the stores syntactically correct.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Rather than Manifest() returning an osbuild.Manifest object, introduce a
new distro.Manifest object which represents it as an opaque, JSON
serializable object. This new type has the following properties:
1) its serialization is compatible with the input to osbuild,
2) any valid osbuild input can be deserialized into it, and
3) marshalling and unmarshaling to and from JSON is lossless.
This means that even as we change the subset of valid osbulid manifests
that we support, we can still load any previous state from disk, and it
will continue to work just as before, even though we can no longer
deserialize it into our internal notion of osbuild.Manifest.
This fixes the underlying problem of which #685 was a symptom.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Any valid ImageBuild must contain a Manifest, so don't allow this to be
nil, simplifying the code a bit in the process.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The manifests we support marshalling/unmarshalling in composer are a
subset of the ones supported by osbuild. osbuild needs to preserve
backwards compatibility, but composer only needs to be concerned with
producing valid manifests at a given time.
A problem with this is that when we marshal our manifests to disk, and
read them out with a future version of composer, we would have to make
sure that the serialization is compatible, or unmarshalling the entire
store will fail. This is what happened when commit 5938276d3b
changed the way we marshal the file source.
Improve on this by only dropping those composes from the store on
unmarshal, whose manifests we are no longer able to unmarshal.
In future commits we will drop that requirement too.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Specifying the image format in the `DiskContainers` part of the snapshot
import process sometimes causes AWS to fail the import due to formatting
issues. However, AWS seems to do a decent job at determining the image
format on its own without specifying the format explicitly.
Signed-off-by: Major Hayden <major@redhat.com>
`log` is `nil`, unless we are in verbose mode. Skip logging if no logger
was passed.
Long-term I think we should reconsider how we do our logging, and always
log these type of unexpected errors, and hence always have the logger be
non-nil. For now, fix the immediate problem.
The fact that loading the store from disk fails on upgrade is
unexpected, and should be fixed separately.
Fixes#685.
Signed-off-by: Tom Gundersen <teg@jklm.no>
