The changes to the TestComposeUnsupportedMountPointV0() test case when
the osbuild/images has been updated to v0.172.0 broke the RHEL nightly
and GA CI pipelines.
Change the test to be backward compatible with osbuild/images <
v0.172.0.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The internal blueprint implementation has been removed from
osbuild/images. Conversion from osbuild/blueprint blueprints to
osbuild/images blueprints is no longer necessary.
The version check is failing on the "dot" version, while trying to
convert "118.1" to an integer. Delete the condition for skipping the
test, because it will never be running on anything older than v83
anyway.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
TestMultilibBlueprintDepsolveV0 unit test relies on 'gsl' package and
the availability of its 32 bit and 64 bit versions in the distribution
repositories. However, the package is no longer available in c10s /
el10. Moreover, there are no 32 bit RPMs in the repositories.
Conditionally compile the test only if 'rhel10' build tag is not
specified. Modify the SPEC file to define 'rhelX' build tag when
compiling tests on RHEL, where X is the major version.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Update the osbuild/images to the version which introduces "dot notation"
for distro release versions.
- Replace all uses of distroregistry by distrofactory.
- Delete local version of reporegistry and use the one from the
osbuild/images.
- Weldr: unify `createWeldrAPI()` and `createWeldrAPI2()` into a single
`createTestWeldrAPI()` function`.
- store/fixture: rework fixtures to allow overriding the host distro
name and host architecture name. A cleanup function to restore the
host distro and arch names is always part of the fixture struct.
- Delete `distro_mock` package, since it is no longer used.
- Bump the required version of osbuild to 98, because the OSCAP
customization is using the 'compress_results' stage option, which is
not available in older versions of osbuild.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
UBI and the oldest support Fedora (37) now all have go 1.19, so we are
cleared to switch.
gofmt now reformats comments in certain cases, so that explains the formatting
changes in this commit.
See https://go.dev/doc/go1.19#go-doc
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Remove all the internal package that are now in the
github.com/osbuild/images package and vendor it.
A new function in internal/blueprint/ converts from an osbuild-composer
blueprint to an images blueprint. This is necessary for keeping the
blueprint implementation in both packages. In the future, the images
package will change the blueprint (and most likely rename it) and it
will only be part of the osbuild-composer internals and interface. The
Convert() function will be responsible for converting the blueprint into
the new configuration object.
This is currently failing (fixed in the next commit). It tests to make
sure that a blueprint with package name globs can be frozen. The
resulting blueprint should replace the glob entries with the expanded
list of packages.
This tests to make sure that package name globs are working during
integration test runs. dnf supports this, and users have been using it,
so testing to make sure it keeps working is important.
Update the internal RepoConfig object to
accept a slice of baseurls rather than a
single field. This change was needed to
align RepoConfig with the dnf spec [1].
Additionally, this change adds custom json
marshal and unmarshal functions to ensure
backwards compatibility with older workers.
Add json tags to the internal rpmmd config
since this is serialized in dnfjson.
Add unit tests to check the serialization
is okay.
[1] See dnf.config
ioutil has been deprecated since go 1.16, this fixes all of the
deprecated functions we are using:
ioutil.ReadFile -> os.ReadFile
ioutil.ReadAll -> io.ReadAll
ioutil.WriteFile -> os.WriteFile
ioutil.TempFile -> os.CreateTemp
ioutil.TempDir -> os.MkdirTemp
All of the above are a simple name change, the function arguments and
results are exactly the same as before.
ioutil.ReadDir -> os.ReadDir
now returns a os.DirEntry but the IsDir and Name functions work the
same. The difference is that the FileInfo must be retrieved with the
Info() function which can also return an error.
These were identified by running:
golangci-lint run --build-tags=integration ./...
Fedora 35 support was dropped, so we can update to a newer Go.
Stable RHEL 8 and 9 and Fedora 36 ships Go 1.18, so let's switch to it.
"//go:build" directives are now apparently enforced by go fmt, so that's why
there were added.
Also, all the github actions were adjusted to use Go 1.18.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Since the LVM support was added to all distros, our disk
related code is adaptive, i.e. we will set the correct BLS
and grub2 prefix if there a `boot` partiton is present in
the layout after all customizations happen, which includes
LVMification.
One thing that was not yet fully working was layouts that
do not yet have a `/boot` partition but allow LVMification.
In that case `NewPartitionTable` and if `/boot` was the
first (or only) customization, would LVMify the partition
which in turn would create the `/boot` partition; but after
`newPT.ensureLVM()` the call to `newPT.createFilesystem`
with `/boot` would try to create another `/boot` mountpoint.
In order to deal with this situation correctly we are now
using a two phase approach: 1) enlarge existing mountpoints
and collect new ones. 2) if there are new ones and LMVify
was allowed, switch to LVM layout. Do a second pass and now
create or enlarge existing partitions, handling `/boot` in
the process.
All calls to rpmmd.Depsolve() are now replaced with the equivalent call
to solver.Depsolve() (or dnfjson.Depsolve() for one-off calls).
Attached an unconfigured dnfjson.BaseSolver to all APIs and server
configurations where rpmmd.RPMMD used to be. This BaseSolver instance
loads the repository credentials from the system and carries the cache
directory, much like the RPMMD field used to do. The BaseSolver is used
to create an initialised (configured) solver with the platform variables
(module platform ID, release ver, and arch) before running a Depsolve()
or FetchMetadata() using the NewWithConfig() method.
The FillDependencies() call in the modulesInfoHandler() of the weldr API
has been replaced by a direct call to the Depsolve() function. This
rpmmd function was only used here. Replacing the rpmmd.Depsolve() call
in rpmmd.FillDependencies() with dnfjson.Depsolve() would have created
an import cycle. The FillDependencies() function could have been moved
to dnfjson, but since it's only used in one place, moving the one-line
function body into the caller is ok.
For testing:
The mock-dnf-json is compiled to a temporary directory during test
initialisation and used for each Depsolve() or FetchMetadata() call.
The weldr API tests now use the mock dnfjson. Each rpmmd_mock.Fixture
now also has a dnfjson_mock.ResponseGenerator.
All API calls in the tests use the proper functions from dnfjson and
only the dnf-json script is mocked. Because of this, some of the
expected results in responses_test had to be changed to match correct
behaviour:
- The "builds" array of each package in the result of a module or
project list is now sorted by version number (ascending) because we
sort the package list in the result of dnfjson by NVR.
- 'check_gpg: true' is added to the expected response of the depsolve
test. The repository configs in the test weldr API specify 'CheckGPG:
True', but the mock responses returned it as false, so the expected
result didn't need to include it. Since now we're using the actual
dnfjson code to convert the mock response to the internal structure,
the repository settings are correctly used to set flag to true for
each package associated with that repository.
- The word "occurred" was mistyped as "occured" in rpmmd and is now
fixed in dnfjson.
NOTE that the tests against the rpmmd fixure for running and waiting are
a bit of a kludge. It is impossible to mock up a running or waiting that
will allow an actual cancel, so the tests check for the internal server
error generated by using the fixure data.
Getting this error means that the API code did it's job and tried to
cancel the compose so that should be a good compromise.
Extend Weldr API to accept a list of denied image types, which should
not be exposed via API for any supported distribution. This
functionality will be needed to not expose image types which can't be
successfully built outside of Red Hat VPN. Example of such images are
the official RHEL EC2 images, which include RHUI client packages not
available publicly.
Image Types are filters when listing available compose types and
creating a new compose using Weldr API.
Extend osbuild-composer configuration to allow specifying the list of
denied Image Types for Weldr API.
Add unit tests for implemented changes.
Add NEWS entry describing the newly introduced functionality.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The system sources allow specification of the rhsm parameter, but it
isn't available in the sources configured over the Weldr API. This patch
implements support for it.
Modify composer to use RepoRegistry, instead of loading the host
repositories, when initializing WeldrAPI.
Modify WeldrAPI to use RepoRegistry, instead of a map of repository
definitions. Make sure that the RepoRegistry method specific to image
type is used in Welder where appropriate. Specifically when depsolving a
Blueprint, which is used to build a specific image type. Update Weldr
API unit tests to reflect the change.
Add a new method to RepoRegistry, allowing to get list of repositories,
which should be used for building an image for a given architecture,
without specifying the exact image type. Add relevant unit tests.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
fedoratest was yet another dummy distribution used by unit tests. After
the rework of test_distro, there is no reason to not use it as the only
distro implementation for testing purposes.
Remove fedoratest distro and replace it with test_distro in all affected
tests.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
testjobqueue did not implement the JobQueue interface correctly (noted
in its package comment), making it impossible to write tests for
JobQueue itself.
Replace its use everywhere with fsjobqueue operating on a temporary
directory.
When gsl with version * was specified in the blueprint,
composer depsolved both x86_64 and i686 version of gsl.
This test case should prevent this from happening.
gsl is used because it has x86_64 and i686 versions on both RHEL and Fedora.
Also, gsl-devel package exists, which is not dependant on gsl and shouldn't
be depsolved.
It was passing it through to the non-system delete function
and not returning an error. This checks for system repos first and
returns a 400, SystemSource error response if it is in the system list.
This changes store.DeleteSource to DeleteSourceByName for v0 use and
DeleteSourceByID for v1 usage.
It includes a new client function DeleteSourceV1, adds a new test, and
converts the tests for the previous Source V1 API commits to use
DeleteSourceV1.
This commit changes the store.GetAllSources to distinguish between
getting the source by the Name field, or by the ID (the key to the map)
using GetAllSourcesByName and ...ByID.
SourceConfig.RepoConfig() now takes an id parameter because SourceConfig
only stores the Name, not the ID.
In weldr I split the sourceInfoHandler into 2 separate functions for v0
and v1 behavior, with the core of the old function refactored as
getSourceConfigs and used by both of them.
This also adds new structs for the SourceResponseV0 and SourceResponseV1
as well as helper functions for converting to/from store.SourceConfig
This commit changes the store.PushSource function to take the key as
well as the SourceConfig so that it can be used for v0 or v1.
It adds helper functions for decoding the toml/json into a new
SourceConfig interface type which lets the core source/new code be
shared between the versions.
It also adds tests for the new API behavior.
This is the first patch in a series to add APIv1 support to the
/projects/source routes. The change involves using the store.Sources key
in a different way (as an id instead of as a duplicate of the struct's
Name field) but does not actually involve changing the Sources json in
the store.
In the V0 API the name of the source was used as the identifier, and
there was no short id. In V1 the source is identified by the API using
a short id, and the Name is just a field in the struct to describe the
source. This will become more obvious with the /projects/source/info
response.
This commit changes the following:
Changes store.ListSources to ListSourcesByName and explicitly pulls the
name from the source struct instead of the key. v0 will use this
function call.
Adds store.ListSourcesById which returns the source key as the
identifier. This is used by v1.
Adds a new weldr.SourcesListV1 response type, even though it is exactly
the same as the V1 response in this specific case. I thought it would be
better to have one called V1 than to reuse the V0 struct and possibly
confuse people.
The /projects/source/list API now lists the sources by name for v0 and id for v1.
A test has been added. You will notice it still uses v0 to push and
delete the sources. These will be updated when the new version of the
functions are added in subsequent commits.
