After introducing Go 1.18 to a project, it's required by law to convert at
least one method to a generic one.
Everyone hates IntToPtr, StringToPtr, BoolToPtr and Uint64ToPtr, so let's
convert them to the ultimate generic ToPtr one.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Fedora 35 is going EOL on Tue 2022-12-13. At the time of writing this commit
message, that's the next day. As we do releases on Wednesdays, the next
release will never find its way to F35 and thus, there's no point in keeping
support for it.
Let's delete everything that relates to Fedora 35. If there's something that
cannot be deleted (e.g. CI containers based on F35), let's upgrade it to F37.
TestCrossArchDepsolve now uses CentOS Stream 8 because RHEL 8.4 cannot read
F37 repository metadata. This is a similar issue to
https://bugzilla.redhat.com/show_bug.cgi?id=2004853 . Basically, newer
repositories can be only read by libmodulemd >= 2.11.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Previously the blueprint change was only present until a reboot. The
change and its version was recorded, but the actual blueprint was not
saved.
This adds saving the blueprint when it is changed, and loading the old
blueprint versions when they are present in the store.
No version change is needed, if the blueprint is not present it is
loaded as an empty blueprint which was already being detected by
checking the length of the Name.
This will allow specific blueprint changes to be recalled after a
restart of the server.
This also includes tests for the new behavior.
When the store is written to disk it simplifies the ImageBuild details
into a simple image type string. This works fine for composes that match
the host's distro but isn't enough detail to load composes made for
other distros, especially if the image type name isn't supported on the
host. This results in cross distro compose results being lost after a
reboot.
This fix uses the distro information from the compose's blueprint to
determine which distro the image type should be loaded from. It assumes
that the architecture matches the hosts' arch -- this is currently
always true but in the future if cross-arch builds are added it will
need to be addressed in a different way.
newComposeFromV0, newComposesFromV0, and newStoreFromV0 now take a
pointer to the full distro registry instead of an Arch, this allows them
to access the correct image types for the distro selected by the
blueprint. When loading the composes from disk the blueprint distro is
loaded from the registry before checking the image type string.
This means that we do not have to change the store version or on disk
format, the only thing changing is how it decides to populate the
ImageBuild when reloading the store.
A number of tests use a fake test distro using fake architecture names.
These tests have been adjusted to use a fake distro registry with
overridden host architecture that matches the fake one.
With an empty or missing version number the commit message would not
include the version (which is set to 0.0.0 by calling Initialize). This
adds a call to Initialize() in the API code before constructing the
commit message. It also moves the check for non-empty blueprint name
into the Initialize call where it belongs.
This fixes a bug where tagging a blueprint would overwrite the blueprint
in the blueprintChanges storage with an older version of the blueprint.
It also adds checks to make sure the blueprint is present in the
blueprintsCommits and blueprintsChanges maps before trying to reference
them -- just in case something in the store gets out of sync.
TagBlueprint has a bug, it replaces the tagged blueprint with the last
untagged blueprint. The test wasn't testing for this, so correct that
before fixing the actual bug.
The map is meant to maintain compatibility with changes in image type
names. Most entries however map to themselves. It's redundant to keep
updating this map with every new image type name and map it to itself.
Instead, check the map for a compat string and if one doesn't exist,
return the original.
Conversely, when the inverse lookup is performed, first check if the
compat string is a valid image type name and only iterate the map if
it's not.
Now we don't need to update the map every time we introduce a new image
type, unless we need to map an old name to a new one.
This is the first step to support embedding container images. Here
we add the `containers []container.Spec` argument to supply images
with resolved container specifications. For now all distros will
return an error in case a container is actually supplied since none
of them currently support embedding containers. NB: also no apis or
tools will actually resolve containers.
The test_distro Manifest, which is used in tests across multiple
packages, was using the old structure. Updated to the v2 structure and
adapted all tests.
The filename of the image as produced by osbuild for a given export is
currently set in each target options type in the `Filename` struct
member. However, the value is not really specific to any target type,
but to the specific export used for the target. For this reason move the
value form target type options to the `Target` struct inside a new
struct `OsbuildArtifact` under the name`ExportFilename`.
The backward compatibility with older implementations of the composer
and workers is kept on the JSON (Un)mashaling level, where the JSON
object is always a super-set of the old and new way of providing the
export filename in the Target.
Completely remove the use of `local` target from all code, which is not
required to keep backward compatibility. The target has not been used in
composer for some time already, but some unit tests still used its data
structures. Mark the target as deprecated and adjust all unit tests that
depended on it.
The backward compatibility is kept mostly to enable long running
osbuild-composer instances, which were upgraded to still read old jobs
from the store.
While a target with the same intention will be reintroduced, the current
`local` target data structures contain many fields which would not be
relevant for the new target.
In addition, while the "local" target will be ever used only by Weldr
API, the name would be a bit misleading. Although the worker usually
runs on the same system when using Weldr API, there is no hard
requirement enforcing this setup. In reality, the worker will be
uploading the image back to the worker server, so there is room for a
better name.
Add a container image type that is based on the existing fedora
container image. There is a delta in terms of the configuration
because osbuild does not yet provide all the neccessary means,
but the package set is already very close.
Add the `gce-rhui` image type intended for Google Compute Engine. The image
uses Google's RHUI infrastructure to access Red Hat content.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add the `gce` image type intended for Google Compute Engine. The image
is BYOS - bring your own subscription and requires registering in order
to access Red Hat content.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
When the host distribution is not known or supported, instead of failing
with an error, print a warning to the log and initialise the API with
the architecture name and distro name.
This enables running the weldr API on unsupported distros for
cross-distro building.
Guards against a nil arch member when initialising the store.
The blueprint name should never be empty, as it can cause other problems
like with the blueprints list results. Return an error if one is pushed
to the store, either as a blueprint commit or as a blueprint workspace.
Also adjusts the new test for the new error.
Related: rhbz#1922845
There is a problem with blueprint changes, once the server is restarted
the previous changes are all lost because they are not serialized to
disk.
This adds test fixture support so that new tests can be added before
fixing the problem. It adds store.FixtureOldChanges with blueprints
changes and empty blueprints.
Related: rhbz#1922845
Clean up some implementation aspects of the Fedora distro definition:
- Do not have default Fedora distro version and use `fedora` as the
package name in all places that use it, instead of `fedora33`.
- Fix bugs when wrong (Fedora 33) values were returned by `OSTreeRef()`
and `Releasever()` for newer Fedora releases.
- Test Fedora 35 in package unit tests.
- Add unit test for `OSTreeRef()` method.
- Use architecture name constants from `distro` package, instead of
string literals.
Fix#1802
Signed-off-by: Tomas Hozza <thozza@redhat.com>
OSBuild Composer can now build the RHEL 8.5 Raw Images. This images are
compressed raw images, i.e. a file that has a partition layout with an
deployed OSTree commit in it. It can be used to flash onto a hard drive
or booted in a virtual machine. An existing OSTree commit needs to
be provided.
The following image new types are supported: edge-raw-image.
This adds a new installer called the "Simplified Installer" for Edge.
In contrast to the existing insaller, which is based on Anaconda, this
new installer based on the CoreOS installer project[1], a small rust
based binary that is executed in the initramfs and will flash a raw
image to a specified installation device. For this a new blueprint
option is introduced. The raw image is created from an existing OSTree
commit and embedded into the resulting bootable iso. When booting the
iso the installation will automatically start witout any interaction
from the user.
NB: As with the existing edge installer, support is currently limited
to x86. The new installer also does not support non-uefi boot.
[1] https://github.com/coreos/coreos-installer
Co-Developed-by: Achilleas Koutsou <achilleas@koutsou.net>
Co-Developed-by: Antonio Murdaca <runcom@linux.com>
Renamed tar-installer to image-installer.
This is a more appropriate name:
- It disassociates the image type from the "tar" image type. The two
should not be perceived to be connected.
- It's more descriptive. The format of the payload (tar) isn't relevant
to the purpose of the image type.
Add new image type definitions `ec2` and `ec2-ha` representing the
official RHEL ec2 image types.
Add a `xzArchivePipeline()`, which returns a pipeline producing a XZ
archive from a file produced by a different pipeline.
Add rpmrepo snapshots for `rhui` and `ha` repositories used to generate
image test cases. `rhui` is used by the `ec2` image and it is available
on x86_64 and aarch64 architectures. `ha` is used by the `ec2-ha` image
and it is available only for x86_64.
The new image type definitions are currently not used by any
API test case.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The system sources allow specification of the rhsm parameter, but it
isn't available in the sources configured over the Weldr API. This patch
implements support for it.
fedoratest was yet another dummy distribution used by unit tests. After
the rework of test_distro, there is no reason to not use it as the only
distro implementation for testing purposes.
Remove fedoratest distro and replace it with test_distro in all affected
tests.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Extend the "Test Distro" implementation and definition to contain two
architectures and make the second architecture contain two image types.
Add New2() function returning another "Test Distro".
Modify the `internal/store` unit tests to reflect changes done to the
"Test Distro".
Signed-off-by: Tomas Hozza <thozza@redhat.com>
This adds support for Packages to the store's json structures so that
they will be preserved across restarts of the osbuild-composer service.
Reading the old format will result in an empty []rpmmd.PackageSpec so
this does not require a composeV1 structure.
This adjusts current tests to account for the new struct member, and
tests osbuild-composer with empty results (eg. existing system will not
have this stored) and with the sets populated by test data.
This adds a list of the depsolved packages to the store's Compose
struct. It is indexed by compose UUID and contains a list of
PackageSpecs that were used to construct the compose. This can assist in
auditing of the composes, or be used to duplicate the compose.
New image type that generates a Boot ISO. The ISO contains a RHEL Edge
commit and an installer. On Boot, it sets up a new RHEL Edge system
with the commit.
The RHEL Edge commit (ostree commit) is downloaded during build from a
URL that should be supplied with the compose request. The commit's hash
and URL need to be added to the Sources list in the Manifest.
Unlike other types, the new image type defines its own "build" package
set that is added to the distro and arch build package lists.
imageTypeS2 implements the distro.ImageType interface buts generates a
Manifest matching the new osbuild v2 schema.
Two new image types are added to the rhel84 distro (x84_64 and aarch64)
for generating OCI containers contain an Edge (ostree) commit and, when
run, start a web serer to serve the commit.
The image type uses the new PackageSets map to define packages (and
excludes) for the image. The old methods (Packages() and
BuildPackages()) are implemented for compatibility with the old
workflow.
The image also defines an extra package set for the container that will
serve the package: "httpd" (and its dependencies).
The distro.ImageType interface has a new method: Exports()
It should return a list of names or IDs of artefacts that should be
exported from osbuild when the job is complete.
For the old image types, this is simply set to "assembler".
