osbuild has a concept of runners now: scripts that set up a build
environment. Update the osbuild submodule to latest master, change
`Pipeline` to to the new buildroot description format, and use the
`org.osbuild.fedora30` runner from the fedora30 distro.
This slightly changes the customizations logic. We now make sure
that each stage is appended exactly once.
customizations.go are now responsible only for the things that are
completely generic, and not per-ouput-type. helpers.go contain more
high-level helpers that combine customziations and per-output-type
defaults.
This does not change the behaviour, though some pipelines are slightly
reordered to make them consistent.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This commandline tools uploads a file to S3, as a proof of concept.
All options are mandatory. Credentials are only read from the
commandline and not from the environment or configuration files.
The next step is to add support for importing from S3 to EC2,
currently the images we produce cannot be imported as-is, so this
requires more research.
To try this out: create an S3 bucket, get your credentials and
call the tool, passing any value as `key`. Note that if the key
already exists, it will be overwritten.
Signed-off-by: Tom Gundersen <teg@jklm.no>
lorax-composer recently introduced API version 1. This commit introduces
very basic support for it. This implementation tries to deduplicate code
for routes with the same behaviour as much as possible. All the differences of
v1 API are marked as TODOs for now and will be implemented in follow-ups PRs.
Make distros export repository information and use those in the weldr
API. This means that repos are only specified once and that the API
returns the right packages when we allow different distros.
Split the error case (no sources specified) into its own function, so
that we can use `source/info/:sources` (note the colon) to get the list
of sources without the leading `/`. This gets rid of two special cases
which made the previous implementation hard to parse.
The naming is confusing: repositories have an `id` and a human readable
`name`. Weldr's sources also have a field called `name`, but
lorax-composer uses that as a way to identify repositories by their id.
Use `id` consistently here as well.
The blueprints freeze test now creates the blueprint with the package
dep-package1 which is mocked and will properly depsolve when the
blueprint is frozen. This test can no longer run externally against
lorax-composer.
This test creates a new blueprint with libsemanage. Libsemanage is
already in the mock rpmmd so when we test the freeze route on this
blueprint, the blueprint will properly depsolve and return the package
with the depsolved version.
The blueprint freeze route returns the blueprint info but each
package will be the package selected by depsolving. So, instead of the
version being the version number with optional wildcards as
/blueprints/info would provide, the version is of the form
`Version-Release.Arch`.
These endpoint are similar in many ways, therefore just one commit. Their
functionality is basically same as in lorax except for error messages and
weird edge cases when handling trailing slashes.
closes#64, closes#65
Add a route to set a blueprint back to its state at a particular change.
The route `blueprints/undo/:blueprint/:commit` requires the blueprint
name and the commit hash for the change that the blueprint should be
reverted too. Also, the commit message for the change created when a
blueprint is pushed is now passed from the api to the store's
PushBlueprint funtion.
Add function to get a single blueprint change given the blueprint name
and the commit hash. Update the PushBlueprint function to allow a commit
message to be passed to it which will be used when adding a change.
The package list is generated on each request for a package so there is
no longer a need to generate the package list in main or to store these
packages in the API object.
The package list is now generated from the base repo and the
user-defined repos. This allows users to add packages not found in the
base repo to their blueprints.
Add function to convert between a user source and a repo which can be
passed to dnf-json. This is neccessary because user-defined sources have
a slightly different format than dnf repos.
1) additional qemu tests for ami, vmdk, vhd, and openstack image types
2) new type of systemd-nspawn tests for tar, ext4, and parititioned disk
types
the systemd-nspawn tests use loopback network interface directly from
the host so it is necessary to tweak the settings of its SSH server.
This is done in a "script" stage using simple "sed" command.
We want to test API methods which calls dnf. Unfortunately, calling dnf
is expensive operation - it requires network access and downloading
a lot of (meta)data. This commit changes the rpmmd implementation
so that it can be mocked.
For each blueprint name passed to the route, a list of the changes to
that blueprint will be returned.
weldr/tests: add blueprint changes test
In order to test blueprint changes a blueprint must be created with a
unique id. Blueprint changes are not deleted when the blueprint is
deleted so in order to test this against lorax the blueprint must have
not been used/tested before. This id is created from a random int. The
test creates and deletes the same blueprint twice to check that each
creation updates the list of changes.
When a blueprint is pushed to the store it will also add a change to the
BlueprintChanges map. Each blueprint has a list of changes mapped to
their commit. This commmit is a hex string based off of the sha1 hash of
the current time so every commit is unique. The message and timestamp
follow the format of lorax-composer.
Add a struct to store changes made to a blueprint. Each change contains
a commit which is a hex string based off of an sha1 hash, a message
describing the change, a revision which will usually be null, a
timestamp, and the blueprint at the time of the change.
When dnf-json dumps the packages from the repos passed to it, it does
not sort the packages. In order to properly list and search the
packages, the package list is now sorted before being returned by the
FetchPackageList function.
The tests works by executing osbuild with predefined pipeline. Then the
image boots and the testing script creates SSH connection to the running
VM. If everything goes fine `systemctl is-system-running` is executed
with result `running` and the test case passed.
The JSON definition of the test case contains also a blueprint that
should generate the desired pipeline, but it didn't work for me, so I'm
including it for future use from the golang unit tests.
The delete source handler now removes the leading "/" from the
parameters passed to it. Not removing the "/" caused sources to not
be deleted from the store since they could not be found when their name
contained a "/" as the first character.
lorax-composer uses the host's repositories for building images. This is
prone to accidental configuration errors and duplicates functionality
(adding custom repositories via the source API is much more explicit).
However, blueprints don't specify the distribution they're based on.
This is something they should do in the future to enable cross-distro
image builds. Until then, read `/etc/os-release` to detect the host OS
and use that as the base distro for a blueprint.
Detection works by concatenating the ID field with a `-` and the
VERSION_ID field. This mandates how additional distributions should
register themselves to the `distro` package.
If composer cannot build the detected distro, fall back to fedora-30.
Use this detection everywhere that fedora-30 was hard-coded before.
Introduce the `distro` package, which contains an interface for OS
implementations. Its main purpose is to convert a blueprint to a
distro-specific pipeline.
Also introduce the `distro/fedora30` package. It is the first
implementation of the distro interface. Most of its code has been copied
with minimal modifications from the blueprint package.
The `blueprint` package is now back to serving a single purpose:
representing a weldr blueprint. It does not depend on the `pipeline`
package anymore.
Change osbuild-composer and osbuild-pipeline to use the new API,
hard-coding "fedora-30". This looks a bit weird now, but is the same
behavior as before.
All test cases now also take an "distro" key in the "compose" object.
Add tests for adding and deleting user defined sources. Update existing
tests for source info. The source being added is the fish repository.
This is the same source that the cockpit-composer tests attempt to add.
The api now supports user defined sources alongside the existing base
repository. The routes /projects/source/new and
/projects/source/delete/<name> have been added.
/projects/source/info/<names> and /projects/source/list have been
updated.
The SourceConfig type is now called from the store instead of being
declared in the api.
The api is initialized with a base repo but users may want to add their
own custom repos. Based on lorax's implementation, the store now
supports sources. The source object is similar to but not the same as
the repos used by dnf. The store can now add, delete, and get info about
sources.
