Commit graph

3344 commits

Author SHA1 Message Date
Lars Karlitski
42f46a5992 distro/rhel82: make openstack's package list constistent with lorax 2019-12-13 00:14:07 +01:00
Martin Sehnoutka
b7b60363dc Modify RHEL8.2 VHD image to boot in Azure 2019-12-12 13:52:21 +01:00
Tom Gundersen
d074395474 distro/rhel-8.2: update VHD images to be closer to lorax
The current one was taken from fedora and that failed both due to an
SELinux bug in RHEL8.2, but also because the list of packages was
wrong for the distro.

We don't enable WAAgent, which we probably should. And there are a few
other config tweaks to cloud-init and WAAgent that we skip too.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-12 13:52:21 +01:00
Lars Karlitski
305b2dda5c rpmmd: add IgnoreSSL key to RepoConfig
This is the opposite of dnf.conf's `sslverify`, because go's default for
booleans is always false. This is error prone: we'd like to default to
true.
2019-12-12 01:27:04 +00:00
Lars Karlitski
2b42612336 distro: insert custom sources into pipeline
This is an additional argument on `distro.Pipeline` right now. In the
future, we probably want this per-blueprint.
2019-12-11 21:19:32 +00:00
Lars Karlitski
d9ad5d7062 weldr: add toml output to projects/source/info 2019-12-11 18:53:10 +00:00
Lars Karlitski
72e1da47ab weldr: make projects/source/new accept text/x-toml
composer-cli sends sources as toml if the original format is toml.
2019-12-11 18:53:10 +00:00
Tom Gundersen
d55abdfd75 osbuild-worker: disable the osbuild cache
We still need to figure out the garbage collection, so for now don't
enable the cache.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-11 15:23:24 +01:00
Tom Gundersen
d33fc5f010 composer: add provisional multi-arch support
The pipeline generation now takes the architecture as an argument.
Currently only x86_64 is supported. The architecture is detected
at start-up, and passed down to each pipeline translation.

For osbuild-pipeline we now requrie the architecture to be passed
in.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-12-11 15:23:24 +01:00
Lars Karlitski
dcc9cdedee distro: rename output of qcow2 image to disk.qcow2
Integration tests currently expect that and they need to be able to run
against lorax-composer (which outputs disk.qcow2) and osbuild-composer.
2019-12-10 20:21:47 +00:00
Lars Karlitski
ea70d9dfc5 blueprint: don't set omitempty for toml groups and modules
The integration tests expect the empty arrays to be there.
2019-12-10 20:21:47 +00:00
Lars Karlitski
0718766fbc weldr: fix reply on compose/info route
composer-cli requires `dependencies.packages` to exist. We can still
leave it empty for now, because it doesn't look at the contents.
2019-12-10 20:21:47 +00:00
Lars Karlitski
b7dac20c74 weldr: add test for blueprint/depsolve route 2019-12-10 20:38:22 +01:00
Lars Karlitski
280a7c3950 distro/rhel82: use latest nightly
Now that osbuild-composer is inserting repository checksums dynamically,
we can use a URL that points to a changing repository.
2019-12-10 20:38:22 +01:00
Lars Karlitski
d3a0b788a2 distro: set the repository checksum dynamically
Instead of having a static repository checksum, set it dynamically from
the metadata that osbuild-composer last saw. This is implemented in
dnf-json, which returns the checksums for each repository on every call.

This enables the use of repositories that change over time, such as
fedora-updates. Note that the osbuild pipeline will break when such a
repository changes. This is intentional: pipelines have to be
reproducible.
2019-12-10 20:38:22 +01:00
Lars Karlitski
75218ad2d9 distro/fedora30: don't include variables in repo URL
Replace those variables with their contents everywhere the repository
URLs are used is error-prone.
2019-12-10 20:38:22 +01:00
Ondřej Budai
6bbc89d5f3 api: implement /compose/delete route 2019-12-10 01:14:36 +01:00
Ondřej Budai
6d15833e4e store: introduce state directory
Prior to this commit outputs directory used by local target was owned by root.
This made impossible for osbuild-composer to delete images. (osbuild-composer
doesn't run as root).

This commit introduces state directory in which osbuild-composer creates
outputs directory. Because this directory is owned by osbuild-composer, it's
able to delete files inside.
2019-12-10 01:14:36 +01:00
Ondřej Budai
4e62f181fa api: implement /compose/logs route
The implementation is just a stub returning always the same tar archive.
The ability to return actual logs will be implemented in the future - osbuild
isn't currently returning any logs.
2019-12-08 00:21:30 +01:00
Ondřej Budai
d7f81b36e6 api: implement /compose/info route
Not everything is implemented in this route but enough to pass the tests.
2019-12-06 10:42:19 +01:00
Ondřej Budai
ad7650384e api: use committed blueprint in POST /compose
lorax-composer apparently doesn't use blueprint from workspace to create a new
compose. This commit implements the same behaviour.
2019-12-05 13:08:50 +01:00
Ondřej Budai
24faab8c18 store: clean up blueprint API
Prior to this commit blueprint getters looked like C-style API with output
parameters. This commit refactors them to more conventional multiple return
values API.
2019-12-05 13:08:50 +01:00
Ondřej Budai
f89a9671be store: add image struct into compose struct
As a part of f4991cb1 ComposeEntry struct was removed from store package.
This change made sense because this struct is connected more with API than
with store - store uses its own Compose struct. In addition, converters
between Compose and ComposeEntry were added. Unfortunately, ComposeEntry
contains ImageSize which was not stored in Compose but retrieved from store
using GetImage method. This made those converters dependent on the store,
which was messy.

To solve this issue this commit adds image struct into Compose struct.
The content of image struct is generated on the worker side - when the worker
sets the compose status to FINISHED, it also sends Image struct with detailed
information about the result.
2019-12-05 09:48:21 +01:00
Lars Karlitski
b2880cacc6 weldr: don't indent toml output
Be consistent with lorax-composer.

The integration tests do a string compare on received toml files.
2019-12-04 19:15:33 +01:00
Lars Karlitski
b1c65bb671 blueprint: tag all struct fields for toml
This is repetitive, but there doesn't seem to be another way to export
lower-case field names.
2019-12-04 19:15:33 +01:00
Lars Karlitski
94bfcc518c weldr: bump blueprint version
According to the same rules as lorax-compposer:

  https://weldr.io/lorax/lorax-composer.html#blueprints
2019-12-04 18:35:18 +01:00
Ondřej Budai
b6307cafd5 api: fix compose routes returning null instead of empty array
Cockpit-composer really doesn't like that
2019-12-03 11:49:35 +01:00
Ondřej Budai
dca7ab2847 api: handle blueprint depsolve error 2019-12-03 11:49:27 +01:00
Lars Karlitski
20bb60f3fd weldr: add support for toml blueprints 2019-12-03 01:27:00 +01:00
Lars Karlitski
1c5adfc682 job: use image.raw.xz filename
RHEL and Fedora used different output formats (.ami vs .raw.xz). The job
package assumed `image.ami`, which failed for RHEL.

Change Fedora to use image.raw.xz as well. This makes it consistent, but
we should also get the filename from the distro at some point.
2019-12-01 00:05:17 +01:00
Lars Karlitski
b53d897019 job: set S3 key to compose id
cockpit-composer currently doesn't require setting the S3 key. Set it to
the compose ID if it's empty.
2019-12-01 00:05:17 +01:00
Ondřej Budai
f4991cb1ca api: Add support for upload API
This commit introduces basic support for upload API. Currently, all the routes
required by cockpit-composer are supported (except for /compose/log).

Also, ComposeEntry struct is moved outside of the store package. I decided
to do it because it isn't connected in any way to store, it's more connected
to API. Due to this move there's currently a known bug that image size is
not returned. This should be solved by moving Image struct inside Compose
struct by follow-up PR.
2019-12-01 00:05:17 +01:00
Ondřej Budai
be1cf79d6a test: introduce helpers for go-cmp
This commit introduces three helpers to ignore dates, uuids and paths when
using cmp.diff() method
2019-12-01 00:05:17 +01:00
Ondřej Budai
613b659b95 test: use google cmp library to do the deep reflect
The cmp library has two advantages:

- it shows diffs between compared values
- it allows ignoring fields based on field name or type
2019-12-01 00:05:17 +01:00
Ondřej Budai
268befa435 store: make ListQueue return value more predictable by sorting it
The return value hasn't been predictable prior to this commit. This is
probably caused by internal implementation of Golang map. This commit
fixes it by sorting the output.
2019-12-01 00:05:17 +01:00
Ondřej Budai
7fbe8453ef api: support wildcard /compose/status 2019-12-01 00:05:17 +01:00
Ondřej Budai
df7e14d5eb target: add new fields to target and cleanup
We need some more fields in target struct to support weldr API. This commit
introduces them. Also a bit of cleanup is done.
2019-12-01 00:05:17 +01:00
Martin Sehnoutka
364ea62f59 Introduce Azure upload test
The test is very simple, it first upload a file with random content and
then downloads it back and compare hashes.
2019-11-29 12:19:40 +01:00
Lars Karlitski
5dad3bfc8e worker: pass build environment to osbuild
Detect it from the host using the distro package.
2019-11-29 00:46:05 +01:00
Lars Karlitski
85e6182bdc distro: don't fall back to fedora-30
Make osbuild-composer use FromHost() directly. Everywhere else needs to
specify the distro explicitly.

Also don't panic when a distro doesn't exist. Instead, return nil. Make
sure all callers check for that.
2019-11-29 00:46:05 +01:00
Lars Karlitski
14ebed46da distro: register distros explicitly
Automatically registering on `init()` is clever, but a bit too magical
and easy to get wrong, because every binary must include all distros
somewhere.

Flip this inside out: distros now have a `New()`, which returns
something that implements the `Distro` interface. The distro package
explicitly creates all of them.

This means that distros cannot import distro itself anymore, because go
forbids import cycles. This only affected `InvalidOutputFormatError`.
Return a generic error for now.
2019-11-29 00:46:05 +01:00
Lars Karlitski
673d3ff14b blueprint: remove unused error
It was moved to the distro package a while ago.
2019-11-29 00:46:05 +01:00
Lars Karlitski
c0516420cc distro/rhel82: move ami output closer to official ec2 image
This includes adding (and excluding) the right packages, setting the right
systemd boot target, fixing kernel options, changing output type to
.raw.xz, and setting the size of the image to 6GB.
2019-11-29 00:46:05 +01:00
Tom Gundersen
7485094b34 pipeline/stage: add unit test for basic (un)marshalling
This makes sure that marshall/unmarshal are inverses of each other
for the most trivial instances of the various stage structs.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-11-28 05:56:11 +01:00
Tom Gundersen
caff96bd4f job/run: never panic on failed job
Return the error code of the osbuild run, and an array of errors,
one for each target provided. If a target fails, all other targets
are still attempted.

If either osbuild or one of the targets retursn an error, the worker
notifies osbuild-composer that the job failed.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-11-28 05:56:11 +01:00
Tom Gundersen
e29b6fe06b target/aws: support setting the image name
This is part of the lorax API, so support forwarding this.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-11-28 05:56:11 +01:00
Tom Gundersen
553edef01d distro/f30: rework in the same way as RHEL8.2
The produced pipelines are unchanged.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-11-28 01:24:05 +01:00
Ondřej Budai
66c57a05b7 api tests: add store fixtures
Prior this commit there wasn't an easy to populate the store. The only way
was to call the weldr API or store methods. This design made testing of
various edges quite hard.

This commit adds store fixtures - an easy way how to define store state
before each test case.

In addition, the fixtures were refactored so that new instances are created
prior each test. Before this change the tests were in some cases dependant
on each other.
2019-11-27 22:27:40 +01:00
Ondřej Budai
a00a0caa70 api tests: refactor weldr and jobqueue api tests to use common helpers
The helper functions in both api packages were more or less same. However,
over time they have been slowly diverging. This commit extract the helpers
into one common package to make the tests more maintainable and
to deduplicate the code.
2019-11-27 22:27:40 +01:00
Lars Karlitski
da311f13eb distro: add rhel82
This takes a different approach to outputs and customizations, which is
much shorter and duplicates less code.

This uses links to internal repositories for now, because 8.2 hasn't
been released yet.

Add a `distro` flag to `osbuild-pipeline`.
2019-11-27 17:08:51 +01:00