Commit graph

1076 commits

Author SHA1 Message Date
Stef Walter
2d331034e5 CONTRIBUTING: Tweak documentation for new contributors
The CONTRIBUTING.MD was very useful. In this commit I'm suggesting
a couple changes that I had to make to get it to work. Hope it
helps.
2020-05-29 22:23:10 +02:00
Stef Walter
20c40b59ac osbuild-composer.spec: Add a dependency on osbuild-ostree
The new functionality that landed in b0cfec767a
needs this dependency in order to function.

Fixes #689
2020-05-29 22:22:04 +02:00
Tom Gundersen
71b69e5ad2 store: don't log if no logger was passed
`log` is `nil`, unless we are in verbose mode. Skip logging if no logger
was passed.

Long-term I think we should reconsider how we do our logging, and always
log these type of unexpected errors, and hence always have the logger be
non-nil. For now, fix the immediate problem.

The fact that loading the store from disk fails on upgrade is
unexpected, and should be fixed separately.

Fixes #685.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-29 22:20:43 +02:00
Ondřej Budai
76c18566fc spec: exclude i686 architecture
RHEL 8 doesn't have golang in the i686 buildroot anymore. Fedora doesn't have
i686 images and kernel anymore. Also, osbuild-composer doesn't have support
for building i686.

Let's exclude i686 for these reasons, it's dead.
2020-05-28 19:44:13 +02:00
Tom Gundersen
b5605642e3 repositories/rhel8: add s390x and ppc64le repositories
These were accidentally left out from
59355c6c91.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-28 07:25:31 -05:00
Ondřej Budai
5f0dffefe4 13
Release osbuild-composer version 13
2020-05-28 09:44:36 +02:00
Brian C. Lane
508e7b40f7 tests: lorax-composer returns 2 errors for these tests
TestComposeEmptyBlueprintV0
TestComposeInvalidCharsBlueprintV0
2020-05-28 08:28:11 +02:00
Brian C. Lane
c000a409a3 tests: Add missing descriptions to test blueprints
These were causing the wrong error when running against lorax-composer.
2020-05-28 08:28:11 +02:00
Tom Gundersen
59355c6c91 distro/rhel8: add preliminary support for ppc64le and s390x
This adds placeholder architecture structs, but no image types are
hooked up.

This will allow osbuild-composer to run on these architectures, and
serve the weldr API, but without support for building any images.

Image type support will be added in the future.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-28 07:34:48 +02:00
Tom Gundersen
a13076a97b distro/rhel8: refactor to be based on fedora-32
This is not a functional change, all produced manifests are as before.
This is in preparation for adding support for more architectures in a
follow-up patch.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-28 07:34:48 +02:00
Tom Gundersen
97b365618a distro/fedora32: minor style fixes
Rename the `Fedora32` type to simply `distribution`, to avoid the
stutter. Move `New()` to the bottom of the file, which is the only
non-generic part. Also make the linter happy.

No functional change.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-28 07:34:48 +02:00
Major Hayden
4dc48b2897 CI: Remove journald_cursor play variable
This variable is no longer needed since the bash script handles the
cursor.

Signed-off-by: Major Hayden <major@redhat.com>
2020-05-28 07:29:35 +02:00
Brian C. Lane
49ca489efa client: Add functions to handle Metadata and Results requests
Includes basic tests to check for errors, but not content.
2020-05-28 00:39:10 +02:00
Brian C. Lane
6b5ab26072 tests: Add unit tests for compose/metadata and compose/results 2020-05-28 00:39:10 +02:00
Brian C. Lane
b07735e2a8 weldr: Add compose/results repsonse
This returns a tar of the metadata json, the logs, and the image if the
compose was successful.
2020-05-28 00:39:10 +02:00
Brian C. Lane
0598be6b9a weldr: Add compose/metadata support
This will return a tar of the compose's json metadata. Inside the tar
the file is named UUID.json
2020-05-28 00:39:10 +02:00
Ondřej Budai
fcd3394a82 spec: bump the osbuild dependency to version 15
The new Fedora IoT image type uses org.osbuild.ostree.commit assembler with
tar option that was introduced in osbuild 15, therefore the dependency version
must be bumped.

This also bumps the submodule to 9cbedc04. That's basically version 15 of
osbuild with one fix related to secrets.
2020-05-28 00:31:30 +02:00
Ondřej Budai
62b3cd68d7 makefile: change the indentation in release target
c886e1be changed the indentation of the NEWS file. This commit changes also
the indentation in the news template produced by `make release`.

Also - in front of the Location was changed to — to please David.
2020-05-28 00:31:30 +02:00
Ondřej Budai
7e12a55455 news: Add missing bcl from the version 12 2020-05-28 00:31:30 +02:00
Tom Gundersen
d9ae1f9204 rpmmd/FetchMedata: fix checksum map
This is currently only used in tests. We broke backwards compatibility
when we split the repo name and the repo id. The tests expect this to be
indexed by name, not index.

This should probably be reworked or dropped, for now revert to the old
behavior.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-28 00:23:54 +02:00
Jacob Kozol
04e3d0ba7b osbuild: update submodule to 9cbedc049671bf2a2eb50caa2759d91acbf7c7f5 2020-05-28 00:23:54 +02:00
Jacob Kozol
2faf8b3861 test/cases: update distro tests
The distro tests will continue to use the internal repos. But, the
manifest should use the new urls format and have the url inside a
map.
2020-05-28 00:23:54 +02:00
Tom Gundersen
b2cd76ef69 rpmmd/RHSMSecrets: don't store as global variable
Read in when instantiating the rpmmd object, and cache it there instead.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-28 00:23:54 +02:00
Tom Gundersen
a4ebf1cd28 rpmmd/repository: reinstate metadata_expire
This is saved on disk, so needs to be read in and passed on.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-28 00:23:54 +02:00
Tom Gundersen
116a7605dd rpmmd: make the on-disk format match the use
The 'id' field in the json files is exposed as 'name' in the weldr API,
and that is now its only use. Rename it to match.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-28 00:23:54 +02:00
Tom Gundersen
48079b3a4d rpmmd/RepoConf: rename Id to Name
This is how it is used in the rest of the code, as a name to represent
the repository in the weldr API. Rename to match its use, and avoid
confusion with the ID passed to dnf-json, which is not the same.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-28 00:23:54 +02:00
Tom Gundersen
00483101c6 rpmmd/Depsolve: rely on the RepoID being the offset into the array
We don't have to go via a map, and also don't have to rely on the ID's
in the passed in repos are the same, by instead relying on RepoID being
the offset into the passed in repos array, as set in the previous patch.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-28 00:23:54 +02:00
Tom Gundersen
3edada6664 rpmmd/toDNFRepoConfig: use an increasing counter for the ID
The ID needs only be unique in the calling function, so replace the UUID
with an increasing counter.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-28 00:23:54 +02:00
Jacob Kozol
5938276d3b osbuild: update files source for rhsm
osbuild manifest's urls should now contain a url and any secrets. The
secrets should contain a name.
2020-05-28 00:23:54 +02:00
Jacob Kozol
387e8c40aa repos: switch rhel-8 repos from nightlies to cdn
The rhel-8 repositories json file is updated to use the cdn instead of
the nightly vpn. Systems running RHEL 8 must have a subscription now.
2020-05-28 00:23:54 +02:00
Jacob Kozol
d19ffb4eb9 rpmmd: add ssl fields to RepoConfig
rpmmd now contains three repo config structs. RepoConfig is for use
throughout osbuild-composer. repository is for reading from our
internal json repository files before creating a RepoConfig.
dnfRepoConfig is is for use within rpmmd and it matches what dnf-json
expects from a repo.

Repos now contain support for rhsm. In order to connect to rhel's
cdn, dnf must pass an sslcacert, sslcakey, and sslclientcert. When a
repo is used for fetching metadata or depsolving it is checked for
rhsm secrets. If secrets are needed they are retrieved from the host
system. Packages requiring rhsm are marked as requiring
"org.osbuild.rhsm" secrets.
2020-05-28 00:23:54 +02:00
Jacob Kozol
8750dc467b dnf-json: add ssl certs to repo
If a repo passed to dnf-json contains an sslcacert, sslclientkey, or
sslclientcert then dnf-json will include those values in that repo in
the dnf base.
2020-05-28 00:23:54 +02:00
Major Hayden
18fe05f686 CI: Use RHEL 8 CDN subscribed instance
Replace RHEL 8.2 nightly images with a RHEL 8 CDN subscribed instance so
we can test with the exact content a customer would have.

Signed-off-by: Major Hayden <major@redhat.com>
2020-05-28 00:22:25 +02:00
Lars Karlitski
5299e7e242 image-types/rhel8: add kvm, azure, and vmware
These are fairly bare-bones right now, but include them for
completeness.
2020-05-27 22:47:14 +02:00
Lars Karlitski
02c913cf92 image-types/rhel8: streamline requirements
We don't need to specify these images again, but only explain some
non-obvious choices about deviation from the requirements.
2020-05-27 22:47:14 +02:00
Lars Karlitski
b2fc0a384c image-types: remove owners 2020-05-27 22:47:14 +02:00
Tom Gundersen
e66368b099 distro: drop fedora-30
We never shipped into fedora-30 and it is now EOL.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-27 16:20:35 +02:00
Tom Gundersen
e2c14b9fe6 distro: drop legacy image types
We do not properly test, and do not have properly defined use-cases for
the ext4-filesystem, partitioned-disk, nor tar image types. Drop them to
focus on delivering the things we car properly test.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-27 16:20:35 +02:00
Lars Karlitski
e06076ed8c worker: add JobArtifact() and DeleteJobArtifacts()
This allows removing the `artifactsDir` from `weldr.API`. It makes more
sense to deal with that directory in one place only.
2020-05-27 16:16:05 +02:00
Major Hayden
b452a31eed Remove CacheDirectory for remote worker
Setting the cache directory to `/var/cache` causes problems on the
remote worker startup and it needs to be removed.

Signed-off-by: Major Hayden <major@redhat.com>
2020-05-26 15:46:29 -05:00
Major Hayden
0921643fa3 Add health check at /status
There are times where it would be good to monitor that osbuild-composer
is up and running. Add a very simple status check that always returns
200/OK. This can be expanded later to verify that other parts of
osbuild-composer are working properly.

Signed-off-by: Major Hayden <major@redhat.com>
2020-05-26 15:34:16 -05:00
Lars Karlitski
e503b0a4d4 worker: pass a temporary store to osbuild
When fdd753615 added `--output-directory` to the invocation of osbuild,
it also removed `--store`.

This was a mistake: osbuild's default store is `.osbuild`, which is not
what we want. Restore the old behavior of passing a temporary directory.
2020-05-26 22:16:52 +02:00
Lars Karlitski
b065d8c304 worker: handle error in defer
There's not much to do when removing the temporary directory fails.
Print a message so that people have the chance to notice.
2020-05-26 22:16:52 +02:00
Lars Karlitski
a1cf3984dc worker: introduce job artifact directory
The `jobs/:job_id/builds/:build_id/image` route was awkward: the
`:jobid` was actually weldr's compose id and `:build_id` was always `0`.

Change it to `jobs/:job_id/artifacts/:name`, where `:job_id` is now a
job id, and `:name` is the name of the artifact to upload. In the
future, it could support uploading more than one artifact.

This allows removing outputs from `store`, which is now back to being a
pure JSON-store. Take care that `weldr` returns (and deletes) images
from the new (or for backwards compatibility, the old) location.

The `org.osbuild.local` target continues to exist as a marker for the
worker to know whether it should upload artifacts.
2020-05-26 10:42:20 +02:00
Tom Gundersen
8f7a9b3439 weldr/compose: add support for ostree-specific options
Similarly to how some image types accept a size-argument, some image
types require  ostree-specific options. For now this is only used by
Fedora IoT on Fedora 32.

composer-cli nor cockpit-composer currently has the ability to pass in
these options, but the change is backwards compatible and existing image
types are unaffected. If the options are omitted for image types that
require them, sensible defaults are chosen.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-25 22:37:29 +02:00
Tom Gundersen
b0cfec767a distro/f32: add iot-commit image type
This produces a Fedora IoT commit tarball, based on
https://github.com/gicmo/ostree-osbuild-demo and the fedora 32 specific
configuration found at
https://pagure.io/fedora-iot/ostree/c/465c8a2c1d1c0cf4b762477af87fb1d82c8ee002?branch=master

The usbguard package is excluded due to some SELinux failure that
remains to be resolved.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-25 22:37:29 +02:00
Tom Gundersen
77ea9d67bf distro/f32: pass ImageOptions to assemblers
Prepare for assemblers needing more image-type specific options.

This is not a functional change.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-25 22:37:29 +02:00
Tom Gundersen
0b3702cb19 distro/Manifest: generalize the size argument
When generating an osbuild manifest for an image type, we take a
customizations struct, which specifies the image-type-independent
customizations to apply. We also take the size argument, which is
specific to the image build and not part of the blueprint.

Introduce a new argument ImageOptions, which for now just wraps the size
argument. These options are specific to the image build/type, and
therefore does not belong with the other customizations.

For now this is a non-functional change, but follow-up commits will
introduce more types of image options.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-25 22:37:29 +02:00
Tom Gundersen
a544fa7eb6 osbuild: add rpm-ostree stage
This adds the wrapper object and a simple test for the rpm-ostree stage.

See the osbuild documentation for details.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-25 22:37:29 +02:00
Tom Gundersen
1c9cb20b77 osbuild: add ostree.commit assembler
This adds the wrapper object and a simple test for the ostree.commit
assembler.

See the osbuild documentation for details.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-05-25 22:37:29 +02:00