Commit graph

701 commits

Author SHA1 Message Date
Brian C. Lane
eb76b9ec8e weldr: Fix the bad blueprint depsolve response
This changes the response to match lorax-composer's behavior. If any of
the blueprints in the list passed to /blueprints/depsolve/... have an
error that error should be appended to the error list, and the blueprint
included in the blueprints list with an empty dependencies section.

It was returning an error 400 and a single error if it hit any depsolve
problems, skipping any other blueprints and returning the wrong
response.

This also adjusts the tests to account for the change.

Fixes #890
2020-07-28 08:18:56 +02:00
Major Hayden
3daa751095 distro/rhel8: increase default image size to 4GB
RHEL 8.3 requires more than 2GB to install on aarch64/ppc64le.

Signed-off-by: Major Hayden <major@redhat.com>
2020-07-27 10:27:15 -05:00
Martin Sehnoutka
3fc03503a5 distro/f32: manually relabel cp in buildroot with install_t
By labeling `cp` with `system_u:object_r:install_exec_t:s0` we allow it
to copy labels unknown to the host.

See also corresponding commit in osbuild:
e80130a830
2020-07-22 10:47:46 +01:00
Martin Sehnoutka
f8f35016d6 osbuild: Add "labels" field into the SELinux stage
The SELinux stage supports labeling specific files with specific labels
like this:
7b0db90c76/stages/org.osbuild.selinux (L37)

This can be useful for preventing some SELinux issues during the image
build. You can find example usage here:
7b0db90c76/test/data/stages/selinux/test_basic.json (L5)

And more reasoning here:
e80130a830
2020-07-22 10:47:46 +01:00
Martin Sehnoutka
636b761b6b distro/f32: add SELinux RPM to build pipeline
This patch adds SELinux stage to build pipeline for f32 just like
we do for RHEL8. This helps with issues like this:
https://github.com/osbuild/osbuild-composer/issues/798
2020-07-22 10:47:46 +01:00
Martin Sehnoutka
60269abdff distro: print more information in case the test fails
This helps when looking for the issue because the previous input did not
mention which specific qcow2 test case failed.
2020-07-22 10:47:46 +01:00
chloenayon
ae3fb5e1c5 distro: enable cloud-init services for openstack and qcow2 images for fedora
Previously, cloud-init was installed on qcow and openstack images but was not
enabled, preventing a user from logging in to these images. Cloud-init, cloud-final,
cloud-init-local and cloud-config services are enabled to make cloud-init work
and to maintain consistency with official fedora 31 and fedora 32 images.

The image tests have been updated to reflect these changes.

Fixes #698
2020-07-18 11:57:05 +02:00
Lars Karlitski
f38d55da88 distro/tests: use test case filename as test name
These tests used the image type as test name, which is ambiguous. Use
the file name for the test case instead.
2020-07-18 10:52:11 +01:00
Tom Gundersen
fbfa191c81 rcm: drop sub-package
The osbuild-composer-rcm package was never finished, not in use and will be replaced by osbulid-composer-koji.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-07-17 19:13:15 +01:00
Tom Gundersen
bca53a596c crypt: add automatic darwin support
Currently one would have to pass a `macos` flag explicitly to build on
OSX, but if we replace `macos` with `darwin` this happens automatically.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-07-17 07:48:22 +02:00
Ondřej Budai
b4a57225a7 api/weldr: add missing modtime to tar returned by /compose/logs
This file was surely not created on 1970-01-01, this commit sets the ModTime
to the current datetime.
2020-07-15 08:58:25 +01:00
Ondřej Budai
454e471af8 api/weldr: bring tars back from the future
By default, go's tar archiver uses USTAR header format. Unfortunately, this
format doesn't support sub-second resolution for ModTime. Go solves this by
*rounding* the time. Sometimes, this creates an archive containing a file
with modtime from the future. When such archive is untarred by GNU tar,
the following message is produced:

tar: bf548dfd-0a90-40e6-bbf2-dcdd82fcbb4e.json: time stamp 2020-07-13
13:34:31 is 0.356223173 s in the future

We have two options here:

1) Use gnu header format that supports sub-second resolution. Unfortunately,
   it seems that not all tar archivers support this format (e.g. 7-zip).

2) The other option is to truncate the date (instead of rounding).

I went with option 2.

Also, this commit adds a test to check that the header is not from the future.
Without this fix, the test is actually failing, I verified this manually.

Fixes #854
2020-07-15 08:58:25 +01:00
Major Hayden
0f32f35756 📦 Use raw image format for AWS
Although the vhdx format is smaller than raw, AWS often has some issues
booting instances from an imported vhdx snapshot. The raw format seems
to boot consistently at AWS.

Fixes #754.

Signed-off-by: Major Hayden <major@redhat.com>
2020-07-02 13:11:11 -05:00
Tom Gundersen
8c7d8a442b osbuild: pass UID/GID as int
These were passed as strings, which is not what osbuild expects.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-29 16:04:38 +02:00
Tom Gundersen
50d469fe45 distro: replace BasePackages() with Packages()
Rather than getting a set of base packages from the ImageType, and then
appending the requested packages from the blueprint, pass the blueprint
into the new Packages() function, and return the full set of packages to
be depsolved.

This allows us to also append packages based on other customizations
too, and use that to append chrony when the timezone is set. This
matches the behavior anaconda had, and there was a TODO item to do this,
which had been overlooked.

Fixes #787.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-29 16:04:38 +02:00
Tom Gundersen
d31e3ebb65 distro: add groups before users
If a user is added to a group, then the group must be added first.
2020-06-29 16:04:38 +02:00
Christian Kellner
408a704669 distro/{rhel8, fedora32}: no fstab for OSTree
How exactly the final file system layout looks like is determined
by the installer (anaconda) and thus can not be known at commit
creation time. Thus creating an /etc/fstab file is unnecessary and
the information in it probably wrong. The file wont be used though
because it will be overwritten during the installation process.
2020-06-21 16:16:14 +02:00
Brian C. Lane
4f3f09fec5 worker: Check OSBuildOutput for nil before using it
It is possible for it to end up as nil so it needs to be checked
everywhere it is used.
2020-06-12 12:45:46 +02:00
Lars Karlitski
be10efeb1e weldr: implement cancel route 2020-06-12 10:00:50 +02:00
Lars Karlitski
27acd03b68 worker: add API support for canceling jobs 2020-06-12 10:00:50 +02:00
Martin Sehnoutka
1e5c31a7ad distro/rhel8: introduce s390x support
This patch introduces support for s390x architecture. It has some
specific requirements like using kernel-cmdline stage before the RPM
stage because RPM scriptlets require these values to be present in the
system already. It also uses zipl stage and bootloader as opposed to all
the other architectures.
2020-06-11 21:50:09 +02:00
Martin Sehnoutka
d20682fcae osbuild: introduce kernel-cmdline stage
This stage can be used to set kernel boot parameters.
2020-06-11 21:50:09 +02:00
Peter Robinson
f32770fed6 distro/rhel8: add RHEL for Edge
Initial definition for RHEL for edge package set on x86.
This is based marginally on the upstream Fedora IoT package set
but slimmed down and adjusted for RHEL package naming.

Based on Tom's patch.

Signed-off-by: Tom Gundersen <teg@jklm.no>
Signed-off-by: Peter Robinson <pbrobinson@gmail.com>
2020-06-11 20:44:17 +02:00
Tom Gundersen
ab276bdb77 tools/test-case-generator: add more options
Allow individual test-cases or sub-sets of test-cases to be generated
more easily.

We allow explicit skipping of image-info generation (and hence the
osbuild run), and also individual image types to be specified.

Also drop distros and image types that are no longer supported.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-10 19:25:54 +02:00
Tom Gundersen
cbf19d1319 distro/rhel8: label the build root
We require the right SELinux labels to be set on some of the tools we
use, make sure the policy is installed and the labels applied in the
buildroot, in the same way as in the target image.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-10 19:25:54 +02:00
Tom Gundersen
5c3f4193e5 osbuild/rpm: set check_gpg per-package
Rather than passing an array of checksums, pass an array of objects
with package properties. One is the `checksum`, which is required, and
in addition to that we now pass the boolean check_gpg, which indicates
that that RPM should have its GPG signature verified.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-10 19:25:54 +02:00
Tom Gundersen
42de929adb rpmmd/Depsolve: expose whether to check GPG signature for each RPM
Currently each repo is annotated with this information, use the mapping
we have from packages to their originating repos to annotate each
PackageSpec with the same information.

This information is not serialized, and not yet used. It may be used to
expose this information to osbuild's org.osbuild.rpm stage.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-10 19:25:54 +02:00
Tom Gundersen
e9cd6409ef rpmmd/PackageSpec: duplicate the type definition
Use the new dnfPackageSpec only for communicating with dnf-json. This
allows us to drop the RepoID from the public type, as that is only meant
to be used internally in `Depsolve()`.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-10 19:25:54 +02:00
Tom Gundersen
6a7e57ce8d rpmmd: allow check_gpg to be configured for system repos
The sources weldr API already supports this, so hook it up to be
represented on disk and in our internal state tracking too.

This does not yet hook this up to be respected by osbuild, which
currently takes this to be unconditionally set to true.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-10 19:25:54 +02:00
Ondřej Budai
3913f419e2 weldr: drop getPkgNameGlob method
Not used anywhere! :-)
2020-06-10 17:40:30 +02:00
Ondřej Budai
fa0d800850 test: add a test-case to prevent bad multilib depsolves
When gsl with version * was specified in the blueprint,
composer depsolved both x86_64 and i686 version of gsl.
This test case should prevent this from happening.
gsl is used because it has x86_64 and i686 versions on both RHEL and Fedora.
Also, gsl-devel package exists, which is not dependant on gsl and shouldn't
be depsolved.
2020-06-10 17:40:30 +02:00
Ondřej Budai
0796342fa1 weldr/api: get the package list for depsolving in a better way
blueprint.GetPackages() method was used to depsolve blueprints prior the
dnf-to-rpm switch. However, it got dropped during the switch. This commit
makes weldr use it again.

The nice effect of this change is that we can drop getPkgNameGlob function and
have only one function for getting package name-versions from a blueprint.

Also, blueprint.GetPackages() works better with * version. Previously, we had
issues with composer depsolving bash of version * to both x86_64 and i686
versions of bash package. GetPackages() converts the package to name-version
of just bash, which dnf-json correctly depsolves to just one architecture. On
the contrary, the previous method converted bash to name-version bash-*.*.*,
which confused dnf-json.

Note that conversion to bash-* is also wrong because this will cause dnf-json
to install all packages with prefix "bash-*".
2020-06-10 17:40:30 +02:00
Ondřej Budai
6e6eae24eb blueprint: do not append "-" to name-version if version is empty
Previously, bash with empty version was converted to name-version "bash-"
that is obviously ill-formed. This commit fixes it.
2020-06-10 17:40:30 +02:00
Ondřej Budai
224bd6b341 rpmmd: rely on dnf-json shebang instead of using PATH python
RHEL doesn't have python3 in PATH by default. Instead it has /usr/libexec/
platform-python. When building the RPM, RHEL correctly mangles dnf-json's
shebang to the platform python. This commit switches osbuild-composer
to use this shebang because it should always respect target platform's
default python binary.

Fixes #745
2020-06-10 16:56:28 +02:00
Ondřej Budai
b93df4b524 rpmmd: require the path to dnf-json to be passed in
RPMMD had hardcoded path to dnf-json helper. This required all executables
using RPMMD to be run in the directory where dnf-json was located. This commit
makes RPMMD take the path to dnf-json as an argument. This allows its
consumers to specify whichever path they want.

Not a functional change
2020-06-10 16:56:28 +02:00
Tom Gundersen
83a63aaf17 distro/rhel8: add tar image type for all architectures
This is a very minimal image type, which is not expected to do anything
with an empty blueprint. It is documented as a way to install RHEL for
Satellite.

This fixes #720.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-08 13:32:59 +02:00
Tom Gundersen
253379a3b9 tests: use more reliable mirrors for Fedora tests
Move away from the round-robin fedora mirrors, and use the most reliable
one we have found (kernel.org).

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-07 13:33:32 +02:00
Martin Sehnoutka
385df479f3 distro/rhe8: introduce ppc64le qcow2 image type
This patch introduces a qcow2 image type for RHEL8 distro and associated
test case.
2020-06-06 16:56:39 +02:00
Martin Sehnoutka
7979a705b6 osbuild: filesystem is not required member of qemu assembler
The input options of qemu assembler do not require filesystem for every
partition. This patch makes it optional and omits the object in case it
is empty.

osbuild schema definition:
162797f206/assemblers/org.osbuild.qemu (L63)
2020-06-05 20:09:20 +02:00
Martin Sehnoutka
673e967b2d osbuild: Change QEMUPartition "Type" to match osbuild
Partition type, as defined in osbuild qemu assembler, is of type string
because it is either UUID or identifier. See here for details:
162797f206/assemblers/org.osbuild.qemu (L88)

Change type of the "Type" struct member to match the definition in
osbuild. This patch is needed for ppc64le support.
2020-06-05 20:09:20 +02:00
Major Hayden
38e4fe8781 Add more emojis to AWS upload logs
Now that we know emojis work properly with osbuild-composer and systemd,
let's add some more.

Signed-off-by: Major Hayden <major@redhat.com>
2020-06-05 19:56:57 +02:00
Martin Sehnoutka
d74fc4e3fa osbuild: introduce zipl stage
zipl is a z initial program loader used with IBM Z systems. This stage
is required to get support for the s390x architecture.
2020-06-05 19:29:16 +02:00
Ondřej Budai
d65057e164 weldr: add support for toml format in /blueprints/freeze
This is needed for lorax parity. When multiple blueprints are being frozen in
toml mode, the API returns an error. This is the same behaviour as in the
/blueprints/info route.

Fixes #667
2020-06-05 13:51:04 +02:00
Ondřej Budai
14a3f2a9d3 weldr: move projects and modules routes towards lorax parity
lorax-composer returns more information about projects and modules. They are
all meaningless, but we need them for parity.

Fixes #664
2020-06-05 10:01:32 +02:00
Martin Sehnoutka
604f7c2a55 osbuild: introduce bootloader struct into qemu assembler
osbuild takes a "bootloader" object as an option to the qemu assembler:
3f14ace5c1/assemblers/org.osbuild.qemu (L43)
we don't use it because for x86_64 with enabled legacy support it
defaults to the right value:
3f14ace5c1/assemblers/org.osbuild.qemu (L482)
but in order to gain support for ppc64le we need to introduce this.
Example usage can be found in samples directory:
3f14ace5c1/samples/f30-ppc64le.json (L819)

This change itself does not alter osbuild-composer output.
2020-06-05 10:01:24 +02:00
Brian C. Lane
0f5b99c754 Fix fonud typo in rpmmd 2020-06-05 08:27:25 +02:00
Jakub Rusz
f5680284df Increase coverage of store.go
Closes #455
2020-06-04 17:35:42 +02:00
Tom Gundersen
27e2d17911 awsupload: fix printing of IDs
The AWS API always uses pointers to strings everywhere, rather than
plain strings. The log messages got this wrong and printed the pointers
rather than dereferencing them.

Don't require users to do a memory dump to get at the AMIs, but instead
print the actual strings. Also change to %s from %v to let the linter
warn us in the future. %s accepts only strings, whereas %v prints
whatever you throw at it.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-06-04 10:05:12 +02:00
Ondřej Budai
761be27f32 upload/aws: fail ImportSnapshot when the snapshot is deleted
When ImportSnapshot fails, it goes into the deleted state. However, the code
didn't consider this state as a failure. This commit fixes that.

Fixes #706
2020-06-03 21:55:46 +02:00
Brian C. Lane
e37b513560 weldr: Add check for deleting system sources
It was passing it through to the non-system delete function
and not returning an error. This checks for system repos first and
returns a 400, SystemSource error response if it is in the system list.
2020-06-03 21:24:47 +02:00