Commit graph

2299 commits

Author SHA1 Message Date
Achilleas Koutsou
5909ca202d disk: boot partition finder as partitiontable method
Moved the function that searches for the boot partition index to the
PartitionTable struct as a method. The method returns -1 if not found
and it's now the responsibility of the caller to handle the case where
it is not found.
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
bc1b45f7fa qemu stage test squash 2021-07-24 15:54:00 +02:00
Achilleas Koutsou
8fc74d1e73 tools/test-case-generators: update distro map
Added RHEL 8.5 image types to distro-arch-imagetype map
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
35f0ef1053 test: regression test installing excluded dependency
Unlike the original regression test which explicitly names the excluded
dependency, this test installs a package that depends on the excluded
package without naming it. Based on the new way we depsolve blueprint
packages in RHEL 8.5, this should now work as well.
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
2444e633fb test: separate regression from base tests
- Rewrote regression.sh to be like base tests with list of regression
  test scripts to be run and fail counts
- Separate regression tests in CI to have better control of distro
  runners
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
ce0fddf4c2 osbuild2: unit tests for new stages 2021-07-24 15:54:00 +02:00
Achilleas Koutsou
dc62275d4d common: helper function for uint64 ptr 2021-07-24 15:54:00 +02:00
Achilleas Koutsou
cba720e63f rhel85: depsolve blueprint packages separately
To avoid packages specified in a blueprint from conflicting with exclude
lists, we depsolve blueprint packages separately and pass them into the
Manifest generator under the new "blueprint" package set key.

This approach has the added benefit that dependencies of packages
specified in the blueprint are not subject to exclusion in addition to
the explicitly named packages.

The OS pipeline which installs the packages for the base system merges
the two package sets before running the RPM stage. The signature of the
function is changed to explicitly require blueprint packages be
specified (though `nil` or empty slice is valid).

The kernel selection test is adapted to merge the package sets before
counting kernel package.

Adaptation of changes in
https://github.com/osbuild/osbuild-composer/pull/1349
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
cc0f75b38f CI: remove rhel-8.4 runner for base tests 2021-07-24 15:54:00 +02:00
Achilleas Koutsou
190f473b04 distro/rhel85: update unit tests 2021-07-24 15:54:00 +02:00
Achilleas Koutsou
d6ec5aa59e distro/rhel85: update ami package set
Based on http://git.app.eng.bos.redhat.com/git/spin-kickstarts.git/tree/rhel8/rhel-8.5-ec2.ks
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
b8d34d7679 distro/rhel85: introduce the ami image type
This is a copy of the ami image type from RHEL 8.4.
Small modifications are necessary.
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
1f005a55b9 distro/rhel85: don't always install bootloader
Don't install bootloader unless `platform` (imageType.arch.legacy) is
defined.
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
9c3c956655 distro/rhel85: aarch64 uefi: true 2021-07-24 15:54:00 +02:00
Achilleas Koutsou
3f0f4ffab9 distro/rhel85: simplify edge types and package sets
- Cleaned up distro-specific edge package sets
- Added edge package set merging in PackageSets() function
- Edge image type definitions are no longer arch specific, just like the
  other image types
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
44a22bbb14 test/data: add aarch64 repos for RHEL 8.5 2021-07-24 15:54:00 +02:00
Achilleas Koutsou
78920efc36 distro/rhel85: partition table fixes
- Automatically find /boot partition (fallback to root)
- Partition sizes need a bit of a shrink
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
cd030c0bd6 CI: enable base tests on RHEL 8.5 2021-07-24 15:54:00 +02:00
Achilleas Koutsou
876a67ccb0 distro/rhel85: introduce the openstack image type 2021-07-24 15:54:00 +02:00
Achilleas Koutsou
ceba83dcca distro/rhel85: introduce the vmdk image type 2021-07-24 15:54:00 +02:00
Achilleas Koutsou
2ef462bdaa distro/rhel85: add dhcp-client to vhd packages
Is no longer installed by default but is required to boot in Azure
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
1ed9008785 distro/rhel85: introduce the vhd image type 2021-07-24 15:54:00 +02:00
Achilleas Koutsou
3eae16badb distro/rhel85: remove installer type for aarch64
The installer image type needs package definitions and testing before it
can be included in aarch64.
Removing for now.
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
f09bf063c8 distro/rhel85: tidy up package sets
- Move all package set definitions to package_sets.go file.
- Do not append image package sets to arch and distro sets; this happens
  automatically in PackageSets() method.
- Fix architecture-specific package sets (boot and build where
  available).
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
dec74dba32 distro/rhel85: add image and qcow pipelines
Live image pipeline: Creates the live image in a file through a loopback
device.

Stages:
- truncate: create the file to hold the image
- sfdisk: partition the device
- mkfs.fat, mkfs.xfs: create the filesystems
- copy: copy the tree from the previous pipeline (the OS pipeline) into
  the directories where the partitions are mounted
- grub2.inst: install the bootloader

QEMU pipeline: Convert the live image from the previous pipeline to a
qcow2 image.
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
6debb62758 osbuild2: separate input base types into new file
Keep input base types / interfaces separately from stages for
readability.
Cleaned up some unused interfaces in the process.
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
e85fc3b48c osbuild2: new stages and osbuild features
Stages:
- org.osbuild.copy
- org.osbuild.truncate
- org.osbuild.sfdisk
- org.osbuild.qemu
- org.osbuild.mkfs.btrfs
- org.osbuild.mkfs.ext4
- org.osbuild.mkfs.fat
- org.osbuild.mkfs.xfs
- org.osbuild.grub2.inst

Stages can now have devices and mounts in addition to options and
inputs.

Devices:
- org.osbuild.loopback

Mounts:
- org.osbuild.btrfs
- org.osbuild.ext4
- org.osbuild.fat
- org.osbuild.xfs
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
4cf26bb628 distro/rhel85: split coreStages() implementation
coreStages() was meant to produce the stages that are common between all
image types. Since there are too many exceptions and differences between
traditional and edge image types, keep the two pipelines separately
instead.

Moved SELinux stages to the end of each pipeline.
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
39371b01ba distro/rhel85: document selinuxStageOptions arg
Renamed the argument to have clearer (and correct) meaning and added a
function docstring to describe the purpose.

The argument is set to 'true' in the build pipeline.
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
708de084aa distro/rhel85: add grub2 stage options 2021-07-24 15:54:00 +02:00
Achilleas Koutsou
bf61ca422f distro/rhel85: introduce the qcow2 image type
- distro definition
- qcow2 package set
- pipeline generation function (unfinished)
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
23de460a66 disk: generate fstab stage options for osbuild v2
Same function as v1 but generates v2 struct for v2 pipelines
2021-07-24 15:54:00 +02:00
Achilleas Koutsou
fd4a8387f7 distro/rhel85: partition table generator function
Copied from RHEL 8.4
2021-07-24 15:54:00 +02:00
Chloe Kaubisch
4c800f29a7 worker: add metrics
use prometheus to gather metrics
2021-07-23 21:54:28 +02:00
Brian C. Lane
b5987a5ca5 Add repository file for RHEL 8.4 2021-07-22 15:34:32 +01:00
Ondřej Budai
e1462e261d mockbuild: add rhel 9
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-07-22 16:46:58 +03:00
Ondřej Budai
3ec917062f ci: upload rpms built in RHEL 8 CDN buildroot into rhel-8-cdn directory
Let's explain how RPMs for RHEL are built:

We use a subscribed RHEL 8.x machine and mock build these on it. Mock
initializes its own buildroot based on the latest RHEL 8 CDN content, see[1].
This means that the minor version of the buildroot is independent of the minor
version of the host.

However, we currently upload RPMs to a directory whose name consists also of
the minor version of the host. Our hosts are currently running RHEL 8.3 so
the RPMs are uploaded into rhel-8.3 directory despite them being built in the
RHEL 8.4 buildroot (RHEL 8 CDN buildroot specifically). This means that
we cannot guarantee that they are installable on RHEL 8.3 which is weird.

This commit adds a special case for hosts that run on subscribed RHEL and
thus build RPMs in a buildroot constructed from RHEL CDN. These RPMs are
now uploaded into rhel-8-cdn directory. This change more accurately reflects
the way we build our RPMs and removes some confusion.

Also, we need to bump osbuild commit so we have a version that already has
the rhel-8-cdn change in it.

[1]: https://github.com/rpm-software-management/mock/blob/main/mock-core-configs/etc/mock/templates/rhel-8.tpl#L37

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-07-21 20:02:03 +02:00
Ondřej Budai
298526d2af mockbuild: register the instance at the beginning
So we can check if the machine is subscribed in the script.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-07-21 20:02:03 +02:00
Thomas Lavocat
cfc91a3eb6 CloudAPI: Invalid package set returns 400
Previously a bad error code was returned, fixes #1477.

Testing:

I have two test cases to test the solution. The first is a request that
makes depsolve crash by changing the dnf-json script by an almost empty
one that only throws an exception. The second one fails because it
requests a non existing package. The former ends with a 500 error and
the later with a 400.

----8<-----

HTTP/1.1 500 Internal Server Error

Failed to depsolve base packages for ami/x86_64/centos-8: ailed to
depsolve base packages for ami/x86_64/centos-8: unexpected end of JSON
input

----8<-----

HTTP/1.1 400 Bad Request

Content-Length: 226Failed to depsolve base packages for
ami/x86_64/centos-8: DNF error occured: MarkingErrors: Error occurred
when marking packages for installation: Problems in request:
missing packages: jesuisunpaquetquinexistepas_idonotexist
2021-07-21 16:21:15 +02:00
Jakub Rusz
76b2579c4b CI: update terraform sha
This change contains increased Openstack runners limit. This will
help running the CI more smoothly.
2021-07-21 13:04:51 +02:00
Achilleas Koutsou
583bfc0b4a test/cases/ostree: save build logs and manifests
Copy log and manifest for each build to the ci-artifacts/ directory to
be stored at the end of the build for troubleshooting.
2021-07-21 11:21:42 +02:00
Achilleas Koutsou
720ffbfbb7 ci: save everything under ci-artifacts
A general purpose artifact save directory where test scripts can add
files to be saved.
2021-07-21 11:21:42 +02:00
Martin Sehnoutka
b7df6ac627 devcontainer: introduce VSCode dev container
Introduce Dockerfile and devcontainer.json specifying how to build and
run a containerized development environment in VSCode.

The configuration is inspired by the osbuild repo where a similar
configuration directory already exists.
2021-07-20 23:04:41 +02:00
Yuxin Sun
471353db60 test: Add Fedora 33 ostree test back
Signed-off-by: Yuxin Sun <yuxisun@redhat.com>
2021-07-20 21:18:02 +02:00
Yuxin Sun
c09638efe5 test: Fedora error/fail logs should not be a must
Signed-off-by: Yuxin Sun <yuxisun@redhat.com>
2021-07-20 09:56:38 +03:00
Tomas Hozza
044fa0ffaf news: document changes to osbuild integration done in PR#1525
Add news entry for changes merged as part of PR#1525 [1].

[1] https://github.com/osbuild/osbuild-composer/pull/1525

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-07-19 21:37:15 +02:00
Tomas Hozza
5359ff8da7 osbuild2: use type aliases for unexported types to prevent recursion
Some osbuild stages added as part of PR#1525 declare unexported types
which are complete copies of types with custom MarshalJSON method,
 to prevent recursion when marshalling to JSON.

Modify relevant osbuild stages to use type alias instead of declaring a
complete type copy.

https://github.com/osbuild/osbuild-composer/pull/1525

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-07-19 21:37:15 +02:00
Tomas Hozza
2be664de38 osbuild2: rename profile_id to profile in authselect stage
Reflect https://github.com/osbuild/osbuild/pull/729 in osbuild-composer.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-07-19 21:37:15 +02:00
sanne
234c51e562 distribution: Spell osbuild-composer correctly 2021-07-16 12:14:46 +02:00
sanne
7555ed7556 worker: Merge configs 2021-07-16 12:14:46 +02:00