Commit graph

3927 commits

Author SHA1 Message Date
Achilleas Koutsou
8be2c5c89a disk: test minimum directory size calculations
/ and /usr have minimum sizes defined (1 GiB and 2 GiB respectively).
When /usr is not defined, the minimum size of /usr gets added to the
minimum size for /.
This new test runs through a few scenarios and checks whether the sizes
fit.
2022-04-29 08:55:36 +02:00
Achilleas Koutsou
b8cd4bb839 disk: add an empty blueprint to the tests
Adjusted tests accordingly
2022-04-29 08:55:36 +02:00
Achilleas Koutsou
ae9960b8db disk: test that all mountpoints are the minimum 1 GiB
This tests that the clampFSSize() function ensures all user-defined
mountpoints are at least 1 GiB.
Added a blueprint with < 1 GiB minsizes to test this.
Testing all blueprints in TestCreatePartitionTable() now.
2022-04-29 08:55:36 +02:00
Achilleas Koutsou
61be84d634 disk: define blueprint map for test
Convenient for iterating and to define more cases.
2022-04-29 08:55:36 +02:00
Achilleas Koutsou
9920d798f5 disk: remove special case for /usr in clampFSSize
The `/usr` case is now handled by the `EnsureDirectorySizes()` function.
2022-04-29 08:55:36 +02:00
Achilleas Koutsou
275e60f199 disk: add default minimum directory sizes
Currently, we only specify a minimum size for
- `/` (1 GiB), and
- `/usr` (2 GiB).

This ensures that
- a separate `/usr` partition is at least 2 GiB,
- `/` is always at least 1 GiB,
- if `/usr` it not a separate partition, `/` is at least 3 GiB.

We could (or should), in the future, make it possible for image types to
override this mapping as part of their default config, for example, if
an image type by default requires a larger `/usr`.
2022-04-29 08:55:36 +02:00
Achilleas Koutsou
efa10e56e1 disk: use constants for size units in tests
Makes the test values more readable (without needing comments).
Some values in the default partition table were fixed, e.g., cases where
we had `Size: 1024000, // 500 MB`.
2022-04-29 08:55:36 +02:00
Christian Kellner
b365b18390 disk: fix ensureLVM for partition tables without /boot
When the partition table did not have a boot partition, we created it
but then _unconditionally_ returned, which meant that we did not create
the LVM skeleton and wrap the root partition. Properly handle this case
and also re-initialize the `rootPath` in this case since we change the
underlying `Partition[]` array in `PartitionTable` object. Add an extra
blueprint with only one customization which exposes this bug.

Co-Authored-By: Achilleas Koutsou <achilleas@koutsou.net>
2022-04-29 08:55:36 +02:00
Sanne Raymaekers
e3f812f6fd Schutzfile: Pin osbuild version to use minimal required caps 2022-04-28 16:38:42 +02:00
Tomas Hozza
81fc032a45 RHEL-90: don't pass blueprint package set separately to pipelines
Blueprint package set is now depsolved together with the OS package set
in a chain. The result is stored in the package specs sets under the OS
package set name.

In reality, the code was able to handle a `nil` package specs to be
passed to pipelines, however some parts were looking for the kernel
version in the blueprint package specs, which would be a bug.

Regenerated affected image test cases.
2022-04-28 14:42:49 +02:00
Tomas Hozza
e6a20699f9 RHEL-90beta: don't pass blueprint package set separately to pipelines
Blueprint package set is now depsolved together with the OS package set
in a chain. The result is stored in the package specs sets under the OS
package set name.

In reality, the code was able to handle a `nil` package specs to be
passed to pipelines, however some parts were looking for the kernel
version in the blueprint package specs, which would be a bug.

Regenerated affected image test cases.
2022-04-28 14:42:49 +02:00
Tomas Hozza
c26a1839d1 RHEL-86: don't pass blueprint package set separately to pipelines
Blueprint package set is now depsolved together with the OS package set
in a chain. The result is stored in the package specs sets under the OS
package set name.

In reality, the code was able to handle a `nil` package specs to be
passed to pipelines, however some parts were looking for the kernel
version in the blueprint package specs, which would be a bug.

Regenerated affected image test cases.
2022-04-28 14:42:49 +02:00
Tomas Hozza
4c707663f2 RHEL-85: don't pass blueprint package set separately to pipelines
Blueprint package set is now depsolved together with the OS package set
in a chain. The result is stored in the package specs sets under the OS
package set name.

In reality, the code was able to handle a `nil` package specs to be
passed to pipelines, however some parts were looking for the kernel
version in the blueprint package specs, which would be a bug.

Regenerated affected image test cases.
2022-04-28 14:42:49 +02:00
Tomas Hozza
800ac5fe0b Weldr: use rpmmd.DepsolvePackageSets() in compose handler
Use `rpmmd.DepsolvePackageSets()` in Weldr API compose request handler,
instead of `rpmmd.Depsolve()`.

Extract common code from `API.allRepositories()` and
`API.allRepositoriesByImageType()` to a new method
`API.payloadRepositories()`.

Modify `API.allRepositoriesByImageType()` to return payload repositories
(repositories defined by user) as a separate slice to enable the use of
`rpmmd.DepsolvePackageSets()`, which requires the package-set-specific
repositories to be passed separately.

Keep using `rpmmd.Depsolve()` in Weldr where appropriate. The
implementation depsolves various simple package sets for multiple API
request handlers and it does not make sense to complicate the code by
moving to `rpmmd.DepsolvePackageSets()`.
2022-04-28 14:42:49 +02:00
Tomas Hozza
21c6c452ef TestDistro: return dummy values from PackageSets related methods
Return dummy values from the following methods:
 - PackageSets
 - PayloadPackageSets
 - PackageSetsChains

Use package set names commonly used by recent distro definitions.
Package sets are based on values used by rpmmd mock implementation.

Adjust two Weldr API unit test check for the dummy values. Without
this fix, these unit tests would start failing after the move to
`rpmmd.DepsolvePackageSets()` in Weldr API compose handler.
2022-04-28 14:42:49 +02:00
Tomas Hozza
42b3512b12 Delete TestImageType methods not required by ImageType interface 2022-04-28 14:42:49 +02:00
Tomas Hozza
b92196a056 kojiapi: use DepsolvePackageSets instead of Depsolve 2022-04-28 14:42:49 +02:00
Tomas Hozza
694230b453 TestDistro_Manifest: use DepsolvePackageSets instead of Depsolve 2022-04-28 14:42:49 +02:00
Tomas Hozza
e819e08098 worker: extend the depsolve job to use DepsolvePackageSets()
Extend the `DepsolveJob` worker job argument to contain package sets
chains and use `DepsolvePackageSets()` for depsolving.
2022-04-28 14:42:49 +02:00
Tomas Hozza
ac8b0b211c osbuild-store-dump: use DepsolvePackageSets instead of Depsolve 2022-04-28 14:42:49 +02:00
Tomas Hozza
906e88ea8c osbuild-pipeline: use DepsolvePackageSets instead of Depsolve 2022-04-28 14:42:49 +02:00
Tomas Hozza
ef4db9edda rpmmd: introduce DepsolvePackageSets() to the RPMMD interface
Add a convenience method `DepsolvePackageSets()` to the `RPMMD`
interface. The method is expected to depsolve all provided package sets
in a chain or separately, based on the provided arguments, and return
depsolved PackageSpecs sets.

The intention is to have a single implementation of how are package sets
depsolved and then use it from all places in composer (API and tools
implementations).

Adjust necessary mock implementations and add a unit test testing the
new interface method implementation.
2022-04-28 14:42:49 +02:00
Tomas Hozza
fee529cd0a distro: introduce PackageSetsChains() method to ImageType
Introduce a new method `PackageSetsChains()` to the `ImageType`
interface, which returns a named lists of package sets, which should be
depolved together in a chain.

Extend all distro implementations with the new method.

Add a unit test ensuring that if an image type defines some package set
name chains, that all of the listed package set names are present in the
package set map returned by the image type.

The method is currently not used anywhere. This is a preparation for
switching from current way of depsolving to the chain depsolving.
2022-04-28 14:42:49 +02:00
Tomas Hozza
04d45a0fce rpmmd: switch Depsolve() internal implementation to chainDepsolve()
Replace the `rpmmdImpl.Depsolve()` internal implementation by a call to
`rpmmdImpl.chainDepsolve()`.
2022-04-28 14:42:49 +02:00
Tomas Hozza
d48da99a12 rpmmd/dnf-json: support chain dependency solving
Add a new `rpmmdImpl` method `chainDepsolve`, which is able to
depsolve multiple chained package sets as separate DNF transactions
layered on top of each other.

This new method allows to depsolve the `blueprint` package set on top of
the base image package set (usually called `packages`).

Introduce a helper function `chainPackageSets` for constructing
arguments to the `chainDepsolve` method based on the provided arguments:
 - slice of package set names to chain as transactions
 - map of package sets
 - slice of system repositories used by all package sets
 - map of package-set-specific repositories

Extend `dnf-json` with a new command `chain-depsolve` allowing to
depsolve multiple transaction in a row, layered on top of each other.

Add unit tests where appropriate.
2022-04-28 14:42:49 +02:00
Achilleas Koutsou
0f0b2072d5 test: update test manifests
Update GPG keys in all RHEL 8.x test manifests.
2022-04-28 09:50:27 +02:00
Achilleas Koutsou
81c62204b9 repositories: update gpg key for all RHEL 8.x repos
Follow-up to 60db6ad06f

The SHA-1 key is no longer supported in RHEL 9.0.  This isn't a problem
for RHEL 8.x in general, but it prevents cross building RHEL 8.x images
on RHEL 9.0, since the host (RHEL 9.0) rpm and openssl cannot import the
older keys and we fail to bootstrap the build root for the new image if
the source repositories use SHA-1 keys.

Related rhbz#2058497 (Comment 18).

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2022-04-28 09:50:27 +02:00
Achilleas Koutsou
389b7c1848 Unify formatting for all repository json files
- 2 space indent
- lists on multiple lines
- newlines at EOF

This was accomplished by simply running each file through `jq` with no
arguments.
It is also equivalent to Python's `json.dump(..., indent=2)` plus the
added newline.
2022-04-28 09:50:27 +02:00
Achilleas Koutsou
55a2e8ddac disk: add function for resizing Entities based on dirs
New function that ensures that a partition can hold the total sum of all
the required sizes of specific directories on the partition.  The
function sums the required directory sizes grouped by their mountpoint
and then resizes the entity path of that Mountable.
2022-04-27 13:49:19 +02:00
Achilleas Koutsou
ce5a28c113 disk: add function for finding entity path for a dir
New function that returns the Entity path for the Mountpoint that
contains a given directory path.
2022-04-27 13:49:19 +02:00
Achilleas Koutsou
75a0a74064 disk: add docstring for resizeEntityBranch()
resizeEntityBranch() is a private function, but it's not immediately
clear how it works so a docstring is useful for internal development.
2022-04-27 13:49:19 +02:00
Achilleas Koutsou
f4d40e3973 Schutzfile: pin osbuild for new inputs schema
Pinning osbuild to 99abc1373d5c6a85eb1a6ca6eaade0ee3c27ec18 which adds
support for the new array-of-objects inputs schema.
2022-04-27 11:51:25 +02:00
Achilleas Koutsou
2b2181867f osbuild2: use source-array-ref for rpm stage inputs
The (new) source-array-ref object is an array of objects, which allows
for input references to have options (metadata) but still maintain their
order when (de)serialised.

Fixes https://github.com/osbuild/osbuild-composer/issues/2577
Related to https://github.com/osbuild/osbuild/pull/1003
2022-04-27 11:51:25 +02:00
Achilleas Koutsou
26a83262a8 osbuild2: new references format for rpm stage inputs 2022-04-27 11:51:25 +02:00
Achilleas Koutsou
b277d88d93 osbuild2: rename RPMStageReferenes to RPMStageSourceOptions
Matches the equivalent object reference in the osbuild schema:
org.osbuild.files/definitions/source-options
2022-04-27 11:51:25 +02:00
Achilleas Koutsou
0a695b21a1 osbuild2: add some docstrings for rpm input types
A few simple docstrings for the RPM input structs.
2022-04-27 11:51:25 +02:00
Sanne Raymaekers
e258947a58 cloudapi/v2: Fix generated GCP name
The generated gcp name had an invalid `.tar.gz` extension. This
extension still needs to be supplied for the object name however.

The integration tests supply the image name rather than relying in the
generated one, which is why this slipped through.
2022-04-26 13:58:54 +02:00
dependabot[bot]
0eff6c3de7 build(deps): bump google.golang.org/api from 0.74.0 to 0.75.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.74.0 to 0.75.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.74.0...v0.75.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-04-25 15:05:03 +02:00
dependabot[bot]
a4434666d1 build(deps): bump github.com/hashicorp/go-retryablehttp
Bumps [github.com/hashicorp/go-retryablehttp](https://github.com/hashicorp/go-retryablehttp) from 0.7.0 to 0.7.1.
- [Release notes](https://github.com/hashicorp/go-retryablehttp/releases)
- [Commits](https://github.com/hashicorp/go-retryablehttp/compare/v0.7.0...v0.7.1)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-retryablehttp
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-04-25 12:06:02 +02:00
dependabot[bot]
ed96faf9e0 build(deps): bump cloud.google.com/go/cloudbuild from 1.0.0 to 1.2.0
Bumps [cloud.google.com/go/cloudbuild](https://github.com/googleapis/google-cloud-go) from 1.0.0 to 1.2.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/dlp/v1.0.0...dlp/v1.2.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/cloudbuild
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-04-25 10:41:47 +02:00
dependabot[bot]
e8a87572a7 build(deps): bump github.com/Azure/go-autorest/autorest
Bumps [github.com/Azure/go-autorest/autorest](https://github.com/Azure/go-autorest) from 0.11.25 to 0.11.27.
- [Release notes](https://github.com/Azure/go-autorest/releases)
- [Changelog](https://github.com/Azure/go-autorest/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Azure/go-autorest/compare/autorest/v0.11.25...autorest/v0.11.27)

---
updated-dependencies:
- dependency-name: github.com/Azure/go-autorest/autorest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-04-25 10:41:07 +02:00
Alexander Todorov
1e520719cb Don't run unsupported tests on 8.6 and 9.0
https://coreos.slack.com/archives/C0235DZB0DT/p1650536467098679?thread_ts=1650512166.141439&cid=C0235DZB0DT
2022-04-22 15:45:02 +02:00
Tomas Hozza
013eb6c9af RHEL-86/EC2 SAP: install ansible-core instead of ansible
Related to https://bugzilla.redhat.com/show_bug.cgi?id=2077356
2022-04-21 16:20:50 +02:00
Tomas Hozza
d474db38d5 RHEL-86/EC2 SAP: install compat-sap-c++-10
Related to https://bugzilla.redhat.com/show_bug.cgi?id=2074114
2022-04-21 16:20:50 +02:00
Tomas Hozza
082ce34a93 RHEL-90/EC2 SAP: install @Server package group
Related to:
- 9.0: https://bugzilla.redhat.com/show_bug.cgi?id=2076763
- 9.2: https://bugzilla.redhat.com/show_bug.cgi?id=2076761
2022-04-21 16:20:50 +02:00
Tomas Hozza
54ba98f8c1 RHEL-86/EC2 SAP: install @Server package group
Related to https://bugzilla.redhat.com/show_bug.cgi?id=2074107
2022-04-21 16:20:50 +02:00
Tomas Hozza
4bba54b1b3 RHEL-90/EC2 SAP: adjust PAM limits configuration
Related to:
- 9.0: https://bugzilla.redhat.com/show_bug.cgi?id=2076759
- 9.2: https://bugzilla.redhat.com/show_bug.cgi?id=2076760
2022-04-21 16:20:50 +02:00
Tomas Hozza
88fc9f9869 RHEL-86/EC2 SAP: adjust PAM limits configuration
Related to https://bugzilla.redhat.com/show_bug.cgi?id=2074101
2022-04-21 16:20:50 +02:00
schutzbot
c2c1f39d78 Post release version bump
[skip ci]
2022-04-20 13:35:43 +00:00
dependabot[bot]
9336d66c32 build(deps): bump cloud.google.com/go/storage from 1.18.2 to 1.22.0
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.18.2 to 1.22.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/storage/v1.18.2...spanner/v1.22.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-04-20 14:27:18 +02:00