Commit graph

3365 commits

Author SHA1 Message Date
Christian Kellner
4f474f1661 disk: introduce ContainsMountpoint helper
Simple wrapper around FindFilesystemForMountpoint that will
return a boolean if a filesystem with the given mountpoint
is defined in the partition table.
2022-02-22 19:23:41 +00:00
Christian Kellner
37d912529c disk: add ForEachFilesystem helper
Add helper that iterates over all filesystems in the partition
table and calls the callback on each one.
Add simple checks for it as well.
2022-02-22 19:23:41 +00:00
Christian Kellner
74f6c59c93 disk: remove RootPartitionIndex
This method was only used by `updateRootPartition` which got
removed as well so this method is unused now.
2022-02-22 19:23:41 +00:00
Christian Kellner
fb7f92aa95 disk: clone partition table before modifying it
The partition table is modified `CreatePartitionTable`, which is
not a problem for the table itself since it is currently passed
by value. However, all shallow copies share the same file system
pointers and `CreatePartitionTable` will modify those as well.
As there is a data race where two concurrent thread modify the
content of the Fileystem object at the same time before writing
the uuids out to the manifest via the stage options and thus the
resulting manifest would be broken.
Therefore we use the new `Clone` methods to make a dee@ copy of
the `PartitionTable` object in the `CreatePartitionTable` method;
This will allow us to pass the `basePartitionTable` object by val
and also return the resulting `PartitionTable` as a pointer.
2022-02-22 19:23:41 +00:00
Christian Kellner
d7df6b3d5c disk: add Clone helpers
Add `Clone` methods to Filesystem and PartitionTable. This can be
used to create a new PartitionTable that should be modified based
on the original.
2022-02-22 19:23:41 +00:00
Christian Kellner
34781214cf disk: methods of Filesystem take it by ref
Convert all methods of `Filesystem` to pass it by reference
not by value.
2022-02-22 19:23:41 +00:00
Christian Kellner
5bf4892077 disk: use pointer receiver for PartitionTable
The `PartitionTable` object contains an array of `Partitions`, which
is shared among all shallow copies of `PartitionTable` instances, as
when it is used via a value receiver in method calls. The individual
objects are thus not fully isolated since modifications of array
elements, or in some cases modifications to the array itself, are
shared and visible between all shallow `PartitionTable` copies. Thus
use a pointer receiver for all methods of `PartitionTable` to make
it explicit that modifications will affect other objects.
2022-02-22 19:23:41 +00:00
Christian Kellner
d415d57f95 disk: create{Partition -> Filesystem}
What this function is actually doing is to create a filesystem,
together with a new partition; for LVM that filesystem could
also be created on a new logical volume though.
2022-02-22 19:23:41 +00:00
Christian Kellner
5693045916 disk: createPartition now appends also it
It makes sense that the function that creates the partition also
adds it to the partition table.
2022-02-22 19:23:41 +00:00
Christian Kellner
147e0c966b disk: remove updateRootPartition
The previous commit made sure that `PartitionTable.RootPartition`
is indeed returning the pointer partition object inside the array
of the `PartitionTable`. Thus changes to the returned object will
now directly affect the object and thus there is no need to call
`PartitionTable.updateRootPartition` anymore.
2022-02-22 19:23:41 +00:00
Christian Kellner
6333d54091 disk: return the correct ptr from RootPartition
When we iterate over the partitions to find the root partition,
we are getting the partition by value so even if we return a
pointer to that then, it is actually a pointer duplicated object.
Return the pointer to the actual partition object instead.
2022-02-22 19:23:41 +00:00
Christian Kellner
c8efc7d282 disk: create file system accessors and use those
In various places we are interested in the root and boot file-
systems. Currently those were accessed via by retrieving the
partition that contain them and the accessing the filesystem
member. Add accessors to `PartitionTable` that directly return
the needed filesystem. This will help if the file system is
stored inside a container like LVM or LUKS instead of directly
on a partition.
2022-02-22 19:23:41 +00:00
Sanne Raymaekers
d589317dcb internal/cloudapi: Log error in manifest job 2022-02-22 14:06:05 +01:00
Djebran Lezzoum
d8fdb03373 dnf-json: Add repo_id to dump package.
The dump function will be used to search packages and as we are implementing third party repositories, adding the repo_id of the package will allow us to identify the package repository used.
2022-02-21 15:46:53 +01:00
Sanne Raymaekers
c4ecbea510 internal/cloud: Allow aws creds from defaults
Defaults according to https://docs.aws.amazon.com/sdk-for-go/api/aws/#Config:
Defaults to a chain of credential providers to search for credentials in
environment variables, shared credential file, and EC2 Instance Roles.

If nothing is specified fall back to whatever instance role.
2022-02-21 15:43:53 +01:00
Ondřej Budai
17e809b5f7 worker: use default transport instead of "blank" one
This allows us to use the default behaviour of http.DefaultTransport
to honor HTTP_PROXY.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-02-21 14:46:49 +01:00
Jakub Rusz
f0f80bd930 ci: update terraform SHA
This brings AWS internal subnet balancing which allows to double our CI
runners in AWS.
2022-02-21 12:21:13 +01:00
Jakub Rusz
f9fb593da3 ci: large runners for some ostree tests
The longer running ostree tests (all except ostree.sh) benefit from
running on large runners because of their long runtime. Changing all to
large runners. Should not be a problem with our increased Openstack
quota.
2022-02-21 12:02:55 +01:00
Tom Gundersen
bb486bcccb cloudapi/v2/compose: don't overwrite customizations struct
The customizations struct contains different kinds of customizations,
such as users and filesystems. If both filesystems and users are
configured, the current code overwrites the user configuration.

Change this by unconditionally allocating the customizations struct
when creating the empty blueprint, and ammending it for each
customization we find.

This means that the customizations will be non-nil, also when it is
empty, which was not the case in the past. This should not change
the behavior of the code.
2022-02-19 11:13:55 +00:00
Tom Gundersen
351d06aecf tests/cases/api.sh: make /var at least 256MB
We (mistakenly) don't enforce a minimum size for /var,
so setting it to 1024 (1kB) causes the image build to fail.

CI does not expose this in a helpful way at the moment,
so this is a bit tricky to debug.

Also skip customizations for the AWS.S3 upload type. Not all the
image types with this upload type support filesystem customizations
and that's as expected. We could make a more fine-grained test in
the future, but testing with a coulpe of targets should be
sufficient.
2022-02-19 11:13:55 +00:00
Chloe Kaubisch
0619ffa588 cloudapi: expose filesystem customizations 2022-02-19 11:13:55 +00:00
ondrejbudai
9c1f552210 Post release version bump
[skip ci]
2022-02-18 18:36:49 +00:00
Ondřej Budai
10c0891fe0 mockbuild: temporarily pin RHEL 9 compose to an older one
A quick and dirty solution just to see the CI green before the release.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-02-18 16:03:20 +01:00
Achilleas Koutsou
2a504db9ed test: drop ANSIBLE_STDOUT_CALLBACK
Leftover from 1a6f6fdc4b
2022-02-18 10:39:08 +00:00
Achilleas Koutsou
12cbae3891 test: wait longer for reboot during ostree rollback test
Co-Authored-By: Xiaofeng Wang <henrywangxf@me.com>
2022-02-18 10:39:08 +00:00
Achilleas Koutsou
22f23ec38e test: update manifests and image info 2022-02-18 10:39:08 +00:00
Achilleas Koutsou
96e09e0322 Update cs9 test repos to latest snapshot 2022-02-18 10:39:08 +00:00
Achilleas Koutsou
f7d8535b16 tools/test-case-generators: add s390x for CS9 to test gen map 2022-02-18 10:39:08 +00:00
Achilleas Koutsou
d1ee43c9ec CI: run installers test on CS9 2022-02-18 10:39:08 +00:00
Achilleas Koutsou
85383ca950 test/cases: add centos OS_VARIANTS to installers test
Set conditional OS_VARIANT, copied from ostree-ng.sh.
Add CentOS Stream 9 variant.
2022-02-18 10:39:08 +00:00
Achilleas Koutsou
32fe534292 distro/rhel90: fix ISO Label for CS9
Should be CentOS-Stream-9-BaseOS-$arch (based on official CS9 ISOs)
instead of CentOS-Stream-9-$arch-dvd, which was the format that CS8
used.
2022-02-18 10:39:08 +00:00
Achilleas Koutsou
d271de346f test/ostree: use production compose for virt boot location 2022-02-18 10:39:08 +00:00
Achilleas Koutsou
a087fc5c86 test/cases: use correct OS_VARIANT for CentOS Stream 9
Correct name is 'centos-stream9'
See 'osinfo-query os'
2022-02-18 10:39:08 +00:00
Achilleas Koutsou
ff992e00f2 tests/libvirt: test uefi on cs9 2022-02-18 10:39:08 +00:00
Achilleas Koutsou
0ec9dcbc63 distro/rhel90: enable s390x for cs9
CentOS Stream 9 supports s390x.
2022-02-18 10:39:08 +00:00
Ondřej Budai
16f6864d58 mockbuild: fix CS9 mockbuild
- name mangling from centos to centos-stream was enabled also for
  centos 9
- CS9 mock configs from mock-core-configs-36.3-1.fc35.noarch were added

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-02-18 10:39:08 +00:00
Achilleas Koutsou
96f3647464 CI: run tests on CentOS Stream 9
Tests need to be on the internal network to fetch the cert for epel for
EL9 (see provision.sh).
Use the large runner for all ostree-based tests since the regular one
seems to be running out of space.
2022-02-18 10:39:08 +00:00
Achilleas Koutsou
6bc4afbcbd test,repositories: add cs9 repositories
Add rpmrepo repositories for testing.
Symlink centos-9 to centos-stream-9.
Add rpmrepo snapshot repositories to test and
test-case-generator repos.
Remove unused repositories from cs9 test repositories.
2022-02-18 10:39:08 +00:00
Achilleas Koutsou
0932fd700b distroregistry: enable CS9
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2022-02-18 10:39:08 +00:00
Brian C. Lane
52e9d45e58 client: Add Cancel function and tests
NOTE that the tests against the rpmmd fixure for running and waiting are
a bit of a kludge. It is impossible to mock up a running or waiting that
will allow an actual cancel, so the tests check for the internal server
error generated by using the fixure data.

Getting this error means that the API code did it's job and tried to
cancel the compose so that should be a good compromise.
2022-02-18 09:31:07 +01:00
Brian C. Lane
c657713181 weldr: Allow cancel on waiting and running composes
Fix the cancel API to allow a waiting compose to be canceled.
This also fixes the cancel return code to be 400, the lorax-composer
behavior was a bug, and using 400 allows composer-cli to properly
display the error.
2022-02-18 09:31:07 +01:00
Alexander Todorov
c877f14379 tests: Use human readable size for mountpoint
Follow up to
https://github.com/osbuild/osbuild-composer/pull/1921 and
https://bugzilla.redhat.com/show_bug.cgi?id=2023635
2022-02-18 09:28:01 +01:00
Simon Steinbeiss
cdddc3f81c worker: Properly log successful image builds
This will make counting successful image builds in Splunk and Cloudwatch
much simpler and robust.
2022-02-18 09:27:35 +01:00
Tomas Hozza
657b7a5833 Move ostreePullStageInputs() from distros to osbuild2
Move the `ostreePullStageInputs()` function duplicated in all
distro definitions to the `osbuild2` package as
`NewOstreePullStageInputs()`.

Delete `stage_inputs.go` from all distro definitions.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-18 09:26:35 +01:00
Tomas Hozza
d946199441 Move bootISOMonoStageInputs() from distros to osbuild2
Move the `bootISOMonoStageInputs()` function duplicated in all
distro definitions to the `osbuild2` package as
`NewBootISOMonoStagePipelineTreeInputs()`.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-18 09:26:35 +01:00
Tomas Hozza
069423ea67 Move rpmStageInputs() from distros to osbuild2
Move the `rpmStageInputs()` function duplicated in all
distro definitions to the `osbuild2` package as
`NewRpmStageSourceFilesInputs()`.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-18 09:26:35 +01:00
Tomas Hozza
8e6826e743 Move OSBuildMetadataToRPMs and PackageMetadataToSignature to osbuild2
Move `OSBuildMetadataToRPMs()` and `PackageMetadataToSignature()`
functions from the `rpmmd` package to `osbuild2` package to prevent
import cycles while de-duplicating `rpmStageInputs()` function from
`stage_inputs.go` of distro definitions.

Rename `PackageMetadataToSignature()` to
`RPMPackageMetadataToSignature()`, since it takes specifically
`RPMPackageMetadata` type as an argument.

Adjust affected parts of code (unit tests, cloudapi, worker).

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-18 09:26:35 +01:00
Tomas Hozza
0bd0f57231 Move qemuStageInputs() from distros to osbuild2
Move the `qemuStageInputs()` function duplicated in most
distro definitions to the `osbuild2` package as
`NewQemuStagePipelineFilesInputs()`.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-18 09:26:35 +01:00
Tomas Hozza
71ae94b40f Move xorrisofsStageInputs() from distros to osbuild2
Move the `xorrisofsStageInputs()` function duplicated in all
distro definitions to the `osbuild2` package as
`NewXorrisofsStagePipelineTreeInputs()`.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-18 09:26:35 +01:00
Tomas Hozza
c9b72033ee Move copyPipelineTreeInputs() from distros to osbuild2
Move the `copyPipelineTreeInputs()` function duplicated in many
distro definitions to the `osbuild2` package as
`NewCopyStagePipelineTreeInputs()`.

This will prevent creating another copy of the code in rhel-84 for
the `gce` image.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-18 09:26:35 +01:00