Commit graph

3509 commits

Author SHA1 Message Date
Christian Kellner
5bf4892077 disk: use pointer receiver for PartitionTable
The `PartitionTable` object contains an array of `Partitions`, which
is shared among all shallow copies of `PartitionTable` instances, as
when it is used via a value receiver in method calls. The individual
objects are thus not fully isolated since modifications of array
elements, or in some cases modifications to the array itself, are
shared and visible between all shallow `PartitionTable` copies. Thus
use a pointer receiver for all methods of `PartitionTable` to make
it explicit that modifications will affect other objects.
2022-02-22 19:23:41 +00:00
Christian Kellner
d415d57f95 disk: create{Partition -> Filesystem}
What this function is actually doing is to create a filesystem,
together with a new partition; for LVM that filesystem could
also be created on a new logical volume though.
2022-02-22 19:23:41 +00:00
Christian Kellner
5693045916 disk: createPartition now appends also it
It makes sense that the function that creates the partition also
adds it to the partition table.
2022-02-22 19:23:41 +00:00
Christian Kellner
147e0c966b disk: remove updateRootPartition
The previous commit made sure that `PartitionTable.RootPartition`
is indeed returning the pointer partition object inside the array
of the `PartitionTable`. Thus changes to the returned object will
now directly affect the object and thus there is no need to call
`PartitionTable.updateRootPartition` anymore.
2022-02-22 19:23:41 +00:00
Christian Kellner
6333d54091 disk: return the correct ptr from RootPartition
When we iterate over the partitions to find the root partition,
we are getting the partition by value so even if we return a
pointer to that then, it is actually a pointer duplicated object.
Return the pointer to the actual partition object instead.
2022-02-22 19:23:41 +00:00
Christian Kellner
c8efc7d282 disk: create file system accessors and use those
In various places we are interested in the root and boot file-
systems. Currently those were accessed via by retrieving the
partition that contain them and the accessing the filesystem
member. Add accessors to `PartitionTable` that directly return
the needed filesystem. This will help if the file system is
stored inside a container like LVM or LUKS instead of directly
on a partition.
2022-02-22 19:23:41 +00:00
Sanne Raymaekers
d589317dcb internal/cloudapi: Log error in manifest job 2022-02-22 14:06:05 +01:00
Djebran Lezzoum
d8fdb03373 dnf-json: Add repo_id to dump package.
The dump function will be used to search packages and as we are implementing third party repositories, adding the repo_id of the package will allow us to identify the package repository used.
2022-02-21 15:46:53 +01:00
Sanne Raymaekers
c4ecbea510 internal/cloud: Allow aws creds from defaults
Defaults according to https://docs.aws.amazon.com/sdk-for-go/api/aws/#Config:
Defaults to a chain of credential providers to search for credentials in
environment variables, shared credential file, and EC2 Instance Roles.

If nothing is specified fall back to whatever instance role.
2022-02-21 15:43:53 +01:00
Ondřej Budai
17e809b5f7 worker: use default transport instead of "blank" one
This allows us to use the default behaviour of http.DefaultTransport
to honor HTTP_PROXY.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-02-21 14:46:49 +01:00
Jakub Rusz
f0f80bd930 ci: update terraform SHA
This brings AWS internal subnet balancing which allows to double our CI
runners in AWS.
2022-02-21 12:21:13 +01:00
Jakub Rusz
f9fb593da3 ci: large runners for some ostree tests
The longer running ostree tests (all except ostree.sh) benefit from
running on large runners because of their long runtime. Changing all to
large runners. Should not be a problem with our increased Openstack
quota.
2022-02-21 12:02:55 +01:00
Tom Gundersen
bb486bcccb cloudapi/v2/compose: don't overwrite customizations struct
The customizations struct contains different kinds of customizations,
such as users and filesystems. If both filesystems and users are
configured, the current code overwrites the user configuration.

Change this by unconditionally allocating the customizations struct
when creating the empty blueprint, and ammending it for each
customization we find.

This means that the customizations will be non-nil, also when it is
empty, which was not the case in the past. This should not change
the behavior of the code.
2022-02-19 11:13:55 +00:00
Tom Gundersen
351d06aecf tests/cases/api.sh: make /var at least 256MB
We (mistakenly) don't enforce a minimum size for /var,
so setting it to 1024 (1kB) causes the image build to fail.

CI does not expose this in a helpful way at the moment,
so this is a bit tricky to debug.

Also skip customizations for the AWS.S3 upload type. Not all the
image types with this upload type support filesystem customizations
and that's as expected. We could make a more fine-grained test in
the future, but testing with a coulpe of targets should be
sufficient.
2022-02-19 11:13:55 +00:00
Chloe Kaubisch
0619ffa588 cloudapi: expose filesystem customizations 2022-02-19 11:13:55 +00:00
ondrejbudai
9c1f552210 Post release version bump
[skip ci]
2022-02-18 18:36:49 +00:00
Ondřej Budai
10c0891fe0 mockbuild: temporarily pin RHEL 9 compose to an older one
A quick and dirty solution just to see the CI green before the release.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-02-18 16:03:20 +01:00
Achilleas Koutsou
2a504db9ed test: drop ANSIBLE_STDOUT_CALLBACK
Leftover from 1a6f6fdc4b
2022-02-18 10:39:08 +00:00
Achilleas Koutsou
12cbae3891 test: wait longer for reboot during ostree rollback test
Co-Authored-By: Xiaofeng Wang <henrywangxf@me.com>
2022-02-18 10:39:08 +00:00
Achilleas Koutsou
22f23ec38e test: update manifests and image info 2022-02-18 10:39:08 +00:00
Achilleas Koutsou
96e09e0322 Update cs9 test repos to latest snapshot 2022-02-18 10:39:08 +00:00
Achilleas Koutsou
f7d8535b16 tools/test-case-generators: add s390x for CS9 to test gen map 2022-02-18 10:39:08 +00:00
Achilleas Koutsou
d1ee43c9ec CI: run installers test on CS9 2022-02-18 10:39:08 +00:00
Achilleas Koutsou
85383ca950 test/cases: add centos OS_VARIANTS to installers test
Set conditional OS_VARIANT, copied from ostree-ng.sh.
Add CentOS Stream 9 variant.
2022-02-18 10:39:08 +00:00
Achilleas Koutsou
32fe534292 distro/rhel90: fix ISO Label for CS9
Should be CentOS-Stream-9-BaseOS-$arch (based on official CS9 ISOs)
instead of CentOS-Stream-9-$arch-dvd, which was the format that CS8
used.
2022-02-18 10:39:08 +00:00
Achilleas Koutsou
d271de346f test/ostree: use production compose for virt boot location 2022-02-18 10:39:08 +00:00
Achilleas Koutsou
a087fc5c86 test/cases: use correct OS_VARIANT for CentOS Stream 9
Correct name is 'centos-stream9'
See 'osinfo-query os'
2022-02-18 10:39:08 +00:00
Achilleas Koutsou
ff992e00f2 tests/libvirt: test uefi on cs9 2022-02-18 10:39:08 +00:00
Achilleas Koutsou
0ec9dcbc63 distro/rhel90: enable s390x for cs9
CentOS Stream 9 supports s390x.
2022-02-18 10:39:08 +00:00
Ondřej Budai
16f6864d58 mockbuild: fix CS9 mockbuild
- name mangling from centos to centos-stream was enabled also for
  centos 9
- CS9 mock configs from mock-core-configs-36.3-1.fc35.noarch were added

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-02-18 10:39:08 +00:00
Achilleas Koutsou
96f3647464 CI: run tests on CentOS Stream 9
Tests need to be on the internal network to fetch the cert for epel for
EL9 (see provision.sh).
Use the large runner for all ostree-based tests since the regular one
seems to be running out of space.
2022-02-18 10:39:08 +00:00
Achilleas Koutsou
6bc4afbcbd test,repositories: add cs9 repositories
Add rpmrepo repositories for testing.
Symlink centos-9 to centos-stream-9.
Add rpmrepo snapshot repositories to test and
test-case-generator repos.
Remove unused repositories from cs9 test repositories.
2022-02-18 10:39:08 +00:00
Achilleas Koutsou
0932fd700b distroregistry: enable CS9
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2022-02-18 10:39:08 +00:00
Brian C. Lane
52e9d45e58 client: Add Cancel function and tests
NOTE that the tests against the rpmmd fixure for running and waiting are
a bit of a kludge. It is impossible to mock up a running or waiting that
will allow an actual cancel, so the tests check for the internal server
error generated by using the fixure data.

Getting this error means that the API code did it's job and tried to
cancel the compose so that should be a good compromise.
2022-02-18 09:31:07 +01:00
Brian C. Lane
c657713181 weldr: Allow cancel on waiting and running composes
Fix the cancel API to allow a waiting compose to be canceled.
This also fixes the cancel return code to be 400, the lorax-composer
behavior was a bug, and using 400 allows composer-cli to properly
display the error.
2022-02-18 09:31:07 +01:00
Alexander Todorov
c877f14379 tests: Use human readable size for mountpoint
Follow up to
https://github.com/osbuild/osbuild-composer/pull/1921 and
https://bugzilla.redhat.com/show_bug.cgi?id=2023635
2022-02-18 09:28:01 +01:00
Simon Steinbeiss
cdddc3f81c worker: Properly log successful image builds
This will make counting successful image builds in Splunk and Cloudwatch
much simpler and robust.
2022-02-18 09:27:35 +01:00
Tomas Hozza
657b7a5833 Move ostreePullStageInputs() from distros to osbuild2
Move the `ostreePullStageInputs()` function duplicated in all
distro definitions to the `osbuild2` package as
`NewOstreePullStageInputs()`.

Delete `stage_inputs.go` from all distro definitions.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-18 09:26:35 +01:00
Tomas Hozza
d946199441 Move bootISOMonoStageInputs() from distros to osbuild2
Move the `bootISOMonoStageInputs()` function duplicated in all
distro definitions to the `osbuild2` package as
`NewBootISOMonoStagePipelineTreeInputs()`.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-18 09:26:35 +01:00
Tomas Hozza
069423ea67 Move rpmStageInputs() from distros to osbuild2
Move the `rpmStageInputs()` function duplicated in all
distro definitions to the `osbuild2` package as
`NewRpmStageSourceFilesInputs()`.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-18 09:26:35 +01:00
Tomas Hozza
8e6826e743 Move OSBuildMetadataToRPMs and PackageMetadataToSignature to osbuild2
Move `OSBuildMetadataToRPMs()` and `PackageMetadataToSignature()`
functions from the `rpmmd` package to `osbuild2` package to prevent
import cycles while de-duplicating `rpmStageInputs()` function from
`stage_inputs.go` of distro definitions.

Rename `PackageMetadataToSignature()` to
`RPMPackageMetadataToSignature()`, since it takes specifically
`RPMPackageMetadata` type as an argument.

Adjust affected parts of code (unit tests, cloudapi, worker).

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-18 09:26:35 +01:00
Tomas Hozza
0bd0f57231 Move qemuStageInputs() from distros to osbuild2
Move the `qemuStageInputs()` function duplicated in most
distro definitions to the `osbuild2` package as
`NewQemuStagePipelineFilesInputs()`.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-18 09:26:35 +01:00
Tomas Hozza
71ae94b40f Move xorrisofsStageInputs() from distros to osbuild2
Move the `xorrisofsStageInputs()` function duplicated in all
distro definitions to the `osbuild2` package as
`NewXorrisofsStagePipelineTreeInputs()`.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-18 09:26:35 +01:00
Tomas Hozza
c9b72033ee Move copyPipelineTreeInputs() from distros to osbuild2
Move the `copyPipelineTreeInputs()` function duplicated in many
distro definitions to the `osbuild2` package as
`NewCopyStagePipelineTreeInputs()`.

This will prevent creating another copy of the code in rhel-84 for
the `gce` image.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-18 09:26:35 +01:00
Tomas Hozza
0efbe0c55e RHEL-90 beta: remove duplicate import in pipelines.go
Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-18 09:26:35 +01:00
Tomas Hozza
8713b8002a Move kernelVerStr() from distros to rpmmd
Move the `kernelVerStr()` function duplicated in many
distro definitions to the `rpmmd` package as
`GetVerStrFromPackageSpecListPanic()`.

I could not come up with a better name, sorry.

This will prevent creating another copy of the code in rhel-84 for
the `gce` image.

This change initially exposed a bug in the original implementation of
`kernelVerStr()`. Since on the first line, we allocate an empty structure
into `kernelPkg` variable, it can never be `nil` and the function never
panicked even if there was no `kernel` package in the PackageSpec list.

Fix all unit tests to provide valid arguments when calling `Manifest()`
method of image types.

Signed-off-by: Tomas Hozza <thozza@redhat.com>

kernelVerStr fixup

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-18 09:26:35 +01:00
Tomas Hozza
a392d71da6 Move kernelCmdlineStageOptions() from distros to osbuild2
Move the `kernelCmdlineStageOptions()` function duplicated in many
distro definitions to the `osbuild2` package as
`NewKernelCmdlineStageOptions()`.

This will prevent creating another copy of the code in rhel-84 for the
`gce` image.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-18 09:26:35 +01:00
Jakub Rusz
43bfcc5389 tests/upgrade: gather more logs
Also disable firwalld AllowZoneDrifting because it's not supported in
RHEL-9.
2022-02-18 09:25:01 +01:00
Ondřej Budai
5d304d2957 packer: make the worker image smaller
This should save us some money. 10 GB is the size of the underlying
RHEL 8.5 AMI so this should be the minimum.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-02-18 09:24:07 +01:00
Jakub Rusz
8331af3ffd ci/nightly: report composer NVR in slack
Storing composer NVR in a file after it's installed and reporting on
Slack. Also reducing duplications of artifacts in .gitlab-ci.yml
2022-02-17 11:14:31 +01:00