Commit graph

5341 commits

Author SHA1 Message Date
Tomáš Hozza
b007b0ea12 internal/fsnode: add API for custom directories and files
Add an internal API for working with custom FS nodes such as Files and
Directories. This implementation is agnostic to external API, such as
Weldr API, Cloud API or osbuild stages. The purpose of it is to be the
common translation layer between all of these "external" APIs and
osbuild.

In this stage, the representation for Files and Directories is added.

The functionality is not yet used by any existing code.

Note about user/group type being `interface{}`:
I considered using the internal `users` representation for users and
groups, but it contains additional information, which are not relevant
for FS node user / group ownership representation. Therefore I didn't
use it. I also considered using separate variables for user / group
name (string) and uid / gid (int64). However, the implementation would
need to ensure that only one of these typed values is set for user /
group or ensure that it refers to the same group / user. My estimate
was that the code ensuring that only one of these typed values is set
would be probably as complex as the current implementation that checks
the types stored in `interface{}` typed variable. And ensuring that
the set user / group name and uid / gid is referring to the same user
/ group is nearly impossible to get right without actually building
the image.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2023-02-22 12:17:36 +01:00
Tomáš Hozza
0e4dabd265 blueprint: delete HasIgnition() function
Unify how are allowed options checked in distro implementation in
relation to Ignition customization. Specifically, delete `HasIgnition()`
function and replace its use by `GetIgnition()` call and checking if it is
`nil`. This approach is consistent with how this is checked for other
customizations.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2023-02-22 12:17:36 +01:00
Tomáš Hozza
317565f2c4 blueprint: delete unused HasFDO() function
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2023-02-22 12:17:36 +01:00
Tomáš Hozza
772a131d35 osbuild/copy: add TODO comment related to GenCopyFSTreeOptions()
The `GenCopyFSTreeOptions()` function has unused argument
`inputPipeline`. After discussing this with @achilleas-k, we determined
that it would be for the best to refactor this function and split out
the part that generates Mounts and Devices and instead return the actual
Stage from it. The reason for splitting out Mounts and Devices is that
these are then reused also by other stages when constructing pipelines
inside composer.

This would be a bigger change, so just adding the comment to the code to
capture this future work.

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2023-02-22 12:17:36 +01:00
Mario Cattamo
b68cf83a8e Update boot location for Fedora 37 edge-commit 2023-02-21 09:46:27 +01:00
Jakub Rusz
004c4e8acc Gitlab CI: Run libvirt.sh tests on GCP
We can now use GCP runners in our CI. Let's test them out by running all
of libvirt tests for starters.
2023-02-20 14:24:09 +01:00
Sanne Raymaekers
96b42b5791 internal/worker: add String to clienterrors.Error struct
This should make the logging better when errors have another error
struct.

Fixes #3272
2023-02-16 00:55:34 +01:00
Juan Abia
5a62453190 Remove cloud cleaner
cloud cleaner is now being executed in it's own repo.
2023-02-15 17:23:35 +01:00
Juan Abia
197cd47e0b test/cases: move CIV options into a variable
In order to provide different options when running CIV from it's own CI,
move them to the variable CIV_OPTIONS
2023-02-15 17:23:35 +01:00
Thomas Lavocat
be48279af8 schutzfile: update OSBuild's ref
Image-info needs to have support for mounting the devices with their
respective mount options. Otherwise it will fail schema validation.
2023-02-14 14:37:10 +01:00
Thomas Lavocat
aba981c508 tools/image-info: bad syntax on rhel8 python
We can't use:

with (a() as b, c() as d):
    pass

On older versions of python.
2023-02-14 14:37:10 +01:00
Thomas Lavocat
15d80a0cd1 tools/image-info: cosmetics, autopep8 2023-02-14 14:37:10 +01:00
Thomas Lavocat
36d890d452 toos/image-info: use SECTOR-SIZE
Instead of numerical value
2023-02-14 14:37:10 +01:00
Thomas Lavocat
0324e02659 tools/image-info: support iso
Add the support for ISOs. Mounting then as a loopack device and then
accessing the content as a tarball
2023-02-14 14:37:10 +01:00
Thomas Lavocat
d0be1ec5a4 tools/image-info: validate with osbuild's schemas
Before we were invoking osbuild's stages/devices/mounter directly
without taking the time to validate that the options sent were actually
valid and supported.
This commit adds the support of the validation schema into image-info so
that we're sure we don't mess with the internals when we call them.
2023-02-14 14:37:10 +01:00
Thomas Lavocat
ffd45d635a tools/image-info: mount with the mount options
Before the new version of image-info was ignoring the mount options for
the devices leading to errors in the resulting json.
2023-02-14 14:37:10 +01:00
Thomas Lavocat
62e6bed23d tools/image-info: clean placeholder try block
Was only there to ease diffing previous commit.
2023-02-14 14:37:10 +01:00
Thomas Lavocat
31e3729236 tools/image-info: use osbuild's internal to mount
On the newest versions of rhel 92, 88, a change in behavior makes the
previous version of image-info failing to mount loopback devices. We've
tracked down this error to be a race condition on udev, yet without
understanding what changed for now.

Osbuild had for some time already a cleaner way to mount partitions.
osbuild has some machinery to opt out of block device handling in udev
48a4419705/devices/org.osbuild.loopback (L69)
Using this fixes the issue at hand.

This changes the way we need to mount all the partitions, including the
LVM ones. This new mechanism might also pave the way to include lusks fs.
2023-02-14 14:37:10 +01:00
Thomas Lavocat
aaddca445d tools/image-info: fix mypy errors 2023-02-14 14:37:10 +01:00
Antonio Murdaca
71ca1d0e9a regenerate manifests
Signed-off-by: Antonio Murdaca <antoniomurdaca@gmail.com>
2023-02-14 12:07:23 +01:00
Antonio Murdaca
6afab3dff2 rhel9: fix conditionals for sysroot.readonly enablement
Signed-off-by: Antonio Murdaca <antoniomurdaca@gmail.com>
2023-02-14 12:07:23 +01:00
Antonio Murdaca
1b9e008c29 rhel8: remove sysroot.readonly functionality
Signed-off-by: Antonio Murdaca <antoniomurdaca@gmail.com>
2023-02-14 12:07:23 +01:00
Brian C. Lane
6846f59ff4 manifest: Add tests for subscriptions
There are 3 different ways for subscriptions to be setup, test the
pipeline to make sure it includes the correct commands, and check the
package set chain to make sure it includes the packages needed to
support the selected subscription mode.
2023-02-09 09:41:15 +01:00
Brian C. Lane
6944d15293 cloudapi: Add a test for the subscription rhc flag
This makes sure that the new flag is accepted by the API endpoint.
2023-02-09 09:41:15 +01:00
Brian C. Lane
b0e388d115 cloudapi: Add subscription option for rhc
When rhc is selected it will install the required packages, register
using rhc and always enable insights.

When rhc is not selected it will use subscription manager for
registration, and optionally enable insights. Also installing required
packages.
2023-02-09 09:41:15 +01:00
Jakub Rusz
3cdfa9d7f0 internal/prometheus: add more buckets for job durations
We were hitting the limit on stage, let's increase it.
2023-02-08 12:33:10 +01:00
Diaa Sami
20c6fad7c2 osbuild-worker/koji: Add logging for koji requests/responses 2023-02-08 11:40:34 +01:00
schutzbot
4e574b6e99 Post release version bump
[skip ci]
2023-02-08 08:25:30 +00:00
Antonio Murdaca
b8ddbd3031 internal/distro/rhel9: enable ignition on 9+
Signed-off-by: Antonio Murdaca <antoniomurdaca@gmail.com>
2023-02-07 13:30:22 +01:00
Antonio Murdaca
5ab00987cd test/cases: add ignition test to edge-raw-image
Signed-off-by: Antonio Murdaca <antoniomurdaca@gmail.com>
2023-02-07 13:30:22 +01:00
Antonio Murdaca
b3f1b0edc8 ignition: drop embedded provisioning url
Signed-off-by: Antonio Murdaca <antoniomurdaca@gmail.com>
2023-02-07 13:30:22 +01:00
Ondřej Budai
2b1facb44d distro/rhel8: don't install missing MSFT key into azure-sap-rhui
The rhui-azure-rhel8-sap-ha package is currently missing
the /etc/pki/rpm-gpg/RPM-GPG-KEY-microsoft-azure-release key.
This makes the image type unbuildable, which causes some of our tests
to fail.

Overlay the generic RHUI config, so the missing key isn't imported.
See CLOUDX-336 for more information.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-02-07 13:16:12 +01:00
Ondřej Budai
bd5bffc8cf distro/rhel8: make sap image config methods private
They are not used in a different package, so they can be private.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-02-07 13:16:12 +01:00
dependabot[bot]
c2a290ef76 build(deps): bump github.com/labstack/echo/v4 from 4.9.0 to 4.10.0
Bumps [github.com/labstack/echo/v4](https://github.com/labstack/echo) from 4.9.0 to 4.10.0.
- [Release notes](https://github.com/labstack/echo/releases)
- [Changelog](https://github.com/labstack/echo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/labstack/echo/compare/v4.9.0...v4.10.0)

---
updated-dependencies:
- dependency-name: github.com/labstack/echo/v4
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-07 09:03:10 +01:00
dependabot[bot]
ff86f29120 build(deps): bump github.com/openshift-online/ocm-sdk-go
Bumps [github.com/openshift-online/ocm-sdk-go](https://github.com/openshift-online/ocm-sdk-go) from 0.1.287 to 0.1.315.
- [Release notes](https://github.com/openshift-online/ocm-sdk-go/releases)
- [Changelog](https://github.com/openshift-online/ocm-sdk-go/blob/main/CHANGES.md)
- [Commits](https://github.com/openshift-online/ocm-sdk-go/compare/v0.1.287...v0.1.315)

---
updated-dependencies:
- dependency-name: github.com/openshift-online/ocm-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-07 09:02:42 +01:00
Xiaofeng Wang
b4f15f13e3 test: Add minimal-raw image test
The minimal-raw image is for Fedora only and feature merged by
PR #3181
2023-02-06 17:58:05 +01:00
Sanne Raymaekers
8f9f0d03c5 repositories: fix rhel-90 repositories
The rhel-90 repos should point to the rhel-90 minor version repos, the
rhel-9 repos should point to the rolling release.
2023-02-06 17:35:54 +01:00
Antonio Murdaca
e34b88d6e4 .github/workflows: pin golangci-lint to v1.51.0
The new "latest" aka v1.51.1 started throwing unused errors during runs,
blocking all the other tests, to quickly unblock that until a proper
fix/rewrite is found, pin the version to the previous one.

Signed-off-by: Antonio Murdaca <antoniomurdaca@gmail.com>
2023-02-06 11:04:06 +01:00
Ondřej Budai
9633dce3bb osbuild-pipeline: add support for RHSM when defining repos
I use this tool quite a lot and I often want to use the CDN content, so
I would very much appreciate RHSM support. :)

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-02-06 10:02:04 +01:00
Achilleas Koutsou
d877b7419e osbuild-dnf-json-tests: remove distro loop
We only test one distro, CentOS 9, so there's no need to loop and spawn
a new t.Run().
2023-02-02 19:01:20 +01:00
Brian C. Lane
356f261bcd dnf-json: disable pylint warnings
The _dnfrepo is full of branches, turn off warning.

The keyfile doesn't use 'with' because the file needs to remain
available when the function exits. Cleanup of persistdir will clean up
the temporary file used for the key.
2023-02-01 10:27:58 +01:00
Brian C. Lane
d5e666cc9c tests: Add tests for new repo gpg support
Add testing for gpgkeys and check_repogpg.
2023-02-01 10:27:58 +01:00
Brian C. Lane
c2577eaea8 Add gpgkey and check_repogpg support to dnf-json
This allows verification of repository metadata signatures.

The gpgkeys field is a list of key urls, or the gpg key itself, starting
with '-----BEGIN PGP PUBLIC KEY BLOCK-----'. These will be written to a
temporary file, and that file:// url will be passed to dnf.
2023-02-01 10:27:58 +01:00
Brian C. Lane
d12447408b Change RepoConfig.GPGKey to an array of key strings
DNF supports more than one GPG key. It is possible that one may be used for
signing packages, and another to sign the repository metadata. This
renamed GPGKey to GPGKeys internally. It does not change the on-disk
repository json format.
2023-02-01 10:27:58 +01:00
Brian C. Lane
a47375a10d Rename gpgkey_urls to gpgkeys
This has never been supported in osbuild-composer, so it is safe to
rename in preparation for adding repo metadata gpg signature checking.
2023-02-01 10:27:58 +01:00
Achilleas Koutsou
7a1d7ae1d7 test: update installer manifests for RHEL 9.x and CS9
Dracut modules and drivers fixed for the dracut stage in all RHEL 9.x
and CS9 Anaconda installers (image-installer and edge-installer).
2023-01-31 11:03:26 +01:00
Achilleas Koutsou
2f96cc7051 manifest: fix dracut modules and drivers for Anaconda installers
When the image definitions were updated to the new framework, I failed
to update the dracut modules for Anaconda installers to match the
existing ones.

The changes in the manifest are at commit
c4af0a1886.
The nvdimm module and the additional drivers were removed.

The nvdimm module in particular is required for http boot but should
only be specified for RHEL 9.  In RHEL 8 it is part of the default set
of modules.

See 02bb7a0b4f and
dc95382ba3 for the original commits that
introduced these changes.
2023-01-31 11:03:26 +01:00
Irene Diez
1dfc82ae4a test/data/manifests: regen for edge container + commit
Signed-off-by: Irene Diez <idiez@redhat.com>
2023-01-30 19:26:44 +01:00
Irene Diez
3b62e15d01 internal/distro/rhel8/edge: add sos package for rhel8.6+ and CS8
Similarly to the change made for rhel9, adding the sos package
gives users a built-in way to gather system logs and debug info.

Signed-off-by: Irene Diez <idiez@redhat.com>
2023-01-30 19:26:44 +01:00
Irene Diez
18aee4e465 internal/distro/rhel9/edge: add sos package
Adds the sos package to the edge commit package set so that
users have built-in way to gather system logs and debug info.

Signed-off-by: Irene Diez <idiez@redhat.com>
2023-01-30 19:26:44 +01:00