Commit graph

2748 commits

Author SHA1 Message Date
Martin Sehnoutka
b2d8d092c9 internal/rpmmd: log repository files loaded during composer startup
This is a confusing part of the startup process. We have changed the naming
and we also added the cross distro building feature. It is unclear which
files are loaded and from where (if /etc or /usr).

Log the files that are loaded so every user can clearly see what
configuration osbuild-composer uses. This complements the log of loaded
configuration.
2021-10-05 14:41:05 +02:00
Tom Gundersen
e76543d779 dnf-json: expire metadata by default
Never expiring metadata by default leads to surprising behavior
especially for our long-running services. The overhead of expiration
is small but noticeable, attempt some compromise.

This should all be revisited to make dnf-json handle caches better
and be more performant.
2021-10-04 16:02:31 +02:00
sanne
149d53c1e4 schutzbot: Clean up non-default storage accounts 2021-10-04 14:30:04 +02:00
Achilleas Koutsou
1a29f084a1 main: IsNotExist() is no longer a valid check 2021-10-04 13:54:53 +02:00
Achilleas Koutsou
c30c4decb7 config: update NonExisting test to check for default 2021-10-04 13:54:53 +02:00
Achilleas Koutsou
8393cfc322 config: don't fail LoadConfig if file doesn't exist
When the config file doesn't exist, don't return because we need to keep
the default and also load from env.
2021-10-04 13:54:53 +02:00
Tomas Hozza
2aedd3da05 Test: regenerate all image test cases
Fix #1793

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
147cec6baa Image tests: use RPMRepo with released RHEL-8.4 content
Fix #1793

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
4854b5730c image-info: fix undefined variable in analyse_directory()
One branch of the analyse_directory() function's if statement used
undefined variable `repo`. Copy its existing definition already used in
the function.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
7b37c92bea image-info: sort partitions list in the report.
Ensure that the list of partitions in the report is always sorted. Use
'partuuid' as the sorting key.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
02b706b673 image-info: use subprocess_check_output() in read_selinux_ctx_mismatch()
Use the `subprocess_check_output()` wrapper function in the
`read_selinux_ctx_mismatch()` parser function, instead of
directly calling `subprocess.check_output()`.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
e91c4a114b image-info: check not installed documentation
Extend the report to list also all documentation files, which are
normally part of the installed RPM packages, but were not installed
on the system. This can happen e.g. when '--excludedocs' option
is used when installing packages using rpm.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
ff2aa771fc image-info: read content of /etc/resolv.conf
Read uncommented lined from /etc/resolv.conf and add them as a list to
the image-info report. The list of lines is present in the report even
if it is empty, so that the report is explicit about the file content
and presence.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
299bd201e6 image-info: read sysctl.d config files from multiple paths
Extend image-info to read sysctl.d configuration files from multiple
paths:
 - /etc/sysctl.d/*.conf
 - /usr/lib/sysctl.d/*.conf

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
fea41e9c3a image-info: read security limits config files from multiple paths
Extend image-info to read security limits configuration files from multiple
paths:
 - /etc/security/limits.conf
 - /etc/security/limits.d/*.conf

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
7c9ecf8a53 image-info: read tmpfiles.d config files from multiple paths
Extend image-info to read tmpfiles.d configuration files from multiple
paths:
 - /etc/tmpfiles.d/*.conf
 - /usr/lib/tmpfiles.d/*.conf

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
70e1b30347 image-info: read systemd service unit drop-ins from multiple paths
Extend image-info to read systemd service unit drop-ins from
multiple paths:
 - /etc/systemd/system/*.service.d
 - /usr/lib/systemd/system/*.service.d

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
90a13c3d24 image-info: read cloud-init configs from multiple paths
Extend image-info to read cloud-init configuration files from
multiple paths:
 - /etc/cloud/cloud.cfg
 - /etc/cloud/cloud.cfg.d/*.cfg

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
9e719a0c33 image-info: read systemd-logind configs from multiple paths
Extend image-info to read systemd-logind configuration files from
multiple paths:
 - /etc/systemd/logind.conf
 - /etc/systemd/logind.conf.d/*.conf
 - /usr/lib/systemd/logind.conf.d/*.conf

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
fb982b20b9 image-info: read dracut configs from multiple paths
Extend image-info to read dracut configuration files from multiple
paths:
 - /etc/dracut.conf.d/*.conf
 - /usr/lib/dracut/dracut.conf.d/*.conf

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
ee197346bb image-info: read modprobe configs from multiple paths
Extend image-infro to read modprobe configuration files from multiple
paths:
 - /etc/modprobe.d/*.conf
 - /usr/lib/modprobe.d/*.conf
 - /usr/local/lib/modprobe.d/*.conf

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
sanne
1e724be8e9 composer: More configuration of how composer is served
Allows you to disable tls
2021-10-02 14:27:02 +02:00
Simon Steinbeiss
fed4b97e4e 35
Release osbuild-composer 35

Signed-off-by: Simon Steinbeiss <simon.steinbeiss@redhat.com>
2021-10-02 09:11:07 +02:00
Achilleas Koutsou
66b76c2ace gitleaks: add allow list for test passwords and keys
Add a .gitleaks.toml file that lists all the test passwords and paths to
keys that should be ignored by the security scanner. This avoids false
positives so we can enable the scanner to catch actual leaks.

Closes #1747

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-10-01 16:56:26 +02:00
Ondřej Budai
af784a11b1 mockbuild: use download.devel of download.eng.bos
download.eng.bos.redhat.com is down, use the mirroring service instead.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-01 15:23:53 +02:00
Ondřej Budai
2070747876 ci: install ansible-core instead of ansible for EL9
ansible-core no longer provides ansible so we need to require specifically
ansible-core.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-01 15:23:53 +02:00
Ondřej Budai
4f8dc76ca7 dnf-json: disable zchunk
See the comment

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-01 15:23:53 +02:00
Diaa Sami
cc54c4deee composer: improve logging 2021-09-30 18:34:05 +02:00
Diaa Sami
27ca55372a cloudapi: Add extra logging & improve existing
Use different logging levels depending on situation
Log HTTP 5xx errors to stdout with internal error details when possible
2021-09-30 18:34:05 +02:00
Diaa Sami
179009fec4 cloudapi: use Logrus as default logger in Echo
And remove log.Logger references
2021-09-30 18:34:05 +02:00
Diaa Sami
75372b98e1 logging: add logrus dependency
Just add the dependency and update relevant files
2021-09-30 18:34:05 +02:00
Jakub Rusz
c112e9d12d ci: don't run nightly pipeline on ga releases
It makes no sesnse to run nightly testing on ga composes. This updates
the regex to exclude ga runners from the scheduled nightly run.
2021-09-30 16:03:14 +02:00
Ondřej Budai
5ed954bcf5 distro/rhel90: use qcow2 compat 1.1 for qcows
There's no reason to use 0.10 since we don't need to support running
this image on RHEL 6 anymore.
See https://bugzilla.redhat.com/show_bug.cgi?id=2008910

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-09-30 15:36:15 +02:00
sanne
106e9b3fc2 tools: Push ubi container to quay.io/app-sre/composer 2021-09-29 22:37:25 +02:00
Gianluca Zuccarelli
b8c14a5689 blueprints: change minsize from int to uint64
thozza pointed out that `int` is platform dependent which results in
a fs size that is too small for 32-bit machines. This commit changes
the filesystem custimizations to use `uint64` instead of `int`
2021-09-29 15:08:12 +02:00
sanne
5239e264bb test/regression: Adapt to new rhel-84-ga runners 2021-09-29 15:07:35 +02:00
sanne
042eb75de6 schutzbot: Also build container on branch pushes
The regression test failed to find the container when not running from a
PR.
2021-09-29 15:07:35 +02:00
Ondřej Budai
9403c6e795 schutzbot: pin osbuild to >=37 on all distros
The bootloader option in ostree.config stage is new in osbuild 37. However,
osbuild 37 contains a bug that prevents it from working in certain cases,
thus we need to pin to a slightly newer commit that contains a fix for it:

bug: https://github.com/osbuild/osbuild/issues/822
fix: https://github.com/osbuild/osbuild/pull/824

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-09-27 20:10:04 +02:00
Ondřej Budai
1b59b2973e spec: bump osbuild depedendency to 37
The bootloader option in ostree.config stage is new in osbuild 37.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-09-27 20:10:04 +02:00
Christian Kellner
7a4488e5c4 distro/rhel85: set bootloader to none for edge
Use the new bootloader config for OSTree deployments to set the
bootloader backend to `none` since we use grub2 and BLS in all
currentlysupported architectures. In fact, not setting it to
`none`, will use the default `auto`, which will result in the
run of `grub2-mkconfig`, which is not what we want and might
even fail for us.
2021-09-27 20:10:04 +02:00
Christian Kellner
f6d4228b00 osbuild2: add bootloader option to ostree.config
Expose the new (osbuild > 36) `sysroot.bootloader` option to
configure the bootloader backend for OSTree deployments.
Normally, this should be set to `none` for modern bootloader
that use the BLS, since the BLS snippets are always generated.
2021-09-27 20:10:04 +02:00
Christian Kellner
920863df70 osbuild2: small doc fix for ostree.config 2021-09-27 20:10:04 +02:00
dependabot[bot]
0091d66b95 build(deps): bump github.com/labstack/echo/v4 from 4.5.0 to 4.6.0
Bumps [github.com/labstack/echo/v4](https://github.com/labstack/echo) from 4.5.0 to 4.6.0.
- [Release notes](https://github.com/labstack/echo/releases)
- [Changelog](https://github.com/labstack/echo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/labstack/echo/compare/v4.5.0...v4.6.0)

---
updated-dependencies:
- dependency-name: github.com/labstack/echo/v4
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-27 14:35:27 +02:00
dependabot[bot]
352128f831 build(deps): bump github.com/aws/aws-sdk-go from 1.40.46 to 1.40.49
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.40.46 to 1.40.49.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.40.46...v1.40.49)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-27 14:34:48 +02:00
sanne
2f328b0e97 workers: Backwards compatible api.openshift.com spec compliance
The main changes are:
- Kind, Href, Id fields for every object returned
- Attach operationIds to each request, return it for errors
- Errors are predefined and queryable
2021-09-27 13:10:05 +01:00
Alexander Todorov
5e206322a2 Disable regression-composer-works-behind-satellite.sh, Refs #1834 2021-09-24 17:03:53 +02:00
Ondřej Budai
3c9c9f81e7 test/ostree: use 8.4 when testing on 8.4
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-09-24 17:03:53 +02:00
Ondřej Budai
3991827f68 ci: add workaround for missing redhat.repo in EC2
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-09-24 17:03:53 +02:00
Ondřej Budai
e64cc2effe schutzfile: remove rhel-8.3
We don't run on RHEL 8.3 anymore.

Note that the koji-osbuild rev-deps test is not run anymore, therefore
it's fine to drop dependants, see:
https://github.com/osbuild/osbuild-composer/issues/1559

We should reintroduce it though.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-09-24 17:03:53 +02:00
Ondřej Budai
aff680542d test/koji: always build the latest RHEL
We use the koji API to build RHEL nightlies, therefore we should build
only development versions in the testsuite.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-09-24 17:03:53 +02:00