Commit graph

2766 commits

Author SHA1 Message Date
Tomas Hozza
7c6c67ef98 Tests/RHEL-9.0: add EC2 and EC2 HA image tests
Because the RHUI client RPMs for RHEL-9 don't exist yet, these test
cases use the RHUI client RPMs for RHEL-8 to satisfy the required
package set. These RPMs contain the entitlement certificate and repo
definitions for RHEL-8 AWS RHUI repos, but RHUI will not work without
the image being imported with the proper Billing Code to AWS. So these
installed RHUI RPMs are technically useless on the RHEL-9 image, but
this approach is good enough to allow us to at least build and test
these images from configuration point of view.

Regenerate all image test cases because of the updated repositories.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-08 16:34:55 +01:00
Tomas Hozza
d11db7ed83 Tests/RHEL-9.0: add repos needed for EC2 and EC2 HA image tests
Temporarily use the RHEL-8 RHUI client repo, because RHUI for RHEL-9
does not exist yet. Using at least RHEL-8 RHUI repo for EC2 images
allows us to at least successfully build them and inspect them.

Update all RHEL-9 repos to the latest snapshot to ensure the consistency
of package split among repositories.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-08 16:34:55 +01:00
Tomas Hozza
9497ef8ca9 Image tests: skip rpm-ostree-1-autovar.conf tmpfiles.d config on Fedora
The content of `/usr/lib/tmpfiles.d/rpm-ostree-1-autovar.conf` is
dynamically created and its lines are not ordered deterministically.
This causes the rpm-ostree based image test cases to fail often on
Fedora, because the diff of the expected and actual image-info report
always produces some different lines.

Add a new option `-skip-tmpfilesd-path` to `osbuild-image-tests`
accepting a tmpfiles.d configuration path, which should be ignored when
comparing the expected and actual image-info report. The option can be
specified multiple times and all paths will be ignored.

Modify the `image_tests.sh` test case to use the new option and ignore the
`/usr/lib/tmpfiles.d/rpm-ostree-1-autovar.conf` file when testing Fedora
images.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-08 16:21:25 +01:00
sanne
9d5c16f623 composer: Don't dump sensitive fields from config 2021-10-08 13:58:47 +02:00
Jakub Rusz
e9c0f45bf7 tests: Update image_tests 2021-10-07 12:41:53 +02:00
sanne
4b48c194a3 templates: Duplicate value in composer config
[skip ci]
2021-10-07 12:18:35 +02:00
sanne
973c1c4795 templates: Port names should be less than 15 characters
[skip ci]
2021-10-07 12:03:21 +02:00
sanne
14370e3c49 templates: Make sure ports are unquoted
[skip ci]
2021-10-07 11:56:02 +02:00
sanne
cda54ef926 cloudapi/v1: Return status created in compose handler 2021-10-07 10:54:12 +02:00
Diaa Sami
12ca5325d6 worker: Use Recover middleware to handle panics
recover from panics such as out-of-bounds array access & nil
pointer access, print a stack trace and return 5xx error
2021-10-06 17:04:52 +02:00
Diaa Sami
22f151df68 worker: Improve logging
Use logrus library for logging
Use appropriate log-level for different log statements
2021-10-06 17:04:52 +02:00
Tomas Hozza
f9fbf5bd5e Regenerate affected image test cases
Regenerate affected image test cases with the image-info fix (affecting
mainly `tar` image type) and with the latest upstream `osbuild` build
(affecting rhel-85 qcow2 images).

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-06 15:17:06 +02:00
Tomas Hozza
9ec2788ac8 generate-all-test-cases: allow specifying additional DNF repos
Add a new option `--repofrompath` allowing to specify additional DNF
repositories, which will be used on the Runner when installing any
packages (such as osbuild).

Extend the `test/README.md` to mention the new option. In addition,
specify some aspects of the script in more detail, because some of
them were not easy to figure out by users.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-06 15:17:06 +02:00
Tomas Hozza
71cfc35b67 image-info: ensure that directory is analysed as read-only
In some cases, e.g. when analysing an extracted `tar` image, it could
happen that the tools used by `image-info` to analyse the image could
modify its content (e.g. create new files which were originally not on
the image). This is especially an issue with `rpm`, which DB backend
seemed to create files in `/var/lib/rpm/` when run.

Ensure that the analysed directory can not be modified by bind-mounting
it as a read only, before the analysis.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-06 15:17:06 +02:00
Jakub Rusz
1a7858c196 tests: enable koji.sh test on RHEL-9 2021-10-06 15:42:36 +03:00
Jakub Rusz
9e7e16c9cf tests/ci: enable vmware.sh and cross-distro.sh on rhel-9 2021-10-06 15:42:36 +03:00
Simon Steinbeiss
a4db11d15f README: Add a link to our developer guide 2021-10-06 12:30:55 +02:00
sanne
4e56f04dd7 templates: Composer OSD template 2021-10-05 16:45:55 +02:00
Martin Sehnoutka
b2d8d092c9 internal/rpmmd: log repository files loaded during composer startup
This is a confusing part of the startup process. We have changed the naming
and we also added the cross distro building feature. It is unclear which
files are loaded and from where (if /etc or /usr).

Log the files that are loaded so every user can clearly see what
configuration osbuild-composer uses. This complements the log of loaded
configuration.
2021-10-05 14:41:05 +02:00
Tom Gundersen
e76543d779 dnf-json: expire metadata by default
Never expiring metadata by default leads to surprising behavior
especially for our long-running services. The overhead of expiration
is small but noticeable, attempt some compromise.

This should all be revisited to make dnf-json handle caches better
and be more performant.
2021-10-04 16:02:31 +02:00
sanne
149d53c1e4 schutzbot: Clean up non-default storage accounts 2021-10-04 14:30:04 +02:00
Achilleas Koutsou
1a29f084a1 main: IsNotExist() is no longer a valid check 2021-10-04 13:54:53 +02:00
Achilleas Koutsou
c30c4decb7 config: update NonExisting test to check for default 2021-10-04 13:54:53 +02:00
Achilleas Koutsou
8393cfc322 config: don't fail LoadConfig if file doesn't exist
When the config file doesn't exist, don't return because we need to keep
the default and also load from env.
2021-10-04 13:54:53 +02:00
Tomas Hozza
2aedd3da05 Test: regenerate all image test cases
Fix #1793

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
147cec6baa Image tests: use RPMRepo with released RHEL-8.4 content
Fix #1793

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
4854b5730c image-info: fix undefined variable in analyse_directory()
One branch of the analyse_directory() function's if statement used
undefined variable `repo`. Copy its existing definition already used in
the function.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
7b37c92bea image-info: sort partitions list in the report.
Ensure that the list of partitions in the report is always sorted. Use
'partuuid' as the sorting key.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
02b706b673 image-info: use subprocess_check_output() in read_selinux_ctx_mismatch()
Use the `subprocess_check_output()` wrapper function in the
`read_selinux_ctx_mismatch()` parser function, instead of
directly calling `subprocess.check_output()`.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
e91c4a114b image-info: check not installed documentation
Extend the report to list also all documentation files, which are
normally part of the installed RPM packages, but were not installed
on the system. This can happen e.g. when '--excludedocs' option
is used when installing packages using rpm.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
ff2aa771fc image-info: read content of /etc/resolv.conf
Read uncommented lined from /etc/resolv.conf and add them as a list to
the image-info report. The list of lines is present in the report even
if it is empty, so that the report is explicit about the file content
and presence.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
299bd201e6 image-info: read sysctl.d config files from multiple paths
Extend image-info to read sysctl.d configuration files from multiple
paths:
 - /etc/sysctl.d/*.conf
 - /usr/lib/sysctl.d/*.conf

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
fea41e9c3a image-info: read security limits config files from multiple paths
Extend image-info to read security limits configuration files from multiple
paths:
 - /etc/security/limits.conf
 - /etc/security/limits.d/*.conf

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
7c9ecf8a53 image-info: read tmpfiles.d config files from multiple paths
Extend image-info to read tmpfiles.d configuration files from multiple
paths:
 - /etc/tmpfiles.d/*.conf
 - /usr/lib/tmpfiles.d/*.conf

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
70e1b30347 image-info: read systemd service unit drop-ins from multiple paths
Extend image-info to read systemd service unit drop-ins from
multiple paths:
 - /etc/systemd/system/*.service.d
 - /usr/lib/systemd/system/*.service.d

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
90a13c3d24 image-info: read cloud-init configs from multiple paths
Extend image-info to read cloud-init configuration files from
multiple paths:
 - /etc/cloud/cloud.cfg
 - /etc/cloud/cloud.cfg.d/*.cfg

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
9e719a0c33 image-info: read systemd-logind configs from multiple paths
Extend image-info to read systemd-logind configuration files from
multiple paths:
 - /etc/systemd/logind.conf
 - /etc/systemd/logind.conf.d/*.conf
 - /usr/lib/systemd/logind.conf.d/*.conf

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
fb982b20b9 image-info: read dracut configs from multiple paths
Extend image-info to read dracut configuration files from multiple
paths:
 - /etc/dracut.conf.d/*.conf
 - /usr/lib/dracut/dracut.conf.d/*.conf

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
Tomas Hozza
ee197346bb image-info: read modprobe configs from multiple paths
Extend image-infro to read modprobe configuration files from multiple
paths:
 - /etc/modprobe.d/*.conf
 - /usr/lib/modprobe.d/*.conf
 - /usr/local/lib/modprobe.d/*.conf

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-04 10:48:21 +02:00
sanne
1e724be8e9 composer: More configuration of how composer is served
Allows you to disable tls
2021-10-02 14:27:02 +02:00
Simon Steinbeiss
fed4b97e4e 35
Release osbuild-composer 35

Signed-off-by: Simon Steinbeiss <simon.steinbeiss@redhat.com>
2021-10-02 09:11:07 +02:00
Achilleas Koutsou
66b76c2ace gitleaks: add allow list for test passwords and keys
Add a .gitleaks.toml file that lists all the test passwords and paths to
keys that should be ignored by the security scanner. This avoids false
positives so we can enable the scanner to catch actual leaks.

Closes #1747

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-10-01 16:56:26 +02:00
Ondřej Budai
af784a11b1 mockbuild: use download.devel of download.eng.bos
download.eng.bos.redhat.com is down, use the mirroring service instead.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-01 15:23:53 +02:00
Ondřej Budai
2070747876 ci: install ansible-core instead of ansible for EL9
ansible-core no longer provides ansible so we need to require specifically
ansible-core.

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-01 15:23:53 +02:00
Ondřej Budai
4f8dc76ca7 dnf-json: disable zchunk
See the comment

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2021-10-01 15:23:53 +02:00
Diaa Sami
cc54c4deee composer: improve logging 2021-09-30 18:34:05 +02:00
Diaa Sami
27ca55372a cloudapi: Add extra logging & improve existing
Use different logging levels depending on situation
Log HTTP 5xx errors to stdout with internal error details when possible
2021-09-30 18:34:05 +02:00
Diaa Sami
179009fec4 cloudapi: use Logrus as default logger in Echo
And remove log.Logger references
2021-09-30 18:34:05 +02:00
Diaa Sami
75372b98e1 logging: add logrus dependency
Just add the dependency and update relevant files
2021-09-30 18:34:05 +02:00
Jakub Rusz
c112e9d12d ci: don't run nightly pipeline on ga releases
It makes no sesnse to run nightly testing on ga composes. This updates
the regex to exclude ga runners from the scheduled nightly run.
2021-09-30 16:03:14 +02:00