Commit graph

2828 commits

Author SHA1 Message Date
Simon de Vlieger
82218c91a1 tox: remove isort
The role of the `isort` check is now performed by `ruff` instead of
`isort`.
2024-03-05 16:10:27 +01:00
Simon de Vlieger
6536a45d0a lint: explicit check
Add an explicit check to the return value of subprocess.run.
2024-03-05 16:10:27 +01:00
Simon de Vlieger
c9739dbd2a lint: sort import(s) 2024-03-05 16:10:27 +01:00
Simon de Vlieger
a779d3268d ruff: configure
Newer versions of ruff support more linters and autoformatting, let's
configure those to do the same as our current linter selection.
2024-03-05 16:10:27 +01:00
Simon de Vlieger
7fc06acffc tox: update ruff dependency
Renames a property that should now live under `lint` to silence a
warning in newer `ruff`.
2024-03-05 16:10:27 +01:00
Achilleas Koutsou
a0f5262701 gitlab: don't update GitHub status from individual jobs
Never update the GitHub status at the end of a job.  Instead, when
everything is done, the 'fail' job will run in the 'finish' stage and
update the status accordingly.
2024-03-05 16:04:20 +01:00
Achilleas Koutsou
f7dd0878b9 gitlab: new pipeline 'fail'
New pipeline called fail that runs in the 'finish' stage but only when a
job fails.  The job updates the github status to indicate failure.
2024-03-05 16:04:20 +01:00
Achilleas Koutsou
b0abe9725d schutzbot/update_github_status.sh: new verb: fail
New script subcommand that explicitly reports a failure.
2024-03-05 16:04:20 +01:00
Ondřej Budai
57bba13752 gitlab: parallelize the manifest db tests even more
Why not, we can run a lot of VMs in parallel, and this will speed up the test
runs.
2024-03-05 12:49:21 +02:00
Ondřej Budai
08d4bbf4dd test/ostree: parallelize the test
Building 7 images on one machine is quite slow. Instead, let's spawn build
them on separate ones to save some time.
2024-03-05 12:49:21 +02:00
Ondřej Budai
6750c0fd6a gitlab: Stop running the ostree tests in the internal network
It's just Fedora, there's nothing internal.
2024-03-05 12:49:21 +02:00
Ondřej Budai
8632269f79 gitlab: Run the ostree tests just on Fedora
These tests are building just Fedora manifests, thus there's basically
no point in running them on RHEL, since we never support building
Fedora on RHEL.
2024-03-05 12:49:21 +02:00
Simon Steinbeiss
b8e0253493 actions: Add a PR best practices check
This pipeline tests:
1. If the PR description is not empty (blocking)
2. If the PR title follows our format (non-blocking)
   `component: This is the change (JIRA-001)`
3. If 1. and 2. are True, it adds a 'best practice' label to the PR
2024-03-05 12:01:10 +02:00
Michael Vogt
ea36e25b09 workflow: run assembler tests in parallel
The assembler tests currently run around 45min, try to run in
parallel to see how much we can win from that.
2024-03-05 07:44:26 +01:00
Michael Vogt
4d2476a26d test: use OSBUILD_TEST_STORE in test_assemblers.py too
Use the OSBUILD_TEST_STORE in the test_assemblers.py file too
and re-use already downloaded sources.
2024-03-05 07:44:26 +01:00
Jakub Rusz
95c1b90d33 Schutzbot: Remove RH-IT-Root-CA 2024-03-04 23:53:36 +01:00
Simon Steinbeiss
75bb706cb0 actions/test: Only test assemblers on Python3.6
Assemblers are only part of v1 manifests, so the only Python version we
need to test is 3.6 (RHEL8).
2024-03-04 14:39:50 +01:00
Simon Steinbeiss
3de3bddc10 actions/sonarqube: Drop SonarQube in favor of Snyk 2024-03-04 14:14:03 +01:00
Ondřej Budai
ec096f449b osbuild-mpp: disable zchunks
See the comment.
2024-03-04 14:06:33 +01:00
Michael Vogt
ba732b8532 workflow, osbuild-mpp: run with cache and use GH cache
Share cache between runs and also put into GH cache accross runs.
2024-03-04 14:06:33 +01:00
Michael Vogt
1ea7b4943c workflow: drop python3.11 from matrix (HMS-3697)
With fedora moving to python3.12 we can stop testing on py311.

As a drive-by this commit also documents why we need to test on
py36 and py39.
2024-03-04 13:51:41 +01:00
Michael Vogt
9baca1fe90 stages: add small unit test for the gzip stage 2024-03-04 11:18:13 +01:00
Luke Yang
cfaabe618f stages/org.osbuild.gzip: add compression level option
Allow compression level to be specified instead of defaulting to 1. This is needed for CoreOS Assembler.
2024-03-04 11:18:13 +01:00
Michael Vogt
30f740ec9f stages: add test for coreos.platform generate_console_settings_file
And a tiny tweak to avoid an empty line at the start of the
`boot/grub2/console.cfg`
2024-02-28 10:37:01 +01:00
Michael Vogt
e9c31c035b stages: tweak process_platforms_json to be slightly shorter
With the test the helper can now be slightly simplified. Because
we only have two results it seems easier to just use them directly
than to store them in an intermediate result struct.
2024-02-28 10:37:01 +01:00
Michael Vogt
6cc7309890 stages: add unit test for coreos.platform process_platforms_json
Add unit test for `coreos.platform` process_platforms_json helper.
2024-02-28 10:37:01 +01:00
Michael Vogt
d38665a2af util: tweak bls.options_append() support no/multiple options
The BLS specification [0] says the `options` field is optional and
can also appear multiple times. This commit tweaks the code to
deal with these corner cases and also adds tests that ensure that
this works correctly.

It also tweaks the file handling to be atomic.

[0] https://uapi-group.org/specifications/specs/boot_loader_specification/
2024-02-28 10:37:01 +01:00
schutzbot
87dce84b90 Post release version bump
[skip ci]
2024-02-28 08:13:32 +00:00
Luke Yang
7709b7caeb stages/org.osbuild.tar: add sparse option
Add a sparse option to the tar stage which make files in the archive sparse if they are sparse on disk. This is needed for CoreOS Assember.
2024-02-27 16:46:19 -05:00
Michael Vogt
b0091c0470 inputs: simplify/comment clean of test_containers_local_inputs_integration
Thanks to Ondrej fot the suggestions
2024-02-27 15:07:42 +01:00
Michael Vogt
82f2414637 sources: tweak ContainersStorageSources.exists to return False
When an images does not exist just return `False` instead of
raising a RuntimeError. If anything else goes wrong (unknown
output or hash mismatch) keep the RuntimeError as this is an
unexpected exception.
2024-02-27 15:07:42 +01:00
Michael Vogt
5ab0b41456 sources: add test for non-existing id in containers-storage 2024-02-27 15:07:42 +01:00
Michael Vogt
0ac05ecb55 sources: tweak docstring for containers-storage 2024-02-27 15:07:42 +01:00
Michael Vogt
c219160b8f osbuild: add warning when lazy umount in containers_storage_source fails 2024-02-27 15:07:42 +01:00
Michael Vogt
b5355c46ac inputs: tweak containers-storage implementation
This commit tweaks the excellent work from Gianluca in PR#1550 a
little bit. Now that the container inputs are their own input
type some of the code that used to be part of the original
`inputs/org.osbuild.containers` can be simplified.
2024-02-27 15:07:42 +01:00
Michael Vogt
882c9efbc1 inputs: add integration test for inputs/org.osbuild.containers-storage
Ensure that `ContainersInput.map()` work correctly with local
image store references.
2024-02-27 15:07:42 +01:00
Michael Vogt
2d71935a91 inputs: add new "inputs_module" fixture 2024-02-27 15:07:42 +01:00
schutzbot
3824c9c868 Post release version bump
[skip ci]
2024-02-26 15:06:35 +00:00
Renata Ravanelli
0b7dcf3cf4 Add coreos.platforms stage
- Process all necessary operations related to CoreOS
platforms is crucial and specific to CoreOS. This step
is essential for CoreOS exclusively.

- Our approach to handling 'platforms.json' may change as we
advance with the OSBuild work. However, we don't have a clear
vision about how it will be in the future yet, particularly as
we also manage similar components within the osbuild composer
to configure cloud parameters. We probably will know better
when we start working with the cloud artifacts.

As a summary, let's add it know to unblock us, and if we find a
better approach in the future, we can always go back and remove it.

Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
2024-02-26 16:02:28 +01:00
Renata Ravanelli
ddf4478ea9 util: Add bls module
- Add functions for appending kernel parameters to the
Boot Loader Specification (BLS) as needed.

Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
2024-02-26 16:02:28 +01:00
Tomáš Hozza
2b868fbe75 Sources/containers-storage: make the code Python 3.6 compliant
The source implementation used `subprocess.run()` argument
`capture_output`, which was added in Python 3.7. Since the minimum
supported Python version for osbuild on RHEL-8 is 3.6, the stage fails
with TypeError.

Example failure: https://artifacts.dev.testing-farm.io/c147b608-c40e-46ed-bf11-6b15ecf718dc/

Signed-off-by: Tomáš Hozza <thozza@redhat.com>
2024-02-25 09:27:23 +01:00
schutzbot
badca79801 Post release version bump
[skip ci]
2024-02-22 11:53:23 +00:00
Michael Vogt
322974695c stages(dracut): add small unittest for initoverlayfs
Small followup for https://github.com/osbuild/osbuild/pull/1586
that includes a basic check that the initoverlayfs option calls
the right binary.
2024-02-22 11:59:21 +01:00
Gianluca Zuccarelli
6b0d2d7a3c stages/ostree-container-deploy: mkdir /var/tmp
This stage crashes when using `containers-storage` if `/var/tmp`
doesn't exist, since the directory is required by `skopeo` for big
files. See:
4ad2f46269/storage/storage_src.go (L163)
2cb3c9d3c2/internal/tmpdir/tmpdir.go (L18)
2024-02-22 11:58:54 +01:00
Achilleas Koutsou
b42e1afddc github: change container storage driver inside test container
The default overlayfs doesn't work inside the container runner which
causes the test to fail with
  Invalid destination name
  containers-storage:localhost/osbuild-skopeo-test-...:
  'overlay' is not supported over overlayfs, a mount_program is
  required: backing file system is unsupported for this graph driver

Changing the driver to vfs solves the issue.
2024-02-21 17:55:37 +01:00
Achilleas Koutsou
e7400d2e59 test/stages: read expected data from manifest
Parse the manifest to dynamically discover the expected data in the
destination so we don't have to keep the test updated if the manifest
changes.
2024-02-21 17:55:37 +01:00
Achilleas Koutsou
700dc8ded3 test/stages: build manifest and check tree
Copy hello.img into the host storage, build the manifest, export the
tree, and check the storage in the tree for the expected files and data.
2024-02-21 17:55:37 +01:00
Achilleas Koutsou
78ea8405a4 test: add test manifest for containers-storage
A manifest (mpp and json) that uses the new source and input with the
skopeo stage.

This depends on the image we store at
./test/data/stages/skopeo/hello.img

The plan is to test this by pulling the hello.img into the host root
storage, build the manifest, delete the image from storage, and check
the tree.
2024-02-21 17:55:37 +01:00
Achilleas Koutsou
423819a80e util/containers: add the checksum to the image data
When parsing a container input, add the checksum to the data as well.

Usually with other inputs, the stage only needs to know the filepath
where it can find the source content.  In most (all, so far) cases, this
is a checksum appended to the content type.

In this case, the filepath is the location of the storage bind mount and
the checksum is needed to retrieve the container.  The name might only
be a destination name (a name to use for storing the container in the
image), so we can't rely on it being valid in the source.
2024-02-21 17:55:37 +01:00
Achilleas Koutsou
591593ea00 testutil: make_container context manager
Make make_container a context manager so we can reliably clean up
containers that were created in tests.
2024-02-21 17:55:37 +01:00