Commit graph

2947 commits

Author SHA1 Message Date
Sanne Raymaekers
b90a5027dc sources(curl): set HTTP proxy through the environment 2024-04-08 11:56:05 +02:00
Michael Vogt
98f5904181 source: add curl test in preparation for #1573
When moving to parallel downloads in curl we will need more
comprehensive tests. This commits starts with building some
infrastructure for this.
2024-04-05 16:42:07 +02:00
Michael Vogt
cee0615f4d testutil: add http_serve_director() test helper
To test the curl sources it is very useful to have a small httpd
server that can serve an arbitrary directory. This helper will
ensure that via:
```python
with with osbuild.testutil.net.http_serve_directory(fake_httpd_root) as httpd:
    port = httpd.server_port
    # download from http://localhost:{port}/<any-path-under-httpd-root>
```
2024-04-05 16:42:07 +02:00
Michael Vogt
2f858d32e4 stages: add "devices/mounts" as allowed inputs for users/selinux
When moving to `bootc install to-filesystem` we will need support
for mounting the deployed disk and writing to the deployment root
this requires that we teach the users and selinux stages to
have them available. This is a first step towards this.

It also adds tests to ensure the options can be passed.
2024-04-05 16:13:42 +02:00
Michael Vogt
ba85d30cee stage(selinux): move to schema_2 to allow adding mounts/devices
This is a preparation to allow adding mounts/devices to the users
stage so that we can eventually support bootc install to-filesystem.
2024-04-05 16:13:42 +02:00
Michael Vogt
72a2334fbe states(users): move to schema_2 to allow adding mounts/devices
This is a preparation to allow adding mounts/devices to the users
stage so that we can eventually support bootc install to-filesystem.

It also adds some smoke tests for the schema to ensure it's still
valid.
2024-04-05 16:13:42 +02:00
schutzbot
35fbf6a377 schutzfile: Update snapshots to 20240327 2024-04-04 13:57:59 +02:00
Michael Vogt
7279c44c53 test: add StageTests.test_zip() to run zip inside a stage too
Similar to the tar test this adds a zip test that runs inside
the stages.
2024-04-04 13:54:34 +02:00
Michael Vogt
91f47ddf03 stages(zip): add test for zip stage
Add a small integration test that runs zip with various options
to ensure it works correctly.
2024-04-04 13:54:34 +02:00
Luke Yang
ac8a2a4f30 stages: add org.osbuild.zip
In CoreOS Assembler, some hyperv artifact we `zip` for compression. This
new stage is modeled after the `org.osbuild.tar` stage with necessary
modifications.
2024-04-04 13:54:34 +02:00
Achilleas Koutsou
7b004a297e test/systemd.unit: Environment option object
Add an org.osbuild.systemd.unit stage using the new format for the
Environment option with two instances to the test manifest.

The contents of the new dropin file at
tree/usr/lib/systemd/system/boltd.service.d/30-boltd-debug.conf are:

[Service]
Environment="G_MESSAGES_DEBUG=all"
Environment="G_MESSAGES_TRACE=none"
2024-04-04 10:36:04 +02:00
Achilleas Koutsou
0dc816c2f9 stages/systemd.unit: multiple Environment options
Update the org.osbuild.systemd.unit stage to also support multiple
Environment options where each is an object with {key: value}.  Enable
the allow_no_value option in configparser so we can add the multiple
entries.
2024-04-04 10:36:04 +02:00
Achilleas Koutsou
e3fd572b94 test/systemd_unit_create: add new options to unit test
Add the new options to the unit test.
2024-04-04 10:36:04 +02:00
Achilleas Koutsou
80c84020dd test/systemd.unit.create: add new options to test
Add the new options to the b.json test and update the diff.

The new file has the following contents:

[Unit]
Description=Create directory
DefaultDependencies=False
ConditionPathExists=|!/etc/myfile
ConditionPathIsDirectory=|!/etc/mydir

[Service]
Type=oneshot
RemainAfterExit=True
ExecStart=mkdir -p /etc/mydir
ExecStart=touch /etc/myfile
Environment="DEBUG=1"
EnvironmentFile=/etc/example.env

[Install]
WantedBy=local-fs.target
RequiredBy=multi-user.target
2024-04-04 10:36:04 +02:00
Achilleas Koutsou
c72f1bc54e test/systemd.unit.create: add empty file to a.json
Add an empty file to the location where the service file will be
created in the b.json version of the test.  This way, we will get a
content hash of the created file which is a slightly better test than
just knowing that it was created.

Note that, in the diff, the "before" checksum is the empty file hash:

    echo -n '' | sha256sum
    e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  -
2024-04-04 10:36:04 +02:00
Achilleas Koutsou
d1fc2cd6b6 test/systemd.unit.create: fix b.mpp.yaml
In 2d2cdd8097 the file was replaced by
the generated json and it went unnoticed in the PR.  Reverted and
updated the options to match the generated json file.
2024-04-04 10:36:04 +02:00
Achilleas Koutsou
2d3c48207b stages/systemd.unit.create: comment for allow_no_value
Add a comment explaining why we need to set allow_no_value=True for the
configparser.
2024-04-04 10:36:04 +02:00
Achilleas Koutsou
77e7c0538a stages/systemd.unit.create: Environment and EnvironmentFile
Support the Environment and EnvironmentFile options in the Service
section of the unit file.
The Environment option is set as an object with keys "key" and "value"
and the key is validated with a pattern.

Updated the stage to special-case the Environment option.
2024-04-04 10:36:04 +02:00
Michael Vogt
79d788ac23 tests: use tmp_path fixture in test_curl_source.py
Tiny tweak to remove some boilerplate related to tmpfile handling.
The pytest `tmp_path` fixture gives us the tmpdir without having
to worry about cleanup etc (and in a slightly more concise way).
2024-04-03 15:06:07 +02:00
Michael Vogt
fb701d6db5 sources: simplify test_curl_source_amend_* tests a little bit
The `amend_secrets()` does not work with real files so there is
no need to mock cachedirs or create fake input files. This commit
just removes those.

It also changes the checksum to `"1"*64` to make it very clear
that the checksum has no significance in this test.
2024-04-03 15:06:07 +02:00
Michael Vogt
fe05b3084b sources: add regression test for issue##1693
There was a regression with the secrets adding of rhsm for the
curl source. This was my mistake (sorry!). Here is a regression
test that would have prevented this (if we have had it earlier).
2024-04-03 13:55:00 +02:00
Michael Vogt
1d4f2dc53b testutil: extract find_one_subclass_in_module() helper
A small refactor to avoid shipping this duplicated code (this
one is easy to extract/reuse).
2024-04-03 11:36:01 +02:00
Michael Vogt
79360b529a sources: add new sources_service fixture
Similar to the previous commit to include a `inputs_service` fixture
this does the same for `source.SourcesService` imports.

Note that we cannot easily share the helpers so we have to life with
a bit of very similar but duplicated code. To fix this we would have
to have a shared confftest.py that pytest can find. Which would mean
that we need to put the tests under a common dir that is reachable
via __init__.py files (which we currently not have because stages,
inputs etc do not have a __init__.py so python does not considers
them modules).
2024-04-03 11:36:01 +02:00
Michael Vogt
7c0e9cfcf7 inputs: add inputs_service fixture
This new fixture avoids some repetitivness when building tests
for input services.
2024-04-03 11:36:01 +02:00
Michael Vogt
5f31ccf9f2 test: add/use new testutil.make_fake_service_fd()
All inputs/sources tests need a fake service fd to instanciate
their services. Consolidate the creation in a single helper.
2024-04-03 11:36:01 +02:00
Pierre-Yves Chibon
cccdf8b784 osbuild-mpp: Expend information in PkgInfo
This way we can extract these information later on.

Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
2024-04-03 10:29:49 +02:00
Ondřej Budai
dc85702553 Disable F40 CI (CVE-2024-3094 response) 2024-03-31 17:10:13 +02:00
Paweł Poławski
ae8f3f4a1e Stage: Schema migration to separated JSON files
In the past input schema for stages has been moved to standalone
JSON files. Looks like org.osbuild.tar was missed during the migration.
2024-03-30 11:03:29 +01:00
schutzbot
9a4f4552f5 Post release version bump
[skip ci]
2024-03-29 12:35:05 +00:00
Andre Marianiello
7e0e30fd8f curl: fix RHSM url retrieval 2024-03-29 13:02:11 +01:00
schutzbot
62e1570ef9 Post release version bump
[skip ci]
2024-03-27 08:18:03 +00:00
Michael Vogt
ec496769c5 stages: allow bootc.install-to-filesystem work without selinux
By default "bootc" will refuse to work on a non-selinux system if
the bootc container requires selinux. This is a sensible approach
in general but for us it's tricky because we want to be able to
generate images when running on developer machines or CI machines
that may not necessarily have selinux. So make bootc more relaxed.
2024-03-26 12:55:11 +01:00
Achilleas Koutsou
7f6dea860a github: add new workflow for checking json files
Reformat all json files in the repository and check that they're
formatted properly.
2024-03-26 01:20:37 +01:00
Achilleas Koutsou
0799328a6a devcontainer: fix json file
Fix the devcontainer.json (trailing comma).
2024-03-26 01:20:37 +01:00
Achilleas Koutsou
e837ebba76 schemas: reformat json files for consistencyo
Formatted with `js --indent 2 .`
2024-03-26 01:20:37 +01:00
Achilleas Koutsou
1366976dd8 test: reformat all json files
Reformat all test json files for consistency.

Formatted with `js --indent 2 .`
2024-03-26 01:20:37 +01:00
Achilleas Koutsou
fa0605b89f stages: reformat meta.json files
Reformat all meta.json files for consistency.

Formatted with `jq --indent 2 .`
2024-03-26 01:20:37 +01:00
Paweł Poławski
fa7f26a229 test: Add stage test trigger to makefile
Stage tests are located in the other place than module or run tests.
For consistency separated target in makefile is added.
2024-03-25 18:39:09 +01:00
Michael Vogt
0a72145b13 stages(chattr): fix rebase artifact 2024-03-25 18:26:53 +01:00
Renata Ravanelli
a847e6314c util: Rename function
- Rename parse_mount to find_mount_root.
 - Address other small changes

Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
2024-03-25 18:26:53 +01:00
Michael Vogt
465e55a860 util: change "assert" to raising an exception with context in parse_location() 2024-03-25 18:26:53 +01:00
Michael Vogt
249107a028 stages,test: fix lint errors and add basic unit tests
Add very simple unit tests as a starting point for the new
parsing functions in `util/parsing.py`.
2024-03-25 18:26:53 +01:00
Renata Ravanelli
6d4d1962eb util: Consolidate parse functions into util
- Move functions to the 'util' to centralize common
functionality, reducing code duplication and improving
maintainability across the codebase.

Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
2024-03-25 18:26:53 +01:00
Michael Vogt
4f8ff6bc99 workflows: improve speed of test_stages and test_assemblers
The other tests have concurrency issues at the moment.

It seems in https://github.com/osbuild/osbuild/pull/1655 we lost
parallel running. The issue seems to be that `env` does not
actually define a shell environment but only a github environment.

This sets commit sets the shell env explicitly to unbreak us
again.
2024-03-23 15:38:34 +01:00
Michael Vogt
c5655c9006 osbuild: make the entire /etc/selinux avaialble for the buildroot
When moving to `bootc install to-filesystem` we need more information
for bootc from /etc/selinux than our current /etc/selinux/targeted/contexts
policy.

This commit makes all of /etc/selinux available which unblocks
the bootc install.
2024-03-22 11:24:35 +01:00
Tom Koscielniak
f9eea44083 Update manifest-db dependency to remove failing tests 2024-03-21 15:46:48 +01:00
Tom Koscielniak
3b7a8aa728 Migrate manifest tests to new RHEL GA version 2024-03-21 15:46:48 +01:00
Michael Vogt
345516e867 osbuild: ensure /var/tmp is a real directory
This is a followup for https://github.com/osbuild/osbuild/pull/1649

Instead of symlinking /var/tmp to /tmp which may be on a tmpfs
this commit puts it on a real filesystem.

This should fix:
https://github.com/osbuild/bootc-image-builder/issues/285
2024-03-20 20:36:26 +01:00
Michael Vogt
0528ccc3f0 osbuild: add support to exclude_paths to setfiles()
This is needed because on a mounted `bootc` container `setfiles`
without excluding `/sysroot` will create many warnings like:
```
setfiles: conflicting specifications for /run/osbuild/tree/sysroot/ostree/repo/objects/00/0ef9ada2ee87792e8ba21afd65aa00d79a1253018832652b8694862fb80e84.file and /run/osbuild/tree/usr/lib/firmware/cirrus/cs35l41-dsp1-spk-prot-103c8b8f-r1.bin.xz, using system_u:object_r:lib_t:s0.
```
but simply excluding this dir fixes them.
2024-03-20 18:05:51 +01:00
Michael Vogt
ba08a524a4 github: add cache key to generate.yml flow
There are recent GH failures because the github cache action
complains:
```
Input required and not supplied: key
```
This is slightly odd as we did not provide a cache key before and
it was fine but *shrug*. We also don't really need a cache key,
we always get the same cache, osbuild is smart enough to figure
it out.
2024-03-20 14:20:43 +01:00