Commit graph

2955 commits

Author SHA1 Message Date
Brian C. Lane
21c1573ae8 generate.yml: Bump actions/checkout to v4 2024-04-10 01:32:51 +02:00
Brian C. Lane
3666aa510a coverity.yml: Bump actions/checkout to v4 2024-04-10 01:32:51 +02:00
Brian C. Lane
922e22758d check.yml: Bump actions/checkout to v4 2024-04-10 01:32:51 +02:00
Paweł Poławski
f3a5267e65 Tests: Add unit tests for org.osbuild.mkfs.xfs stage 2024-04-09 23:56:06 +02:00
Michael Vogt
38bcef9378 stages(mkdir): allow to pass in devices/mounts
This allows to combine `org.osbuild.mkdir` with the `osbuild.deployment`
mount and with the upcoming `org.osbuild.bind` mount. The use case is
that we need to create the dir `/var/home` so that `useradd` from inside
a ostree root works (there /home is a symlink and useradd will not
follow the symlink and create a dir in the target by itself).

This allows to write:
```json
        {
          "type": "org.osbuild.mkdir",
          "options": {
            "paths": [
	      {
		"path": "/var/home"
	      }
	    ]
          },
          "devices": {
            "disk": {
              "type": "org.osbuild.loopback",
              "options": {
                "filename": "disk.raw",
                "partscan": true
              }
            }
          },
          "mounts": [
            {
              "name": "part4",
              "type": "org.osbuild.ext4",
              "source": "disk",
              "target": "/",
              "partition": 4
            },
            {
              "name": "part3",
              "type": "org.osbuild.ext4",
              "source": "disk",
              "target": "/boot",
              "partition": 3
            },
            {
              "name": "part2",
              "type": "org.osbuild.fat",
              "source": "disk",
              "target": "/boot/efi",
              "partition": 2
            },
            {
              "name": "ostree.deployment",
              "type": "org.osbuild.ostree.deployment",
              "options": {
                "source": "mount",
                "deployment": {
                  "default": true
                }
              }
            },
            {
              "name": "bind",
              "type": "org.osbuild.bind",
	      "target": "tree://",
	      "options": {
		"source": "mount://"
	      }
            }
          ]
        },
```
to fix this.
2024-04-09 17:12:20 +02:00
Paweł Poławski
488eee7bc0 Tests: Add unit tests for org.osbuild.mkfs.btrfs stage 2024-04-09 00:39:11 -07:00
Michael Vogt
34ad069757 sources(curl): tweak tests to use monkeypatch.setenv()
Using pytests support for changing setenv() in tests makes things
a little bit more concise.
2024-04-09 03:03:38 +02:00
Michael Vogt
b9b296a7e5 testutil: add AtomicCounter() as a threadsafe counter
The existing code in the reqs counting is not really thread safe,
this commit fixes that.
2024-04-09 03:02:45 +02:00
Sanne Raymaekers
b90a5027dc sources(curl): set HTTP proxy through the environment 2024-04-08 11:56:05 +02:00
Michael Vogt
98f5904181 source: add curl test in preparation for #1573
When moving to parallel downloads in curl we will need more
comprehensive tests. This commits starts with building some
infrastructure for this.
2024-04-05 16:42:07 +02:00
Michael Vogt
cee0615f4d testutil: add http_serve_director() test helper
To test the curl sources it is very useful to have a small httpd
server that can serve an arbitrary directory. This helper will
ensure that via:
```python
with with osbuild.testutil.net.http_serve_directory(fake_httpd_root) as httpd:
    port = httpd.server_port
    # download from http://localhost:{port}/<any-path-under-httpd-root>
```
2024-04-05 16:42:07 +02:00
Michael Vogt
2f858d32e4 stages: add "devices/mounts" as allowed inputs for users/selinux
When moving to `bootc install to-filesystem` we will need support
for mounting the deployed disk and writing to the deployment root
this requires that we teach the users and selinux stages to
have them available. This is a first step towards this.

It also adds tests to ensure the options can be passed.
2024-04-05 16:13:42 +02:00
Michael Vogt
ba85d30cee stage(selinux): move to schema_2 to allow adding mounts/devices
This is a preparation to allow adding mounts/devices to the users
stage so that we can eventually support bootc install to-filesystem.
2024-04-05 16:13:42 +02:00
Michael Vogt
72a2334fbe states(users): move to schema_2 to allow adding mounts/devices
This is a preparation to allow adding mounts/devices to the users
stage so that we can eventually support bootc install to-filesystem.

It also adds some smoke tests for the schema to ensure it's still
valid.
2024-04-05 16:13:42 +02:00
schutzbot
35fbf6a377 schutzfile: Update snapshots to 20240327 2024-04-04 13:57:59 +02:00
Michael Vogt
7279c44c53 test: add StageTests.test_zip() to run zip inside a stage too
Similar to the tar test this adds a zip test that runs inside
the stages.
2024-04-04 13:54:34 +02:00
Michael Vogt
91f47ddf03 stages(zip): add test for zip stage
Add a small integration test that runs zip with various options
to ensure it works correctly.
2024-04-04 13:54:34 +02:00
Luke Yang
ac8a2a4f30 stages: add org.osbuild.zip
In CoreOS Assembler, some hyperv artifact we `zip` for compression. This
new stage is modeled after the `org.osbuild.tar` stage with necessary
modifications.
2024-04-04 13:54:34 +02:00
Achilleas Koutsou
7b004a297e test/systemd.unit: Environment option object
Add an org.osbuild.systemd.unit stage using the new format for the
Environment option with two instances to the test manifest.

The contents of the new dropin file at
tree/usr/lib/systemd/system/boltd.service.d/30-boltd-debug.conf are:

[Service]
Environment="G_MESSAGES_DEBUG=all"
Environment="G_MESSAGES_TRACE=none"
2024-04-04 10:36:04 +02:00
Achilleas Koutsou
0dc816c2f9 stages/systemd.unit: multiple Environment options
Update the org.osbuild.systemd.unit stage to also support multiple
Environment options where each is an object with {key: value}.  Enable
the allow_no_value option in configparser so we can add the multiple
entries.
2024-04-04 10:36:04 +02:00
Achilleas Koutsou
e3fd572b94 test/systemd_unit_create: add new options to unit test
Add the new options to the unit test.
2024-04-04 10:36:04 +02:00
Achilleas Koutsou
80c84020dd test/systemd.unit.create: add new options to test
Add the new options to the b.json test and update the diff.

The new file has the following contents:

[Unit]
Description=Create directory
DefaultDependencies=False
ConditionPathExists=|!/etc/myfile
ConditionPathIsDirectory=|!/etc/mydir

[Service]
Type=oneshot
RemainAfterExit=True
ExecStart=mkdir -p /etc/mydir
ExecStart=touch /etc/myfile
Environment="DEBUG=1"
EnvironmentFile=/etc/example.env

[Install]
WantedBy=local-fs.target
RequiredBy=multi-user.target
2024-04-04 10:36:04 +02:00
Achilleas Koutsou
c72f1bc54e test/systemd.unit.create: add empty file to a.json
Add an empty file to the location where the service file will be
created in the b.json version of the test.  This way, we will get a
content hash of the created file which is a slightly better test than
just knowing that it was created.

Note that, in the diff, the "before" checksum is the empty file hash:

    echo -n '' | sha256sum
    e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  -
2024-04-04 10:36:04 +02:00
Achilleas Koutsou
d1fc2cd6b6 test/systemd.unit.create: fix b.mpp.yaml
In 2d2cdd8097 the file was replaced by
the generated json and it went unnoticed in the PR.  Reverted and
updated the options to match the generated json file.
2024-04-04 10:36:04 +02:00
Achilleas Koutsou
2d3c48207b stages/systemd.unit.create: comment for allow_no_value
Add a comment explaining why we need to set allow_no_value=True for the
configparser.
2024-04-04 10:36:04 +02:00
Achilleas Koutsou
77e7c0538a stages/systemd.unit.create: Environment and EnvironmentFile
Support the Environment and EnvironmentFile options in the Service
section of the unit file.
The Environment option is set as an object with keys "key" and "value"
and the key is validated with a pattern.

Updated the stage to special-case the Environment option.
2024-04-04 10:36:04 +02:00
Michael Vogt
79d788ac23 tests: use tmp_path fixture in test_curl_source.py
Tiny tweak to remove some boilerplate related to tmpfile handling.
The pytest `tmp_path` fixture gives us the tmpdir without having
to worry about cleanup etc (and in a slightly more concise way).
2024-04-03 15:06:07 +02:00
Michael Vogt
fb701d6db5 sources: simplify test_curl_source_amend_* tests a little bit
The `amend_secrets()` does not work with real files so there is
no need to mock cachedirs or create fake input files. This commit
just removes those.

It also changes the checksum to `"1"*64` to make it very clear
that the checksum has no significance in this test.
2024-04-03 15:06:07 +02:00
Michael Vogt
fe05b3084b sources: add regression test for issue##1693
There was a regression with the secrets adding of rhsm for the
curl source. This was my mistake (sorry!). Here is a regression
test that would have prevented this (if we have had it earlier).
2024-04-03 13:55:00 +02:00
Michael Vogt
1d4f2dc53b testutil: extract find_one_subclass_in_module() helper
A small refactor to avoid shipping this duplicated code (this
one is easy to extract/reuse).
2024-04-03 11:36:01 +02:00
Michael Vogt
79360b529a sources: add new sources_service fixture
Similar to the previous commit to include a `inputs_service` fixture
this does the same for `source.SourcesService` imports.

Note that we cannot easily share the helpers so we have to life with
a bit of very similar but duplicated code. To fix this we would have
to have a shared confftest.py that pytest can find. Which would mean
that we need to put the tests under a common dir that is reachable
via __init__.py files (which we currently not have because stages,
inputs etc do not have a __init__.py so python does not considers
them modules).
2024-04-03 11:36:01 +02:00
Michael Vogt
7c0e9cfcf7 inputs: add inputs_service fixture
This new fixture avoids some repetitivness when building tests
for input services.
2024-04-03 11:36:01 +02:00
Michael Vogt
5f31ccf9f2 test: add/use new testutil.make_fake_service_fd()
All inputs/sources tests need a fake service fd to instanciate
their services. Consolidate the creation in a single helper.
2024-04-03 11:36:01 +02:00
Pierre-Yves Chibon
cccdf8b784 osbuild-mpp: Expend information in PkgInfo
This way we can extract these information later on.

Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
2024-04-03 10:29:49 +02:00
Ondřej Budai
dc85702553 Disable F40 CI (CVE-2024-3094 response) 2024-03-31 17:10:13 +02:00
Paweł Poławski
ae8f3f4a1e Stage: Schema migration to separated JSON files
In the past input schema for stages has been moved to standalone
JSON files. Looks like org.osbuild.tar was missed during the migration.
2024-03-30 11:03:29 +01:00
schutzbot
9a4f4552f5 Post release version bump
[skip ci]
2024-03-29 12:35:05 +00:00
Andre Marianiello
7e0e30fd8f curl: fix RHSM url retrieval 2024-03-29 13:02:11 +01:00
schutzbot
62e1570ef9 Post release version bump
[skip ci]
2024-03-27 08:18:03 +00:00
Michael Vogt
ec496769c5 stages: allow bootc.install-to-filesystem work without selinux
By default "bootc" will refuse to work on a non-selinux system if
the bootc container requires selinux. This is a sensible approach
in general but for us it's tricky because we want to be able to
generate images when running on developer machines or CI machines
that may not necessarily have selinux. So make bootc more relaxed.
2024-03-26 12:55:11 +01:00
Achilleas Koutsou
7f6dea860a github: add new workflow for checking json files
Reformat all json files in the repository and check that they're
formatted properly.
2024-03-26 01:20:37 +01:00
Achilleas Koutsou
0799328a6a devcontainer: fix json file
Fix the devcontainer.json (trailing comma).
2024-03-26 01:20:37 +01:00
Achilleas Koutsou
e837ebba76 schemas: reformat json files for consistencyo
Formatted with `js --indent 2 .`
2024-03-26 01:20:37 +01:00
Achilleas Koutsou
1366976dd8 test: reformat all json files
Reformat all test json files for consistency.

Formatted with `js --indent 2 .`
2024-03-26 01:20:37 +01:00
Achilleas Koutsou
fa0605b89f stages: reformat meta.json files
Reformat all meta.json files for consistency.

Formatted with `jq --indent 2 .`
2024-03-26 01:20:37 +01:00
Paweł Poławski
fa7f26a229 test: Add stage test trigger to makefile
Stage tests are located in the other place than module or run tests.
For consistency separated target in makefile is added.
2024-03-25 18:39:09 +01:00
Michael Vogt
0a72145b13 stages(chattr): fix rebase artifact 2024-03-25 18:26:53 +01:00
Renata Ravanelli
a847e6314c util: Rename function
- Rename parse_mount to find_mount_root.
 - Address other small changes

Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
2024-03-25 18:26:53 +01:00
Michael Vogt
465e55a860 util: change "assert" to raising an exception with context in parse_location() 2024-03-25 18:26:53 +01:00
Michael Vogt
249107a028 stages,test: fix lint errors and add basic unit tests
Add very simple unit tests as a starting point for the new
parsing functions in `util/parsing.py`.
2024-03-25 18:26:53 +01:00