Commit graph

2687 commits

Author SHA1 Message Date
Michael Vogt
94d8a1357f test: check that mkfs.fat has the -g option in test_fat
Older versions of RHEL/Centos do not have `mkfs.fat -g` yet so
this test will fail. Detect this and skip the test if mkfs.fat
is too old (see 7af2f1a for the original commit).
2024-01-18 08:44:15 +01:00
Michael Vogt
ff10cdb4db test: export schemas in testing_libdir_fixture
When constructing a minimal environment for osbuild the exported
dirs lacked "schemas" so the test_exports.py test failed on RHEL8.

This commit adds it (and also "assemblers" for good measure). With
that the test will pass.
2024-01-18 08:44:07 +01:00
Dusty Mabe
94b10403a0 schutzbot: add dustymabe SSH key to team_ssh_keys
This will allow me to access a CI runner for debugging a CI failure.
2024-01-17 19:54:19 +01:00
Michael Vogt
84059544e4 test: fix test_schema_validation_containers_storage_conf
The test starts failing because a new version of jsonschema (4.21.0)
changed the error messages for `minProperties: 1`.

To fix this we just use a regex and check for both possible values.
As a drive-by the commit also improves the error output in case
the match is not found.
2024-01-17 13:58:22 +01:00
schutzbot
c4ff215149 Post release version bump
[skip ci]
2024-01-16 13:48:47 +00:00
Michael Vogt
e0bbb3397d test: update test_mount.py for lsblk --json differences
Older version of `lsblk --json` will not have the plural `mountpoints`
but only a singular `mountpoint`. But newer version lost the
singular `mountpoint` in the json. Adjust the test accordingly.
2024-01-16 13:51:30 +01:00
Michael Vogt
d96f94fede test: fix new mount tests under rhel8
The new `create_image_with_partitions()` helper fails under rhel8
currently. The reason is that `mkfs.ext4 -E offset=` will warn
in older versions about a partition table and require user input.

This got fixed `e2fsprogs` 1.46.3 in Jul 2021 but RHEL8 still
has 1.45.

[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989612
2024-01-16 13:51:30 +01:00
Dusty Mabe
d50705249a osbuild-mpp: fix float correction
I mistakenly thought the .strip('.0') would strip off just '.0` but
in reality it strips off all instances of `.` or `0` on the end so
'4075520.0'.strip('.0') => '407552', which is clearly wrong.

This is a fixup for 5207e92.
2024-01-15 11:53:21 +01:00
Dusty Mabe
7d73c0e741 manifests: move osname to a var in fcos manifest
This makes it easier to see/change the value globally.
2024-01-15 11:53:21 +01:00
Dusty Mabe
d9fe4221b0 manifests: move boot/root fs uuid/label to var in fcos manifest
This makes it easier to see the important unique values in our
manifest.
2024-01-15 11:53:21 +01:00
Dusty Mabe
ac8a57c5a3 manifests: set sysroot.readonly config to true in fcos manifest
This is set in Fedora CoreOS today. Let's make it match here.
2024-01-15 11:53:21 +01:00
Dusty Mabe
7cf98cfbf1 manifests: remove partition UUIDs from fcos container manifest
We don't set the UUIDs today in COSA/create_disk.sh so let's not
statically set them here either. This means they'll get set to
something random.
2024-01-15 11:53:21 +01:00
Luke Yang
d089b429a7 stages/org.osbuild.ostree.config: support bls-append-except-default
Support setting the sysroot.bls-append-except-default value in the
OSTree config. This is used by CoreOS to support configuration used
for GRUB password support https://github.com/coreos/fedora-coreos-tracker/issues/1333
2024-01-11 12:55:12 -05:00
Michael Vogt
73ec3122f2 stages,test: update tests for new selinux.setfiles() calling 2024-01-11 10:23:25 +01:00
Michael Vogt
467a23ffa7 stages(selinux): use osbuild.util.selinux.setfiles()
Instead of calling `setfiles` manually just reuse the existing
helper. The tests validate that there is no functional change.
2024-01-11 10:23:25 +01:00
Michael Vogt
18159eceec test: add selinux stage test 2024-01-11 10:23:25 +01:00
Dusty Mabe
83a14886d3 add --break for requesting a debug shell
Similar to rd.break for dracut this allows a user to specify:

- --break or --break=*
    - to get a shell before each stage is run
- --break=stage.name
    - to get a shell each time the stage with that name is run
    - example: --break=org.osbuild.copy
- --break=stage.id
    - to get a shell each time the stage with that ID is run
    - get the ID for the stages for your manifest by running
      osbuild on the manifest with --inspect
    - example: --break=dc6e3a66fef3ebe7c815eb24d348215b9e5e2ed0cd808c15ebbe85fc73181a86

and get a bash shell where they can inspect the environment to debug
and develop OSBuild stages.
2024-01-10 08:39:27 -08:00
Michael Vogt
962b7f4d4b test: add test for export option 2024-01-10 11:24:01 +01:00
Dusty Mabe
8cce659ec5 stages/bootupd: support installing to a partition of a device
For ppc64le we need to pass in a partition (i.e. /dev/loop0p1) rather
than the root device (/dev/loop0) to the --device argument of bootupctl.
Let's add a partition field and find the device node based on the user
specified partition.

On ppc64le this would look something like:

```
      - type: org.osbuild.bootupd
        options:
          bios:
            device: disk
            partition:
              mpp-format-int: '{image.layout[''POWERPC-PREP-BOOT''].partnum}'
          static-configs: true
          deployment:
            osname: fedora-coreos
            ref: ostree/1/1/0
        devices:
          disk:
            type: org.osbuild.loopback
            options:
              filename: disk.img
              partscan: true
        mounts:
          - name: root
            type: org.osbuild.xfs
            source: disk
            partition:
              mpp-format-int: '{image.layout[''root''].partnum}'
            target: /
          - name: boot
            type: org.osbuild.ext4
            source: disk
            partition:
              mpp-format-int: '{image.layout[''boot''].partnum}'
            target: /boot
```
2024-01-10 01:15:18 +01:00
Michael Vogt
492dc6021c stages(bootupd): refactor test to separately test behaviors
It now tests  bind_mounts and bootupd behavior separately.
2024-01-10 01:15:18 +01:00
Michael Vogt
dacf5733ea stages(bootupd): add tests for existing behavior
Also refactor bind mounts into a helper.
2024-01-10 01:15:18 +01:00
Michael Vogt
eb657b08b3 stages(bootupd): add schema test 2024-01-10 01:15:18 +01:00
Dusty Mabe
fdd5d25c1e tests/fedora-coreos-container: switch to testing FCOS
The stable stream currently doesn't have a new enough bootupd to pass
the tests for the bootupd stage. Let's update to `:testing` for now
and we'll switch back to `:stable` later.
2024-01-10 01:15:18 +01:00
Renata Ravanelli
cdcb4598bb create org.osbuild.bootupd stage
Add the bootupd stage to install GRUB on both BIOS and UEFI systems,
ensuring that your bootloader stays up-to-date.

Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
2024-01-10 01:15:18 +01:00
Michael Vogt
c03e6be52a stages(grub2): allow pulling efi binaries from alternative efi roots
The EFI binaries are currently pulled from a hardcoded path in the
buildroot. When moving to containers as buildroots this will no
longer work as they have an alternative layout. This is an easy
"fix" - make the location of the `EFI/` directory configurable.

This allows us set `efi_src_dir` to `/usr/lib/bootupd/updates/EFI/`
and keep our existing `bootc-image-builder` workflow.

Note that this may actually not be the desired solution and instead
we want the new `bootupd`: https://github.com/osbuild/osbuild/pull/1519
2024-01-09 19:43:01 +01:00
Michael Vogt
cb02d0a4bc stages(container-deploy): ensure /var/tmp is available
This commit ensures that `/var/tmp` is available. Skopeo expects
this dir but the bwrap environment starts with a very minimal
`/var` so `/var/tmp` may not be available.
2024-01-09 12:54:11 +01:00
Dusty Mabe
c62e555202 tools/osbuild-mpp: run _process_format() for mpp-embed dict
So we can use something like mpp-format-string inside of mpp-embed. An
example would be below where we want to substitute the ociarchive var
in using `mpp-format-string: 'file://{ociarchive}'`.

```
version: '2'
mpp-vars:
  ociarchive: /path/to/fedora-coreos-39.20240104.dev.2-ostree.x86_64.ociarchive
pipelines:
  - name: oci-archive
    stages:
      - type: org.osbuild.copy
        inputs:
          inlinefile:
            type: org.osbuild.files
            origin: org.osbuild.source
            mpp-embed:
              id: fcos.ociarchive
              url:
                mpp-format-string: 'file://{ociarchive}'
        options:
          paths:
            - from:
                mpp-format-string: input://inlinefile/{embedded['fcos.ociarchive']}
              to: tree:///fcos.ociarchive
```
2024-01-05 18:57:43 +01:00
Gianluca Zuccarelli
a2b2a02add sources/skopeo: add storage location
For local images copied from an image store other than the default, we
need to be able to specify the `storage-location`. This commit enables
this functionality.

Jira: https://issues.redhat.com/browse/HMS-3235
2024-01-05 16:42:51 +01:00
Michael Vogt
ac5653d9f1 stages: rework container-deploy stage to not use tmp storage
This commit reworks the `org.osbuild.container-deploy` stage to
not use a tmp storage when mounting the container image. This
is needed because of [0] but it should generally be fine because
inside the stages the real /var is a tmpfs (which is why we
triggered the bug in the first place).

[0] https://github.com/containers/storage/issues/1779
2024-01-03 11:25:48 -08:00
Michael Vogt
9a8b0ddb61 test: add test for org.osbuild.container-deploy 2024-01-03 11:25:48 -08:00
Ondřej Budai
fdc5bf3b98 stages: org.osbuild.container-deploy 2024-01-03 11:25:48 -08:00
Michael Vogt
ad8fd2f532 testutil: extrace new make_fake_tree() helper
Extract a new helper `make_fake_tree()` that generalizes the existing
helper `make_fake_input_tree()`. The later will always create the
content under `{basedir}/tree` which is convinient for input tree
based tests but too specialized when using it in different contexts.

The existing `make_fake_input_tree()` is preserved unchanged and
becomes just a tiny wrapper.
2024-01-03 11:25:48 -08:00
schutzbot
25d198da3c Post release version bump
[skip ci]
2024-01-03 08:14:11 +00:00
Michael Vogt
8c95bd9dd7 test: fix autopep8 issues 2024-01-02 19:31:31 +01:00
Michael Vogt
530afa566f test: fix isort issues 2024-01-02 19:31:31 +01:00
Michael Vogt
57b5c7994e test: fix all pylint issues 2024-01-02 19:31:31 +01:00
Michael Vogt
7f3dbb2e2d tox.ini: run lint on test/ dir too 2024-01-02 19:31:31 +01:00
Michael Vogt
4a86688870 test: deal with missing sfdisk correctly 2024-01-02 18:40:31 +01:00
Michael Vogt
8bb399d269 runners: add debian runner (based on ubuntu1804) 2024-01-02 18:40:31 +01:00
schutzbot
5f97fcf032 schutzfile: Update snapshots to 20240101 2024-01-02 13:24:02 +01:00
Michael Vogt
1b297ad0ba test: add more output when ensure_mtime() assert fails 2023-12-22 19:46:48 +01:00
Achilleas Koutsou
54dfe70b41 github: run tests on push again
We've temporarily disabled the merge queue because our tests often
require retries to go all-green and this isn't possible to do on the
queue, meaning that it's close to impossible to get a PR merged.
Run tests on push so that they run in main when a PR is merged.

This reverts commit 63feab7d86.
2023-12-22 19:28:12 +01:00
Michael Vogt
75054f9db4 Schutzfile: update manifest-db to latest version with support for the new "partition" field 2023-12-22 10:18:29 -05:00
Michael Vogt
70c3539044 devcontainer: install parted for test_mount_with_partition 2023-12-22 10:18:29 -05:00
Michael Vogt
8b475bb3f2 test: extend test_mount.py to mount 2 partitions 2023-12-22 10:18:29 -05:00
Michael Vogt
e76e0e92d6 osbuild: take partition into account in calc_id and add test 2023-12-22 10:18:29 -05:00
Michael Vogt
9cf68394d9 test: add mount service test with partitions 2023-12-22 10:18:29 -05:00
Michael Vogt
651a0ff047 test: add test that checks that partition are added to mount stage 2023-12-22 10:18:29 -05:00
Michael Vogt
827b238764 format/v2: add new "partition" field to "describe_mount" 2023-12-22 10:18:29 -05:00
Dusty Mabe
ce8408a9c6 mounts: support mounting partitions
This allows us to map in a whole disk as a loopback device with parition
scanning rather than slicing up the disk and creating several loopback
devices. Something like this:

```
      - type: org.osbuild.copy
        inputs:
          tree:
            type: org.osbuild.tree
            origin: org.osbuild.pipeline
            references:
              - name:tree
        options:
          paths:
            - from: input://tree/
              to: mount://root/
        devices:
          efi:
            type: org.osbuild.loopback
            options:
              filename: disk.img
              start:
                mpp-format-int: '{image.layout[''EFI-SYSTEM''].start}'
              size:
                mpp-format-int: '{image.layout[''EFI-SYSTEM''].size}'
          boot:
            type: org.osbuild.loopback
            options:
              filename: disk.img
              start:
                mpp-format-int: '{image.layout[''boot''].start}'
              size:
                mpp-format-int: '{image.layout[''boot''].size}'
          root:
            type: org.osbuild.loopback
            options:
              filename: disk.img
              start:
                mpp-format-int: '{image.layout[''root''].start}'
              size:
                mpp-format-int: '{image.layout[''root''].size}'
        mounts:
          - name: root
            type: org.osbuild.xfs
            source: root
            target: /
          - name: boot
            type: org.osbuild.ext4
            source: boot
            target: /boot
          - name: efi
            type: org.osbuild.fat
            source: efi
            target: /boot/efi
```

now becomes a little more simple:

```
      - type: org.osbuild.copy
        inputs:
          tree:
            type: org.osbuild.tree
            origin: org.osbuild.pipeline
            references:
              - name:tree
        options:
          paths:
            - from: input://tree/
              to: mount://root/
        devices:
          disk:
            type: org.osbuild.loopback
            options:
              filename: disk.img
              partscan: true
        mounts:
          - name: root
            type: org.osbuild.xfs
            source: disk
            partition:
              mpp-format-int: '{image.layout[''root''].partnum}'
            target: /
          - name: boot
            type: org.osbuild.ext4
            source: disk
            partition:
              mpp-format-int: '{image.layout[''boot''].partnum}'
            target: /boot
          - name: efi
            type: org.osbuild.fat
            source: disk
            partition:
              mpp-format-int: '{image.layout[''EFI-SYSTEM''].partnum}'
            target: /boot/efi
```

Fixes https://github.com/osbuild/osbuild/issues/1495
2023-12-22 10:18:29 -05:00