Commit graph

2682 commits

Author SHA1 Message Date
Michael Vogt
e0bbb3397d test: update test_mount.py for lsblk --json differences
Older version of `lsblk --json` will not have the plural `mountpoints`
but only a singular `mountpoint`. But newer version lost the
singular `mountpoint` in the json. Adjust the test accordingly.
2024-01-16 13:51:30 +01:00
Michael Vogt
d96f94fede test: fix new mount tests under rhel8
The new `create_image_with_partitions()` helper fails under rhel8
currently. The reason is that `mkfs.ext4 -E offset=` will warn
in older versions about a partition table and require user input.

This got fixed `e2fsprogs` 1.46.3 in Jul 2021 but RHEL8 still
has 1.45.

[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989612
2024-01-16 13:51:30 +01:00
Dusty Mabe
d50705249a osbuild-mpp: fix float correction
I mistakenly thought the .strip('.0') would strip off just '.0` but
in reality it strips off all instances of `.` or `0` on the end so
'4075520.0'.strip('.0') => '407552', which is clearly wrong.

This is a fixup for 5207e92.
2024-01-15 11:53:21 +01:00
Dusty Mabe
7d73c0e741 manifests: move osname to a var in fcos manifest
This makes it easier to see/change the value globally.
2024-01-15 11:53:21 +01:00
Dusty Mabe
d9fe4221b0 manifests: move boot/root fs uuid/label to var in fcos manifest
This makes it easier to see the important unique values in our
manifest.
2024-01-15 11:53:21 +01:00
Dusty Mabe
ac8a57c5a3 manifests: set sysroot.readonly config to true in fcos manifest
This is set in Fedora CoreOS today. Let's make it match here.
2024-01-15 11:53:21 +01:00
Dusty Mabe
7cf98cfbf1 manifests: remove partition UUIDs from fcos container manifest
We don't set the UUIDs today in COSA/create_disk.sh so let's not
statically set them here either. This means they'll get set to
something random.
2024-01-15 11:53:21 +01:00
Luke Yang
d089b429a7 stages/org.osbuild.ostree.config: support bls-append-except-default
Support setting the sysroot.bls-append-except-default value in the
OSTree config. This is used by CoreOS to support configuration used
for GRUB password support https://github.com/coreos/fedora-coreos-tracker/issues/1333
2024-01-11 12:55:12 -05:00
Michael Vogt
73ec3122f2 stages,test: update tests for new selinux.setfiles() calling 2024-01-11 10:23:25 +01:00
Michael Vogt
467a23ffa7 stages(selinux): use osbuild.util.selinux.setfiles()
Instead of calling `setfiles` manually just reuse the existing
helper. The tests validate that there is no functional change.
2024-01-11 10:23:25 +01:00
Michael Vogt
18159eceec test: add selinux stage test 2024-01-11 10:23:25 +01:00
Dusty Mabe
83a14886d3 add --break for requesting a debug shell
Similar to rd.break for dracut this allows a user to specify:

- --break or --break=*
    - to get a shell before each stage is run
- --break=stage.name
    - to get a shell each time the stage with that name is run
    - example: --break=org.osbuild.copy
- --break=stage.id
    - to get a shell each time the stage with that ID is run
    - get the ID for the stages for your manifest by running
      osbuild on the manifest with --inspect
    - example: --break=dc6e3a66fef3ebe7c815eb24d348215b9e5e2ed0cd808c15ebbe85fc73181a86

and get a bash shell where they can inspect the environment to debug
and develop OSBuild stages.
2024-01-10 08:39:27 -08:00
Michael Vogt
962b7f4d4b test: add test for export option 2024-01-10 11:24:01 +01:00
Dusty Mabe
8cce659ec5 stages/bootupd: support installing to a partition of a device
For ppc64le we need to pass in a partition (i.e. /dev/loop0p1) rather
than the root device (/dev/loop0) to the --device argument of bootupctl.
Let's add a partition field and find the device node based on the user
specified partition.

On ppc64le this would look something like:

```
      - type: org.osbuild.bootupd
        options:
          bios:
            device: disk
            partition:
              mpp-format-int: '{image.layout[''POWERPC-PREP-BOOT''].partnum}'
          static-configs: true
          deployment:
            osname: fedora-coreos
            ref: ostree/1/1/0
        devices:
          disk:
            type: org.osbuild.loopback
            options:
              filename: disk.img
              partscan: true
        mounts:
          - name: root
            type: org.osbuild.xfs
            source: disk
            partition:
              mpp-format-int: '{image.layout[''root''].partnum}'
            target: /
          - name: boot
            type: org.osbuild.ext4
            source: disk
            partition:
              mpp-format-int: '{image.layout[''boot''].partnum}'
            target: /boot
```
2024-01-10 01:15:18 +01:00
Michael Vogt
492dc6021c stages(bootupd): refactor test to separately test behaviors
It now tests  bind_mounts and bootupd behavior separately.
2024-01-10 01:15:18 +01:00
Michael Vogt
dacf5733ea stages(bootupd): add tests for existing behavior
Also refactor bind mounts into a helper.
2024-01-10 01:15:18 +01:00
Michael Vogt
eb657b08b3 stages(bootupd): add schema test 2024-01-10 01:15:18 +01:00
Dusty Mabe
fdd5d25c1e tests/fedora-coreos-container: switch to testing FCOS
The stable stream currently doesn't have a new enough bootupd to pass
the tests for the bootupd stage. Let's update to `:testing` for now
and we'll switch back to `:stable` later.
2024-01-10 01:15:18 +01:00
Renata Ravanelli
cdcb4598bb create org.osbuild.bootupd stage
Add the bootupd stage to install GRUB on both BIOS and UEFI systems,
ensuring that your bootloader stays up-to-date.

Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
2024-01-10 01:15:18 +01:00
Michael Vogt
c03e6be52a stages(grub2): allow pulling efi binaries from alternative efi roots
The EFI binaries are currently pulled from a hardcoded path in the
buildroot. When moving to containers as buildroots this will no
longer work as they have an alternative layout. This is an easy
"fix" - make the location of the `EFI/` directory configurable.

This allows us set `efi_src_dir` to `/usr/lib/bootupd/updates/EFI/`
and keep our existing `bootc-image-builder` workflow.

Note that this may actually not be the desired solution and instead
we want the new `bootupd`: https://github.com/osbuild/osbuild/pull/1519
2024-01-09 19:43:01 +01:00
Michael Vogt
cb02d0a4bc stages(container-deploy): ensure /var/tmp is available
This commit ensures that `/var/tmp` is available. Skopeo expects
this dir but the bwrap environment starts with a very minimal
`/var` so `/var/tmp` may not be available.
2024-01-09 12:54:11 +01:00
Dusty Mabe
c62e555202 tools/osbuild-mpp: run _process_format() for mpp-embed dict
So we can use something like mpp-format-string inside of mpp-embed. An
example would be below where we want to substitute the ociarchive var
in using `mpp-format-string: 'file://{ociarchive}'`.

```
version: '2'
mpp-vars:
  ociarchive: /path/to/fedora-coreos-39.20240104.dev.2-ostree.x86_64.ociarchive
pipelines:
  - name: oci-archive
    stages:
      - type: org.osbuild.copy
        inputs:
          inlinefile:
            type: org.osbuild.files
            origin: org.osbuild.source
            mpp-embed:
              id: fcos.ociarchive
              url:
                mpp-format-string: 'file://{ociarchive}'
        options:
          paths:
            - from:
                mpp-format-string: input://inlinefile/{embedded['fcos.ociarchive']}
              to: tree:///fcos.ociarchive
```
2024-01-05 18:57:43 +01:00
Gianluca Zuccarelli
a2b2a02add sources/skopeo: add storage location
For local images copied from an image store other than the default, we
need to be able to specify the `storage-location`. This commit enables
this functionality.

Jira: https://issues.redhat.com/browse/HMS-3235
2024-01-05 16:42:51 +01:00
Michael Vogt
ac5653d9f1 stages: rework container-deploy stage to not use tmp storage
This commit reworks the `org.osbuild.container-deploy` stage to
not use a tmp storage when mounting the container image. This
is needed because of [0] but it should generally be fine because
inside the stages the real /var is a tmpfs (which is why we
triggered the bug in the first place).

[0] https://github.com/containers/storage/issues/1779
2024-01-03 11:25:48 -08:00
Michael Vogt
9a8b0ddb61 test: add test for org.osbuild.container-deploy 2024-01-03 11:25:48 -08:00
Ondřej Budai
fdc5bf3b98 stages: org.osbuild.container-deploy 2024-01-03 11:25:48 -08:00
Michael Vogt
ad8fd2f532 testutil: extrace new make_fake_tree() helper
Extract a new helper `make_fake_tree()` that generalizes the existing
helper `make_fake_input_tree()`. The later will always create the
content under `{basedir}/tree` which is convinient for input tree
based tests but too specialized when using it in different contexts.

The existing `make_fake_input_tree()` is preserved unchanged and
becomes just a tiny wrapper.
2024-01-03 11:25:48 -08:00
schutzbot
25d198da3c Post release version bump
[skip ci]
2024-01-03 08:14:11 +00:00
Michael Vogt
8c95bd9dd7 test: fix autopep8 issues 2024-01-02 19:31:31 +01:00
Michael Vogt
530afa566f test: fix isort issues 2024-01-02 19:31:31 +01:00
Michael Vogt
57b5c7994e test: fix all pylint issues 2024-01-02 19:31:31 +01:00
Michael Vogt
7f3dbb2e2d tox.ini: run lint on test/ dir too 2024-01-02 19:31:31 +01:00
Michael Vogt
4a86688870 test: deal with missing sfdisk correctly 2024-01-02 18:40:31 +01:00
Michael Vogt
8bb399d269 runners: add debian runner (based on ubuntu1804) 2024-01-02 18:40:31 +01:00
schutzbot
5f97fcf032 schutzfile: Update snapshots to 20240101 2024-01-02 13:24:02 +01:00
Michael Vogt
1b297ad0ba test: add more output when ensure_mtime() assert fails 2023-12-22 19:46:48 +01:00
Achilleas Koutsou
54dfe70b41 github: run tests on push again
We've temporarily disabled the merge queue because our tests often
require retries to go all-green and this isn't possible to do on the
queue, meaning that it's close to impossible to get a PR merged.
Run tests on push so that they run in main when a PR is merged.

This reverts commit 63feab7d86.
2023-12-22 19:28:12 +01:00
Michael Vogt
75054f9db4 Schutzfile: update manifest-db to latest version with support for the new "partition" field 2023-12-22 10:18:29 -05:00
Michael Vogt
70c3539044 devcontainer: install parted for test_mount_with_partition 2023-12-22 10:18:29 -05:00
Michael Vogt
8b475bb3f2 test: extend test_mount.py to mount 2 partitions 2023-12-22 10:18:29 -05:00
Michael Vogt
e76e0e92d6 osbuild: take partition into account in calc_id and add test 2023-12-22 10:18:29 -05:00
Michael Vogt
9cf68394d9 test: add mount service test with partitions 2023-12-22 10:18:29 -05:00
Michael Vogt
651a0ff047 test: add test that checks that partition are added to mount stage 2023-12-22 10:18:29 -05:00
Michael Vogt
827b238764 format/v2: add new "partition" field to "describe_mount" 2023-12-22 10:18:29 -05:00
Dusty Mabe
ce8408a9c6 mounts: support mounting partitions
This allows us to map in a whole disk as a loopback device with parition
scanning rather than slicing up the disk and creating several loopback
devices. Something like this:

```
      - type: org.osbuild.copy
        inputs:
          tree:
            type: org.osbuild.tree
            origin: org.osbuild.pipeline
            references:
              - name:tree
        options:
          paths:
            - from: input://tree/
              to: mount://root/
        devices:
          efi:
            type: org.osbuild.loopback
            options:
              filename: disk.img
              start:
                mpp-format-int: '{image.layout[''EFI-SYSTEM''].start}'
              size:
                mpp-format-int: '{image.layout[''EFI-SYSTEM''].size}'
          boot:
            type: org.osbuild.loopback
            options:
              filename: disk.img
              start:
                mpp-format-int: '{image.layout[''boot''].start}'
              size:
                mpp-format-int: '{image.layout[''boot''].size}'
          root:
            type: org.osbuild.loopback
            options:
              filename: disk.img
              start:
                mpp-format-int: '{image.layout[''root''].start}'
              size:
                mpp-format-int: '{image.layout[''root''].size}'
        mounts:
          - name: root
            type: org.osbuild.xfs
            source: root
            target: /
          - name: boot
            type: org.osbuild.ext4
            source: boot
            target: /boot
          - name: efi
            type: org.osbuild.fat
            source: efi
            target: /boot/efi
```

now becomes a little more simple:

```
      - type: org.osbuild.copy
        inputs:
          tree:
            type: org.osbuild.tree
            origin: org.osbuild.pipeline
            references:
              - name:tree
        options:
          paths:
            - from: input://tree/
              to: mount://root/
        devices:
          disk:
            type: org.osbuild.loopback
            options:
              filename: disk.img
              partscan: true
        mounts:
          - name: root
            type: org.osbuild.xfs
            source: disk
            partition:
              mpp-format-int: '{image.layout[''root''].partnum}'
            target: /
          - name: boot
            type: org.osbuild.ext4
            source: disk
            partition:
              mpp-format-int: '{image.layout[''boot''].partnum}'
            target: /boot
          - name: efi
            type: org.osbuild.fat
            source: disk
            partition:
              mpp-format-int: '{image.layout[''EFI-SYSTEM''].partnum}'
            target: /boot/efi
```

Fixes https://github.com/osbuild/osbuild/issues/1495
2023-12-22 10:18:29 -05:00
Dusty Mabe
f6d0a4a9ac devices/loopback: enable partition scanning as an option
We can now add an entire device and then get the partitions added
to our environment for use, rather than to have to map each partition
in to a separate loopback device.

This is a prep patch for https://github.com/osbuild/osbuild/issues/1495
2023-12-22 10:18:29 -05:00
Dusty Mabe
c3b52ba240 osbuild-mpp: add partnum field to the partitions
Partitions are usually indexed starting a 1 so an index starting
at 0 is confusing (i.e. you never say mount the filesystem on
partition 0). Let's add a partnum field that can be used in
definitions instead.
2023-12-22 10:18:29 -05:00
Achilleas Koutsou
63feab7d86 github: run tests on merge queue
Don't run checks on push to main since the tip of the merge queue is the
same.
2023-12-21 18:47:23 +01:00
Michael Vogt
2d72dc125d test: rename TestFileSystemMountService->FakeFileSystemMountService
The rename avoids warnings from pytest. It confuses this for
a unittest class apparently. And it's really a `Fake` service
anyway.
2023-12-21 16:07:35 +01:00
Michael Vogt
a6fba858ab osbuild: honor OSBUILD_EXPORT_FORCE_NO_PRESERVE_OWNER env
To workaround the issue that inside macOS containers the ownership
cannot be preserved we introduce a new environment that can be
used to forcefully relax the use of `cp -a`.

I did it via an environment instead of a commandline option mostly
because `github.com/osbuild/images/osbuild:RunOBuild()` already has
`extraEnv` option.
2023-12-20 09:28:39 +01:00