Commit graph

347 commits

Author SHA1 Message Date
Paweł Poławski
91c6352c80 tests: Update stage tests readme
Readme was pointing to the dead ling with manifest v2 documentation.
I have redirected it to the documentation presented in repo.
2024-05-13 11:19:26 -07:00
Achilleas Koutsou
1f0f18d281 test/stages/systemd: add -.mount unit to test
Create a unit using an inline file called -.mount with the following
content:

  [Unit]
  Before=local-fs.target
  After=blockdev@dev-disk-by\x2duuid-af34257d\x2d3e14\x2d4a51\x2db91d\x2dc430a956dcba.target

  [Mount]
  What=/dev/disk/by-uuid/af34257d-3e14-4a51-b91d-c430a956dcba
  Where=/
  Type=ext4
  Options=rw,noatime

  [Install]
  RequiredBy=local-fs.target

and enable it in the systemd stage to test that we can enable units with
a - prefix.
2024-04-23 19:59:44 +02:00
Achilleas Koutsou
420aacb82e test: regenerate test repository metadata
In 599d3a8730 the repository files weren't
all updated and the metadata was inconsistent.
2024-04-16 07:38:18 +02:00
Michael Vogt
599d3a8730 tools: rename "nothing" to "pkg-with-no-deps" to make it a bit clearer
Tweak the package name a bit to more it very explicit what it's about.
2024-04-11 12:45:25 +02:00
Achilleas Koutsou
bc08eed1ef test: add repository metadata for osbuild-depsolve-dnf
Add two test rpm metadata directories that can be served as RPM repos.
One was copied from osbuild/images and contains the repository metadata
for CentOS Stream 9 BaseOS.
The second was created by building a simple spec file into an RPM and
creating the metadata using createrepo.
2024-04-10 16:22:13 -07:00
Michael Vogt
7279c44c53 test: add StageTests.test_zip() to run zip inside a stage too
Similar to the tar test this adds a zip test that runs inside
the stages.
2024-04-04 13:54:34 +02:00
Luke Yang
ac8a2a4f30 stages: add org.osbuild.zip
In CoreOS Assembler, some hyperv artifact we `zip` for compression. This
new stage is modeled after the `org.osbuild.tar` stage with necessary
modifications.
2024-04-04 13:54:34 +02:00
Achilleas Koutsou
7b004a297e test/systemd.unit: Environment option object
Add an org.osbuild.systemd.unit stage using the new format for the
Environment option with two instances to the test manifest.

The contents of the new dropin file at
tree/usr/lib/systemd/system/boltd.service.d/30-boltd-debug.conf are:

[Service]
Environment="G_MESSAGES_DEBUG=all"
Environment="G_MESSAGES_TRACE=none"
2024-04-04 10:36:04 +02:00
Achilleas Koutsou
80c84020dd test/systemd.unit.create: add new options to test
Add the new options to the b.json test and update the diff.

The new file has the following contents:

[Unit]
Description=Create directory
DefaultDependencies=False
ConditionPathExists=|!/etc/myfile
ConditionPathIsDirectory=|!/etc/mydir

[Service]
Type=oneshot
RemainAfterExit=True
ExecStart=mkdir -p /etc/mydir
ExecStart=touch /etc/myfile
Environment="DEBUG=1"
EnvironmentFile=/etc/example.env

[Install]
WantedBy=local-fs.target
RequiredBy=multi-user.target
2024-04-04 10:36:04 +02:00
Achilleas Koutsou
c72f1bc54e test/systemd.unit.create: add empty file to a.json
Add an empty file to the location where the service file will be
created in the b.json version of the test.  This way, we will get a
content hash of the created file which is a slightly better test than
just knowing that it was created.

Note that, in the diff, the "before" checksum is the empty file hash:

    echo -n '' | sha256sum
    e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  -
2024-04-04 10:36:04 +02:00
Achilleas Koutsou
d1fc2cd6b6 test/systemd.unit.create: fix b.mpp.yaml
In 2d2cdd8097 the file was replaced by
the generated json and it went unnoticed in the PR.  Reverted and
updated the options to match the generated json file.
2024-04-04 10:36:04 +02:00
Achilleas Koutsou
1366976dd8 test: reformat all json files
Reformat all test json files for consistency.

Formatted with `js --indent 2 .`
2024-03-26 01:20:37 +01:00
Sayan Paul
548d2a51ea stage/unit-create:new systemd unit config params
Added new systemd unit config : ConditionPathIsDirectory

Signed-off-by: Sayan Paul <paul.sayan@gmail.com>
2024-03-19 10:19:26 +01:00
Sayan Paul
2d2cdd8097 stage/systemd-unit-create:paremeterize file location
User can now customize the systemd unit load path.
User can select between etc or usr , defaults to 'usr'.
Also user can customize the scope of the service between global
or system, defaults to system.

Signed-off-by: Sayan Paul <paul.sayan@gmail.com>
2024-03-07 14:04:00 +02:00
Luke Yang
cfaabe618f stages/org.osbuild.gzip: add compression level option
Allow compression level to be specified instead of defaulting to 1. This is needed for CoreOS Assembler.
2024-03-04 11:18:13 +01:00
Renata Ravanelli
0b7dcf3cf4 Add coreos.platforms stage
- Process all necessary operations related to CoreOS
platforms is crucial and specific to CoreOS. This step
is essential for CoreOS exclusively.

- Our approach to handling 'platforms.json' may change as we
advance with the OSBuild work. However, we don't have a clear
vision about how it will be in the future yet, particularly as
we also manage similar components within the osbuild composer
to configure cloud parameters. We probably will know better
when we start working with the cloud artifacts.

As a summary, let's add it know to unblock us, and if we find a
better approach in the future, we can always go back and remove it.

Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
2024-02-26 16:02:28 +01:00
Achilleas Koutsou
78ea8405a4 test: add test manifest for containers-storage
A manifest (mpp and json) that uses the new source and input with the
skopeo stage.

This depends on the image we store at
./test/data/stages/skopeo/hello.img

The plan is to test this by pulling the hello.img into the host root
storage, build the manifest, delete the image from storage, and check
the tree.
2024-02-21 17:55:37 +01:00
Sayan Paul
e858dc72c3 stage/systemd-unit:stage to create systemd unit file
Add systemd unit files in osbuild stage

This stage creates systemd unit file in `/usr/lib/systemd/system/`.
The stage accepts filename which must end with `.service`.Section
`Unit` , `Service` , `Install` accepts various parameters as per
the systemd documentaion.`systemd-analyze verify` is be performed
after the .service file is created to check for potential errors.

Signed-off-by: Sayan Paul <paul.sayan@gmail.com>
2024-02-16 11:55:11 +01:00
Dusty Mabe
f334ca92c6 manifests: convert a few more ostree deployments
These were missed in the initial conversion.
2024-02-12 11:25:11 -05:00
Michael Vogt
c0f670f035 test: add manifests for bootc.install-to-filesystem
This commit adds example manifests for a bootc.install-to-filesystem
system. It does not do more with them because running a full test
requires a working podman which is difficult to use inside our
GH runners that are already running inside docker.
2024-02-09 12:03:09 +01:00
Luke Yang
619a64f0bd stages/org.osbuild.qemu: make qcow2 compression optional
Modify the stages/org.osbuild.qemu stage such that compression is
optional. This resolves the image size differences between an image
built with coreos assember vs osbuild, as discussed in:
https://github.com/coreos/fedora-coreos-tracker/issues/1653#issuecomment-1928342241
2024-02-07 20:31:10 -05:00
Dusty Mabe
e1cbf92673 ostree: add convenience function for using default OSTree deployment
This adds a `default: true` option for all cases where OSTree
information is specified in schemas and allows for the information
to be picked up from the filesystem.

This is a safe operation because when building disk images there is
no known case where having two deployments makes sense. In the case
there ever were a case then the osname, ref, and serial options still
exist and can be used.

Co-authored-by: Luke Yang <luyang@redhat.com>
Co-authored-by: Michael Vogt <michael.vogt@gmail.com>
2024-02-07 18:50:38 -05:00
Achilleas Koutsou
6e12f08a29 test: add bootupd.gen-metadata stage test
Since the stage depends on quite a specific tree state (ostree prepped
tree with boot files), we can't really unit test it any simpler than
generating a tree with and without running the stage and diffing the
tree.
2024-02-07 12:10:01 +01:00
Dusty Mabe
011cf6851b stages(ostree.config): support setting sysroot.bootprefix
See https://github.com/ostreedev/ostree/pull/2705 and also
https://github.com/osbuild/osbuild/issues/1566.
2024-02-07 11:28:24 +01:00
Dusty Mabe
151b6e48ae stages(sgdisk): support label option
We only support `gpt` here so it would seem this option doesn't
make much sense to add, but it will make it so that the mpp-define-images
from osbuild-mpp can be passed in to `org.osbuild.sgdisk` just as it
can be passed in today to `org.osbuild.sfdisk`.
2024-02-06 17:44:31 +01:00
Nikita Dubrovskii
fc185dae8c support user-defined partition numbers for GPT disks
Partitions by default are indexed starting at 1, but in
some cases, such as CoreOS for IBM Z, it may be usefull
to set the 'partnum' for GPT disks explicitly, without
creating dummy partitions.

Now user can define an image:

```
    mpp-define-images:
      - id: image
        size: 10737418240
        table:
          uuid: 00000000-0000-4000-a000-000000000001
          label: gpt
          partitions:
            - name: boot
              type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
	      partnum: 3
              size: 786432
            - name: root
              type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
	      partnum: 4
              size: 4194304
```

So target disk would look like:

```
    Disklabel type: gpt
    Disk identifier: 00000000-0000-4000-A000-000000000001
    Device        Start     End Sectors  Size Type
    /dev/loop0p3   2048  788479  786432  384M Linux filesystem
    /dev/loop0p4 788480 4982783 4194304    2G Linux filesystem
```

This patch updates the osbuild-mpp tool and the sgdisk and sfdisk
stages to support this.

Co-authored-by: Dusty Mabe <dusty@dustymabe.com>
2024-02-06 17:44:31 +01:00
Luke Yang
477a21043e Create stages/org.osbuild.chattr stage
Add or remove the immutable bit to the specified mount directory.

The need we have for this right now is for the CoreOS builds where
the immutable bit being set on an OSTree deployment root doesn't
survive the `cp -a --reflink=auto` in the org.osbuild.copy stage when
being copied from the directory tree into the mounted XFS filesystem
we created on the disk image. Thus we have to workaround this loss
of attribute by applying the attribute directly on the mounted
filesystem from the disk.
2024-01-31 16:46:01 -05:00
Dusty Mabe
408b101799 manifests: support container_imgref var in FCOS manifest
This makes it easier to see/change the value globally.
2024-01-19 17:15:23 -05:00
Dusty Mabe
7d73c0e741 manifests: move osname to a var in fcos manifest
This makes it easier to see/change the value globally.
2024-01-15 11:53:21 +01:00
Dusty Mabe
d9fe4221b0 manifests: move boot/root fs uuid/label to var in fcos manifest
This makes it easier to see the important unique values in our
manifest.
2024-01-15 11:53:21 +01:00
Dusty Mabe
ac8a57c5a3 manifests: set sysroot.readonly config to true in fcos manifest
This is set in Fedora CoreOS today. Let's make it match here.
2024-01-15 11:53:21 +01:00
Dusty Mabe
7cf98cfbf1 manifests: remove partition UUIDs from fcos container manifest
We don't set the UUIDs today in COSA/create_disk.sh so let's not
statically set them here either. This means they'll get set to
something random.
2024-01-15 11:53:21 +01:00
Luke Yang
d089b429a7 stages/org.osbuild.ostree.config: support bls-append-except-default
Support setting the sysroot.bls-append-except-default value in the
OSTree config. This is used by CoreOS to support configuration used
for GRUB password support https://github.com/coreos/fedora-coreos-tracker/issues/1333
2024-01-11 12:55:12 -05:00
Dusty Mabe
fdd5d25c1e tests/fedora-coreos-container: switch to testing FCOS
The stable stream currently doesn't have a new enough bootupd to pass
the tests for the bootupd stage. Let's update to `:testing` for now
and we'll switch back to `:stable` later.
2024-01-10 01:15:18 +01:00
Renata Ravanelli
cdcb4598bb create org.osbuild.bootupd stage
Add the bootupd stage to install GRUB on both BIOS and UEFI systems,
ensuring that your bootloader stays up-to-date.

Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
2024-01-10 01:15:18 +01:00
Dusty Mabe
ce8408a9c6 mounts: support mounting partitions
This allows us to map in a whole disk as a loopback device with parition
scanning rather than slicing up the disk and creating several loopback
devices. Something like this:

```
      - type: org.osbuild.copy
        inputs:
          tree:
            type: org.osbuild.tree
            origin: org.osbuild.pipeline
            references:
              - name:tree
        options:
          paths:
            - from: input://tree/
              to: mount://root/
        devices:
          efi:
            type: org.osbuild.loopback
            options:
              filename: disk.img
              start:
                mpp-format-int: '{image.layout[''EFI-SYSTEM''].start}'
              size:
                mpp-format-int: '{image.layout[''EFI-SYSTEM''].size}'
          boot:
            type: org.osbuild.loopback
            options:
              filename: disk.img
              start:
                mpp-format-int: '{image.layout[''boot''].start}'
              size:
                mpp-format-int: '{image.layout[''boot''].size}'
          root:
            type: org.osbuild.loopback
            options:
              filename: disk.img
              start:
                mpp-format-int: '{image.layout[''root''].start}'
              size:
                mpp-format-int: '{image.layout[''root''].size}'
        mounts:
          - name: root
            type: org.osbuild.xfs
            source: root
            target: /
          - name: boot
            type: org.osbuild.ext4
            source: boot
            target: /boot
          - name: efi
            type: org.osbuild.fat
            source: efi
            target: /boot/efi
```

now becomes a little more simple:

```
      - type: org.osbuild.copy
        inputs:
          tree:
            type: org.osbuild.tree
            origin: org.osbuild.pipeline
            references:
              - name:tree
        options:
          paths:
            - from: input://tree/
              to: mount://root/
        devices:
          disk:
            type: org.osbuild.loopback
            options:
              filename: disk.img
              partscan: true
        mounts:
          - name: root
            type: org.osbuild.xfs
            source: disk
            partition:
              mpp-format-int: '{image.layout[''root''].partnum}'
            target: /
          - name: boot
            type: org.osbuild.ext4
            source: disk
            partition:
              mpp-format-int: '{image.layout[''boot''].partnum}'
            target: /boot
          - name: efi
            type: org.osbuild.fat
            source: disk
            partition:
              mpp-format-int: '{image.layout[''EFI-SYSTEM''].partnum}'
            target: /boot/efi
```

Fixes https://github.com/osbuild/osbuild/issues/1495
2023-12-22 10:18:29 -05:00
Luke Yang
5fc3b565a2 create org.osbuild.ostree.aleph stage
Similar to the aleph file created for builds of FCOS based on ostree
commit inputs, this adds an aleph file that contains information about
the initial deployment of data when the disk image was built

A new stage is preferred here as both the org.osbuild.ostree.deploy
and org.osbuild.ostree.deploy.container stages need an aleph file and
use of the aleph file may depend on the project/product. For example,
right now CoreOS is the only project that uses an aleph file, but others
may want it in the future.
2023-12-19 17:58:39 +01:00
Dusty Mabe
96ee2e4bc8 test: update stage diffs for F39 bump
Diffs for stage tests have changed after the sources were updated.
Update them to match expected behaviour. This was mostly done with
some form of:

```
foo=update-crypto-policies;
sudo tools/gen-stage-test-diff --libdir . --store /var/osbuild/store/ test/data/stages/$foo > test/data/stages/$foo/diff.json
```

For the dracut one I had to figure out what new kernel was used
and the new modules and update the vanilla.json file to get the
test to pass.

For the rpm one I had to also update the metadata.json with something
like:

```
sudo python3 -m osbuild --libdir .  --store /var/osbuild/store/ --export tree \
    --output-directory /var/osbuild/out/ test/data/stages/rpm/b.json --json   \
        | jq .metadata >test/data/stages/rpm/metadata.json
```
2023-12-19 17:58:39 +01:00
Dusty Mabe
e077a4ceba manifests/fedora-vars: update to fedora 39
For https://github.com/osbuild/osbuild/pull/1475 we need to pick up
https://bodhi.fedoraproject.org/updates/FEDORA-2023-a3737d3bee so let's
bump to the latest Fedora 39 snapshot.
2023-12-19 17:58:39 +01:00
Alexander Larsson
69028fc016 org.osbuild.systemd: Support masking generators
This adds a new key masked_generators, similar to masked_services,
which masks systemd generators from running at boot, by creating
symlinks to /dev/null in /etc/systemd/systemd-generators, as
described in:
 https://www.freedesktop.org/software/systemd/man/latest/systemd.generator.html#Description

This will be useful for the automotive project, as it allows disabling
of unsupported things like sysv or rc.local legacy support, while
improving boot performance.
2023-12-15 10:01:00 -08:00
Gianluca Zuccarelli
6d9c6e0bde stages:oscap.autotailor: add key/value overrides
This commit extends the current support for OpenSCAP
tailoring by accepting an array of key/value overrides.
Users will be able to specify override values for specific
rules that will update the value when remediating the
image.
2023-12-04 08:49:47 +01:00
Ondřej Budai
7882db2543 test/data/manifests: add erofs-utils to Fedora buildroot
The next commit will add a stage test that requires erofs-utils. Let's add it
into the buildroot in a separate commit, so the history is more readable.
2023-12-01 10:36:06 +01:00
Michael Vogt
627199543f test: add erofs stage test
Add a simple stage test for the erofs stage. It uses dump.erofs
instead of mounting the file because the kernel in the GH runners
do not support mounting erofs just yet.
2023-12-01 10:36:06 +01:00
Dusty Mabe
e191dc1d0d tools/osbuild-mpp: set part ID from name if missing
For our Fedora CoreOS disk images we set the partition labels (name)
for the partitions. This is also supported using the primitives here
in OSBuild, but it wasn't obvious that I needed to set the name in
the mpp-define-images definition. Let's set the name there, but let's
also allow osbuild-mpp to set the `id`, which is what is used later
to access that partition from the `name` too if `id` isn't set.

This means we allow something like:

   - name: BIOS-BOOT
     type: 21686148-6449-6E6F-744E-656564454649
     bootable: true
     uuid: FAC7F1FB-3E8D-4137-A512-961DE09A5549
     size: 100

rather than requiring something like:

   - id: BIOS-BOOT
     name: BIOS-BOOT
     type: 21686148-6449-6E6F-744E-656564454649
     bootable: true
     uuid: FAC7F1FB-3E8D-4137-A512-961DE09A5549
     size: 100
2023-11-29 10:37:08 +01:00
Dusty Mabe
5ad3f968e1 manifests: add example metal4k disk image for Fedora CoreOS
The created disk image would be written to 4k native disks.
2023-11-29 10:37:08 +01:00
Dusty Mabe
2e1f6e2553 tools/osbuild-mpp: support defining multiple image layouts
Right now you can only define a single image, lets add mpp-define-images
and accept a list.
2023-11-29 10:37:08 +01:00
Dusty Mabe
b824e1e57a manifests: fedora-coreos-container: human friendly partitioning
Let's define the partitioning for the example here in a more human
friendly way using GiB and MiB terms. This yields no change in the
generated json.
2023-11-29 10:37:08 +01:00
Gianluca Zuccarelli
9cd8fc979b test: fix broken oscap remediation tests
The oscap remediation tests broke with 810a48d. This commit adds the
fixes to get the GitHub actions to go green again.
2023-11-24 14:28:34 +00:00
Simon de Vlieger
e7db40831f test: stage tests -> stage integration tests
Quick rename to have our wording be in-line with the new differences
between stage unit tests and stage integration tests; also being applied
to the guides.
2023-11-24 13:58:04 +01:00
Miguel Martín
f3d740aaf8 stages: add org.osbuild.update-crypto-policies stage
This stage calls `update-crypto-policies` to set the
policy applicable for the various cryptographic back-ends,
such as SSL/TLS libraries.

Signed-off-by: Miguel Martín <mmartinv@redhat.com>
2023-11-15 17:15:10 +01:00