Commit graph

872 commits

Author SHA1 Message Date
Joe
132cbef123 Add AppArmor support to debian-forge
- Add debian-forge-apparmor package with AppArmor stage support
- Create example AppArmor stage (org.osbuild.apparmor)
- Update workflow to build 9 packages total
- Add AppArmor manifest example for Debian Atomic
- Update todo with complete package structure
2025-08-29 17:45:28 -07:00
robojerk
eb18f1a514 Add dynamic apt-cacher-ng configuration system for collaborators
Some checks are pending
Checks / Spelling (push) Waiting to run
Checks / Python Linters (push) Waiting to run
Checks / Shell Linters (push) Waiting to run
Checks / 📦 Packit config lint (push) Waiting to run
Checks / 🔍 Check for valid snapshot urls (push) Waiting to run
Checks / 🔍 Check JSON files for formatting consistency (push) Waiting to run
Generate / Documentation (push) Waiting to run
Generate / Test Data (push) Waiting to run
Tests / Unittest (push) Waiting to run
Tests / Assembler test (legacy) (push) Waiting to run
Tests / Smoke run: unittest as normal user on default runner (push) Waiting to run
2025-08-26 15:52:43 -07:00
robojerk
502e1469ae Move composer scripts to root directory and add comprehensive Debian Atomic support
Some checks failed
Checks / Spelling (push) Has been cancelled
Checks / Python Linters (push) Has been cancelled
Checks / Shell Linters (push) Has been cancelled
Checks / 📦 Packit config lint (push) Has been cancelled
Checks / 🔍 Check for valid snapshot urls (push) Has been cancelled
Checks / 🔍 Check JSON files for formatting consistency (push) Has been cancelled
Generate / Documentation (push) Has been cancelled
Generate / Test Data (push) Has been cancelled
Tests / Unittest (push) Has been cancelled
Tests / Assembler test (legacy) (push) Has been cancelled
Tests / Smoke run: unittest as normal user on default runner (push) Has been cancelled
2025-08-23 08:02:45 -07:00
robojerk
3f639d537a Start OSBuild Composer integration testing: organize test files, create integration test script 2025-08-22 21:27:21 -07:00
robojerk
abea5a1380 Complete major testing milestones for Debian atomic system
Some checks are pending
Checks / Spelling (push) Waiting to run
Checks / Python Linters (push) Waiting to run
Checks / Shell Linters (push) Waiting to run
Checks / 📦 Packit config lint (push) Waiting to run
Checks / 🔍 Check for valid snapshot urls (push) Waiting to run
Checks / 🔍 Check JSON files for formatting consistency (push) Waiting to run
Generate / Documentation (push) Waiting to run
Generate / Test Data (push) Waiting to run
Tests / Unittest (push) Waiting to run
Tests / Assembler test (legacy) (push) Waiting to run
Tests / Smoke run: unittest as normal user on default runner (push) Waiting to run
- Add multi-stage workflow testing and validation
- Add error handling and recovery testing
- Add image generation testing (ISO, QCOW2, RAW)
- Validate complete build pipeline end-to-end
- Mark multiple TODO items as complete
- Maintain 1:1 OSBuild compatibility throughout
2025-08-22 21:00:14 -07:00
robojerk
b689f3e868 Document OSBuild architecture and enhance debootstrap stage
Some checks are pending
Checks / Spelling (push) Waiting to run
Checks / Python Linters (push) Waiting to run
Checks / Shell Linters (push) Waiting to run
Checks / 📦 Packit config lint (push) Waiting to run
Checks / 🔍 Check for valid snapshot urls (push) Waiting to run
Checks / 🔍 Check JSON files for formatting consistency (push) Waiting to run
Generate / Documentation (push) Waiting to run
Generate / Test Data (push) Waiting to run
Tests / Unittest (push) Waiting to run
Tests / Assembler test (legacy) (push) Waiting to run
Tests / Smoke run: unittest as normal user on default runner (push) Waiting to run
- Create comprehensive OSBuild architecture documentation
- Document buildroot isolation and object store patterns
- Enhance debootstrap stage to follow OSBuild patterns exactly
- Add proper metadata generation and filesystem mounting
- Maintain 1:1 compatibility with OSBuild design
- Mark foundational architecture tasks as complete
2025-08-22 20:49:53 -07:00
robojerk
48c31fa24f Implement enhanced build orchestration and artifact management
Some checks are pending
Checks / Spelling (push) Waiting to run
Checks / Python Linters (push) Waiting to run
Checks / Shell Linters (push) Waiting to run
Checks / 📦 Packit config lint (push) Waiting to run
Checks / 🔍 Check for valid snapshot urls (push) Waiting to run
Checks / 🔍 Check JSON files for formatting consistency (push) Waiting to run
Generate / Documentation (push) Waiting to run
Generate / Test Data (push) Waiting to run
Tests / Unittest (push) Waiting to run
Tests / Assembler test (legacy) (push) Waiting to run
Tests / Smoke run: unittest as normal user on default runner (push) Waiting to run
- Add build status tracking with state machine
- Implement build logging and monitoring system
- Add build progress tracking and cancellation support
- Create artifact management system with SQLite database
- Fix stage file extensions for proper Python imports
- Enhance resource allocation with actual resource tracking
- Add comprehensive testing for all components
2025-08-22 18:45:17 -07:00
robojerk
6a744c6c5b Add sbuild and debian source stages for Debian package building and source management
Some checks are pending
Checks / Spelling (push) Waiting to run
Checks / Python Linters (push) Waiting to run
Checks / Shell Linters (push) Waiting to run
Checks / 📦 Packit config lint (push) Waiting to run
Checks / 🔍 Check for valid snapshot urls (push) Waiting to run
Checks / 🔍 Check JSON files for formatting consistency (push) Waiting to run
Generate / Documentation (push) Waiting to run
Generate / Test Data (push) Waiting to run
Tests / Unittest (push) Waiting to run
Tests / Assembler test (legacy) (push) Waiting to run
Tests / Smoke run: unittest as normal user on default runner (push) Waiting to run
2025-08-22 18:22:25 -07:00
robojerk
6a17af5a62 Add Debian apt config and ostree deploy stages, update apt-cacher-ng to 192.168.1.101:3142 2025-08-22 18:18:48 -07:00
robojerk
85e0c04d21 Add apt-cacher-ng support to Debian stages for improved build performance 2025-08-22 18:14:00 -07:00
robojerk
31162116f8 Add Debian-specific OSBuild stages: apt, debootstrap, ostree-commit
Some checks are pending
Checks / Spelling (push) Waiting to run
Checks / Python Linters (push) Waiting to run
Checks / Shell Linters (push) Waiting to run
Checks / 📦 Packit config lint (push) Waiting to run
Checks / 🔍 Check for valid snapshot urls (push) Waiting to run
Checks / 🔍 Check JSON files for formatting consistency (push) Waiting to run
Generate / Documentation (push) Waiting to run
Generate / Test Data (push) Waiting to run
Tests / Unittest (push) Waiting to run
Tests / Assembler test (legacy) (push) Waiting to run
Tests / Smoke run: unittest as normal user on default runner (push) Waiting to run
2025-08-22 18:11:39 -07:00
Jonathan Lebon
b63306616a stages/coreos.live-artifacts: rework volume ID handling
In https://github.com/osbuild/osbuild/pull/2148, we changed the logic to
generate the volume ID from the data in `/usr/lib/os-release` to sever
the reliance on metadata in the embedded treefile that will no longer
exist.

This had no effect in FCOS, but had an effect in RHCOS, which
went from a volume ID of `rhcos-...` to `rhel-coreos-...`.
This was considered harmless at the time, but in fact ended
up affecting the OpenShift Assisted Image Service. See e.g.
https://github.com/openshift/assisted-image-service/pull/477 which
attempted to adapt that code. But in the end, it felt safer and less
work to just revert back to the previous volume ID. So here we are.

But we still don't want to go back to using the embedded treefile.
Instead, we now have access to the OS name to use as a label on the
container image. This label gets serialized into the aleph during the
creation of the metal image (via the `org.osbuild.ostree.aleph` stage)
which we have access here when mounting the metal image via loopback.

So pick it up from there and use that. But in case it's missing,
fallback to the previous logic rather than hard fail to make this easier
to ratchet in.
2025-08-22 01:55:45 +02:00
Michael Vogt
16f40b01d3 stages: extend test_bootc_install_to_fs test for new options
The org.osbuild.bootc.install_to_fs stage got a few new
options in PR#2152. Add them to our tests too.
2025-08-19 01:10:18 +02:00
jbtrystram
e930eeb519 stages/bootc.install: make boot and root mount spec customizable
Allow passing custom mount specs for boot and root. Optional fields.
2025-08-15 13:01:44 +02:00
jbtrystram
825045a914 stages/bootc.install-to-filesystem: parametrize stateroot value
This adds an extra option to make the stateroot name customizable.

Fixes https://github.com/osbuild/osbuild/issues/2151
2025-08-15 13:01:44 +02:00
Achilleas Koutsou
e31084e520 stages/tar: add --numeric-owner option
The numeric-owner option omits the inclusion of user and group names in
the archive metadata.  This is often desirable since name and group
mappings can change the ownership of files during extraction.

The test uses the tarfile module to check that the uname and gname
attributes in the tar archive itself are empty, which is the intended
effect of enabling numeric-owner [1].

RHEL-102854

[1] https://www.gnu.org/software/tar//manual/html_section/Attributes.html
2025-08-12 19:44:14 +02:00
Brian C. Lane
e3236a8b34 org.osbuild.grub2.iso.legacy: Add support for default menu selection
Currently the grub2 menu defaults to the first entry. This adds support
for setting the default to a later entry. The default in the official
boot.iso is entry 1 -- booting with the iso checksum check.

This includes a test for the new behavior.
2025-07-29 10:09:14 -07:00
Brian C. Lane
f49621ce44 org.osbuild.grub2.iso: Add support for default menu selection
Currently the grub2 menu defaults to the first entry. This adds support
for setting the default to a later entry. The default in the official
boot.iso is entry 1 -- booting with the iso checksum check.

This includes a test for the new behavior.
2025-07-29 10:09:14 -07:00
jbtrystram
f4698da5bb stages/ignition: parametrize the path to boot
Allow passing a mount to specify where to write the igntion.firstboot
file.
This keeps the default `tree:///` value to not break existing stages.
2025-07-28 21:14:02 +02:00
Brian C. Lane
de22369919 org.osbuild.isolinux: Add support for optional fips menu
On RHEL 9.7+ and on RHEL 10.1+ we need to be able to include a menu that
boots the installer environment with fips=1 on the cmdline. This adds an
optional menu entry controlled by the "fips" boolean.

This also includes a new test for the menus with and without fips
included.

Related: RHEL-104075
2025-07-25 22:52:19 +02:00
Brian C. Lane
d085681911 org.osbuild.grub2.iso.legacy: Add support for optional fips menu
On RHEL 9.7+ and on RHEL 10.1+ we need to be able to include a menu that
boots the installer environment with fips=1 on the cmdline. This adds an
optional menu entry controlled by the "fips" boolean.

This also includes a test for the new behavior.

Related: RHEL-104075
2025-07-25 22:52:19 +02:00
Brian C. Lane
9a563f8306 org.osbuild.grub2.iso: Add support for optional fips menu
On RHEL 9.7+ and on RHEL 10.1+ we need to be able to include a menu that
boots the installer environment with fips=1 on the cmdline. This adds an
optional menu entry controlled by the "fips" boolean.

This also includes a new test for the menus with and without fips
included.

Related: RHEL-104075
2025-07-25 22:52:19 +02:00
Bipin B Narayan
7593b65dc7 live-artifacts: read os name from /usr/lib/os-release
With importing ociarchive, the treefile.json doesn't contain osname.
2025-07-16 14:43:18 +02:00
Simon de Vlieger
fd6801e380 stages/vagrant: allow configuring synced folders
When guest additions are not installed in VirtualBox Vagrant boxes then
the default shared `/vagrant` directory must be set to `rsync`,
otherwise Vagrant fails to start as the shared directory cannot be
mounted with the `vboxfs` filesystem.

Let's expand the schema to allow for `synced-folders` (currently only
accepted under the `virtualbox` variant of the schema) to allow setting
the bare subset of relevant options to configure this from `images`.

Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
2025-07-11 17:13:49 +02:00
Achilleas Koutsou
a4061f831e stages/dnf4.versionlock: respect SOURCE_DATE_EPOCH
Respect the SOURCE_DATE_EPOCH environment variable for reproducible
builds.

https://reproducible-builds.org/docs/source-date-epoch/
2025-07-07 17:19:25 +02:00
Achilleas Koutsou
ece618d0d0 stages/test: add unit tests for the dnf4.versionlock stage 2025-07-07 17:19:25 +02:00
Achilleas Koutsou
06344b6827 stages: add a new dnf4.versionlock stage
New stage that writes a versionlock.list to lock packages in the format
used by the dnf4 versionlock plugin.
2025-07-07 17:19:25 +02:00
Leonardo Rossetti
4af53e4c6e add ssh_pwauth to org.osbuild.cloud-init
Signed-off-by: Leonardo Rossetti <lrossett@redhat.com>
2025-07-04 21:52:00 +02:00
Jelle van der Waa
f7ef1d6464 stages/vagrant: add virtualbox support
This adds support generating a virtualbox vagrant image. It differs from
libvirt by requiring an xml file and a vmdk image.

When the provider is set to libvirt it is required to pass a
`virtualbox` configuration section to this stage which must include the
mac address.

Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
2025-07-04 14:09:47 +02:00
Simon de Vlieger
ae70eb08d2 stages/ovf: test vbox_mac_address
Test that different calls result in different mac addresses (with the
same prefix), also test that when the correct options are passed that
the mac address is returned as is.

Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
2025-07-01 15:16:35 +02:00
Simon de Vlieger
54ac04e001 stage/ovf: mac address pattern
Adds a pattern for the mac address, without delimiters; as this is the
direct input for the virtualbox mac address property.

Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
2025-07-01 15:16:35 +02:00
Simon de Vlieger
02537089d8 stage/ovf: additional schema tests
Adds tests to more thoroughly look at the vmware and virtualbox schemas
by passing wrong types.

Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
2025-07-01 15:16:35 +02:00
Simon de Vlieger
19c38cb38f stages/ovf: allow setting mac address for vbox
The MAC address will need to be the same as the one being used by the
Vagrant stage *if* it's being used. This leaves it up to images to
generate the correct MAC address and set it in both places.

When the OVF stage is used separately it's still fine to use a random
MAC address.

Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
2025-07-01 15:16:35 +02:00
Simon de Vlieger
1cb0f26b09 stages/ovf: virtualbox's ostype as option
Let the user of the stage set the os type for the virtualbox bits.

Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
2025-07-01 15:16:35 +02:00
Simon de Vlieger
ce2bda719c stages/ovf: vmware's ostype as option
Let the user of the stage set the os type for the vmware bits.

Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
2025-07-01 15:16:35 +02:00
Simon de Vlieger
9559cd0528 stages/ovf: write virtualbox
Expand the written XML to include information as used by VirtualBox.
This should not affect any other use cases of the OVF document that is
generated and is purely extra information consumed by VirtualBox.

Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
2025-07-01 15:16:35 +02:00
Simon de Vlieger
d85ead3956 stages/ovf: reformat
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
2025-07-01 15:16:35 +02:00
Alexander Larsson
b7c0520392 stages: Add tests for org.osbuild.write-device
Signed-off-by: Alexander Larsson <alexl@redhat.com>
2025-06-25 11:12:25 +02:00
Alexander Larsson
30ca57de2c stages: Add org.osbuild.write-device stage
This stage writes a file to a device using dd. This is a
rewrite/backport of one of the stages in osbuild-auto.

The osbuild-auto stage is used in automotive-image-builder to write
the aboot image to the "boot_a" partition, to allow android boot
systems to boot. We will want similar functionallity in
bootc-image-builder, so it is important to upstream this.

Signed-off-by: Alexander Larsson <alexl@redhat.com>
2025-06-25 11:12:25 +02:00
Karolina Surma
7390f91592 Mark tests needing TOML-writing library with a custom marker
This way they can be conveniently skipped with `pytest -m "not
tomlwrite"` in environments where such libraries aren't available.
2025-06-23 19:51:29 +02:00
Karolina Surma
516301c02b Skip tests creating system units in RPM environment
Attempting to run them during the RPM build ends with an error:
'Failed to create directory '/run/systemd/': Permission denied'
2025-06-23 19:51:29 +02:00
Simon de Vlieger
d566c68f94 stages/wsl-distribution: stage to write wsl config
This is a new stage to be able to write a (subset) of relevant keys into
the `wsl-distribution` configuration file.

The file, and these options, are necessary for the enablement of fancy
logos and experiences in WSL2.

Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
2025-06-19 10:22:57 +02:00
Simon de Vlieger
bc99513c0a stages/test: kickstart different messages
On Python 3.6 we're getting a *different* error message from other
versions in CI. Let's accept both versions.

Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
2025-06-16 09:31:49 -07:00
Achilleas Koutsou
0968ee8d81 stages/grub2: add compat_version for fixed behaviour
For the terminal_input, terminal_output, and serial variables in the
GRUB_CFG_TEMPLATE, the keys were not set when those variables were not
defined in the options.  This caused the template variables to show up
in the final file itself, which could break the config.  For example,
the following line was being generated for one of our images:

    serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1${terminal_input}${terminal_output}

Setting the keys to an empty string when they're not defined solves the
issue.

Since this is a stage behaviour change, the new behaviour is toggled by
an option that maintains backwards compatibility when not set.
This introduces a new convention of adding a `compat_version` number to
the stage options to control the behaviour.  The value defaults to 1,
which is the old (broken) behaviour.  Manifest generators should always
set it to 2 to get the correct behaviour.
2025-06-13 14:14:57 +02:00
Simon de Vlieger
42281231b0 stages/containers: don't validate keys
In Python 3.14 configparser started validating keys. We use hacky bits
to write duplicate keys which means we have delimiters in our keys which
is now no longer allowed.

Let's explicitly turn off key validation for now as suggested by
mhroncok in [1].

[1]: https://github.com/osbuild/osbuild/issues/2109#issuecomment-2964585958

Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
2025-06-12 15:31:34 +02:00
Simon de Vlieger
44cb7934db stages/systemd: don't validate keys
In Python 3.14 configparser started validating keys. We use hacky bits
to write duplicate keys which means we have delimiters in our keys which
is now no longer allowed.

Let's explicitly turn off key validation for now as suggested by
mhroncok in [1].

[1]: https://github.com/osbuild/osbuild/issues/2109#issuecomment-2964585958

Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
2025-06-12 15:31:34 +02:00
Brian C. Lane
a828ef95d6 stages/erofs: Add exclude_paths support
Add the ability to exclude files and directories from the erofs
image. This uses the mkfs.erofs --exclude-regex feature, so simple
matches or regexes can be used.
2025-06-11 20:22:23 +02:00
Brian C. Lane
86c89a2421 stages/squashfs: Add exclude_paths support
Add the ability to exclude files and directories from the squashfs
image. This uses the mksquashfs -regex -e FILES... feature, so simple
matches or regexes can be used.

This also adds a new test for squashfs, based on the existing test for
erofs.
2025-06-11 20:22:23 +02:00
Simon de Vlieger
a464815ea8 stages/tar: allow chosen compression
The `org.osbuild.tar` stage only supports auto compression which is
based on the filename. For several (newer) artifacts such as Vagrant
(`.box`) and WSL (`.wsl`) we want to explicitly give the compression
algorithm.

I've chosen a (few) commonly used compression algorithms. If others are
needed they are one-line followups away.

Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
2025-06-03 18:18:23 +02:00
Achilleas Koutsou
0ac83fd421 stages/kickstart: post-installation scripts
Add a new %post option to the kickstart stage that supports adding
multiple post blocks to a kickstart file, with all the options supported
by the directive.
2025-05-14 09:06:40 +02:00