Commit graph

2444 commits

Author SHA1 Message Date
Ondřej Budai
b94048dfd4 mounts/org.osbuild.btrfs: add support for compression
Also, add it to the sample manifest.
2023-08-25 12:35:56 +02:00
Ondřej Budai
3bbded4607 org.osbuild.mkfs.btrfs: add support for -m argument
Osbuild doesn't support creating btrfs over multiple devices, thus it should
be fine to have only single and dup in the schema (other options are for raid)
2023-08-25 12:35:56 +02:00
Ondřej Budai
dae3f884bc samples/fedora-boot-btrfs: convert to use subvolumes
We now have a full support for subvolumes in osbuild, let's use
it in the sample.

Co-authored-by: Christian Kellner <christian@kellner.me>
2023-08-25 12:35:56 +02:00
Ondřej Budai
0952ae9933 mounts/org.osbuild.btrfs: add support for mounting subvolumes
Co-authored-by: Christian Kellner <christian@kellner.me>
2023-08-25 12:35:56 +02:00
Ondřej Budai
724183b35c stages: add org.osbuild.btrfs.subvol
A trivial stage to create subvolume on a btrfs partition.
2023-08-25 12:35:56 +02:00
Ondřej Budai
f145a877f6 test/data: include bootable fedora image with btrfs
So we have a quick reference how to work with btrfs-based images.

Co-authored-by: Christian Kellner <christian@kellner.me>
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2023-08-25 12:35:56 +02:00
schutzbot
36516f9fea schutzfile: Update snapshots to 20230824 2023-08-24 18:14:55 +02:00
schutzbot
271f5eff31 Post release version bump
[skip ci]
2023-08-23 15:27:18 +00:00
Sanne Raymaekers
433515cff8 schutzbot: unregister test hosts 2023-08-21 18:46:55 +02:00
Sayan Paul
61da6124ed stages/systemd:support for user service dropin
systemd unit stage will now be able to add user unit dropin,
this is done by adding an additional field:unit-type
which is set to system by default. It also adds ability
to update the unit config with ConditionPathExists.
Relevant testing updated for the new workflow.

Signed-off-by: Sayan Paul <paul.sayan@gmail.com>
2023-08-16 13:23:16 +02:00
Sayan Paul
ed926ecc71 stages/systemd:support timer units
unit pattern updated to incoporate timer units

Signed-off-by: Sayan Paul <paul.sayan@gmail.com>
2023-08-16 13:23:16 +02:00
Brian C. Lane
463d799dfd Remove unused repos as reported by ./tools/check-runners
This removes these unused repos:

fedora-35
fedora-36
rhel-8.8
rhel-9.2

Note that that script also reports that rhel-8.7 and rhel-9.1 do not have
Schutzfile entries. That is because they are used in deploy.sh which
does not use Schutzfile to lookup the urls.
2023-08-16 10:40:05 +02:00
Brian C. Lane
68f4c5bd5f tests: Add a check for valid snapshot urls
This pulls the list of snapshots from the rpmrepo API, greps the
codebase for all uses of rpmrepo.osbuild.org that look like a snapshot
name, and then checks to make sure they are still valid.

This is the same script as osbuild-composer, except that it also skips
checking ./test/data/stages/ which has rpm urls that look like snapshot
urls.
2023-08-16 10:40:05 +02:00
schutzbot
694e956104 Post release version bump
[skip ci]
2023-08-16 08:18:35 +00:00
schutzbot
e8bd5e9c8e schutzfile: Update snapshots to 20230815 2023-08-15 09:50:45 +02:00
Ondřej Budai
330d9d8188 ci: simplify the Python linters
We now have a Makefile target for linting, let's use it in the CI.

Running the linters is pretty fast, so we can just do it in one container.
2023-08-15 03:11:40 +02:00
Ondřej Budai
f075b743e8 test: remove the src module
It's already empty, let's just remove it and the test-src Makefile target
with it.

If you wish to lint the sources, there's the new `lint` target.
2023-08-15 03:11:40 +02:00
Ondřej Budai
dc0dbb0f09 Makefile: Add a lint target
The new target just runs all the linters in parallel.
2023-08-15 03:11:40 +02:00
Simon de Vlieger
161fe789af dnf4.mark: mark packages in DNF state database
This adjustment allows the definition of the mark with the RPMs and runs
DNF after installing the RPMs to put the proper markings in the DNF
state database. See #455.

This ensures that packages don't get removed during `autoremove` leading
to broken systems.
2023-08-15 00:37:06 +02:00
Jakub Rusz
4ddfe5ed0f Add jrusz ssh key to list of team keys 2023-08-14 17:39:27 +02:00
Jakub Rusz
9017408366 Update fedora-39 runners and repositories 2023-08-14 17:39:27 +02:00
Brian C. Lane
a4ee2f47b7 terraform: Use new rawhide and new RHEL 9.3 images
This uses Fedora's 20230809.n.2 cloud image with dnf5 5.1.1-1,
and new RHEL 9.3 Image Builder.
2023-08-10 13:04:14 +02:00
Brian C. Lane
44c28c8c16 autopep8: Update with changes to make autopep8 -a -a -a happy 2023-08-10 13:04:14 +02:00
Brian C. Lane
a7541ad8b4 tox: Pin pycodestyle to 2.10.0
They changed their API in 2.11.0 and broke autopep8
See:
https://github.com/hhatto/autopep8/issues/689
and
https://github.com/hhatto/autopep8/pull/696
2023-08-10 13:04:14 +02:00
Brian C. Lane
2047a8efbc autopep8: Increase aggressive level 2023-08-10 13:04:14 +02:00
Ondřej Budai
c47850b8ae manifests/fedora-ostree-native-container: bump max_layers to 4
rpm-ostree crashes when max_layers is set to <4. Work around that
by bumping the max_layers argument.

See https://github.com/coreos/rpm-ostree/issues/4530
2023-08-08 12:41:17 +02:00
Ondřej Budai
6e254c95af stages/ostree.encapsulate: use new entrypoint on rpm-ostree >=2023.2
rpm-ostree 2023.2 dropped the `rpm-ostree container-encapsulate` entrypoint.
Instead, we have to use `rpm-ostree compose container-encapsulate`.

Adjust the code that it selects the correct entrypoint based on the rpm-ostree
version.

Since the new stage now requires python-yaml, add it to the buildroot and
regenerate all manifests.
2023-08-08 12:41:17 +02:00
Ondřej Budai
09fa9e9def test/data/stages: Document the changes to the tests 2023-08-08 12:41:17 +02:00
Ondřej Budai
fbe6e3ed97 manifests: remove no longer needed f34-build-v2
It's just no longer required by anything, yay!
2023-08-08 12:41:17 +02:00
Ondřej Budai
59edb5ea6c test/data/stages: update to the new structure
This commit changes these manifests to use the new fedora-vars.ipp and
fedora-build-v2.ipp infrastructure to remove all hardcoded Fedora versions.

Note that this is currently limited only to stage tests that already use v2
manifests. v1 manifests will come later.

Notes:
The buildroot needs additional packages: zstd and openscap-utils. Thus,
all manifests had to be regenerated.

GPG keys were added where missing.

The oscap.remediation stage now creates a bunch of files with timestamps in
their names under /var/lib/authselect/backups. Thus, the newly introduced
`added_directories` directive is used to handle them.

I reviewed all changes in `diff.json` files, and they all seem sane given
that we jumped 4 releases forward.
2023-08-08 12:41:17 +02:00
Ondřej Budai
5045bde6b7 tools/tree-diff: make its output stable
I don't see a reason why not to make the default output from tree-diff always
stable.

The stage tests already do that before running the comparison.
2023-08-08 12:41:17 +02:00
Ondřej Budai
04c545c39c tools: introduce a tool for generating diffs for stage tests
There hasn't ever been an official tool for this and everyone was just using
the output from the test to generate the diff. This commit introduces a proper
tool for generating these files.
2023-08-08 12:41:17 +02:00
Ondřej Budai
16294997d3 test_stages: allow specifying a whole added directory
This commit adds support for a new field in tree diffs fed to the stage tests.
This is useful when you care that a directory is added, but you don't care
about its content.

Since this is useful only for the expected tree diff, it's supported only
there. The actual tree diff doesn't support the new field.
2023-08-08 12:41:17 +02:00
Ondřej Budai
f0c5dfb3a3 manifests/fedora-ostree-tarball: update to the new structure
This commit changes this manifest to use the new fedora-vars.ipp and
fedora-build-v2.ipp infrastructure to remove all hardcoded Fedora versions.
2023-08-08 12:41:17 +02:00
Ondřej Budai
111fc97035 manifests/fedora-ostree-container: update to the new structure
This commit changes this manifest to use the new fedora-vars.ipp and
fedora-build-v2.ipp infrastructure to remove all hardcoded Fedora versions.

We are also able to remove the container manifests from the Makefile,
because they are no longer depending on the old f34-build-v2.json.
2023-08-08 12:41:17 +02:00
Ondřej Budai
9a9834a309 manifests/fedora-ostree-bootiso*: update to the new structure
This commit changes this manifest to use the new fedora-vars.ipp and
fedora-build-v2.ipp infrastructure to remove all hardcoded Fedora versions.

An auto-generated ostree ref and isolabel was added to fedora-vars.

smc-meera-fonts was retired, so this commit changes it to rit-meera-new-fonts.
See
a8fe0d2620

icfg and reiserfs-utils are gone too, and they don't seem to be neccessary for
the image, so they were removed as well. Please correct me, if I am wrong.
2023-08-08 12:41:17 +02:00
Ondřej Budai
aeec48619b manifests/fedora-container: update to the new structure
This commit changes this manifest to use the new fedora-vars.ipp and
fedora-build-v2.ipp infrastructure to remove all hardcoded Fedora versions.
2023-08-08 12:41:17 +02:00
Ondřej Budai
82b7053983 manifests/fedora-vars: update to fedora 38
What? I can just edit variables in one files and all(*) manifests get updated
content? That's impressive.

(*) We will be able to do all once we migrate all to the new format. For now,
    the usual disclaimer applies:

    This change is applied only to manifests based on fedora-build-v2 for now.
2023-08-08 12:41:17 +02:00
Ondřej Budai
baca25f7cb manifests: fix module_platform_id
module_platform_id has a format of `platform:f34`. I have no idea why just
`f34` is used. Let's fix it.

This change is applied only to manifests based on fedora-build-v2 for now.
2023-08-08 12:41:17 +02:00
Ondřej Budai
ed16da70eb manifests: factor repositories and gpg keys out
They are the same everywhere, we can just factor them out and save some bytes.

This change is applied only to manifests based on fedora-build-v2 for now.
2023-08-08 12:41:17 +02:00
Ondřej Budai
9286515821 manifests: move vars to a separate file
This change is applied only to manifests based on fedora-build-v2 for now.
2023-08-08 12:41:17 +02:00
Ondřej Budai
64cc289bf7 manifests: remove vars from fedora-build-v2
Defining variables in the build pipeline is potentially dangerous, because
it can lead to the buildroot using a different package set than the other
pipelines.

Thus, this commit removes the variables from the build manifest. Since now on,
the variables must be defined before the build manifest is included.

Since the build manifest is no longer buildable on its own, let's remove it
from mpp.yaml to ipp.yaml. This is a convention set by the automotive team:
ipp manifests are not meant to be directly buildable, they are only useful
for embedding.
2023-08-08 12:41:17 +02:00
Ondřej Budai
f777eb5ac2 convert all mpp.yaml files to proper YAML
This is the second step of the conversion. All mpp.yaml json files were loaded
and dumped again with pyyaml. It was tweaked to keep the order of keys and to
save multi-line strings in the "flow" style. This was done, so the GPG keys
are kept on one line. Otherwise, they take up too much visual space.

There is no functional change in this commit.
2023-08-08 12:41:17 +02:00
Ondřej Budai
d827be2fb2 rename all .mpp.json files to .mpp.yaml
osbuild-mpp supports yaml files. The advantage of yaml is that it's less
verbose and allows comments. This way, we can start documenting all the
test manifests in this repository.

This is the first step: This commit just renames the files, so we keep the
history intact. Remember: YAML is a super-set(*) of JSON, so this is fine.

There is no functional change in this commit.

(*) There are some caveats, but we don't need to care in this case.
2023-08-08 12:41:17 +02:00
Ondřej Budai
2d23f9c17c samples: remove empty el8-ostree-bootiso.json
No idea why this is here.
2023-08-08 12:41:17 +02:00
Colin Walters
6f81c59894 ostree.encapsulate: It's rpm-ostree, not ostree
We have a lot of copypasta of `def ostree()` but in this
case it's actually rpm-ostree, and we should be clear about that.

(Also in the future this code will need to change
 to use the non-deprecated entrypoint, see
 https://github.com/coreos/rpm-ostree/pull/4527 )
2023-08-05 23:42:50 +02:00
Simon de Vlieger
427e82e0c0 osbuild: --checkpoint can now use globs
When developing or rebuilding manifests a lot it is common to want to
checkpoint everything to the store. It seems we all have small shell
scripts hanging around for this.

Let `--checkpoint` take a shell-like glob such as `--checkpoint="*"` to
checkpoint everything.

Note that there's a behavioral change here; previously `osbuild
--checkpoint=a` would error if that specific checkpoint wasn't found.
Now `osbuild` will only error if nothing was selected by the passed
globs.
2023-08-04 19:59:11 +02:00
schutzbot
14d31633a4 schutzfile: Update snapshots to 20230801 2023-08-02 13:59:51 +02:00
schutzbot
b8836948c5 Post release version bump
[skip ci]
2023-08-02 08:15:32 +00:00
Simon de Vlieger
10e7046301 ci: drop some Python versions
This limits the list to the Python versions found in current Fedoras,
RHELs, and CentOS Streams.
2023-08-01 15:01:13 +02:00