The "main" branch is failing right now in tests. The reason is
that we do not have a merge queue and when
https://github.com/osbuild/osbuild/pull/1715
was merged we had no test for `osbuild-depsolve-dnf` yet.
We have one now (THANK YOU achilleas-k) and it shows an issue :)
This commit fixes the issue.
Depsolver test that starts a temporary file server and queries it using
osbuild-depsolve-dnf.
Generates all combinations of repositories configured through the
depsolve-dnf request or the repositories directory and runs the test
cases. The results should be the same regardless of combination.
Test repos are defined with a fake gpg key on the request or repo config
and check if it is read correctly and attached to the repo configs in
the response. The name of the repo is appended to each repo's gpg key
so we can make sure that repo option values don't get swapped.
Some of the repository properties in the request were named differently
than the equivalent properties in the dnf repository configuration.
This can introduce bugs and confusion.
One such issue already existed with osbuild/images using 'gpgcheck' in
the request, osbuild-depsolve-dnf5 checking for 'check_gpg', and the dnf
repository configuration property being 'gpgcheck'. This didn't cause
any bad behaviour because osbuild/images reused the original (internal)
configuration to set the property in stages and depsolving isn't
affected by the value of this property.
Change the request properties to match the dnf repository configuration
to avoid confusion: gpgcheck, repo_gpgcheck, and sslverify. Users of
osbuild-depsolve-dnf5 should use property names that match dnf. Use
the same names in the response.
To maintain the same behaviour for SSL verification, a missing sslverify
default to True. The previous property had the opposite meaning,
ignore_ssl, and defaulted to False.
Add the full gpg keys to the repository configs in the response.
On each repository object from dnf, the gpg keys are URLs, either
file:// or http(s)://. We need to resolve these and return them with
in the response.
When the URL is a file:// path, and it comes from a .repo config file,
we assume that the path is relative to the root_dir, so we prepend it to
the path in the file. This is so that repo configs in OS root trees can
be used unmodified. However, when a key is defined in the request, we
should assume that the path is valid, either because it was defined by
the caller as a URL, or because it was defined in-line in the request
and osbuild-depsolve-dnf5 wrote it to the persistdir itself.
A new exception is defined to identify errors during this process.
Support loading repositories from a root tree instead of supplying them
with the request. The repositories should be in the standard yum repo
format. Both repository sources can be defined simultaneously, but at
least one is required.
The root_dir is expected to contain files necessary for depsolving in
the standard paths.
These files are:
- Repository (.repo) configurations in <root_dir>/etc/yum.repos.d/
- GPG key files in <root_dir>/etc/pki/rpm-gpg/
- This will be used to resolve gpg key paths specified in the .repo
files that are relative to the root_dir.
- (Optional) Custom dnf config variables in <root_dir>/etc/dnf/vars or
<root_dir>/usr/share/dnf5/vars.d.
- This is used by CentOS Stream to set the value of $stream.
Custom repository configurations in arbitrary (non-root) paths will have
to follow this directory structure.
A new variable is added to the request, `releasever`, which is mandatory
when using `root_dir`. This variable is used in repository URLs and GPG
key paths. In the default case, dnf reads this variable by inspecting
the rpm database. We will override it in the Solver the same way we
override the arch and basearch for variable substitution. In the
future, we will make this variable mandatory in all cases, which will
make the variable available for repo configs defined in the request as
well.
The root_dir is used in three ways:
- Set the base.conf.installroot
- Set the base.conf.varsdir to <root_dir>/usr/share/dnf5/vars.d and
<root_dir>/etc/dnf/vars to read resolve custom variables when loading
repositories.
- Call create_repos_from_dir() with <root_dir>/etc/yum.repos.d.
base.setup() should be called before loading repositories otherwise
substitutions might not work.
See https://github.com/rpm-software-management/dnf5/issues/1374#issuecomment-2038995031
Some of the repository properties in the request were named differently
than the equivalent properties in the dnf repository configuration.
This can introduce bugs and confusion.
One such issue already existed with osbuild/images using 'gpgcheck' in
the request, osbuild-depsolve-dnf checking for 'check_gpg', and the dnf
repository configuration property being 'gpgcheck'. This didn't cause
any bad behaviour because osbuild/images reused the original (internal)
configuration to set the property in stages and depsolving isn't
affected by the value of this property.
Change the request properties to match the dnf repository configuration
to avoid confusion: gpgcheck, repo_gpgcheck, and sslverify. Users of
osbuild-depsolve-dnf (osbuild/images) should use property names that
match dnf. Use the same names in the response.
To maintain the same behaviour for SSL verification, a missing sslverify
default to True. The previous property had the opposite meaning,
ignore_ssl, and defaulted to False.
Add the full gpg keys to the repository configs in the response.
On each repository object from dnf, the gpg keys are URLs, either
file:// or http(s)://. We need to resolve these and return them with
in the response.
When the URL is a file:// path, and it comes from a .repo config file,
we assume that the path is relative to the root_dir, so we prepend it to
the path in the file. This is so that repo configs in OS root trees can
be used unmodified. However, when a key is defined in the request, we
should assume that the path is valid, either because it was defined by
the caller as a URL, or because it was defined in-line in the request
and osbuild-depsolve-dnf wrote it to the persistdir itself.
A new exception is defined to identify errors during this process.
When generating package sources and rpm stage metadata for a manifest
from a list of packages, we need to associate repository configuration
options to each package [1]. Previously, a caller had all the
repository configurations because they were part of the request, so
packages could be associated with all the repository options by the
repository ID. Now, osbuild-depsolve-dnf will use repositories loaded
from a directory that the caller shouldn't have to read, so returning
all repository configurations in the response makes it possible to
get all package metadata from the response.
This changes the whole structure of the response to a depsolve request.
Previously, we returned an array of packages. Now we return an object
with two keys:
- packages: the array of packages as before
- repositories: an object mapping repository IDs to repository
configurations.
Each package contains the repository ID it comes from (as before), under
`repo_id`. This can be used to get repository configurations and
determine gpg keys and SSL certs for each package.
The new structure avoids duplicating values across all the (sometimes
hundreds) of packages.
[1] 92497c7b1f/pkg/dnfjson/dnfjson.go (L499-L507)
Support loading repositories from a root tree instead of supplying them
with the request. The repositories should be in the standard yum repo
format. Both repository sources can be defined simultaneously, but at
least one is required.
The root_dir is expected to contain files necessary for depsolving in
the standard paths.
These files are:
- Repository (.repo) configurations in <root_dir>/etc/yum.repos.d/
- GPG key files in <root_dir>/etc/pki/rpm-gpg/
- This will be used to resolve gpg key paths specified in the .repo
files that are relative to the root_dir.
- (Optional) Custom dnf config variables in <root_dir>/etc/dnf/vars or
<root_dir>/etc/yum/vars.
- This is used by CentOS Stream to set the value of $stream.
Custom repository configurations in arbitrary (non-root) paths will have
to follow this directory structure.
A new variable is added to the request, `releasever`, which is mandatory
when using `root_dir`. This variable is used in repository URLs and GPG
key paths. In the default case, dnf reads this variable by inspecting
the rpm database. We will override it in the Solver the same way we
override the arch and basearch for variable substitution. In the
future, we will make this variable mandatory in all cases, which will
make the variable available for repo configs defined in the request as
well.
The root_dir is used in two ways:
- Set the base.conf.reposdir to <root_dir>/etc/yum.repos.d.
- Call update_from_etc() with root_dir to read custom variables in
<root_dir>/etc/yum/vars and <root_dir>/etc/dnf/vars.
Add an example render to test/demo how the json-seq based progress
works. It needs the python `tqdm` package for the actual rendering.
See the output with:
```
$ sudo OSBUILD_TEST_STORE=/var/tmp/osbuild-test-store \
python3 -m osbuild --libdir=. --monitor=JSONSeqMonitor --export image \
--output-dir=/tmp/output-dir ./test/data/manifests/fedora-boot.json | ./tools/osbuild-json-seq-progress-example-renderer
```
The code in `check-snapshots` will print "No snapshots cache found
at ..." regardless of the error that happens when trying to open
the file. This can be misleading if e.g. the issue is permissions
to open the file or the file is corrupted. So make the exception
more targeted and only catch FileNotFound error and let python
how the full error for the other cases. Obviously this can be
done in many ways so I'm happy to tweak and e.g. keep catching
all exception but print the value etc.
Partitions by default are indexed starting at 1, but in
some cases, such as CoreOS for IBM Z, it may be usefull
to set the 'partnum' for GPT disks explicitly, without
creating dummy partitions.
Now user can define an image:
```
mpp-define-images:
- id: image
size: 10737418240
table:
uuid: 00000000-0000-4000-a000-000000000001
label: gpt
partitions:
- name: boot
type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
partnum: 3
size: 786432
- name: root
type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
partnum: 4
size: 4194304
```
So target disk would look like:
```
Disklabel type: gpt
Disk identifier: 00000000-0000-4000-A000-000000000001
Device Start End Sectors Size Type
/dev/loop0p3 2048 788479 786432 384M Linux filesystem
/dev/loop0p4 788480 4982783 4194304 2G Linux filesystem
```
This patch updates the osbuild-mpp tool and the sgdisk and sfdisk
stages to support this.
Co-authored-by: Dusty Mabe <dusty@dustymabe.com>
I mistakenly thought the .strip('.0') would strip off just '.0` but
in reality it strips off all instances of `.` or `0` on the end so
'4075520.0'.strip('.0') => '407552', which is clearly wrong.
This is a fixup for 5207e92.
So we can use something like mpp-format-string inside of mpp-embed. An
example would be below where we want to substitute the ociarchive var
in using `mpp-format-string: 'file://{ociarchive}'`.
```
version: '2'
mpp-vars:
ociarchive: /path/to/fedora-coreos-39.20240104.dev.2-ostree.x86_64.ociarchive
pipelines:
- name: oci-archive
stages:
- type: org.osbuild.copy
inputs:
inlinefile:
type: org.osbuild.files
origin: org.osbuild.source
mpp-embed:
id: fcos.ociarchive
url:
mpp-format-string: 'file://{ociarchive}'
options:
paths:
- from:
mpp-format-string: input://inlinefile/{embedded['fcos.ociarchive']}
to: tree:///fcos.ociarchive
```
Partitions are usually indexed starting a 1 so an index starting
at 0 is confusing (i.e. you never say mount the filesystem on
partition 0). Let's add a partnum field that can be used in
definitions instead.
`mpp-define-images` can create an image file, using `losetup` to deal
with non-standard sector sizes requires root. Not all users run
`osbuild-mpp` as root.
While I am not a fan of "suddenly sudo" based on the input manifest this
does alleviate builds breaking for manifests with default sector sizes
when non-root.
For our Fedora CoreOS disk images we set the partition labels (name)
for the partitions. This is also supported using the primitives here
in OSBuild, but it wasn't obvious that I needed to set the name in
the mpp-define-images definition. Let's set the name there, but let's
also allow osbuild-mpp to set the `id`, which is what is used later
to access that partition from the `name` too if `id` isn't set.
This means we allow something like:
- name: BIOS-BOOT
type: 21686148-6449-6E6F-744E-656564454649
bootable: true
uuid: FAC7F1FB-3E8D-4137-A512-961DE09A5549
size: 100
rather than requiring something like:
- id: BIOS-BOOT
name: BIOS-BOOT
type: 21686148-6449-6E6F-744E-656564454649
bootable: true
uuid: FAC7F1FB-3E8D-4137-A512-961DE09A5549
size: 100
Now you can specify a sector_size in `mpp-define-images` to support
creating a 4k native disk image (sector_size=4096).
This does use a loopback device, which means osbuild-mpp also needs
to run as root, when previously that wasn't necessary.
If you do math in mpp-format-int it could end up getting converted
to a float. Of course if you end up with a decimal value that isn't
`.0` that's a problem for an int, but if it is `.0` let's handle it
gracefully.
For example, math like this could end up with a value with `.0`:
mpp-format-int: "{bios_boot_size_mb * 1024 * 1024 / sector_size_bytes}"
I'm trying to debug some failures, and having no feedback as to
what file we're parsing or what code is evaluated when something
fails makes it hard to debug manifests.
This adds some nice error messages that will help.
Instead of running osbuild as a binary use `python3 -m osbuild`
(just like in `test/test.py:compile()`) so that it will use
osbuild fromgit and can be run from a checkout without the need
for an installed osbuild.
Instead of using the F34-based manifests, let's switch to F38. I tried my
best to import the vars into the new manifest, but I don't think that's
currently supported.
The `./tools` dir was not part of the LINTABLES in the `tox.ini`
which meant that pep8/pylint etc checks were not run on the tools
there.
This commit adds it and fixes the issues that `make lint` found.
