This commit tweaks build() to be mypy clean without the need to
call assert. This drops the map() and instead we use the existing
dict-like access of the manifest to get the pipeline. In practise
this should not happen but lets be prepared.
Note that a small tweak for the error is needed to make it clear
what is happening.
Thanks to Simon for raising this.
This commit makes the returns of the of {Build,Download}Result.as_dict()
explicit. Ideally this would just be a dataclass and dataclass.asdict()
but because we need to support python3.6 this is not possible today.
This commit adds error reporting from source download errors
to the monitor. It reuses the `BuildResult` for symmetry but
we probably want to refactor this a bit to make source handling
a bit more similar to stages.
In order to avoid having to rely on the output of `osbuild --json`
when using `--progress=JSONSeqMonitor` the monitor needs to include
the `osbuild.pipeline.BuildResult` for each individual stage.
This commit adds those to the montior.
This commit relaxes the "mirror" schema regex from only allow
hashes to the repo ID regex as described in
https://dnf.readthedocs.io/en/latest/conf_ref.html#description
This is needed because in e.g. bootc-image-builder we use
"SetRootDir()" to get the repository information from an existing
bootc image and there the repo_id (that is then used as the mirror_id)
is written in a human friendly way instead of hex.
Thanks to Tomáš Hozza for the suggestions and background info, see
https://github.com/osbuild/osbuild/pull/1974#issuecomment-2579325272
This commit adds support for the `org.osbuild.mtls` sources and
adds a test.
Note that it reuses the `OSBUILD_SOURCES_CURL_` - this may or
may not be desired, it is certainly simpler this way and librepo
is using libcurl as well internally but it feels a bit messy so
maybe we want to rename the env?
This commit fixes the pylint errors and as a drive-by tweaks some naming
in `fetch_all()`. Maybe they became a bit verbose along the way,
idk.
Thanks also to Tomáš Hozza for the dict.setdefault() suggestion.
This commit adds unit a basic test for librepo and also updates
the code to the latest requirements to provide "fetch_all".
It also fixes a bug in the handling of org.osbuild.rhsm secrets.
This commit just ran:
```
find -iname "*.json" -print -exec sh -c 'jq --indent 2 . {} | sponge {}' \;
```
to ensure that the new test json data from librepo is in the format
the linter expects.
Most modules do not support both schema versions. This is masked by
module type code in get_schema() in most cases, but really should not be
tested. This skips running check_moduleinfo if the module doesn't
support the version. eg. org.osbuild.librepo only supports v2.
This helps prevent testing against invalid manifest data. It runs on the
source's manifest data, using the highest schema version parsed for the
source.
These tests test downloading packages using mirrorlist:
* Using a single mirrorlist repo
* Using two mirrorlist repos
* A bad checksum (or file) from the download which is expected to fail
* Two mirrorlist paths with the first one returning a 404, expected to
succeed with the 2nd mirrorlist path.
NOTE: The metalink xml file points to a repomd.xml file which does not
exist. In practice librepo doesn't download this, it only uses the url
to construct the mirror urls, so it is not present.
These tests test downloading packages using metalink:
* Using a single metalink repo
* Using two metalink repos
* A bad checksum (or file) from the download which is expected to fail
* Two metalink paths with the first one returning a 404, expected to
succeed with the 2nd metalink path.
Using a metalink or mirrorlist along with the package paths and
checksums allows them to be reliably downloaded even when mirrors are
not all in sync. It will retry with a new mirror until it succeeds, or
has tried all of the mirrors.
This commit adds support to run the tests on Debian/Ubuntu systems.
Here the binary is called `grub-mkimage` instead of `grub2-mkimage`,
in addition the "grub-pc-bin" package must be installed there.
Don't include the "location" offset, and use just a path for the prefix
section to set the path of the grub2 modules on the iso. eg.
{
"filename": "eltorito.img",
"platform": "i386-pc",
"core": {
"type": "mkimage",
"partlabel": "gpt",
"filesystem": "iso9660"
},
"prefix": {
"path": "/boot/grub2/"
}
}
The mount ID must be unique. So far, we were using the device as the ID
for the mount because that was unique to each mount. With btrfs
subvolumes however, the device and partition are the same for all, so we
need another way to differentiate.
Btrfs volumes typically only contain subvolumes instead of (parts of)
the OS tree directly. In our images in particular, this is always the
case. When searching for root to find /etc/fstab, search through the
subvolumes on a btrfs volume for the file and return the path to the
root subvolume.
Co-authored-by: Michael Vogt <michael.vogt@gmail.com>
Always set partition=None for the kwargs of the Mount() constructor.
The previous code was added for backwards compatibility with older
versions of the Mount() constructor that didn't include the 'partition
argument. It's safe to remove now because:
1. It's been long enough that we wont run osbuild-image-info with an old
version of osbuild.
2. The tool is packaged with osbuild so there is no version drift and no
compatibility issues.
When the fstab file isn't found, the root_tree will never be set after
being initialised to "" and an exception is raised "The root filesystem
tree is not mounted". It's a lot clearer if the failure happens closer
to the root cause, which is that fstab wasn't found and there are no
fstab entries to iterate through and find the root filesystem.
When iterating partitions to mount, skip any with filesystem type
"swap". This is done in two places:
1. When mounting partitions to find /etc/fstab.
2. When mounting partitions and volumes to analyse the tree.
When iterating through partitions, store the fstype along with the other
information. This will be useful for identifying btrfs partitions,
which we will need to scan for subvolumes, and for identifying swap
partitions, so we can avoid trying to mount them.
Run isort for imports.
Pylint: wrong-import-order / C0411
Solves the following linter warnings:
- standard import "pathlib" should be placed before third party import
"yaml"
- standard import "collections.OrderedDict" should be placed before
third party imports "yaml", "jsonschema"
- standard import "typing.Dict" should be placed before third party
imports "yaml", "jsonschema"
Fix default arg values.
Pylint: dangerous-default-value / W0102
- Using mutable default values ([]) for function arguments is considered
dangerous.
Rename format variable.
Pylint: redefined-builtin / W0622
- 'format' is a built-in function.
Use f-strings instead of formatting where possible.
Pylint: consider-using-f-string / C0209
Remove unnecessary else after returns.
Pylint: no-else-return / R1705
Remove unnecessary else after continue.
Pylint: no-else-continue / R1724
Set the encoding (utf-8) for all calls to open().
Pylint: unspecified-encoding / W1514
Disable the too-many-branches and too-many-statements warnings for
append_partitions() and append_filesystem(). We can refactor the
functions to make them smaller later, but for now we're addressing only
the simpler issues.
Initialise with dict literal instead of call to function.
Pylint: use-dict-literal / R1735
Use implicit truthiness for glob instead of len().
Pylint: use-implicit-booleaness-not-len / C1802
Rename ambiguous variable 'l' to 'line'.
pycodestyle: ambiguous-variable-name (E741)
Merge comparisons with 'in'.
Pylint: consider-using-in / R1714
This creates an iso matching the current method used in Fedora where it
uses grub2 for BIOS and UEFI booting. Pass the path to the grub2 hybrid
mbr to the stage in the 'grub2mbr' field. eg.
"grub2mbr": "/usr/lib/grub/i386-pc/boot_hybrid.img"
It turned out that in many cases, stages need to join two absolute
paths, the pipeline tree path and the path on a booted system. However,
the standard `os.path.join()` function can't handle such situation as
just prepending the root to the subsequent paths.
Add a new helper function, which is able to join any paths together,
regardless if any of them is absolute or not. If the root is not
absolute, the result will be made absolute to the filesystem root `/`.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Previously, the stage would try to call `os.scandir()` on non-existing
profile directories, which results in an exception. While the
directories that it scans are currently created by the TuneD RPM package
when installed, it it much nicer to gracefully handle their potential
non-existence, than to fail with an exception.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
The stage schema previously accepted an empty string as a profile name.
This would not work in practice, therefore extend the schema to require
non-empty strings.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
`read_boot_entries()` could previously fail when trying to split lines
in bootloader entries, which contained only "\n" and became empty
string after stripping whitespace characters. This is the case e.g. on
F41 images.
Moreover, bootloader entries can contain comments as lines starting with
"#", which were previously not ignored by the function and would end up
in the parsed entry and could potentially fail to be split.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add unit test for osbuild-image-info's `read_boot_entries()` function,
to ensure that it can handle various situations that can happen in the
real world.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
