The change in commit ed33869430 to
use hashlib.file_digest breaks with older python, because
it was added in 3.11.
This change reverts back to hashing all the data in the case
where file_digest doesn't exist.
This allows using e.g. mpp-eval in the resolve-image operation, similar
to how it is now possible in the mpp-depsolve handling.
We want this so we can inject the list of images from a list variable
(that can then be mpp-join:ed, etc).
Currently if one uses `mpp-embed` with URLs, osbuild-mpp still wants to
download the full file just so it can hash it. Make this more efficient
by hashing from the stream instead, which `hashlib` natively supports.
This also makes osbuild-mpp work with large artifacts in environments
that may not have enough space to temporarily save the data.
This commit allows to the user to define the "container-transport"
variable when they are defining a container image to be resolved by the
osbuild-mpp tool. When the user defines the "container-storage" as
"container-transport" osbuild-mpp will look for the image in the local
container-storage.
Signed-off-by: Roberto Majadas <rmajadas@redhat.com>
Partitions by default are indexed starting at 1, but in
some cases, such as CoreOS for IBM Z, it may be usefull
to set the 'partnum' for GPT disks explicitly, without
creating dummy partitions.
Now user can define an image:
```
mpp-define-images:
- id: image
size: 10737418240
table:
uuid: 00000000-0000-4000-a000-000000000001
label: gpt
partitions:
- name: boot
type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
partnum: 3
size: 786432
- name: root
type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
partnum: 4
size: 4194304
```
So target disk would look like:
```
Disklabel type: gpt
Disk identifier: 00000000-0000-4000-A000-000000000001
Device Start End Sectors Size Type
/dev/loop0p3 2048 788479 786432 384M Linux filesystem
/dev/loop0p4 788480 4982783 4194304 2G Linux filesystem
```
This patch updates the osbuild-mpp tool and the sgdisk and sfdisk
stages to support this.
Co-authored-by: Dusty Mabe <dusty@dustymabe.com>
I mistakenly thought the .strip('.0') would strip off just '.0` but
in reality it strips off all instances of `.` or `0` on the end so
'4075520.0'.strip('.0') => '407552', which is clearly wrong.
This is a fixup for 5207e92.
So we can use something like mpp-format-string inside of mpp-embed. An
example would be below where we want to substitute the ociarchive var
in using `mpp-format-string: 'file://{ociarchive}'`.
```
version: '2'
mpp-vars:
ociarchive: /path/to/fedora-coreos-39.20240104.dev.2-ostree.x86_64.ociarchive
pipelines:
- name: oci-archive
stages:
- type: org.osbuild.copy
inputs:
inlinefile:
type: org.osbuild.files
origin: org.osbuild.source
mpp-embed:
id: fcos.ociarchive
url:
mpp-format-string: 'file://{ociarchive}'
options:
paths:
- from:
mpp-format-string: input://inlinefile/{embedded['fcos.ociarchive']}
to: tree:///fcos.ociarchive
```
Partitions are usually indexed starting a 1 so an index starting
at 0 is confusing (i.e. you never say mount the filesystem on
partition 0). Let's add a partnum field that can be used in
definitions instead.
`mpp-define-images` can create an image file, using `losetup` to deal
with non-standard sector sizes requires root. Not all users run
`osbuild-mpp` as root.
While I am not a fan of "suddenly sudo" based on the input manifest this
does alleviate builds breaking for manifests with default sector sizes
when non-root.
For our Fedora CoreOS disk images we set the partition labels (name)
for the partitions. This is also supported using the primitives here
in OSBuild, but it wasn't obvious that I needed to set the name in
the mpp-define-images definition. Let's set the name there, but let's
also allow osbuild-mpp to set the `id`, which is what is used later
to access that partition from the `name` too if `id` isn't set.
This means we allow something like:
- name: BIOS-BOOT
type: 21686148-6449-6E6F-744E-656564454649
bootable: true
uuid: FAC7F1FB-3E8D-4137-A512-961DE09A5549
size: 100
rather than requiring something like:
- id: BIOS-BOOT
name: BIOS-BOOT
type: 21686148-6449-6E6F-744E-656564454649
bootable: true
uuid: FAC7F1FB-3E8D-4137-A512-961DE09A5549
size: 100
Now you can specify a sector_size in `mpp-define-images` to support
creating a 4k native disk image (sector_size=4096).
This does use a loopback device, which means osbuild-mpp also needs
to run as root, when previously that wasn't necessary.
If you do math in mpp-format-int it could end up getting converted
to a float. Of course if you end up with a decimal value that isn't
`.0` that's a problem for an int, but if it is `.0` let's handle it
gracefully.
For example, math like this could end up with a value with `.0`:
mpp-format-int: "{bios_boot_size_mb * 1024 * 1024 / sector_size_bytes}"
I'm trying to debug some failures, and having no feedback as to
what file we're parsing or what code is evaluated when something
fails makes it hard to debug manifests.
This adds some nice error messages that will help.
The `./tools` dir was not part of the LINTABLES in the `tox.ini`
which meant that pep8/pylint etc checks were not run on the tools
there.
This commit adds it and fixes the issues that `make lint` found.
This stage is similar to ostree.deploy, but deploys from a container
image rather than from an OSTree commit by using the `ostree container
image deploy` command. An example stage definition could look like:
```
- type: org.osbuild.ostree.deploy.container
options:
osname: fedora-coreos
target_imgref: ostree-remote-registry:fedora:quay.io/fedora/fedora-coreos:stable
mounts:
- /boot
- /boot/efi
kernel_opts:
- rw
- console=tty0
- console=ttyS0
- ignition.platform.id=qemu
- '$ignition_firstboot'
inputs:
images:
type: org.osbuild.containers
origin: org.osbuild.source
mpp-resolve-images:
images:
- source: quay.io/fedora/fedora-coreos
tag: stable
```
Co-authored-by: Dusty Mabe <dusty@dustymabe.com>
This adds an early return to the `_process_ostree_commits` function to
prevent an empty ostree commit object from being created in the deploy
stage which causes violations to the deploy stage input schema.
Add the translation logic to handle the attrs field of sfdisk as
supported by org.osbuild.sfdisk and documented in its schema. With the
schema taking an int array, some translation is required to populate the
sfdisk command appropriately.
Amend the example schema to reflect the change.
Signed-off-by: Eric Chanudet <echanude@redhat.com>
Add support for resolving manifest lists in osbuild-mpp.
Adds an `index` boolean field to the container image struct for
mpp-resolve-images. When enabled, the preprocessor will also store the
manifest-list digest as a separate skopeo-index source and add it to the
skopeo stage under the `manifest-lists` input.
According to the OCI Image Index specification the mediaType field is not mandatory
Assume that it is a list if mediaType is not set while manifests field is
Signed-off-by: Ygal Blum <ygal.blum@gmail.com>
For some reasons I forgot to fix those in the previous runs. Fix a
linter and pep8 warning.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
Avoid using `os` or `input` as symbol names, since they shadow
previously defined symbols.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
Make sure to correctly implement virtual functions and raise
NotImplementedError, as suggested by the standard library. Pylint will
otherwise complain about unavailable functions being accessed.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
The `problem` member is only defined if a problem-mark is present. If
the mark is not set, we get no details on the error whatsoever and have
to just print a generic error message.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
The __getitem__() function incorrectly returns `None` if the
partition-type is not found. Make sure to always return IndexError if
the requested value is outside the supported range.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
This mergest the handling of process_stages() and process_format() into
just one process_format(), which incrementally tracks the call stack
of the formating, which allows it to detect when it is hitting a stage
and can call _process_stage().
This means it is possible to mix things like mpp-if and mpp-join with
stages.
For example, you can do complex combinations like:
pipelines:
- name: rootfs
stages:
mpp-join:
- - type: org.ostree.foo
- mpp-if: use_bar
then:
type: org.osbuild.bar
- mpp-if: use_extra_stages
then:
mpp-eval: extra_stages
This is particularly useful if you included something and you want
to mpp-join something that was set in a variable.
Both file embedding and depsolves start by iterating over all stages,
and we want to add another similar one for container installs, so
break out the iteration over the containers so that it is done in
one place only.
This introduces a new dependency resolver to osbuild-mpp for Arch Linux
which uses the pacman package manager. The used solver is determined by
the `solver` field in the `mpp-depsolve` object inside the manifest
file, if it does not exists it falls back to the DepSolver for dnf/rpm.
Co-Authored-By: Jelle van der Waa <jvanderwaa@redhat.com>
