The code in `check-snapshots` will print "No snapshots cache found
at ..." regardless of the error that happens when trying to open
the file. This can be misleading if e.g. the issue is permissions
to open the file or the file is corrupted. So make the exception
more targeted and only catch FileNotFound error and let python
how the full error for the other cases. Obviously this can be
done in many ways so I'm happy to tweak and e.g. keep catching
all exception but print the value etc.
Partitions by default are indexed starting at 1, but in
some cases, such as CoreOS for IBM Z, it may be usefull
to set the 'partnum' for GPT disks explicitly, without
creating dummy partitions.
Now user can define an image:
```
mpp-define-images:
- id: image
size: 10737418240
table:
uuid: 00000000-0000-4000-a000-000000000001
label: gpt
partitions:
- name: boot
type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
partnum: 3
size: 786432
- name: root
type: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
partnum: 4
size: 4194304
```
So target disk would look like:
```
Disklabel type: gpt
Disk identifier: 00000000-0000-4000-A000-000000000001
Device Start End Sectors Size Type
/dev/loop0p3 2048 788479 786432 384M Linux filesystem
/dev/loop0p4 788480 4982783 4194304 2G Linux filesystem
```
This patch updates the osbuild-mpp tool and the sgdisk and sfdisk
stages to support this.
Co-authored-by: Dusty Mabe <dusty@dustymabe.com>
I mistakenly thought the .strip('.0') would strip off just '.0` but
in reality it strips off all instances of `.` or `0` on the end so
'4075520.0'.strip('.0') => '407552', which is clearly wrong.
This is a fixup for 5207e92.
So we can use something like mpp-format-string inside of mpp-embed. An
example would be below where we want to substitute the ociarchive var
in using `mpp-format-string: 'file://{ociarchive}'`.
```
version: '2'
mpp-vars:
ociarchive: /path/to/fedora-coreos-39.20240104.dev.2-ostree.x86_64.ociarchive
pipelines:
- name: oci-archive
stages:
- type: org.osbuild.copy
inputs:
inlinefile:
type: org.osbuild.files
origin: org.osbuild.source
mpp-embed:
id: fcos.ociarchive
url:
mpp-format-string: 'file://{ociarchive}'
options:
paths:
- from:
mpp-format-string: input://inlinefile/{embedded['fcos.ociarchive']}
to: tree:///fcos.ociarchive
```
Partitions are usually indexed starting a 1 so an index starting
at 0 is confusing (i.e. you never say mount the filesystem on
partition 0). Let's add a partnum field that can be used in
definitions instead.
`mpp-define-images` can create an image file, using `losetup` to deal
with non-standard sector sizes requires root. Not all users run
`osbuild-mpp` as root.
While I am not a fan of "suddenly sudo" based on the input manifest this
does alleviate builds breaking for manifests with default sector sizes
when non-root.
For our Fedora CoreOS disk images we set the partition labels (name)
for the partitions. This is also supported using the primitives here
in OSBuild, but it wasn't obvious that I needed to set the name in
the mpp-define-images definition. Let's set the name there, but let's
also allow osbuild-mpp to set the `id`, which is what is used later
to access that partition from the `name` too if `id` isn't set.
This means we allow something like:
- name: BIOS-BOOT
type: 21686148-6449-6E6F-744E-656564454649
bootable: true
uuid: FAC7F1FB-3E8D-4137-A512-961DE09A5549
size: 100
rather than requiring something like:
- id: BIOS-BOOT
name: BIOS-BOOT
type: 21686148-6449-6E6F-744E-656564454649
bootable: true
uuid: FAC7F1FB-3E8D-4137-A512-961DE09A5549
size: 100
Now you can specify a sector_size in `mpp-define-images` to support
creating a 4k native disk image (sector_size=4096).
This does use a loopback device, which means osbuild-mpp also needs
to run as root, when previously that wasn't necessary.
If you do math in mpp-format-int it could end up getting converted
to a float. Of course if you end up with a decimal value that isn't
`.0` that's a problem for an int, but if it is `.0` let's handle it
gracefully.
For example, math like this could end up with a value with `.0`:
mpp-format-int: "{bios_boot_size_mb * 1024 * 1024 / sector_size_bytes}"
I'm trying to debug some failures, and having no feedback as to
what file we're parsing or what code is evaluated when something
fails makes it hard to debug manifests.
This adds some nice error messages that will help.
Instead of running osbuild as a binary use `python3 -m osbuild`
(just like in `test/test.py:compile()`) so that it will use
osbuild fromgit and can be run from a checkout without the need
for an installed osbuild.
Instead of using the F34-based manifests, let's switch to F38. I tried my
best to import the vars into the new manifest, but I don't think that's
currently supported.
The `./tools` dir was not part of the LINTABLES in the `tox.ini`
which meant that pep8/pylint etc checks were not run on the tools
there.
This commit adds it and fixes the issues that `make lint` found.
This script reads the RUNNER entries from .gitlab-ci.yml and the
snapshot repo names from Schutzfile. It then prints which runners don't
have entries in Schutzfile and which Schutzfile entries don't have
runners.
This is slightly complicated by the fact that centos-stream reports
itself as centos. So when checking for missing entries they are
considered to be the same thing. eg. If centos is a RUNNER and
centos-stream is in Schutzfile it doesn't report anything missing.
The import of `dnf-json` came from `osbuild-composer`, as we sadly
have/had multiple copies this wasn't the *right* version as it was
lacking previous work to disable weak dependencies.
This stage is similar to ostree.deploy, but deploys from a container
image rather than from an OSTree commit by using the `ostree container
image deploy` command. An example stage definition could look like:
```
- type: org.osbuild.ostree.deploy.container
options:
osname: fedora-coreos
target_imgref: ostree-remote-registry:fedora:quay.io/fedora/fedora-coreos:stable
mounts:
- /boot
- /boot/efi
kernel_opts:
- rw
- console=tty0
- console=ttyS0
- ignition.platform.id=qemu
- '$ignition_firstboot'
inputs:
images:
type: org.osbuild.containers
origin: org.osbuild.source
mpp-resolve-images:
images:
- source: quay.io/fedora/fedora-coreos
tag: stable
```
Co-authored-by: Dusty Mabe <dusty@dustymabe.com>
Import the `dnf-json` Python script that is included in the "images" and
"osbuild-composer" repository into "osbuild".
`dnf-json` provides dependency solving capabilities by taking in some
JSON and then outputting the package set that JSON resolves to. Since it
is a Python script it moves into a subpackage of `osbuild`.
The script is renamed to `osbuild-depsolve-dnf`.
This adds an early return to the `_process_ostree_commits` function to
prevent an empty ostree commit object from being created in the deploy
stage which causes violations to the deploy stage input schema.
This pulls the list of snapshots from the rpmrepo API, greps the
codebase for all uses of rpmrepo.osbuild.org that look like a snapshot
name, and then checks to make sure they are still valid.
This is the same script as osbuild-composer, except that it also skips
checking ./test/data/stages/ which has rpm urls that look like snapshot
urls.
There hasn't ever been an official tool for this and everyone was just using
the output from the test to generate the diff. This commit introduces a proper
tool for generating these files.
The schemas allowed for `inputs` changed a while back; most new
manifests use a different format than what `osbuild-dev` initially
understood.
This adds all schemas.
Add the translation logic to handle the attrs field of sfdisk as
supported by org.osbuild.sfdisk and documented in its schema. With the
schema taking an int array, some translation is required to populate the
sfdisk command appropriately.
Amend the example schema to reflect the change.
Signed-off-by: Eric Chanudet <echanude@redhat.com>
`os.scandir()` can accept file descriptors only since Python 3.7. The
tool would previously fail with exception when run using Python 3.6.
The solution is to provide a path, which is done using a symlink in
procfs (this is already used within the tool).
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
Add support for resolving manifest lists in osbuild-mpp.
Adds an `index` boolean field to the container image struct for
mpp-resolve-images. When enabled, the preprocessor will also store the
manifest-list digest as a separate skopeo-index source and add it to the
skopeo stage under the `manifest-lists` input.
According to the OCI Image Index specification the mediaType field is not mandatory
Assume that it is a list if mediaType is not set while manifests field is
Signed-off-by: Ygal Blum <ygal.blum@gmail.com>
