Instead of using stderr for the ostree subprocess command
capture its output so that in the case of an error we get
properly return the error output. With the old behavior
all the `ostree` command output would land in the journal
of the worker.
Source, for compatability reasons, have two modes: download only
and download and export. The difference is the arguments that
are passed to the source: For download only, the `output` param
is empty. In this case also `checksums` *can* be empty and if so
it means everything, i.e. the commits, should be fetched. The
latter was not properly handled so far. Adjust the logic, which
now closely mimics that of the `org.osbuild.curl` source to fix
this case.
Also catch exceptions invoking `ostree` and properly return them
via the json error messaging.
This is a partial revert of commit d584a1e, which converted the
dynamically generate stage tests to sub tests. The problem with
the latter is that they can't be individually run, which makes
testing changes to specific stages cumbersome to develop.
Therefore switch back to a model where the stage tests are
dynamically generated via a class decorator.
In the output test, check that for a successful pipeline run all
the elements are present: main result, assembler result, stage
result.
NB: Build result is hard to test because we would need to actually
build a valid build root.
If a pipeline has an assembler and that assembler failed, the
overall status of the build also needs to be marked as failed.
This used to be the case, but a bug got introduced when the
format abstraction code was added.
This is, like the stage with the same name, an assembler that
will exit with an error code (default 255, but can be specified
via the assembler options). It is mostly useful for testing.
Set the "GRUB_CMDLINE_LINUX" variable in /etc/default/grub to the
kernel command line options. This is used by `grub2-mkconfig` to
assemble the full kernel command line when generating the menu
entires. NB: `GRUB_CMDLINE_LINUX` does NOT include the root fs
bits (`root=...`), since that is generated by grub2-mkconfig
itself.
Do the check if there is a /etc/machine-id before moving /etc to
/usr/etc, because otherwise /etc/machine-id will obviously not
exist and thus the detection is broken.
Instead of including SELinux labels for the content layers via the
`--selinux` tar option, make sure selinux labels are not included by
using the `--no-selinux` option.
The inclusion of the labels was a mistake, since they should be
determined by the target system because selinux labels are not
namespaced. On RHEL/Fedora the SELinux label used is something like
`system_u:object_r:container_ro_file_t:s0` for all the files in the
container.
Including the label was leading to permission problems because
the files had a different label on the host and programs inside
the container get `EACCES`, i.e. Permission denied, errors when
accessing files with the different label.
Interestingly this does not happen on Fedora 33 but only on RHEL.
One possibility is that the overlayfs kernel driver in RHEL is
behaving differently on RHEL than on Fedora.
The pylorax implementation of the template running code supports
globbing, as well as `--exclude` and `--optional` commands. These
are handled independently for each `installpkg` command, so that
requesting the installation of firmware packages in one command
together with an exclude of `*alsa` does indeed only exclude any
alsa firmware packages but not any other alsa packages. The
previous version of this script would just build an global list
for excludes which has a drastically different result because
a global exclude of "*alsa" would result in a global exclusion of
all matching packages and probably a dependency error.
Therefore, add support for dnf based filtering of packages on a
per `installpkg` command bases, very much like pylorax does.
Add a simple helper that is meant to gather the list of packages
to be installed via a lorax template that uses the "installpkg"
directives. A prominent example is the 'runtime-install.tmpl'
script from lorax-templates-generic, used to create boot isos.
This builds osbuild in F33aarch64 as well as RHEL8.4 and CentOS8. No tests are currently run.
With the mockbuilds in place we will be able to run composer CI against osbuild master, when necessary.
Extend the current ostree container test case to use the new
bootiso manifest to build a bootable iso. This is done in the
existing container test case in order to share the build root
and ostree commit results and speed up testing. In the future,
the test infrastructure should be extended so that the cache
can be optionally shared between test cases.
Port the org.osbuild.tar assembler to a new assembler like stage,
that takes a tree input. The only real change is that instead of
having a compression argument, the compression is now based on
the file ending.
Add support for the 'liveimg' kickstart command, which can be used
with tar payloads included in the installer image, to install
pre-built image archives.
This new manifests creates an boot iso which contains an ostree
commit that is build in the same manifest. The boot iso can then
be installed via the ISO.
New stage that uses the implantisomd5(1) to implant MD5 checksums
into an ISO. This is then used by a dracut module in the installer
ISO to check the installation medium.
Add a new stage that uses the `xorrisofs`(1) command line utility
to assemble a. The iso can be made bootable by specifying a
combination of the `boot` and `efi` options.
Add a new stage that prepares a bootable file system tree suitable
for writing to an ISO file system. It currently only supports
EFI and PC-BIOS boot. It takes a tree input which will be wrapped
into a ext4 file-system wrapped into a squashfs image.
Add a new stage that uses the recently added lorax template
helpers to execute such a template. The template itself will
be search in the build root, but the command of the script
will operate on the tree.
A new module that can parse and execute Lorax script templates,
which are mako template based files that support a limited set
of commands, like "install", "remove" and such.
The module provides helper functions to parse such templates
and execute them by providing a re-implementation of a subset
of the commands. All commands needed for running the post
installationtemplates were implemented.
Add a stage to create a buildstamp file, which is required by
anaconda to properly function, since it configures varies
aspects of the installation target (product, build arch, ...).
Add a stage to configure anaconda. For now only the enabled
kickstart modules can be configured. This is done by dropping
a file "90-osbuild.conf" in `/etc/anaconda/conf.d`.
There was a bug in mke2fs (fixed in versionv 1.45.7, with commit
6fa8edd0) where mkfs.ext4 would fail because the default config,
created on the fly, would contain a syntax error. The program
would abort with:
Syntax error in mke2fs config file (<default>, line #22)
Unknown code prof 17
To avoid this error, we try to bind mount the config from the build
root.
Since pipelines can now be uniquely addressed via their names,
add the ability to checkpoint via the pipeline name. This will
effectively checkpoint the last stage of a pipeline.
For format v1 manifests, the build pipeline is called "build",
the main pipeline is called "tree" and the pipeline for the
assembler is called "assembler".
This is to test the functionality of the new `saved_entry` grub2
config file. Ideally an integration test would install a new non-
default kernel and check that it does not get selected. Something
for the future.
The current grub configuration has no logic to select the default
boot entry, which means the next boot entry is chosen by grub2
by the sort order of available entries. This will break tooling
like grub2-set-default and grub2-reboot that set on a specific
variable `saved_entry` in the grub environment file `grubenv` to
to point to the select boot entry.
Change our grub configuration file to use that `saved_entry` var
to select the next boot entry to unbreak the tooling.
In addition to the aforementioned tools, the saved_entry is also
updated by the kernel install scripts when new kernel is installed,
but only if the newly installed kernel is the selected default.
Thus not respecting the `saved_entry` variable might cause a bug
where a newly installed non-default kernel gets selected by grub2.
Use the new immutable image infrastructure from `osbuild/containers`.
While at it, also switch over to the new github-actions helper, now that
we no longer run `systemd-nspawn` in our tests.
The old image was renamed from `ghci-osbuild` to `osbuild-ci` to avoid
accidentally replacing old images. The new infrastructure uses immutable
images, so downstream will no longer get automatic updates, unless the
`latest` tags are used.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
Although MPP supported having multiple repo entries for each
depsolve block, in reality that didn't actually work because
the same (global) baseurl was used for all packages.
Now the basurl can still be (otionally) specified globally,
but also overridden by the repo block and the package paths
will be relative to that.
In addition to the required base layer, provided via the the input
of the same name, the oci-archive stage now accepts up to nine
additional layers that get added on top of each other, sorted in
ascending order, i.e. `layer.1` to `layer.9`.
Adapt the `fedora-ostree-container` example manifest so that the
ostree commit is now in a separate layer, which makes it possible
to share the base layer between different commits container.
