Now that as a result of commit 4d2f15f all symlinks have been
dropped from the individual module paths, the search for module
contents can be simplified again.
In case `--libdir` is not specified on the command line, and thus
`args.libdir` is `None`, pass the standard `/usr/lib/osbuild` path
to the meta.Index constructor. Otherwise no schema information can
be found.
By using a small Jenkins pipeline in the repository, we can define
almost all of our testing parameters in the repo itself and not inside
Jenkins. 🥳
This also allows us to use the GitHub Branch Source plugin and
auto-discover new repositories without `ok to test` bombs in
pull requests.
Signed-off-by: Major Hayden <major@redhat.com>
The truthiness of the `Schema` object itself now contains the
schema validation as well, i.e. schema is only valid if schema
information is present and said information passes validation.
Up until now it was verified that loading the `STAGE_OPTS` data
works, i.e. that it contains valid JSON, but the schema itself
was not verified. Use the new `Schema.check` method to do that.
The _validator member of `Schema` is used as an indicator whether
the provided schema is valid. The `check` method will, in case
that _validator is not set attempt to validate the schema data,
if present and set the _validator member if schema data is set and
validation has passed. On failure, i.e. missing schema information
or invalid schema data, the ValidationResult will contain the
respective error.
This option will print the manifest in JSON, including all the ids,
to stdout. It will not build the pipeline, but the input manifest
will be validated and if that fails the validation result will be
return in JSON.
Add a option to all description methods to include the respective
ids in the description. Defaults to False to preserve the original
output which is used in the tests.
Change all the schemata to not allow additional properties. This
should help with misspelled properties as well as missing schema
information in the stage itself.
Done via a small python3 script:
--- 8< --- 8< --- 8< --- 8< --- 8< --- 8< --- 8< --- 8< ---
import os
import sys
def list_stages(base):
return [(base, f) for f in os.listdir(base) if f.startswith("org.osbuild")]
stages = list_stages("stages")
stages += list_stages("assemblers")
def find_line(lines, start):
for i, l in enumerate(lines):
if l.startswith(start):
return i
return None
NOADD = '"additionalProperties": false'
for stage in stages:
with open(f"{stage[0]}/{stage[1]}", "r") as f:
print(f"{stage[0]}/{stage[1]}", file=sys.stderr)
data = f.readlines()
i = find_line(data, 'STAGE_OPTS = """')
if i:
data.insert(i+1, NOADD + ",\n")
else:
i = find_line(data, 'STAGE_OPTS = ""')
if i:
data[i] = f'STAGE_OPTS = """\n'
data.insert(i+1, NOADD + "\n")
data.insert(i+2, '"""\n')
with open(f"{stage[0]}/{stage[1]}", "w") as f:
f.writelines(data)
In the "test_tar" test the compression can be None, which in turn
will be serialized as "compression: null" in the options dict.
However, this is not a valid option according to the schema. The
schema could be adapted to allow for this but it is probably
better to just omit empty or null values.
Install the schema to %{_datadir}/osbuild/schema and provide a
link from %{pkgdir}/schema to that location so that the osbuild
library can easily access the schemata.
Validate the options of stages and assembler of the pipeline
before running it. A validation failure will abort the run.
Errors are printed in human readable unless `--json` is passed;
For each error a human readable message together with a path
to the object with the error is given. The syntax of the path
is such it can be used via the `jq` command to select the item.
Add checks for the new validation API, specifically check that
errors in the main manifest, the stages and stage options,
assembler and assembler options. Also ensure that there are no
duplicated errors due to pipeline nesting (e.g. build pipelines).
This new module contains utilities that help to introspect parts
that constitute the inner parts of osbuild, i.e. its stages
and assembler (which is also considered a type of stage in
this context). It contains the `StageInfo` class that can that
contains meta-information about the individual stage, such as
a short information (`info`), a longer description (`desc`) and
its JSON schema. A new Schema class represents schema data and
has a `validation` method that can be used to validate that json
data conforms to said schema.
A `Index` class can be used to obtain `StageInfo` and `Schema`
for entities identified via `klass` and `name`.
A top level `validate` method is introduced that can validate
manifest data.
Internally it uses the `jsonschema` package so add that as a
requirement and Install this dependency in the CI.
Using the "$schema" without any specific version has been
deprecated[1] and will lead to warnings like:
DeprecationWarning: The metaschema specified by
$schema was not found. Using the latest draft to validate,
but this will raise an error in the future
Fix this by pointing $schema to "draft-07", which is the latest
version fully supported the jsonschema python package, that is
being used internally.
[1]
"The possibility to declare $schema without specific version
(http://json-schema.org/schema#) was deprecated after Draft 4
and should no longer be used."
https://json-schema.org/understanding-json-schema/reference/schema.html
This reverts commit 33844711cd.
There are systems were our runners have no standard python3 location
available. They will fix the environment before invoking any further
utilities. Therefore, we cannot rely on `python3 foo.py` to work in our
ad-hoc containers.
This simply reverts the behavior back to using the shebang.
Drop the `osbuild -> ../osbuild` symlink from all module directories.
We now properly initialize the PYTHONPATH to provide the imported
osbuild module from the host environment. Therefore, these links are no
longer needed.
The sources run from the host environment, so they should just pick them
up from the environment the same way osbuild itself does.
Prepare the PYTHONPATH of the build-root container to include the path
to the osbuild library. This way, we no longer need any symlinks or
bind-mounts for the individual modules.
Use the python-interpreter explicitly to invoke the runners. This works
around inconsistencies between scripts imported from the host, and the
interpreter taken from a build-root.
This commit turns on the `cleanup_composer_directories` option to clean up
the osbuild-composer directories during the time the services are stopped
(when ansible-osbuild is about to deploy the new versions of the
services).
Taken from osbuild/osbuild-composer#575, thanks to @major!
Jenkins now uses a non-root user for its agents, so we will need to use
sudo for some commands.
Stolen from osbuild/osbuild-composer#581, big thanks to @major!
With `--keep-unit` we now run with the privileges and resources of the
caller. We no longer require external services to extend our privileges.
This also means we no longer have to configure our unit sandbox
manually, but simply rely on kernel sandboxing to do the right thing.
Create a preliminary NEWS entry for the upcoming v13 release. Lots of
fixes all over the place, but most of them not worth mentioning for the
release. However, we do have some feature improvements as well.
Although we are doing a good job of finding osbuild breakage when we
test osbuild-composer, we aren't finding osbuild-composer breakage
when we test osbuild.
Run the image tests from osbuild-composer whenever we do CI for osbuild.
Fixes#362.
Signed-off-by: Major Hayden <major@redhat.com>
Split off the argument parser as well as the manifest parser into
helper functions. Drop the pylint hints from the main function now that
it is considerably smaller.
This extracts the CLI entrypoint into `main_cli.py` and prepares the
codebase for the introduction of additional entrypoints. This should
not contain any functional changes.
The idea behind this is to add `main_api.py` (and maybe more in the
future), which will be similar to `main_cli.py` but contain the
`osbuild-api` entrypoint. This will make all entrypoints nicely symetric
and the only difference will be `setup.py` selecting the right
entrypoint for each executable, as well as `__main__.py` selecting the
entrypoint for the module itself (which we will keep to the CLI for
compatibility).
