Commit graph

1465 commits

Author SHA1 Message Date
Christian Kellner
d5c5947271 test/data: generate the fff... uuid
Instead of writing 32 `f` chars, we can now generate it. The
future is here.
2021-07-02 11:28:43 +02:00
Christian Kellner
8a27de0d10 tools/osbuild-mpp: include nevra in package list
Include the nevra in the list of dep-solved packages. Use that for the
package info. Adapt the boot manifest accordingly.
2021-07-02 11:28:43 +02:00
Christian Kellner
ff76b6fade tools/osbuild-mpp: small whitespace fixes
Since David is on leave, I have to do them. Also Move the pylint
comment above the line, not next to it.
2021-07-02 11:28:43 +02:00
Christian Kellner
5ac1e004a6 spec: ship osbuild-mpp in new -tools
Include the `osbuild-mpp` tool, that is now installed via `setup.py`
in a new `osbuild-tools` package.
2021-07-02 11:28:43 +02:00
Christian Kellner
a247b8cc1e setup: install osbuild-mpp
Install the Manifest Pre-Processor tool.
2021-07-02 11:28:43 +02:00
Christian Kellner
44fec6cd91 tools/mpp: rename mpp.py to osbuild-mpp
Prepare the optional installation of the script via a more unique
and langauge independent name.
2021-07-02 11:28:43 +02:00
Christian Kellner
e4f39c5e03 devcontainer: add pyyaml dependency
This is needed for the pylint to work correctly with the new
cloud-init stage.
While there, add some more utilities useful for debugging. Also
add glibc-langpack-en needed for `en_GB`.
2021-07-01 23:29:03 +02:00
Alexander Larsson
53af328b7c mpp: Use the right format for the file secrets
I was getting:

.sources.org.osbuild.curl.items.sha256:02c8f76fd8b13972c541e4fa3bd2101681b7d4baa1320e3101bc4167b5531685:
  {'url': 'https://cdn.redhat.com/content/dist/rhel8/8/aarch64/baseos/os/Packages/e/elfutils-libelf-0.182-3.el8.aarch64.rpm', 'secrets': 'org.osbuild.rhsm'} is not valid under any of the given schemas

Because the secrets key should have an object, not a string value.
2021-07-01 18:15:00 +02:00
Alexander Larsson
e4eb3e0fd0 Use the new formating features to simplify the test manifests
This uses size computations to simplify the partition size/offsets
and the depsolv results to get the kernel version.

This makes no changes to the resulting json files.
2021-07-01 15:06:57 +02:00
Alexander Larsson
bdfef651eb mpp: Set a variable to the set of depsolved rpms
This allows you to format strings based on what will get installed.
A common example here is extracting the kernel version for the
dracut stage.
2021-07-01 15:06:57 +02:00
Alexander Larsson
e541c1c196 mpp: Add support for formating strings
This lets a mpp manifest define some variables and then use
these variables inside python f-strings to expand things in a
flexible way. This allows a single value such as `rootfs_size` to
be expanded in various places, including using computations to
e.g. define the partition offsets/sizes.
2021-07-01 15:06:57 +02:00
Achilleas Koutsou
ba3467d747 test/data: update nginx.conf manifests
- Update to v2 manifest
- Add build stage
2021-07-01 10:49:38 +02:00
Achilleas Koutsou
7e1f2fa9b9 test/data: update chmod manifests
- Update to v2 manifest
- Add build stage
- Create file from inline data and use it for testing chmod
2021-07-01 10:49:38 +02:00
Achilleas Koutsou
6636d390d6 stages/nginx.conf: SCHEMA_2 2021-07-01 10:49:38 +02:00
Achilleas Koutsou
c77767c788 stages/chmod: SCHEMA_2 2021-07-01 10:49:38 +02:00
Christian Kellner
fbbefa0f48 tools/mpp: remove separate mpp tools
Those got unified in tools/mpp.py, which should be used now instead.
2021-06-30 20:26:07 +01:00
Christian Kellner
68bd2c5339 Makefile: use new mpp tool
Use the new unified mpp tool instead of the two separate tools.
2021-06-30 20:26:07 +01:00
Christian Kellner
3b7320d114 tools/mpp: remove elif after return
Pylint will complain otherwise.
2021-06-30 20:26:07 +01:00
Christian Kellner
1202085883 tools/mpp: move main code into main function
Move all the global code into a `main` function and call that. This
fixes a lot of pylint warnings where variable names were re-used
from the main, and thus global, context.
2021-06-30 20:26:07 +01:00
Christian Kellner
802f401069 tools/mpp: refactor dep-solving
Create a DepSolver class that carries the global state such as dirs
and subscription information, as well as local state, like the 
repositories and basedir. The latter can be reset so the class can
easily be re-used for all dep-solve sections.
This avoids having any global state.
2021-06-30 20:26:07 +01:00
Christian Kellner
77c5c8e8a6 tools/mpp: silence bare-except warning
In that case we don't care what exactly went wrong.
2021-06-30 20:26:07 +01:00
Christian Kellner
f37c83ec25 tools/mpp: raise exception from None
Raise the `ValueError` that is thrown when we are unable to get
the secrets from `None`.
2021-06-30 20:26:07 +01:00
Christian Kellner
c600ea60a4 tools/mpp: don't compare None with ==
Either use `is` or the truth-y-ness directly.
2021-06-30 20:26:07 +01:00
Christian Kellner
bb620d727d tools/mpp: use python3 style super invocation
In Python3 there is no need for arguments in the `super` invocation
itself.
2021-06-30 20:26:07 +01:00
Christian Kellner
fdd757dd28 tools/mpp: sort the source urls
Sort the checksum: urls dictionary via the url, so that no matter
where the urls came from (import or any dep-solve section), the
checksum: url dict is the same for the same set of urls.
2021-06-30 20:26:07 +01:00
Christian Kellner
ec68369daf tools/mpp: support search dirs for manifests
Add support for additional paths that are searched when trying to
load a manifest. Currently only the path of the manifest that has
the include is searched. With this changed additional directories
will be included after that, in the order they were given ton the
command line.
2021-06-30 20:26:07 +01:00
Christian Kellner
2230dfe566 mpp: use simple urls if possible
If no `secrets` is required to download a package we can use the
simple checksum: url mapping, instead of the checksum: {url: }
one.
2021-06-30 20:26:07 +01:00
Christian Kellner
1b4f96c4e7 mpp: unify add package code path
Both v1 and v2 code paths used the same logic to add the resolved
dependencies to the `source_urls` dict, which is already stored
in the base class. Move the logic into the base class too.
2021-06-30 20:26:07 +01:00
Christian Kellner
4add7c69ff mpp: turn comment into doc string 2021-06-30 20:26:07 +01:00
Christian Kellner
3298a6461e mpp: small whitespace fixes
Be PEP-8 complaint.
2021-06-30 20:26:07 +01:00
Alexander Larsson
ab453bf81a mpp: combine depsolve and import into mpp.py
Rewrite image pre-processor to single tool so that it is easier to use.
Now also supports `ignore-weak-deps` when dep-solving and supports
relative paths for local files.
Also create a symlink to the osbuild package, so that the tools can be
run from the source checkout and have access to the osbuild package.
2021-06-30 20:26:07 +01:00
Jakub Rusz
e5d599d8ee ci: add Gitlab CI
This is an inital commit that enables the use of Gitlab CI.
2021-06-30 19:54:37 +02:00
Jakub Rusz
803c698f9a ci: Remove Jenkins in favor of Gitlab CI
Jenkinsfile and the trigger are no longer neccessary since
we're moving all testing to Gitlab CI.
2021-06-30 19:54:37 +02:00
Christian Kellner
ace07f2656 stages/tar: support choosing the format
Add support different archive formats. The list is taken from what `tar`
currently supports.
2021-06-30 17:36:40 +02:00
Christian Kellner
3ab543937c test/stages: include checks for the xz stage
Compress a file, delivered via the inline source, and check it exists.
2021-06-30 12:06:30 +02:00
Christian Kellner
dfda290d6a stages: add org.osbuild.xz to compress files
Add a new stage that will take a file from the input and compress
it via xz.
2021-06-30 12:06:30 +02:00
Christian Kellner
5d22a672b4 format/v1: remove now unused get_ids function
This was only used in a test utility method. Its use was replaced
and thus it is now longer needed.
2021-06-30 12:06:30 +02:00
Christian Kellner
427266c583 test: support v2 format for tree-id detection
In the function `treeid_from_manifest`, use dynamic format detection
instead of hard-coding the use of the version 1 format. Additionally,
directly access the `tree` pipeline and its `id` instead of going
via the `get_ids` helper function, which is only present in v1.py.
2021-06-30 12:06:30 +02:00
Christian Kellner
7576191c2d sources/inline: fix schema
The top-level node "items" was not defined and the required property
"encoding" was wrongly called "method".
2021-06-30 12:06:30 +02:00
Tomas Hozza
88da3beabc stages/authselect: new stage to select system identity and auth sources
Add a new `org.osbuild.authselect` stage for configuring system identity
and authentication sources using `authselect`. The stage runs
`authselect select` command from the image in a chroot to set the
desired 'profile_id' profile. Optionally, a list of specific profile
features to enable can be passed using the 'features' option.

Add a test case for the new stage.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-06-30 09:17:08 +02:00
Tomas Hozza
6140ba1130 stages/keymap: add option to configure X11 keyboard
Add a new option 'x11-keymap' for configuring the X11 keyboard settings.
The value of the option is a dictionary with keys representing settings
which can be configured. Currently, only the 'layouts' setting can be
configured. Its value is a list of strings, representing the specific
layouts, which should be configured for the X11 keyboard.

Update the stage test case with the new option 'x11-keymap'.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-06-29 17:57:18 +02:00
Achilleas Koutsou
0f530b4d18 test/data/stages: tests for new stages
org.osbuild.chmod:
  - chmod the chmod binary to a-x
org.osbuild.nginx.conf
2021-06-28 19:07:42 +02:00
Achilleas Koutsou
64794b20d3 test/util_path: add basic utility function test 2021-06-28 19:07:42 +02:00
Achilleas Koutsou
158d6da415 stages: add org.osbuild.chmod
Add new stage to change the mode bits of files in the tree.
2021-06-28 19:07:42 +02:00
Achilleas Koutsou
a411ba2270 stages: add org.osbuild.nginx.conf
Add new stage for writing an nginx configuration file.
2021-06-28 19:07:42 +02:00
Achilleas Koutsou
901de63fb9 util/path: add path checker helper function
Checks if one path is a child of a second one. Useful for checking if
paths defined in a manifest exist inside the tree.
Optionally checks if the target path exists.
2021-06-28 19:07:42 +02:00
Christian Kellner
704d5d305a buildroot: mount /sys as read-only
This will prevent any modification of anything in `/sys`. It will
also prevent `udevadm tigger` to run, which needs /sys writeable.
This is a desired effect, since uevents are not delivered to the
contained environment, so `udevadm trigger` might hang.
2021-06-28 13:39:25 +01:00
Tomas Hozza
5dc72038ef stages/chrony: extend chrony stage to allow additional directives
The `org.osbuild.chrony` stage currently supports only a single option
'timeservers' which does not allow specifying additional options for the
configured timeservers. The option can not be easily extended to allow
specifying additional options and at the same time keep the backward
compatibility with old manifests.

The need for a lower-level stage option allowing to configure additional
options of the 'server' directive is required by RHEL AMI images, which
use 'maxpoll', 'minpoll' and 'prefered' options.

Extend the `org.osbuild.chrony` stage with two additional options. The
'servers' option accepts a list of dictionaries specifying timeservers
to be configured using the 'server' directive, including a subset of the
directive options. The 'servers' option can not be used at the same time
in the stage options as the 'timeservers' option.

The second added option is 'leapsectz' corresponding with a directive of
the same name. The value of the option is a string. If the provided
string is empty, then all occurrences of the 'leapsectz' directive are
removed from the `chrony.conf`. Otherwise, the 'leapsectz' directive is
added to the `chrony.conf` with the provided value, while all original
occurrences of the option are removed.

Add a new stage test case under `test/data/stages/chrony-servers` to
test the new variant of the stage options. The reason is that the
'timeservers' option conflicts with 'servers' option, which makes it
impossible to test both of them in the same test case.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-06-26 10:28:03 +01:00
Tomas Hozza
e96366fb10 stages/cloud-init: add new stage for configuring cloud-init
Add new `org.osbuild.cloud-init` stage, which currently allows to create
configuration files for cloud-init under `/etc/cloud/cloud.cfg.d`. The
stage supports only a very limited subset of cloud-init configuration
options, which is covering needs of RHEL AMI images.

The schema mandates that if the 'configuration_files' option is
specified, then at least one configuration file must be defined. In
addition each section of the configuration must contain at least one
property (section or configuration option).

Add `python3-pyyaml` package to the `F34-build` testing manifest,
because it is required for running and testing the new stage.
Regenerate all affected manifests.

Add test for the new stage.

Update the `osbuild-ci` container image used for testing to a new tag,
which includes python3-pyyaml, the dependency of the new stage.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-06-24 16:55:18 +02:00
Tomas Hozza
751fd5cd28 stages/dracut.conf: add new stage for creating dracut config files
Add new `org.osbuild.dracut.conf` stage. The option
'configuration_files' allows to create dracut configuration files under
`/usr/lib/dracut/dracut.conf.d` and thus make the
configuration persistent. The stage supports only a subset of all
configuration options allowed in dracut configuration. The intention is
to provide almost functional parity with the options supported by
`org.osbuild.dracut` stage.

The schema mandates that at least one configuration file must be defined
in the stage options. In addition, each configuration file must contain
at least one configuration option.

Add test for the new stage.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-06-24 12:43:00 +02:00