Commit graph

1766 commits

Author SHA1 Message Date
Christian Kellner
5828da8d62 test/data: use ostree.deployment in fedora image 2021-10-30 15:32:44 +01:00
Christian Kellner
8b5fe600c1 mounts: add new ostree.deployment service
Create a new OSTree deployment mount service that will set up bind
mounts inside the tree very much as it is done by OSTree in early
boot. This allows any stage to transparently work with OSTree
deployments.
2021-10-30 15:32:44 +01:00
Christian Kellner
a27b3f78bc mounts: include tree directory in arguments
Include the path to the tree for all mount services, so that future
services can e.g. bind mount something inside the tree.
2021-10-30 15:32:44 +01:00
Christian Kellner
f50146d3c5 mounts: allow empty returns from service
Allow mount services to return None, which means they have not 
actually mounted anything within the mount root. This might be
because they have bind mounted directories within the tree.
These mounts do not need any path translation.
2021-10-30 15:32:44 +01:00
Christian Kellner
08c1fbad4b mounts: separate file system mount service
Separate the current `MountService` into the more generic base mount
service and a specialized one for file systems.
2021-10-30 15:32:44 +01:00
Christian Kellner
7e776a0763 meta: allow mounts for all stages
If a stage has not itself defined the `mounts` property, allow any
mounts. This is in preparation to support specialized mounts, such
as bind mounts or ostree deployment mounts to transparently work
with any stage.
NB: devices are not allowed so this will not be applicable for the
current filesystem mounts.
2021-10-30 15:32:44 +01:00
Christian Kellner
5b1cd2b1c5 schema/v2: make mount source and target optional
The previous commit gave the individual mounts more control over the
source and target properties. Do not require them at the global
schema but hand the control if they are optional over to the modules.
2021-10-30 15:32:44 +01:00
Christian Kellner
02404ced94 mounts: change schema meta information
Define the mount schema in the actual mounts at a higher level. This
is in preparation to give the modules more control over the `source`
and `target` properties.
2021-10-30 15:32:44 +01:00
Christian Kellner
5694743ca6 mounts: introduce new mount manager class
Introduce a new specialized service manager class `MountManager` to
manage mounts. It uses the newly introduced `DeviceManager` to look
up devices and stores the reference to the mount point root path.
See the commit that introduced the `DeviceManager` for more info.
2021-10-30 15:32:44 +01:00
Christian Kellner
2447172125 devices: add device path helper functions
Add new helper functions that can translate from a managed device
to its path. One is relative and one is the absolute path on the
host, i.e. to the device node on the host.
2021-10-30 15:32:44 +01:00
Christian Kellner
a5e07cf506 devices: introduce new device manager class
Introduce a new class to manage devices, `DeviceManger` and move the
code to open devices from the `Device` here. The main insight of why
the logic should be place here is that certain information is needed
to open the devices, independently of specific type: the path to the
device node directory, `devpath`, the actual `tree` and the service
manager instance to start the actual service. Instead of passing all
this information again and again to the `Device` class, we now have
a specialized (service) manager class for devices that has all the
needed information all the time. Additionally, the special handling
of parent devices is moved from the pipeline to the service manager,
which is where it belongs.
This will make even more sense for mounts, where the `DeviceManger`
can then be passed to access the individual devices.
Port the test to use the `DeviceManager`.
2021-10-30 15:32:44 +01:00
Christian Kellner
94d8fb1ff7 test/data: add RHEL 7 manifests
Add RHEL 7.9 example manifests. Add them to a `rhel` sub-directory in
the test/data/manifests directory since we cannot re-generate them
in the normal github actions, because they require access to RHEL
content.
2021-10-30 00:54:09 +01:00
Christian Kellner
074741cc27 test/stages: add check for parted stage
Include a new test that writes a partition table to a disk and
then reads it back via `sfdisk` compares it against an layout
that was generated.
NB: This test needs `sfdisk` with `--json` support on the use host.
2021-10-30 00:54:09 +01:00
Christian Kellner
f951a4931e runners: add rhel7 runner
Create a runner for RHEL7. The one thing to note is that RHEL 7
makes use of ld.so.confd snippets and one important for us is
to include `/usr/lib64/iscsi` needed by qemu-img. Otherwise this
is a fairly simple and straight forward runner.
2021-10-30 00:54:09 +01:00
Christian Kellner
f16b606716 stages/grub2.legacy: new stage for non-bls config
Configure grub2 but instead of using the Bootloader Specification (BLS)
it uses traditional menu entries for the individual boot entries. This
is needed since RHEL7 does not have grub2 with BLS support.
2021-10-30 00:54:09 +01:00
Christian Kellner
da5150084e stages/parted: new stage to partition a device
Like the existing sfdisk stage, the parted stage can be used to create
a partition table on the specified device. In contrast to sfdisk, it
does not support uuids, neither for specifying the partition types,
nor the actual uuid of the partition. The current implementation only
supports GPT.
This stage is meant to be used on older systems, like RHEL 7, where
sfdisk exists but does not support GPT (or --json).
2021-10-30 00:54:09 +01:00
Christian Kellner
4a7e49c20b pipeline: don't bind-mount /boot from the host
When setting up the build root, only bind mount the `/boot` dir
from the supplied build tree, if the build tree is not the host
itself, since we never want to leak any host specific data and
the `/boot` directory should never be needed when building the
build root. The only reason `/boot` is mounted at all is for
the grub2 stage to copy efi binaries to the tree since they
directly installed to `/boot` by the respective bootloader
packages.
2021-10-30 00:16:03 +01:00
Christian Kellner
ccb26806fc buildroot: make mounting /boot optional
Currently, we take to paths from the root file system supplied
to the `BuildRoot` class: `/boot` and `/usr`. The reason for
mounting `/boot` is that grub2 and shim install efi binaries
there and for certain images we want to copy the binaries from
the build root and not install the respective packages.
However, if we build to build root itself, we probably don't
want the mount the hosts' `/boot` since we don't want to copy
anything from there. This change should give us the ability to
do exactly that.
2021-10-30 00:16:03 +01:00
Christian Kellner
d8a8dc84d6 setup.cfg: increase max-statements to 75
The default is 50, which we hit in a few places already. Let's do
75, equally arbitrary but gives us a bit more wiggle room.
2021-10-30 00:16:03 +01:00
Christian Kellner
de7fd7b402 runners: add new centos9 runner
Actually, rename the rhel90 runner to the centos9 runner, and
make the former a link to the latter, since in RHEL 9, CentOS
is the upstream and RHEL the downstream.
2021-10-30 00:14:16 +01:00
Ondřej Budai
a39d5af339 ci: remove 8.5 nightly testing
It no longer makes sense because:

- we don't make any changes to 8.5
- we don't regenerate test manifests for 8.5
- osbuild-composer for 8.5 is in the rhel-8.5.0 branch

Also, the latest-8.5.0 symlink was removed, which broke the CI.
2021-10-26 12:39:35 +02:00
Christian Kellner
c42c19356c mpp: fix long options
The correct way to specify long options is as separate arguments,
not as one argument separated by comma.
2021-10-25 18:01:25 +02:00
Alexander Larsson
611b6846e8 osbuild-mpp: Set the "arch" variable to the current rpm arch 2021-10-25 15:17:04 +02:00
Alexander Larsson
1425f66449 osbuild-mpp: Better handling of variable defaults and propagation
We now allow there to be default variables (none so far) which gets
installed as the initial value of manifest.vars.

Additionally, when including a file the default comes from the current
state of the including manifest, allowing the included manifest to use
variables from the base file.
2021-10-25 15:17:04 +02:00
Alexander Larsson
41db488aaa osbuild-mpp: Better handling of -D overrides
We expand variables in various places during load, like when importing
files, etc. This means that the current approach of merging the
overrides into `manifest.vars` at the end doesn't work.

Instead we track overrides completely separate, and when we do the
expansion that always replaces whatever is in `manifest.vars`.
2021-10-25 15:17:04 +02:00
Alexander Larsson
2d16f1243b osbuild-mpp: Allow using formating in depsolver node
This is very useful so that mpp-join can create a merged
package list.
2021-10-25 15:17:04 +02:00
Alexander Larsson
cedb2426c7 osbuild-mpp: Add mpp-join
This allows you to join a set of lists, its very nice to use
e.g. when constructing the package-list for a depsolve node.
2021-10-25 15:17:04 +02:00
Alexander Larsson
01fbc8c136 osbuild-mpp: Add mpp-eval
This:
 {"mpp-eval": "foo+bar"}

Is essentially the same as using mpp-format with a trivial format
string:

 {"mpp-format-int": "{foo+bar}"}

However, it is less to type, clearer to read, and supports
returning more complex types. For example, you can have a variable
that is a dict and expand that using eval.
2021-10-25 15:17:04 +02:00
Simon Steinbeiss
a33f1a975b Fix GitHub Action tag pattern
GitHub Actions currently doesn't support regular expressions, but
instead only "filter patterns".
https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#filter-pattern-cheat-sheet
2021-10-25 15:13:03 +02:00
Ondřej Budai
8cd9fd79c6 mockbuild: reuse mock repos from the system ones
There's no need to define the repositories twice, let's just reuse the system
repository.

This change was already successfully applied in composer.
2021-10-21 17:48:08 +02:00
Ondřej Budai
b02dc8a7ab mockbuild: rotate a variable name
The previous one is no longer defined.
2021-10-21 17:48:08 +02:00
Ondřej Budai
74d1806ab0 gitlab: don't save journal 2021-10-21 17:48:08 +02:00
Ondřej Budai
f01d522dbf deploy: update to the latest composer commit
We rotated some variable names in composer, thus we need to adjust..
2021-10-21 17:48:08 +02:00
Ondřej Budai
3e402f126c ci: don't register the runners
They all subscribed before deploy.sh is even run.
2021-10-21 17:48:08 +02:00
Ondřej Budai
2f1b5d72dd trigger-gitlab: do not interpret the fetch_pulls outputs
Here documents are by default interpreted. Quoting the delimiter prevents
that.

See https://stackoverflow.com/questions/27920806/how-to-avoid-heredoc-expanding-variables
2021-10-21 11:35:08 +02:00
Simon Steinbeiss
4ff59e2086 Bump version numbers ahead of release
This is necessary for the new simplified release process and is done
ahead of time once for the upcoming release now.
After osbuild 40 this will be done by the GitHub composite action.
2021-10-19 11:21:54 +02:00
Simon Steinbeiss
4ea2915ef7 Switch to simple upstream releases
This commit changes our release process from the model of having a
release commit (and pull request) which also updated the NEWS.md file
and bumped the versions in the osbuild.spec and setup.py files to simply
pushing a tag.

After the tag (containing the release notes) is pushed, a GitHub
composite action is triggered that creates a GitHub release with the
contents of the git release tag. Furthermore the bumping of the version
number now always has to happen directly after a release to avoid having
to push a(n untested) commit to main for the release and this is also
handled by the GitHub composite action.

Finally packit pushes directly to dist-git now on pushing the release
tag, so no pull-request needs to be reviewed and merged anymore.
2021-10-19 11:21:54 +02:00
Christian Kellner
d76db390c8 stages/grub2.inst: ensure /var/tmp exists
Ensure that `/var/tmp` exists in the build root before trying to
create a file there. It seems to not be there on RHEL 7 build
roots.
2021-10-13 17:10:09 +02:00
Christian Kellner
bf5a7cdbb0 devices/loopback: remove extra "'" from print
There is an extra `'` character in the print messages that shows
which file we are using.
2021-10-13 17:10:09 +02:00
Jakub Rusz
221381b2de tests/ci: Switch to testing on 8.4 GA
Updating terraform sha and switching to 8.4 GA runners.
Also updating DISTRO_CODE for each runner.
2021-10-12 13:29:37 +02:00
Jakub Rusz
b065a456d6 tests: enable testing on RHEl-8.5 and RHEL-9.0
Also update osbuild-composer commit to use newer manifests.
2021-10-12 13:29:37 +02:00
Sanne Raymaekers
79f842d4e7 .github: Write PR data to a file first in trigger-gitlab
Using echo breaks if any PR body contains a `'` character.
2021-10-07 10:07:41 +02:00
Simon Steinbeiss
11dea50050 README: Add a link to our developer guide 2021-10-06 20:03:48 +01:00
Thomas Lavocat
00125e9fdb 39
Release osbuild 39

Signed-off-by: Thomas Lavocat <tlavocat@redhat.com>
2021-10-06 14:00:21 +02:00
Tomas Hozza
2e8ef3eadd packit: enable builds on ppc64le
Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-10-06 08:24:33 +02:00
Achilleas Koutsou
8076f2beb2 CI: rename rhel-8.5 runners to rhel-8.5-nightly
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-10-05 15:57:45 +02:00
Achilleas Koutsou
3de57ae9a7 mockbuild: add RHEL 8.6
- CI runners
- Subscription override for mock templates
- Terraform SHA for 8.6 runners

Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-10-05 15:57:45 +02:00
Christian Kellner
03411015a2 docs: document osbuild and selinux integration
Add a first outline of the integration points of osbuild and
selinux, from two perspectives: how osbuild interacts with
the host and how osbuild labels the target artefact and the
resulting peculiarties.
2021-10-01 11:02:32 +02:00
Christian Kellner
ebb7a982f8 ci: trigger gitlab from checks not tests
Use the "Checks" workflow to trigger gitlab; this workflow should
be much quicker to complete and thus the gitlab ci will trigger
earlier leading to a more parallel ci run.
2021-09-29 20:07:59 +02:00
Christian Kellner
ead48226fc ci: split out checks from tests
Split out all checks that verify documentation, spelling and that
test data is update to date from the actual unit and integration
tests.
2021-09-29 20:07:59 +02:00