Commit graph

1527 commits

Author SHA1 Message Date
Christian Kellner
9a33fabd51 assembler/ostree.commit: fix copying of links
Commit 92cc269 fixed a bug where `/var` was copied into `/var`
resulting in `/var/var`. Sadly the fix broke copying links,
like `bin -> usr/bin`, where now the content of the link would
be copied but not the link itself. Use the `-t` command line
flag for `cp` which should ensure that we copy links as links
but also copy the contents for `/var` should the target dir,
i.e. `/var` already exist.
2021-07-16 10:32:12 +02:00
Christian Kellner
e8bf211e0c test/data: move ostree repo to the iso root
Instead of having the OSTree repo in the anaconda root squashfs,
put the it to the root of the iso. This has several advantages:
first and foremost, we don't have to make a huge squahfs file-
system. We don't have to compress the repo. Additionally, the
repo is now easily accessible by mounting the iso.
2021-07-15 23:08:51 +01:00
Christian Kellner
a19e261dbe test/data: move kickstart to the iso root
Intead of overwriting the standard ananconda kickstart file, we
move it to the root of the iso. To have anaconda pick it up an
additional kernel command line parameter is required.
2021-07-15 23:08:51 +01:00
Christian Kellner
c77170a924 test/data: create core user via kickstart
Create the users via the new users support that was added to the
kickstart stage, instead of having them in the OSTree commit.
2021-07-15 23:08:51 +01:00
Christian Kellner
9d3c420a38 test/data: update x11 pkgs for ostree-bootiso
F34+ has the xorg-x11-server-utils package split up. Install the
individual new packages we need for anaconda.

See Anaconda commit 55371c996861a47da9504b31118858dd3257a8b0
2021-07-15 23:08:51 +01:00
Christian Kellner
887e1bd8aa stages/kickstart: support for users and groups
Add support for creating groups and users during the installation
via the corresponding kickstart directives.
2021-07-15 23:08:51 +01:00
Christian Kellner
403dd5e2ec tools/lorax-templates-pkg: set executable bit 2021-07-15 23:08:51 +01:00
Ondřej Budai
19c4b43952 spec: make format of file modes consistent 2021-07-15 16:23:01 +01:00
Ondřej Budai
e9da5ccdc2 spec: make json schemas non-executable
rpmbuild actually complains about it:

*** WARNING: ./usr/share/osbuild/schemas/osbuild1.json is executable but has no shebang, removing executable bit
*** WARNING: ./usr/share/osbuild/schemas/osbuild2.json is executable but has no shebang, removing executable bit
2021-07-15 16:23:01 +01:00
Tomas Hozza
7617cb9a44 stages/authselect: rename profile_id option to profile
Rename the `profile_id` option to `profile` in the stage schema. This is
a follow-up to PR#696 comment after is has been merged [1].

[1] https://github.com/osbuild/osbuild/pull/696#discussion_r661253293

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2021-07-15 17:17:50 +02:00
Christian Kellner
cc39d5a8b4 test/data: use more variables in ostree-bootiso
Use variables for various often occuring contants, like the
release, ref and isolabel. Additionally, use variables for
the snapshot so it can be changed easily.
2021-07-14 23:51:34 +02:00
Christian Kellner
a9b3fbd92b test/data: auto-format ostree-bootiso
Re-indent and format the bootiso mpp. No actual content changes.
2021-07-14 23:51:34 +02:00
Christian Kellner
c4db24c481 tools/mpp: substitute vars in mpp blocks
Allow the manifest variables, defined via mpp-vars, to be used from
within the mpp blocks. For this template strings are used, where
variables are marked via `$`. We cannot use the `mpp-format` logic
easily there, since that is processed after other mpp directives
have been processed.
As a result remove the built-in substitution from support from dnf
dep-solving, since we had to post-process the resulting urls with
variable substitution afterwards. Now that is covered with this
more generic mechanism.
2021-07-14 23:51:34 +02:00
Christian Kellner
4a00895ba5 tools/mpp: sort import
Makes David happy, which makes us happy.
2021-07-14 23:51:34 +02:00
Christian Kellner
fd1b688f78 tools/mpp: single mpp node retrieval method
Introduce a helper method that will return the mpp node of a given
name or `None` if it does not exist. Delete it from the parent
dictionary if it did.

Reorder the typing imports as well to be sorted below the non-
`from` imports.
2021-07-14 23:51:34 +02:00
Christian Kellner
c774e76079 test/data: remove duplicated baseurl entries
We don't need to define them twice, remove them everwhere where
that was the case.
2021-07-14 23:51:34 +02:00
Christian Kellner
fdb0faca86 tools/mpp: extract vars initialization
Move the initialization for manifest vars out of `__init__`.
2021-07-14 23:51:34 +02:00
Ondřej Budai
4c3c7d7851 ci: enable tests on Fedora aarch64
because why not (:

RHEL might be more difficult, let's revisit it later.
2021-07-14 23:51:05 +02:00
Ondřej Budai
ea7eee8db6 ci: update composer to make tests more robust
f8fd3d04 switched F33 image tests to use rpmrepo rather than kernel mirrors
that are sometimes desynced. This should make the testsuite more stable.
2021-07-14 23:51:05 +02:00
Javier Martinez Canillas
6901c4dfc5 test/data: use UTF-8 locale on all manifests
The manifests set a "en_US" locale but this causes gnome-terminal to not
run due a non UTF-8 locale being used, which is an unsupported config:

gnome-terminal-server[1899]: Non UTF-8 locale (ISO-8859-1) is not supported!

Reported-by: Stephen Smoogen <ssmoogen@redhat.com>
2021-07-14 23:50:24 +02:00
Javier Martinez Canillas
2bc0088767 README: improve Build section
The current Build section is misleading, since running the commands that
are mentioned there will lead to only the osbuild module and osbuild-mpp
tool to be installed.

But none of the other required artifacts (sources, stages, schemas, etc)
will, causing the installed osbuild to not work at all. Instead, have an
Install section that explains how osbuild can be installed from RPMs.
2021-07-14 14:37:03 +02:00
Javier Martinez Canillas
897d702d2e docs: add --export option to the osbuild man page
This is used to export an image but isn't present in the osbuild man page.
2021-07-14 14:35:30 +02:00
Ondřej Budai
8396e73372 ci: upload rpms built in RHEL 8 CDN buildroot into rhel-8-cdn directory
Let's explain how RPMs for RHEL are built:

We use a subscribed RHEL 8.x machine and mock build these on it. Mock
initializes its own buildroot based on the latest RHEL 8 CDN content, see[1].
This means that the minor version of the buildroot is independent of the minor
version of the host.

However, we currently upload RPMs to a directory whose name consists also of
the minor version of the host. Our hosts are currently running RHEL 8.3 so
the RPMs are uploaded into rhel-8.3 directory despite them being built in the
RHEL 8.4 buildroot (RHEL 8 CDN buildroot specifically). This means that
we cannot guarantee that they are installable on RHEL 8.3 which is weird.

This commit adds a special case for hosts that run on subscribed RHEL and
thus build RPMs in a buildroot constructed from RHEL CDN. These RPMs are
now uploaded into rhel-8-cdn directory. This change more accurately reflects
the way we build our RPMs and removes some confusion.

[1]: https://github.com/rpm-software-management/mock/blob/main/mock-core-configs/etc/mock/templates/rhel-8.tpl#L37
2021-07-12 19:02:21 +02:00
Ondřej Budai
41e01326f0 mockbuild: register the host as soon as possible
So we can check if we are subscribed in the variable definitions.

I believe that registering the instance will go away soon nevertheless.
2021-07-12 19:02:21 +02:00
Ondřej Budai
54f233ac96 mockbuild: use REPO_BUCKET when defining the repo's base URL
Just deduplication of code.
2021-07-12 19:02:21 +02:00
Christian Kellner
e92b40952b samples: replace with symlink to test data
Instead of maintaining a separate set of samples that by now are very
much outdated (using Fedora 31 or older), make the samples directory
a symlink to the test data. Manifests in there are indeed tested and
maintained.
The error and noop samples are also covered in unit tests, so no need
for extra samples there either.

CI: remove the sample validation since all the test data manifests
    are actually built.
2021-07-12 18:44:50 +02:00
Christian Kellner
d09f07a24c test/data: remove users from ostree-image
Let's not have users in the ostree commit itself since we have
zezere ignition enabled for user provisioning.
2021-07-12 18:21:05 +02:00
Christian Kellner
284e73a018 test/data: enable greenboot and zezere ignition
Enable greenboot for health checks and zezere ignition so we get
user provisioning like Fedora IoT. Also explicitly opt into rngd
and sshd.
2021-07-12 18:21:05 +02:00
Christian Kellner
75e3856967 test/data: use new ostree stages
Use the new ostree stages instead of the old monolithic one.
2021-07-12 18:21:05 +02:00
Christian Kellner
0d625f34ea stages/fstab: add support for OSTree
Add experimental support for writing the fstab file to `/etc` in on
OSTree deployment. Experimental here means that options might be
replaced in the near future with a different mechanism and are thus
not considered stable API.
2021-07-12 18:21:05 +02:00
Christian Kellner
3d197247b2 stages: add org.osbuild.ostree.selinux
Fix SELinux labels for an OSTree deployment.
2021-07-12 18:21:05 +02:00
Christian Kellner
467f313cee stages: add org.osbuild.ostree.remotes
Configure OSTree remotes for a repository.
2021-07-12 18:21:05 +02:00
Christian Kellner
98d0a856df stages: add org.osbuild.ostree.fillvar
Pre-populate /var directory for a given deployment.
2021-07-12 18:21:05 +02:00
Christian Kellner
d284bc0ef2 stages: add org.osbuild.ostree.deploy
Create an OSTree deployment[1] for a given ref.
2021-07-12 18:21:05 +02:00
Christian Kellner
d793ffd805 stages: add org.osbuild.ostree.config
Change OSTree configuration for a repository via `ostree config`.
2021-07-12 18:21:05 +02:00
Christian Kellner
ee124df336 stages: add org.osbuild.ostree.init-os
Initializes a new stateroot (see [1]) for the OS with the
name `osname`.
2021-07-12 18:21:05 +02:00
Christian Kellner
784d29218f stages: add org.osbuild.ostree.init-fs
Creates the basic file system layout for an OSTree based system.
2021-07-12 18:21:05 +02:00
Christian Kellner
54fe67ece5 utils/ostree: deployment path helper
Add a simple helper method that returns the path for a deployment,
given the sysroot, the osname, the reference or commit and the
deployment serial. Path might not exist.
`
2021-07-12 18:21:05 +02:00
Christian Kellner
dc201d45fd utils/ostree: add rev_parse helper function
Add a simple helper function that wraps `ostree rev-parse` to make
it easy to resolve an OSTree reference given a repository.
2021-07-12 18:21:05 +02:00
Christian Kellner
368d0a5c18 tools/mpp: support multiple image definitions
Support more than one image definition by introducing a `id` attribute
which can be used to specify how the variable for the definition is
called.
2021-07-09 19:50:38 +01:00
Christian Kellner
042a3424e4 tools/mpp: support skip_if_unavailable
Add `skip_if_unavailable` as an supported attribute
for the repository configuration.
2021-07-09 19:50:38 +01:00
Christian Kellner
c5c5550ad5 stages/qemu: remove mixed indention
Only spaces, no tabs!

Reported-by: Achilleas Koutsou <achilleas@koutsou.net>
2021-07-09 19:50:38 +01:00
Christian Kellner
0c871c26c0 objectstore: use recursive bind mounts
When bind-mounting the tree for i/o, use recursive bind mounts.
This could be needed in the case that `/usr` is not one single
mount but assembled from different ones. Normally this should
not be the case but we want to support in, just in case.
Conversely, when unmounting, do so recursively too.
NB: This should not make any differences for trees that we have
built ourselves since they don't contain any mounts.
2021-07-09 18:09:37 +01:00
Christian Kellner
2b4e913e1e objectstore: only bind-mount /usr for host trees
The only thing we should ever need from the host is `/usr`. Therefore
instead of bind-mounting the entirety that is `/`, just bind-mount
`/usr`.
2021-07-09 18:09:37 +01:00
Christian Kellner
d2c4888843 test/data: use mpp-define-image in ostree-image
Use the new partiton layout support in MPP. NB: start data as
well as the size of the last partition have been omitted now,
since `sfdisk` will figure those out for us.
2021-07-08 22:47:45 +02:00
Christian Kellner
d39e3a239e tools/mpp: support for defining partitions
It is now possible to define a partition layout via `mpp-define-image`.
The defined layout is actually written to a temporary sparse file and
read back via `sfdisk`, so that all partition data like `size` and
`start` include actual padding and such. The `image` variable will be
defined with `size` and `layout` keys, the latter containing the
partition layout data. It can be accessed via the "String expansion"
mechanism.
2021-07-08 22:47:45 +02:00
Christian Kellner
4456e1f276 stages: add org.osbuild.mkdir
Can create one or more directories, optionally also their parents.
Add some basic tests as well.
2021-07-08 21:14:21 +01:00
Christian Kellner
3a69bf39b7 host: catch BrokenPipeError while sending replies
Catch the BrokenPipeError exception when sending a reply. This will
happen when the other side closes their side of the connection/pipe
so in that case we just break out of the serve loop.
2021-07-08 15:01:33 +01:00
Christian Kellner
254c1cd9fb api: remove host side arguments facility
Now that arguments are transmitted via a mapped, i.e. bind-mounted,
file instead of using the jsoncomm RPC mechanism, all the methods
related to the latter can be removed from API.
2021-07-08 15:01:33 +01:00
Christian Kellner
affd384669 pipeline: deliver stage arguments via a file
Instead of using the jsoncomm API to transmit stagge arguments,
write them out to a file that is then mapped into the container.
The `api.arguments` function is re-written just read that file
from within the container.
2021-07-08 15:01:33 +01:00