A usual step in creating OS file system trees is to apply the
correct SELinux labels for all files and directories. This is
done by the org.osbuild.selinux stage, which internally uses the
setfiles command in order to do so. The SELiunx policy to be
used for this operation is the one of the newly created system,
not the host one. It therefore can contain labels that are not
known on the host. The kernel will prevent setting invalid,
i.e. unknown, labels unless the caller has the CAP_MAC_ADMIN
capability. By default, setfiles is executed in the setfiles_t
domain, where it lacks that capability. Therefore a custom
osbuild SELinux policy was created, with a special transition
rule that will execute setfiles in the setfiles_mac_t
domain. All stages, sources and assemblers as well as the main
binary are label with the new osbuild_exec_t label.
Additionally, allow a transition from osbuild_t to install_t by
using `anaconda_domtrans_install`, so that ostree and
rpm-ostree, which are labeled as install_exec_t, can transition
to the install_t domain when called form osbuild. Update the
spec file to build the policy and include it in a new
osbuild-selinux sub-package.
In python 3.6 the value of `__origin__` for typing.List[str] is
typing.List. This then changed to the actual `list` type in later
versions. Accept both versions.
The `capture_output` argument for subprocess.run was added in 3.7,
but want to support 3.6 as well. Change all the usages of it with
`stdout=subprocess.PIPE` that will have the same effect, at least
for stdout.
The retry logic was meant to work around issues where a round-robin
redirect of mirrors gave us random mirrors of varying quality. This was
not used in practice, rather fixed mirrors were always used (either
hard-coded as basurl, or resolved from metalink).
The retry logic meant that when we did hit very slow mirrors we would
time-out and retry, potentially failing altogether, even though the data
was coming. Each retry would not help, as the mirror was anyway the
same. As a result our CI gave us avoidable false negative test results
some of the time.
The proper solution to this is to gain support for librepo and metalinks
to adopt the same retry logic that dnf uses.
For now, improve on the retry logic by retrying until a max total time,
rather than an increasing timeout on each try. Up the given timeouts to
be one minute to connect and five minutes to complete the download. This
avoids hanging forever if the mirror is truly broken, but still gives
more time to finish the download than each iteration in the old code
did.
There are no new tests for this, as before this change the tests mostly
passed, and after it they will hopefully still mostly pass (but more
often).
Signed-off-by: Tom Gundersen <teg@jklm.no>
This replaces the round-robin mirror at fedoraproject.org, as that was
proving to be quite unreliable.
This is a short-term fix before add metalink support.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Move the last remaining test into the correct subdir. With this done,
all our tests run in one of the 3 groups:
* `make test-src`
Run tests against the source-code, including linters.
* `make test-mod`
Run unit-tests on the individual python modules. This needs no
special permissions (unless noted in each test) or runtime
environments. It is meant to be fast and easy to run in all
circumstances.
* `make test-run`
Run tests that execute the osbuild pipeline. This requires
superuser privileges and will likely take a while. Furthermore,
this might produce large artifacts.
Align the makefile targets with the test-targets (`module` -> `mod`,
etc.). This way, we have consistent names everywhere.
While at it, move the `make test-run` invocation closer to the others.
Move the stage-tests over to the new test-infrastructure. This moves
the test invocation into `./test/run/test_stages.py`, so it is invoked
as part of the runtime-tests. Secondly, the test-data is stored in
./test/data/stages/ so the path is relative to
TestBase.locate_test_data().
While at it, this also drops the dynamic class modifications and instead
uses subTest(). This simplifies the code quite a bit and avoids
dynamically creating python code.
Move the `test_osbuild.py` test into the module-test directory. This
test contains just a bunch of basic functionality tests for a selection
of osbuild modules. Hence, it can be run together with the other module
tests.
Move `test_objectstore` into the module-level tests. This allows us to
run it as part of `make test-module.
Make sure to properly guard it as root-only module.
This moves the `sources_tests` into ./test/data/sources/ and makes the
entire test use `locate_test_data()` to get relative paths for their
accesses.
This further improves our test cases to support running from
installments rather than local checkouts. We need access to ./test/data
guarded, so we can install packages and still have the tests access the
correct paths.
This also adjusts the HTTP-Server we use in the test to serve data
relative to a path it is handed. I now chose `./test/data`, which will
easily allow us to re-use the same HTTP-Server in the future for other
tests that require it.
We want to extend our base-class to support extensions to
unittest.TestCase, so make sure we inherit from it.
Adjust all callers to no longer inherit from TestCase, since this is now
done automatically by TestBase.
Move the `tree-diff` tool into ./tools, which is our new place for tools
used by the test-suite or during development.
The only hard-coded user is the TestBase, so fix its path to the tool
so the test-suite will continue to find it.
Run the MPP tools in the CI and verify the committed test-data did not
change and is up-to-date.
This runs `make test-data` and then simply uses `git diff --exit-code`
to trigger a CI failure if there are any differences in ./test/data.
This adds a new MPP which supports importing pipelines from another
file. It simply looks for "mpp-import-pipeline" tags at the same
position where we would expect a "pipeline" tag. It then uses the "path"
attribute in it to find a manifest. From this manifest, the "sources"
are merged back into the original sources, and the "pipeline" is taken
verbatim to replace the "mpp-import-pipeline".
The idea is to allow importing build-pipelines from other files into our
test manifests, without duplicating the build-pipeline everywhere.
Add a new trivial runtime-test which simply runs a no-op pipeline. This
is a fast, trivial test that simply verifies osbuild is properly setup
and accessible.
Remove the explicit no-op test from the CI, now that the test-suite has
it as well.
Major changes are the support for ignition, that assembler results
wont be automatically committed anymore, a new org.osbuild.copy
stage, the removal of the `tree_id` & `output_id` as well as full
manifest validation due to the addition of source schema validation.
Recently downloading rpms from the fau.de mirror is failing quite
often with timeouts. Replace its only usage in the rpm stages
test with the kernel.org mirror.
Add a new stage option `initramfs-args`, that will in turn be set
on the Treefile that is then passed to rpm-ostree. This will pass
the options on to dracut when building the initramfs module. NB:
the `ostree` dracut module will also always be automatically
included by rpm-ostree.
Add the initramfs-args Treefile option that can be used to pass
arguments to drauct via rpm-ostree. NB: the ostree module will
always be automatically be included by rpm-ostree.
This stage will create a file '/boot/ignition.firstboot' that will,
with the help of support in grub, trigger ignition on the first
boot. The `network` option can be used to overwrite the default
network configuration set in grub2.
Add support for ignition[1] via a new `ignition` stage option. If
enabled, a new section is added to the main grub.cfg that will
create a 'ignition_firstboot' variable meant to be included in the
kernel command line configuration.
The grub.cfg snippet was taken from 'src/grub.cfg' of Fedora CoreOS
Assembler[2] at ec05cde20d3449fab8e4c76493ffa1ebd9b0b626 but with
PR #1373 applied to not hard-code the dhcp options.
[1] https://github.com/coreos/ignition
[2] https://github.com/coreos/coreos-assembler/
When adding a new user with a uid that already exists, the "-o",
option needs to be added. Always do so when the uid is specified.
A use case for this is e.g. an installer iso with a `install`
user that has 0 (like root) for its uid (and gid).
Support setting uids, gids with values of `0` as well as passwords
and descriptions with the empty string, by explicitly checking the
value of each against `None`, because simple `if` conditionals are
false for those.
The content hash of each RPM is already verified, so verifying
signatures again is not necessary if the manifest generation is trusted,
and verifying signatures does not help if the manifest generation is
not.
Let us follow what DNF does and default to not verify signatures, but in
order to preserve features already in use we still allow opting in to
verifying signatures as before on a per RPM basis.
This will make it possible to install unsigned RPMs, or a mixed of
signed and unsigned RPMs.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Add support for querying information about sources: add the mapping
from name to directory and accept "Source" as a module name. Adapt
the ModuleInfo schema property to handle the different styles for
stage-like schemata as well as sources now.
Extract the grub.cfg redirect config as GRUB_REDIRECT_TEMPLATE,
meant to be used via python's string.Template class. Document
its intended use and also the template options.
Extract the strings for grub.cfg into a GRUB_CFG_TEMPLATE multi-line
string and turn it into a template meant to be used via python's
string.Template class. Document it, especially the template options.
Instead of having two different places within conditionals where
the configuration is written, have only one common. In the case
of hybrid boot, in the end there will be two grub configuration
files: the canonical one in /boot/grub2/grub.cfg and a redirect
one, in the EFI directory that will redirect to the canonical.
In case of legacy only, only the canonical one in the default
location (/boot/grub2/grub.cfg) will be written.
For EFI only mode, only /boot/efi/EFI/<vendor>/grub.cfg will
be written with the main grub configuration data.
Thus the writing of the main grub configuration will now always
be written in exactly one place (now line 319).
Move the write_grub_cfg_redirect to the new GrubConfig object as
write_redirect. Add a `separate_boot` property to be used by the
new write_redirect. Remove the corresponding variable and also
the `grub_fs` variable since that is now all handled by the
GrubConfig object.
The reason behind this is to combine all the necessary state in the
object instead of passing it all to the write_grub_cfg function.
The idea is that as more things will get configurable, say the
timeout or ignition support, more things need to be passed to it
and thus it is better to an object where these config options can
be set and then combined when writing the config.
Fedora CoreOS[1] uses a dracut module[2] together with a systemd
generator[3] to mount the file system, including the root one.
Thus neither '/etc/fstab' nor a `root=` kernel command line
option is needed. Support that use case by making the 'rootfs'
option optional.
[1] https://github.com/coreos/fedora-coreos-config/tree/testing-devel/
[2] overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree
[3] overlay.d/05core/usr/lib/systemd/system-generators/coreos-boot-mount-generator
The ignition-dracut module for Fedora CoreOS and anaconda both have
code to populate '/var' via systemd-tmpfiles. In images that where
said dracut module is not used, but '/var' needs to be populate, it
can no be done by setting the `populate_var` option.
We currently run pylint on all files we find in the checkout. This is
particularly annoying when we have osbuild-stores in the checkout, which
then contain lots of python files in their stored trees.
Change the pylint test to use `git ls-tree` to find all files in the
index and then only run pylint against the subset that we are interested
in.
