curl uses strtod from the C standard library to convert the --max-time's value
from string to double. However, this is what strtod expects:
nonempty sequence of decimal digits optionally containing decimal-point
character (as determined by the current C locale)
Yeah, unfortunately, the decimal-point character is determined by the current
C locale. For example, Czech and German locale uses a comma as the
decimal-point character.
For reasons I don't fully understand, Python thinks it's running on en_US
locale, even though LC_NUMERIC is set to cs_CZ, so it uses a full stop as the
decimal-point character when converting float to string. However, as written
before, curl fails to parse this because it expects comma.
The fix I chose is simple: Use math.ceil, so only an integer can be passed to
curl. Why ceil? Because --max-time == 0 sounds fishy. math.ceil should return
an integer (and it does in Python 3.8) but the documentation is not 100% clear
on this topic, so let's be paranoid and also convert it to int after the
ceiling.
It should be possible to use the stage to only set the default
target or disable services. There is no need to always require
having `enabled_services`.
Add the ability to mask services, which is done e.g. when building
installers. See systemctl(1) for more information about masked
services.
Modify the existing stage test to include a test for it.
Add the scaffolding and various targets to run a coverity analysis,
including downloading the tool, running the "build", archiving
the results and submitting it to coverity for analysis.
The downloading and submitting need COVERITY_{EMAIL, TOKEN} to be
defined in the environment.
Co-authored-by: David Rheinsberg <david.rheinsberg@gmail.com>
Runner are invoked to prepare the execution of stages and assemblers
inside the container. The setup tasks are specific to the distribution
and maybe the version of it, therefore specific runners are used for
each distribution+version combination.
The build the first (most nested) build root, `/usr` is taken from the
host to bootstrap the container. On RHEL, the python interpreter to be
used for software that belongs to the platform is platform-python, as
it provides a stable API. Therefore the RHEL runners should use that
instead of relying on the presence of /usr/bin/python3.6, which might
not be installed and is indeed not installed by default.
Add a new `labels` option that can contain `path`: `label` pairs
to overwrite the default labels for `path`.
NB: These manually set labels will not survive a relabeling and
are most useful to adjust policy in the buildroot, e.g. for `cp`
to be able to copy labels unknown to the host, by labeling it
as `system_u:object_r:install_exec_t:s0`.
Add a simple check for the selinux check by building the f32-base
image with an added selinux stage. Use the options from a test
json file and verify the labels against a set of labels given in
the aforementioned test file.
Only generate stage tests for sub-directories in stages_tests
that contain a diff.json. This should allow us to have specialized
stage tests that don't use the current {a, b}.json & diff.json
pattern.
Verify the rpm-ostree.input hash is set correctly for the repository
itself as well. This will in turn also verify that the repository
is existent and can be accessed.
Install rpm-ostree in the (mpp-)f32-build.json build manifest, so
ostree commits can be created using the same build root. Update
all affected manifests.
Support excluding packages when dep-solving; needed for e.g. the
ostree commits to not create a rescue kernel by excluding the
'dracut-config-rescue' package.
Using the network block device (nbd) kernel module to test all
the non-raw image formats often caused tests to fail due to nbd
not being stable itself (see below).
Instead convert non-raw images to the raw format via qemu-img
convert and mount those with loop-back devices. All the testing
code itself stays the same.
Example nbd error messages:
kernel: block nbd15: NBD_DISCONNECT
kernel: block nbd15: Disconnected due to user request.
kernel: print_req_error: 89 callbacks suppressed
kernel: blk_update_request: I/O error, dev nbd15, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
kernel: buffer_io_error: 134 callbacks suppressed
kernel: Buffer I/O error on dev nbd15, logical block 0, async page read
kernel: blk_update_request: I/O error, dev nbd15, sector 1 op 0x0:(READ) flags 0x0 phys_seg 7 prio class 0
Split up the partition table test into reading the partition table
and then asserting it has the correct entries. Prepares the usage
of the partition information later.
Major feature is the new custom SELinux policy to fix labeling.
Important bug fixes to the files source, the rpm stage. Also
restore Python 3.6 support.
This brings it in line with the JSON schema. Also fix a bug where the
wrong root was used for checking the signatures.
Signed-off-by: Tom Gundersen <teg@jklm.no>
When osbuild is invoked via the osbuild composer worker, the latter
is run in the unconfined_service_t domain, so add a rule that
allows that transition.
When applying labels inside the container that are unknown to the
host, the process needs to have the CAP_MAC_ADMIN capability in order
to do so, otherwise the kernel will prevent setting those unknown
labels. See the previous commit for more details.
A usual step in creating OS file system trees is to apply the
correct SELinux labels for all files and directories. This is
done by the org.osbuild.selinux stage, which internally uses the
setfiles command in order to do so. The SELiunx policy to be
used for this operation is the one of the newly created system,
not the host one. It therefore can contain labels that are not
known on the host. The kernel will prevent setting invalid,
i.e. unknown, labels unless the caller has the CAP_MAC_ADMIN
capability. By default, setfiles is executed in the setfiles_t
domain, where it lacks that capability. Therefore a custom
osbuild SELinux policy was created, with a special transition
rule that will execute setfiles in the setfiles_mac_t
domain. All stages, sources and assemblers as well as the main
binary are label with the new osbuild_exec_t label.
Additionally, allow a transition from osbuild_t to install_t by
using `anaconda_domtrans_install`, so that ostree and
rpm-ostree, which are labeled as install_exec_t, can transition
to the install_t domain when called form osbuild. Update the
spec file to build the policy and include it in a new
osbuild-selinux sub-package.
In python 3.6 the value of `__origin__` for typing.List[str] is
typing.List. This then changed to the actual `list` type in later
versions. Accept both versions.
The `capture_output` argument for subprocess.run was added in 3.7,
but want to support 3.6 as well. Change all the usages of it with
`stdout=subprocess.PIPE` that will have the same effect, at least
for stdout.
The retry logic was meant to work around issues where a round-robin
redirect of mirrors gave us random mirrors of varying quality. This was
not used in practice, rather fixed mirrors were always used (either
hard-coded as basurl, or resolved from metalink).
The retry logic meant that when we did hit very slow mirrors we would
time-out and retry, potentially failing altogether, even though the data
was coming. Each retry would not help, as the mirror was anyway the
same. As a result our CI gave us avoidable false negative test results
some of the time.
The proper solution to this is to gain support for librepo and metalinks
to adopt the same retry logic that dnf uses.
For now, improve on the retry logic by retrying until a max total time,
rather than an increasing timeout on each try. Up the given timeouts to
be one minute to connect and five minutes to complete the download. This
avoids hanging forever if the mirror is truly broken, but still gives
more time to finish the download than each iteration in the old code
did.
There are no new tests for this, as before this change the tests mostly
passed, and after it they will hopefully still mostly pass (but more
often).
Signed-off-by: Tom Gundersen <teg@jklm.no>
This replaces the round-robin mirror at fedoraproject.org, as that was
proving to be quite unreliable.
This is a short-term fix before add metalink support.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Move the last remaining test into the correct subdir. With this done,
all our tests run in one of the 3 groups:
* `make test-src`
Run tests against the source-code, including linters.
* `make test-mod`
Run unit-tests on the individual python modules. This needs no
special permissions (unless noted in each test) or runtime
environments. It is meant to be fast and easy to run in all
circumstances.
* `make test-run`
Run tests that execute the osbuild pipeline. This requires
superuser privileges and will likely take a while. Furthermore,
this might produce large artifacts.
Align the makefile targets with the test-targets (`module` -> `mod`,
etc.). This way, we have consistent names everywhere.
While at it, move the `make test-run` invocation closer to the others.
Move the stage-tests over to the new test-infrastructure. This moves
the test invocation into `./test/run/test_stages.py`, so it is invoked
as part of the runtime-tests. Secondly, the test-data is stored in
./test/data/stages/ so the path is relative to
TestBase.locate_test_data().
While at it, this also drops the dynamic class modifications and instead
uses subTest(). This simplifies the code quite a bit and avoids
dynamically creating python code.
Move the `test_osbuild.py` test into the module-test directory. This
test contains just a bunch of basic functionality tests for a selection
of osbuild modules. Hence, it can be run together with the other module
tests.
Move `test_objectstore` into the module-level tests. This allows us to
run it as part of `make test-module.
Make sure to properly guard it as root-only module.
This moves the `sources_tests` into ./test/data/sources/ and makes the
entire test use `locate_test_data()` to get relative paths for their
accesses.
This further improves our test cases to support running from
installments rather than local checkouts. We need access to ./test/data
guarded, so we can install packages and still have the tests access the
correct paths.
This also adjusts the HTTP-Server we use in the test to serve data
relative to a path it is handed. I now chose `./test/data`, which will
easily allow us to re-use the same HTTP-Server in the future for other
tests that require it.
