This was deprecated in favor of always having the source in the
manifest. Remove the command line option and the corresponding
code that would override the sources definitions.
Update the docs accordingly.
Integrate with codecov. Define a threshold of 5% to pass. Coverage
is cumulative, i.e. all the tests send their coverage to codecov,
which will integrate them all into a total.
This runner is used by both CentOS 8 and CentOS Stream. CentOS is kinda weird
because it specifies only number 8 in VERSION_ID in /etc/os-release unlike
RHEL. Also the ID is the same for CentOS 8 and CentOS Stream.
This should work fine for now though:
CentOS 8 is currently based on RHEL 8.3 and CentOS Stream on devel version of
RHEL 8.4. For both RHEL 8.3 and 8.4 we use the RHEL 8.2 runner so it should be
safe to assume that it's OK to base the CentOS 8 runner also on the RHEL 8.2
one.
We might need to tweak this at some point but I suggest dealing with it when
that time comes.
The "/run/osbuild" path is used as the default runpath by the
BuildRoot, which creates it on demand. The only other place
is the API (`BaseAPI`) to create the socket directories in,
but that is now also created on-demand. Additionally, the
API are only run after the build root has been set up so that
directory would already exist.
When creating the socket directory, i.e. in the case that it was
not specified directly, ensure the parent directories exist.
Make it possible to override that parent directory.
We do want to turn the dependency generator off for runners,
because they are tied to the specific platform, which might, if
not disabled, introduce dependencies for that platform to the
general package. An prominent example is platform-python used
by the RHEL runner.
On the other hand, we do want to pick up the dependency for the
stages and assemblers, i.e. /usr/bin/python3, because they need
to be able to run on the host, since the host provides the root
file-system for the initial build container, the build host.
Add an additional comment to the shebang mangling exception to
explain that due to the combination of dependency generator and
the disabling of shebang mangler for assembler and stages an
additionally dependency on /usr/bin/python3 will be added on RHEL,
and that this is what we indeed want.
Pin the osbuild-composer that schutzbot runs a reverse dependency test
against. This allows to control which exact version to test against, and
ensures that PRs against osbuild always run against the same version.
Now that osbuild-composer's CI uploads RPMs to a predictable destination
(the same one that osbuild uses), we can use that instead of rebuilding
osbuild-composer on every CI run. This should speed up the mockbuild
stage considerably.
Pin it to v24 now.
Drop setting fastestmirror, disabling weak dependencies, and removal of
modular repositories.
Try to install as close to what people do in production, which means
sticking to the defaults.
It was only used once, to retry dnf. This is not necessary, because dnf
already has retrying logic. We're also not using `retry` on any of the
other calls to dnf in this script.
Now that the repository URLs are predictable, don't use Jenkins' stash
feature to pass the repo file between stages.
Instead, simply create the repo file where it is needed, in deploy.sh.
The length of these is not predictable. It depends on the shortest
unique prefix in the repository and git configuration.
Just use the full one, which also makes it easier to copy the id from
`git log` or GitHub.
Change the repository path on S3 to a more predictable one. We really
only need the name of the project (static osbuild for this repository),
the name of the distro (use the same as osbuild-composer's API for
consistency) and the commit SHA.
In particular, drop the PR number / branch name. Also don't remove the
dots from version numbers. All places we're using them in (paths and
URLs) support dots.
For example, osbuild commit xxxxxxx for fedora-33 on x86_64 will result
in this URL:
osbuild/fedora-33/x86_64/xxxxxxx
Jenkins has been configured to use the latest commit on a pull request
(instead of merging to master) for a long time now. Rename the variable
to reflect that.
Explicit re-raise the BufferError exception in recv from the orignal
JSONDecodeError, so the latter gets recorded as the underlying cause.
Uncovered by pylint 2.6.0: W0707: "Not using raise from makes the
traceback inaccurate, because the message implies there is a bug in
the exception-handling code itself, which is a separate situation
than wrapping an exception."
All runners stopped calling `api.setup_stdio` (commit c40b414), and
thus all output of runners and also modules is now redirected to a
pipe (created via Popen and subprocess.PIPE for stdout).
Text was read from that pipe via `stdout.read(4096)`, which means
that it is now buffered in chunks of 4096, where it previously was
line buffered in the case that osbuild was run in the terminal and
--json was not specified. This is very annoying for anyone wanting
to follow osbuild's output in real-time.
Restore the previous behavior by using `os.read`, which should be
a small wrapper around read(3), which does not block until all the
requested data is available but returns early (short reads). This
means, new text will be forwarded as soon is it is available in the
pipe. Increase the read buffer to 32768 while at it, which is what
Popen is using in Python 3.9.
Add two simple tests to check that the osbuild executable fails with the
right exit codes when passed an invalid manifest or checkpoint.
This reuses test.OSBuild, which is extended to raise CalledProcessError
if needed.
osbuild_cli() sometimes returned an exit code, but at the end called
sys.exit() directly. The idea was probably to always return the code
with which the executable should exit.
Make this consistent and call sys.exit() in __main__.py, with the value
returned by osbuild_cli().
Highlights are metadata fields are return for packages in the rpm
stage, the spec file has been fixed to not mangle shebangs anymore.
Generally, CI and testing has been improved.
Metadata information can easily become very big, like in the case
of the package metadata of the org.osbuild.rpm stage, quite likely
exceeding the configured maximum package length of the underlying
socket. To avoid potential issues here, transfer the actual data
by writing it to a temporary file and sending a open fd over.
osbuild composer upstream has dropped support for Fedora 31 since
this is very close to end-of-life. Since we use composer master
in our CI this change actually broke it.
s3cmd does not work properly with python 3.9 (used on Fedora 33):
Problem: <class 'AttributeError: 'xml.etree.ElementTree.Element'
object has no attribute 'getchildren'
S3cmd: 2.1.0
python: 3.9.0rc1 (default, Aug 12 2020, 00:00:00)
Use the one from the official repositories, which in case of F33
has a downstream fix for the error.
See rhbz#1884607 and s3tools/s3cmd#1137
Patch based on osbuild-composer commit 1a69a891 by Ondřej Budai.
Instead of hard-coding the use of the "org.osbuild.linux" runner,
use the new `osbuild.pipeline.detect_host_runner` function to
dynamically detect the runner for the host system. That should fix
the tests on RHEL systems, where python3 is by default not present
and even if it is manually installed, is an indirection via
alternatives (i.e. a link to /etc/alternatives), which must be
explicitly configured in the build root container for the host.
