The description of a pipeline is format dependent and thus needs
to be located at the specific format module.
Temporarily remove two tests; they should be added back to a format
specific test suit.
Instead of having the pipeline and the source option as separate
arguments, the load function now takes the full manifest, which
has those two items combined.
Instead of importing the load, load_build functions into the osbuild
namespace and using it via that, use the load function via the module
that provides them, i.e. the formats.v1 module.
The validation of the manifest descritpion is eo ipso format
specific and thus belongs into the format specific module.
Adapt all usages throughout the codebase to directly use the
version 1 specific function.
Extract the code that loads a pipeline from a pipeline description,
i.e. a manifest, into a new module inside a new 'formats' package.
The idea is to have different descriptions, i.e. different formats,
for the same internal representation. This allows changing the
internal representation, i.e. data structures, but still having the
same external description.
Later a new description might be added that better matches the new
internal representation.
This was deprecated in favor of always having the source in the
manifest. Remove the command line option and the corresponding
code that would override the sources definitions.
Update the docs accordingly.
Integrate with codecov. Define a threshold of 5% to pass. Coverage
is cumulative, i.e. all the tests send their coverage to codecov,
which will integrate them all into a total.
This runner is used by both CentOS 8 and CentOS Stream. CentOS is kinda weird
because it specifies only number 8 in VERSION_ID in /etc/os-release unlike
RHEL. Also the ID is the same for CentOS 8 and CentOS Stream.
This should work fine for now though:
CentOS 8 is currently based on RHEL 8.3 and CentOS Stream on devel version of
RHEL 8.4. For both RHEL 8.3 and 8.4 we use the RHEL 8.2 runner so it should be
safe to assume that it's OK to base the CentOS 8 runner also on the RHEL 8.2
one.
We might need to tweak this at some point but I suggest dealing with it when
that time comes.
The "/run/osbuild" path is used as the default runpath by the
BuildRoot, which creates it on demand. The only other place
is the API (`BaseAPI`) to create the socket directories in,
but that is now also created on-demand. Additionally, the
API are only run after the build root has been set up so that
directory would already exist.
When creating the socket directory, i.e. in the case that it was
not specified directly, ensure the parent directories exist.
Make it possible to override that parent directory.
We do want to turn the dependency generator off for runners,
because they are tied to the specific platform, which might, if
not disabled, introduce dependencies for that platform to the
general package. An prominent example is platform-python used
by the RHEL runner.
On the other hand, we do want to pick up the dependency for the
stages and assemblers, i.e. /usr/bin/python3, because they need
to be able to run on the host, since the host provides the root
file-system for the initial build container, the build host.
Add an additional comment to the shebang mangling exception to
explain that due to the combination of dependency generator and
the disabling of shebang mangler for assembler and stages an
additionally dependency on /usr/bin/python3 will be added on RHEL,
and that this is what we indeed want.
Pin the osbuild-composer that schutzbot runs a reverse dependency test
against. This allows to control which exact version to test against, and
ensures that PRs against osbuild always run against the same version.
Now that osbuild-composer's CI uploads RPMs to a predictable destination
(the same one that osbuild uses), we can use that instead of rebuilding
osbuild-composer on every CI run. This should speed up the mockbuild
stage considerably.
Pin it to v24 now.
Drop setting fastestmirror, disabling weak dependencies, and removal of
modular repositories.
Try to install as close to what people do in production, which means
sticking to the defaults.
It was only used once, to retry dnf. This is not necessary, because dnf
already has retrying logic. We're also not using `retry` on any of the
other calls to dnf in this script.
Now that the repository URLs are predictable, don't use Jenkins' stash
feature to pass the repo file between stages.
Instead, simply create the repo file where it is needed, in deploy.sh.
The length of these is not predictable. It depends on the shortest
unique prefix in the repository and git configuration.
Just use the full one, which also makes it easier to copy the id from
`git log` or GitHub.
Change the repository path on S3 to a more predictable one. We really
only need the name of the project (static osbuild for this repository),
the name of the distro (use the same as osbuild-composer's API for
consistency) and the commit SHA.
In particular, drop the PR number / branch name. Also don't remove the
dots from version numbers. All places we're using them in (paths and
URLs) support dots.
For example, osbuild commit xxxxxxx for fedora-33 on x86_64 will result
in this URL:
osbuild/fedora-33/x86_64/xxxxxxx
Jenkins has been configured to use the latest commit on a pull request
(instead of merging to master) for a long time now. Rename the variable
to reflect that.
Explicit re-raise the BufferError exception in recv from the orignal
JSONDecodeError, so the latter gets recorded as the underlying cause.
Uncovered by pylint 2.6.0: W0707: "Not using raise from makes the
traceback inaccurate, because the message implies there is a bug in
the exception-handling code itself, which is a separate situation
than wrapping an exception."
All runners stopped calling `api.setup_stdio` (commit c40b414), and
thus all output of runners and also modules is now redirected to a
pipe (created via Popen and subprocess.PIPE for stdout).
Text was read from that pipe via `stdout.read(4096)`, which means
that it is now buffered in chunks of 4096, where it previously was
line buffered in the case that osbuild was run in the terminal and
--json was not specified. This is very annoying for anyone wanting
to follow osbuild's output in real-time.
Restore the previous behavior by using `os.read`, which should be
a small wrapper around read(3), which does not block until all the
requested data is available but returns early (short reads). This
means, new text will be forwarded as soon is it is available in the
pipe. Increase the read buffer to 32768 while at it, which is what
Popen is using in Python 3.9.
Add two simple tests to check that the osbuild executable fails with the
right exit codes when passed an invalid manifest or checkpoint.
This reuses test.OSBuild, which is extended to raise CalledProcessError
if needed.
osbuild_cli() sometimes returned an exit code, but at the end called
sys.exit() directly. The idea was probably to always return the code
with which the executable should exit.
Make this consistent and call sys.exit() in __main__.py, with the value
returned by osbuild_cli().
Highlights are metadata fields are return for packages in the rpm
stage, the spec file has been fixed to not mangle shebangs anymore.
Generally, CI and testing has been improved.
