Since low-level primitives (os.read) are used to read from the stdout
pipe, manual text decoding was necessary there anyway. The `encoding`
argument meant that we could forgo the manual decoding for the call
to `communicate`. But this meant that text handling is not uniform.
Therefore, remove the `encoding` argument from the `Popen` call and
manual decode all the text.
Host services are a way to provide functionality to stages that is
restricted to the host and not directly available in the container,
such as providing input to stages, devices access and mounting.
This commit introduces a `ServiceManager` class that can be used to
start and (automatically) stop host service, as well as a `Service`
base class together with a `ServiceClient` class that be used to
implement host services and communicate with them. Refer to the doc
string of the module for more information.
Often, a message is being sent and followed by a call to `recv`
to wait for a reply. Create a simple helper `send_and_recv` that
does both in one method.
Add a simple check for that helper to the tests.
Add a new constructor method that allows creating a `Socket` from
an existing file-descriptor of a socket. This might be need when
the socket was passed to a child process.
Add a simple test for the new constructor method.
Add a new constructor method, `Socket.new_pair`, to create a pair
of connected sockets (via `socketpair`) and wrap both sides via
`jsoncomm.Socket`.
Add a simple test to check it.
For schema version 2 of modules, the `definitions` node, as defined in
the module itself, won't be at the `options` level but at the level of
the `properties` node. Look for a `definitions` at that `properties`
level and move it to the top, if found.
Work around a bug on aarch64[1] where `qemu-img` would hang
about a third of the time when converting images. To be able
to activate the work-around based on the environment, i.e.
only on certain distributions, introduce an environment
variable, `OSBUILD_QEMU_IMG_COROUTINES`, that is set in the
runner and then picked up in the assembler.
[1] https://bugs.launchpad.net/qemu/+bug/1805256
When new ostree related stages and the new ostree input was added
they were included in the main package since all the modules were
manually listed in the corresponding exclude/include sections.
Change that by using wildcards, since all ostree related modules
should start with the org.osbuild.ostree* pattern.
When parsing the module file, parse the JSON directly from the AST
node, because the AST node contains the line number of the schema
in the module and thus we can resolve the correct line number for
errors within the JSON. Convert the `JSONDecodeError` to a
`SyntaxError` which results in an overall better exception message:
Before:
Traceback (most recent call last):
File "/workspaces/osbuild/osbuild/meta.py", line 331, in get_schema
opts = self._make_options(version)
[...]
File "/usr/lib64/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in
double quotes: line 2 column 1 (char 14)
After:
Traceback (most recent call last):
File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
[...]
raise SyntaxError(msg, detail) from None
File "stages/org.osbuild.ostree.init-fs", line 31
additionalProperties: False
^
SyntaxError: Invalid schema: Expecting property name enclosed in ...
Define the mapping of modules and their paths at the `ModuleInfo` class
level instead of having it inline in a function. This makes it possible
to use it from other places in the code.
Disable host-only mode when running dracut and generate reproducible
images by default.
Suggested-by: gicmo
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Move from using 'zram' to 'zram-generator-defaults' in the ostree bootiso
testing manifest. More information is available in Fedora 33 Change
document [1].
Add org.osbuild.kernel-cmdline stage to fedora-boot.json manifest
because of change in how grub handles the kernel command line arguments
[2].
GRUB2 Stage 2 checksums in assemblers test are updated. The change have
been verified by building the fedora-boot.json manifest with each checked
filesystem and booting the image in QEMU with legacy mode.
[1] https://fedoraproject.org/wiki/Changes/SwapOnZRAM
[2] https://github.com/osbuild/osbuild-composer/pull/982#issuecomment-697356929
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The previous version covered too few use cases, more specifically a
single subscription. That is of course not the case for many hosts, so
osbuild needs to understand subscriptions.
When running org.osbuild.curl source, read the
/etc/yum.repos.d/redhat.repo file and load the system subscriptions from
there. While processing each url, guess which subscription is tied to
the url and use the CA certificate, client certificate, and client key
associated with this subscription. It must be done this way because the
depsolving and fetching of RPMs may be performed on different hosts and
the subscription credentials are different in such case.
More detailed description of why this approach was chosen is available
in osbuild-composer git: https://github.com/osbuild/osbuild-composer/pull/1405
RHEL 8.4 is now GA, so we don't need any extra tests for it. This should also
make the CI more reliable because having two distros with the same DISTRO_CODE
caused some tests to fail randomly (they used the same intermediate
artifacts).
Define a set of pre-defined ostree related annotations that can
and should be used to indicate that a container image contains
an OSTree commit. This can be used by other tools to inspect and
extract the commit more easily.
Add support for arbitrary manifest annotations: allow anything
with the exception of the `org.osbuild` and `org.opencontainer`
prefixes. The former is reserved by us, the latter by the OCI
image specification. The latter specifies a set of pre defined
keys, which are not yet supported by osbuild but will be in the
future, partly via more generic options (creation time).
Add a simple tool that will spit out a valid org.osbuild.inline
source entry that encodes the given file. Currently always uses
base64 as encoding and sha256 for the hashing.
Add a new source for transporting binary data within the source
entry itself. The data is ascii encoded in the `data` property
of the inline source item, with the encoding that is used being
specified in the `encoding` property.
Now that there is a common utility function to verify the checksum
of a file, use that.
Also fix the json schema entry for the property to have to correct
minium and maximum digest length, given the supported algorithm,
which is 32 (md5) and 128 (sha512) characters.
Test that `checksum.verify_file` works correctly, which internally
uses the only other utility function `checksum.hexdigest_file`.
Check all algorithms currently supported by the `org.osbuild.curl`
source.
Create a setup.cfg file and move the pylint settings from .pylintrc
here. This file can be used to configure other tooling as well, so
it is in general more useful as a central configuration place.
Add basic checks for the ostree source, which includes a successful
pull of a commit, an empty source entry and one where the specified
commit is non-existant. For this create a simple commit in a ostree
repo is checked in. The commit was created via:
mkdir "/tmp/data"
echo "Hello World" > /tmp/data/hello.txt
ostree init --repo test/data/sources/org.osbuild.ostree/data/repo \
--mode=archive
ostree commit --repo test/data/sources/org.osbuild.ostree/data/ \
--branch "test/ostree" /tmp/data \
--timestamp="1995-05-13 12:34:56 +0000"
This should give an commit with the following commit id:
d6243b0d0ca3dc2aaef2e0eb3e9f1f4836512c2921007f124b285f7c466464d8
Since the `sources.SourcesServer` has been removed, nothing is
using the export functionality anymore. Inputs are now used to
make content in the store available to stages. Remove all the
export logic from org.osbuild.ostree.
Since the `sources.SourcesServer` has been removed, nothing is
using the export functionality anymore. Inputs are now used to
make content in the store available to stages. Remove all the
export logic from org.osbuild.curl.
The usage of the `sources.SourcesServer` and `sources.get` have
been removed from `Stage.run`, which was the only usage throughout
osbuild and thus it is not needed anymore and can be removed.
All sources are now pre-fetched before any pipeline and thus any
stage is being built. Additionally, in the version 1 foramt, all
stages that were using source are converted to use inputs when
the manifest is loaded. Thus, nothing should use `source.get`
and thus the sources API (`SourcesServer`) anymore.
Parameterize `test_sources` via `pytest.mark.parameterize`, so that
now the product of source and test cases for that source is visible
to pytest and thus also the caller.
Instead of testing the legacy `SourcesServer` and `sources.get`,
test the `Source.download` method, which is the new and exclusive
way how sources are used in osbuild. For this, the complete info
for a specific source is now included the specific test case.
For the `org.osbuild.curl` source this means that the respective
information is moved from `sources.json`, which is then unused
and thus removed. The test case that checks for an unknown
checksum is also removed because `Source.download` just fetches
everything instead of a subset.
Use `pytest`, which is a dependency of osbuild since a few versions,
instead of `unittest` for the source tests. The main reason is to be
able to use `pytest.mark.parameterize` in the near future to easily
generate a test matrix.
Add a new option `qcow2_comapt` which can be used explicitly
select the compatibility level of the qcow2 file format. Qemu
version 1.1 introduced extensions to the format that became
the default with 1.7, which are not readable by qemu < 1.1.
Thus if the resulting qcow2 should be read by such older qemu
versions, the compatibility level needs to be set to 0.10.
According to the OCI Image Format Specification[1] history entries
for layers in the container are optional; but when trying to push
a container quay.io via skopeo (copy oci-archive:… docker://quay)
it will fail with "Cannot convert an image with 0 history entries".
This seems to come from the containers/image[2] library when the
container is converted back from the docker distribution format
to oci-archive on quay.io. Thus it seems that when skopeo converts
the image to the docker format for the distribution it does not
fill any the history entries, which are then assumed and required
to be there when converting back.
To fix this, insert history entries for each layer that is created.
[1] https://github.com/opencontainers/image-spec/blob/master/config.md
[2] https://github.com/containers/image/
Only include a very specific set of extended attributes:
- user.*: user specified extended attributes
- security.ima: Integrity Measurement Architecture (IMA)
- security.capability: Linux capabilities(7)
This follows what containers/storage[1] and containers/buildah[2]
are doing. It is important to note that we DO NOT want selinux
related extended attributes (`security.selinux`) in there, which
seems to be pulled in by some versions of `tar` even if that was
seemingly excluded via `--no-selinux`. Therefore we also exclude
selinux and xattrs explicitly from the wrapping container to
make sure they are never included.
[1] 35ebda8ae2/pkg/archive/archive.go (L399)
[2] 214e4c9335/copier/xattrs.go (L19)
