Currently stdin is taken to be the pipeline to be built, this allows
it to be instead a map containing the suorces and the pipeline.
We would imagine passing around the sources and pipeline together, so
this just makes the behavior of osbuild more closely match the intended
use and semantics of the sources configuration.
This keeps backwards compatibility for now, but that may be dropped as
soon as osbuild-composer no longer relies on the old behavior.
Disable too-many-{branches,statements} pylint warnings in __main__.py.
These do not seem helpful, but could be reenabled if we drop some
options in the future.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Make the sources options a static property of the pipeline, in
particular of each stage, rather than being passed in on `run()`.
This more closely matches the intended semantics of sources and
pipeline having similar lifetimes and being fairly coupled together.
The difference between the pipeline and the sources is that the
sources do not contribute to identifying the pipeline (they are not
part of the hash for the pipeline id), and they could be swapped
out without changing the output image (as long as they are valid).
However, a pipeline without A sources object would not be useful,
and typically the pipeline and the sources are generated, passed
around and used together.
This is different from the build environment and the secrets object,
which both are specific to either the host or the caller, unlike
the pipeline which should be universal.
This changes the `load()` function to take a `manifest`, which is
a map containing both the pipeline and the sources.
Note that the semantics of the build-env parameter remains unchanged:
It shares the sources with the rest of the pipeline. We may want to
reconsider this in future commits, as the build-env is specific to
the host, whereas the regular pipeline is not.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Appart from giving us a hard time on s390x, this feature did not seem
to have a measurable effect. Moreover, O_DIRECT is not supported by
tmpfs so without this patch we could not use tmpfs as backing store,
which does speed up image generation considerably.
Drop the flag and and rather put the store on tmpfs in order to speed
things up.
This source adds support for downloaded files. The files are
indexed by their content hash, and the only option is their URL.
The main usecase for this will be downloading rpms. Allowing depsolving
to be done outside of osbuild, network access to be restricted and
downloaded rpms to be reused between runs.
Each source is now passed two additional arguments, a cache directory
and an output directory. Both are in the source's namespace, and
the source is responsible for managing them. Each directory may
contain contents from previous runs, but neither is ever guaranteed
to do so.
Downloaded contents may be saved to the cache and resued between
runs, and the requested content should be written to the output dir.
If secrets are used, the source must only ever write contents to
the output that corresponds to the available secrets (rather than
contents from the cache from previous runs).
Each stage is passed an additional argument, a sources directory.
The directory is read-only, and contains a subdirectory named after
each used source, which will contain the requseted contents when
the `Get()` call returns (if the source uses this functionality).
Based on a patch by Lars Karlitski.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Add a new `--checkpoint` option, which can be provided multiple
times, that indicate after which stages a the current stage of
the tree should be committed to the object store; the tree id
will be the treesum of the tree at that point and a reference
is created with the id of the stage at the point.
The argument to `--checkpoint` is the id of the stage. If not
all the given checkpoints can be found the execution will be
aborted.
When the tree is committed to the objects directory of the object-
store, it is done via rename(3). The two possible errors that can
be raised in case that a non-empty tree with the same name already
exist is [EEXIST] or [ENOTEMPTY]. The latter was already ignored
but the former was not. At least on btrfs former will be raised
File "/home/gicmo/Code/src/osbuild/osbuild/objectstore.py",
os.rename(tree.root, output_tree)
FileExistsError: [Errno 17] File exists: 'store/tmpyyi3yvie/tree' -> 'store/objects/…'
Add a new method to the ObjectStore that takes a path to a file
system tree, which is currently being built, and commits it to
the store and references it via a given object_id.
The tree is copied to a temporary location (co-located in the
store to enable fast copying via reflinks) and then atomically
moved into the ObjectStore's objects path via rename(3).
Extract the code from ObjectStore.new that will commit the filled
tree to the store into its own method so it can be used from a
future method to snapshot trees at random points in time.
Introduce a small `TreeObject` class that is the representation of
a tree during its construction. It supports calculating its treesum
as well initialize the new tree with an existing one.
This makes sure all disk access is backed by the same disk. We may
want this for performance reasons (avoiding moving across disks), but
also to experiment with different backing stores for all disk access.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Currently /var was always backed by /var/tmp, but we may want to
control exactly what it is backed by. The default is the same, so
this is not a behavioral change.
The dnf stage wants to import `osbuild.sources` but currently the
osbuild module is not available in the stages. Apply the same hack
done in the Assembler also in for the stages, i.e. bind mount the
osbuild module to the stages/osbuild.
This happens rarely when the same loop device is used in rapid
succession. The kernel flushes the page cache asynchronously, which
means that it might not be cleared yet when a new file is bound.
`set_status` checks if the cache is clear (`set_fd` doesn't).
Handle this by trying a different device when `set_status` returns
`EBUSY`.
Fixes#177
Don't wait until python's garbage collector closes the file descriptors
to loop devices. Close them when the `LoopServer` context manager exits,
after an assembler has finished running.
The z Initial Program Loader (zipl) when creating the bootmap in
bootmap_creat (src/zipl/bootmap.c) wants to create a device node
via misc_temp_dev (bootmap_create:1141) for the device that it
is installing the bootloader to[1]. Currently access to loopback
devices is allowed from within the container (it is used to mount
the image), but only read/write access. On s390x also allow the
creation of device nodes, so zipl can do its work and install
the bootloader stages on the "disk".
[1] zipl source at commit dcce14923c3e9615df53773d1d8a3a22cbb23b96
Add a new command line option `--secrets`, which accepts a JSON file
that is structured similarly to a source file. It is should contain data
that is necessary to fetch content, but shouldn't appear in any logs.
This might (hopefully) fix a race in destructing the asyncio.EventLoop
that's used in all API classes, which leads to warnings about unhandled
exceptions on CI.
This also puts their creation closer to where the client-side sockets
are created.
Pipelines encode which source content they need in the form of
repository metadata checksums (or rpm checksums). In addition, they
encode where they fetch that source content from in the form of URLs.
This is overly specific and doesn't have to be in the pipeline's hash:
the checksum is enough to specify an image.
In practice, this precluded using alternative ways of getting at source
packages, such as local mirrors, which could speed up development.
Introduce a new osbuild API: sources. With it, a stage can query for a
way to fetch source content based on checksums.
The first such source is `org.osbuild.dnf`, which returns repository
configuration for a metadata checksum. Note that the dnf stage continues
to verify that the content it received matches the checksum it expects.
Sources are implemented as programs, living in a `sources` directory.
They are run on the host (i.e., uncontained) right now. Each source gets
passed options, which are taken from a new command line argument to
osbuild, and an array of checksums for which to return content.
This API is only available to stages right now.
The recent changes removed the {Assembler,Stage}Failed exceptions,
which includes them being thrown from Stage.run and Assembler.run.
Instead result dictionaries are returned even on errors. But the
object store, used as a context manager, relies on exceptions to
detect the error case and thus needs them to cleanup the temporary
objects. Without those exceptions the temporary objects end up in
the store even when the sage or assembler failed.
Restore the old behavior by throwing a generic BuildError exception
from the Stage and Assembler, which will be caught directly in the
pipeline and converted to a result dict.
The socket that the osbuild and loop apis should talk on are passed into
their `__init__` function. The caller should be responsible for closing
those sockets.
This already happens in all current callers.
This fixes a non-fatal error on RHEL's python 3.6, because it was
calling `socket.close` on an already-closed socket:
Traceback (most recent call last):
File "/usr/lib64/python3.6/asyncio/base_events.py", line 529, in __del__
self.close()
File "/usr/lib64/python3.6/asyncio/unix_events.py", line 63, in close
super().close()
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 99, in close
self._close_self_pipe()
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 109, in _close_self_pipe
self._remove_reader(self._ssock.fileno())
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 268, in _remove_reader
key = self._selector.get_key(fd)
File "/usr/lib64/python3.6/selectors.py", line 189, in get_key
return mapping[fileobj]
File "/usr/lib64/python3.6/selectors.py", line 70, in __getitem__
fd = self._selector._fileobj_lookup(fileobj)
File "/usr/lib64/python3.6/selectors.py", line 224, in _fileobj_lookup
return _fileobj_to_fd(fileobj)
File "/usr/lib64/python3.6/selectors.py", line 41, in _fileobj_to_fd
raise ValueError("Invalid file descriptor: {}".format(fd))
ValueError: Invalid file descriptor: -1
Commit 82a2be53d introduced a new return type from `Pipeline.run()`. It
changed the caller in `__main__.py`, but missed that the build pipeline
uses the same function.
A pipeline run only returned logs in the `StageFailed` and
`AssemblerFailed` exceptions. Remove those and always return structured
data instead.
It only returns data for stages that actually ran (i.e., didn't come
from the cache). This is similar to the output in interactive mode.
Also change osbuildtest to be able to deal with output that is larger
than the pipe buffer by using subprocess.communicate().
The workaround of manually linking /lib64 -> /usr/lib64 inside the
container that is needed on s390 is also required on ppc64 because
here the dynamic linker is set to /lib64/ld64.so.2 and the /lib64
link is not created.
Work around a combination of systemd not creating the link from
/lib64 -> /usr/lib64 (see systemd issue #14311) and the dynamic
linker is being set to (/lib/ld64.so.1 -> /lib64/ld64.so.1)
Therefore we manually create the link before calling nspawn
osbuild currently throws an error when not passing a build environment
on the command line, because the runner is unset. This is annoying on
hosts which only need a runner set, but no build pipeline.
To simplify running osbuild in this common case, introduce
`org.osbuild.host`, which is a runner that is defined to work on the
host that osbuild is installed on. Use this runner by default and
include a symlink to the right runner in the Fedora and RHEL packages.
Also add `runners/org.osbuild.host` to `.gitignore`, so that developers
can set the symlink when running osbuild from the source directory.
Fixes#171
We've been using a generic `osbuild-run`, which sets up the build
environment (and works around bugs) for all build roots. It is already
getting unwieldy, because it tries to detect the OS for some things it
configures. It's also about to cause problems for RHEL, which doesn't
currently support a python3 shebang without having /etc around.
This patch changes the `build` key in a pipeline to not be a pipeline
itself, but an object with `runner` and `pipeline` keys. `pipeline` is
the build pipeline, as before. `runner` is the name of the runner to
use. Runners are programs in the `runners` subdirectory.
Three runners are included in this patch. They're copies of osbuild-run
for now (except some additions for rhel82). The idea is that each of
them only contains the minimal setup code necessary for an OS, and that
we can review what's needed when updating a build root.
Also modify the `--build-pipeline` command line switch to accept such a
build object (instead of a pipeline) and rename it accordingly, to
`--build-env`.
Correspondingly, `OSBUILD_TEST_BUILD_PIPELINE` → `OSBUILD_TEST_BUILD_ENV`.
`osbuild-run` sets up the build root so that programs can be run
correctly in it. It should be run for all programs, not just stages and
assemblers (even though they're the only consumers right now).
Also, conceptually, `osbuild-run` belongs to the build root. We'll
change its implementation based on the build root in a future commit.
The buildroot already sets up `/run/osbuild/api`. It makes sense to have
it manage libdir as well.
A nice side benefit of this is a simplification of the Stage and
Assembler classes, which grew quite complex and contained duplicate
code.
Use the new the osbuild API to setup the standard input/output
inside the container, i.e. replace stdin, stdout, and stderr with
sockets provided by the host.
Introduce an osbuild API that can be used by the container to talk
to the osbuild host. It currently supports one method 'setup-stdio'
which should be used by the container to setup its standard input/
output so the stages can transparently do i/o with the osbuild host
via stdio.
The input data (args) is written to a temp-file backed buffer. The
output is either the host's stdout directly or another temp-file
backed buffer; the latter is re-opened (via /proc/self/fd) to get
another file-descriptor for the container, so in theory the host
and the container could do i/o to the same buffer independently.
Expose the flags, address parameter of the underlying sock.sendmsg
method, in order to be able to explicitly specify the recipient of
the message; as needed in connection-less mode.
Python 3.2 renamed array.fromstring to array.frombytes, but kept
the former as an, now deprecated, alias. Use the canonical form
which indeed better describes what is going on.
In case osbuild is invoked without libdir parameter, the osbuild files
are not propagated into the buildroot container and therefore all
pipelines containing buildroot fail.
Example:
```
$ sudo osbuild --store /var/osbuild/ qcow2-pipeline.json
...
execv(/usr/lib/osbuild/osbuild-run) failed: No such file or directory
```
Unfortunately this is only the first error. Once you fix it, you realize
that also the symlink from "assemblers" directory is missing and
therefore you cannot import osbuild because it is not available anywhere
in the path. This is why I had to bind the osbuild module from host to
the build container.
If dir_fd wasn't passed, create_device() openend it to `/dev` and forgot
about closing it. To fix this, it would have to gain logic to only close
the fd if it wasn't passed in.
Side-step the problem by removing dir_fd, since nothing is using it
right now. We can add it back if something needs it.
Closing the socket is the responsibility of whoever opened it.
Fix this in the only user (qemu assembler) by using socket() in a `with`
block, which closes the socket on exit.
Storytime! I tried to run multiple osbuilds at once. It failed when
unmounting the buildtree. Weird. It turned out the buildtree was not
there anymore when osbuild tried to unmount it. But who unmounted it?
We need to deep dive into mount-types.
Nowadays, the / directory is shared-mounted by systemd. See:
https://serverfault.com/questions/868682/implications-of-mount-make-private
This has interesting implications, see the following example:
we start osbuild1 with /var/tmp/os1 as its store
osbuild1 creates /var/tmp/os1/tmp
osbuild1 bind-mounts / onto /var/tmp/os1/tmp
we start osbuild2 with /var/tmp/os2 as its store
osbuild2 creates /var/tmp/os2/tmp
osbuild2 bind-mounts / onto /var/tmp/os2/tmp
Now, the shared-mounting goes into effect:
The second mount-event gets propagated into the first mount, where it
creates another mount, so we get something like this:
/var/tmp/os1/tmp/var/tmp/os2/tmp
But this is just a start! Imagine running three osbuilds at once.
The event would get propagated to those 3 mounts created by two
osbuilds, creating 3 extra mounts, 7 in total.
It turns out this mounting strategy creates an *exponential number* of
mounts. Crazy, right?
This commit mounts the root inside build root using private bind, which
doesn't propagate bind-events. This solves the problem with the
exponential growth.
But the original problem was different, mount points were disappearing.
So how does this fix solve the problem?
Honestly, I don't know. Something with mount-event propagation is
probably responsible, but I cannot imagine how it is actually affecting
the unbinding.
