Since Object knows its base now, the initialization of the tree
with the content of its base can be delayed until the moment
someone wants to actually modify the tree, thus implementing
copy on write semantics. For this a new `write` method is added
that will initialize the base and return the writable tree. It
should be used instead of `path` whenever the a client wants to
write to the tree of the Object.
Adapt the pipeline and the tests to use the new `write` method
in all the appropriate places.
NB: since the intention can not be inferred when using `path`
directly, the Object is still being initialized there.
When a new Object is created it can have a `base`, i.e. another
object that is already committed to the store, which is then used
to initialize the tree of the new object. That is, the contents
of the new Object will be based on the contents of the existing.
The initialization of an Object with its base (if any) was done
by the ObjectStore. Move all of that logic inside `Object`:
The Object will store its base, which `Object.init` will use to
initialize itself. Additionally, if `Object.path` is accessed
`init` is being called as well to make sure it is properly
initialized, i.e. the tree initialized with the base content.
Refactor the `ObjectStore.snapshot` method to take an `Object` not
a plain filesystem tree, so the latter is more encapsulated from
the ObjectStore user (e.g. the pipeline) and prepares a unified
code-path for `snapshot` and `commit` in the future.
Now that Object manages its work directory itself, re-create the
latter when the its tree is moved, i.e. when the object is being
committed to the store. This means that after the object has been
written to the store it is in the same state is if it was new and
can be used in the very same way.
If the move itself fails (the rename(2) fails), the tree and its
contents is cleaned up with the reset of the work directory.
Rename the `move` method to `store_tree` to better reflect how the
method should be used, i.e. to store the tree corresponding to the
Object instance.
When a new object is being created, a corresponding temporary
directory is created within the file system tree of the object
store, which shall be called the "work dir". Within that dir a
well-known directory ('tree') is created that then is the root
of the filesystem tree[1] that clients use to store the tree
or the resulting image in.
Previously, the work dir was managed, i.e. created and cleaned
up (via a context manager) by the ObjectStore. Now the Object
itself manages the tree and thus the lifetime of the work dir
is more directly integrated and controlled by it. As a result
the Object itself is now a context manager. On exit of the
context the work dir is cleaned up.
[1] For the assembler this is the output directory that will
contain the final image.
Instead of just returning the path of the temporary object that is
created in .new() the actual instance of the new `Object` is being
returned, which can then provide a richer interface for clients
than a plain directory path.
Keep are reference to the parent store, which this object is tied
to. It is currently not yet used directly but is a preparation for
a closer Object and ObjectStore integration that will happen in
commits to follow.
As the name implies, the ObjectStore stores objects, which can be
trees but also everything an Assembler can make of the input tree,
like qcow2 images, tarballs and other non tree-like outputs.
Therefore rename the TreeObject to Object to better reflect that it
is representing any object, not only trees, in the store.
Detect the host dynamically from os-release(5) instead of relying on the
`org.osbuild.host` symlink.
It is awkward to install a symlink that tells osbuild which distro is is
running on, when there is a standard way to detect this.
This makes it easier to run osbuild from sources and removes the need to
include every host in the spec file. The latter became hard to do,
because there's no obvious way to distinguish RHEL minor releases.
Currently stdin is taken to be the pipeline to be built, this allows
it to be instead a map containing the suorces and the pipeline.
We would imagine passing around the sources and pipeline together, so
this just makes the behavior of osbuild more closely match the intended
use and semantics of the sources configuration.
This keeps backwards compatibility for now, but that may be dropped as
soon as osbuild-composer no longer relies on the old behavior.
Disable too-many-{branches,statements} pylint warnings in __main__.py.
These do not seem helpful, but could be reenabled if we drop some
options in the future.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Make the sources options a static property of the pipeline, in
particular of each stage, rather than being passed in on `run()`.
This more closely matches the intended semantics of sources and
pipeline having similar lifetimes and being fairly coupled together.
The difference between the pipeline and the sources is that the
sources do not contribute to identifying the pipeline (they are not
part of the hash for the pipeline id), and they could be swapped
out without changing the output image (as long as they are valid).
However, a pipeline without A sources object would not be useful,
and typically the pipeline and the sources are generated, passed
around and used together.
This is different from the build environment and the secrets object,
which both are specific to either the host or the caller, unlike
the pipeline which should be universal.
This changes the `load()` function to take a `manifest`, which is
a map containing both the pipeline and the sources.
Note that the semantics of the build-env parameter remains unchanged:
It shares the sources with the rest of the pipeline. We may want to
reconsider this in future commits, as the build-env is specific to
the host, whereas the regular pipeline is not.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Appart from giving us a hard time on s390x, this feature did not seem
to have a measurable effect. Moreover, O_DIRECT is not supported by
tmpfs so without this patch we could not use tmpfs as backing store,
which does speed up image generation considerably.
Drop the flag and and rather put the store on tmpfs in order to speed
things up.
This source adds support for downloaded files. The files are
indexed by their content hash, and the only option is their URL.
The main usecase for this will be downloading rpms. Allowing depsolving
to be done outside of osbuild, network access to be restricted and
downloaded rpms to be reused between runs.
Each source is now passed two additional arguments, a cache directory
and an output directory. Both are in the source's namespace, and
the source is responsible for managing them. Each directory may
contain contents from previous runs, but neither is ever guaranteed
to do so.
Downloaded contents may be saved to the cache and resued between
runs, and the requested content should be written to the output dir.
If secrets are used, the source must only ever write contents to
the output that corresponds to the available secrets (rather than
contents from the cache from previous runs).
Each stage is passed an additional argument, a sources directory.
The directory is read-only, and contains a subdirectory named after
each used source, which will contain the requseted contents when
the `Get()` call returns (if the source uses this functionality).
Based on a patch by Lars Karlitski.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Add a new `--checkpoint` option, which can be provided multiple
times, that indicate after which stages a the current stage of
the tree should be committed to the object store; the tree id
will be the treesum of the tree at that point and a reference
is created with the id of the stage at the point.
The argument to `--checkpoint` is the id of the stage. If not
all the given checkpoints can be found the execution will be
aborted.
When the tree is committed to the objects directory of the object-
store, it is done via rename(3). The two possible errors that can
be raised in case that a non-empty tree with the same name already
exist is [EEXIST] or [ENOTEMPTY]. The latter was already ignored
but the former was not. At least on btrfs former will be raised
File "/home/gicmo/Code/src/osbuild/osbuild/objectstore.py",
os.rename(tree.root, output_tree)
FileExistsError: [Errno 17] File exists: 'store/tmpyyi3yvie/tree' -> 'store/objects/…'
Add a new method to the ObjectStore that takes a path to a file
system tree, which is currently being built, and commits it to
the store and references it via a given object_id.
The tree is copied to a temporary location (co-located in the
store to enable fast copying via reflinks) and then atomically
moved into the ObjectStore's objects path via rename(3).
Extract the code from ObjectStore.new that will commit the filled
tree to the store into its own method so it can be used from a
future method to snapshot trees at random points in time.
Introduce a small `TreeObject` class that is the representation of
a tree during its construction. It supports calculating its treesum
as well initialize the new tree with an existing one.
This makes sure all disk access is backed by the same disk. We may
want this for performance reasons (avoiding moving across disks), but
also to experiment with different backing stores for all disk access.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Currently /var was always backed by /var/tmp, but we may want to
control exactly what it is backed by. The default is the same, so
this is not a behavioral change.
The dnf stage wants to import `osbuild.sources` but currently the
osbuild module is not available in the stages. Apply the same hack
done in the Assembler also in for the stages, i.e. bind mount the
osbuild module to the stages/osbuild.
This happens rarely when the same loop device is used in rapid
succession. The kernel flushes the page cache asynchronously, which
means that it might not be cleared yet when a new file is bound.
`set_status` checks if the cache is clear (`set_fd` doesn't).
Handle this by trying a different device when `set_status` returns
`EBUSY`.
Fixes#177
Don't wait until python's garbage collector closes the file descriptors
to loop devices. Close them when the `LoopServer` context manager exits,
after an assembler has finished running.
The z Initial Program Loader (zipl) when creating the bootmap in
bootmap_creat (src/zipl/bootmap.c) wants to create a device node
via misc_temp_dev (bootmap_create:1141) for the device that it
is installing the bootloader to[1]. Currently access to loopback
devices is allowed from within the container (it is used to mount
the image), but only read/write access. On s390x also allow the
creation of device nodes, so zipl can do its work and install
the bootloader stages on the "disk".
[1] zipl source at commit dcce14923c3e9615df53773d1d8a3a22cbb23b96
Add a new command line option `--secrets`, which accepts a JSON file
that is structured similarly to a source file. It is should contain data
that is necessary to fetch content, but shouldn't appear in any logs.
This might (hopefully) fix a race in destructing the asyncio.EventLoop
that's used in all API classes, which leads to warnings about unhandled
exceptions on CI.
This also puts their creation closer to where the client-side sockets
are created.
Pipelines encode which source content they need in the form of
repository metadata checksums (or rpm checksums). In addition, they
encode where they fetch that source content from in the form of URLs.
This is overly specific and doesn't have to be in the pipeline's hash:
the checksum is enough to specify an image.
In practice, this precluded using alternative ways of getting at source
packages, such as local mirrors, which could speed up development.
Introduce a new osbuild API: sources. With it, a stage can query for a
way to fetch source content based on checksums.
The first such source is `org.osbuild.dnf`, which returns repository
configuration for a metadata checksum. Note that the dnf stage continues
to verify that the content it received matches the checksum it expects.
Sources are implemented as programs, living in a `sources` directory.
They are run on the host (i.e., uncontained) right now. Each source gets
passed options, which are taken from a new command line argument to
osbuild, and an array of checksums for which to return content.
This API is only available to stages right now.
The recent changes removed the {Assembler,Stage}Failed exceptions,
which includes them being thrown from Stage.run and Assembler.run.
Instead result dictionaries are returned even on errors. But the
object store, used as a context manager, relies on exceptions to
detect the error case and thus needs them to cleanup the temporary
objects. Without those exceptions the temporary objects end up in
the store even when the sage or assembler failed.
Restore the old behavior by throwing a generic BuildError exception
from the Stage and Assembler, which will be caught directly in the
pipeline and converted to a result dict.
The socket that the osbuild and loop apis should talk on are passed into
their `__init__` function. The caller should be responsible for closing
those sockets.
This already happens in all current callers.
This fixes a non-fatal error on RHEL's python 3.6, because it was
calling `socket.close` on an already-closed socket:
Traceback (most recent call last):
File "/usr/lib64/python3.6/asyncio/base_events.py", line 529, in __del__
self.close()
File "/usr/lib64/python3.6/asyncio/unix_events.py", line 63, in close
super().close()
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 99, in close
self._close_self_pipe()
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 109, in _close_self_pipe
self._remove_reader(self._ssock.fileno())
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 268, in _remove_reader
key = self._selector.get_key(fd)
File "/usr/lib64/python3.6/selectors.py", line 189, in get_key
return mapping[fileobj]
File "/usr/lib64/python3.6/selectors.py", line 70, in __getitem__
fd = self._selector._fileobj_lookup(fileobj)
File "/usr/lib64/python3.6/selectors.py", line 224, in _fileobj_lookup
return _fileobj_to_fd(fileobj)
File "/usr/lib64/python3.6/selectors.py", line 41, in _fileobj_to_fd
raise ValueError("Invalid file descriptor: {}".format(fd))
ValueError: Invalid file descriptor: -1
Commit 82a2be53d introduced a new return type from `Pipeline.run()`. It
changed the caller in `__main__.py`, but missed that the build pipeline
uses the same function.
A pipeline run only returned logs in the `StageFailed` and
`AssemblerFailed` exceptions. Remove those and always return structured
data instead.
It only returns data for stages that actually ran (i.e., didn't come
from the cache). This is similar to the output in interactive mode.
Also change osbuildtest to be able to deal with output that is larger
than the pipe buffer by using subprocess.communicate().
The workaround of manually linking /lib64 -> /usr/lib64 inside the
container that is needed on s390 is also required on ppc64 because
here the dynamic linker is set to /lib64/ld64.so.2 and the /lib64
link is not created.
Work around a combination of systemd not creating the link from
/lib64 -> /usr/lib64 (see systemd issue #14311) and the dynamic
linker is being set to (/lib/ld64.so.1 -> /lib64/ld64.so.1)
Therefore we manually create the link before calling nspawn
osbuild currently throws an error when not passing a build environment
on the command line, because the runner is unset. This is annoying on
hosts which only need a runner set, but no build pipeline.
To simplify running osbuild in this common case, introduce
`org.osbuild.host`, which is a runner that is defined to work on the
host that osbuild is installed on. Use this runner by default and
include a symlink to the right runner in the Fedora and RHEL packages.
Also add `runners/org.osbuild.host` to `.gitignore`, so that developers
can set the symlink when running osbuild from the source directory.
Fixes#171
We've been using a generic `osbuild-run`, which sets up the build
environment (and works around bugs) for all build roots. It is already
getting unwieldy, because it tries to detect the OS for some things it
configures. It's also about to cause problems for RHEL, which doesn't
currently support a python3 shebang without having /etc around.
This patch changes the `build` key in a pipeline to not be a pipeline
itself, but an object with `runner` and `pipeline` keys. `pipeline` is
the build pipeline, as before. `runner` is the name of the runner to
use. Runners are programs in the `runners` subdirectory.
Three runners are included in this patch. They're copies of osbuild-run
for now (except some additions for rhel82). The idea is that each of
them only contains the minimal setup code necessary for an OS, and that
we can review what's needed when updating a build root.
Also modify the `--build-pipeline` command line switch to accept such a
build object (instead of a pipeline) and rename it accordingly, to
`--build-env`.
Correspondingly, `OSBUILD_TEST_BUILD_PIPELINE` → `OSBUILD_TEST_BUILD_ENV`.
`osbuild-run` sets up the build root so that programs can be run
correctly in it. It should be run for all programs, not just stages and
assemblers (even though they're the only consumers right now).
Also, conceptually, `osbuild-run` belongs to the build root. We'll
change its implementation based on the build root in a future commit.
The buildroot already sets up `/run/osbuild/api`. It makes sense to have
it manage libdir as well.
A nice side benefit of this is a simplification of the Stage and
Assembler classes, which grew quite complex and contained duplicate
code.
Use the new the osbuild API to setup the standard input/output
inside the container, i.e. replace stdin, stdout, and stderr with
sockets provided by the host.
Introduce an osbuild API that can be used by the container to talk
to the osbuild host. It currently supports one method 'setup-stdio'
which should be used by the container to setup its standard input/
output so the stages can transparently do i/o with the osbuild host
via stdio.
The input data (args) is written to a temp-file backed buffer. The
output is either the host's stdout directly or another temp-file
backed buffer; the latter is re-opened (via /proc/self/fd) to get
another file-descriptor for the container, so in theory the host
and the container could do i/o to the same buffer independently.
