Add a new output-directory argument which specifies where to store
result objects. For now, this is purely optional and simply copies from
the old `output_id` into the specified directory. This allows a
backwards compatible transition towards removing any external access to
the osbuild cache.
Note that this has still lots of room for improvements:
* We only support assembler-output for now, but we could also easily
support entire trees as output, in case no assembler was selected.
Alternatively, we could introduce a "copy" assembler, that just
outputs the input tree.
* This parameter is optional, but should really be mandatory. There
is little reason to have the default behavior just dropping any
generated content. This would be a breaking change, though.
* We could move data out of a temporary object-store entry, rather
than copy it. But again, for backwards-compatibility, we leave the
latest store-object intact and do not move things out of it.
* We could now transition towards never committing anything to the
store, not even output IDs, unless explicitly checkpointed.
Add /boot to be mounted from the build tree into the build root,
because the EFI binaries for grub are stored in there and for
ostree grub2 support those need to be copied too.
Add a small wrapper around the setfiles(8) utility that can be used
to set the security context fields on one or multiple provided paths,
given a specification. The root of the file system tree can be given
via `root` and all elements of `paths` will be interpreted as
relative to that root.
Add a helper, `parse_config`, to parse a selinux configuration file,
see selinux(8), and return a dictionary containing the configuration
data in key, value pairs. This, in turn, can be fed into the other
helper method, `config_get_policy`, to get the effective policy or
`None` if SELinux is disabled or the policy type is not configured.
Add a new test suite that checks the basic functionality of the
helpers above.
When using rpm-ostree compose, a Treefile[1] controls various
aspects of its behaviour. Since rpm-ostree will, at least in
the beginning, be used to post-process and committing the tree
add a helper class to ease the creation of correct Treefiles.
The docstring of the Treefile contains the information in which
phases ('install', 'postprocess', 'commit') the option is used,
as of rpm-ostree commit 1cf0d557ae8059e689b1fed670022727e9842288
Add basic checks for the ostree.Treefile helper. Some of the
tests require rpm-ostree to be installed.
[1] https://rpm-ostree.readthedocs.io/en/stable/manual/treefile/
Causes a problem with ostree-osbuild on CI (travis) otherwise:
Traceback (most recent call last):
File "osbuild-ostree", line 345, in <module>
sys.exit(main())
File "osbuild-ostree", line 337, in main
return build(args)
File "osbuild-ostree", line 257, in build
output_id, commit_id = build_commit(builddir, args)
File "osbuild-ostree", line 162, in build_commit
r = pipeline.run(store.store,
File "/home/travis/build/gicmo/ostree-osbuild-demo/osbuild/osbuild/pipeline.py", line 358, in run
r = self.assemble(object_store,
File "/home/travis/build/gicmo/ostree-osbuild-demo/osbuild/osbuild/pipeline.py", line 314, in assemble
r = self.assembler.run(input_dir,
File "/home/travis/build/gicmo/ostree-osbuild-demo/osbuild/osbuild/pipeline.py", line 148, in run
osbuild_module_path = os.path.dirname(importlib.util.find_
Move the whole result handling of the assembler outside the context
manager; this includes the cleanup of the object in the error case
which would conflict with the ongoing write operation inside the
context manager and thus lead to a crash:
Traceback (most recent call last):
File "/usr/bin/osbuild", line 11, in <module>;
load_entry_point('osbuild==10', 'console_scripts', 'osbuild')()
File "/usr/lib/python3.7/site-packages/osbuild/__main__.py", line 99, in main
secrets=secrets
File "/usr/lib/python3.7/site-packages/osbuild/pipeline.py", line 362, in run
libdir)
File "/usr/lib/python3.7/site-packages/osbuild/pipeline.py", line 324, in assemble
output.cleanup()
File "/usr/lib/python3.7/site-packages/osbuild/objectstore.py", line 160, in cleanup
self._check_writer()
File "/usr/lib/python3.7/site-packages/osbuild/objectstore.py", line 178, in _check_writer
raise ValueError("Write operation is ongoing")
ValueError: Write operation is ongoing
Instead of using the chattr binary, which adds another dependency
use what amounts to ioctl(fd, ,FS_IOC_SETFLAGS, ~FS_IMMUTABLE_FL),
to clear the immutable flag. Constants are taken from linux/fs.h.
The tree, which is created by stages and assemblers, might contain
immutable files, which for Python 3 currently (version 3.8) leads
to errors when the tempfile.TemporaryDirectory is being cleaned up.
Therefore, manually cleanup the tree directory, if it exists, via
shutil.rmtree with a custom onerror handler that also removes the
immutable bit on permission errors.
osbuild can now take only manifests as its input (the legacy input format
was dropped in e48c2f1). This commit changes all remaining occurrences of
"pipeline" to "manifest" when describing the osbuild input.
This drops support for passing in non-manifest style pipelines
directly. It used to be that we directly pass in the pipeline
description, but it got changed to a proper manifest format in:
commit e48c2f178c
Author: Tom Gundersen <teg@jklm.no>
Date: Thu Feb 13 17:44:54 2020 +0100
osbuild: allow the sources to be passed in on stdin
With 2 releases in between, we are now far enough to drop the old
format. All code has been converted, our API guarantee is not in place,
yet, so lets just drop the legacy code and fully commit to the
manifest.
Fixes#265.
If there is a build pipeline specified, always build it, even if
there are no accompanying stages. If we short-circuit earlier and
ignore the build pipeline section, errors in the build pipeline
would not be caught at all.
The `build_stages` method short-circuits and returns early in case
any of the stages fail to build and returns None for the tree, and
build tree, therefore both of those can immediately cleaned up at
that point.
For this add a small helper `cleanup` that will call the cleanup
method for all supplied arguments, after filtering out None values.
Delay the cleanup of the build tree of the build pipeline, and
first check the result and only cleanup the tree when the build
did not fail, because in that case both returned trees will be
None and trying to cleanup them up will result in an exception.
Therefore, also don't clean up `tree` in the error case.
If the final object, image, artifact, already exists in the store,
short-circuit and return directly from `Pipeline.run`. Otherwise
the situation might arise that the final result is in the store,
but the tree (and build trees) are not and thus the tree would be
built, just to be thrown away when the assembler phase detects
that the final output already exists.
Extract the code that assembles the tree into its own method as
it was previously done for the stages. This should make the new
method as well as `Pipeline.run` method easier to read.
Refactor the building of stages and the build tree so that no auto
commit is done at the end of the build pipeline anymore, i.e. the
respective build tree(s) are not commit to the store unless that
was explicitly enabled via a checkpoint.
NB: `objectstore.Object`s are used not via a context manager
anymore, because they are returned from the `build_stages` method
to make the code easier to use and read. Cleanup of Objects during
a KeyboardInterrupt exception (Ctrl-C) are handled by using the
ObjectStore with a context manager, which on exit of the context
will cleanup all objects. Due to a big in python[1] this is indeed
more robust than using `with object_store.new() as tree` because
that is translated[2] to something like:
1: mgr = (EXPR)
2: exit = type(mgr).__exit__
3: value = type(mgr).__enter__(mgr)
-> 4: # NOTE: KeyboardInterrupt here will "leak" value
5: try:
6: [...]
7: finally:
8: if exc:
9: exit(mgr, None, None, None)
Which can leave the tree initialized but not cleaned up if the
KeyboardInterrupt happens exactly line 4.
[1] https://bugs.python.org/issue29988
[2] https://www.python.org/dev/peps/pep-0343/
Simple new object that should expose the root file system with the
same API as `objectstore.Object` but as read-only. This means that
the `read` call works exactly as for `Object` but `write` raises
an exception.
Add tests to specifically check the read-only properties.
Keep track of all created objects via weak references. Add support
to use ObjectStore as context manager and ensure that all objects
are cleaned up when the context is exited.
Instead of creating temporary directories at the root of the store
create them in a sub-directory called 'tmp'. This should make it
easy to cleanup left-over (temporary) dirs in case of crashes.
Additionally, it has the nice side effect that it is possible to
check that there are no objects that are still in-flight, i.e. not
cleaned-up.
Turn `ObjectStore.new` into a plain method, since `Object` itself
can be used as a context manager, which is now directly returned,
instead of internally wrapped in a `with` statement and then
yielded. Thus for callers of the method nothing changes and the
behavior of `with objectstore.new() as x` is exactly the same.
Only commit checkpoints to the object store if the run of the
stage or assembler was successful. Otherwise we commit a empty,
corrupted or old tree to the store. Any subsequent run might
then pick up that bogus tree as a starting point.
When marking stages for checkpointing, let us make use of the local set
datastructure we already allocate, rather than iterating over it
linearly.
Apart from the negligible performance improvement, it makes the code
quite a lot simpler.
We generally surround function definitions with newlines. Make sure
this is also true for local function definitions.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
This modifies the help-strings for parameters in `osbuild --help`.
Rather than using the identifier to describe its purpose, make it
describe its type. That is, this changes:
--sources=SOURCES => --sources=FILE
The option-name should already describe the purpose, so lets use the
argument-name for the type. This also improves on the stuttering when
reading the output.
We already do that for options that take directories as arguments. For
some reasons, we did not do that for options that take file-paths.
It is arguable whether this should be `PATH` or `FILE`. The latter has
the advantage that it makes clear that it is not a directory. It should
be obvious that `FILE` allows all kinds of paths.
Lastly, this does not update the positional arguments (in our case just
`PIPELINE`), since I did not conclude on the best way to make it
self-documenting. `PIPELINE-FILE` sounds convoluted.
Rename the function to `describe_os()`. We do no actual detection, nor
verification here. That is, the return value of this function is in no
way guaranteed to be a valid runner. That is, error-handling needs to
be done in the caller. Make this clear by renaming the function.
Note: Currently, in case no runner exists for the OS, we end up with:
execv(...) failed: No such file or directory
This needs to be fixed in the future.
Reading from an `Object` via `read` already uses a context manager
to manage the read-only bind mount and also maintain a count of
currently active readers. With this an attempt to start a new
`write` operation while readers were active can be detected and
an exception is throw. Since `write` was not introducing a context
the inverted situation, i.e. reads while a write is ongoing, was
not possible to detect.
This commit therefore introduces a context also for `.write` so
that we can enforce the policy to have either many readers but no
writers, or just one writer and no readers.
A bind mount is also used for write (in read-write mode) to hide
the internal path of the tree.
The exception that was thrown by {stage.run, assembler.run} was a
necessary ingredient that in combination with the context manager
around `Objectstore.new` made sure that tree the object was only
auto-committed to the store when there was no error during the
executing of any of the `.run` methods.
Now that the auto-commit feature got removed and committing of any
object to the store is explicitly done via `objectstore.commit`,
the whole exception throwing and handling can be removed. Status
reporting was already done in `BuildResult.success` and the new
code will use that to exit the function early on stage/asm errors.
Refactor `get_buildtree` to do input/output via `Object`, i.e. by
creating a new `Object`, setting its base accordingly and then
use its `read` and `write` methods. This is what `ObjectStore.get`
does as well.
In the case that there is no build pipeline, use the mount helpers
of `objectstore` instead of the custom mount calls.
Do not automatically commit the last stage of the pipeline to the
store. The last stage is most likely not what should be cached,
because it will contain all the individual customization and thus
be very likely different for different users. Instead, the dnf or
rpm stages have a higher chance of being the same and thus are
better candidates for caching.
Technically this change is done via two big changes that build
upon new features introduces in the previous commits, most notably
the copy on write semantics of Object and that input/output is
being done via `objectstore.Object` instead of plain paths. The
first of the two big changes is to create one new `Object` at
the beginning of `pipeline.run` and use that, in write mode via
`Object.write` across invocations of `stage.run` calls, with
checkpoints being created after each stage on demand.
The very same `Object` is then used in read mode via `Object.read`
as the input tree for the Assembler. After the assembler is done
the resulting image/tree is manually committed to the store.
The other big change is to remove the `ObjectStore.commit` call
from the `ObjectStore.new` method and thus the automatic commit
after the last stage is gone.
NB: since the build tree is being retrieved in `get_buildtree`
from the store, a checkpoint for the last stage of the build
pipeline is forced for now. Future commits will refactor will
do away with that forced commit as well.
Change osbuildtest.TestCase to always create a checkpoint at
the final tree (the last stage of the pipeline), since tests
need it to check the tree contents.
Instead of using custom bind-mount based logic in ObjectStore.get,
use a combination of Object + `Object.read` with the supplied base
(that can be None), which will lead to exactly the same outcome.
Provide a way to read the current contents of the object, in a way
the follows the copy-on-write semantics: If `base` is set but the
object has not yet been written to, the `base` content will be
exposed. If no base is set or the object has been written to, the
current (temporary) tree will be exposed. In either way it is done
via a bind mount so it is assured that the contents indeed can only
be read from, but not written to.
The code also currently make sure that there is no write operation
started as long as there is at least one reader.
Additionally, also introduce checks that the object is intact, i.e.
not cleaned up, for all operations that require such a state.
Analogous to `_path`, it is not possible to identify the intended
mode of the i/o operation from using `open` (whether it is a read
or a write operation) and thus make it an internal method and only
use it for read operations.
Since it is hard to infer the intended modus of the i/o operation,
i.e. whether it is going to be a read or a write from accessing the
`path` property make it an internal method. Do not initialize the
method on property access but return the writable tree, if Object
is initialized, the path to its base tree otherwise.
Adapt all the usage internally: Use `path` for read operations and
initialize the object and then directly use `_tree` for write ops.
As a result of the previous commits that implement copy on write
semantics, `commit` can now be used to create snapshots. Whenever
an Object is committed, its tree is moved to the store and it is
being reset, i.e. a new clean workdir is created and the old one
discarded. The moved tree is then set as the base of the reset
Object. On the next call to `write` the moved tree will be copied
over and forms the basis of the Object again. Should nobody want
to write to Object after the snapshot, i.e. the `commit`, no copy
will be made.
NB: snapshots/commits will act now act as synchronization points:
if a object with the same treesum, i.e. the very same content
already exists, the move (i.e. `store_tree`) will gracefully fail
and the existing content will be set as the base for Object.
Since Object knows its base now, the initialization of the tree
with the content of its base can be delayed until the moment
someone wants to actually modify the tree, thus implementing
copy on write semantics. For this a new `write` method is added
that will initialize the base and return the writable tree. It
should be used instead of `path` whenever the a client wants to
write to the tree of the Object.
Adapt the pipeline and the tests to use the new `write` method
in all the appropriate places.
NB: since the intention can not be inferred when using `path`
directly, the Object is still being initialized there.
When a new Object is created it can have a `base`, i.e. another
object that is already committed to the store, which is then used
to initialize the tree of the new object. That is, the contents
of the new Object will be based on the contents of the existing.
The initialization of an Object with its base (if any) was done
by the ObjectStore. Move all of that logic inside `Object`:
The Object will store its base, which `Object.init` will use to
initialize itself. Additionally, if `Object.path` is accessed
`init` is being called as well to make sure it is properly
initialized, i.e. the tree initialized with the base content.
Refactor the `ObjectStore.snapshot` method to take an `Object` not
a plain filesystem tree, so the latter is more encapsulated from
the ObjectStore user (e.g. the pipeline) and prepares a unified
code-path for `snapshot` and `commit` in the future.
Now that Object manages its work directory itself, re-create the
latter when the its tree is moved, i.e. when the object is being
committed to the store. This means that after the object has been
written to the store it is in the same state is if it was new and
can be used in the very same way.
If the move itself fails (the rename(2) fails), the tree and its
contents is cleaned up with the reset of the work directory.
Rename the `move` method to `store_tree` to better reflect how the
method should be used, i.e. to store the tree corresponding to the
Object instance.
When a new object is being created, a corresponding temporary
directory is created within the file system tree of the object
store, which shall be called the "work dir". Within that dir a
well-known directory ('tree') is created that then is the root
of the filesystem tree[1] that clients use to store the tree
or the resulting image in.
Previously, the work dir was managed, i.e. created and cleaned
up (via a context manager) by the ObjectStore. Now the Object
itself manages the tree and thus the lifetime of the work dir
is more directly integrated and controlled by it. As a result
the Object itself is now a context manager. On exit of the
context the work dir is cleaned up.
[1] For the assembler this is the output directory that will
contain the final image.
Instead of just returning the path of the temporary object that is
created in .new() the actual instance of the new `Object` is being
returned, which can then provide a richer interface for clients
than a plain directory path.
Keep are reference to the parent store, which this object is tied
to. It is currently not yet used directly but is a preparation for
a closer Object and ObjectStore integration that will happen in
commits to follow.
As the name implies, the ObjectStore stores objects, which can be
trees but also everything an Assembler can make of the input tree,
like qcow2 images, tarballs and other non tree-like outputs.
Therefore rename the TreeObject to Object to better reflect that it
is representing any object, not only trees, in the store.
