Keep track of all created objects via weak references. Add support
to use ObjectStore as context manager and ensure that all objects
are cleaned up when the context is exited.
Instead of creating temporary directories at the root of the store
create them in a sub-directory called 'tmp'. This should make it
easy to cleanup left-over (temporary) dirs in case of crashes.
Additionally, it has the nice side effect that it is possible to
check that there are no objects that are still in-flight, i.e. not
cleaned-up.
Turn `ObjectStore.new` into a plain method, since `Object` itself
can be used as a context manager, which is now directly returned,
instead of internally wrapped in a `with` statement and then
yielded. Thus for callers of the method nothing changes and the
behavior of `with objectstore.new() as x` is exactly the same.
This rips out the `PIPELINE` section from osbuild(1) and instead adds a
new osbuild-manifest(5) man-page. This new man-page contains a rather
formal definition of the manifest, with a separate section for each
part of a manifest.
The man-page is exhaustive, in that it describes all available options.
However, it does *NOT* document the available stages, runners, and
assemblers. It does document the available (and supported) sources.
This should serve as an example how to document available stages and
assemblers in the future.
Note that it is not clear whether we should document these right now.
Once we decided to support the available stages for a reasonable
time-frame, we can start on documenting them as well.
This adds a non-binding, documentational-only json-schema to
schemas/osbuild1.json which describes the format of the pipeline
manifest taken as input to osbuild. This is currently for
documentational purposes, but is definitely open to be used for actual
runtime verification.
The manifest does not describe options to assemblers, stages, or
sources. These are left as arbitrary json-objects and need separate
validation, if required. Note that most stages already contain an
embedded schema for their parameters.
This extends the EXAMPLES section with more examples, reduces their
complexity, and also restructures the layout to make it nicer to read
in the resulting TROFF file.
We will mention this example in our man-page, so make sure it actually
works. This imports all sources into the pipeline definition and
adjusts the syntax to match what we expect.
It is not used anywhere and might be confusing to newcomers, because it
only contains the (private) osbuild library, without the command line
tool and runners/stages/assemblers.
Prior to this patch, `make rpm` would produce rpms that have the latest
tag as their versions. This was confusing, because one could never know
which contents are in a locally built rpm.
Change this so that the is version always based on the commit hash of
HEAD. This is easy: the golang macros read a `%commit` macro when it
exists and do this for us.
To simplify more, only define `%_topdir` to ./rpmbuild and use
rpmbuild's known directory structure (SPEC, SOURCES, RPMS, ...)
otherwise, to make it easier to find build results.
Build the specfile, tarball, source rpms, and rpms with `make rpm`,
without separate sub-targets. We can reintroduce them if they're needed
somewhere.
The rule for the tag name is to have a 'v' prefix, update the
release instruction to reflect that. Also remove the Packit
header, because we don't use Packit at all anymore.
The MANIFEST.in file is used to include `LICENSE` in our python source
distribution. However, we never use this. In our spec-files we use the
rpm `%license` macro which copies from the source checkout, anyway.
Lets drop this file so we can forget about it.
Only commit checkpoints to the object store if the run of the
stage or assembler was successful. Otherwise we commit a empty,
corrupted or old tree to the store. Any subsequent run might
then pick up that bogus tree as a starting point.
We already have a makefile for maintenance work. Merge the
`bump-version.sh` script into `Makefile` as a new target so we avoid
placing scripts spread around the repository.
While at it, this commit also uses immediate assignment for
shell-evaluated make variables. This ensures the shell commands are
only executed once, and then are guaranteed to be of the same value for
the remainder of the execution.
When marking stages for checkpointing, let us make use of the local set
datastructure we already allocate, rather than iterating over it
linearly.
Apart from the negligible performance improvement, it makes the code
quite a lot simpler.
We generally surround function definitions with newlines. Make sure
this is also true for local function definitions.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
Even on ubuntu we can build rpm-based pipelines without bootstrapping
via fedora 27. Drop the build env from the travis config and from our
samples directory.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Using a metalink resolves to a specific mirror at runtime, and
downloads each rpm from that repository.
We want to move to using the org.osbuild.files source, which means
that we must save the url to each rpm in the source definition, which
will be determined by which mirror is used to generate the config.
If we use metalinks to generate the source configuration, the mirror
used will be arbitrary. Instead, we want to pick the best mirror
explicitly, ideally in a way that is independent of the location
depsolving happens in (which will be different from the location
the rpms are downloaded to).
We can choose explicitly by passing baseurl rather than metalink
to dnf, so move in that direction now by replacing all metalinks
by baseurls in our dnf configuration.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We now support sources and pipelines being passed to osbuild as one.
This will make the transformation from dnf to rpm stage simpler, as
the source object will then be different for each stage, so having
a shared one as now would be cumbersome.
Signed-off-by: Tom Gundersen <teg@jklm.no>
As long as this matches the build environment, this does not make
a differenece, but let us not depend on this.
This will be useful when automatically transforming dnf to rpm
pipelines, as the platform_module_id is needed as input to
osbuild-composer's dnf-json tool.
Performed using this script:
```
cat $1 | jq '(.stages[]? | select(.name == "org.osbuild.dnf") | .options.module_platform_id) |= . + "platform:f30"' | sponge $1
cat $1 | jq '(.build.pipeline.stages[]? | select(.name == "org.osbuild.dnf") | .options.module_platform_id) |= . + "platform:f30"' | sponge $1
```
Signed-off-by: Tom Gundersen <teg@jklm.no>
With the introduction of man-pages, this now moves the usage
information, examples, and pipeline descriptions into the man-page. With
this out of the way, the README is restructured with information on how
to install the project, what dependencies exist, and directions on
where to find further information or contact upstream.
Signed-off-by: David Rheinsberg <david.rheinsberg@gmail.com>
This adds a man-page `osbuild(1)` which documents the `osbuild`
executable. We use the python-native doc-utils and rely on
reStructuredText as documentation language.
While it is common with python libraries to use sphinx for documenting
the project, this commit explicitly does not import a full sphinx
documentation. Instead, it only adds the `rst` sources for man-pages.
These can be built by distributions via a simple:
rst2man docs/<manpage>.rst docs/<output-file>
This modifies the help-strings for parameters in `osbuild --help`.
Rather than using the identifier to describe its purpose, make it
describe its type. That is, this changes:
--sources=SOURCES => --sources=FILE
The option-name should already describe the purpose, so lets use the
argument-name for the type. This also improves on the stuttering when
reading the output.
We already do that for options that take directories as arguments. For
some reasons, we did not do that for options that take file-paths.
It is arguable whether this should be `PATH` or `FILE`. The latter has
the advantage that it makes clear that it is not a directory. It should
be obvious that `FILE` allows all kinds of paths.
Lastly, this does not update the positional arguments (in our case just
`PIPELINE`), since I did not conclude on the best way to make it
self-documenting. `PIPELINE-FILE` sounds convoluted.
This adds a new runner for Arch Linux. For now this simply links to the
blank linux runner, which works perfectly fine to bootstrap more
complex build pipelines.
Note that if we ever end up with more complex pipelines native to Arch
Linux, we might have to update this runner as well, since even on Arch
/etc must be pre-populated. Regardless, the blank linux runner serves
as a nice base and allows us to easily bootstrap osbuild on foreign
distros.
Now with `os-release` using `linux` as default ID+VERSION string, we
have a proper fallback name for our blank runner. Rename the blank
runner to `org.osbuild.linux`. It now serves as default fallback for
anything not further specified.
This adds a new runner called `org.osbuild.blank`, which assumes /usr
is pre-populated and ready to go. It does not perform any OS setup. It
only initializes the environment and executes the stage.
This runner allows easy bootstrapping of new systems. It assumes our
ideal setup where `/usr` describes a host system in its entirety,
without any local policy applied. Thus, this runner is also what we
ultimately want to work towards as a default. This might not happen
anytime soon, though, given how `passwd`, `ldconfig`, `nss`, etc. still
depend on prepopulated caches in `/etc`.
This adds two more os-release tests. One contains an empty os-release
file. We verify it is correctly parsed and ends up with the default
value.
The second one is the official Arch Linux os-release file. We verify
that we correctly end up with the rolling-release name.
Rename the function to `describe_os()`. We do no actual detection, nor
verification here. That is, the return value of this function is in no
way guaranteed to be a valid runner. That is, error-handling needs to
be done in the caller. Make this clear by renaming the function.
Note: Currently, in case no runner exists for the OS, we end up with:
execv(...) failed: No such file or directory
This needs to be fixed in the future.
Reading from an `Object` via `read` already uses a context manager
to manage the read-only bind mount and also maintain a count of
currently active readers. With this an attempt to start a new
`write` operation while readers were active can be detected and
an exception is throw. Since `write` was not introducing a context
the inverted situation, i.e. reads while a write is ongoing, was
not possible to detect.
This commit therefore introduces a context also for `.write` so
that we can enforce the policy to have either many readers but no
writers, or just one writer and no readers.
A bind mount is also used for write (in read-write mode) to hide
the internal path of the tree.
The exception that was thrown by {stage.run, assembler.run} was a
necessary ingredient that in combination with the context manager
around `Objectstore.new` made sure that tree the object was only
auto-committed to the store when there was no error during the
executing of any of the `.run` methods.
Now that the auto-commit feature got removed and committing of any
object to the store is explicitly done via `objectstore.commit`,
the whole exception throwing and handling can be removed. Status
reporting was already done in `BuildResult.success` and the new
code will use that to exit the function early on stage/asm errors.
Verify that `Object` can have multiple readers but as long as at
least one reader is active, no one can write to it. Also checks
that after `Object` has left the context it is not writable anymore.
Verify the copy on write semantics of `objectstore.Object`, i.e.
content will only be copied at the moment a client wants to write
to `Object`. This also checks that `Object.base` works.
Modify the CI to execute the unit tests in a privileged container
because `Object.read()` works internally by bind mounting a path.
The mount operation needs at least CAP_SYS_ADMIN and overwriting
the file permissions CAP_DAC_OVERRIDE.
Refactor `get_buildtree` to do input/output via `Object`, i.e. by
creating a new `Object`, setting its base accordingly and then
use its `read` and `write` methods. This is what `ObjectStore.get`
does as well.
In the case that there is no build pipeline, use the mount helpers
of `objectstore` instead of the custom mount calls.
Do not automatically commit the last stage of the pipeline to the
store. The last stage is most likely not what should be cached,
because it will contain all the individual customization and thus
be very likely different for different users. Instead, the dnf or
rpm stages have a higher chance of being the same and thus are
better candidates for caching.
Technically this change is done via two big changes that build
upon new features introduces in the previous commits, most notably
the copy on write semantics of Object and that input/output is
being done via `objectstore.Object` instead of plain paths. The
first of the two big changes is to create one new `Object` at
the beginning of `pipeline.run` and use that, in write mode via
`Object.write` across invocations of `stage.run` calls, with
checkpoints being created after each stage on demand.
The very same `Object` is then used in read mode via `Object.read`
as the input tree for the Assembler. After the assembler is done
the resulting image/tree is manually committed to the store.
The other big change is to remove the `ObjectStore.commit` call
from the `ObjectStore.new` method and thus the automatic commit
after the last stage is gone.
NB: since the build tree is being retrieved in `get_buildtree`
from the store, a checkpoint for the last stage of the build
pipeline is forced for now. Future commits will refactor will
do away with that forced commit as well.
Change osbuildtest.TestCase to always create a checkpoint at
the final tree (the last stage of the pipeline), since tests
need it to check the tree contents.
