A single workflow allows us to re-use artifacts throughout all jobs.
Also, we can fail the tests early if there's a linting issue before
we spend time waiting for RPMs to build.
Signed-off-by: Major Hayden <major@redhat.com>
By default, GitHub Actions stops running all RPM builds if one fails.
This means that a temporary repository issue (like what is happening
with F32 today) stops all RPM builds.
Set the `fail-fast` option to `false` to disable this behavior.
Signed-off-by: Major Hayden <major@redhat.com>
Make RPM building more similar to how RPMs will be built in koji.
This downloads the specfile from github at the given commit, appends
the commit sha to the specfile and uses spectools to download the
correct sources from github.
This should be equivalent to what is done in the makefile, the only
behavioral difference is that the rpms are now versioned based on
the git sha they are built from.
The main purpose of this change is to avoid any differences between
the CI and the real RPMs due to bugs in the Makefile. Correctly
versioned RPMs will also be handy for testing/debugging/deployment.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Now that RPMs are built natively in Fedora containers via GitHub
Actions, we no longer need to build them in Travis CI.
Signed-off-by: Major Hayden <major@redhat.com>
This changes the name of the source tarball, but apart from that has
no effect as-is.
However, if the `commit` global is set, then the source tarball is
expected to be the standard git-archive at that commit, and the
release field is set to contain the date and the git sha. This is
useful for when we want to build rpms between releases.
This works analogously to how osbulid-composer currently works.
See <https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation>
for details.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Building rpm directly on Ubuntu is somewhat possible but it's fragile and
it cannot cover all quirks of rpm build. This commit switches the rpm build
tests to using fedora containers. This isn't also the best solution but
imho it's much better than the previous one.
Enabling the fastestmirror plugin allows dnf to choose the fastest
(also usually the closest) mirror to the instance of osbuild. It
has no effect on builds that force the use of a specific server
or mirror.
Signed-off-by: Major Hayden <major@redhat.com>
Currently stdin is taken to be the pipeline to be built, this allows
it to be instead a map containing the suorces and the pipeline.
We would imagine passing around the sources and pipeline together, so
this just makes the behavior of osbuild more closely match the intended
use and semantics of the sources configuration.
This keeps backwards compatibility for now, but that may be dropped as
soon as osbuild-composer no longer relies on the old behavior.
Disable too-many-{branches,statements} pylint warnings in __main__.py.
These do not seem helpful, but could be reenabled if we drop some
options in the future.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Make the sources options a static property of the pipeline, in
particular of each stage, rather than being passed in on `run()`.
This more closely matches the intended semantics of sources and
pipeline having similar lifetimes and being fairly coupled together.
The difference between the pipeline and the sources is that the
sources do not contribute to identifying the pipeline (they are not
part of the hash for the pipeline id), and they could be swapped
out without changing the output image (as long as they are valid).
However, a pipeline without A sources object would not be useful,
and typically the pipeline and the sources are generated, passed
around and used together.
This is different from the build environment and the secrets object,
which both are specific to either the host or the caller, unlike
the pipeline which should be universal.
This changes the `load()` function to take a `manifest`, which is
a map containing both the pipeline and the sources.
Note that the semantics of the build-env parameter remains unchanged:
It shares the sources with the rest of the pipeline. We may want to
reconsider this in future commits, as the build-env is specific to
the host, whereas the regular pipeline is not.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The dnf stage was very verbose, probably overly so. Rather than
benig completely silent, let's print basic progress indication
so we have some idea what's going on.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Appart from giving us a hard time on s390x, this feature did not seem
to have a measurable effect. Moreover, O_DIRECT is not supported by
tmpfs so without this patch we could not use tmpfs as backing store,
which does speed up image generation considerably.
Drop the flag and and rather put the store on tmpfs in order to speed
things up.
Drop the rpm downloading and instead use the files source. This gives
us caching for free, and is the last missing step before we can
deprecate the dnf stage.
The main benefit of the rpm over the dnf stage is that we pin the package
versions rather than the repo metadata version. This will allow us to
support continuously changing repositories as individual packages are much
less likely to change than the repos iteself, and old packages are meant
to stay around for some time, unlike the repo metadata which is instantly
swapped out.
Depsolving is also slow on the first run, which we were always hitting as
the depsolving was always happening in a fresh container.
Based on a patch by Lars Karlitski.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This source adds support for downloaded files. The files are
indexed by their content hash, and the only option is their URL.
The main usecase for this will be downloading rpms. Allowing depsolving
to be done outside of osbuild, network access to be restricted and
downloaded rpms to be reused between runs.
Each source is now passed two additional arguments, a cache directory
and an output directory. Both are in the source's namespace, and
the source is responsible for managing them. Each directory may
contain contents from previous runs, but neither is ever guaranteed
to do so.
Downloaded contents may be saved to the cache and resued between
runs, and the requested content should be written to the output dir.
If secrets are used, the source must only ever write contents to
the output that corresponds to the available secrets (rather than
contents from the cache from previous runs).
Each stage is passed an additional argument, a sources directory.
The directory is read-only, and contains a subdirectory named after
each used source, which will contain the requseted contents when
the `Get()` call returns (if the source uses this functionality).
Based on a patch by Lars Karlitski.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Add a new `--checkpoint` option, which can be provided multiple
times, that indicate after which stages a the current stage of
the tree should be committed to the object store; the tree id
will be the treesum of the tree at that point and a reference
is created with the id of the stage at the point.
The argument to `--checkpoint` is the id of the stage. If not
all the given checkpoints can be found the execution will be
aborted.
Check the very basic operations of the object store, i.e. creating
multiple trees with different object_ids works and results in the
correct number of objects and references.
When the tree is committed to the objects directory of the object-
store, it is done via rename(3). The two possible errors that can
be raised in case that a non-empty tree with the same name already
exist is [EEXIST] or [ENOTEMPTY]. The latter was already ignored
but the former was not. At least on btrfs former will be raised
File "/home/gicmo/Code/src/osbuild/osbuild/objectstore.py",
os.rename(tree.root, output_tree)
FileExistsError: [Errno 17] File exists: 'store/tmpyyi3yvie/tree' -> 'store/objects/…'
Add a new method to the ObjectStore that takes a path to a file
system tree, which is currently being built, and commits it to
the store and references it via a given object_id.
The tree is copied to a temporary location (co-located in the
store to enable fast copying via reflinks) and then atomically
moved into the ObjectStore's objects path via rename(3).
Extract the code from ObjectStore.new that will commit the filled
tree to the store into its own method so it can be used from a
future method to snapshot trees at random points in time.
Introduce a small `TreeObject` class that is the representation of
a tree during its construction. It supports calculating its treesum
as well initialize the new tree with an existing one.
This makes sure all disk access is backed by the same disk. We may
want this for performance reasons (avoiding moving across disks), but
also to experiment with different backing stores for all disk access.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Currently /var was always backed by /var/tmp, but we may want to
control exactly what it is backed by. The default is the same, so
this is not a behavioral change.
The dnf stage wants to import `osbuild.sources` but currently the
osbuild module is not available in the stages. Apply the same hack
done in the Assembler also in for the stages, i.e. bind mount the
osbuild module to the stages/osbuild.
