Add a new assembler that takes a file system tree that is already
conforming to the ostree system layout[1], creates a new repository
in archive mode and commits the file system tree to it. Afterwards,
a reference is created with the value supplied in `ref`.
The repository is located at the `/repo` directory and additional
metadata is /compose.json which contain the compose information.
Currently uses rpm-ostree to do the actual committing. In the future
this might change to plain ostree.
[1] https://ostree.readthedocs.io/en/stable/manual/adapting-existing/
A stage that uses rpm-ostree compose to do post-processing of the
tree. Among other things the main steps are:
- moves /etc to /usr/etc
- move /boot to /usr/lib/ostree-boot
- potentially moving /var/lib/rpm
- re-creates the initramfs (dracut)
- stores the treefile at /usr/share/rpm-ostree/treefile.json
- adds altfiles module to nss
- Recompiles SELinux policy (semodule -nB)
- Migrates /usr/etc/{passwd, group} to /usr/lib/
- Postprocess SELinux policy
- Convert /var to tmpfiles.d
- Prepares symlinks
- /usr/local -> /var/usrlocal
- /var/lib/alternatives -> /usr/lib/alternatives
- /var/lib/vagrant -> /usr/lib/vagrant
- copies the rpmdb
Based on commit 1cf0d557ae8059e689b1fed670022727e9842288 (rpm-ostree)
Add a small wrapper around the setfiles(8) utility that can be used
to set the security context fields on one or multiple provided paths,
given a specification. The root of the file system tree can be given
via `root` and all elements of `paths` will be interpreted as
relative to that root.
Add a helper, `parse_config`, to parse a selinux configuration file,
see selinux(8), and return a dictionary containing the configuration
data in key, value pairs. This, in turn, can be fed into the other
helper method, `config_get_policy`, to get the effective policy or
`None` if SELinux is disabled or the policy type is not configured.
Add a new test suite that checks the basic functionality of the
helpers above.
When using rpm-ostree compose, a Treefile[1] controls various
aspects of its behaviour. Since rpm-ostree will, at least in
the beginning, be used to post-process and committing the tree
add a helper class to ease the creation of correct Treefiles.
The docstring of the Treefile contains the information in which
phases ('install', 'postprocess', 'commit') the option is used,
as of rpm-ostree commit 1cf0d557ae8059e689b1fed670022727e9842288
Add basic checks for the ostree.Treefile helper. Some of the
tests require rpm-ostree to be installed.
[1] https://rpm-ostree.readthedocs.io/en/stable/manual/treefile/
Now that stages no longer access the network, drop CA certificate
setup.
In the future, we may want to restrict all network access to the
container, but that requires more work.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This has now been entirely replaced by the rpm stage in all its
users. The dnf stage does not fit very nicely into the osbuild
module, in particular it requires direct network access, which
we would like to avoid.
Signed-off-by: Tom Gundersen <teg@jklm.no>
For the sake of backwards compatibility, legacy support was enabled
by default. Flip this around, so that leaving the parameter out
means disabling it.
This is more intuitive, and will pave the way for dropping support
for the value being a bool in the future.
`osbuild-composer` always passes the argumnet explicitly, though
still always as a boolean.
Signed-off-by: Tom Gundersen <teg@jklm.no>
VHDX is the best format for uploading to AWS, thus this commit adds the
support for it. Pros over other formats supported by AWS:
- vmdk - doesn't work, qemu-img probably needs some special options
- vhd - the image size gets round up (I can get only a >=7GB volume from
a 6GB image)
- ova - just a wrapper over vmdk/vhd/vhdx adding some metadata
- raw - no compression, the images are huge
Also, the format specification is open, therefore I can't see any issues
with it.
Test a deployment of osbuild along with osbuild-composer for each PR.
More testing could be stacked on top.
Signed-off-by: Major Hayden <major@redhat.com>
Causes a problem with ostree-osbuild on CI (travis) otherwise:
Traceback (most recent call last):
File "osbuild-ostree", line 345, in <module>
sys.exit(main())
File "osbuild-ostree", line 337, in main
return build(args)
File "osbuild-ostree", line 257, in build
output_id, commit_id = build_commit(builddir, args)
File "osbuild-ostree", line 162, in build_commit
r = pipeline.run(store.store,
File "/home/travis/build/gicmo/ostree-osbuild-demo/osbuild/osbuild/pipeline.py", line 358, in run
r = self.assemble(object_store,
File "/home/travis/build/gicmo/ostree-osbuild-demo/osbuild/osbuild/pipeline.py", line 314, in assemble
r = self.assembler.run(input_dir,
File "/home/travis/build/gicmo/ostree-osbuild-demo/osbuild/osbuild/pipeline.py", line 148, in run
osbuild_module_path = os.path.dirname(importlib.util.find_
Make sure we prune the caches after each stage-test to keep our disk
footprint small. This does considerably increase build-times since we
no longer share cached entries. However, the current CI builds simply
run out of disk-space.
Once we use separate output-directories we will be able to drop the
automatic checkpointing from the tests, and thus effectively get the
same behavior. Until then, lets prune the caches explicitly.
The `osbuild` executable requires that the exact same version of the
corresponding python library is installed, but this was not enforced
in the RPM package. Thus a old version of osbuild could be installed
alongside an older version of python3-osbuild, which results in an
osbuild crash (see below).
Therefore, enforce that both installed packages have matching
versions by specifying the exact version for the `python3-osbuild`
dependency of the `osbuild` package.
Example osbuild crash if versions are not matching:
Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 583, in _build_master
ws.require(__requires__)
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 900, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (osbuild 10 (/usr/lib/python3.8/site-packages), Requirement.parse('osbuild==11'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/osbuild", line 6, in <module>
from pkg_resources import load_entry_point
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3252, in <module>
def _initialize_master_working_set():
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3235, in _call_aside
f(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3264, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 585, in _build_master
return cls._build_from_requirements(__requires__)
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 786, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'osbuild==11' distribution was not found and is required by the application
The sources.json file contains repositories for x86_64 and s390,
this patch adds the same for aarch64 Fedora 30 so that the example
pipeline can be build easily.
The "legacy" option in stages/org.osbuild.grub2 defaults to True:
`legacy = options.get("legacy", True)`
If both legacy and uefi is enable, the stage will try to create a hybrid
image, but that fails for aarch64. This patch makes sure we use uefi
mode only which fixed this issue.
I tested the patch by hand together with the fix from this PR:
https://github.com/osbuild/osbuild/pull/308
The GUID Partition Table (GPT) layout supports assigning UUIDs for
individual partitions. Add support for specifying those in the
partition description.
Move the whole result handling of the assembler outside the context
manager; this includes the cleanup of the object in the error case
which would conflict with the ongoing write operation inside the
context manager and thus lead to a crash:
Traceback (most recent call last):
File "/usr/bin/osbuild", line 11, in <module>;
load_entry_point('osbuild==10', 'console_scripts', 'osbuild')()
File "/usr/lib/python3.7/site-packages/osbuild/__main__.py", line 99, in main
secrets=secrets
File "/usr/lib/python3.7/site-packages/osbuild/pipeline.py", line 362, in run
libdir)
File "/usr/lib/python3.7/site-packages/osbuild/pipeline.py", line 324, in assemble
output.cleanup()
File "/usr/lib/python3.7/site-packages/osbuild/objectstore.py", line 160, in cleanup
self._check_writer()
File "/usr/lib/python3.7/site-packages/osbuild/objectstore.py", line 178, in _check_writer
raise ValueError("Write operation is ongoing")
ValueError: Write operation is ongoing
Instead of using the chattr binary, which adds another dependency
use what amounts to ioctl(fd, ,FS_IOC_SETFLAGS, ~FS_IMMUTABLE_FL),
to clear the immutable flag. Constants are taken from linux/fs.h.
The tree, which is created by stages and assemblers, might contain
immutable files, which for Python 3 currently (version 3.8) leads
to errors when the tempfile.TemporaryDirectory is being cleaned up.
Therefore, manually cleanup the tree directory, if it exists, via
shutil.rmtree with a custom onerror handler that also removes the
immutable bit on permission errors.
osbuild can now take only manifests as its input (the legacy input format
was dropped in e48c2f1). This commit changes all remaining occurrences of
"pipeline" to "manifest" when describing the osbuild input.
This drops support for passing in non-manifest style pipelines
directly. It used to be that we directly pass in the pipeline
description, but it got changed to a proper manifest format in:
commit e48c2f178c
Author: Tom Gundersen <teg@jklm.no>
Date: Thu Feb 13 17:44:54 2020 +0100
osbuild: allow the sources to be passed in on stdin
With 2 releases in between, we are now far enough to drop the old
format. All code has been converted, our API guarantee is not in place,
yet, so lets just drop the legacy code and fully commit to the
manifest.
Fixes#265.
Recent qemu version will warn with our current code:
qemu-system-x86_64: -accel kvm:hvf:tcg: Don't use ':' with -accel,
use -M accel=... for now instead
Since this might result in hard-errors, lets just follow the advice and
use the `-M` switch.
RPM is smart about dependencies - it goes over all shebangs of packaged files
and adds Requires to them. Prior this commit osbuild package depended on
python 3.6, because rhel runner has /usr/bin/python3.6 as shebang.
This commit fixes it by turning off dependency generators for our internal
stuff: assemblers, runners and stages. They run in a container, so I think
there's no reason to generate dependencies from them.
Now that containers are being built daily in GitHub Actions from
the container[0] repository, we can use those containers. That will
speed up the GitHub Actions for osbuild and reduce errors from
unreleased Fedora versions.
[0] https://github.com/osbuild/containers
Signed-off-by: Major Hayden <major@redhat.com>
Continue our effort to move to Github-actions. This imports the runtime
tests from Travis into Github-actions. The `test_boot` test is still
left on travis, since it requires stacked KVM, which is not yet
available on github-actions.
Drop the filter on `master`. This prevents us from running the CI on
forked repositories. In particular, it prevents us from pushing
branches to github-forks and have the CI run on them. This is very nice
to have as development tool, as it allows running the real CI without
opening a PR.
If there is a build pipeline specified, always build it, even if
there are no accompanying stages. If we short-circuit earlier and
ignore the build pipeline section, errors in the build pipeline
would not be caught at all.
If the local checkout has no git tags, the contributor list cannot be
generated. Just omit the information in that case. We can try to be
more clever here, but lets fix this first. I could not come up with any
reasonable solution to improve on this easily. Hence, the source
includes a comment explaining that you might just have to fetch the git
tags first.
The `build_stages` method short-circuits and returns early in case
any of the stages fail to build and returns None for the tree, and
build tree, therefore both of those can immediately cleaned up at
that point.
For this add a small helper `cleanup` that will call the cleanup
method for all supplied arguments, after filtering out None values.
Delay the cleanup of the build tree of the build pipeline, and
first check the result and only cleanup the tree when the build
did not fail, because in that case both returned trees will be
None and trying to cleanup them up will result in an exception.
Therefore, also don't clean up `tree` in the error case.
This extends on our model how we do releases. It introduces `NEWS.md`
as the authoritative source of our release-notes. It is pre-populated
with the release-notes from the previous 'v9' release, and contains a
suggestion for the upcoming 'v10'.
Furthermore, this adds `make release` as a simple checklist target that
contains instructions how to create a new release. Note that it is a
passive make-target which has no side-effects at all. It only prints
release information.
With this in place, we can drop `RELEASE.md`, as all information is now
combined in `make release`.
The used format of `-X, --long=VALUE` is not a valid option-list entry,
even though it is very commonly used all over the linux man-pages. Use
the supported format of `-X VALUE, --long=VALUE`, which will format
correctly in the man-page and html outputs.
For reference, these formats are valid in RST option-lists:
-a Short option
-c arg Short option with arg.
--long Long option.
-2, --two Aliases on a single line.
-f FILE, --file=FILE Aliases with arguments.
/V VMS/DOS-style option.
Add a 10s connection timeout for each file transfer. Also add an
increasing max timeout for a given file transfer (30s to 180s).
Also increase the retries to 10 and the concurrent threads to 15.
Hopefully this should make things a bit more stable in the face of
bad mirrors. We were encountering mirrors that would hang either
on connect or download at such slow speeds that they might as well
have stalled (~1kB in 45s).
Follow-up patches will provide a more long-term solution, by
allowing the same mirror selection as dnf currently uses.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Use `make man` rather than hard-coding man-page creation in the
spec-file. Furthermore, install all man-pages, not just the base one.
The commands are adjusted to account for possibly more man-pages being
added. Tree-wide the only place to adjust for new man-pages being added
is the `%files` list in the spec-file.
