The build pipeline, is a sub-pipeline used to generate the build
tree to use rather than the current root directory. This can be
nested arbitrarily deep, but ultimately we will fall back to the
current logic when no build property is found.
Just like the tree after the last stage of a regular pipeline ends
up in the object store, so does currently each build tree (as the
build sub-pipeline really is just a regular pipeline in its own
right). We may want to avoid both these instances of the implicit
storing semantics, and rather make it something the caller opts-in
to. However, for now that is left as a future optimization.
Signed-off-by: Tom Gundersen <teg@jklm.no>
On some hosts, systemd-tmpfiles will generate an nsswitch.conf
configuring DNS to be done via systemd-resolved, but this will
require the container to be booted and resolved to be running.
In other cases, a proper fall-back is configured, so this is not
a problem, but on some hosts this means DNS does not work.
Conversely, the default behavior with no nsswitch.conf at all
works just fine, always using nss-dns.
Let's simply delete the file if it is there, and rely on the
default.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This also changes the structure of the object store, though the
basic idea is the same.
The object store contains a directory of objects, which are content
addressable filesystem trees. Currently we only ever use their
content-hash internally, but the idea for this is basically Lars
Karlitski and Kay Sievers' `treesum()`. We may exopse this in the
future.
Moreover, it contains a directory of refs, which are symlinks named
by the stage id they correspond to (as before), pointing to an object
generated from that stage-id.
The ObjectStore exposes three method:
`has_tree()`: This checks if the content store contains the given tree.
If so, we can rely on the tree remaining there.
`get_tree()`: This is meant to be used with a `with` block and yields
the path to a read-only instance of the tree with the given id. If the
tree_id is passed in as None, an empty directory is given instead.
`new_tree()`: This is meant to be used with a `with` block and yields
the path to a directory in which the tree by the given id should be
created. If a base_id is passed in, the tree is initialized with the
tree with the given id. Only when the block is exited successfully
is the tree written to the content store, referenced by the id in
question.
Use this in Pipeline.run() to avoid regenerating trees unneccessarily.
In order to trigger a regeneration, the content store must currently
be manually flushed.
Update the travis test to run the noop pipeline twice, verifying that
the stage is only run the first time.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Rather than hard-coding this to /, let the caller provide the
directory path to use.
In the past, we needed to give special treatment to /, as it had
to be bind-mounted before being used by nspawn, to work around a
check they had, refusing to use the host root in the container.
We no longer pass the directory directly to nspawn, but rather
mount the subdirs we want ourselves, so that no longer applies.
The callers pass in /, so the behavior is unchanged.
Signed-off-by: Tom Gundersen <teg@jklm.no>
packit is a service for continous delivery into Fedora repositories. It
should help us synchronize upstream repository on Github with downstream
repository on src.fedoraproject.org.
Travis uses Ubuntu, which does not ship dnf, so introduce a yum
stage that allows us to test actual generation of trees on Travis.
We use this to generate a tree containing the tools necessary to
create abritrary Fedora-based build images in the future. We base
this on Fedora 27, as that is the last version that is installable
using yum rather than dnf.
In the future, once we support pipelines with nested build-images,
rather than just using the host OS as the build image, this will
allow us to bootstrap arbitrary pipelines on Travis.
Signed-off-by: Tom Gundersen <teg@jklm.no>
We want the same functionality, but we now impleent it ourselves.
In addition to bind-mounting in /usr into the target container
(which is all nspawn does), we also add /bin, /sbin, /lib and
/lib64, if they exist and are not symlinks (presuambly into
/usr).
This means we can work on distros who have not implemented the
usr-move, like Ubuntu Bionic (used by Travis).
Signed-off-by: Tom Gundersen <teg@jklm.no>
Call update-ca-certificates if the binary is found, generating SSL
certificates in /etc in i similar way on Debian-based systems as
is being done on RedHat-based ones.
Signed-off-by: Tom Gundersen <teg@jklm.no>
This is a RHism, that is not available on Debian-based systems.
Do not make it a hard reqirement, as pipelines may be able to
function just fine without it.
In a follow-up commit we will also check for the Debian-based
equivalent.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Let's always use the latest available Ubuntu release for our CI, we
are interested in potentially building old images, and using old
images as bulid images, but having an old distro as host is not
necessarily an aim. If we want to test with a greater diversity of
distros (which we do), we should do that in VM's, this should just
be for the simple/quick case.
Also restructure a bit to allow for more (named) tests.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The underlying filesystem was mounted in __init__ and unmonuted in
__exit__/__del__. This meant that if the same object was reused in
several `with` clauses, only the first one would work as intended.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Support the LOOP_SET_DIRECT_IO ioctl, which alows us to control
whether or not a loopback device should perform its own buffering
or rely on the one done by the underlying backing file.
Enabling this should improve both throughput and memory consumption,
it is not currently hooked up as more testing would be required.
This way the assemblers/stages are valid in isolation, even without
osbulid installed system-wide. This would be needed to have this work
when --libdir is not the system-wide one, as the library would
otherwise not be in sys.path.
Signed-off-by: Tom Gundersen <teg@jklm.no>
RPM packages are now kept in output directory after build so that we
know exactly which packages to copy to the test. The test directory now
contains special directory for RPMs. Fedora developer portal is
referenced from README file.
Stop guessing if we're in the source directory by looking if a `stages`
subdirectory exists. Instead, assume that osbuild is installed on the
host.
If `--libdir` is given, mount the libdir into `/run/osbuild/lib` (alas,
we can't overwrite `/usr/libexec/osbuild`) and run osbuild from there.
Thus, running from source must now be done like this:
# python3 -m osbuild --libdir . [other args]
The repository now contains a Vagrantfile for running the testing script
against an RPM package created locally using `make rpm`. To run this
test use `make vagrant-test`. setup.py was also modified to adhere to
packaging guidelines and not to install system-level executables.
The lincense is now included in the Python package using the MANIFEST.in
file.
This really only makes sense if we are running systemd as PID1
inside the container, but we are not booting a system, just using
it as a glorified chroot.
This means entering the namespaces from the outside will be a bit
more cumbersome, but that was not used much and was never reliable
to begin with.
Signed-off-by: Tom Gundersen <teg@jklm.no>
