Commit graph

145 commits

Author SHA1 Message Date
Martin Sehnoutka
d975effc42 improve vagrant test and its documentation
RPM packages are now kept in output directory after build so that we
know exactly which packages to copy to the test. The test directory now
contains special directory for RPMs. Fedora developer portal is
referenced from README file.
2019-07-25 12:46:53 +02:00
Lars Karlitski
5b50dec8c5 osbuild: add -l/--libdir parameter
Stop guessing if we're in the source directory by looking if a `stages`
subdirectory exists. Instead, assume that osbuild is installed on the
host.

If `--libdir` is given, mount the libdir into `/run/osbuild/lib` (alas,
we can't overwrite `/usr/libexec/osbuild`) and run osbuild from there.
Thus, running from source must now be done like this:

    # python3 -m osbuild --libdir . [other args]
2019-07-24 12:55:48 +02:00
Lars Karlitski
100dfd4f90 vagrant: also install python3-osbuild 2019-07-24 12:55:48 +02:00
Lars Karlitski
e1770a2386 pylint: disable too-many-arguments rule
5 arguments is not enough.
2019-07-24 12:55:48 +02:00
Martin Sehnoutka
e23fdb2b45 move stages and assemblers into /usr/libexec/ 2019-07-23 20:21:53 +02:00
Martin Sehnoutka
3c19420b3e introduce spec file and related build scripts
The repository now contains a Vagrantfile for running the testing script
against an RPM package created locally using `make rpm`. To run this
test use `make vagrant-test`. setup.py was also modified to adhere to
packaging guidelines and not to install system-level executables.
The lincense is now included in the Python package using the MANIFEST.in
file.
2019-07-23 15:22:40 +02:00
Martin Sehnoutka
c5ec09a230 enable running osbuild test from local checkout
It wasn't possible to run python3 -m osbuild as the OSBUILD command for
tests. With this change it will be.
2019-07-23 00:06:31 +02:00
Lars Karlitski
1e92e56b49 osbuild: remove systemResourcesFromEtc
It is a kludge that doesn't fit into osbuild's model. It's also not
necessary for any hacks anymore.
2019-07-19 13:31:49 +02:00
Lars Karlitski
00f4e6ed8b osbuild-run: generate /etc/pki/tls/certs/ca-bundle.crt
There's no need to bind mount the full /etc/pki from the host. This file
can be generated from /usr.
2019-07-19 13:31:49 +02:00
Lars Karlitski
f89ef6539d test and samples: don't include /etc/grub.d from the host
This is not necessary anymore, because we're generating a grub config
ourselves.
2019-07-19 13:31:49 +02:00
Tom Gundersen
96ea4e5698 BuildRoot: do not register with systemd-machined
This really only makes sense if we are running systemd as PID1
inside the container, but we are not booting a system, just using
it as a glorified chroot.

This means entering the namespaces from the outside will be a bit
more cumbersome, but that was not used much and was never reliable
to begin with.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 01:27:17 +02:00
Tom Gundersen
670d51a746 BuildRoot: drop unused device permissions
We only need permissions for loop devices, not for loop-control
and not for partitions.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
7274847711 Assembler: no longer mount devtmpfs in the container
Move the only assembler that relied on this to use LoopClient instead.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
d855c6c35e Assembler: add the remoteloop API to the assembler build roots
Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
c124ab264b loop: add helpers to use IPC to create loop devices
loop.py is a simple wrapper around the kernel loop API. remoteloop.py
uses this to create a server/clinet pair that communicates over an
AF_UNIX/SOCK_DGRAM socket to allow the server to create loop devices
for the client.

The client passes a fd that should be bound to the resulting loop
device, and a dir-fd where the loop device node should be created.
The server returns the name of the device node to the client.

The idea is that the client is run from whithin a container without
access to devtmpfs (and hence /dev/loop-control), and the server
runs on the host. The client would typically pass its (fake) /dev
as the output directory.

For the client this will be similar to `losetup -f foo.img --show`.

[@larskarlitski: pylint: ignore the new LoopInfo class, because it
only has dynamic attributes. Also disable attribute-defined-outside-init,
which (among other problems) is not ignored for that class.]

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
5d5766a98a BuildRoot: include the osbulid module in the containers
This way we have access to all our helper libraries from the stages
and assemblers.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
e2dc0c5081 BuildRoot: add support for an API sockets dir
Add a directory to each BuildRoot potentially containing a set of
sockets. Also add a helper to create a named bound socket in a given
BuildRoot.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
40dfee9855 assemblers/qcow2: don't rely on partition loopback devices
Create a loopback device for the raw partiton, rather than relying on
the partition devices the kernel puts in /dev. This requires us to
specify the part_msdos module directly as grub2-install now seems
unable to detect the partition table type.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
18ebce3016 Stage: don't bind-mount devtmpfs into the stage buildroot
We only use devices in Assemblers so this is not needed.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
8994e7f803 libdir: allow osbulid to be built from the tree
This was a regression from when osbuild was made into a module.
Restore the old behavior.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Martin Sehnoutka
b218430bfa add __main__ file for convenient CLI usage 2019-07-18 10:17:39 +02:00
Martin Sehnoutka
1f9752ef0f fix tests 2019-07-18 10:17:39 +02:00
Martin Sehnoutka
23680736bb create osbuild package 2019-07-18 10:17:39 +02:00
Martin Sehnoutka
70aeaec9ed drop unused partition_table_id option completely 2019-07-17 13:08:22 +02:00
Martin Sehnoutka
f04cb3836f fix warnings generated by pylint 2019-07-17 13:08:22 +02:00
Martin Sehnoutka
082bb267df add pylint and travis CI configuration 2019-07-17 13:08:22 +02:00
Lars Karlitski
6e8eb23e12 osbuild.py: make throwing exceptions optional
When using osbuild in non-interactive mode, returned results are needed
in the failure and the success case. Allow opting out of throwing an
exception, similar to `subprocess.run`.
2019-07-17 10:36:13 +02:00
Lars Karlitski
fc3ae0c72b osbuild.py: always save the last tree of a pipeline
Also fix crash when pipeline has no base and is empty.
2019-07-17 10:36:13 +02:00
Lars Karlitski
bedd71fc5c osbuild.py/Pipeline: more explicit handling of base
Now that `pipeline.stages` are actual objects, we can look up the id of
the last one to save it.
2019-07-17 10:36:13 +02:00
Lars Karlitski
241583b612 osbuild.py/Assembler: only check for output_dir once 2019-07-17 10:36:13 +02:00
Lars Karlitski
0d58d5c60a osbuild.py: don't expose tmpfs helper 2019-07-17 10:36:13 +02:00
Lars Karlitski
781fc73176 osbuild.py: make StageFailed and AssemblerFailed consistent
This way they can both be used as ducks in an exception handler.
2019-07-17 10:36:13 +02:00
Lars Karlitski
baa5da6abe osbuild.py: clean up Pipeline API
Separate deserialization into a new `osbuid.load()` function.

Pass objects to `Pipeline.run()` instead of keeping it in an attribute
on the pipeline.
2019-07-17 10:36:13 +02:00
Sehny
36f20b3e48 make the compression algorithm explicit 2019-07-16 10:25:42 +02:00
Lars Karlitski
8e04d527f1 osbuild.py: make storing resulting tree optional 2019-07-15 15:24:50 +02:00
Martin Sehnoutka
7a9925ec99 tar: derive compression algorithm from the file name 2019-07-15 14:52:48 +02:00
Tom Gundersen
7ea901928d assembler/qcow2: only require loopback for grub2
Create and instantiate the ext4 filesystem directly on the device, without
ever mounting it. This means that only grub2 now requires loopback devices
to function properly.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-11 15:01:44 +02:00
Tom Gundersen
a428572382 stages/io.weldr.grub2: drop unused functions and options
Since we no longer use grub2-mkconfig, but write static configuration
we can drop most of the helpers.

The partitin table id was never used in the first place. We use
filesystem UUIDs, not partition UUIDs to name our root/boot partitions.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-11 15:01:44 +02:00
Martin Sehnoutka
0dd84e66d4 raise AssemblerFailed in the Assembler class 2019-07-11 14:06:24 +02:00
Martin Sehnoutka
60312c3dae package osbuild-run in setup.py 2019-07-11 13:59:30 +02:00
Martin Sehnoutka
c8ca8c0460 use correct exception 2019-07-11 13:59:30 +02:00
Martin Sehnoutka
4afef01626 look for stages in /usr/lib/osbuild instead of /usr/lib 2019-07-11 12:34:32 +02:00
Martin Sehnoutka
a837cc5e82 make /run/osbuild in the pipeline class 2019-07-11 12:34:32 +02:00
Lars Karlitski
6fb843f7f6 add setup.py (#12) 2019-07-11 11:32:49 +02:00
Lars Karlitski
75aab4600c osbuild.py: separate BuildRoot.run() from Stage.run()
Make BuildRoot.run() a method that bind-mounts a binary into the build
root and runs it. It has the same API as subprocess.run().

Move stage- and assembler-specific code to Stage.run() and
Assembler.run().
2019-07-09 18:04:11 +02:00
Tom Gundersen
65151e22ff osbuild.py: assign ids to stages rather than to pipelines
Compute a hash based on the content of a stage, together with the
hash of its parent stage.

The output of a pipeline is saved by the id of the last stage.

This is largely equivalent to the current logic, where it is the
pipeline that contains the id, but this means that the ids are
indepedent of how pipelines are split, the only thing that matters
is the sequence of stages, not whether or not they are in one or
several interdependent pipelines.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-09 12:41:26 +02:00
Tom Gundersen
f5b1c80fb2 osbuild.py: run each stage/assembler in a fresh buildroot
Rather than making the buildroot shared for a pipeline, run each
stage/assembler in a fresh one.

This avoids state to leak between the stages/assemblers.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-09 12:41:26 +02:00
Tom Gundersen
7105bc91c7 osbuild.py: introduce Stage/Assembler classes
This moves some logic from the BuildRoot and some  from the Pipeline
class, into dedicated classes for stage and assembler. No functional
change.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-09 12:41:26 +02:00
Tom Gundersen
33daee4b0a osbuild.py/BuildRoot: explode stage/assembler dicts
run_{stage,assembler}() take stage/assembler dicts repsectively,
expoled them in the caller and pass in each argument instead.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-09 12:41:26 +02:00
Tom Gundersen
bfae7c3089 osuild.py/BuildRoot/getSystemResourceFromEtc: don't pass the whole assemble or stage object in
Only pass in the relevant array, making the type of the arguments
clearer and prepare to expoled the argumnets passed to run_{stage,assembler}.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-09 12:41:26 +02:00