Commit graph

39 commits

Author SHA1 Message Date
Lars Karlitski
0d58d5c60a osbuild.py: don't expose tmpfs helper 2019-07-17 10:36:13 +02:00
Lars Karlitski
781fc73176 osbuild.py: make StageFailed and AssemblerFailed consistent
This way they can both be used as ducks in an exception handler.
2019-07-17 10:36:13 +02:00
Lars Karlitski
baa5da6abe osbuild.py: clean up Pipeline API
Separate deserialization into a new `osbuid.load()` function.

Pass objects to `Pipeline.run()` instead of keeping it in an attribute
on the pipeline.
2019-07-17 10:36:13 +02:00
Lars Karlitski
8e04d527f1 osbuild.py: make storing resulting tree optional 2019-07-15 15:24:50 +02:00
Martin Sehnoutka
0dd84e66d4 raise AssemblerFailed in the Assembler class 2019-07-11 14:06:24 +02:00
Martin Sehnoutka
c8ca8c0460 use correct exception 2019-07-11 13:59:30 +02:00
Martin Sehnoutka
4afef01626 look for stages in /usr/lib/osbuild instead of /usr/lib 2019-07-11 12:34:32 +02:00
Martin Sehnoutka
a837cc5e82 make /run/osbuild in the pipeline class 2019-07-11 12:34:32 +02:00
Lars Karlitski
75aab4600c osbuild.py: separate BuildRoot.run() from Stage.run()
Make BuildRoot.run() a method that bind-mounts a binary into the build
root and runs it. It has the same API as subprocess.run().

Move stage- and assembler-specific code to Stage.run() and
Assembler.run().
2019-07-09 18:04:11 +02:00
Tom Gundersen
65151e22ff osbuild.py: assign ids to stages rather than to pipelines
Compute a hash based on the content of a stage, together with the
hash of its parent stage.

The output of a pipeline is saved by the id of the last stage.

This is largely equivalent to the current logic, where it is the
pipeline that contains the id, but this means that the ids are
indepedent of how pipelines are split, the only thing that matters
is the sequence of stages, not whether or not they are in one or
several interdependent pipelines.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-09 12:41:26 +02:00
Tom Gundersen
f5b1c80fb2 osbuild.py: run each stage/assembler in a fresh buildroot
Rather than making the buildroot shared for a pipeline, run each
stage/assembler in a fresh one.

This avoids state to leak between the stages/assemblers.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-09 12:41:26 +02:00
Tom Gundersen
7105bc91c7 osbuild.py: introduce Stage/Assembler classes
This moves some logic from the BuildRoot and some  from the Pipeline
class, into dedicated classes for stage and assembler. No functional
change.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-09 12:41:26 +02:00
Tom Gundersen
33daee4b0a osbuild.py/BuildRoot: explode stage/assembler dicts
run_{stage,assembler}() take stage/assembler dicts repsectively,
expoled them in the caller and pass in each argument instead.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-09 12:41:26 +02:00
Tom Gundersen
bfae7c3089 osuild.py/BuildRoot/getSystemResourceFromEtc: don't pass the whole assemble or stage object in
Only pass in the relevant array, making the type of the arguments
clearer and prepare to expoled the argumnets passed to run_{stage,assembler}.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-09 12:41:26 +02:00
Tom Gundersen
c31a22ac30 osbuild.py: remove some newlines
When nesting deeply, excessive newlines make the code harder to
read.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-03 13:11:37 +02:00
Tom Gundersen
cebed27cd9 osbuild: drop the concept of an input_dir
This removes the possibility of passing in arbitrary input data. We
now restrict ourselves to explicitly specified files/directories or
a base tree given by its pipeline id.

This drops the tar/tree stages/assemblers, as the tree/untree ones
are implicit in osbuild, and if we wish to also support compressed
trees, then we should add that to osbuild core as an option.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-03 13:11:37 +02:00
Tom Gundersen
ffffb87dea osbuild.py/pipeline: introduce the concept of a base pipeline
This allows one pipline to build on top of another. When the pipeline
id of one pipeline is specified in another, the tree is initialized
with the output of the given pipeline.

The caller must ensure that the base pipeline has alreday been run,
and its content is in the content-store.

This renders the io.weldr.untree stage and the --input argument both
redundant.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-03 13:11:37 +02:00
Tom Gundersen
e607053c32 osbuild.py/pipeline: add the concept of a content store
Whenever an assembler is not specified, the output tree is instead
saved to the content store, in a directory named after the pipeline
id.

This should render the io.weldr.tree assembler redundant.

In order to build the samples as before, specify the content store
as the input directory to build any pipeline that uses the
io.weldr.untree stage.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-03 13:11:37 +02:00
Tom Gundersen
f25cffa151 osbuild.py/pipeline: assign a unique id to every pipeline
This uniquely identifies a pipeline based on its content. Pipelines
are considered equal modulo whitespace and the order of object
elements.

The intention is that two runs of a pipeline with the same id
generates functionaly equivalent ids. It is up to the writers
of stages and pipelines to ensure this property holds.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-03 13:11:37 +02:00
Tom Gundersen
1219f1dc55 osbuild.py: introduce a Pipeline class
This is not a functional change, it is just a wrapper class around the
pipeline state.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-03 13:11:37 +02:00
Tom Gundersen
3b93fb6431 osbuild.py: fix run_assembler() API
The `interactive` argument was left out from commit
767b249b2d.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-01 22:56:06 +02:00
Lars Karlitski
d8ff04cde6 osbuild.py: return output from stages 2019-07-01 17:01:26 +02:00
Lars Karlitski
767b249b2d osbuild: move run() into osbuild.py
This allows for running a pipline from python and for non-interactive
mode, in which all output is captured.
2019-07-01 17:01:26 +02:00
Tom Gundersen
e9363f96ed osbuild.py: give assemblers a full devtmpfs
This follows what is done for the stages. We intend to look into
making this unnecessary, but for now allow full access so we can
work on the individual assemblers.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-06-19 19:49:53 +02:00
Tom Gundersen
0646b48bb3 osbulid.py/nspawn: bindmount /dev rather than use a synthetic one
We need dynamic device node creation when we create loop devices
and their partitions. It would be possible to mknod them, but
just using the host /dev seems more straight-forward. Access to
the various nodes still needs to be granted explictly as we are
in the devices CGroup.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-06-17 14:49:53 +02:00
Tom Gundersen
566911cc27 osbuild.pi: grant access to loop device nodes
We are in the devices namespace, so we must explicitly grant
permission to devices and/or device classes if we are to use
them.

We rely on being able to create and access loop devices in
order to create partitions and filesystems.

/dev/loop-control allows us to create new loop devices backed by
files. The block-loop class allows access to the created block
devices to set up a partition table and the block-blkext class
allows access to the partitions to create and mount filesystems.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-06-16 22:48:16 +02:00
Tom Gundersen
253ae27eca osbuild.py/tmpfs: mount with the right mode
Mount tmpfs by default as 0755. This only really makes a differente
when we copy the whole tree.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-06-16 13:58:13 +02:00
Lars Karlitski
92f3af94f6 stage api: pass options in a separate key
This avoids name clashes between osbuild and stage options.
2019-06-16 12:07:27 +02:00
Lars Karlitski
2d487fe685 osbuild.py: add systemResourcesFromEtc key
Some programs have resources in /etc (for example, /etc/pki or grub's
config scripts). Give stages a way to access these.
2019-06-14 20:29:14 +02:00
Lars Karlitski
bd87038210 osbuild.py: separate tree from build root
There's no reason to conflate the two. This allows us to build on
something other than a tmpfs.
2019-06-14 19:54:42 +02:00
Lars Karlitski
d43fe388a0 osbuild.py: clarify argument name 2019-06-14 19:46:37 +02:00
Lars Karlitski
c0769fc318 Add assemblers/ directory 2019-06-14 19:46:37 +02:00
Lars Karlitski
aea9dd1752 osbuild.py: don't use /tmp in the build root 2019-06-14 19:34:19 +02:00
Lars Karlitski
bc36adc1f1 osbuild.py: clean up bind mounts for stages
Allow input_dir for assemblers, but mount the tree read-only.
2019-06-14 19:32:05 +02:00
Lars Karlitski
e6dd428107 osbuild.py: rename run-stage to osbuild-run
Run all programs in the build root through osbuild-run. The things it
sets up are probbaly needed by everything.
2019-06-14 18:45:55 +02:00
Lars Karlitski
ce0b01e93d osbuild: remove --sit
It's not really useful because it's at the wrong place, after a stage
has torn down all mounts. It also makes the code more complex for too
little benefit.
2019-06-14 18:38:13 +02:00
Lars Karlitski
7ee6571640 osbuild.py: use readonly bind mounts when possible 2019-06-13 21:12:51 +02:00
Lars Karlitski
2dbd177b0f osbuild.py: add BuildRoot.run_assembler()
This is the canonical way to run an assembler.

Also improve error handling by introducing a StageFailed exception.
2019-06-13 21:07:23 +02:00
Lars Karlitski
b36c8135ae osbuild: split BuildRoot into a reusable module 2019-06-13 20:01:53 +02:00