Commit graph

121 commits

Author SHA1 Message Date
Martin Sehnoutka
f04cb3836f fix warnings generated by pylint 2019-07-17 13:08:22 +02:00
Martin Sehnoutka
082bb267df add pylint and travis CI configuration 2019-07-17 13:08:22 +02:00
Lars Karlitski
6e8eb23e12 osbuild.py: make throwing exceptions optional
When using osbuild in non-interactive mode, returned results are needed
in the failure and the success case. Allow opting out of throwing an
exception, similar to `subprocess.run`.
2019-07-17 10:36:13 +02:00
Lars Karlitski
fc3ae0c72b osbuild.py: always save the last tree of a pipeline
Also fix crash when pipeline has no base and is empty.
2019-07-17 10:36:13 +02:00
Lars Karlitski
bedd71fc5c osbuild.py/Pipeline: more explicit handling of base
Now that `pipeline.stages` are actual objects, we can look up the id of
the last one to save it.
2019-07-17 10:36:13 +02:00
Lars Karlitski
241583b612 osbuild.py/Assembler: only check for output_dir once 2019-07-17 10:36:13 +02:00
Lars Karlitski
0d58d5c60a osbuild.py: don't expose tmpfs helper 2019-07-17 10:36:13 +02:00
Lars Karlitski
781fc73176 osbuild.py: make StageFailed and AssemblerFailed consistent
This way they can both be used as ducks in an exception handler.
2019-07-17 10:36:13 +02:00
Lars Karlitski
baa5da6abe osbuild.py: clean up Pipeline API
Separate deserialization into a new `osbuid.load()` function.

Pass objects to `Pipeline.run()` instead of keeping it in an attribute
on the pipeline.
2019-07-17 10:36:13 +02:00
Sehny
36f20b3e48 make the compression algorithm explicit 2019-07-16 10:25:42 +02:00
Lars Karlitski
8e04d527f1 osbuild.py: make storing resulting tree optional 2019-07-15 15:24:50 +02:00
Martin Sehnoutka
7a9925ec99 tar: derive compression algorithm from the file name 2019-07-15 14:52:48 +02:00
Tom Gundersen
7ea901928d assembler/qcow2: only require loopback for grub2
Create and instantiate the ext4 filesystem directly on the device, without
ever mounting it. This means that only grub2 now requires loopback devices
to function properly.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-11 15:01:44 +02:00
Tom Gundersen
a428572382 stages/io.weldr.grub2: drop unused functions and options
Since we no longer use grub2-mkconfig, but write static configuration
we can drop most of the helpers.

The partitin table id was never used in the first place. We use
filesystem UUIDs, not partition UUIDs to name our root/boot partitions.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-11 15:01:44 +02:00
Martin Sehnoutka
0dd84e66d4 raise AssemblerFailed in the Assembler class 2019-07-11 14:06:24 +02:00
Martin Sehnoutka
60312c3dae package osbuild-run in setup.py 2019-07-11 13:59:30 +02:00
Martin Sehnoutka
c8ca8c0460 use correct exception 2019-07-11 13:59:30 +02:00
Martin Sehnoutka
4afef01626 look for stages in /usr/lib/osbuild instead of /usr/lib 2019-07-11 12:34:32 +02:00
Martin Sehnoutka
a837cc5e82 make /run/osbuild in the pipeline class 2019-07-11 12:34:32 +02:00
Lars Karlitski
6fb843f7f6 add setup.py (#12) 2019-07-11 11:32:49 +02:00
Lars Karlitski
75aab4600c osbuild.py: separate BuildRoot.run() from Stage.run()
Make BuildRoot.run() a method that bind-mounts a binary into the build
root and runs it. It has the same API as subprocess.run().

Move stage- and assembler-specific code to Stage.run() and
Assembler.run().
2019-07-09 18:04:11 +02:00
Tom Gundersen
65151e22ff osbuild.py: assign ids to stages rather than to pipelines
Compute a hash based on the content of a stage, together with the
hash of its parent stage.

The output of a pipeline is saved by the id of the last stage.

This is largely equivalent to the current logic, where it is the
pipeline that contains the id, but this means that the ids are
indepedent of how pipelines are split, the only thing that matters
is the sequence of stages, not whether or not they are in one or
several interdependent pipelines.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-09 12:41:26 +02:00
Tom Gundersen
f5b1c80fb2 osbuild.py: run each stage/assembler in a fresh buildroot
Rather than making the buildroot shared for a pipeline, run each
stage/assembler in a fresh one.

This avoids state to leak between the stages/assemblers.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-09 12:41:26 +02:00
Tom Gundersen
7105bc91c7 osbuild.py: introduce Stage/Assembler classes
This moves some logic from the BuildRoot and some  from the Pipeline
class, into dedicated classes for stage and assembler. No functional
change.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-09 12:41:26 +02:00
Tom Gundersen
33daee4b0a osbuild.py/BuildRoot: explode stage/assembler dicts
run_{stage,assembler}() take stage/assembler dicts repsectively,
expoled them in the caller and pass in each argument instead.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-09 12:41:26 +02:00
Tom Gundersen
bfae7c3089 osuild.py/BuildRoot/getSystemResourceFromEtc: don't pass the whole assemble or stage object in
Only pass in the relevant array, making the type of the arguments
clearer and prepare to expoled the argumnets passed to run_{stage,assembler}.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-09 12:41:26 +02:00
Martin Sehnoutka
5ca90dc052 mount filesystems as rw
d-bus service was unable to start with the previous setup
2019-07-09 10:29:48 +02:00
Martin Sehnoutka
824cba1c08 introduce testing script 2019-07-09 10:29:48 +02:00
Martin Sehnoutka
9ea6150ee7 introduce disabled services into the systemd stage 2019-07-09 10:29:48 +02:00
Lars Karlitski
e9fb1ded15 assemblers: resurrect io.weldr.targz
This was dropped in cebed27cd. However, a tar of a tree is useful in its
own right as an output format.
2019-07-08 14:22:56 +02:00
Lars Karlitski
34d620feea stages/dnf: don't use configparser to write dnf.conf
configparser writes strings with quotes and lists with enclosing
brackets, both of which may not appear in dnf.conf.

dnf.conf(5) defines the format loosely.
2019-07-08 14:22:25 +02:00
Tom Gundersen
c31a22ac30 osbuild.py: remove some newlines
When nesting deeply, excessive newlines make the code harder to
read.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-03 13:11:37 +02:00
Tom Gundersen
cebed27cd9 osbuild: drop the concept of an input_dir
This removes the possibility of passing in arbitrary input data. We
now restrict ourselves to explicitly specified files/directories or
a base tree given by its pipeline id.

This drops the tar/tree stages/assemblers, as the tree/untree ones
are implicit in osbuild, and if we wish to also support compressed
trees, then we should add that to osbuild core as an option.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-03 13:11:37 +02:00
Tom Gundersen
ffffb87dea osbuild.py/pipeline: introduce the concept of a base pipeline
This allows one pipline to build on top of another. When the pipeline
id of one pipeline is specified in another, the tree is initialized
with the output of the given pipeline.

The caller must ensure that the base pipeline has alreday been run,
and its content is in the content-store.

This renders the io.weldr.untree stage and the --input argument both
redundant.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-03 13:11:37 +02:00
Tom Gundersen
e607053c32 osbuild.py/pipeline: add the concept of a content store
Whenever an assembler is not specified, the output tree is instead
saved to the content store, in a directory named after the pipeline
id.

This should render the io.weldr.tree assembler redundant.

In order to build the samples as before, specify the content store
as the input directory to build any pipeline that uses the
io.weldr.untree stage.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-03 13:11:37 +02:00
Tom Gundersen
f25cffa151 osbuild.py/pipeline: assign a unique id to every pipeline
This uniquely identifies a pipeline based on its content. Pipelines
are considered equal modulo whitespace and the order of object
elements.

The intention is that two runs of a pipeline with the same id
generates functionaly equivalent ids. It is up to the writers
of stages and pipelines to ensure this property holds.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-03 13:11:37 +02:00
Tom Gundersen
1219f1dc55 osbuild.py: introduce a Pipeline class
This is not a functional change, it is just a wrapper class around the
pipeline state.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-03 13:11:37 +02:00
Tom Gundersen
9f5ce324b0 stages/grub2: use static grub.cfg/grubenv files
Afetr discussions with Javier Martinez Canillas and Alberto Ruiz
from the grub team, it was clear that we want move to a static
configuration shipped in the grub rpm in the future. For now,
we use our own, but the aim is to use the same as upstream.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-02 20:05:19 +02:00
Tom Gundersen
a45f0dc8e5 samples/qcow2: reuse the base-with-grub2 stage
Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-02 18:49:33 +02:00
Tom Gundersen
3b93fb6431 osbuild.py: fix run_assembler() API
The `interactive` argument was left out from commit
767b249b2d.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-01 22:56:06 +02:00
Tom Gundersen
037f439d01 stages/grub2: fix defaults/grub
This file was missing new-lines.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-01 22:18:05 +02:00
Tom Gundersen
02797f1e9a samples/base: use fedora 30
The boot loader snippets were not being generated on f29, we may want
to revisit that, but for now let's work against f30.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-01 22:18:05 +02:00
Tom Gundersen
0252f42305 assemblers/loop_dev: take the size and optionally the offset as arguments
By default the whole image file is attechd to the loopback device, but
we want to only attach a slice, namely individual partitions.

This is a noop, just adds the parameters to the helpers.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-01 22:18:05 +02:00
Lars Karlitski
d8ff04cde6 osbuild.py: return output from stages 2019-07-01 17:01:26 +02:00
Lars Karlitski
767b249b2d osbuild: move run() into osbuild.py
This allows for running a pipline from python and for non-interactive
mode, in which all output is captured.
2019-07-01 17:01:26 +02:00
Tom Gundersen
ecaed3bbfa stages/grub2: don't set up a partition table
We only need the filesystem with the correct fs-UUID to chroot into,
there is no need to set up a whole partition table.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-06-27 16:22:30 +02:00
Tom Gundersen
779e5c40ea stages/grub2: set default entry
Without a default and/or timeout set, grub refuses to boot.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-06-26 13:46:19 +02:00
Tom Gundersen
d9488e5f8b losetup: make sure image sizes are multiples of 512K sectors
Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-06-26 12:57:20 +02:00
Tom Gundersen
b540eb5ee7 losetup: create and attach images to loopback devices in one operation
Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-06-26 12:45:35 +02:00
Tom Gundersen
4b3c1e922c samples/qcow2: reuse the base image so we don't have to recreate all the time
Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-06-26 12:31:56 +02:00