Commit graph

159 commits

Author SHA1 Message Date
Martin Sehnoutka
1ca4d8e6bb introduce keymap stage 2019-07-29 12:39:38 +02:00
Lars Karlitski
2e0981f1ab qcow2: remove partition_table_id
It's not necessary to specify it.
2019-07-29 12:39:10 +02:00
Martin Sehnoutka
593c6de385 add timezone stage 2019-07-29 12:05:42 +02:00
Martin Sehnoutka
693cd4e6b0 drop Darwin support, noone uses it anymore 2019-07-29 12:05:42 +02:00
Martin Sehnoutka
0812baa352 update url in the specfile 2019-07-29 09:41:28 +02:00
Lars Karlitski
4ebdc33e5e samples: fix base ids
I broke them with the io.weldr → org.osbuild move.
2019-07-28 21:27:26 +02:00
Tom Gundersen
4cb424acf7 travis: move to bionic
Let's always use the latest available Ubuntu release for our CI, we
are interested in potentially building old images, and using old
images as bulid images, but having an old distro as host is not
necessarily an aim. If we want to test with a greater diversity of
distros (which we do), we should do that in VM's, this should just
be for the simple/quick case.

Also restructure a bit to allow for more (named) tests.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-27 11:49:04 +02:00
Lars Karlitski
f4862457a3 rename io.weldr to org.osbuild (#39) 2019-07-26 09:40:55 +02:00
Lars Karlitski
265968f5c6 spec: only install regular files in stages/ and assemblers (#36)
The previous commit added a symlink to `../osbuild`, which is helpful
when developing, but shouldn't be included in the rpm.
2019-07-26 08:58:58 +02:00
Tom Gundersen
98ce5a7595 TmpFs: do not mount in __init__
The underlying filesystem was mounted in __init__ and unmonuted in
__exit__/__del__. This meant that if the same object was reused in
several `with` clauses, only the first one would work as intended.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-25 23:55:43 +02:00
Tom Gundersen
79b2f37cbc loop: add direct-io support
Support the LOOP_SET_DIRECT_IO ioctl, which alows us to control
whether or not a loopback device should perform its own buffering
or rely on the one done by the underlying backing file.

Enabling this should improve both throughput and memory consumption,
it is not currently hooked up as more testing would be required.
2019-07-25 23:55:43 +02:00
Tom Gundersen
70f2ec7227 README: update to match current state 2019-07-25 23:53:09 +02:00
Tom Gundersen
a17ecd0fca stages/assemblers: symlink the osbulid python library
This way the assemblers/stages are valid in isolation, even without
osbulid installed system-wide. This would be needed to have this work
when --libdir is not the system-wide one, as the library would
otherwise not be in sys.path.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-25 21:28:23 +02:00
Martin Sehnoutka
03a0e4b41c make vagrant-test idempotent 2019-07-25 15:47:53 +02:00
Martin Sehnoutka
d975effc42 improve vagrant test and its documentation
RPM packages are now kept in output directory after build so that we
know exactly which packages to copy to the test. The test directory now
contains special directory for RPMs. Fedora developer portal is
referenced from README file.
2019-07-25 12:46:53 +02:00
Lars Karlitski
5b50dec8c5 osbuild: add -l/--libdir parameter
Stop guessing if we're in the source directory by looking if a `stages`
subdirectory exists. Instead, assume that osbuild is installed on the
host.

If `--libdir` is given, mount the libdir into `/run/osbuild/lib` (alas,
we can't overwrite `/usr/libexec/osbuild`) and run osbuild from there.
Thus, running from source must now be done like this:

    # python3 -m osbuild --libdir . [other args]
2019-07-24 12:55:48 +02:00
Lars Karlitski
100dfd4f90 vagrant: also install python3-osbuild 2019-07-24 12:55:48 +02:00
Lars Karlitski
e1770a2386 pylint: disable too-many-arguments rule
5 arguments is not enough.
2019-07-24 12:55:48 +02:00
Martin Sehnoutka
e23fdb2b45 move stages and assemblers into /usr/libexec/ 2019-07-23 20:21:53 +02:00
Martin Sehnoutka
3c19420b3e introduce spec file and related build scripts
The repository now contains a Vagrantfile for running the testing script
against an RPM package created locally using `make rpm`. To run this
test use `make vagrant-test`. setup.py was also modified to adhere to
packaging guidelines and not to install system-level executables.
The lincense is now included in the Python package using the MANIFEST.in
file.
2019-07-23 15:22:40 +02:00
Martin Sehnoutka
c5ec09a230 enable running osbuild test from local checkout
It wasn't possible to run python3 -m osbuild as the OSBUILD command for
tests. With this change it will be.
2019-07-23 00:06:31 +02:00
Lars Karlitski
1e92e56b49 osbuild: remove systemResourcesFromEtc
It is a kludge that doesn't fit into osbuild's model. It's also not
necessary for any hacks anymore.
2019-07-19 13:31:49 +02:00
Lars Karlitski
00f4e6ed8b osbuild-run: generate /etc/pki/tls/certs/ca-bundle.crt
There's no need to bind mount the full /etc/pki from the host. This file
can be generated from /usr.
2019-07-19 13:31:49 +02:00
Lars Karlitski
f89ef6539d test and samples: don't include /etc/grub.d from the host
This is not necessary anymore, because we're generating a grub config
ourselves.
2019-07-19 13:31:49 +02:00
Tom Gundersen
96ea4e5698 BuildRoot: do not register with systemd-machined
This really only makes sense if we are running systemd as PID1
inside the container, but we are not booting a system, just using
it as a glorified chroot.

This means entering the namespaces from the outside will be a bit
more cumbersome, but that was not used much and was never reliable
to begin with.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 01:27:17 +02:00
Tom Gundersen
670d51a746 BuildRoot: drop unused device permissions
We only need permissions for loop devices, not for loop-control
and not for partitions.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
7274847711 Assembler: no longer mount devtmpfs in the container
Move the only assembler that relied on this to use LoopClient instead.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
d855c6c35e Assembler: add the remoteloop API to the assembler build roots
Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
c124ab264b loop: add helpers to use IPC to create loop devices
loop.py is a simple wrapper around the kernel loop API. remoteloop.py
uses this to create a server/clinet pair that communicates over an
AF_UNIX/SOCK_DGRAM socket to allow the server to create loop devices
for the client.

The client passes a fd that should be bound to the resulting loop
device, and a dir-fd where the loop device node should be created.
The server returns the name of the device node to the client.

The idea is that the client is run from whithin a container without
access to devtmpfs (and hence /dev/loop-control), and the server
runs on the host. The client would typically pass its (fake) /dev
as the output directory.

For the client this will be similar to `losetup -f foo.img --show`.

[@larskarlitski: pylint: ignore the new LoopInfo class, because it
only has dynamic attributes. Also disable attribute-defined-outside-init,
which (among other problems) is not ignored for that class.]

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
5d5766a98a BuildRoot: include the osbulid module in the containers
This way we have access to all our helper libraries from the stages
and assemblers.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
e2dc0c5081 BuildRoot: add support for an API sockets dir
Add a directory to each BuildRoot potentially containing a set of
sockets. Also add a helper to create a named bound socket in a given
BuildRoot.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
40dfee9855 assemblers/qcow2: don't rely on partition loopback devices
Create a loopback device for the raw partiton, rather than relying on
the partition devices the kernel puts in /dev. This requires us to
specify the part_msdos module directly as grub2-install now seems
unable to detect the partition table type.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
18ebce3016 Stage: don't bind-mount devtmpfs into the stage buildroot
We only use devices in Assemblers so this is not needed.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
8994e7f803 libdir: allow osbulid to be built from the tree
This was a regression from when osbuild was made into a module.
Restore the old behavior.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Martin Sehnoutka
b218430bfa add __main__ file for convenient CLI usage 2019-07-18 10:17:39 +02:00
Martin Sehnoutka
1f9752ef0f fix tests 2019-07-18 10:17:39 +02:00
Martin Sehnoutka
23680736bb create osbuild package 2019-07-18 10:17:39 +02:00
Martin Sehnoutka
70aeaec9ed drop unused partition_table_id option completely 2019-07-17 13:08:22 +02:00
Martin Sehnoutka
f04cb3836f fix warnings generated by pylint 2019-07-17 13:08:22 +02:00
Martin Sehnoutka
082bb267df add pylint and travis CI configuration 2019-07-17 13:08:22 +02:00
Lars Karlitski
6e8eb23e12 osbuild.py: make throwing exceptions optional
When using osbuild in non-interactive mode, returned results are needed
in the failure and the success case. Allow opting out of throwing an
exception, similar to `subprocess.run`.
2019-07-17 10:36:13 +02:00
Lars Karlitski
fc3ae0c72b osbuild.py: always save the last tree of a pipeline
Also fix crash when pipeline has no base and is empty.
2019-07-17 10:36:13 +02:00
Lars Karlitski
bedd71fc5c osbuild.py/Pipeline: more explicit handling of base
Now that `pipeline.stages` are actual objects, we can look up the id of
the last one to save it.
2019-07-17 10:36:13 +02:00
Lars Karlitski
241583b612 osbuild.py/Assembler: only check for output_dir once 2019-07-17 10:36:13 +02:00
Lars Karlitski
0d58d5c60a osbuild.py: don't expose tmpfs helper 2019-07-17 10:36:13 +02:00
Lars Karlitski
781fc73176 osbuild.py: make StageFailed and AssemblerFailed consistent
This way they can both be used as ducks in an exception handler.
2019-07-17 10:36:13 +02:00
Lars Karlitski
baa5da6abe osbuild.py: clean up Pipeline API
Separate deserialization into a new `osbuid.load()` function.

Pass objects to `Pipeline.run()` instead of keeping it in an attribute
on the pipeline.
2019-07-17 10:36:13 +02:00
Sehny
36f20b3e48 make the compression algorithm explicit 2019-07-16 10:25:42 +02:00
Lars Karlitski
8e04d527f1 osbuild.py: make storing resulting tree optional 2019-07-15 15:24:50 +02:00
Martin Sehnoutka
7a9925ec99 tar: derive compression algorithm from the file name 2019-07-15 14:52:48 +02:00