Commit graph

173 commits

Author SHA1 Message Date
Martin Sehnoutka
3bbe17073d download sources from github as <name>-<version>.tar.gz
packit produces tarballs with this name, so it is easier to download
them from github like this
2019-08-01 16:17:11 +02:00
Martin Sehnoutka
97abdbed39 enable copr builds triggered in pull-requests
packit provide us with possibility to automatically build RPM package
for each PR, this configuration change enable the feature
2019-08-01 16:17:11 +02:00
Martin Sehnoutka
28e33c07ce introduce support for user-defined kernel options 2019-08-01 14:59:37 +02:00
Martin Sehnoutka
a095eb01c3 add packit config
packit is a service for continous delivery into Fedora repositories. It
should help us synchronize upstream repository on Github with downstream
repository on src.fedoraproject.org.
2019-07-31 17:38:03 +02:00
Tom Gundersen
faaa5ffa10 travis: test rpm generation
Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-31 01:34:31 +02:00
Tom Gundersen
ef31294505 travis: only install relevant dependencies for each job
This speeds up the simple tests considerably.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-31 01:34:31 +02:00
Tom Gundersen
8b659ae638 travis: add a test for a yum-based pipeline
Travis uses Ubuntu, which does not ship dnf, so introduce a yum
stage that allows us to test actual generation of trees on Travis.

We use this to generate a tree containing the tools necessary to
create abritrary Fedora-based build images in the future. We base
this on Fedora 27, as that is the last version that is installable
using yum rather than dnf.

In the future, once we support pipelines with nested build-images,
rather than just using the host OS as the build image, this will
allow us to bootstrap arbitrary pipelines on Travis.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-31 01:34:31 +02:00
Tom Gundersen
fa7a5b985e travis: add an osbuild test
Actually test the tools. This simply runs a noop stage followed by
a noop assembler.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-31 01:34:31 +02:00
Tom Gundersen
659ce42c83 BuildRoot: don't use nspawn's --volatile mode
We want the same functionality, but we now impleent it ourselves.

In addition to bind-mounting in /usr into the target container
(which is all nspawn does), we also add /bin, /sbin, /lib and
/lib64, if they exist and are not symlinks (presuambly into
/usr).

This means we can work on distros who have not implemented the
usr-move, like Ubuntu Bionic (used by Travis).

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-31 01:34:31 +02:00
Tom Gundersen
52e2374bb6 travis: run osbuild-run through pylint
This was most likely an oversight.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-31 01:34:31 +02:00
Tom Gundersen
fd72ed80ba osbuild-run: generate SSL certificates also on Debian-based systems
Call update-ca-certificates if the binary is found, generating SSL
certificates in /etc in i similar way on Debian-based systems as
is being done on RedHat-based ones.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-31 01:34:31 +02:00
Tom Gundersen
d08aed8f12 osbuild-run: make update-ca-trust optional
This is a RHism, that is not available on Debian-based systems.
Do not make it a hard reqirement, as pipelines may be able to
function just fine without it.

In a follow-up commit we will also check for the Debian-based
equivalent.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-31 01:34:31 +02:00
Tom Gundersen
9fa827ab04 osbuild-run: refactor into functions
Prepare for follow-up patches adding more functionality.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-31 01:34:31 +02:00
Martin Sehnoutka
0cf93934eb unify messages about files to be removed 2019-07-29 20:13:36 +02:00
Martin Sehnoutka
1ca4d8e6bb introduce keymap stage 2019-07-29 12:39:38 +02:00
Lars Karlitski
2e0981f1ab qcow2: remove partition_table_id
It's not necessary to specify it.
2019-07-29 12:39:10 +02:00
Martin Sehnoutka
593c6de385 add timezone stage 2019-07-29 12:05:42 +02:00
Martin Sehnoutka
693cd4e6b0 drop Darwin support, noone uses it anymore 2019-07-29 12:05:42 +02:00
Martin Sehnoutka
0812baa352 update url in the specfile 2019-07-29 09:41:28 +02:00
Lars Karlitski
4ebdc33e5e samples: fix base ids
I broke them with the io.weldr → org.osbuild move.
2019-07-28 21:27:26 +02:00
Tom Gundersen
4cb424acf7 travis: move to bionic
Let's always use the latest available Ubuntu release for our CI, we
are interested in potentially building old images, and using old
images as bulid images, but having an old distro as host is not
necessarily an aim. If we want to test with a greater diversity of
distros (which we do), we should do that in VM's, this should just
be for the simple/quick case.

Also restructure a bit to allow for more (named) tests.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-27 11:49:04 +02:00
Lars Karlitski
f4862457a3 rename io.weldr to org.osbuild (#39) 2019-07-26 09:40:55 +02:00
Lars Karlitski
265968f5c6 spec: only install regular files in stages/ and assemblers (#36)
The previous commit added a symlink to `../osbuild`, which is helpful
when developing, but shouldn't be included in the rpm.
2019-07-26 08:58:58 +02:00
Tom Gundersen
98ce5a7595 TmpFs: do not mount in __init__
The underlying filesystem was mounted in __init__ and unmonuted in
__exit__/__del__. This meant that if the same object was reused in
several `with` clauses, only the first one would work as intended.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-25 23:55:43 +02:00
Tom Gundersen
79b2f37cbc loop: add direct-io support
Support the LOOP_SET_DIRECT_IO ioctl, which alows us to control
whether or not a loopback device should perform its own buffering
or rely on the one done by the underlying backing file.

Enabling this should improve both throughput and memory consumption,
it is not currently hooked up as more testing would be required.
2019-07-25 23:55:43 +02:00
Tom Gundersen
70f2ec7227 README: update to match current state 2019-07-25 23:53:09 +02:00
Tom Gundersen
a17ecd0fca stages/assemblers: symlink the osbulid python library
This way the assemblers/stages are valid in isolation, even without
osbulid installed system-wide. This would be needed to have this work
when --libdir is not the system-wide one, as the library would
otherwise not be in sys.path.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-25 21:28:23 +02:00
Martin Sehnoutka
03a0e4b41c make vagrant-test idempotent 2019-07-25 15:47:53 +02:00
Martin Sehnoutka
d975effc42 improve vagrant test and its documentation
RPM packages are now kept in output directory after build so that we
know exactly which packages to copy to the test. The test directory now
contains special directory for RPMs. Fedora developer portal is
referenced from README file.
2019-07-25 12:46:53 +02:00
Lars Karlitski
5b50dec8c5 osbuild: add -l/--libdir parameter
Stop guessing if we're in the source directory by looking if a `stages`
subdirectory exists. Instead, assume that osbuild is installed on the
host.

If `--libdir` is given, mount the libdir into `/run/osbuild/lib` (alas,
we can't overwrite `/usr/libexec/osbuild`) and run osbuild from there.
Thus, running from source must now be done like this:

    # python3 -m osbuild --libdir . [other args]
2019-07-24 12:55:48 +02:00
Lars Karlitski
100dfd4f90 vagrant: also install python3-osbuild 2019-07-24 12:55:48 +02:00
Lars Karlitski
e1770a2386 pylint: disable too-many-arguments rule
5 arguments is not enough.
2019-07-24 12:55:48 +02:00
Martin Sehnoutka
e23fdb2b45 move stages and assemblers into /usr/libexec/ 2019-07-23 20:21:53 +02:00
Martin Sehnoutka
3c19420b3e introduce spec file and related build scripts
The repository now contains a Vagrantfile for running the testing script
against an RPM package created locally using `make rpm`. To run this
test use `make vagrant-test`. setup.py was also modified to adhere to
packaging guidelines and not to install system-level executables.
The lincense is now included in the Python package using the MANIFEST.in
file.
2019-07-23 15:22:40 +02:00
Martin Sehnoutka
c5ec09a230 enable running osbuild test from local checkout
It wasn't possible to run python3 -m osbuild as the OSBUILD command for
tests. With this change it will be.
2019-07-23 00:06:31 +02:00
Lars Karlitski
1e92e56b49 osbuild: remove systemResourcesFromEtc
It is a kludge that doesn't fit into osbuild's model. It's also not
necessary for any hacks anymore.
2019-07-19 13:31:49 +02:00
Lars Karlitski
00f4e6ed8b osbuild-run: generate /etc/pki/tls/certs/ca-bundle.crt
There's no need to bind mount the full /etc/pki from the host. This file
can be generated from /usr.
2019-07-19 13:31:49 +02:00
Lars Karlitski
f89ef6539d test and samples: don't include /etc/grub.d from the host
This is not necessary anymore, because we're generating a grub config
ourselves.
2019-07-19 13:31:49 +02:00
Tom Gundersen
96ea4e5698 BuildRoot: do not register with systemd-machined
This really only makes sense if we are running systemd as PID1
inside the container, but we are not booting a system, just using
it as a glorified chroot.

This means entering the namespaces from the outside will be a bit
more cumbersome, but that was not used much and was never reliable
to begin with.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 01:27:17 +02:00
Tom Gundersen
670d51a746 BuildRoot: drop unused device permissions
We only need permissions for loop devices, not for loop-control
and not for partitions.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
7274847711 Assembler: no longer mount devtmpfs in the container
Move the only assembler that relied on this to use LoopClient instead.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
d855c6c35e Assembler: add the remoteloop API to the assembler build roots
Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
c124ab264b loop: add helpers to use IPC to create loop devices
loop.py is a simple wrapper around the kernel loop API. remoteloop.py
uses this to create a server/clinet pair that communicates over an
AF_UNIX/SOCK_DGRAM socket to allow the server to create loop devices
for the client.

The client passes a fd that should be bound to the resulting loop
device, and a dir-fd where the loop device node should be created.
The server returns the name of the device node to the client.

The idea is that the client is run from whithin a container without
access to devtmpfs (and hence /dev/loop-control), and the server
runs on the host. The client would typically pass its (fake) /dev
as the output directory.

For the client this will be similar to `losetup -f foo.img --show`.

[@larskarlitski: pylint: ignore the new LoopInfo class, because it
only has dynamic attributes. Also disable attribute-defined-outside-init,
which (among other problems) is not ignored for that class.]

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
5d5766a98a BuildRoot: include the osbulid module in the containers
This way we have access to all our helper libraries from the stages
and assemblers.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
e2dc0c5081 BuildRoot: add support for an API sockets dir
Add a directory to each BuildRoot potentially containing a set of
sockets. Also add a helper to create a named bound socket in a given
BuildRoot.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
40dfee9855 assemblers/qcow2: don't rely on partition loopback devices
Create a loopback device for the raw partiton, rather than relying on
the partition devices the kernel puts in /dev. This requires us to
specify the part_msdos module directly as grub2-install now seems
unable to detect the partition table type.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
18ebce3016 Stage: don't bind-mount devtmpfs into the stage buildroot
We only use devices in Assemblers so this is not needed.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Tom Gundersen
8994e7f803 libdir: allow osbulid to be built from the tree
This was a regression from when osbuild was made into a module.
Restore the old behavior.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-07-19 00:51:12 +02:00
Martin Sehnoutka
b218430bfa add __main__ file for convenient CLI usage 2019-07-18 10:17:39 +02:00
Martin Sehnoutka
1f9752ef0f fix tests 2019-07-18 10:17:39 +02:00