Commit graph

6182 commits

Author SHA1 Message Date
Tom Gundersen
5a82c95c9a blueprint/qcow2: move wanted pipeline into sepratate file
This will allow us to run tests on the generated images from the
same files in the future.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-21 13:35:05 +02:00
Tom Gundersen
d302ac6d09 blueprint/output/qcow2: add unit tests
Use the autogenerated test framework from VSCode (with minor fixes to
make them compile).

As we do not yet support customizations in blueprint, all the tests are
trivial.

I was not entirely sure about the best way to encode the wanted pipeline
output. I currently represent it as a JSON string and unmarshal it into
the object to compare with.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-18 15:57:31 +02:00
Ondřej Budai
deb763c13b dnf-json: Remove custom dnf cache directory
When osbuild-composer is run as systemd service, we don't want to write
anything into working directory. Currently, we write dnf cache into it.
Instead, let's just use the default dnf cache directory.
2019-10-14 23:45:53 +02:00
Ondřej Budai
8644213bb7 Remove process termination in case of error from runDNF function
log.Fatalf not only writes into a log, but also exits the process.
Instead, we want the caller to handle the error.

Also, the logged message is imho wrong.
2019-10-14 23:43:04 +02:00
Ondřej Budai
44088d9a6b Redirect dnf-json stderr to osbuild-composer stderr
When something wrong happens, we want to know about it.
2019-10-14 23:43:04 +02:00
Lars Karlitski
abd958f463 Add README and LICENSE files 2019-10-14 10:31:04 +02:00
Tom Gundersen
029dd0a81b build-sys: rework makefiles
Unify into one make file, with two targets:

build:
  This unconditionally builds the binaries that wolud be shipped in the package,
  this is marked as a phony target, and relies on the go compiler to know what
  is up-to-date and what needs to be rebuilt.

install:
  This installs binaries and config files into the file system and runs any
  scripts to set things up correctly, as if the rpm had been installed.

This drops the old `run` target, which could still be done from the commandline,
but we want to be able to rely on systemd features, so it is not something we
want to test or support.

To run osbuild-composer from a checkout one should now do:
```
$ make build
$ sudo make install
```

This has the same effect as installing the RPM would have, starting and stopping
the sokcets/services is still left to the user.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-11 17:40:31 +02:00
Jacob Kozol
d5830a4794 tests: delete blueprints after tests
At the end of each set of tests the blueprint created is deleted.
This tests the delete route as well as removing blueprints
created while testing external apis such as lorax.
2019-10-11 14:16:19 +02:00
Ondřej Budai
6e790bc62d osbuild-composer: Fix imports broken by new module name
We've merged some stuff with old module name used in imports. This
commit fixes it.
2019-10-09 08:06:07 +02:00
Ondřej Budai
15b82a15d2 osbuild-composer: Rename module to github.com/osbuild/osbuild-composer
This should be the best practice according to other popular go projects:
- https://github.com/prometheus/prometheus
- https://github.com/syncthing/syncthing
- https://github.com/drone/drone
- https://github.com/hashicorp/terraform

Also, this change fixes go get command (it currently fails due to bad package
name).
2019-10-08 21:44:57 +02:00
Tom Gundersen
b0d9423b73 osbuild-pipeline: add tool to print pipelines to stdout
This tool outputs the pipeline associated with each output type. In
the future this should grow to take a blueprint as an input, and
also allow the arch/os/version/variant to be configured.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-08 21:03:45 +02:00
Tom Gundersen
22254637f8 blueprint: generate real f30 images for all the output types
This is still just a placeholder, and in particular the live CD support
is not right.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-08 21:03:45 +02:00
Tom Gundersen
4845826048 blueprint: add proper error handling for querying invalid output types
Otherwise an invalid output type would crash composer.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-08 21:03:45 +02:00
Tom Gundersen
976d59cfda pipeline: add rawfs assembler support
Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-08 21:03:45 +02:00
Tom Gundersen
35055b3176 pipeline/assembler: complete the qcow2/qemu rename
This was an oversight.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-08 21:03:45 +02:00
Tom Gundersen
6a40d77ed3 pipeline/stages/dnf: repositories are no longer named
Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-08 21:03:45 +02:00
Tom Gundersen
7e78b4051d jobqueue: add back tests for the API
Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-07 10:37:43 +02:00
Tom Gundersen
94e1e6f42b store: add error types and rudimentary error handling
This will allow us to better distinguish between different error
conditions in the API.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-07 10:37:43 +02:00
Tom Gundersen
2e4063c78b job: drop the job package
This was only used internally in the store package, so moved the
one type that was still used and remove the rest.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-07 10:37:43 +02:00
Tom Gundersen
68ade23fcc store: make state (de)serialization internal
This hides the state hanlding in the store package. As before, it
can be disabled by passing `nil` instead of the path to the state
file.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-07 10:37:43 +02:00
Tom Gundersen
ad637b46f8 osbuild-composer: rename the state file state.json
It is no longer specific to the weldr API, so drop the weldr prefix.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-07 10:37:43 +02:00
Tom Gundersen
b6c63b6d59 store: don't initialize the store with an example
We probably want to ship in a pristine state, so let's leave this out
for now.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-07 10:37:43 +02:00
Tom Gundersen
9cb140795e store: add {Push,Pop}Compose methods and hide channel
Wrap the channel in Pop and Push methods, so it is not exposed to
the callers. PushCompose replaces the old AddCompose for consistency,
and PopCompose simply reads from the other end of the channel.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-07 10:37:43 +02:00
Tom Gundersen
3ff4f59fc7 store: pass the store to the jobqueue API
Drop the jobUpdates channel, and instead add an UpdateCompose method
to the store, which updates the status of a compose directly.

This allows us to report back errors directly, rather than having to
mirror the staet in the jobqueue API.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-07 10:37:43 +02:00
Tom Gundersen
89fd2e6037 store: move creation to main.go
Let the weldr API take the store as an argument, rather than create it
itself. This will allow us to share the store with the jobqueue API in
follow-up patches.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-07 10:37:43 +02:00
Tom Gundersen
4fcb5d66fa store: make the Store a package of its own
This encapsulates all the state of osbuild, no longer just the state specific
to the weldr API. Make it a first class type, preparing for direct access from
both the weldr and jobqueue APIs.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-07 10:37:43 +02:00
Tom Gundersen
2e979c8b82 weldr/store: add getImage
This allows images to be downloaded for completed composes.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-07 10:37:43 +02:00
Tom Gundersen
0880014edf jobqueue: cleanup API a bit and unify the two job stores
Let the store in weldr be the only one that keeps state, and push
updates directly there. This fixes a bug where there was an ID mismatch.

Change the API to not let the caller pick the UUID, but provide it
in the response. Use the same UUID as is used to identify composes,
this makes it simpler to trace what is going on.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-07 10:37:43 +02:00
Tom Gundersen
7f82ef4043 weldr/store: correctly update compose status
The entries in the map are stored by value, we must write them back
after modifying them.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-07 10:37:43 +02:00
Tom Gundersen
02ec086e51 weldr/compose: add support for listing the contents of the queue
This either returns all elements in the queue, or only those matching
a list of UUIDs.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-07 10:37:43 +02:00
Tom Gundersen
b1688cda76 blueprint: add infrastructure for translating to blueprints
Each output type will have its own translator from a blueprint to
a pipeline.

In the future, we may also want to distinguish between different
base distributions and architectures, but for now we keep it
simple and hardcode to a given one.

Some placeholder output formats are added, these have not been
tested or reserached, but are included to show how the it will
all fit together.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-05 20:44:32 +02:00
Tom Gundersen
949d3464b5 weldr: split blueprint into its own package
This will contain all the translation logic from blueprints to pipelines,
which will constitute the main part of composer, so let's separate that
out from the API.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-05 20:44:32 +02:00
Lars Karlitski
329964f2ab image-info: add test
Add two kinds of tests, with one case each:

1. Run image-info against an osbuild pipeline. Uses osbuild
from a submodule to make an image from the pipeline.

2. Run image-info against an existing image, fetched from the internet.
2019-10-05 14:47:35 +02:00
Lars Karlitski
2854b2938e image-info: don't use rpm --verify with --root
RPM uses the host's passwd instead of that in the target to get uid
and gid mappings.

Run it in a chroot to work around this bug.
2019-10-05 14:47:35 +02:00
Jacob Kozol
cefea2d387 test: run api tests against lorax
osbuild-composer's tests can be run against lorax-composer's api in
order to ensure parity with lorax. This can be accomplished by passing
the environment variable LORAX=true into the tests. Otherwise, the
tests run as usual against osbuild-composer's api.
2019-10-04 22:17:44 +02:00
msehnout
7cf8c2e875 Remove RuntimeDirectory
This directory is currently only used for unix-domain sockets but they are created by systemd with the socket unit and therefore this line is redundant and maybe even causes problems.

We want the sockets to live even when the service is down (either because of error or intentional suspension) and it seems that setting the RuntimeDirectory to the directories that contain the sockets makes systemd remove both directories and unlink the sockets when the service fails and that is exactly what we don't want.
2019-10-04 11:17:10 +02:00
Lars Karlitski
5040e6b427 image-info: sort bootmenu entries
The order doesn't matter, but should be the same between different runs
of image-info.
2019-10-04 01:51:07 +02:00
Martin Sehnoutka
4dbba6398a fix worker service template
It is not properly named so it does not work for starting multiple
instances. This should fix it.
2019-10-04 01:49:16 +02:00
Tom Gundersen
859bc0ad2e pipeline: clean up and document
This finishes the initial version of the pipline package, adding
documentation, but still missing unittests.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-03 12:49:21 +02:00
Tom Gundersen
41c6f5dd0b image-info: include partition table id
Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-10-03 12:35:51 +02:00
Martin Sehnoutka
ed185b41ce Integration of osbuild composer with systemd
osbuild-composer now uses socket activation instead of hardcoded paths
in the code. osbuild-worker is an http client therefore it uses only
service unit. osbuild-worker must be started after the socket is
created. osbuild-composer service requires osbuild-worker to run, because without
it no jobs can be started.

osbuild-composer is executed as a regular user (newly created
_osbuild-composer user) as opposed to the worker which must run as root
in order to execute osbuild itself
2019-10-02 17:56:59 +02:00
Jacob Kozol
d23d57bc31 Update api and store for parity with lorax
Add parameters to the api responses that while not displayed by
cockpit-composer are used in props validation or blueprint updates.
2019-10-02 15:25:02 +02:00
Lars Karlitski
b170ea036c image-info: include output from rpm --verify
This shows the changes an image has relative to what its rpm database
thinks is installed. Output is:

    "rpm-verify": {
        "missing": [ <missing files> ],
        "changed": { <map from filename to rpm attribute octet > }
    }

Alas, this makes running image-info slower.
2019-10-01 17:05:10 +02:00
Lars Karlitski
01b7402ce2 image-info: include partition label and uuid 2019-09-30 22:28:28 +02:00
Lars Karlitski
9867846ff4 tools/image-info: consistent use of - over _
And remove a stray comment and whitespace.
2019-09-30 22:28:28 +02:00
Tom Gundersen
e43955a4c5 image-info: include the image format
Use qemu-img to query the image format and include that in the report.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-09-30 15:15:14 +02:00
Lars Karlitski
bf3d7fcdca image-info: add basic support for multiple partitions
For each partition, find out if its the root or boot partition and
gather only the relevant information. Make sure that we don't get
information from /boot twice.
2019-09-30 13:32:53 +02:00
Lars Karlitski
172a23bdb2 image-info: bootloader=unknown instead of creashing 2019-09-30 13:32:53 +02:00
Lars Karlitski
457f5f697e image-info: assume bootable=False if key is not present
sfdisk doesn't include the "bootable" key in its output when a partition
is not marked as bootable.
2019-09-30 13:32:53 +02:00
Tom Gundersen
96f3cbf655 weldr/store/compose: store information exposed in the API
Use the exact same status strings as is used in the API,
making it clearer that they are the same (and avoiding any
translation). Remember the creation/start/finish timestamps.
And store the output type.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2019-09-29 14:51:22 +02:00