Commit graph

775 commits

Author SHA1 Message Date
Lars Karlitski
2435163fc9 worker: move running osbuild into separate function
Setting up a command to run is quite involved. Separate that from the
logic of running it.
2020-04-06 12:11:54 +02:00
Lars Karlitski
ada058f2c3 jobqueue: move Job struct to client.go
`Job` is now a simple data struct, which is only used to return
multiple arguments from `AddJob()`. Move it to that file.
2020-04-06 12:11:54 +02:00
Lars Karlitski
1ece08414c jobqueue: move Job.Run() to the worker
This makes the jobqueue package independent of forking osbuild, the
choices for which (exact invocation, location of the cache directory)
should be made in the worker.
2020-04-06 12:11:54 +02:00
Lars Karlitski
d3b9a3515d worker: inline handleJob()
It's a small function that's only called once.
2020-04-06 12:11:54 +02:00
Lars Karlitski
db5dd1ee2c worker: remove redundant UpdateJob() call
A job is already set to be running when it is returned from the API (see
Store.PopJob()).
2020-04-06 12:11:54 +02:00
Lars Karlitski
54b63a645f jobqueue: remove JSON tags on Job
Job is not used for serialization anymore.
2020-04-06 12:11:54 +02:00
Lars Karlitski
ab2c8aff08 jobqueue: make import declarations consistent
The usual style is three blocks: standard library, external
dependencies, internal dependencies.

Also run through `go fmt`, which sorts each of those blocks
alphabetically.
2020-04-06 12:11:54 +02:00
Lars Karlitski
1f06d78362 jobqueue: rename ID to ComposeID in job structs
It's not an id of the job, but the compose id.
2020-04-06 12:11:54 +02:00
Lars Karlitski
269988a737 jobqueue: start moving API types to json.go
Similar to `weldr/json.go`, this file will contain structs that are
shared between client and server and only meant to be used to
serialize from and to JSON.

This immediate change allows us to make fields in `Job` private in a
future commit.
2020-04-06 12:11:54 +02:00
Ondřej Budai
9a96908c1e upload/aws: delete the s3 object after the snapshot is imported
After the snapshot is imported to EC2, there's no point in having the s3
object anymore. If the s3 object is not deleted, the user will be charged
for it periodically, while the object doesn't provide any value.
2020-04-04 02:49:00 +02:00
Major Hayden
2f94a87dab 🎲 Enable rcm integration tests
Now that Jenkins is hooked up to the repository, let's run the rcm
tests after we deploy all of the services into a VM.

Signed-off-by: Major Hayden <major@redhat.com>
2020-04-03 19:38:10 +00:00
Tom Gundersen
8797cd4704 common: drop unused image types
The Alibaba and GCE image types were not used, and they were
causing problems when translated from and to strings as they were
mapping to the same internal image type. The end result was that
AWS jobs were shown as Alibaba or GCE in the UI.

This fixes #369.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-04-03 08:46:50 +02:00
Major Hayden
b57f70e3e5 🔌 Plug osbuild-composer into Jenkins
Let's throw schutz to the wind and add Jenkins to the osbuild-composer
repository. Jenkins will execute the `jenkins/run_tests.sh` script and that
will deploy osbuild, osbuild-composer, and cockpit-composer on Fedora 31
and 32.

We can add extra integration tests gradually after this is merged.

Signed-off-by: Major Hayden <major@redhat.com>
2020-04-02 21:11:37 +00:00
Jakub Rusz
8861c6dbb9 tests: increase coverage of internal/crypt
fixes #444
2020-04-02 20:12:06 +02:00
Jakub Rusz
4f160447ee tests: use asserts in states_test.go 2020-04-02 09:17:09 +02:00
Jakub Rusz
1cf2af10d0 tests: add coverage for common/helpers.go
This also changes the helpers.go to enable mocking all the different architectures.
2020-04-02 09:14:40 +02:00
Ondřej Budai
09109b4f8e 9
Release osbuild-composer version 9
2020-04-01 15:56:57 +02:00
Ondřej Budai
c081844954 ci/github: bump the checkout action to version 2
actions/checkout@v1 sometimes fails, bumping the version should fix it. See:
https://github.com/actions/checkout/issues/23
2020-04-01 15:03:48 +02:00
Ondřej Budai
bbd71efeff distro/rhel: remove dracut-config-generic from build packages
The build pipeline doesn't run dracut nor install kernel, the package is
therefore useless in it.
2020-04-01 12:29:59 +02:00
Ondřej Budai
c48728a049 distro/rhel: move dracut-config-generic to bootloader packages
This change makes the dracut-config-generic installation in par with
Fedora. Also, by having it in bootloader packages, it is not installed in
non-bootable images, where it provides no value.
2020-04-01 12:29:59 +02:00
Ondřej Budai
f14224ede1 distro/fedora: add dracut-config-generic to bootloader packages
Dracut is unfortunately very host-dependant by default. The package
dracut-config-generic forces it use a generic configuration instead of a
configuration generated from the host environment.

This change should make the image generation more reproducible. For example
it was not possible to boot ami images built on Travis on AWS prior this
commit.

Also, the tests were re-generated in this commit.
2020-04-01 12:29:59 +02:00
Jiri Kortus
b0e28689fb Add unit test for systemd stage
Refs: #406
2020-04-01 09:52:10 +02:00
Lars Karlitski
3a957f4b16 store: don't add custom repositories in PushCompose()
The weldr API already passes custom repositories, so they end up in the
list twice. This is a bit awkward, because weldr passes something to the
store that the store already knows about. It only works this way right
now though, because the rcm API doesn't need these repositories at all.
It expects all repositories to be passed.
2020-04-01 00:29:13 +02:00
Lars Karlitski
cef7267a2d store: create compose id from PushCompose
Every caller had to make their own uuid and pass that into
PushCompose(). Make PushCompose() create the id and return it.
2020-04-01 00:29:13 +02:00
Lars Karlitski
a6338410bc rcm: remove outdated comment
PushCompose() does not inject a local target anymore.
2020-04-01 00:29:13 +02:00
Lars Karlitski
eb10cf3124 store: remove unused arguments from PushCompose 2020-04-01 00:29:13 +02:00
Jiri Kortus
e516180640 Add unit tests for SELinux stage
Refs: #406
2020-04-01 00:26:22 +02:00
Jiri Kortus
44b501d401 Add unit test for timezone stage
Refs: #406
2020-04-01 00:25:59 +02:00
Jiri Kortus
8dafb64f1b Add unit tests for script stage
Refs: #406
2020-04-01 00:25:34 +02:00
Jiri Kortus
8c808831d3 Add unit test for users stage
Refs: #406
2020-04-01 00:24:39 +02:00
David Rheinsberg
6319d52fe4 docs: import changelog as NEWS.md
Import a new `NEWS.md` file with the changelog for each release. This is
targetted at distributors, so it should contain information on new
features, modified features, and anything important to packagers. This
is not a replacement for the git-changelog, but rather a human-readable
instruction to packagers what needs to be changed when using the new
release.

Additionally, copy `make release` from *osbuild* to help in creating new
release entries in NEWS.md.
2020-04-01 00:22:59 +02:00
David Rheinsberg
1c32a30092 docs: rework README and align with osbuild
This aligns the layout of `README.md` with the osbuild project. It now
has a clear structure with individual sections that we can continue to
improve on.

This tries to salvage most of the information that was previously in
README.md, but removes the bits and pieces that were either moved into
the osbuild-composer(7) man-page, or feels misplaced in a project
readme.
2020-04-01 00:22:59 +02:00
David Rheinsberg
d344e967e5 build: rework Makefile for man-page target
This adds `make man` with SRCDIR and BUILDDIR support. This is copied
from *osbuild*. Since `make man` will be the official way to build the
man-pages, lets also import `make help` and some documentation for the
Makefile.
2020-04-01 00:22:59 +02:00
David Rheinsberg
b36227e4da docs: add osbuild-composer(7)
Create a new man-page source for `osbuild-composer(7)`. This will be
the overview man-page for composer, as well as a good entrypoint to
learn about the entire osbuild setup.

For now, just include some basic information, as well as copy the bits
from README.md that make sense.

Contributions welcome! This could be a lot more verbose and helpful.
2020-04-01 00:22:59 +02:00
David Rheinsberg
c5293edbab docs: break lines in markdown
Avoid overlong lines in markdown files. This does not affect the
rendered output, but makes it a lot easier to navigate through the file
from a text-editor when editing. Furthermore, it improves *Diffs* of
our commits, and allows reading markdown without rendering it first.
2020-04-01 00:22:59 +02:00
Jiri Kortus
ea02b33d2b Add unit test for RPM stage
Refs: #406
2020-04-01 00:19:43 +02:00
Jakub Rusz
703039495b tests: add coverage for internal/blueprint/customizations.go 2020-03-31 16:47:00 +02:00
Jakub Rusz
4807b64dea tests: add coverage for compose_result.go 2020-03-31 16:32:34 +02:00
Jakub Rusz
2b87ca41b5 Separate nested structure in compose_result.go
This separated the nested structure into their own struct types.
2020-03-31 16:32:34 +02:00
Ondřej Budai
28d0c4f640 tools: remove accidentally added line in test case generator
The line slipped in f945c505 when testing the changes. Let's remove it,
it breaks the script.
2020-03-31 14:40:37 +02:00
Lars Karlitski
c5823d92b4 spec: build test binaries against installed libraries
On Fedora, the `%gobuild` macro turns off go modules and sets gopath to
installed libraries (in `/usr/share/gocode`).

When building the test binaries, we used `go test -c` directly, which
downloaded the libraries into GOPATH. That's wasteful and doesn't work
in mock (without internet). Replicate what `%gobuild` does by setting
`GO111MODULE=off` and `GOPATH` to the correct value.

RHEL uses modules as intended. No change is necessary.
2020-03-31 09:48:23 +02:00
Jiri Kortus
600a0f50a9 Add unit test for locale stage
Refs: #406
2020-03-31 09:35:20 +02:00
Jiri Kortus
c4d5edaf0d Add unit tests for Pipeline struct
Refs: #406
2020-03-31 09:35:11 +02:00
Brian C. Lane
107ecde90b store: Fix setting interrupted composes to failed at startup
If a compose was interrupted by restarting osbuild-composer it should be
set to failed at startup. This was not working because imgBuild is a
temporary variable, the value stored in ImageBuild needs to be modified
directly.
2020-03-31 09:34:25 +02:00
Jiri Kortus
f5ad337c62 Add unit test for keymap stage
Refs: #406
2020-03-30 20:45:33 +02:00
Jiri Kortus
c2ed6da92e Add unit test for hostname stage
Refs: #405
2020-03-30 20:44:16 +02:00
Lars Karlitski
144570026d .github: switch to codecov.io
coveralls doesn't work from GitHub actions. Its "github" service type
uses the GITHUB_TOKEN from the action, which only has read access when
invoked from a forked repository.

codecov gets this right: it validates that an uploaded coverage file
originated from a GitHub action run by asking GitHub, and then uses its
OAuth credentials (through the Marketplace App) to comment and set
status.

Also, coveralls requires a third-party package to convert go's coverage
report to a format it understands. codecov detects the format
server-side. It also handles go's coverage format better: it highlights
lines with "some coverage" in yellow (coveralls has no concept of this).
2020-03-30 20:43:34 +02:00
Tom Gundersen
7f6d98ef74 repositories/RHEL: add trailing newline to the GPG keys
rpmkeys in RHEL-8.1 fails if there is no trailing newline, so just
add it to all the keys for consistency. The fedora keys already
had this.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-30 20:05:25 +02:00
Tom Gundersen
b6ebe6ec1e repositories/RHEL: use repositories for the latest releases
Also include the GPG keys. When we first started testing we were
testing against nightlies, whose packages were not signed. But
when using released content (which is signed) the GPG keys are
required.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-30 16:09:41 +02:00
Martin Sehnoutka
92145822d9 distro: rename ListArchs to ListArches
In PR#395 we discussed the spelling of archs vs. arches and we agreed to
use arches. This patch only renames the public method `ListArchs`in the
`Distro` interface.
2020-03-30 10:41:02 +02:00