Commit graph

696 commits

Author SHA1 Message Date
Lars Karlitski
32d7c68cf1 image-types: start describing image types
This is the start to document the image types that osbuild-composer
produces on a higher level than the code in the `distro` package.

The idea is to document the purpose for each of the image types as well
as the reasons why deviating from a standard install is necessary.
2020-03-26 19:16:30 +01:00
Tom Gundersen
895a4769e3 osbuild: remove dead code
The helper functions were never used, we should aim to use the
osbuild types just as regular structs for serialization purposes.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-26 00:31:37 +01:00
Alexander Todorov
7967ecbbf8 tests: Make TestSources distro independent. Refs #315
by calling `composer-cli sources info` on a user defined repository
2020-03-25 16:30:52 +01:00
Ondřej Budai
f6b3c4a994 test/images: regenerate the test cases
Previous commit changed the output of image-info, therefore the cases must be
regenerated.
2020-03-25 16:14:43 +01:00
Ondřej Budai
a040369513 tools/image-info: deduplicate and sort the services
systemctl list-unit-files doesn't produce machine readable output.
parse_unit_files isn't very good at reading it and can produce duplicate
records.

This commit fixes it by deduplicating and sorting the units. This is a bit
hacky solution, but should work just alright. In the future we might
dump list-unit-files and do the job ourself, but let's not recreate the
systemd logic for now.
2020-03-25 16:14:43 +01:00
Ondřej Budai
9b3375ccca tools/image-info: fix crash when parsing output from systemd >245
systemd 245 added a third column to systemctl list-unit-files, causing
the unpacking to fail.
2020-03-25 16:14:43 +01:00
Ondřej Budai
4c93d5a045 tools/image-info: fix warning when running with python 3.8
Python 3.8 introduced this warning:

tools/image-info:53: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if line[0] is '#':

This commit fixes this warning. More info in Python 3.8 release notes:
https://docs.python.org/3.8/whatsnew/3.8.html#changes-in-python-behavior
2020-03-25 16:14:43 +01:00
Lars Karlitski
3b5d5a73d3 worker: drop default port
We require passing the address from the unit file. Do the same for the
socket, using host:port syntax.

Overriding the port was broken before, because we unconditionally
appended ":8700" to every address.
2020-03-25 14:05:44 +01:00
Lars Karlitski
f8982f4a1a worker: don't hard code path to unix domain socket
Introduce a mandatory argument `address`, which is interpreted as a path
to a unix socket when `-unix` is given or a network address otherwise.

Move the default path to the service file.

Add a more useful usage message when passing `-help` or no arguments.
2020-03-25 14:05:44 +01:00
Lars Karlitski
b5432e78b9 worker: move ComposerClient to jobqueue package
This moves the client code into the same package as the server code,
which makes it easier to change (and version) the two in sync. Also, it
will allow to make some structs private to the jobqueue package and to
test `Client`.

Also rename it to jobqueue.Client.
2020-03-25 14:05:44 +01:00
Lars Karlitski
cb4421b69f worker: remoteAddress → address 2020-03-25 14:05:44 +01:00
Lars Karlitski
94183d14a8 worker: split NewClient()
Use the default dialing functions for tcp connections and set the tls
config on the transport directly. This makes the code easier to follow,
because the only special case is overriding the DialContext() for unix
connections.
2020-03-25 14:05:44 +01:00
Lars Karlitski
845ba6e8e5 worker: don't hard code upload URL
This doesn't work with remote workers.
2020-03-25 14:05:44 +01:00
Lars Karlitski
9e71df234a worker: load tls certificates once on startup 2020-03-25 14:05:44 +01:00
Lars Karlitski
16cd243300 worker: set remoteAddress once on startup 2020-03-25 14:05:44 +01:00
Ondřej Budai
ef7e8e0772 drop packit configuration file
Packit is not used currently, therefore let's drop its configuration from the
repository. The change can be always reverted if we change our minds.
2020-03-25 11:43:39 +01:00
Ondřej Budai
0d5c2d141c spec: add notes to remove the lines related to renaming in F34
The packaging guidelines requires that these provides and obsoletes stays
in the spec file for 2 fedora versions. The name change was introduced in
Fedora 32, there the lines should be removed in F34.
2020-03-25 11:43:39 +01:00
Ondřej Budai
247e8cb205 update readmes to use the new osbuild-composer package 2020-03-25 11:43:39 +01:00
Ondřej Budai
b45119c471 spec: sync the line order and whitespace changes
In the new spec file, the tests-related code is always at the bottom of the
containing block to make the conditional code easier to read. Do the same
in the old spec file.

Also, in the new spec file, some whitespaces are different (rpmlint reported
mix used of tabs and spaces), let's do the same changes in the old spec file.
2020-03-25 11:43:39 +01:00
Ondřej Budai
cdc4248909 spec: use the standard way to represent a go dependency
This isn't a functional change.
2020-03-25 11:43:39 +01:00
Ondřej Budai
64181e7ba0 ci/rpm: use the old spec file for f31
Prior this commit the rpm for fedora 31 was built using the new spec file.
The old spec file should be used for that Fedora version.
2020-03-25 11:43:39 +01:00
Ondřej Budai
48a51611ec makefile: use the new spec file for rpm targets
This commit switches the rpm and srpm targets to use the new spec file.
Also, it introduces old-rpm and old-srpm targets to build an rpm from
the old spec file.

Note that the rpm target builds also the tests subpackages (they are not
built by default).
2020-03-25 11:43:39 +01:00
Ondřej Budai
0b08eb1904 spec: add spec file for osbuild-composer package
We successfully renamed the package in Fedora to osbuild-composer
(RHBZ #1815577). This commit adds the new spec file.

Changes: the tests subpackage is not built by default, to build it
--with tests parameter must be passed to rpmbuild.

Let's hope both spec files stay in sync.
2020-03-25 11:43:39 +01:00
Lars Karlitski
ee752b0ab8 tree-wide: panic when json marshalling fails
According to the new guidelines in docs/errors.md.

Note that this does not include code that marshals to a writer that
might fail (when a connection drops, for example).
2020-03-25 10:22:16 +01:00
Lars Karlitski
ad1a12f7aa tree-wide: use uuid.MustParse() when possible
According to the new guidelines in docs/errors.md.
2020-03-25 10:22:16 +01:00
Lars Karlitski
9625c589b0 docs: add errors.md
docs/errors.md are guidelines for how we handle errors. It's meant to be
expanded as we figure out more rules.
2020-03-25 10:22:16 +01:00
Brian C. Lane
899d38422c weldrcheck: Adding integration tests for modules 2020-03-25 10:01:14 +01:00
Brian C. Lane
b5da15e2d8 client: Add /modules/ support 2020-03-25 10:01:14 +01:00
Brian C. Lane
d530d9a87e weldr: Fix projects/depsolve route handling
Return a 400 with json error message when no packages are included, with
or without trailing /
2020-03-25 10:01:14 +01:00
Brian C. Lane
3bddfd3449 weldrcheck: Adding integration tests for projects 2020-03-25 10:01:14 +01:00
Brian C. Lane
271d27a41d client: Add /projects/ support 2020-03-25 10:01:14 +01:00
Brian C. Lane
5ac3cb3f46 rpmmd: Fix the format of BuildTime to match the weldr API
The format of the BuildTime returned by /projects/list and /modules/list
does not include the 'Z' at the end. This fixed the format and adjusts
the tests.
2020-03-25 10:01:14 +01:00
Lars Karlitski
ce6dc53a12 weldr: also pass custom sources to Store.PushCompose()
Also move assembly of a list of all repositories (base + custom sources)
into its own function.
2020-03-24 22:29:41 +01:00
Lars Karlitski
3544590036 store: move adding the local target to weldr
The automatic local target is only needed when accessing the API via
weldr.

In the store, the target was only added when `stateDir` was not `nil`.
This is only used for testing which doesn't exercise the branch in
weldr. Thus, the same check is not needed there.
2020-03-24 22:29:41 +01:00
Lars Karlitski
7594cb262e store: remove job.ImageType
It is not used.

We probably need it at some point to decide which worker to give which
job to.
2020-03-24 22:29:41 +01:00
Lars Karlitski
25e926bc36 store: use Job.ImageBuildID to fetch image build
ImageBuildID is an index into Compose.ImageBuilds. Use that directly
instead of looping over ImageBuilds and matching on OutputType.
2020-03-24 22:29:41 +01:00
Lars Karlitski
f386ae43dc jobqueue: remove Job.OutputType
It is not used. The worker does not need to know the name of the
originating output type. It has all information it needs in the
manifest.
2020-03-24 22:29:41 +01:00
Martin Sehnoutka
e43a30e4cf osbuild-dnf-json-tests: Introduce depsolving test
This test makes sure we can run depsolve for all build packages sets and
base packages sets for all image types for all architectures for all
defined Fedora versions. In could run the same for RHEL, but I haven't
yet implemented it, because such tests cannot run in public Internet.
2020-03-24 20:45:30 +01:00
Martin Sehnoutka
e1e5c1bb0b distro: include xz in distro build packages
xz compression is used for the ami and tar output types, it should be in
the buildroot. This fixes Weldr integration test (/cmd/osbuild-tests) on
ARM architecture (aarch64).

The fact that it worked on x86_64 was just a happy coincidence because
we require grub2-pc which in turn requires dracut which requires xz. We
should not rely on these implicit dependencies because we need xz
unconditionaly, therefore adding it to build packages for all platforms.
2020-03-24 20:45:30 +01:00
Martin Sehnoutka
7e69299259 dnf-json: allow passing arch as an argument
dnf can do cross-arch depsolving, but in case repositories for multiple
arches are provided and it is running on a different architecture than
the image build requires, it can lead to errors.

This patch makes sure that we only include packages from the
architecture we want.

It uses substitutions as defined in dnf documentation:
https://dnf.readthedocs.io/en/latest/api_conf.html#dnf.conf.Conf.substitutions
Unfortunately the docs are very sparse on details and the Fedora docs
are not updated any more:
https://docs.fedoraproject.org/en-US/Fedora/26/html/System_Administrators_Guide/sec-Using_DNF_Variables.html
Also dnf team is migrating the code to libdnf:
https://github.com/rpm-software-management/libdnf
which does not yet have any documentation.
2020-03-24 20:45:30 +01:00
Martin Sehnoutka
9d2dacbcab internal: pass architecture from the APIs to dnf-json
The following commit will introduce support for forced architecture in
dnf-json. The APIs already have this kind of information, so we can
simply pass it to the Depsolve and FetchMetadata functions.
2020-03-24 20:45:30 +01:00
Jakub Rusz
6f1c461c32 tests: use testify for existing blueprint_test functions 2020-03-24 20:37:10 +01:00
Jakub Rusz
833e798f1b tests: test for blueprint.GetPackages()
This also tests Package with "*" specified as version
2020-03-24 20:37:10 +01:00
Alexander Todorov
4c39223497 tests: Add coverage notes and link to Coveralls in docs
also updates the path to the resulting RPMs b/c that seems to have
chagned.
2020-03-24 20:27:16 +01:00
Tom Gundersen
a385d91565 osbuild-pipeline: print more helpful info in case invalid input is given
Use the List*() functions to print the valid options in case an invalid
one is given.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-24 15:46:57 +01:00
Tom Gundersen
f903601ec4 distro: add ListArchs() to Distro interface
Similar to ListImageTypes() add a helper to enumerate the supported
architectures.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-24 15:46:57 +01:00
Tom Gundersen
4a7519807e common/Architecture: drop unused code
The intended use for this will be repalced by Arch objects, currently
it was unused, so drop it.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-24 15:46:57 +01:00
Tom Gundersen
c66db42677 weldr/upload: uploadRequestToTarget() cannot fail
Drop the error return and simplify code accordingly.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-24 14:13:03 +01:00
Tom Gundersen
87b0bb6e5d distro: mass cleanup
Delete unused methods and make types and fields private where
possible. Some code is moved around, but apart from that there
is no change in behavior.

The naming of the distros were moved back into the distro
packages as the common types now only had one user, and this
allowed us to drop some redundant error checking.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-24 14:13:03 +01:00
Tom Gundersen
cf2ad51243 compose/ImageBuild: drop the distro field
This likely needs to be reintroduced in some fashion, but it was
unused, and when we reintroduced it it should be as a real Distro
object.

For now, drop it.

Signed-off-by: Tom Gundersen <teg@jklm.no>
2020-03-24 14:13:03 +01:00