Move OSTree option handling outside of the weldr API to make it usable
by other packages. New subpackage at internal/ostree.
Add support for ostree options ("Ref" and "URL") in the Cloud API.
Validate OSTree options and resolve the parent reference the same way as
in the Weldr API.
Unlike the Weldr API, the Cloud API doesn't support specifying the
Parent reference directly.
The exports list is included in the job information on the queue.
Modify composer to use RepoRegistry, instead of loading the host
repositories, when initializing WeldrAPI.
Modify WeldrAPI to use RepoRegistry, instead of a map of repository
definitions. Make sure that the RepoRegistry method specific to image
type is used in Welder where appropriate. Specifically when depsolving a
Blueprint, which is used to build a specific image type. Update Weldr
API unit tests to reflect the change.
Add a new method to RepoRegistry, allowing to get list of repositories,
which should be used for building an image for a given architecture,
without specifying the exact image type. Add relevant unit tests.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
fedoratest was yet another dummy distribution used by unit tests. After
the rework of test_distro, there is no reason to not use it as the only
distro implementation for testing purposes.
Remove fedoratest distro and replace it with test_distro in all affected
tests.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Validates the ref only when supplied through the API (i.e., doesn't
validate built-in defaults).
Regex matches ostree internal and cockpit-composer UI validation.
Added test case to compose API test.
Replacing repeated calls to u.Parse() with path.Join() on the URL's
path. This method handles certain edge cases differently:
- location not ending in / (http://example.org/repo):
- with the old method, the subsequent parsing of "refs/heads/" would
overwrite the path segment of the original URL, resulting in
http://example.org/refs/heads
- with the new method, "refs/heads" is appended to the location and
a / is added between the two parts if necessary.
- ref begins with / (location: http://example.org/repo/, ref: /ref):
- with the old method, the final parsing of ref would overwrite the
path segment of the URL, resulting in http://example.org/ref
- with the new method, the ref is appended and a / is added between
parts where necessary (same as above).
- ref is a full URL
(location: http://example.org/repo/, ref: http://example.com):
- with the old method, u.Parse(ref) would completely overwrite the
existing URL in u.
- with the new method, the ref is added as a sanitised URL path
resulting in http://example.org/refs/heads/http:/example.com.
The last one will probably result in an error in either case, but it's
probably less incorrect to coerce the ref argument into a path.
The response status code of the GET request is checked as well to
provide an appropriate error message if it is not 200 (OK).
If the data in the response is not a valid hex string, the error message
from the DecodeString() method isn't returned directly and it is
replaced by a more useful message. The original error message is
discarded.
This adjusts current tests to account for the new struct member, and
tests osbuild-composer with empty results (eg. existing system will not
have this stored) and with the sets populated by test data.
This adds the compose's dependency list which was previously missing
from the osbuild-composer implementation of the WELDR API.
The dependencies used for the compose are saved, at compose time, in the
store. They are returned as part of the compose/info results, the 'deps'
field.
We need to add the URL to the manifest as an ostree source repo so that
osbuild can pull the commit to embed it in the boot ISO for the new
rhel-edge-installer image type.
osbuild now supports using the `--export` flag (can be invoked multiple
times) to request the exporting of one or more artefacts. Omitting it
causes the build job to export nothing.
The Koji API doesn't support the new image types (yet) so it simply uses
the "assembler" name, which is the final stage of the old (v1)
Manifests.
This replaces Packages() and BuildPackages() by returning a map of
package sets, the semantics of which is up to the distro to define.
They are meant to be depsolved and the result returned back as a
map to Manifest(), with the same keys.
No functional change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Note that this doesn't actually test for the ostree fields, I'm not sure
if that's possible with this test framework. But it does make sure that
a test compose won't try to fetch the url.
For now this is simply used to resolve the parent commit, in case
one is not provided. In the future it will be used by new image
types to actually pull content from.
This extends the weldr API, so that future work does not have to
modify that.
The logic we now implement for the ostree commit image types is:
If the URL is provided, but the parent commit is not. The parent
commit is taken to be the current HEAD of the ostree repo at the
given url, with the given (or default) ref.
This only provides a small optional convenience, but we will
soon introduce image types where the URL of the repository is
required.
This commit still needs testing.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Rather than setting this automagically, expose it to the caller. For
now the only caller we have simply passes it back in, so this is a
noop.
In follow-up commits this will be used to resolve the parent commit.
This is tested by verifying that the generated manifests do not
change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
There is some confusion surrounding the format of the source TOML that
can be sent to the server. The format it accepts doesn't match the
output from composer-cli which includes the source id in [] eg.
[k8s]
name = "kubernetes packages"
...
This patch changes the parsing to allow the id to be set as 'id = "k8s"'
or passed as a map in [k8s]. If the id is passed in the body it takes
precedence over the map name.
System repos cannot be overridden by users, return an error if they try
to push a source with the same name/id as a system source.
Resolves: rhbz#1915359
testjobqueue did not implement the JobQueue interface correctly (noted
in its package comment), making it impossible to write tests for
JobQueue itself.
Replace its use everywhere with fsjobqueue operating on a temporary
directory.
I thought rand in Go is auto-seeded but I was wrong, see [1].
This commit adds seed initialization.
[1]: https://golang.org/pkg/math/rand/#Seed
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Imagine this situation: You have a RHEL system booted from an image produced
by osbuild-composer. On this system, you want to use osbuild-composer to
create another image of RHEL.
However, there's currently something funny with partitions:
All RHEL images built by osbuild-composer contain a root xfs partition. The
interesting bit is that they all share the same xfs partition UUID. This might
sound like a good thing for reproducibility but it has a quirk.
The issue appears when osbuild runs the qemu assembler: it needs to mount all
partitions of the future image to copy the OS tree into it.
Imagine that osbuild-composer is running on a system booted from an imaged
produced by osbuild-composer. This means that its root xfs partition has this
uuid:
efe8afea-c0a8-45dc-8e6e-499279f6fa5d
When osbuild-composer builds an image on this system, it runs osbuild that
runs the qemu assembler at some point. As I said previously, it will mount
all partitions of the future image. That means that it will also try to
mount the root xfs partition with this uuid:
efe8afea-c0a8-45dc-8e6e-499279f6fa5d
Do you remember this one? Yeah, it's the same one as before. However, the xfs
kernel driver doesn't like that. It contains a global table[1] of all xfs
partitions that forbids to mount 2 xfs partitions with the same uuid.
I mean... uuids are meant to be unique, right?
This commit changes the way we build RHEL 8.4 images: Each one now has a
unique uuid. It's now literally a unique universally unique identifier. haha
[1]: a349e4c659/fs/xfs/xfs_mount.c (L51)
Most of the worker API is now untyped, but keep Enqueu() typed to
ensure the job objects match the names in the queue. This means we
must add a version of Enqueue() for each job type we support.
Similarly to the recent changes to Dequeue(), let the caller unmarshal the
return JSON. This allows us to pass the result on without being able
to unmarshal it.
In follow-up patches, we will pass results of jobs to dependent jobs,
but the worker API does not know about the different job types, nor how
to unmarshal them.
The worker server was heavily tied to OSBuildJob(Result). Untie it so
that it can deal with different job types in the future.
This necessitates a change in the jobqueue: Dequeue() now returns the
job type, as well as job arguments as json.RawMessage. This is so that
the server can wait on multiple job types with different argument
types.
The weldr, composer, and koji APIs continue to use only "osbuild" jobs.
Workers reported status via an `osbuild.Result`, which only includes
osbuild output. Make it report OSBuildJobResult instead, which was meant
to be used for this purpose and is already used as the result type in
the jobqueue.
While at it, add any errors produced by targets into this struct, as
well as an overall success flag.
Note that this breaks older workers returning the result of an osbuild
job to a new composer. I think this is fine in this case, for two
reasons:
1. We don't support running different versions of the worker and
composer in the weldr API, and remote workers aren't widely used yet.
2. Both osbuild.Result and worker.OSBuildJobResult have a top-level
`Success` boolean. Thus, logs are lost in such cases, but the overall
status of the compose is not.
Add "image_name" and "stream_optimized" fields to the osbuild job as
replacement for the local target options. The former signifies the name
of the uploaded artifact and whether an artifact should be uploaded at
all (only weldr API). The latter will be deprecated at some point, when
osbuild itself can make streamoptimized vmdk images.
This change separates what have always been two distinct concepts:
artifacts that are reported back to the composer node (in practice
always running on the same machine), and upload targets to clouds and
such. Separating them makes it easier to add job types that only allow
one upload target while keeping artifacts.
Keep the local target around, so that jobs that are scheduled can still
be run after an upgrade.
ComposeState is only used by the weldr API.
Drop the JSON marshaller and unmarshaller, because ComposeState is not
used in an JSON-exported field anymore.
This state is specific to weldr. Previous commits removed it from the
other APIs, because they use different values.
Move the conversion into the weldr API.
In the same way `osbuild.Manifest` is the input to the osbuild API,
`osbuild.Result` is the output. Move it to the `osbuild` package where
it belongs.
This is not a functional change.
Signed-off-by: Tom Gundersen <teg@jklm.no>
vCenter requires images to be uploaded as vmdk StreamOptimized. Lorax
always produced images on this format, so we should make sure to do the
same for our VMWare images.
Allow LocalTarget to request the images produced by osbuild be converted
to be streamOptimized before saving in composer, and hook the weldr API
up to enable this option for vmdk images.
Ideally this should simply be an option in osbuild, but that would
require some more work, which we will not manage in time for RHEL8.3.
Therefore do this minimal fix.
Note that that means the images produced by our manifests (including in
our image-test test cases) are not on the format that the weldr API
returns, so the tests we run on them would also, for now, need to
convert before uploading to vCenter.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Change the translation from our internal structs to the structs used for
weldr serialization to drop account details. These must obviously be
passed in to configure an upload, but exposing them in the logs may be
surprising.
There is no notion of user accounts in the weldr API, and the state
should not be considered private. However, this is likely to take people
by surprise, so let us guard the secrets entrusted to us.
Fixes#907.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Improve the message returned by osbuild-composer when a user asks for
logs of a compose that is still running.
Signed-off-by: Major Hayden <major@redhat.com>
The API was crashing if the freeze request was called on a non-existent
blueprint. This changes it to return an empty string, matching
lorax-composer's behavior (since the output is toml it shouldn't return
json).
This changes the response to match lorax-composer's behavior. If any of
the blueprints in the list passed to /blueprints/depsolve/... have an
error that error should be appended to the error list, and the blueprint
included in the blueprints list with an empty dependencies section.
It was returning an error 400 and a single error if it hit any depsolve
problems, skipping any other blueprints and returning the wrong
response.
This also adjusts the tests to account for the change.
Fixes#890
By default, go's tar archiver uses USTAR header format. Unfortunately, this
format doesn't support sub-second resolution for ModTime. Go solves this by
*rounding* the time. Sometimes, this creates an archive containing a file
with modtime from the future. When such archive is untarred by GNU tar,
the following message is produced:
tar: bf548dfd-0a90-40e6-bbf2-dcdd82fcbb4e.json: time stamp 2020-07-13
13:34:31 is 0.356223173 s in the future
We have two options here:
1) Use gnu header format that supports sub-second resolution. Unfortunately,
it seems that not all tar archivers support this format (e.g. 7-zip).
2) The other option is to truncate the date (instead of rounding).
I went with option 2.
Also, this commit adds a test to check that the header is not from the future.
Without this fix, the test is actually failing, I verified this manually.
Fixes#854
Rather than getting a set of base packages from the ImageType, and then
appending the requested packages from the blueprint, pass the blueprint
into the new Packages() function, and return the full set of packages to
be depsolved.
This allows us to also append packages based on other customizations
too, and use that to append chrony when the timezone is set. This
matches the behavior anaconda had, and there was a TODO item to do this,
which had been overlooked.
Fixes#787.
Signed-off-by: Tom Gundersen <teg@jklm.no>
The sources weldr API already supports this, so hook it up to be
represented on disk and in our internal state tracking too.
This does not yet hook this up to be respected by osbuild, which
currently takes this to be unconditionally set to true.
Signed-off-by: Tom Gundersen <teg@jklm.no>
