Previously, the partition table definition was defined inside an assembler.
This has an issue though: The partitions and filesystems are needed at several
other places, e.g. grub2 stage and fstab stage. As the partition table was
basically hardcoded, this didn't matter - we could just use constants
in these stages. Not ideal but it worked.
This commit changes the behaviour: A partition table is firstly created and
then it's passed to the assembler function where complete assembler options
are created out of it.
To make this change as small as possible, osbuild.QEMUAssemblerOptions type
is used to encode the partition table for now.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
require.JSONEqf cannot handle diffs of such a big entity as a manifest is.
It just prints an empty string.
This commit unmarshalls the manifests instead and then uses the cmp library
to make a very nice and readable diff.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The kernel options are updated to remove the read only option "ro" from
the qcow2 and amazon image types. Also, the qcow2's kernel options are
updated to only set console=ttyS0 once. It was declared twice which is
redundant so now it is set for both tty0 and ttyS0.
rng-tools is no longer included as a package in the RHEL 8.4 image. This
package is both removed from being an included package and also
specifically declared as excluded. The test manifests are updated.
The RHEL 8.4 specific runner was introduced in osbuild 22, released
on the 8th of October 2020. It should by now be in relevant Fedora
releases and RHEL 8.4.
Adapt the corresponding test cases.
Previously, the compose status returned failure as soon as possible.
koji-osbuild considers the job as done when its status == failure and proceeds
with uploading the logs to koji and marking the job as failed. However, not
all osbuild-composer jobs might be done at this point so the logs might be
incomplete making the debugging hard.
This commit changes the behaviour: Now, the compose status is pending until
ALL jobs belonging to it are finished.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Previously, we had no clue what errors were catched by the default echo's
error handler. Thus, in the case of an error, we were basically blind. Let's
log all errors so we can investigate them later.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
%#v was my bad understanding of Go's error formatting. Let's use the standard
%v that gives saner and human-readable error messages.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Hybrid boot, i.e. MBR+UEFI, is only supported on x86_64, but not
on aarch64, which only supports UEFI. Therefore, disable legacy,
i.e. MBR, boot for aarch64 and remove the corresponding BIOS boot
partition.
Since hybrid boot is always used on x86_64, the "uefi" parameter
is removed completely, together with the "legacy only" option on
x86_64. Thus we are left with a simple one to one mapping of
architecture to qemu assembler stage options in `qemuAssembler`.
Previously, baseurl was required in openapi.yaml. In order to add support
for metalink and mirrorlist repos as well, make all optional, since openapi
does not support mutually exclusive parameters. Instead, enforce this logic
in server.go, and if no repo has been specified, return a 400 bad request error.
test.API is actually just http.Handler, let's use this interface directly
instead of defining our own one.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
These packages are architecture-specific. Including them in the generic
package list broke image builds for everything but x86_64.
The corresponding architecture-dependant packages were already included
in `arch.bootloaderPackages` (some via dependencies).
Fixes#1103
The previous code was smelling a bit (e.g. Server.server field) so I decided
to rewrite it in the style of the much nicer koji server.
Not a functional change.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
cockpit-composer can now build rhel 8.4 images. Our distro name for
rhel 8.4 is rhel-84 unlike prior rhel releases which fall
under the umbrella name rhel-8. rhel 8.4 still uses the same
repos as the rest of the rhel 8 releases but points to a different
nightly repo for testing purposes. Test cases are added. The changes
between rhel 8.3 and 8.4 are as follows:
There is now a hybrid boot partition scheme for x86_64. x86_64 images
now use uefi boot and have 3 gpt partitions: a small unformated
partition for mbr compatibility, an efi boot partition of type vfat, and
a root partition of type xfs. The packages grub2-efi-x64 and shim-x64
are added as bootloader packages for all x86_64 images.
For qcow2 images ro is added as a kernel option and the following
packages are added (+) or removed (-):
+ dosfstools
+ efi-filesystem
+ efivar
+ efivar-libs
+ grub2-efi-x64
+ shim-x64
- rhn-client-tools
- rhnlib
- rhnsd
- rhn-setup
Serializing an interface does not work, let us simply use the string
representation and treat the empty string as no error. This is
compatible with the current API in the success case, and fixes the
error case, which is currently broken.
Also extend the test matrix for the kojiapi to ensure that all the
different kinds of errors can be serialized correctly and leads to
the correct status being returned.
Fixes#1079 and #1080.
Soon, we want to begin tagging the jobs with the name of its submitter.
The simplest way to add a tag to a job is to put it into its type string.
However, as we don't know (and don't want to know) the submitters' names when
osbuild-composer is initialized, we need to be able to push arbitrary job
types into the jobqueue.
This commit therefore lifts the restriction that a jobqueue accepts only
a predefined set of job types. Now, jobqueue clients can push jobs of
arbitrary names.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Now that all interaciton with the koji API happens in the workers
we can drop koji configuration from composer itself. This means
that composer no longer needs to be provisioned with kerberos
credentials, and does not need to know about which koji servers
the workers support.
This is no longer returned when creating a compose, as it is obtained
asynchronously.
The TaskID is still returned, and is always set to 0. This is not right,
and should either be fixed or dropped. The caller should know the TaskID
(if they have one), so this is redundant and currently unused.
The status of a job may depend on the status of its dependenices,
as we do not repeat for instance the failed state in each dependent
job.
Return also the list of dependencies so these can be queried too.
This removes the restriction of only having a single build per compose
and uses the new job types to schedule the broken-appart build.
A small change in behavior is introduced: the koji build ID is not
known when the call to `compose` returns, so it is always set to
`0`. In the future we should remove this from the API, and instead
rely on the status call to return this information, when it is
known.
The status route will be updated in follow-up commits to reflect the
changes introduced here.
Most of the worker API is now untyped, but keep Enqueu() typed to
ensure the job objects match the names in the queue. This means we
must add a version of Enqueue() for each job type we support.
We must special-case the treatment of architecture, to select the
correct remote worker for any job that requires a specific
architecture. For now this means any jobs that run osbuild.
The three new job types osbuild-koji, koji-init, and koji-finalize
allows the different tasks to be split appart and in particular for
there to be several builds on different architectures as part of a
given compose.
In addition to the arguments passed when scheduling a job, a job now
also takes the results of its dependencies as additional arguments. We
call these dynamic arguments for the lack of a better term.
The immediate use-case for this is to allow koji jobs to be split up
as follows:
- koji-init: Creates a koji build, and returns us a token.
- osbuild-koji: one job per architecture, depending on koji-init
having succeeded. Builds the image, and uploads it to koji,
returning metadata about the image produced.
- koji-finalize: uses the token from koji-init and the metadata
from osbuild-koji to import the build into koji if it succeeded
or mark it as failed if it failed.
Similarly to the recent changes to Dequeue(), let the caller unmarshal the
return JSON. This allows us to pass the result on without being able
to unmarshal it.
In follow-up patches, we will pass results of jobs to dependent jobs,
but the worker API does not know about the different job types, nor how
to unmarshal them.
Once a job has been enqueued, there is no way to query its dependencies.
This makes dequeue more symmetric to enqueue by returning the
dependencies that were passed to enqueue, allowing the caller to
query the dependencies and their results.
Signed-off-by: Tom Gundersen <teg@jklm.no>
While dependencies are purely internal, sorting and pruning them is a
reasonable optimization. However, we wish to expose them in follow-up
commits and then we want them to remain unchanged from the input.
Nothing in the internal logic seems to rely on the fact the dependencies
were sorted.
Signed-off-by: Tom Gundersen <teg@jklm.no>
Add an API route that returns logs for a specific compose.
For now, this contains the result of the job, in JSON. The idea is to
put more and more of this information into structured APIs. This is a
first step to make logs available at all.
Amend koji-compose.py to check that the route exist and contains as many
"image_logs" as images that were requested (currently always 1).
Based on a patch by Chloe Kaubisch <chloe.kaubisch@gmail.com>.
The worker server was heavily tied to OSBuildJob(Result). Untie it so
that it can deal with different job types in the future.
This necessitates a change in the jobqueue: Dequeue() now returns the
job type, as well as job arguments as json.RawMessage. This is so that
the server can wait on multiple job types with different argument
types.
The weldr, composer, and koji APIs continue to use only "osbuild" jobs.
Move the fact that the worker is requesting jobs of type "osbuild" out
of the client library.
For one, require consumers to pass accepted job types to RequestJobs()
and allow querying for the job type with the new Type() function.
Also, make OSBuildArgs() and Update() generic, requiring to pass an
argument that matches the job type.
Workers reported status via an `osbuild.Result`, which only includes
osbuild output. Make it report OSBuildJobResult instead, which was meant
to be used for this purpose and is already used as the result type in
the jobqueue.
While at it, add any errors produced by targets into this struct, as
well as an overall success flag.
Note that this breaks older workers returning the result of an osbuild
job to a new composer. I think this is fine in this case, for two
reasons:
1. We don't support running different versions of the worker and
composer in the weldr API, and remote workers aren't widely used yet.
2. Both osbuild.Result and worker.OSBuildJobResult have a top-level
`Success` boolean. Thus, logs are lost in such cases, but the overall
status of the compose is not.
Add "image_name" and "stream_optimized" fields to the osbuild job as
replacement for the local target options. The former signifies the name
of the uploaded artifact and whether an artifact should be uploaded at
all (only weldr API). The latter will be deprecated at some point, when
osbuild itself can make streamoptimized vmdk images.
This change separates what have always been two distinct concepts:
artifacts that are reported back to the composer node (in practice
always running on the same machine), and upload targets to clouds and
such. Separating them makes it easier to add job types that only allow
one upload target while keeping artifacts.
Keep the local target around, so that jobs that are scheduled can still
be run after an upgrade.
The server hasn't used common.ImageBuildState to mark a job as
successful or failed for a long time. Instead, it's using the job's
return argument for that. (Jobs don't have a high-level concept of
failing).
Drop the check in the server, and always send "FINISHED" from the client
for backwards compatibility.
ComposeState is only used by the weldr API.
Drop the JSON marshaller and unmarshaller, because ComposeState is not
used in an JSON-exported field anymore.
This state is specific to weldr. Previous commits removed it from the
other APIs, because they use different values.
Move the conversion into the weldr API.
