Expose a more detailed job status result - specifically, include upload status
alongside image status. Expand openapi.yml accordingly and add an UploadStatus
field to the OSBuildJobResult struct. At the moment, only represent the
"success" and "failure" states of UploadStatus - to differentiate between
"pending" and "running" would involve significant design decisions and should be
addressed in a separate commit.
There is some confusion surrounding the format of the source TOML that
can be sent to the server. The format it accepts doesn't match the
output from composer-cli which includes the source id in [] eg.
[k8s]
name = "kubernetes packages"
...
This patch changes the parsing to allow the id to be set as 'id = "k8s"'
or passed as a map in [k8s]. If the id is passed in the body it takes
precedence over the map name.
See the comment in tools.go, I cannot fully explain what's happening here.
Somehow, Go 1.14 wants to use the vendored version of oapi-codegen but
without this file, oapi-codegen isn't vendored so the generation fails.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
System repos cannot be overridden by users, return an error if they try
to push a source with the same name/id as a system source.
Resolves: rhbz#1915359
The last imagefactory nightly did not contain dnf-plugin-spacewalk,
fwupd, nss, or udisks2. These packages are now excluded. The
udisks2.service and mdmonitor.service are no longer enabled. Also, the
fwupd-refresh, mdcheck_continue, mdcheck_start, and mdmonitor-oneshot
timers are no longer listed as disabled services.
The packages in the last imagefactory nightly differ from ours. The
following packages are now added:
oddjob
oddjob-mkhomedir
psmisc
authselect-compat
rng-tools
dbxtool
Also, the rngd and nfs-convert services are enabled.
Adding the tag called `Name` to the AMI ensures that the name appears in
the *Name* column inside AWS' web console.
Fixes#1171.
Signed-off-by: Major Hayden <major@redhat.com>
An image only had a systemd stage added if its blueprint contained
services or if its image type contained enabled services. The systemd
stage is now also added if the image type contains disabled services or
a default target.
The RHEL 8.4 qcow2 image type now specifies the multi-user default target.
In order to test this the image-info tool now includes the default
target in its output. Image test manifests are updated to include this
change.
Send requests to the compose/{id}, compose/{id}/logs, and
compose/{id}/manifests using job IDs for non-finalize type jobs to test
the type verification.
The type verification introduced in the previous commit is now also used
when retrieving the job status (GET /compose/{id}) and the logs (GET
/compose/{id}/logs).
In these cases, job retrieval needs to be performed twice:
1. First the job parameters are retrieved (Job()) to check the type.
2. Then the job result is retrieved (JobStatus()) for the status or
logs.
This makes it unlikely (essentially impossible) that the retrieval will
fail with "not found" on the second retrieval (JobStatus()), but it's
still a good sanity check for the system.
Verifies the Koji job types when retrieving Init and Build jobs as well.
When a job's arguments are retrieved (for the /manifests API endpoint),
the incoming ID should correspond to a Finalize Job. The new
worker.Job() method helps us verify the type and produce an error if the
wrong type is found.
Similarly, the dependencies of a Finalize Job should be in order (Init
Job first followed by Build Jobs). The types are validated while
iterating the dependency list.
Added convenience functions that check the retrieved job type and return
the initialised struct or an error if the ID is not found or does not
match the type.
Currently the getInitJob() function isn't used but it will be useful
later.
JobArgs() function replaced with more general Job() function that
returns all the parameters used to originally define a job during
Enqueue(). This new function enables access to the type of a job in the
queue, which wasn't available until now (except when Dequeueing).
1. Test retrieving the job arguments from the worker using an
OSBuildJob.
Very basic test with an empty manifest.
2. Test retrieving job manifests through the Koji API. Uses the existing
TestCompose test. The returned manifests are empty for all cases.
Add and implement an API endpoint that returns the manifests for a given
osbuild-koji job. The route returns an array of manifests, one for each
image in the request.
Retrieves the arguments of each Build Job to extract the Manifest.
Wraps jobqueue.JobArgs() method and unmarshals data into the provided
concrete struct value. The struct should match the requested job type of
the given ID.
JobArgs() returns the arguments submitted with a job in raw form.
Since the structure of the args are opaque to job queue, it's the
responsibility of the caller to deserialize the arguments.
Args retrieval is added to the existing TestArgs() function.
RHEl 8.4 guest images need to have the default timezone of EST/EDT
unless the user specifies one in their blueprint. New York is a major
location for this timezone.
testjobqueue did not implement the JobQueue interface correctly (noted
in its package comment), making it impossible to write tests for
JobQueue itself.
Replace its use everywhere with fsjobqueue operating on a temporary
directory.
When rpmmd's Depsolve function is called we need to pass in the image
type's excluded packages. These excluded packages are retrieved when we
get the packages we include from each image type.
Fedora 33 and rawhide got an updated version of the azblob library. Sadly, it
introduced a non-compatible API change. This commit does the same thing as
a67baf5a did for kolo/xmlrpc:
We now have two wrappers around the affected part of the API. Fedora 32 uses
the wrapper around the old API, whereas Fedora 33 and 34 (and RHEL with its
vendored deps) use the wrapper around the new API. The switch is implemented
using go build flags and spec file magic.
See a67baf5a for more thoughts.
Also, there's v0.11.1-0.20201209121048-6df5d9af221d in go.mod, why?
The maintainers of azblob probably tagged a wrong commit with v0.12.0 which
breaks go. The long v0.11.1-.* version is basically the proper v0.12.0 commit.
See https://github.com/Azure/azure-storage-blob-go/issues/236 for more
information.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
I thought rand in Go is auto-seeded but I was wrong, see [1].
This commit adds seed initialization.
[1]: https://golang.org/pkg/math/rand/#Seed
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Imagine this situation: You have a RHEL system booted from an image produced
by osbuild-composer. On this system, you want to use osbuild-composer to
create another image of RHEL.
However, there's currently something funny with partitions:
All RHEL images built by osbuild-composer contain a root xfs partition. The
interesting bit is that they all share the same xfs partition UUID. This might
sound like a good thing for reproducibility but it has a quirk.
The issue appears when osbuild runs the qemu assembler: it needs to mount all
partitions of the future image to copy the OS tree into it.
Imagine that osbuild-composer is running on a system booted from an imaged
produced by osbuild-composer. This means that its root xfs partition has this
uuid:
efe8afea-c0a8-45dc-8e6e-499279f6fa5d
When osbuild-composer builds an image on this system, it runs osbuild that
runs the qemu assembler at some point. As I said previously, it will mount
all partitions of the future image. That means that it will also try to
mount the root xfs partition with this uuid:
efe8afea-c0a8-45dc-8e6e-499279f6fa5d
Do you remember this one? Yeah, it's the same one as before. However, the xfs
kernel driver doesn't like that. It contains a global table[1] of all xfs
partitions that forbids to mount 2 xfs partitions with the same uuid.
I mean... uuids are meant to be unique, right?
This commit changes the way we build RHEL 8.4 images: Each one now has a
unique uuid. It's now literally a unique universally unique identifier. haha
[1]: a349e4c659/fs/xfs/xfs_mount.c (L51)
Now that we have an abstract partition table definition, we can use it to
generate org.osbuild.fstab stage options.
This is extremely nice because it removes magic contains.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Using osbuild.QEMUAssemblerOptions to encode a partition table was weird.
This commit introduces a disk package that contains data types for defining
partition tables. Also, there's a handy function to convert the abstact
partition table to osbuild.QEMUAssemblerOptions.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Previously, the partition table definition was defined inside an assembler.
This has an issue though: The partitions and filesystems are needed at several
other places, e.g. grub2 stage and fstab stage. As the partition table was
basically hardcoded, this didn't matter - we could just use constants
in these stages. Not ideal but it worked.
This commit changes the behaviour: A partition table is firstly created and
then it's passed to the assembler function where complete assembler options
are created out of it.
To make this change as small as possible, osbuild.QEMUAssemblerOptions type
is used to encode the partition table for now.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
require.JSONEqf cannot handle diffs of such a big entity as a manifest is.
It just prints an empty string.
This commit unmarshalls the manifests instead and then uses the cmp library
to make a very nice and readable diff.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The kernel options are updated to remove the read only option "ro" from
the qcow2 and amazon image types. Also, the qcow2's kernel options are
updated to only set console=ttyS0 once. It was declared twice which is
redundant so now it is set for both tty0 and ttyS0.
rng-tools is no longer included as a package in the RHEL 8.4 image. This
package is both removed from being an included package and also
specifically declared as excluded. The test manifests are updated.
The RHEL 8.4 specific runner was introduced in osbuild 22, released
on the 8th of October 2020. It should by now be in relevant Fedora
releases and RHEL 8.4.
Adapt the corresponding test cases.
Previously, the compose status returned failure as soon as possible.
koji-osbuild considers the job as done when its status == failure and proceeds
with uploading the logs to koji and marking the job as failed. However, not
all osbuild-composer jobs might be done at this point so the logs might be
incomplete making the debugging hard.
This commit changes the behaviour: Now, the compose status is pending until
ALL jobs belonging to it are finished.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Previously, we had no clue what errors were catched by the default echo's
error handler. Thus, in the case of an error, we were basically blind. Let's
log all errors so we can investigate them later.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
%#v was my bad understanding of Go's error formatting. Let's use the standard
%v that gives saner and human-readable error messages.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
