Mistakenly removed in 4577ac0717. Composer
itself does the authentication, not the gateway, therefore we do need
the auth exclude.
Added a comment to explain why it's attached to the api socket and not a
separate listener.
Two new tests, one for OSBuild and one for Koji jobs. Both follow the
same flow:
- Enqueue a job that doesn't specify PipelineNames (oldJob)
- Enqueue a job that does specify PipelineNames (newJob)
- Read the job data for the oldJob and check that the default
PipelineNames were added
- Read the job data for the newJob and check that it's unchanged
- Finish oldJob and add results without specifying PipelineNames
- Finish newJob and add results with PipelineNames
- Read the oldJob result and check that the default PipelineNames were
added
- Read the newJob result and check that it's unchanged
This is meant to test several scenarios that can occur when upgrading
the service:
1. The existing jobqueue has old jobs in it that were queued before the
PipelineNames were part of the data structure. The worker should be
able to read these and add the fallback data.
2. New jobs are added while old jobs still exist in the queue and the
worker can read both types.
3. The existing jobqueue has old finished jobs in it that were finished
and had results written before the PipelineNames were part of the
result data structure. The worker should be able to read these and
add the fallback data.
4. New jobs are finished and results are written while old jobs still
exist in the queue and the worker can read both result types.
When worker reading data into the job and result types, check if the
PipelineNames are populated and, if not, add the fallback values from
distro.
This makes it simpler to work with job queues that contain old data
before the introduction of the PipelineNames. In any situation where the
job or result data are read, the reader can assume that the
PipelineNames are non-nil and that if they belong to an old job, they
have the fallback names.
This assumption goes hand-in-hand with the change in v2 format for
osbuild results, since old jobs that don't have PipelineNames set *must*
contain results in the old format for the names to be valid.
Pipeline names are added to each job before adding to the queue. When a
job is finished, the names are copied to the Result object as well. This
is done for both OSBuild and Koji jobs.
The pipeline names in the result are primarily used to separate package
lists into build and payload/image packages in two cases:
1. Koji builds: for reporting the build root and image package lists to
Koji (in Koji finalize).
2. Cloud API (v1 and v2): for reporting the payload packages in the
metadata request.
The pipeline names are also used to print the system log output in the
order in which pipelines are executed. This still isn't used when
printing the OSBuild Result (osbuild2.Result.Write()) and we still rely
on sorting by pipeline name
(see https://github.com/osbuild/osbuild-composer/pull/1330).
The names of the pipelines that make up a Manifest for a job are
attached to the job data that is stored in the queue. The pipelines are
separated into Build and Payload.
This information is useful for identifying the build pipeline results
and metadata and for the order of the pipelines as they appeared in the
manifest.
Moved and adapted tests from osbuild1 to osbuild2.
Moved test data from osbuild1 to osbuild2.
Added conversion tests for v1 to v2.
Added full v2 result raw data from successful build.
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
Each image type now implements BuildPipelines(), which returns a list of
pipeline names that set up the build environment, and
PayloadPipelines(), which returns a list of pipeline names that create
the OS image (all non-build pipeline names).
Older distros that produce v1 manifests should call the distro Fallback
functions to return the common defaults.
A Fallback function for the Exports() method is also added and called by
older distros.
All image types that produce v2 manifests (distros after RHEL 8.4)
should include the information in the image type definition and should
not rely on fallbacks for default values.
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
- koji-finalize:
Use v2 result type to collect RPM metadata.
The separation between the "build" pipeline and the rest is based on the
pipeline name, which isn't completely reliable since pipeline names can
be arbitrary.
Koji will fail a build if it specifies duplicate packages, so the RPM
lists are deduplicated. The "build" pipeline package list is also
deduplicated in case there are multiple build stages in the same
pipeline.
- osbuild:
Use v2 result type for printing build result to log.
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
Convert osbuild1.Result{} to osbuild2.Result{}.
For the Metadata objects, it assumes they are directly convertible: the
stage metadata structs have the same members.
The old conversion code from v2 to v1 is removed and the equivalent
conversion logic is moved to osbuild2:
- version detection based on stub
- custom unmarshaller that calls conversion function if v1 result is
detected
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
Implementing writer for osbuild2 Result type.
Since Go maps don't have stable ordering, sorting by the pipeline name
provides stable output.
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
Reading stage metadata using osbuild's v2 result format.
For RPM stages we only want the core (OS) RPMs (not the build root
RPMs). Skip the build pipeline by name, but this should be handled
better since names are arbitrary.
Using type switch to convert metadata types instead of relying on the
type string of the stage result.
The rpmmd helper function isn't used anymore since that requires two
conversion passes (osbuild.StageMetadata -> rpmmd.RPM ->
cloudapi.PackageMetadata).
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
Function renamed to better fit the argument element type
(StageMetadata).
Argument is a map to fit the pipeline metadata in the result object.
Signature function is made public to be reused in the cloud API
conversion.
Metadata test raw value updated to v2 result format.
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
We will soon need to dequeue a job using its ID. This commit adds ability
to do that to the Jobqueue interface. As always, the fsjobqueue implementation
is slightly naive but it should fine for the usecases that it's designed for.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Previously, we deleted empty channels when a job was dequeued. This is
completely wrong because there still might be some clients waiting for
a job. This commit removes the cleanup and adds a regression test.
Note that this has the potential to leak memory if we ever use a lot of
job types. Currently, we have just handful of them, so this is fine.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Previously, we only needed the log directory to be writeable.
In newer versions of nginx, it also needs to create directories in
/var/lib/nginx, so we make that directory writeable and traversable as
well.
