We actually need 2 * 16 connections at minimum (one worker waits for two
jobs). Let's bump the maximum connection count even moar.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
By default, pgxpool.Pool has 4 connections (or number of cpus if higher).
Currently, we have 3 replicas, that means max 3*4=12 DB connections.
The dequeue operation is actually blocking - when a worker is waiting for
a job, one connection is blocked. My theory is that with 16 workers, we just
don't have enough connections that causes all sorts of weird slowdowns.
This commit bumps the number of connection from one replica to 10, therefore
we should be at 30 connections in total.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
The stage now allows for customizations specific to YUM or DNF. So far
it is just an alias to the same definition, meaning that composer can
use exactly the same structures for both.
Ref: https://github.com/osbuild/osbuild/pull/876
Before dereferencing the method receiver in Write(), check if the object
is nil and return early.
Fixes#2002
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
The CDN repositories require client side SSL certificates, it would have
to be specified in httpd configuration which is technically possible but
in practise it is more troubles than benefits.
It is similar case for the RH internal repositories, there is a new
issue every time we enable the test with them.
Stop wasting dev&qe time and disable the test for anything but rpmrepo
snapshots. It is better to have it running at least for few distros than
to have it disabled globally.
Also display logs for the HTTP proxy to see if there were any errors.
Rework the stage options data validation to be done in constructor
methods, instead of when being marshalled to JSON.
Add validation of values passed to constructor methods for modprobe
command structures.
Add validation of the configuration filename based on stage schema.
Related to issue #1785.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Add support for the 'install' modprobe command in the modprobe osbuild
stage implementation.
Extend unit tests to verify marshalling the stage options into JSON.
Related to https://github.com/osbuild/osbuild/pull/867.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Use newly added RHUI-4 repo snapshots for all RHEL-9 EC2* image test
cases. This includes RHEL-9.0 and RHEL-9.0-Beta images. The removal
of installed repos on EC2-HA and EC2-SAP images is expected, as RHUI
client RPMs for these variants are empty for 9.0 Beta. This should
change for GA once there are updated RHUI clients RPMs available.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Running the job in this case is basically undefined, so let's just skip it
in order to not break anything.
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
Mistakenly removed in 4577ac0717. Composer
itself does the authentication, not the gateway, therefore we do need
the auth exclude.
Added a comment to explain why it's attached to the api socket and not a
separate listener.
Two new tests, one for OSBuild and one for Koji jobs. Both follow the
same flow:
- Enqueue a job that doesn't specify PipelineNames (oldJob)
- Enqueue a job that does specify PipelineNames (newJob)
- Read the job data for the oldJob and check that the default
PipelineNames were added
- Read the job data for the newJob and check that it's unchanged
- Finish oldJob and add results without specifying PipelineNames
- Finish newJob and add results with PipelineNames
- Read the oldJob result and check that the default PipelineNames were
added
- Read the newJob result and check that it's unchanged
This is meant to test several scenarios that can occur when upgrading
the service:
1. The existing jobqueue has old jobs in it that were queued before the
PipelineNames were part of the data structure. The worker should be
able to read these and add the fallback data.
2. New jobs are added while old jobs still exist in the queue and the
worker can read both types.
3. The existing jobqueue has old finished jobs in it that were finished
and had results written before the PipelineNames were part of the
result data structure. The worker should be able to read these and
add the fallback data.
4. New jobs are finished and results are written while old jobs still
exist in the queue and the worker can read both result types.
When worker reading data into the job and result types, check if the
PipelineNames are populated and, if not, add the fallback values from
distro.
This makes it simpler to work with job queues that contain old data
before the introduction of the PipelineNames. In any situation where the
job or result data are read, the reader can assume that the
PipelineNames are non-nil and that if they belong to an old job, they
have the fallback names.
This assumption goes hand-in-hand with the change in v2 format for
osbuild results, since old jobs that don't have PipelineNames set *must*
contain results in the old format for the names to be valid.
Pipeline names are added to each job before adding to the queue. When a
job is finished, the names are copied to the Result object as well. This
is done for both OSBuild and Koji jobs.
The pipeline names in the result are primarily used to separate package
lists into build and payload/image packages in two cases:
1. Koji builds: for reporting the build root and image package lists to
Koji (in Koji finalize).
2. Cloud API (v1 and v2): for reporting the payload packages in the
metadata request.
The pipeline names are also used to print the system log output in the
order in which pipelines are executed. This still isn't used when
printing the OSBuild Result (osbuild2.Result.Write()) and we still rely
on sorting by pipeline name
(see https://github.com/osbuild/osbuild-composer/pull/1330).
The names of the pipelines that make up a Manifest for a job are
attached to the job data that is stored in the queue. The pipelines are
separated into Build and Payload.
This information is useful for identifying the build pipeline results
and metadata and for the order of the pipelines as they appeared in the
manifest.
Moved and adapted tests from osbuild1 to osbuild2.
Moved test data from osbuild1 to osbuild2.
Added conversion tests for v1 to v2.
Added full v2 result raw data from successful build.
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
Each image type now implements BuildPipelines(), which returns a list of
pipeline names that set up the build environment, and
PayloadPipelines(), which returns a list of pipeline names that create
the OS image (all non-build pipeline names).
Older distros that produce v1 manifests should call the distro Fallback
functions to return the common defaults.
A Fallback function for the Exports() method is also added and called by
older distros.
All image types that produce v2 manifests (distros after RHEL 8.4)
should include the information in the image type definition and should
not rely on fallbacks for default values.
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
- koji-finalize:
Use v2 result type to collect RPM metadata.
The separation between the "build" pipeline and the rest is based on the
pipeline name, which isn't completely reliable since pipeline names can
be arbitrary.
Koji will fail a build if it specifies duplicate packages, so the RPM
lists are deduplicated. The "build" pipeline package list is also
deduplicated in case there are multiple build stages in the same
pipeline.
- osbuild:
Use v2 result type for printing build result to log.
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
Convert osbuild1.Result{} to osbuild2.Result{}.
For the Metadata objects, it assumes they are directly convertible: the
stage metadata structs have the same members.
The old conversion code from v2 to v1 is removed and the equivalent
conversion logic is moved to osbuild2:
- version detection based on stub
- custom unmarshaller that calls conversion function if v1 result is
detected
Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
