Setting of the `crashkernel` option to the appropriate value is now done
by the `kexec-tools` package when installed and when any new kernel is
installed.
Regenerate relevant image test cases.
Fix#1819
Fix rhbz#2006692
Signed-off-by: Tomas Hozza <thozza@redhat.com>
This commit introduces the collection of error
metrics since it is now possible to differentiate
between internal errors and user input errors.
Additionally, the error status is reported for
job duration metrics.
For simplicity, the collection of the job metrics
was carried out in the job queue. This was only
being done in the dbqueue and not in the fsqueue.
This pr refactors the metric collection and moves
the job metrics to the worker server, by adding a
wrapper function to enqueueing jobs so that the
metrics only have to be recorded in one place when
queueing a job.
Refactor the current metric collection to make use
of re-usable functions, since some of the same queries
are repeated. This will also make it easier to move
the collection of metrics from the job queue.
The internal GCP package used `pkg.go.dev/google.golang.org/api` [1] to
interact with Compute Engine API. Modify the package to use the new and
idiomatic `pkg.go.dev/cloud.google.com/go` [2] library for interacting
with the Compute Engine API. The new library have been already used to
interact with the Cloudbuild and Storage APIs. The new library was not
used for Compute Engine since the beginning, because at that time, it
didn't support Compute Engine.
Update go.mod and vendored packages.
[1] https://github.com/googleapis/google-api-go-client
[2] https://github.com/googleapis/google-cloud-go
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Disable loging in via password authentication since this is an
official Amazon marketplace requirement
Linux-based AMIs must not allow SSH password authentication.
Disable password authentication via your sshd_config file by
setting PasswordAuthentication to NO.
Section "Security policies" from
https://docs.aws.amazon.com/marketplace/latest/userguide/product-and-ami-policies.html
Disable loging in via password authentication since this is an
official Amazon marketplace requirement
Linux-based AMIs must not allow SSH password authentication.
Disable password authentication via your sshd_config file by
setting PasswordAuthentication to NO.
Section "Security policies" from
https://docs.aws.amazon.com/marketplace/latest/userguide/product-and-ami-policies.html
Disable loging in via password authentication since this is an
official Amazon marketplace requirement
Linux-based AMIs must not allow SSH password authentication.
Disable password authentication via your sshd_config file by
setting PasswordAuthentication to NO.
Section "Security policies" from
https://docs.aws.amazon.com/marketplace/latest/userguide/product-and-ami-policies.html
Clean up some implementation aspects of the Fedora distro definition:
- Do not have default Fedora distro version and use `fedora` as the
package name in all places that use it, instead of `fedora33`.
- Fix bugs when wrong (Fedora 33) values were returned by `OSTreeRef()`
and `Releasever()` for newer Fedora releases.
- Test Fedora 35 in package unit tests.
- Add unit test for `OSTreeRef()` method.
- Use architecture name constants from `distro` package, instead of
string literals.
Fix#1802
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The QEMU assembler in Fedora distro definition for UEFI systems used
longer than allowed label for the VFAT filesystem of the EFI System
Partition. The maximum allowed label length is 11 characters.
This worked before with dosfstools, but in 2018, they added a label
validation [1]. This change got into the v4.2 release of dosfstools,
released in Jan 2021. And subsequently since F34, this new version of
dosfstools is present in Fedora repositories.
[1] ca54953476
Signed-off-by: Tomas Hozza <thozza@redhat.com>
This is for now only supported for koji builds.
A lot of code moved around for this, but functionally not much changed. PostCompose() now only parses the input, and queueing of all the jobs have been factored out in separate functions. PostCompose() is mostly agnostic to koji/non-koji requests.
Replace Job() and JobStatus() with typesafe versions, and introduce JobType()
for the rare instances where we don't know the type up front.
Additionally, catch a few more error cases:
- if OSBuildResult is nil, then we failed to invoke osbuild
- make sure the same JobResult handling is done for osbuild-koji, as for osbuild
This only extends the API, the backend can still only deal with composes of a single build.
I aimed to keep the API practically backwards compatible, i.e., no current consumer of it should notice the change. I hope I didn't mess that up.
fixup: image statuses
In addition to individual image status, have an
overall status that captures success or failure
of the compose as a whole.
This is not as fine grained, and only distinguishes
between "pending", "failure" and "success".
This captures other jobs than the image builds, which
is relevant for the koji composes, which consists also
of koji-init and koji-finalize, in addition to the build
jobs.
For now upload requests are required if and only if we are not
using koji. When using the koji integration the produced artifacts
are uploaded to koji only. In the future we may want to support
also uploading to the cloud providers.
Extend the compose endpoints to have minimal koji support.
This is intended to replace the current koji API so that it
can be consumed through api.openshift.com.
We may need to use several SSO providers, so extend our
configuration to allow that.
Based on PoC from Sanne:
```
package main
import (
"net/http"
"log"
"github.com/openshift-online/ocm-sdk-go/authentication"
"github.com/openshift-online/ocm-sdk-go/logging"
)
type H struct{}
func (h *H) ServeHTTP(w http.ResponseWriter, r *http.Request) {
log.Println("HURRAY")
}
func main() {
logBuilder := logging.NewGoLoggerBuilder()
logger, err := logBuilder.Build()
if err != nil {
panic(err)
}
aH, err := authentication.NewHandler().
KeysURL("https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/certs").
KeysURL("https://identity.api.openshift.com/auth/realms/rhoas/protocol/openid-connect/certs").
Logger(logger).Next(&H{}).Build()
if err != nil {
panic(err)
}
log.Fatal(http.ListenAndServe(":8080", aH))
}
```
It should be `1048576` (exactly 512 MiB), like it is for all other
distributions. It somehow got mingled in when the distribution was
forked off from 8.5/9.0 beta (1048676 to 1048576 strongly suggests
a sed command was involved, so we blame that).
When we compute the overall status of a koji compose, the individual
build jobs are checked. Currently, a job is considered a failure, if
a build job has output (`OSBuildOutput`) and the output's `Success`
field is `false`. But `OSBuildOutput` will be `nil` when osbuild
crashed or refused the manifest input. Therefore the job status is
a failure if `OSBuildOutput` is `nil`, since if osbuild was run,
and the run was successful we must have a non-`OSBuildOutput` field.
Add a special cases for the root user to the work-around for ssh keys in
OSTree commits.
See 93e54cd872 for the original,
equivalent change in RHEL 8.6.
The unification of the partition table also introduced uuids and
types in uuid form for partition tables in dos layout, sill used
on PPC64LE and s390x. The org.osbuild.sfdisk stage did work with
that but produced a `/boot` partition with the wrong type, which
grub2 refused to read from and thus prevented boot. Fix this by
removing uuids from the dos partition tables.
Reported-by: Jakub Rusz <jrusz@redhat.com>
Since the workers will use structured error messages
going forward, it is necessary to maintain backwards
compatability for there errors in composer. Tests have
been added to the various apis to ensure that each api
checks for both kinds of errors, old and new.
Implement the structured errors as defined by the worker client.
Every error for each of the job types now returns a structured
error with a reason and a specific error code. This will make
it possible to differentiate between 4xx errors and 5xx errors.
This commit refactors the way errors are implemented in the workers,
but maintains backwards compatability in composer by checking for
both kinds of errors.
Define worker errors to give more structured
error messages. The error api is:
id: VALIDATION_ERROR_NUMBER, reason: STRING, details: { issues: [{...}, {...}] }
The api was agreed upon with osbuild so that,
in future, osbuild errors will share the same
structure
Port all of the pipeline refactoring done to RHEL-90 to RHEL-86. Both
distros now use the same approach.
Regenerate all RHEL-8.6 and CentOS 8 image test cases.
[1] https://git.centos.org/centos/kickstarts/tree/master
Signed-off-by: Tomas Hozza <thozza@redhat.com>
RHSM configuration is now applied conditionally only on RHEL. The same
applies to the customization to subscribe the system on first boot.
The reason is that the CentOS `@core` package group does not contain
`subscription-manager`. Thus it is not installed on CentOS Stream by
default and also CentOS 8 image definitions don't apply any changes
to the RHSM configuration [1].
In addition, make sure to not install any subscription-manager
packages on CentOS Stream images.
Regenerate all CentOS 8 image test cases.
[1] https://git.centos.org/centos/kickstarts/tree/master
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Enhance the `osPipeline` to add necessary stages to the returned
pipeline, in case the image is RPM OSTree based. As a result, delete the
`ostreeTreePipeline` and replace its uses by `osPipeline`.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Make the `osPipeline` self-contained in the sense, that no stages are
added to the returned pipeline outside of the function and the returned
pipeline is usable as returned.
Modify the `osPipeline` to add Kernel Cmdline, FSTab and bootloader
config stages to the pipeline if a valid partition table was passed to
the function. As the last one, the SELinux stage is appended to the
returned pipeline.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Move the EC2 SAP image specific configuration from `ec2SapPipelines`
to the EC2 SAP default image configurations data structure. As a
result, remove the `ec2SapPipelines` and `rhelEc2SapPipelines` entirely
and use `rhelEc2Pipelines` for all RHEL EC2 images.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Move the x86_64 specific configuration from `ec2X86_64BaseTreePipeline`
to x86_64-specific image configurations for EC2 / AMI images. As a
result, remove the `ec2X86_64BaseTreePipeline` entirely and replace it
with `osPipeline`.
Regenerate image test cases. While there are changed in the manifests,
the actual image configuration didn't change at all and thus the
`image-info` report was not changed.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Move all hard-coded image configuration from the `ec2BaseTreePipeline`
function to the `ImageConfig` structure and update the respective EC2
images default configuration structure.
Update `osPipeline` and `ostreeTreePipeline` to handle all of the new
configuration values from `ImageConfig`.
Completely remove the `ec2BaseTreePipeline` and replace it with
`osPipeline`.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Move the RHSM configuration settings to `ImageConfig` structure and use
when handling subscriptions in `osPipeline`, `ec2BaseTreePipeline` and
`ostreeTreePipeline` functions.
Regenerate image test cases. While there are changed in the manifests,
the actual image configuration didn't change at all and thus the
`image-info` report was not changed.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Introduce a new data structure `ImageConfig` holding the default OS
configuration applied when building an image. The structure can be used
to hold the default image configuration on the distribution level with
possible overrides defined on the image-type level.
As a starting point, move hard-coded default values and configuration
common for `osPipeline`, `ec2BaseTreePipeline` and `ostreeTreePipeline`
to the distribution and image-type default image configuration. This is
preparing the ground for merging all of these three pipeline functions
into `osPipeline`, which will produce the appropriate OS pipeline based
on the image-type configuration and the fact if it is rpmOstree or not.
Regenerate affected EC2 and AMI manifests. There is however no change in
the resulting image configuration and image-info report.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Bring the Tar stage implementation on par with the current osbuild
schema. Specifically add the 'format' and 'root-node' options to the
stage options structure.
Add stage options validation along with appropriate unit tests.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
