Commit graph

1890 commits

Author SHA1 Message Date
Christian Kellner
6300a80c84 util/udev: fix path for udev device inhibitor
The udev inhibitor rules are checking for `device-$major:$minor`
but we created them with `f"device-{major}-{minor}"`. So they
did indeed not actually work. Fix that.
2022-02-27 18:47:17 +01:00
Tomas Hozza
8102f20d23 Add RHEL-9.1 runner
Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-02-25 16:42:22 +01:00
ochosi
97905f6e0c Post release version bump
[skip ci]
2022-02-23 19:13:10 +00:00
Christian Kellner
8f75d1e7a1 stages: add new org.osbuild.fdo stage
This stage is needed to write down the FDO DIUN pub key root certs
needed to talk to the manufacturer server to grab the device credentials
for provisioning and later onboarding

Co-Authored-By: Antonio Murdaca <runcom@linux.com>
2022-02-23 15:32:02 +00:00
Jakub Rusz
89666714f4 rpmbuild: run on centos-9
Also update terraform SHA and EPEL installation for mockbuild.
2022-02-22 18:05:51 +00:00
Christian Kellner
d8bfe13fdd stages/oci-archive: fix creation time format
According to OCI image specification the `created` property needs
to be in the format describe in RFC 3339, section 5.6 "Internet Date /
Time Format". The suffix is the "time-offset" which is either "Z" or
"time-numoffset".
Before commit 25b5679[2] we the datetime object, created via `utcnow`
did not have timezone information so "Z" was manually appended. This
was changed with commit 25b5679 and now the timezone information was
indeed included, which also meant that `isoformat` now included the
offset, i.e. `+00:00`. But the `Z` was still appended and the result
was `+00:00Z`, something that is not standard compliant. Fix this
by removing the extra `Z`.

[1] https://datatracker.ietf.org/doc/html/rfc3339#section-5.6
[2] 25b567990b
2022-02-22 10:04:08 +00:00
Thomas Lavocat
1ceb096594 host: add support for emitting signals
Add support for emitting signals to host.Service which can be used to
transmit data back to the client during an ongoing method call. This
provides the possibility for the services to send information to their
client counterpart while running. The signal can take file descriptors
as extra parameters to send data on separate files.
2022-02-22 10:38:43 +01:00
Antonio Murdaca
f768781589 stages: add luks remove-key stage
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2022-02-16 15:43:19 +01:00
Antonio Murdaca
25ecd12b3d stages: add clevis-luks-bind stage
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2022-02-16 15:43:19 +01:00
chloenayon
5a7d3eee86 Post release version bump
[skip ci]
2022-02-16 14:16:11 +00:00
Alexander Larsson
e9d02a27bb skopeo stage: remove overlay/backingFsBlockDev file after install
Each time the overlay backend runs on an xfs fs it creates the
file "overlay/backingFsBlockDev" in the containers storage directory.

It is not needed in the image as skopeo recreates it each boot, and
since it is a block device its existance means we can't store the
resulting tree in ostree. Lets just get rid of it.
2022-02-15 19:13:00 +01:00
Christian Kellner
62dab7be3d ci: add test_executable to test matrix
I don't see a reason as not run the test. Do so.
2022-02-15 13:25:23 +00:00
Christian Kellner
f7af3042ca test: fix executable test
During the rework done in commit "use and require explicit exports"
with commit id 7ae4a7e78, the test got overlooked. Add an empty
list of checkpoints to the `obs.compile` invocation as to actually
trigger the osbuild invocation.

Reported-By: Thomas Lavocat <tlavocat@redhat.com>
2022-02-15 13:25:23 +00:00
Roy Golan
ed82f33c8f Fix type orci-archive/oci-archive
Signed-off-by: Roy Golan <rgolan@redhat.com>
2022-02-14 14:15:19 +01:00
Alexander Larsson
7c9944ee18 Update the skope test stage
Drop format and file options and convert the hello.img from
docker to oci format.
2022-02-10 14:43:17 +01:00
Alexander Larsson
1152edcf31 inputs/org.osbuild.containers: Drop format and file options
We hardcode oci-archive for now and use the first file in the pipeline
tree as the archive. Long term we may want to get this info from the
metadata of the oci-archive stage instead.
2022-02-10 14:43:17 +01:00
Alexander Larsson
13c0c1e59e test: Add skopeo tests
There is a source test that installs a pre-build, embeded image file
and ensure all the right files are installed. This uses the vfs driver
because then it works everywhere, including the CI (which doesn't do
overlayfs).

Then the is a source test that downloads a minimal image from
a faked registry on localhost.

For the registy API to work the "/v2" entry-point in the webserver has
to be at the root, so there is a symlink in test/data:

  v2 -> sources/org.osbuild.skopeo/data/v2

But otherwise the data is localized to sources/org.osbuild.skopeo.
2022-02-10 14:43:17 +01:00
Alexander Larsson
693e44e412 sources test: Support custom mimetypes in http server
By creating a `foo.mimetype` file you can override the mimetype returned
for the file `foo`.
2022-02-10 14:43:17 +01:00
Alexander Larsson
dbaed75b46 osbuild-mpp: Support mpp-resolve-image for container images 2022-02-10 14:43:17 +01:00
Alexander Larsson
66cc2900c9 obuild-mpp: Add process_stage() helper
Both file embedding and depsolves start by iterating over all stages,
and we want to add another similar one for container installs, so
break out the iteration over the containers so that it is done in
one place only.
2022-02-10 14:43:17 +01:00
Alexander Larsson
46a228df38 Add support for installing containers in images
This adds a stage called org.osbuild.skopeo that installs docker and
oci archive files into the container storage of the tree being
constructed.

The source can either be a file from another pipeline, for example one
created with the existing org.osbuild.oci-archive stage, or it can
be using the new org.osbuild.skopeo source and org.osbuild.containers
input, which will download an image from a registry and install that.

There is an optional option in the install stage that lets you
configure a custom storage location, which allows the use of the
additionalimagestores option in the container storage.conf
to use a read-only image stores (instead of /var/lib/container).

Note: skopeo fails to start if /etc/containers/policy.json is
not available, so we bind mount it from the build tree to the
buildroot if available.
2022-02-10 14:43:17 +01:00
Alexander Larsson
b6629de7b2 Add skopeo to f34-build-v2 manifest
This will be needed to test the container support.
Also update the test data
2022-02-10 14:43:17 +01:00
Alexander Larsson
0c4b83e6e9 Update to the latest osbuild-ci container (with skopeo) 2022-02-10 14:43:17 +01:00
Jakub Rusz
d878824696 ci: remove fedora-33 and add fedora-35 2022-02-09 10:52:21 +01:00
Jakub Rusz
df189a7301 ci: update to 8.5 ga runners 2022-02-09 10:52:21 +01:00
Alexander Larsson
d57eeb38b9 Pass source-epoch to stages only if set
The client side does meta.get("source-epoch", default), but for
this to work we need to have the key unset if not specified,
but currently we set it to None.

Also, make sure the check for "not None" is explicit, because
we do consider a value of `0` to be a valid source-epoch.
2022-02-09 09:58:49 +01:00
Alexander Larsson
1897eaf8bc org.osbuild.oci-archive: Try to create more reproducible images
This tries to make the various tar-balls produced by the stage more
likely to be identical in separate runs. We do this by sorting
the names and removing some unnecessary metadata for the files.

The most important thing to get right is the layer tarball, because
that is what defines the container id. We sort the names to avoid
random differences, and drop ctimes and atimes because these are
generally just set to the something near the current build time which
is not useful to encode in the container image. This is as opposed
to the mtime which generally comes from e.g. the rpms that where
installed in the pipeline.

For the actual archive tarball we can standardize metadata even more,
because none of the metadata are used when consuming the archive.
2022-02-09 09:58:49 +01:00
Alexander Larsson
25b567990b oci-archive stage: Use source-epoch as creation data
If the source-epoch is specified, this is used as the creation date
instead of the current time.
2022-02-09 09:58:49 +01:00
Alexander Larsson
0ab4a6d401 Pipeline: Pass down the source-epoch to stages via the meta key
This can be used by stages that wish to support more reproducible
builds.
2022-02-09 09:58:49 +01:00
Alexander Larsson
b31c91d671 v2: Add source-epoch key in pipeline declaration and pass to buildroot
If this is set it is passed down to all stages and set as
SOURCE_DATE_EPOCH in the buildroot environment. This implements
the spec at:
  https://reproducible-builds.org/docs/source-date-epoch/
2022-02-09 09:58:49 +01:00
Alexander Larsson
e516bf7898 BuildRoot: Support adding custom env vars
We will need this later to set SOURCE_DATE_EPOCH in the build.
2022-02-09 09:58:49 +01:00
Thomas Lavocat
31fb124716 bootiso: add the option to compress using lz4 2022-02-08 16:02:36 +01:00
Ondřej Budai
172c4bcfaf runners: add rhel-87
As always, it can be just a symlink to rhel-82
2022-02-08 11:37:05 +01:00
Jakub Rusz
860a9bb49f tests/ci: fix sonarqube analysis for main branch
Only use the branch options if it's actually running in a branch and
analyse main without them, sonarqube otherwise doesn't recognize the
main branch.
2022-02-04 12:57:36 +02:00
jkozol
ee46f89647 Post release version bump
[skip ci]
2022-02-02 17:03:08 +00:00
Christian Kellner
0f670829a3 util/linux: fix BLK_IOC_FLSBUF on ppc64le
ioctl contants are platform dependent. It should be the same on
x86, aarch64 and s390x but it is indeed different on ppc64le.
This lead to the call to `ioctl_blockdev_flushbuf` actually
raising an exception of `OSError: [Errno 22] Invalid argument`.

The constant was calculated with a little python snippet that
in theory could also go directly into the code, but for now
the simpler condition in this patch is enough.

The snippet is a port of the defines from the Linux kernel,
specifically /usr/include/asm-generic/ioctl.h.

    class IOConstants:
        """IO Commands for Linux"""
        if platform.machine() == "ppc64le":
            NRBITS = 8
            TYPEBITS = 8
            SIZEBITS = 13

            DIR_NONE = 1
        else:
            NRBITS = 8
            TYPEBITS = 8
            SIZEBITS = 14

            DIR_NONE = 0

        NRSHIFT = 0
        TYPESHIFT = NRSHIFT+NRBITS
        SIZESHIFT = TYPESHIFT+TYPEBITS
        DIRSHIFT = SIZESHIFT+SIZEBITS

        @classmethod
        def make(cls, directory, iotype, nr, size):
            return ((directory << cls.DIRSHIFT) |
                    (iotype << cls.TYPESHIFT) |
                    (nr << cls.NRSHIFT) |
                    (size << cls.SIZESHIFT))

        @classmethod
        def make_dir_none(cls, iotype, nr):
            return cls.make(cls.DIR_NONE, iotype, nr, 0)

This is used to get the value for `BLKFLSBUF` taken from the
include `/usr/include/linux/fs.h`:

    #define BLKFLSBUF  _IO(0x12,97)	/* flush buffer cache */

The value is then obtained via:

    print("0x%x" % IOConstants.make_dir_none(0x12,97))
    0x20001261
2022-02-01 16:28:56 +01:00
Jakub Rusz
087b403042 ci: make jobs interruptible
This will cancel old running pipelines if a new one is created.
2022-01-27 13:01:35 +02:00
Jakub Rusz
b38524b7ee test/ci: fix sonarqube run on main
Minor oversight from previous commit. Specify clone depth in
.gitlab-ci.yml file instead.
2022-01-26 13:51:18 +02:00
Jakub Rusz
8bc4bff80c tests/ci: add sonarqube scan
Adding sonarqube scan to the pipeline.
2022-01-25 13:23:36 +02:00
ochosi
597759c18f Post release version bump
[skip ci]
2022-01-19 14:52:39 +00:00
Thomas Lavocat
bb30ffa062 Add the option of compressing using lz4
Using lz4 in squashfs reduces the time to complete the stage from 446s
to 4s on my laptop while making the produced iso going from 1.6G to
2.0G.
2022-01-11 13:20:11 +01:00
thozza
d19d1b498d Post release version bump
[skip ci]
2022-01-07 11:14:37 +00:00
Christian Kellner
26a250ca6e test/data: don't write cmdline in grub2
In the fedora-boot manifest, do not write the kernel command line
to the `grubenv` file.
2022-01-06 15:09:33 +00:00
Christian Kellner
ee96b11faf stages/grub2: ability to not write kernel cmdline
Currently we always write the kernel command line to the `grubenv`
file, if only to include the root device. Starting with Fedora 33
and thus RHEL 9, the kernel command line included statically in
the BLS snippets and the grubenv `kernelopts` variable not used.
Instead one of the {/usr/lib,/etc}/kernel/cmdline files is read
and the parameters in them used during the creation of the BLS
snippets.
Therefore we add a new `write_cmdline` option that, if set to
FALSE, will prevent us from writing the kernel command line.
2022-01-06 15:09:33 +00:00
Christian Kellner
d4f275e024 stages/grub2: fix whitespace in docstring
For you, David.
2022-01-06 15:09:33 +00:00
Christian Kellner
7a2b8ac107 stages/grub2: extract uuid js schema definition
The uuid schema is duplicated in a few places. Extract it and use
references to it instead.
2022-01-06 15:09:33 +00:00
Christian Kellner
94e9f62f63 test/osbuild: check devices, mounts schema
Also check that the schema is valid for devices and mounts.
2022-01-06 15:09:33 +00:00
Jelle van der Waa
e3515caf01 stages: Remove excess whitespace between words 2022-01-06 16:01:26 +01:00
Jelle van der Waa
3d4be5f059 stages/org.osbuild.pacman: Make /dev/stdin available
Installing the mkinitcpio kernel preset's requires /dev/stdin to be
available for calling install.
2022-01-06 16:01:26 +01:00
Jelle van der Waa
97202e53b0 stages: Respect grub_home for grub legacy boot
Additionally add a new QEMU image output target for testing
grub2/mkinitcpio stages.
2022-01-06 16:01:26 +01:00