Each time the overlay backend runs on an xfs fs it creates the
file "overlay/backingFsBlockDev" in the containers storage directory.
It is not needed in the image as skopeo recreates it each boot, and
since it is a block device its existance means we can't store the
resulting tree in ostree. Lets just get rid of it.
During the rework done in commit "use and require explicit exports"
with commit id 7ae4a7e78, the test got overlooked. Add an empty
list of checkpoints to the `obs.compile` invocation as to actually
trigger the osbuild invocation.
Reported-By: Thomas Lavocat <tlavocat@redhat.com>
We hardcode oci-archive for now and use the first file in the pipeline
tree as the archive. Long term we may want to get this info from the
metadata of the oci-archive stage instead.
There is a source test that installs a pre-build, embeded image file
and ensure all the right files are installed. This uses the vfs driver
because then it works everywhere, including the CI (which doesn't do
overlayfs).
Then the is a source test that downloads a minimal image from
a faked registry on localhost.
For the registy API to work the "/v2" entry-point in the webserver has
to be at the root, so there is a symlink in test/data:
v2 -> sources/org.osbuild.skopeo/data/v2
But otherwise the data is localized to sources/org.osbuild.skopeo.
Both file embedding and depsolves start by iterating over all stages,
and we want to add another similar one for container installs, so
break out the iteration over the containers so that it is done in
one place only.
This adds a stage called org.osbuild.skopeo that installs docker and
oci archive files into the container storage of the tree being
constructed.
The source can either be a file from another pipeline, for example one
created with the existing org.osbuild.oci-archive stage, or it can
be using the new org.osbuild.skopeo source and org.osbuild.containers
input, which will download an image from a registry and install that.
There is an optional option in the install stage that lets you
configure a custom storage location, which allows the use of the
additionalimagestores option in the container storage.conf
to use a read-only image stores (instead of /var/lib/container).
Note: skopeo fails to start if /etc/containers/policy.json is
not available, so we bind mount it from the build tree to the
buildroot if available.
The client side does meta.get("source-epoch", default), but for
this to work we need to have the key unset if not specified,
but currently we set it to None.
Also, make sure the check for "not None" is explicit, because
we do consider a value of `0` to be a valid source-epoch.
This tries to make the various tar-balls produced by the stage more
likely to be identical in separate runs. We do this by sorting
the names and removing some unnecessary metadata for the files.
The most important thing to get right is the layer tarball, because
that is what defines the container id. We sort the names to avoid
random differences, and drop ctimes and atimes because these are
generally just set to the something near the current build time which
is not useful to encode in the container image. This is as opposed
to the mtime which generally comes from e.g. the rpms that where
installed in the pipeline.
For the actual archive tarball we can standardize metadata even more,
because none of the metadata are used when consuming the archive.
ioctl contants are platform dependent. It should be the same on
x86, aarch64 and s390x but it is indeed different on ppc64le.
This lead to the call to `ioctl_blockdev_flushbuf` actually
raising an exception of `OSError: [Errno 22] Invalid argument`.
The constant was calculated with a little python snippet that
in theory could also go directly into the code, but for now
the simpler condition in this patch is enough.
The snippet is a port of the defines from the Linux kernel,
specifically /usr/include/asm-generic/ioctl.h.
class IOConstants:
"""IO Commands for Linux"""
if platform.machine() == "ppc64le":
NRBITS = 8
TYPEBITS = 8
SIZEBITS = 13
DIR_NONE = 1
else:
NRBITS = 8
TYPEBITS = 8
SIZEBITS = 14
DIR_NONE = 0
NRSHIFT = 0
TYPESHIFT = NRSHIFT+NRBITS
SIZESHIFT = TYPESHIFT+TYPEBITS
DIRSHIFT = SIZESHIFT+SIZEBITS
@classmethod
def make(cls, directory, iotype, nr, size):
return ((directory << cls.DIRSHIFT) |
(iotype << cls.TYPESHIFT) |
(nr << cls.NRSHIFT) |
(size << cls.SIZESHIFT))
@classmethod
def make_dir_none(cls, iotype, nr):
return cls.make(cls.DIR_NONE, iotype, nr, 0)
This is used to get the value for `BLKFLSBUF` taken from the
include `/usr/include/linux/fs.h`:
#define BLKFLSBUF _IO(0x12,97) /* flush buffer cache */
The value is then obtained via:
print("0x%x" % IOConstants.make_dir_none(0x12,97))
0x20001261
Currently we always write the kernel command line to the `grubenv`
file, if only to include the root device. Starting with Fedora 33
and thus RHEL 9, the kernel command line included statically in
the BLS snippets and the grubenv `kernelopts` variable not used.
Instead one of the {/usr/lib,/etc}/kernel/cmdline files is read
and the parameters in them used during the creation of the BLS
snippets.
Therefore we add a new `write_cmdline` option that, if set to
FALSE, will prevent us from writing the kernel command line.
The sysconfig stage currently does not produce expected results when
used multiple times within the same pipeline. Specifically, the stage
always truncates respective configuration files for properties `kernel`
and `network`, if if these are not set in the stage options. Due to this
reason, the outcome of the image builds may depend on the order of
multiple occurrences of the sysconfig stage.
The following two pipeline snippets would produce different
configuration files content:
Configuration files are truncated:
```
{
"type": "org.osbuild.sysconfig",
"options": {
"kernel": {
"update_default": true,
"default_kernel": "kernel"
},
"network": {
"networking": true,
"no_zero_conf": true
}
}
},
{
"type": "org.osbuild.sysconfig",
"options": {
"network-scripts": {
"ifcfg": {
"eth0": {
"bootproto": "dhcp",
"device": "eth0",
"ipv6init": false,
"onboot": true,
"peerdns": true,
"type": "Ethernet",
"userctl": true
}
}
}
}
},
```
No configuration files are truncated:
```
{
"type": "org.osbuild.sysconfig",
"options": {
"network-scripts": {
"ifcfg": {
"eth0": {
"bootproto": "dhcp",
"device": "eth0",
"ipv6init": false,
"onboot": true,
"peerdns": true,
"type": "Ethernet",
"userctl": true
}
}
}
}
},
{
"type": "org.osbuild.sysconfig",
"options": {
"kernel": {
"update_default": true,
"default_kernel": "kernel"
},
"network": {
"networking": true,
"no_zero_conf": true
}
}
},
```
Change the stage to not touch respective configuration files if the
`kernel` and `network` properties are not set in the stage options.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
The treesum of a filesystem tree is the content hash of all its
files, its directory structure and file metadata.
By storing trees by their treesum we avoid storing duplicates of
identical trees, at the cost of computing the hashes for every
commit to the store.
This has limited benefit as the likelihood of two trees being
identical is slim, in particular when we already have the ability
to cache based on pipeline/stage ID (i.e., we can avoid rebuilding
trees if the pipelines that built them were the same).
Drop the concept of a treesum entirely, even though I very much
liked the idea in theory...
Signed-off-by: Tom Gundersen <teg@jklm.no>
Pacman is the default package manager for Arch Linux and derivates, the
pacman.conf stage generate a valid pacman.conf configuration file.
Co-Authored-By: Jelle van der Waa <jvanderwaa@redhat.com>
