Commit graph

2108 commits

Author SHA1 Message Date
Simon de Vlieger
7d8b2302c0 git: ignore isort commit 2022-09-12 13:32:51 +02:00
Simon de Vlieger
ea6085fae6 osbuild: run isort on all files 2022-09-12 13:32:51 +02:00
Simon de Vlieger
98daf1a87d osbuild: setup isort autochecking
The `test-src` makefile target now also runs isort against all source
files and reports errors.
2022-09-12 13:32:51 +02:00
Simon de Vlieger
a5be1cc4d2 linting: use-implicit-booleaness-not-comparison
Newer warning from pylint, also consistent with how we do things
elsewhere. Note that this only applies to one file in the tests but
disabling it would be very weird for such a small fix.
2022-09-12 10:52:09 +02:00
Simon de Vlieger
321587ad41 linting: fix use-dict-literal
Use the dict literal `{}` instead of `dict()`, this is a warning in
newer pylints and its consistent with what we do elsewhere.
2022-09-12 10:52:09 +02:00
Simon de Vlieger
873a071d43 osbuild: share terminal formats between files 2022-09-09 21:43:56 +02:00
Simon de Vlieger
5cb2da55f1 test: run mypy in test-src not in GH actions 2022-09-09 18:51:57 +02:00
Simon de Vlieger
e330e82cf0 lint: appease autopep8 2022-09-09 15:33:29 +02:00
Simon de Vlieger
b07aca5d86 modules: explicit encodings for open() 2022-09-09 15:33:29 +02:00
Simon de Vlieger
38d2ab685c test: explicit encodings for open() 2022-09-09 15:33:29 +02:00
Simon de Vlieger
3703328751 osbuild: explicit encodings for open()
Provides explicit encodings for all calls to `open()`, this is a newer
pylint warning but also just makes sense to do.
2022-09-09 15:33:29 +02:00
Ygal Blum
42552e0436 osbuild-mpp: Add url option to mpp-embed
Allow the user to pass a URL to be embeded as org.osbuild.curl input

Signed-off-by: Ygal Blum <ygal.blum@gmail.com>
2022-09-06 14:28:06 +03:00
Alexander Larsson
f4bfce05e3 org.osbuild.oci-archive: Support setting Entrypoint
This allows setting Entrypoint (as well as Cmd) in the oci image,
as per the spec:

  https://github.com/opencontainers/image-spec/blob/main/config.md

Note: These two are not equivalent, the Cmd part is replaced by the
argument on the `docker run` commandline, whereas the entrypoint is
kept from the config, so it is important to expose both of these.
2022-09-05 13:10:47 +02:00
Simon Steinbeiss
bc72803c9a packit: Enable Bodhi updates workflow 2022-09-01 15:22:22 +02:00
Achilleas Koutsou
4bc6e226ea rpmbuild: add fedora-37 2022-09-01 11:31:58 +02:00
Achilleas Koutsou
f699720dbd sources/curl: quote URL paths before downloading
Some package versions [1] can contain carets and other characters that
curl doesn't like.  These need to be URL encoded.

Interestingly, the documented way of replacing components in a parsed
URL from urllib in Python is by calling the (seemingly private)
`_replace()` method [2].

[1] https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/#_snapshots
[2] https://docs.python.org/3/library/urllib.parse.html#url-parsing
2022-08-31 22:28:54 +01:00
schutzbot
1782f2fea9 Post release version bump
[skip ci]
2022-08-31 08:16:19 +00:00
Thomas Lavocat
f38ddb5f20 test: manifests testing on each PR
On each PR test that osbuild don't break the image generation by testing
against the manifest-db that the produced image-infos are still the same
as the one stored.
2022-08-30 14:42:13 +02:00
schutzbot
d467e88f99 Post release version bump
[skip ci]
2022-08-26 17:11:12 +00:00
Christian Kellner
ba218f781d pipeline: include mounts in stage checksum
When calculating the checksum of the stage, the mount options were
not included. This was maybe deliberate, because if the mounts of
a stage change, it is very likely that previous stages change too.
But the introduction of non-device mounts, like ostree.deployment,
have changed the setting, since the content of the tree will be
different if that mount is applied or not. And even for the device
based mounts it will change the tree if e.g. a device is mounted
at at different path but otherwise is formatted with the very same
options. In the worst case we miss a few cache hits due to changes
in the mount setup that don't lead to tree changes, but that will
rarely happen in practice.
2022-08-23 19:29:33 +01:00
Sayan Paul
28854f452a Add greenboot configuration management via osbuild
The script will update /etc/greenboot/greenboot.conf if user passes
the parameter in the config. Right now this only tweaks one config but
it will/can be expanded if other use cases come.

Signed-off-by: Antonio Murdaca <runcom@linux.com>
2022-08-23 16:53:55 +02:00
Achilleas Koutsou
b87eaf6032 stages/ostree.remotes: support contenturl
OSTree remotes can be configured with a contenturl that is separate from
the metadata url.  When specified, the contenturl is used for content
and the url is used only for metadata.
Updated the description of the url property for clarity.
2022-08-22 20:17:40 +02:00
Achilleas Koutsou
2bff83364b stages/ostree.remotes: support gpgkeypath
OSTree remotes can be configured to import keys from a file on disk or a
directory containing multiple keys.

When specified, explicitly enables gpg-verify=true.
2022-08-22 20:17:40 +02:00
Ondřej Budai
135b20a487 runners: add fedora38
It's that time of the year again.
2022-08-18 14:22:44 +01:00
schutzbot
cf8ebb46b2 Post release version bump
[skip ci]
2022-08-17 08:17:53 +00:00
Christian Kellner
07d3605385 manifests: set db path in ostree commit
rpm-ostree is setting the database path to /usr/share/rpm during
the build. Do so as well.
2022-08-15 18:22:59 +02:00
Christian Kellner
a967c61d17 stages/rpm: ability to set database path
Add a new option `dbpath` that controls the database path of the
rpm database.
2022-08-15 18:22:59 +02:00
Christian Kellner
8c00907461 stages/rpm.macros: new-line terminate values
Needed if we want to specify multiple values and even for the
single value use case this is the right thing to do.
2022-08-15 18:22:59 +02:00
Christian Kellner
b33f178a11 stages/rpm.macros: allow setting of _dbpath
Allow setting the database path via the `_dbpath` variable.
2022-08-15 18:22:59 +02:00
Christian Kellner
c28269f8d9 manifests: enable cloud-init services in commit
It is instabled, but not a enabled via presets so we have to
explicitly enable it.
2022-08-13 19:21:52 +01:00
Christian Kellner
5570e2a92e manifests: upgrade ostree commit, image to F36
Fedora 34 is end of life and we ought to be testing with newer
releases so that we catch issues like the authselect one, that
is detailed in "stages/rpm: support marking install as ostree".
2022-08-13 19:21:52 +01:00
Christian Kellner
899fdcd5d9 manifests: set rpm/ostree_booted for commit
See "stages/rpm: support marking install as ostree" for more
information why this is needed on Fedora 36 and newer.
2022-08-13 19:21:52 +01:00
Christian Kellner
10255dbdcc test/manifests: base ostree image on commit
Instead of re-defining the ostree-commit in the ostree-image.mpp
file, import the needed pipelines to build the commit and use
that.
2022-08-13 19:21:52 +01:00
Christian Kellner
d2bcaffbdc test/manifests: add update repos in ostree-commit
Include modular and updates repositories for ostree commit and the
new build manifest.
2022-08-13 19:21:52 +01:00
Christian Kellner
073359166d test/manifests: add generic fedora build manifest
Basically a copy of f34-build-v2 but uses variables. Use it form
fedora-ostree-commit.
2022-08-13 19:21:52 +01:00
Christian Kellner
056a33041b manifests: use variables in ostree-commit
Instead of hard-coding the release and the snapshot, use variables,
which should help updating to newer releases and snapshots.
2022-08-13 19:21:52 +01:00
Christian Kellner
4b11989ce3 manifests: convert ostree commit to format v2
Convert the manifest to use version 2 format. Version 1 is really
not used much anymore because composer was completely ported to
using v2. Welcome to the future, ostree commit.
2022-08-13 19:21:52 +01:00
Christian Kellner
c3b172e16b manifests/ostree-commit: update package list
Include cryptsetup and lvm2 so that commit can be deployed to
images that are encrypted and have lvm2 layout. Additionally,
include greenboot. Remove the explicit dependency on ostree
and systemd-udev, which are automatically pulled in.
2022-08-13 19:21:52 +01:00
Christian Kellner
16f1c560cc stages/rpm: support marking install as ostree
An ostree system can be identified as such by the presence of a
marker file: /run/ostree-booted. The `rpm-ostree` tool also
creates this marker during the _installation_ of the system[1,2].
Recently, starting with F36, the authselect package has taken
has become mandatory[3] and is now owning the nsswitch config.
An rpm-ostree system, which has parts of the user database in
/usr, release on the nss-altfiles to read these databases. The
necessary entries are added during the post-processing, which
is called in our `org.osbuild.ostree.preptree` stage[4]. This
installation is skipped though if the nsswitch is the file is
a symlink, indicating that it is owned by some other package,
like authselect. So the F36 authselect change first broke rpm-
ostree[6]. The fix was to check for `/run/ostree-booted` in
the authselect scriptlet and special case this situation[7,8].
Now, our `org.osbuild.rpm` stage does not yet have the ability
to pretend it is a running ostree system and thus we did not
get the special treatment resulting in nss-altfiles not being
enabled in our ostree commits. Therefore the passwd database in
/usr was not read and a lot of daemons and programs without a
valid user, like e.g. `sshd`.
This change introduces a new option, `ostree_booted` that if
set, will create the `/run/ostree-booted` marker and thus our
installation phase will get the same treatments from packages
as rpm-ostree. Hopefully.

[1] 730bec87b1/rust/src/builtins/compose/mod.rs (L24)
[2] 6211d1452e/src/app/rpmostree-compose-builtin-tree.cxx (L501)
[3] https://fedoraproject.org/wiki/Changes/Make_Authselect_Mandatory
[4] 7993c6f565/rust/src/composepost.rs (L635)
[5] d614caeca1/f/0010-spec-fix-detection-of-ostree-system.patch
[6] https://bugzilla.redhat.com/show_bug.cgi?id=2034360
[7] https://src.fedoraproject.org/rpms/authselect/c/d614caeca1a68f55542aefd0d76bda2691c85d24?branch=f36
[8] https://github.com/authselect/authselect/issues/48
2022-08-13 19:21:52 +01:00
Christian Kellner
776bab46ae stages/rpm: convert the mount code to python
Convert the code that sets up the bind mounts to /dev, /sys/ and
/proc from a bash script to python by using the new mount helper
that were moved from the `objectstore`.
2022-08-13 19:21:52 +01:00
Christian Kellner
8f95154a8c stages/rpm: extract machine-id code into function
Extract the code that creates the fake machine id, if it does not
exist, into its own function. Also convert the shell code into
Python code.
2022-08-13 19:21:52 +01:00
Christian Kellner
2e09e7937c objectstore, move {u,}mount methods to util.mnt
Move the mount and umount helpers to the new mount utility module.
No semantic change in the function.
2022-08-13 19:21:52 +01:00
Christian Kellner
a43ea66be2 util/mnt: rename var to avoid future clash
Rename the local `mount` variable to `mnt` so that we can have
a top level `mount` function.
2022-08-13 19:21:52 +01:00
Christian Kellner
b49f3f91f9 util/mnt: extract MountGuard into new module
Extract the `MountGuard` class from all stages that defined it into
a new `mnt` utility module.
2022-08-13 19:21:52 +01:00
Alexander Larsson
7c0f0c1052 osbuild-mpp: Add comment detailing the stage formating 2022-08-08 14:21:13 +02:00
Alexander Larsson
24e48d2f71 osbuild-mpp: Use empty string for pipeline name if missing
This matches what we did before last commit.
2022-08-08 14:21:13 +02:00
Alexander Larsson
fc2697927a osbuild-mpp: Allow use of mpp-* operations for stages
This mergest the handling of process_stages() and process_format() into
just one process_format(), which incrementally tracks the call stack
of the formating, which allows it to detect when it is hitting a stage
and can call _process_stage().

This means it is possible to mix things like mpp-if and mpp-join with
stages.

For example, you can do complex combinations like:

pipelines:
- name: rootfs
  stages:
    mpp-join:
      - - type: org.ostree.foo
        - mpp-if: use_bar
          then:
            type: org.osbuild.bar
      - mpp-if: use_extra_stages
        then:
          mpp-eval: extra_stages

This is particularly useful if you included something and you want
to mpp-join something that was set in a variable.
2022-08-08 14:21:13 +02:00
Alexander Larsson
ed99aa4bfa osbuild-mpp: Store solver_factory in manifest
We do this instead of passing it around to simplify the APIs so
we don't have to track it. This will help for later changes.
2022-08-08 14:21:13 +02:00
Christian Kellner
561122dd30 ci: add autopep8 check to checks.yml
Add to new autopep8 check to the checks suit. Ideally we would
move the `test.src` test over from tests.yml, but running that
in an updated container leads to a whole lot of new warnings,
most of them related to "unspecified encoding" in `open` calls.
2022-08-05 09:41:05 +02:00
Christian Kellner
5c8d11ebe1 test/pylint: check error code instead of exception
Instead of using `subprocess` with `check=True`, which will
echo the command used, including all the files it checked,
check the error code and if non-zero, fail the test with a
nice small error message.
2022-08-05 09:41:05 +02:00