particle-os/debian-forge
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
3052 commits 2 branches 0 tags 60 MiB
Find a file
Michael Vogt 2a17756f45 Revert "runners: clean up temp files before exiting the runner" 
This reverts commit bc04bfc366.

The `remove_tmpfiles()` helper is nice but it is also problematic
because it creates extra output after the command was run and
created output. E.g. a test failure on centos stream9 [0]
```
            r = root.run(["stat", "--format=%a", "/var/tmp"], monitor)
            assert r.returncode == 0
>           assert r.stdout.strip().split("\n")[-1] == "1777"
E           AssertionError: assert '/usr/lib/tmp... such process' == '1777'
E
E             - 1777
E             + /usr/lib/tmpfiles.d/rpcbind.conf:2: Failed to resolve user 'rpc': No such process
```
Here the output from "stat" is not the last output because the
rempve_tmpfiles runs `systemd-tmpfiles --clean --remove` which
produces some noisy output after stat was run.

This was found by @thozza (thanks!) and discussed in osbuild PR#1785.

There are various ways to fix this, the one is to use the
`--graceful` option of systemd-tmpfiles. However that only got added in
systemd v256 and centos-stream9 has v252 so that is sadly not an option.

Plus even when avaialble it will produce some informational output like
```
All rules containing unresolvable specifiers will be skipped.
```

Another way would be to sent the output from systemd-tmpfiles cleanup
to /dev/null. Not really great as we will not know about real problems
or warnings that we should care about.

None of the option above is good. So I started looking at the tmpfiles.d
rules and the cleanup and why we are doing it. It was added relatively
recently in https://github.com/osbuild/osbuild/pull/1458 and after
some medidiation not having it seems to do no harm (details below). The
tl;dr is that the buildroot is created inside bubblewrap and the
dirs that `--clean` and `--remove` touch are already tmpdirs created
just for the buildroot so the cleanup in the runner is redundant
(and because the cleanup is now run for each buidlroot.run() command
there *might* be unintended conequences but the current rules seem
to not have any).

In detail, the tmpfiles_cleanup() does two things:
1. `--clean`
It will remove files that are older then the given age
in tmpfiles.d. The tmpfiles in centos9 give me the following ages:
```
$ systemd-tmpfiles --cat-config|grep -E '[0-9]+d$'
d /var/lib/systemd/pstore 0755 root root 14d
d /var/lib/systemd/coredump 0755 root root 3d
q /tmp 1777 root root 10d
q /var/tmp 1777 root root 30d
D! /tmp/.X11-unix 1777 root root 10d
D! /tmp/.ICE-unix 1777 root root 10d
D! /tmp/.XIM-unix 1777 root root 10d
D! /tmp/.font-unix 1777 root root 10d
```
Given that we run our commands inside a bubblewrap environment and
give it a fresh /run, /tmp, /var [1] there really should be no long
lived things and even if there are they are cleaned up from the
buildroot itself

2. `--remove`
It will remove files marked for removal in tmpdfiles.d. Running
it on a centos9 env it yields for me:
```
$ systemd-tmpfiles --cat-config|grep -E '^[rRD]'
R /var/tmp/dnf*/locks/*
r /var/cache/dnf/download_lock.pid
r /var/cache/dnf/metadata_lock.pid
r /var/lib/dnf/rpmdb_lock.pid
r /var/log/log_lock.pid
r! /forcefsck
r! /fastboot
r! /forcequotacheck
D! /var/lib/containers/storage/tmp 0700 root root
D! /run/podman 0700 root root
D! /var/lib/cni/networks
R! /var/tmp/container_images*
D     /run/rpcbind 0700  rpc  rpc  -  -
D /run/sudo/ts 0700 root root
R! /tmp/systemd-private-*
R! /var/tmp/systemd-private-*
r! /var/lib/systemd/coredump/.#*
D! /tmp/.X11-unix 1777 root root 10d
D! /tmp/.ICE-unix 1777 root root 10d
D! /tmp/.XIM-unix 1777 root root 10d
D! /tmp/.font-unix 1777 root root 10d
r! /tmp/.X[0-9]*-lock
```
which is also covered by the bwrap cleanup.

[0] https://artifacts.dev.testing-farm.io/2d07b8f3-5f52-4e61-b1fa-5328a0ff1058/#artifacts-/plans/unit-tests
[1] https://github.com/osbuild/osbuild/blob/main/osbuild/buildroot.py#L218
2024-05-20 11:55:24 -07:00
.devcontainer devcontainer: fix json file 2024-03-26 01:20:37 +01:00
.github workflow: bump osbiuld-ci container to include podman, et al. 2024-05-02 12:09:44 +02:00
assemblers tox: add tox 2023-08-01 15:01:13 +02:00
data devices: add custom udev rule inhibitor mechanism 2021-12-09 00:44:21 +00:00
devices devices/loopback: enable partition scanning as an option 2023-12-22 10:18:29 -05:00
docs doc: document the new json-sequence monitor 2024-03-13 17:27:39 +01:00
inputs testutil: extract find_one_subclass_in_module() helper 2024-04-03 11:36:01 +02:00
mounts mounts: implement new org.osbuild.bind mount 2024-04-11 17:40:21 +02:00
osbuild Revert "runners: clean up temp files before exiting the runner" 2024-05-20 11:55:24 -07:00
runners Revert "runners: clean up temp files before exiting the runner" 2024-05-20 11:55:24 -07:00
schemas schemas: reformat json files for consistencyo 2024-03-26 01:20:37 +01:00
schutzbot Start building RPMs on 9.5 nightly 2024-05-03 11:09:38 +02:00
selinux docs: document osbuild and selinux integration 2021-10-01 11:02:32 +02:00
sources sources/curl: Use our own User-Agent 2024-04-30 03:10:44 +02:00
stages stages/systemd.unit.create: validation function 2024-05-17 14:26:14 +02:00
test Test/buildroot: fix checking /var/tmp mode 2024-05-16 09:58:38 +02:00
tools osbuild-depsolve-dnf5: Fix sslverify setting 2024-04-27 00:29:02 +02:00
.bandit lint: provide bandit configuration 2023-04-23 21:44:46 +02:00
.editorconfig editorconfig: include markdown specifications 2020-10-23 16:29:50 +02:00
.git-blame-ignore-revs git: ignore isort commit 2022-09-12 13:32:51 +02:00
.gitignore tox: add tox 2023-08-01 15:01:13 +02:00
.gitlab-ci.yml Start building RPMs on 9.5 nightly 2024-05-03 11:09:38 +02:00
.mypy.ini util/containers: read host config 2024-02-21 17:55:37 +01:00
.packit.yaml Packit: make COPR builds for new releases in a dedicated project 2023-11-30 14:23:15 +01:00
.ruff.toml meta: rename -meta.json to .meta.json 2024-03-14 13:37:57 +01:00
LICENSE Revert "Fill in the license template" 2019-11-18 12:23:10 +01:00
Makefile Makefile: make sure that make test-all really runs all tests 2024-04-18 08:35:05 +02:00
osbuild.spec Post release version bump 2024-05-08 08:13:38 +00:00
README.md Fix developer guide link in README.md 2024-02-20 23:45:20 -08:00
requirements.txt Makefile: use pytest for nicer output 2020-12-04 18:24:48 +01:00
samples samples: replace with symlink to test data 2021-07-12 18:44:50 +02:00
Schutzfile schutzfile: Update snapshots to 20240514 2024-05-16 07:13:05 -07:00
setup.cfg tox: add tox 2023-08-01 15:01:13 +02:00
setup.py Post release version bump 2024-05-08 08:13:38 +00:00
tox.ini tox,workflow: pass tests to run via tox {posargs} instead of env 2024-04-16 15:16:49 +02:00

README.md

OSBuild

Build-Pipelines for Operating System Artifacts

OSBuild is a pipeline-based build system for operating system artifacts. It defines a universal pipeline description and a build system to execute them, producing artifacts like operating system images, working towards an image build pipeline that is more comprehensible, reproducible, and extendable.

See the osbuild(1) man-page for details on how to run osbuild, the definition of the pipeline description, and more.

Project

Principles

  1. OSBuild stages are never broken, only deprecated. The same manifest should always produce the same output.
  2. OSBuild stages should be explicit whenever possible instead of e.g. relying on the state of the tree.
  3. Pipelines are independent, so the tree is expected to be empty at the beginning of each.
  4. Manifests are expected to be machine-generated, so OSBuild has no convenience functions to support manually created manifests.
  5. The build environment is confined against accidental misuse, but this should not be considered a security boundary.
  6. OSBuild may only use Python language features supported by the oldest target distribution.

Contributing

Please refer to the developer guide to learn about our workflow, code style and more.

Requirements

The requirements for this project are:

  • bubblewrap >= 0.4.0
  • python >= 3.6

Additionally, the built-in stages require:

  • bash >= 5.0
  • coreutils >= 8.31
  • curl >= 7.68
  • qemu-img >= 4.2.0
  • rpm >= 4.15
  • tar >= 1.32
  • util-linux >= 235
  • skopeo

At build-time, the following software is required:

  • python-docutils >= 0.13
  • pkg-config >= 0.29

Testing requires additional software:

  • pytest

Build

Osbuild is a python script so it is not compiled. To verify changes made to the code use included makefile rules:

  • make lint to run linter on top of the code
  • make test-all to run base set of tests
  • sudo make test-run to run extended set of tests (takes long time)

Installation

Installing osbuild requires to not only install the osbuild module, but also additional artifacts such as tools (i.e: osbuild-mpp) sources, stages, schemas and SELinux policies.

For this reason, doing an installation from source is not trivial and the easier way to install it is to create the set of RPMs that contain all these components.

This can be done with the rpm make target, i.e:

make rpm

A set of RPMs will be created in the ./rpmbuild/RPMS/noarch/ directory and can be installed in the system using the distribution package manager, i.e:

sudo dnf install ./rpmbuild/RPMS/noarch/*.rpm

Repository

License

  • Apache-2.0
  • See LICENSE file for details.