In CoreOS Assembler, some hyperv artifact we `zip` for compression. This
new stage is modeled after the `org.osbuild.tar` stage with necessary
modifications.
Add an org.osbuild.systemd.unit stage using the new format for the
Environment option with two instances to the test manifest.
The contents of the new dropin file at
tree/usr/lib/systemd/system/boltd.service.d/30-boltd-debug.conf are:
[Service]
Environment="G_MESSAGES_DEBUG=all"
Environment="G_MESSAGES_TRACE=none"
Add the new options to the b.json test and update the diff.
The new file has the following contents:
[Unit]
Description=Create directory
DefaultDependencies=False
ConditionPathExists=|!/etc/myfile
ConditionPathIsDirectory=|!/etc/mydir
[Service]
Type=oneshot
RemainAfterExit=True
ExecStart=mkdir -p /etc/mydir
ExecStart=touch /etc/myfile
Environment="DEBUG=1"
EnvironmentFile=/etc/example.env
[Install]
WantedBy=local-fs.target
RequiredBy=multi-user.target
Add an empty file to the location where the service file will be
created in the b.json version of the test. This way, we will get a
content hash of the created file which is a slightly better test than
just knowing that it was created.
Note that, in the diff, the "before" checksum is the empty file hash:
echo -n '' | sha256sum
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -
In 2d2cdd8097 the file was replaced by
the generated json and it went unnoticed in the PR. Reverted and
updated the options to match the generated json file.
This is needed because on a mounted `bootc` container `setfiles`
without excluding `/sysroot` will create many warnings like:
```
setfiles: conflicting specifications for /run/osbuild/tree/sysroot/ostree/repo/objects/00/0ef9ada2ee87792e8ba21afd65aa00d79a1253018832652b8694862fb80e84.file and /run/osbuild/tree/usr/lib/firmware/cirrus/cs35l41-dsp1-spk-prot-103c8b8f-r1.bin.xz, using system_u:object_r:lib_t:s0.
```
but simply excluding this dir fixes them.
Instead of requiring only one of the properties to be present require at
least one of them being present; some stages specify both schema
versions (`org.osbuild.rpm`)
Instead of just mocking the binary also write a log of the way
it got called so that tests can use this to check if the right
options are passed.
Note that the API should be improved here, instead of returning
a "naked" path to the calllog file there should be a class wrapping
it. And of course there should be tests.
The current `make_container()` helper is a bit silly (which is
entirely my fault). It requires a container tag as input but all
tests end up creating a random number for this input. So instead
just remove the input and return the container_id from the podman
build in the contextmanager and use that.
Based on the feedback from Tomáš in [0] this commit adds tests
that ensure consistent behavior between the python and the json
loader.
It's not 100% because the python is extremly leaniant and does
not even check if the required pieces of the json are there.
I.e. it will load a module without a SCHEMA or SCHEMA_2 variable
and the json loader code will warn about the issue but not
raise an error.
Fwiw, I have no strong opinion here but I do lean slightly towards
staying close to the original code (but both approaches of failing
with an exectption and continue with a warning have good arguments).
[0] https://github.com/osbuild/osbuild/pull/1618#discussion_r1521141148
Instead of always parsing the python stage to load meta information
allow the user of a new `{stage}-meta.json` file. This is a first
step towards allowing modules to be written in a different language
than python. It also has some practical advantages:
- slightly faster as it avoids calling python to output the schemas
- easier to write schemas as this can be done in a real json editor
now
- more extensible in a future where stages maybe binaries with
shlib dependencies that are only satisfied in the buildroot
but not on the host
This reverts commit 158acaac78.
With https://github.com/osbuild/bootc-image-builder/pull/238 the
original reason to call mknod goes away so we can just revert
it. osbuild now requires not only the loop device but also uses
`losetup --partscan` quite a lot now so the mknod approach becomes
impractical and the consumers of osbuild in a container should
just setup devtmpfs.
Since we support python3.6 we cannot assume that dicts are ordered
in any way. To ensure the `id` is still always valid we pass
sort_keys=True to json.dump().
Thanks to Simon!
Generate log messages with origin "org.osbuild.main" when
pipelines/stages start and finish. This way a higher level
frontend can display high level progress coming from this
origin and filter out e.g. stages based log messages (that
are usually quite technical as they are just stdout/stderr
from the stages).
Tweak the Progress class to be simpler. Given that progress does
not need to support arbitrary depth but only has a single level
the class now just exposes "sub_progress" to the caller.
When the main progress is advanced the sub_progress is now fully
deleted instead of just reset. The rational is that when the main
progress is done and advances a step it is very likely that a
new sub_progress is required and it's most likely an error if
the same sub_progress will get re-used.
This means that `reset()` can be removed as it's not used anymore
(and YAGNI). We can add it back when we have a use-case.
It also change the code so that "total" starts with 0 instead
of `None` (principle of least surprise). This means that now
`progress.incr()` is called in the JSONSeqMonitor() for
`finish()` and `result()` to indicate that the pipeline/stage
is finished.
This commit tweaks Context a bit so that any write will automatically
reset the `_id`. This ensures that we do not forget to reset `_id`
when the code changes.
It also tweaks the naming a bit, before there was a "setter" for
origin and functions to set "pipeline" and "stage". They are all
functions now with a "set_" prefix for symetry mostly.
The class LogLine() is purely used as a dataclass with no state
and the only function on it is `as_dict()`. This got refactored
into a new function `log_entry()` because there is no need for
this to be a class. The function that takes the same inputs.
This is a follow up to #1550 where we enabled a `rw` permissions mode,
which is not ideal since it would theoretically be possible to set both
`ro` and `rw` modes at the same time. This commit fixes the issue by only
allowing one option at a time.
Fixes#1588
User can now customize the systemd unit load path.
User can select between etc or usr , defaults to 'usr'.
Also user can customize the scope of the service between global
or system, defaults to system.
Signed-off-by: Sayan Paul <paul.sayan@gmail.com>
The `test_assembler.py` hardcods some filesystem and partition
UUIDs. This leads to hard to diagnose test failures when the
test is run in parallel. The btrfs and xfs filesystem drivers
will see the same uuid for multi created images and error sometimes with
someting like:
```
Mar 06 10:22:54 top kernel: BTRFS error: device /dev/loop104 belongs to fsid aff010e9-df95-4f81-be6b-e22317251033, and the fs is already mounted, scanned by mount (123856)
```
Its a race that only happens when two images are checked at the
same time.
This commit fixes the issue by just using a randomized UUID in
the test_assemblers.py. It also re-enables running the test in
parallel (which make it run a lot faster, from 34min to 14min).
The BLS specification [0] says the `options` field is optional and
can also appear multiple times. This commit tweaks the code to
deal with these corner cases and also adds tests that ensure that
this works correctly.
It also tweaks the file handling to be atomic.
[0] https://uapi-group.org/specifications/specs/boot_loader_specification/
- Process all necessary operations related to CoreOS
platforms is crucial and specific to CoreOS. This step
is essential for CoreOS exclusively.
- Our approach to handling 'platforms.json' may change as we
advance with the OSBuild work. However, we don't have a clear
vision about how it will be in the future yet, particularly as
we also manage similar components within the osbuild composer
to configure cloud parameters. We probably will know better
when we start working with the cloud artifacts.
As a summary, let's add it know to unblock us, and if we find a
better approach in the future, we can always go back and remove it.
Signed-off-by: Renata Ravanelli <rravanel@redhat.com>
A manifest (mpp and json) that uses the new source and input with the
skopeo stage.
This depends on the image we store at
./test/data/stages/skopeo/hello.img
The plan is to test this by pulling the hello.img into the host root
storage, build the manifest, delete the image from storage, and check
the tree.
Add systemd unit files in osbuild stage
This stage creates systemd unit file in `/usr/lib/systemd/system/`.
The stage accepts filename which must end with `.service`.Section
`Unit` , `Service` , `Install` accepts various parameters as per
the systemd documentaion.`systemd-analyze verify` is be performed
after the .service file is created to check for potential errors.
Signed-off-by: Sayan Paul <paul.sayan@gmail.com>
The new `testutil.mock_command` context manager can be used to
mock commands in PATH and replace them with arbitrary shell
scripts. This is useful in testing to e.g. simulate exact error
conditions that would be hard to trigger otherwise or to replace
long running commands with faked results.
Example:
```
fake_cmd = textwrap.dedent("""\
do-something
""")
with mock_command("some-cmd", fake_cmd):
your_code
```
