Other commands such as `manifest`, and `build` auto detect the
distribution if none is given. `describe` is the odd one out that
requires `--distro`. Let's also autoselect there.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
Rename the `list-images` command to `list`. We don't have `-image(s)` in
our other subcommands so this is for consistency.
We keep a `list-images` alias behind for compatibility reasons.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
When running `image-builder` warnings are emitted during manifest
generation. Depending on the definitions or customizations packages
can be excluded from groups which leads to:
```
No match for group package "dracut-config-rescue"
```
Interspersing with normal output. Let's pass along another byte buffer
for the warnings from manifest generation to be written into.
Note that this also needs plumbing [1] to land in `images` first.
[1]: https://github.com/osbuild/images/pull/1384
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
Add a check which leverages the osbuild/images@check-spec-deps-action
action to check that the SPEC files requires at least the minimum
versions for dependencies specified by the `osbuild/images`.
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
This commit allows controlling the `osbuild --cache-max-size`
option. By default it will set the cache to 20GiB but allows
this to be controlled by the user.
Thanks to Simon for raising this.
Change the progress imports to refer to `pkg/progress` inside this
repository as it was imported from `bootc-image-builder`.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
Moves the files imported from `bootc-image-builder` directly under `pkg`
so they can be imported in reverse. Also fix up any import paths at the
same time.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
This commit adds a new `BuildLog` option to the `OSBuildOptions`
that can be used to generate a streamed buildlog (e.g. to a file
or a websocket).
This will be used in `ibcli` with a new `--with-buildlog` option.
This commit adds a `OSBuildOptions` struct that can be used to
pass (optional) options to the `progress.RunOSBuild()` helper.
This make it easier to expand with more options.
This commit changes the progress parser (again) to deal with
errors from the osbuild json progress scanner. On errors we
will now exit right away and potentially kill osbuild but
provide an error message that hints how to workaround the
issue.
The original code assumed we get transient errors like json
decode issues. However it turns out that this is not always
the case, some errors from a bufio.Scanner are unrecoverable
(like ErrTooBig) and trying again just leads to an endless
loop. We can also not "break" and wait for the build to finish
because that would appear like the progress is broken
forever and we would still have to report an error (just
much later).
This commit changes the way `os.Stderr` is mocked so that higher
level consumes of the libary can use helpers can replace os.Stderr
(like `testutil.CaptureStdio()` is doing). The existing approach
assigns the "real" os.Stderr to osStderr so early that it cannot
be changed later.
This commit fixes a silly mistake from PR#810. The issue is that
in #810 we started to collect the osbuild stdout/stderr so that
we can show crashes from osbuild or other unexpected output.
However when a stage fails this is reported via the json progress
and not directly on stdout/stderr - this was missed when #810
was done.
This commit does a short term fix by collecting the buildlog again
and showing it in the error and also updates the test to be more
realistic. However we really need a test that actually tests
the real behavior, ideally a real osbuild run with a stage error
so that we can be sure we capture this (criticial!) functionality.
This commit tweaks an issue that potentially an incorrect status
from osbuild would fail the build with a bad error message and
without us getting the full buildlog.
This commit adds catpure of os.Std{out,err} when running osbuild so
that we capture the error log and can display it as part of
an error from osbuild, e.g. when osbuild itself crashes.
This gives us more accurate error reporting if anything fails
during the osbuild building.
This commit makes the previously internal `progress` package an
external API. The is the result of the discussion in images
PR#1150 where we decided that the progress module is not a great
fit for the "images" library.
We want to share this code between bootc-image-builder and
image-builder-cli now. In the future we will also want to
use it in the `worker-executor` in `osbuild-composer` to
parse the stream data from `osbuild`.
We plan to merge bootc-image-builder and image-builder-cli
medium term so importing code from bootc-image-buider in
image-builder cli is not that stange.
When we (longer-term) use this code the `worker-executor`
we will need to think about this again and maybe put it
back into images. However this commit unblocks us without
making anything worse.
Moves the files imported from `bootc-image-builder` to the appropriate
path under `pkg/` so they can be imported reverse. Also fix up the
import paths that are in these files to refer to their new locations.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
Pulling container images before building would break in the case of
authenticated images on podman machine, since the auth file lives on the
host and not podman machine and won't know about it.
This commit deprecates the `--local` flag and warns users when it is
passed to the CLI so that this won't break things for anyone who might
already be using the flag. This change means that the user will have to
ensure that the container is pulled to the local container store before
initiating the build.
Co-authored-by: Ondřej Budai <obudai@redhat.com>
This commit checks early if cross architecture building support via
`qemu-user-static` (or similar tooling) is missing and errors in
a more user friendly way.
Note that there is no integration test right now because testing
this for real requires mutating the very global state of
`echo 0 > /proc/sys/fs/binfmt_misc/qemu-aarch64`
which would make the test non-parallelizable and even risks
failing other cross-arch tests running on the same host (because
binfmt-misc is not namespaced (yet)).
The getContainerSize() was not using some of the modern go helpers.
So let's use `exec.Command().Output()` and introduce a new
`util.OutputErr()` helper that will be able to also show stderr to
the user if the Output() call returns an error.
I ran into the `--local` option not working at least in my
setup with the current container image (which for some reason
hasn't been updated, and predates the latest tip commit
which talks about btrfs)
On this current test system (MacOS + podman 5 + default podman-machine)
things do work with the tip commit.
However...I don't quite understand the need to *both* try to
run `systemd-detect-virt` *and* parse `/proc/self/mountinfo`.
(BTW, the logic for `insideContainer` was really confusing me because
it only returned `true` if we were *not* in a container...I wonder
if that was really intentional?)
Anyways, I think the goal here is just a friendly direct error
message if it doesn't look like the mount is there, which we can
do by just checking for the file path. If it isn't mounted
from the host then nothing will be there.
Signed-off-by: Colin Walters <walters@verbum.org>
When the container storage is checked it may come from a btrfs
subvolume. In /proc/self/mountinfo this looks like:
```
1364 1345 0:33 /root/var/lib/containers/storage /var/lib/containers/storage rw,relatime - btrfs /dev/mapper/luks-1356123c-c7b9-1bd7-a8bf-e13d4cbb5500 rw,seclabel,compress=zstd:1,ssd,discard=async,space_cache,subvolid=257,subvol=/root
```
as a quick fix just compare the suffix of the root instead of the
full path.
Thanks for Ondrej!
Ensure to error early when the user is not passing the required
```
-v /var/lib/containers/storage:/var/lib/containers/storage
```
when doing a local build.
Ensure that the disk size of a container is taken into account
when the image is generated. The current heuristic is that we
just double the container size.
The test will not build an image just generate a manifest and
check that the image file is generated with the expected size.
The way osbuild works is to synthesize a filesystem tree in the
store, then copy it to the disk. This ensures the label for the
store is `root_t` which ends up being the labeling for
the "infrastructure" bits in the `/ostree` repository in the
target root.
This in turn is blocking a lot of things.
Closes: https://github.com/osbuild/bootc-image-builder/issues/149
This ensures that the new `partscan` feature in osbuild works. By
default the containers only have a static snapshot of /dev on a
tmpfs. This means that anything later added by losetup will be
missing inside the container.
It also means that https://github.com/osbuild/osbuild/pull/1468
can be reverted.
In most cases e.g. we do want to show stdout/stderr, and
it's handy to have a debug log when we're running
a subprocess.
While we're here, switch to just forking `cp` in
the setup code.
Signed-off-by: Colin Walters <walters@verbum.org>
Allow users to define the seed that's used for manifest generation.
Regenerating an image with a given seed leads to the same manifest
(provided depsolving does the same).
The seed is normally mostly used to generate random filesystem UUIDs.
This will need a bunch of documentation in a follow up since the use
cases are meant to be advanced but it can really speed up multi-type and
rebuilds of the same image.
Signed-off-by: Simon de Vlieger <supakeen@redhat.com>
This commit tweaks the handling of the `--output-name` option so
that is a name with the same extension as the image is passed that
is just silently ignored. Its a common issue that first time
users run:
```console
$ image-builder build --output-name foo.qcow2 qcow2
```
which currently leads to a foo.qcow2.qcow2. With this commit
the expected "foo.qcow2" will appear.
This commit adds the content of the output directory when a
build is finished. This is a convenient feature to make it easier
for the users.
Thanks to Simon for suggesting this!
This simplifies the way that ibcli determines if the manifest
needs a bootstrap stage or not. The logic is simple:
- when doing `image-builder manifest` we never include one
- when doing `image-builder build --arch` we add one if the
target arch and host arch differ
Instead of passing a callback this is now just a simple bool
option. Thanks to Achilleas for suggesting this.
Co-Authored-by: Achilleas Koutsou <achilleas@koutsou.net>
Add strict checking for toml keys in blueprints. This allows us
to error early if there are unknown keys in a toml blueprint and
helps our users by spotting e.g. typos early.
This is similar to
https://github.com/osbuild/bootc-image-builder/pull/549
(thanks Ondrej!).
