particle-os/debian-forge-composer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
6252 commits 1 branch 0 tags 688 MiB
Commit graph

6252 commits

This branch
This branch
All branches
Author SHA1 Message Date
Sanne Raymaekers
ab6fd9e53e cmd/osbuild-jobsite-builder: actually assign the stdout buffer 
The buffer needs to be assigned to the process' stdout for the buffer to
fill up.
 2024-02-15 18:28:07 +01:00
Simon de Vlieger
b9584099ab jobsite/manager: create export directory 
Be a bit more friendly and actually create the export directory instead
of assuming it exists.
 2024-02-15 16:23:55 +01:00
Sanne Raymaekers
1150f0f27e cmd/osbuild-jobsite: capture osbuild's stdout 
Write osbuild's stdout in the progress step. The manager can just copy
it to stdout and the executor will be able to parse the output into an
osbuild result.
 2024-02-15 14:31:24 +01:00
Sanne Raymaekers
ee6b198b0a cloud/awscloud: remove restricting egress rule from SG 
The machine still needs to be able to fetch sources, so just keep the
default 0.0.0.0/0 rule.
 2024-02-15 14:23:18 +01:00
Sanne Raymaekers
3400c5c20d templates/packer: setup vector in osbuild-executor 2024-02-15 12:05:17 +01:00
Sanne Raymaekers
a9010479e3 templates/packer: set -builder-path to /var/cache/osbuild-builder 
The builder uses `/run/osbuild` as a default path for this argument. Yet
this directory doesn't exist when the builder writes the manifest. But
osbuild should own this directory, not the builder.

Furthermore `/run` is a tmpfs, so the executor might run into memory
issues if we use `/run` as the store and output directory (on the "host"
workers these are in `/var/cache`).

While `/tmp` might seem like a good candidate on RHEL, it's a tmpfs on
Fedora, so it's also to be avoided.
 2024-02-15 01:02:10 +01:00
Sanne Raymaekers
04ce056b6a templates/packer: let the executor listen on all interfaces 2024-02-14 22:55:54 +01:00
Sanne Raymaekers
bfd1978ad8 osbuildexecutor/aws.ec2: pass the manifest to the job manager 2024-02-14 20:28:22 +01:00
Sanne Raymaekers
8e6717fa1b cloud/awscloud: take instance type from host 
InstanceRequirements is very flakey, the create fleet request fails
almost consistently with the same error.

To continue with testing use a fixed instance type for now. As a
followup we can expand the instance type selection logic or figure out
what was wrong with the InstanceRequirements.
 2024-02-14 18:15:25 +01:00
Sanne Raymaekers
8a1d66a0bd cloud/awscloud: max 4 overrides are allowed when creating a fleet 
```
InvalidParameterValue: Your request contains more than the maximum allowed number of InstanceRequirements (4)
```
 2024-02-14 15:24:42 +01:00
Sanne Raymaekers
f03dac5834 templates/packer: don't subscribe fedora executor 2024-02-14 13:45:52 +01:00
Sanne Raymaekers
7fd150b938 cloud/awscloud: specify subnets when creating secure instance 
For non-default VPCs, AWS needs the subnets it can launch the instance
in, otherwise it will try to launch the instance in the default VPC,
even if the supplied security groups are attached to a non-default VPC.

Furthermore there can only be 1 subnet specified per availability zone,
so query the subnets in the VPC of the host (as the instance needs to be
launched in the same network), and pick 1 of the VPC's subnets per AZ.
 2024-02-14 13:45:52 +01:00
Sanne Raymaekers
d5fd1bbbf0 templates/packer: add worker-executor service to image 2024-02-14 13:45:52 +01:00
Sanne Raymaekers
a9066ee793 templates/packer: remove worker_builder script from init service 
This script doesn't actually exist.
 2024-02-14 13:45:52 +01:00
Diaa Sami
f08d1f6068 composer: send error log messages to sentry 2024-02-14 10:38:42 +01:00
Sanne Raymaekers
d0caac9d69 templates/packer: use aws.ec2 osbuild executor 2024-02-14 09:54:11 +01:00
Sanne Raymaekers
4111ca4e0e templates/packer: add worker-executor service 2024-02-14 09:54:11 +01:00
Sanne Raymaekers
e7cadb16af osbuildexecutor: add aws.ec2 executor 
This executor spins up an instance which can only contact the host, and
uses the osbuild-jobsite manager & builder to invoke osbuild.
 2024-02-14 09:54:11 +01:00
Sanne Raymaekers
a2fb1bfc61 cloud/awscloud: add userdata to secure instance 
This way the `worker-initialization.service` knows to spin up the
builder instead of the worker.
 2024-02-14 09:54:11 +01:00
Sanne Raymaekers
3db88960c2 cloud/awscloud: add ability to run a secure instance to awscloud 
This instance can only contact the host, and requires this host to be
running on AWS itself with the appropriate IAM role.
 2024-02-14 09:54:11 +01:00
Sanne Raymaekers
05a45ed233 cloud/awscloud: add ec2metadata client 2024-02-14 09:54:11 +01:00
Sanne Raymaekers
e10424de2f osbuildexecutor: introduce osbuildexecutor.Executor interface 
Wrap the current osbuildexecutor.Executor in an interface so it's easier
to add different executors, which for instance can run osbuild in a VM.
 2024-02-14 09:54:11 +01:00
Sanne Raymaekers
9e85050633 osbuild-worker: add osbuild_executor config option 2024-02-14 09:54:11 +01:00
Diaa Sami
c9c51613a4 composer: glitchtip integration 2024-02-13 14:57:57 +01:00
Simon de Vlieger
95b4979d88 jobsite/builder: move process into builder 
Moves the background process into the builder struct in preparation for
using `sync.Once`.
 2024-02-12 12:03:38 +01:00
Simon de Vlieger
3cf2531ae2 jobsite: go-staticcheck appeasement 
Log and errors should not end with punctuation.
 2024-02-12 12:03:38 +01:00
Simon de Vlieger
7b7de50e23 jobsite/builder: enable --json 
Use `--json`; this causes `osbuild` to buffer all output and then print
a huge blob of JSON at the end of the command.
 2024-02-12 12:03:38 +01:00
Simon de Vlieger
748db5410e jobsite/manager: always retry on some errors 
On network layer errors we will retry the request until the timeout
expires.
 2024-02-12 12:03:38 +01:00
Simon de Vlieger
46ace22007 jobsite/manager: rename pipeline and export 
A better descriptive name externally is to use `export` and
`export-file`.
 2024-02-12 12:03:38 +01:00
Simon de Vlieger
42892754fd jobsite/builder: no caching for osbuild. 
As builders are not re-used caching has no use and can likely only
lead to disk-size issues. This can be re-enabled when we have an
outside-of-the-builder cache.
 2024-02-12 12:03:38 +01:00
Simon de Vlieger
b9b476c430 osbuild/builder: check deferred serve. 
Make sure to keep track of the deferred error for serve.
 2024-02-12 12:03:38 +01:00
Simon de Vlieger
323d3a1f9c jobsite/builder: rename. 
Consistency, the agent is now called builder.
 2024-02-12 12:03:38 +01:00
Simon de Vlieger
7548870e91 jobsite/builder: check deferred close. 
Check the deffered close by providing a name return value.
 2024-02-12 12:03:38 +01:00
Simon de Vlieger
c7788798c6 jobsite/builder: wrap HTTP handlers for errors. 
Instead of going directly fatal inside the handlers this commit provides
a wrapper function allowing handlers to instead return an `error`; this
will allow for checking deferred errors.
 2024-02-12 12:03:38 +01:00
Simon de Vlieger
401cb01c8d jobsite/manager: PathEscape URL parameter. 2024-02-12 12:03:38 +01:00
Simon de Vlieger
2a422321e5 rpm: add jobsite worker and manager 
This adds the `jobsite-worker` and `jobsite-manager` to the `-worker`
package.
 2024-02-12 12:03:38 +01:00
Simon de Vlieger
0bd66cf8e8 jobsite: manager and builder 
WIP
 2024-02-12 12:03:38 +01:00
schutzbot
b6b399e55e Post release version bump 
[skip ci]
 2024-02-07 08:22:32 +00:00
Xiaofeng Wang
fc81b89cde test: remove workarounds for fixed bugs 
Bugs have been fixed, workarounds are not needed anymore.
 2024-02-07 08:52:30 +01:00
dependabot[bot]
232bce95b5 build(deps): bump actions/upload-artifact from 3 to 4 
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3 to 4.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-02-06 12:28:59 +01:00
dependabot[bot]
7b5c4eb526 build(deps): bump actions/setup-go from 4 to 5 
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 4 to 5.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-02-06 12:26:58 +01:00
Tomáš Hozza
f6d35ac5ad Update osbuild/images to v0.35.0 
Signed-off-by: Tomáš Hozza <thozza@redhat.com>
 2024-02-06 12:20:08 +01:00
Sanne Raymaekers
44426bb48f templates/dashboards: add community stage service to orgs 2024-02-05 11:38:53 +01:00
Simon Steinbeiss
add35b86ce DEPLOYING/HACKING.md: Consistently use inline refs 
The footnote or endnote style links sometimes don't work well with
DocuSaurus, i.e. they're not resolved correctly and lead to build
errors.
https://github.com/osbuild/osbuild.github.io/actions/runs/7776075785/job/21202788034?pr=36
 2024-02-05 10:56:45 +01:00
Simon Steinbeiss
1d1b410654 HACKING/DEPLOYING.md: Markdown syntax fixes 2024-02-04 17:08:57 +01:00
Sanne Raymaekers
de548c36f3 pkg/jobqueue: fix worker status update query 
The workers table should be updated, not the heartbeats. Currently every
worker is reregisterig every minute.
 2024-02-02 15:24:57 +01:00
Achilleas Koutsou
3e31ebfbb1 osbuild-composer: add iot-bootable-container to weldr deny list 
We only want to build this image type in Fedora Koji.
 2024-02-02 10:34:41 +01:00
Achilleas Koutsou
bbbb3278f8 test: update fedora-39 test repositories 
Required for bootc for iot-bootable-container.
 2024-02-02 10:34:41 +01:00
Achilleas Koutsou
76a699653a test: add iot-bootable-container to api tests 
The image type supports uploading to both s3 and container registries.
Right now we only care about getting it in Koji, so let's just add it to
the s3 upload section.

The container registry test also embeds containers in the image, which
the iot-bootable-container doesn't support (yet).
 2024-02-02 10:34:41 +01:00
Achilleas Koutsou
52d4b196a3 cloudapi: enable iot-bootable-container image type 2024-02-02 10:34:41 +01:00