Implement the base type of the new entity system, i.e. `Entity`,
for all relevant components:
- PartitionTable
- Partition
- Filesystem
Co-Authored-By: Achilleas Koutsou <achilleas@koutsou.net>
Add a generic entity data model that can be used to navigate and operate
on any specific disk image layout. Since the latter allows for arbitrary
nesting of different container types, such as LUKS2, LVM2, and with file
systems that can have sub-volumes, the entity model is very generic.
Co-Authored-By: Achilleas Koutsou <achilleas@koutsou.net>
Split up disk.go into smaller more manageable pieces: finish
by moving `Partition` to its own file.
Co-Authored-By: Christian Kellner <christian@kellner.me>
Split up disk.go into smaller more managable pieces: continue
with moving `Filesystem` to its own file.
Co-Authored-By: Christian Kellner <christian@kellner.me>
Split up disk.go into smaller more managable pieces: start with
extracting `PartitionTable` and all its methods to a new file.
Co-Authored-By: Christian Kellner <christian@kellner.me>
Use bytes internally everywhere and convert to sectors only when writing
the options for the stages.
Changed the AlignUp() method to not do the alignment if the input is
already aligned. This changes the behaviour when the size is 0, but
that's not a realistic use case. Updated unit tests to match.
Manifests are unaffected.
Co-Authored-By: Christian Kellner <christian@kellner.me>
Instead of just reserving a fixed amount of sectors for the
partition header and footer, calculate the size of the
header and footer dynamically when the layout is GPT.
Return an error if the maximum numbers of partitions has been
reached and thus creating further partitions would result in
errors.
Currently we limit MBR partition types to 4 as we dont support
logical partitions and GPT layouts to 128. According to the
UEFI specificatio (2.8) a minimum of 16384 bytes are reserved
for the partition entries array. Each entry is 128 bytes wide
thus resulting in at least 128 entries; we choose this to be
the maximum as well for now.
This is needed so that we can do different things depending on the
given layout; this will be used in tests for now only. Only GPT
allows for arbitrary number of partitions and once we assert this
in code we will need to adjust the tests accordingly.
NB: This method might be removed again in the future, once generic
LVM support is added everywhere and the ability to differentiate
between MBR and GPT layouts is not needed anymore.
Do not rely on `distro.imageOptions` having any size information,
i.e. `Size` being `0`. Instead use `imageType.Size()` and the
information in the blueprint customization to calculate the size.
This makes the individual distro definitions idenpendent of the
API entry points that currently calculate the size, e.g.:
internal/cloudapi/v1/v1.go:PostCompose line 184
internal/cloudapi/v2/v2.go:PostCompose line 197
internal/kojiapi/server.go:PostCompose line 135
internal/weldr/api.go:composeHandler line 2289
Align partitions during their re-layout. Specifically, we align the
first partition which removes the need for the hard coded magic
number of 2048, which represents the proper alignment for a grain
size of 1MB given a sector size of 512 (our setup).
NB: Since all our partitions should sizes that are themselves
aligned, i.e. evenly dividable by the default grain (1MB) this
will not actually change any manifest.
Add a utility method that can be used to align data to the next "grain"
which is currently defined as 1 MiB -- the default used by sfdisk.
Add corresponding tests.
Instead of hard coding a padding of 100 sectors for all layouts, i.e.
MBR and GTP, adjust the needed space depending on the layout: for MBR
we don't need to reserve any space at all since it does not have a
secondary header. For GTP we reserve 33 sectors, as indicated in the
UEFI specific, which allows for the header itself and up to 128 entries.
To not modify the layout of already released distributions, like RHEL
8.4 and 8.5, a new member called `ExtraPadding` is added to `Partition
Table` and then used in the corresponding layouts to preserve the
existing padding of 100.
The method assert.GreaterOrEqual(t, a, b) asserts that a >= b,
but the arguments were (expected, actual), which is the wrong
way around since we want to assure that the actual size is at
least the expected size, i.e. actual >= expected and thus the
argument should be (actual, expected).
The partition table size (pt.Size) is in sectors while our checks and
customizations are in bytes, so we need to convert when comparing.
Co-Authored-By: Achilleas Koutsou <achilleas@koutsou.net>
Apply the minimum size specified in the file system customizations
also to existing partitions. Specifically, this now allows to set
a minimum size also for the root partition.
Allow the sector size to be specified on the partition table level by
introducing a new `SectorSize` field. Modify the conversion helpers
to use that new field with a fallback to default sector size in case
the field is `0`.
Add simple helper methods that convert between bytes and sectors. This
is a method of `PartitionTable`, since the sector size can in theory
be per partition table and this prepares for that case.
Modify the signature of `CreatePartitionTable` so that it is
possible to return errors from the function. This is not yet
used, but will be in the near future. Change all call sites
accordingly: in most cases we can just bubble up the error.
Instead of generating the UUIDs directly when new partitions are
created and separately for the boot and root partition, use the
new `PartitionTable.GenerateUUIDs` method to generate all UUIDs
that are missing in one go. Since this changes the order in
which the uuids are generated the test manifests UUIDs changed
and needed to be updated:
I used to following patch to get the updated manifests:
--- a/internal/distro/distro_test_common/distro_test_common.go
+++ b/internal/distro/distro_test_common/distro_test_common.go
@@ -105,6 +105,12 @@ func TestDistro_Manifest(t *testing.T, pipelinePath string, prefix string, regis
require.NoError(t, err)
diff := cmp.Diff(expected, actual)
+ if diff != "" {
+ tt.Manifest = got
+ data, _ := json.MarshalIndent(tt, "", " ")
+ path := filepath.Join("/tmp", filepath.Base(fileName))
+ _ = ioutil.WriteFile(path, data, 0644)
+ }
require.Emptyf(t, diff, "Distro: %s\nArch: %s\nImage type: %s\nTest case file: %s\n", d.Name(), arch.Name(), imageType.Name(), fileName)
}
})
And the following fish snippet to update the existing ones, using the
jq and sponge utilities:
for file in /tmp/rhel_85-*.json
set filename (basename $file)
jq -s '.[0].manifest = .[1].manifest | .[0]' test/data/manifests/$filename /tmp/$filename | sponge test/data/manifests/$filename
end
Add a new helper to `PartitionTable` that generates all necessary
uuids for uuid fields that are empty but should not. These are
the file system uuids and, in case of a GPT layout, partition ids.
Pass the `basePartitionTable` argument of `CreatePartitionTable`.
Now that we clone the partition table at the beginning of the
method there is no need to pass a copy of the partition table.
Simple wrapper around FindFilesystemForMountpoint that will
return a boolean if a filesystem with the given mountpoint
is defined in the partition table.
The partition table is modified `CreatePartitionTable`, which is
not a problem for the table itself since it is currently passed
by value. However, all shallow copies share the same file system
pointers and `CreatePartitionTable` will modify those as well.
As there is a data race where two concurrent thread modify the
content of the Fileystem object at the same time before writing
the uuids out to the manifest via the stage options and thus the
resulting manifest would be broken.
Therefore we use the new `Clone` methods to make a dee@ copy of
the `PartitionTable` object in the `CreatePartitionTable` method;
This will allow us to pass the `basePartitionTable` object by val
and also return the resulting `PartitionTable` as a pointer.
The `PartitionTable` object contains an array of `Partitions`, which
is shared among all shallow copies of `PartitionTable` instances, as
when it is used via a value receiver in method calls. The individual
objects are thus not fully isolated since modifications of array
elements, or in some cases modifications to the array itself, are
shared and visible between all shallow `PartitionTable` copies. Thus
use a pointer receiver for all methods of `PartitionTable` to make
it explicit that modifications will affect other objects.
What this function is actually doing is to create a filesystem,
together with a new partition; for LVM that filesystem could
also be created on a new logical volume though.
The previous commit made sure that `PartitionTable.RootPartition`
is indeed returning the pointer partition object inside the array
of the `PartitionTable`. Thus changes to the returned object will
now directly affect the object and thus there is no need to call
`PartitionTable.updateRootPartition` anymore.
When we iterate over the partitions to find the root partition,
we are getting the partition by value so even if we return a
pointer to that then, it is actually a pointer duplicated object.
Return the pointer to the actual partition object instead.
In various places we are interested in the root and boot file-
systems. Currently those were accessed via by retrieving the
partition that contain them and the accessing the filesystem
member. Add accessors to `PartitionTable` that directly return
the needed filesystem. This will help if the file system is
stored inside a container like LVM or LUKS instead of directly
on a partition.
The dump function will be used to search packages and as we are implementing third party repositories, adding the repo_id of the package will allow us to identify the package repository used.
Defaults according to https://docs.aws.amazon.com/sdk-for-go/api/aws/#Config:
Defaults to a chain of credential providers to search for credentials in
environment variables, shared credential file, and EC2 Instance Roles.
If nothing is specified fall back to whatever instance role.
The longer running ostree tests (all except ostree.sh) benefit from
running on large runners because of their long runtime. Changing all to
large runners. Should not be a problem with our increased Openstack
quota.
The customizations struct contains different kinds of customizations,
such as users and filesystems. If both filesystems and users are
configured, the current code overwrites the user configuration.
Change this by unconditionally allocating the customizations struct
when creating the empty blueprint, and ammending it for each
customization we find.
This means that the customizations will be non-nil, also when it is
empty, which was not the case in the past. This should not change
the behavior of the code.
We (mistakenly) don't enforce a minimum size for /var,
so setting it to 1024 (1kB) causes the image build to fail.
CI does not expose this in a helpful way at the moment,
so this is a bit tricky to debug.
Also skip customizations for the AWS.S3 upload type. Not all the
image types with this upload type support filesystem customizations
and that's as expected. We could make a more fine-grained test in
the future, but testing with a coulpe of targets should be
sufficient.
