go.mod: update osbuild/images to v0.168.0

tag v0.165.0
Tagger: imagebuilder-bot <imagebuilder-bots+imagebuilder-bot@redhat.com>

Changes with 0.165.0

----------------
  *  distro: move rhel9 into a generic distro (osbuild/images#1645)
    * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger
  * Revert "distro: drop `ImageType.BasePartitionTable()`" (osbuild/images#1691)
    * Author: Michael Vogt, Reviewers: Simon de Vlieger, Tomáš Hozza
  * Update dependencies 2025-07-20 (osbuild/images#1675)
    * Author: SchutzBot, Reviewers: Achilleas Koutsou, Simon de Vlieger
  * defs: add missing `bootstrap_containers` (osbuild/images#1679)
    * Author: Michael Vogt, Reviewers: Simon de Vlieger, Tomáš Hozza
  * disk: handle adding `PReP` partition on PPC64/s390x (HMS-8884) (osbuild/images#1681)
    * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger
  * distro: bring per-distro checkOptions back (osbuild/images#1678)
    * Author: Michael Vogt, Reviewers: Simon de Vlieger, Tomáš Hozza
  * distro: cleanups in the pkg/distro/generic area (osbuild/images#1686)
    * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger
  * distro: move rhel8 into a generic distro (osbuild/images#1643)
    * Author: Michael Vogt, Reviewers: Nobody
  * distro: small followups for PR#1682 (osbuild/images#1689)
    * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger, Tomáš Hozza
  * distro: unify transform/match into a single concept (osbuild/images#1682)
    * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Tomáš Hozza
  * distros: de-duplicate runner build packages for centos10 (osbuild/images#1680)
    * Author: Michael Vogt, Reviewers: Simon de Vlieger, Tomáš Hozza
  * github: disable Go dep updates through dependabot (osbuild/images#1683)
    * Author: Achilleas Koutsou, Reviewers: Simon de Vlieger, Tomáš Hozza
  * repos: include almalinux 9.6 (osbuild/images#1677)
    * Author: Simon de Vlieger, Reviewers: Lukáš Zapletal, Tomáš Hozza
  * rhel9: wsl distribution config (osbuild/images#1694)
    * Author: Simon de Vlieger, Reviewers: Michael Vogt, Sanne Raymaekers
  * test/manifests/all-customizations: don't embed local file via URI (osbuild/images#1684)
    * Author: Tomáš Hozza, Reviewers: Achilleas Koutsou, Brian C. Lane

— Somewhere on the Internet, 2025-07-28

---

tag v0.166.0
Tagger: imagebuilder-bot <imagebuilder-bots+imagebuilder-bot@redhat.com>

Changes with 0.166.0

----------------
  * customizations/subscription: conditionally enable semanage call (HMS-8866) (osbuild/images#1673)
    * Author: Sanne Raymaekers, Reviewers: Achilleas Koutsou, Michael Vogt
  * distro/rhel-10: versionlock shim-x64 in the azure-cvm image  (osbuild/images#1697)
    * Author: Achilleas Koutsou, Reviewers: Michael Vogt, Simon de Vlieger
  * manifestmock: move container/pkg/commit mocks into helper (osbuild/images#1700)
    * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger
  * rhel9: `vagrant-libvirt`, `vagrant-virtualbox` (osbuild/images#1693)
    * Author: Simon de Vlieger, Reviewers: Michael Vogt, Sanne Raymaekers
  * rhel{9,10}: centos WSL refinement (HMS-8922) (osbuild/images#1690)
    * Author: Simon de Vlieger, Reviewers: Ondřej Budai, Sanne Raymaekers, Tomáš Hozza

— Somewhere on the Internet, 2025-07-29

---

tag v0.167.0
Tagger: imagebuilder-bot <imagebuilder-bots+imagebuilder-bot@redhat.com>

Changes with 0.167.0

----------------
  * RHEL/Azure: drop obsolete WAAgentConfig keys [RHEL-93894] and remove loglevel kernel option [RHEL-102372] (osbuild/images#1611)
    * Author: Achilleas Koutsou, Reviewers: Michael Vogt, Ondřej Budai, Sanne Raymaekers
  * Update dependencies 2025-07-27 (osbuild/images#1699)
    * Author: SchutzBot, Reviewers: Achilleas Koutsou, Simon de Vlieger
  * distro/rhel9: set default_kernel to kernel-uki-virt (osbuild/images#1704)
    * Author: Achilleas Koutsou, Reviewers: Ondřej Budai, Simon de Vlieger
  * distro: drop legacy loaders and update tests (osbuild/images#1687)
    * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Tomáš Hozza
  * distro: fix issues with yaml distro definitions and enable yaml checks (osbuild/images#1702)
    * Author: Achilleas Koutsou, Reviewers: Michael Vogt, Ondřej Budai, Simon de Vlieger

— Somewhere on the Internet, 2025-07-30

---

tag v0.168.0
Tagger: imagebuilder-bot <imagebuilder-bots+imagebuilder-bot@redhat.com>

Changes with 0.168.0

----------------
  * distro: fix bug in variable substitution for static distros (osbuild/images#1710)
    * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger
  * rhel{9,10}: azure for non-RHEL (HMS-8949) (osbuild/images#1707)
    * Author: Simon de Vlieger, Reviewers: Achilleas Koutsou, Michael Vogt

— Somewhere on the Internet, 2025-07-30

---
This commit is contained in:
Achilleas Koutsou 2025-07-30 15:46:51 +02:00
parent fad3b35d49
commit 6497b7520d
856 changed files with 72834 additions and 136836 deletions

View file

@ -0,0 +1,64 @@
package rawfilelock
import (
"os"
)
type LockType byte
const (
ReadLock LockType = iota
WriteLock
)
type FileHandle = fileHandle
// OpenLock opens a file for locking
// WARNING: This is the underlying file locking primitive of the OS;
// because closing FileHandle releases the lock, it is not suitable for use
// if there is any chance of two concurrent goroutines attempting to use the same lock.
// Most users should use the higher-level operations from internal/staging_lockfile or pkg/lockfile.
func OpenLock(path string, readOnly bool) (FileHandle, error) {
flags := os.O_CREATE
if readOnly {
flags |= os.O_RDONLY
} else {
flags |= os.O_RDWR
}
fd, err := openHandle(path, flags)
if err == nil {
return fd, nil
}
return fd, &os.PathError{Op: "open", Path: path, Err: err}
}
// TryLockFile attempts to lock a file handle
func TryLockFile(fd FileHandle, lockType LockType) error {
return lockHandle(fd, lockType, true)
}
// LockFile locks a file handle
func LockFile(fd FileHandle, lockType LockType) error {
return lockHandle(fd, lockType, false)
}
// UnlockAndClose unlocks and closes a file handle
func UnlockAndCloseHandle(fd FileHandle) {
unlockAndCloseHandle(fd)
}
// CloseHandle closes a file handle without unlocking
//
// WARNING: This is a last-resort function for error handling only!
// On Unix systems, closing a file descriptor automatically releases any locks,
// so "closing without unlocking" is impossible. This function will release
// the lock as a side effect of closing the file.
//
// This function should only be used in error paths where the lock state
// is already corrupted or when giving up on lock management entirely.
// Normal code should use UnlockAndCloseHandle instead.
func CloseHandle(fd FileHandle) {
closeHandle(fd)
}

View file

@ -0,0 +1,49 @@
//go:build !windows
package rawfilelock
import (
"time"
"golang.org/x/sys/unix"
)
type fileHandle uintptr
func openHandle(path string, mode int) (fileHandle, error) {
mode |= unix.O_CLOEXEC
fd, err := unix.Open(path, mode, 0o644)
return fileHandle(fd), err
}
func lockHandle(fd fileHandle, lType LockType, nonblocking bool) error {
fType := unix.F_RDLCK
if lType != ReadLock {
fType = unix.F_WRLCK
}
lk := unix.Flock_t{
Type: int16(fType),
Whence: int16(unix.SEEK_SET),
Start: 0,
Len: 0,
}
cmd := unix.F_SETLKW
if nonblocking {
cmd = unix.F_SETLK
}
for {
err := unix.FcntlFlock(uintptr(fd), cmd, &lk)
if err == nil || nonblocking {
return err
}
time.Sleep(10 * time.Millisecond)
}
}
func unlockAndCloseHandle(fd fileHandle) {
unix.Close(int(fd))
}
func closeHandle(fd fileHandle) {
unix.Close(int(fd))
}

View file

@ -0,0 +1,48 @@
//go:build windows
package rawfilelock
import (
"golang.org/x/sys/windows"
)
const (
reserved = 0
allBytes = ^uint32(0)
)
type fileHandle windows.Handle
func openHandle(path string, mode int) (fileHandle, error) {
mode |= windows.O_CLOEXEC
fd, err := windows.Open(path, mode, windows.S_IWRITE)
return fileHandle(fd), err
}
func lockHandle(fd fileHandle, lType LockType, nonblocking bool) error {
flags := 0
if lType != ReadLock {
flags = windows.LOCKFILE_EXCLUSIVE_LOCK
}
if nonblocking {
flags |= windows.LOCKFILE_FAIL_IMMEDIATELY
}
ol := new(windows.Overlapped)
if err := windows.LockFileEx(windows.Handle(fd), uint32(flags), reserved, allBytes, allBytes, ol); err != nil {
if nonblocking {
return err
}
panic(err)
}
return nil
}
func unlockAndCloseHandle(fd fileHandle) {
ol := new(windows.Overlapped)
windows.UnlockFileEx(windows.Handle(fd), reserved, allBytes, allBytes, ol)
closeHandle(fd)
}
func closeHandle(fd fileHandle) {
windows.Close(windows.Handle(fd))
}

View file

@ -0,0 +1,147 @@
package staging_lockfile
import (
"fmt"
"os"
"path/filepath"
"sync"
"github.com/containers/storage/internal/rawfilelock"
)
// StagingLockFile represents a file lock used to coordinate access to staging areas.
// Typical usage is via CreateAndLock or TryLockPath, both of which return a StagingLockFile
// that must eventually be released with UnlockAndDelete. This ensures that access
// to the staging file is properly synchronized both within and across processes.
//
// WARNING: This struct MUST NOT be created manually. Use the provided helper functions instead.
type StagingLockFile struct {
// Locking invariant: If stagingLockFileLock is not locked, a StagingLockFile for a particular
// path exists if the current process currently owns the lock for that file, and it is recorded in stagingLockFiles.
//
// The following fields can only be accessed by the goroutine owning the lock.
//
// An empty string in the file field means that the lock has been released and the StagingLockFile is no longer valid.
file string // Also the key in stagingLockFiles
fd rawfilelock.FileHandle
}
const maxRetries = 1000
var (
stagingLockFiles map[string]*StagingLockFile
stagingLockFileLock sync.Mutex
)
// tryAcquireLockForFile attempts to acquire a lock for the specified file path.
func tryAcquireLockForFile(path string) (*StagingLockFile, error) {
cleanPath, err := filepath.Abs(path)
if err != nil {
return nil, fmt.Errorf("ensuring that path %q is an absolute path: %w", path, err)
}
stagingLockFileLock.Lock()
defer stagingLockFileLock.Unlock()
if stagingLockFiles == nil {
stagingLockFiles = make(map[string]*StagingLockFile)
}
if _, ok := stagingLockFiles[cleanPath]; ok {
return nil, fmt.Errorf("lock %q is used already with other thread", cleanPath)
}
fd, err := rawfilelock.OpenLock(cleanPath, false)
if err != nil {
return nil, err
}
if err = rawfilelock.TryLockFile(fd, rawfilelock.WriteLock); err != nil {
// Lock acquisition failed, but holding stagingLockFileLock ensures
// no other goroutine in this process could have obtained a lock for this file,
// so closing it is still safe.
rawfilelock.CloseHandle(fd)
return nil, fmt.Errorf("failed to acquire lock on %q: %w", cleanPath, err)
}
lockFile := &StagingLockFile{
file: cleanPath,
fd: fd,
}
stagingLockFiles[cleanPath] = lockFile
return lockFile, nil
}
// UnlockAndDelete releases the lock, removes the associated file from the filesystem.
//
// WARNING: After this operation, the StagingLockFile becomes invalid for further use.
func (l *StagingLockFile) UnlockAndDelete() error {
stagingLockFileLock.Lock()
defer stagingLockFileLock.Unlock()
if l.file == "" {
// Panic when unlocking an unlocked lock. That's a violation
// of the lock semantics and will reveal such.
panic("calling Unlock on unlocked lock")
}
defer func() {
// Its important that this happens while we are still holding stagingLockFileLock, to ensure
// that no other goroutine has l.file open = that this close is not unlocking the lock under any
// other goroutine. (defer ordering is LIFO, so this will happen before we release the stagingLockFileLock)
rawfilelock.UnlockAndCloseHandle(l.fd)
delete(stagingLockFiles, l.file)
l.file = ""
}()
if err := os.Remove(l.file); err != nil && !os.IsNotExist(err) {
return err
}
return nil
}
// CreateAndLock creates a new temporary file in the specified directory with the given pattern,
// then creates and locks a StagingLockFile for it. The file is created using os.CreateTemp.
// Typically, the caller would use the returned lock file path to derive a path to the lock-controlled resource
// (e.g. by replacing the "pattern" part of the returned file name with a different prefix)
// Caller MUST call UnlockAndDelete() on the returned StagingLockFile to release the lock and delete the file.
//
// Returns:
// - The locked StagingLockFile
// - The name of created lock file
// - Any error that occurred during the process
//
// If the file cannot be locked, this function will retry up to maxRetries times before failing.
func CreateAndLock(dir string, pattern string) (*StagingLockFile, string, error) {
for try := 0; ; try++ {
file, err := os.CreateTemp(dir, pattern)
if err != nil {
return nil, "", err
}
file.Close()
path := file.Name()
l, err := tryAcquireLockForFile(path)
if err != nil {
if try < maxRetries {
continue // Retry if the lock cannot be acquired
}
return nil, "", fmt.Errorf(
"failed to allocate lock in %q after %d attempts; last failure on %q: %w",
dir, try, filepath.Base(path), err,
)
}
return l, filepath.Base(path), nil
}
}
// TryLockPath attempts to acquire a lock on an specific path. If the file does not exist,
// it will be created.
//
// Warning: If acquiring a lock is successful, it returns a new StagingLockFile
// instance for the file. Caller MUST call UnlockAndDelete() on the returned StagingLockFile
// to release the lock and delete the file.
func TryLockPath(path string) (*StagingLockFile, error) {
return tryAcquireLockForFile(path)
}

View file

@ -0,0 +1,243 @@
package tempdir
import (
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"github.com/containers/storage/internal/staging_lockfile"
"github.com/sirupsen/logrus"
)
/*
Locking rules and invariants for TempDir and its recovery mechanism:
1. TempDir Instance Locks:
- Path: 'RootDir/lock-XYZ' (in the root directory)
- Each TempDir instance creates and holds an exclusive lock on this file immediately
during NewTempDir() initialization.
- This lock signifies that the temporary directory is in active use by the
process/goroutine that holds the TempDir object.
2. Stale Directory Recovery (separate operation):
- RecoverStaleDirs() can be called independently to identify and clean up stale
temporary directories.
- For each potential stale directory (found by listPotentialStaleDirs), it
attempts to TryLockPath() its instance lock file.
- If TryLockPath() succeeds: The directory is considered stale, and both the
directory and lock file are removed.
- If TryLockPath() fails: The directory is considered in active use by another
process/goroutine, and it's skipped.
3. TempDir Usage:
- NewTempDir() immediately creates both the instance lock and the temporary directory.
- TempDir.StageDeletion() moves files into the existing temporary directory with counter-based naming.
- Files moved into the temporary directory are renamed with a counter-based prefix
to ensure uniqueness (e.g., "0-filename", "1-filename").
- Once cleaned up, the TempDir instance cannot be reused - StageDeletion() will return an error.
4. Cleanup Process:
- TempDir.Cleanup() removes both the temporary directory and its lock file.
- The instance lock is unlocked and deleted after cleanup operations are complete.
- The TempDir instance becomes inactive after cleanup (internal fields are reset).
- The TempDir instance cannot be reused after Cleanup() - StageDeletion() will fail.
5. TempDir Lifetime:
- NewTempDir() creates both the TempDir manager and the actual temporary directory immediately.
- The temporary directory is created eagerly during NewTempDir().
- During its lifetime, the temporary directory is protected by its instance lock.
- The temporary directory exists until Cleanup() is called, which removes both
the directory and its lock file.
- Multiple TempDir instances can coexist in the same RootDir, each with its own
unique subdirectory and lock.
- After cleanup, the TempDir instance cannot be reused.
6. Example Directory Structure:
RootDir/
lock-ABC (instance lock for temp-dir-ABC)
temp-dir-ABC/
0-file1
1-file3
lock-XYZ (instance lock for temp-dir-XYZ)
temp-dir-XYZ/
0-file2
*/
const (
// tempDirPrefix is the prefix used for creating temporary directories.
tempDirPrefix = "temp-dir-"
// tempdirLockPrefix is the prefix used for creating lock files for temporary directories.
tempdirLockPrefix = "lock-"
)
// TempDir represents a temporary directory that is created in a specified root directory.
// It manages the lifecycle of the temporary directory, including creation, locking, and cleanup.
// Each TempDir instance is associated with a unique subdirectory in the root directory.
// Warning: The TempDir instance should be used in a single goroutine.
type TempDir struct {
RootDir string
tempDirPath string
// tempDirLock is a lock file (e.g., RootDir/lock-XYZ) specific to this
// TempDir instance, indicating it's in active use.
tempDirLock *staging_lockfile.StagingLockFile
tempDirLockPath string
// counter is used to generate unique filenames for added files.
counter uint64
}
// CleanupTempDirFunc is a function type that can be returned by operations
// which need to perform cleanup actions later.
type CleanupTempDirFunc func() error
// listPotentialStaleDirs scans the RootDir for directories that might be stale temporary directories.
// It identifies directories with the tempDirPrefix and their corresponding lock files with the tempdirLockPrefix.
// The function returns a map of IDs that correspond to both directories and lock files found.
// These IDs are extracted from the filenames by removing their respective prefixes.
func listPotentialStaleDirs(rootDir string) (map[string]struct{}, error) {
ids := make(map[string]struct{})
dirContent, err := os.ReadDir(rootDir)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, fmt.Errorf("error reading temp dir %s: %w", rootDir, err)
}
for _, entry := range dirContent {
if id, ok := strings.CutPrefix(entry.Name(), tempDirPrefix); ok {
ids[id] = struct{}{}
continue
}
if id, ok := strings.CutPrefix(entry.Name(), tempdirLockPrefix); ok {
ids[id] = struct{}{}
}
}
return ids, nil
}
// RecoverStaleDirs identifies and removes stale temporary directories in the root directory.
// A directory is considered stale if its lock file can be acquired (indicating no active use).
// The function attempts to remove both the directory and its lock file.
// If a directory's lock cannot be acquired, it is considered in use and is skipped.
func RecoverStaleDirs(rootDir string) error {
potentialStaleDirs, err := listPotentialStaleDirs(rootDir)
if err != nil {
return fmt.Errorf("error listing potential stale temp dirs in %s: %w", rootDir, err)
}
if len(potentialStaleDirs) == 0 {
return nil
}
var recoveryErrors []error
for id := range potentialStaleDirs {
lockPath := filepath.Join(rootDir, tempdirLockPrefix+id)
tempDirPath := filepath.Join(rootDir, tempDirPrefix+id)
// Try to lock the lock file. If it can be locked, the directory is stale.
instanceLock, err := staging_lockfile.TryLockPath(lockPath)
if err != nil {
continue
}
if rmErr := os.RemoveAll(tempDirPath); rmErr != nil && !os.IsNotExist(rmErr) {
recoveryErrors = append(recoveryErrors, fmt.Errorf("error removing stale temp dir %s: %w", tempDirPath, rmErr))
}
if unlockErr := instanceLock.UnlockAndDelete(); unlockErr != nil {
recoveryErrors = append(recoveryErrors, fmt.Errorf("error unlocking and deleting stale lock file %s: %w", lockPath, unlockErr))
}
}
return errors.Join(recoveryErrors...)
}
// NewTempDir creates a TempDir and immediately creates both the temporary directory
// and its corresponding lock file in the specified RootDir.
// The RootDir itself will be created if it doesn't exist.
// Note: The caller MUST ensure that returned TempDir instance is cleaned up with .Cleanup().
func NewTempDir(rootDir string) (*TempDir, error) {
if err := os.MkdirAll(rootDir, 0o700); err != nil {
return nil, fmt.Errorf("creating root temp directory %s failed: %w", rootDir, err)
}
td := &TempDir{
RootDir: rootDir,
}
tempDirLock, tempDirLockFileName, err := staging_lockfile.CreateAndLock(td.RootDir, tempdirLockPrefix)
if err != nil {
return nil, fmt.Errorf("creating and locking temp dir instance lock in %s failed: %w", td.RootDir, err)
}
td.tempDirLock = tempDirLock
td.tempDirLockPath = filepath.Join(td.RootDir, tempDirLockFileName)
// Create the temporary directory that corresponds to the lock file
id := strings.TrimPrefix(tempDirLockFileName, tempdirLockPrefix)
actualTempDirPath := filepath.Join(td.RootDir, tempDirPrefix+id)
if err := os.MkdirAll(actualTempDirPath, 0o700); err != nil {
return nil, fmt.Errorf("creating temp directory %s failed: %w", actualTempDirPath, err)
}
td.tempDirPath = actualTempDirPath
td.counter = 0
return td, nil
}
// StageDeletion moves the specified file into the instance's temporary directory.
// The temporary directory must already exist (created during NewTempDir).
// Files are renamed with a counter-based prefix (e.g., "0-filename", "1-filename") to ensure uniqueness.
// Note: 'path' must be on the same filesystem as the TempDir for os.Rename to work.
// The caller MUST ensure .Cleanup() is called.
// If the TempDir has been cleaned up, this method will return an error.
func (td *TempDir) StageDeletion(path string) error {
if td.tempDirLock == nil {
return fmt.Errorf("temp dir instance not initialized or already cleaned up")
}
fileName := fmt.Sprintf("%d-", td.counter) + filepath.Base(path)
destPath := filepath.Join(td.tempDirPath, fileName)
td.counter++
return os.Rename(path, destPath)
}
// Cleanup removes the temporary directory and releases its instance lock.
// After cleanup, the TempDir instance becomes inactive and cannot be reused.
// Subsequent calls to StageDeletion() will fail.
// Multiple calls to Cleanup() are safe and will not return an error.
// Callers should typically defer Cleanup() to run after any application-level
// global locks are released to avoid holding those locks during potentially
// slow disk I/O.
func (td *TempDir) Cleanup() error {
if td.tempDirLock == nil {
logrus.Debug("Temp dir already cleaned up")
return nil
}
if err := os.RemoveAll(td.tempDirPath); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("removing temp dir %s failed: %w", td.tempDirPath, err)
}
lock := td.tempDirLock
td.tempDirPath = ""
td.tempDirLock = nil
td.tempDirLockPath = ""
return lock.UnlockAndDelete()
}
// CleanupTemporaryDirectories cleans up multiple temporary directories by calling their cleanup functions.
func CleanupTemporaryDirectories(cleanFuncs ...CleanupTempDirFunc) error {
var cleanupErrors []error
for _, cleanupFunc := range cleanFuncs {
if cleanupFunc == nil {
continue
}
if err := cleanupFunc(); err != nil {
cleanupErrors = append(cleanupErrors, err)
}
}
return errors.Join(cleanupErrors...)
}