Upgrade and roll back an AOS host

AOS records two independent generation axes:

• an image generation owns an A/B root slot, UKI, kernel, initrd, base module library, evaluator, and expected measurements;
• a configuration generation owns the evaluated manifest, retained inputs, package projection, and EROFS /etc lower activated on the running image.

Image state is under /var/lib/profiles/image; configuration state is under /var/lib/profiles/system. Advancing one axis does not silently rewrite the other.

These paths become durable on measured-boot systems only after Secure Boot key enrollment and the first enforcing boot have created the TPM-sealed /var. Do not stage an image update while the machine is still using the disposable plaintext /var provided in UEFI Setup Mode.

Use Secure Boot and verify package trust explains why the image catalog, signed registry graph, immutable root, and package generations are distinct but connected verification steps.

#Prepare the rollout

The system registry must contain one package with the same name as the running sysroot, normally aos, with sysroot = true. Its signed metadata must publish the raw OTA payload, both slot-specific UKIs, module ABI, and the root-hash, expected-measurement, and Secure Boot facts required by the target policy. Recovery-enabled images additionally publish both recovery UKIs, their fixed loader entries, recovery ABI, and the exact authenticated bundle manifest.

Registry and channel policy establishes rollout direction. The upgrade resolver selects the first enabled same-name sysroot whose version differs from the running image; it does not infer semantic-version ordering.

Before changing a host:

1. restrict it to the intended registry and channel;
2. verify the image, UKIs, root hashes, and expected measurements in the signed catalog;
3. synchronize system metadata;
4. record both generation axes and the running slot;
5. preview the candidate.

apm update --system
apm image list
apm config rollback --list
apm image upgrade --dry-run

The dry run resolves the selected candidate and reports the plan without downloading, writing a root slot, changing the boot default, or activating configuration. Use apm switch --dry-run as documented in the configuration guide to preview a host.nix configuration transaction, including /etc, unit, closure, and provider-resolution changes.

#Stage an A/B image upgrade

apm image upgrade

APM verifies the registry graph and Secure Boot policy, imports the authenticated OTA payload, copies the currently needed evaluator closure to the persistent store overlay, writes the inactive root and, for a verity image, its hash slot, and stages the inactive normal UKI. A recovery-enabled update then publishes and read-back-verifies only the matching recovery UKI and its uncounted loader entry before making the counted normal UKI discoverable last. The recovery copy paired with the running known-good slot is not touched. The running image and active configuration are unchanged until reboot.

Before any inactive root write, APM re-authenticates the retained recovery UKI and every installed normal UKI against the immutable Secure Boot db snapshot in the immutable running image. Signed UKI identity, rather than editable image-state slot fields, determines each normal UKI's slot. Digest and generation records are supporting evidence and must agree; they are not signature authority.

Activation modes are:

Mode	Behavior
no mode flag	Stage the inactive slot and print a reboot advisory
--reboot	Stage, then request a full reboot
--drain	Drain workloads before a requested --reboot

Every A/B selection authenticates the running generation record against its immutable toplevel metadata, then compares that identity with the candidate. Both images must declare the same nonempty stateVersion and a canonical image-owned native executor store path. AOS currently has no state migration runner for an image transition, so a different stateVersion is rejected.

Changing the native executor is supported only with the combined --drain --reboot mode. Before draining or selecting the image, APM scans all retained configuration generations and rejects the transition if any native ability transaction lacks terminal evidence under the running executor. Advisory staging and --reboot without --drain cannot cross an executor boundary.

An A/B installation made before toplevels recorded these two fields has one authenticated migration path. The first transition must use --drain --reboot. APM requires both new immutable fields to be absent, requires the legacy image record to name no native executor, and first authenticates that it is the image that actually booted from the baked toplevel pointer, root slot, and verity hash. It then re-authenticates the record against the immutable toplevel, base library, UKI identity, and os-release. APM takes the legacy state version only from the authenticated AOS_STATE_VERSION in that os-release, requires the candidate to preserve it, checks every retained configuration generation for unfinished native transactions, and then records that state version durably before staging. It does not invent an executor identity for the legacy image. A partial new identity or any disagreement among the legacy record, boot identity, immutable metadata, and os-release is rejected.

For a qualified drained reboot, image state.json exposes the durable active_rollout record with schema aos.image-rollout/v1. It binds the exact candidate, prior image, and stateVersion; its status advances from staged to candidate_booted after early boot authenticates the selected image. Boot blessing then requires a strict successful configuration activation. If the manifest includes native ability activation, its committed activation proof must also name a nonempty native ability transaction.

A healthy candidate moves that same record to last_rollout with status succeeded and clears active_rollout. A strict health failure marks the active record health_failed and requests the next counted reboot. That status survives every remaining candidate attempt. After boot counting falls back, the prior image must authenticate, re-evaluate, and become authoritative before AOS moves the record to last_rollout as health_failed and clears the active transition. Operators can therefore distinguish a qualified success from an automatic fallback without treating an intermediate reboot as a final result.

The candidate UKI carries an sd-boot boot-counting suffix. Each unsuccessful attempt decrements its counter; exhaustion demotes the candidate and falls back to the other slot. A candidate is blessed only after it boots, re-evaluates the host configuration against its own ABI-pinned base library, commits a matching configuration generation, reaches the TPM ready phase, and passes local verification of the generation quote against the live PCR 7/11/12 values and the published image PCR 11. A failed ready transition leaves evaluation and boot blessing inactive.

On installations with redundant EFI System Partitions, the ESP selected by firmware is authoritative for boot assessment. AOS identifies it from the systemd-boot EFI variable, mounts it read-only, and refuses to bless a boot if that identity is unavailable or not configured. After a successful blessing, the stable bootloader, UKIs, loader configuration, and sealed credential are synchronized to every configured replica. This ordering prevents an undecremented copy of a failed counted entry from being promoted again.

For the initially installed image, the expected ready-phase value comes from a build-produced measurement sidecar signed by the PCR-policy key and bound to the exact UKI hash. AOS verifies it before importing it into durable image state. Later registry images obtain the same authority from their signed release catalog; a live PCR reading is never promoted into either source.

#Verify an image transition

After reboot:

cat /proc/cmdline
cat /etc/os-release
cat /var/lib/profiles/image/state.json
cat /var/lib/profiles/system/state.json
readlink /var/lib/profiles/system/current
cat /run/aos/activation.json

systemctl is-system-running
systemctl --failed
journalctl -b -p warning

Confirm that running names the expected image, pending has been cleared, the current configuration's image_gen_parent matches it, and the activation record describes the same committed transaction. Verify application health in addition to systemd state.

#Roll back configuration

List configuration generations and preview a target:

apm config rollback --list
apm config rollback --generation N --dry-run

Both --list and --dry-run authenticate retained manifests, current policy, provider routes, credential references, artifacts, and the running module ABI. The JSON form emits aos.retained-activatability/v1 reports with stable reason codes. A blocked dry run exits without activating or repairing state.

Apply it:

apm config rollback --generation N

Without --generation, APM chooses the most recent earlier configuration. When its module ABI matches the running image, rollback validates the retained manifest and switches directly. Across an ABI boundary, direct activation is refused: APM re-evaluates the retained host.nix, instance facts, and exact authenticated package module inputs against the running image, then commits a new compatible configuration generation. This replay requires no registry round trip because each generation retains its cfgsrc inputs.

#Roll back the image

List and preview the image axis separately:

apm image list
apm image rollback --generation N --dry-run

Image reports additionally authenticate the installed UKI, immutable running identity, state-format and native-executor boundary, and unfinished rollout ownership. The read-only probe does not perform the one-time legacy image-state migration; an accepted real rollback performs that migration later while it still holds the switch lock.

Select the older image as the durable next boot, optionally rebooting in the same operation:

apm image rollback --generation N
apm image rollback --generation N --reboot

The running kernel does not change until reboot. On the selected image's first boot, AOS re-evaluates the exact inputs retained by the active configuration generation against that image's base library. This preserves the machine's current intent instead of reverting to its original provisioning metadata. AOS commits the rebound configuration before making the image the durable successful default.

#Interpret configuration activation results

Checked activation publishes a transaction-bound record after the pointer and /etc swap. Its activation_exit field has this meaning; the outer apm command can still report a generic failure from provider orchestration:

activation_exit	State after the command	Operator action
0	New generation is live and healthy	Complete application checks
5	New generation is live; stale mount cleanup warned	Inspect mounts and schedule cleanup
6	New generation is committed but one or more units failed	Inspect failed units; roll back configuration if service is impaired
1-3	Failure occurred before the /etc swap	Previous generation remains live; inspect the reported phase
4	/etc swap or post-swap evidence is indeterminate	Use console access and the recovery procedure

Graph recovery does not treat a degraded or stale activation record as a completed transaction. Re-running the same transaction retries its package and activation work rather than silently skipping it.

#Install a selected sysroot

For controlled staging, select the registry and sysroot package explicitly:

apm image install aos --registry acme --dry-run
apm image install aos --registry acme --yes

This accepts exactly one package marked sysroot = true and stages its A/B image. For ordinary machine-wide packages, use apm install --system PACKAGE. See Manage packages.

#Keep scopes separate

/var/lib/profiles/image             A/B image generations
/var/lib/profiles/system            configuration generations
/var/lib/profiles/system   ordinary machine-wide package generations
/var/lib/profiles/per-user/$USER    user package generations

Rolling back an image does not pick an arbitrary old configuration; boot-time re-evaluation establishes a compatible one. Conversely, configuration or user package rollback does not replace the running kernel or root slot.

apm clean --generations --keep N prunes the invoking user's package profile. Add --system to prune both ordinary machine-wide package generations and configuration generations, keeping the latest N of each plus each profile's current generation. Configuration pruning is serialized with activation and releases its cfg/ and cfgsrc/ roots; a later apm gc --system can reclaim the now unreachable store paths. A/B image-generation pruning remains unavailable. aos gc --list-generations refers to an unrelated Nix profile.