Operations

Historical fleet state — 2026-08-18

As of 2026-08-18: proxies 4/4 + VIP run the Guard generation (3d0138d5…), object servers are the 2026-08-18 BUG-1 build (471e8b73…), drain 0, VIP on swift2. Consistency daemons are 12/13 enabled + active — expirer was unfrozen 2026-08-17 after AUTH_test queue surgery; swift-account-reaper stays frozen (why). S3 dual-oracle stands at 49/8, not GREEN (post-GUARD fail set identical to post-GATE). Stance IMPLEMENTED_SUBSET_ONLY. Account-root 405 and version-index CAS are live. Object BUG-1 (tagging/restore-only POST on a locked object) is live. worm_clock_max_offset_ms stays 0. Binary genealogy: Releases · upgrade/rollback recipes: Runbooks · scoring: Dual-oracle.

Earlier cold/restore closeout (LocalDir, not tape/Glacier, default off): Cold + Contabo S3 delivery.

Alerting & DR (2026-08-17)

• Alerting: Prometheus rules are live on swift1 and aligned to the 12/13 daemon set (FrozenDaemonStarted watches only the still-frozen reaper). Alertmanager + a local JSONL sink (/var/log/peregrine-alerts.jsonl) are running on loopback; a Feishu / human receiver is a documented swap-in, not yet connected. Rules are versioned in tools/monitoring/.
• DR: weekly cron is installed on all four nodes (Mondays 03:05–03:20 UTC, 35-day retention). Config-plane bundles stay off git; encrypted offsite copies of the 2026-08-17 set live on Drive with the passphrase only in the operator Keychain. Procedure: tools/dr/DR-RESTORE.md. Disk posture after the 2026-08-17 cleanup: all four root disks ≤ 76%, journald capped at 500 MB fleet-wide.

Health at a glance

A cluster health check should report services, the load balancer, rings, capacity, replication passes, accounting, and data integrity — and print UNKNOWN for anything it cannot determine rather than scoring a failed probe as fine. The reference health.sh folds a per-node service census, HAProxy backend state, ring device counts, per-device usage, the latest replicator / reconstructor / updater pass, the account rollup vs container truth, and a read-back integrity check into one screen.

Lab topology, EC binary requirements, and subscription cutover rules: Lab cluster.

Replication & reconstruction

• Object replicator — suffix-hash push, handoff revert, rsync / rsync-over-ssh.
• Reconstructor (EC) — rebuilds this node’s missing fragment: gathers ndata coherent peer archives, reconstructs at the node’s fragment index, and persists it durably.

EC self-heal

In the historical 2026-08-18 fleet drill, deleting fragment #2 of a live object left the degraded read serving correct data (2+1 tolerates one loss), and the reconstructor rebuilt the exact fragment on its owner node within ≤15 s, with all three fragments back at the original timestamp. This dated canary is not a current G6 or G7 acceptance result.

Node-down HA drill

Drive a real failover from the swift-console Lab:

POST /lab/api/node/down   {"node":"swift2","ttl_secs":300}

POST /lab/api/node/up     {"node":"swift2"}

Historical 2026-08-18 result: with one node fully down, replication and erasure coding each served 100% of that drill’s reads and writes, with EC writes surviving via handoffs. This is a bounded dated result, not a current G7 pass or an availability SLO.