Incidents & freezes

The AUTH_test routing incident, the frozen daemon set, leftover-resource policy, and the standing hard boundaries.

Production credibility comes from what a project writes down about its failures. These are Peregrine’s, with their current containment state.

2026-08-14 — AUTH_test invalid-bucket routing

What happened. A historical S3 routing hole translated requests carrying invalid bucket names into Swift-native requests against the internal AUTH_test account. The account-reaper subsequently unlinked affected .data files under that account. The damaged data was test data.

Containment and fix. The routing hole was closed the same day by the 887be43 hotfix generation (invalid buckets rejected before any Swift routing; empty-key writes answered honestly) and the fix was carried forward — superseded, not reverted — by the Wave-2 → Size generations that are dual-oracle verified live. The account itself is fenced by the account_freeze middleware (configuration), verified S3-side by the 2026-08-14 s3api-frozen-accounts run.

Standing decisions.

Forensic evidence stays: /root/work/peregrine-auth-test-incident-20260814/ on the build node. No database resurrection, no disk carving, no provider snapshot — the data was test data; the evidence is what is preserved.
The write-side forensic loop was not closed to a tidy conclusion and is recorded that way. Unfinished is written as unfinished.

The frozen daemons — 12/13 as of 2026-08-17

During incident response, 13 consistency daemons across the fleet were set disabled + inactive (ENABLE=NONE) to stop background mutation while the data path stayed up.

The dedicated Wave-6 recovery (2026-08-17) re-enabled 11 of 13 across all four nodes in staged batches — auditors first (read-only), then replicators and the EC reconstructor, then updaters, sharder and reconciler — canary node first, one full pass observed per unit, fleet after. Result: zero quarantine growth, zero object loss, zero crash-loops, disk flat, a ~55-minute quiet window to close. Evidence: /root/work/peregrine-daemon-unfreeze-20260817/ (swift3, 56 files).

swift-object-expirer was unfrozen the same evening after the AUTH_test/fsuite-10720/willexpire queue row was deleted on all three container replicas (object hashdir untouched). One unit stays frozen:

Unit	Why it stays frozen
`swift-account-reaper`	ring majority for `AUTH_test` still carries `DELETED`; `delay_reaping` was raised to 604800 on-node as defense in depth, but starting the reaper remains a user-gated decision

Known steady state after recovery, recorded rather than hidden: 357 incident-era async pendings pointed at deleted AUTH_test containers retry flat (correct Swift semantics, zero growth); four legacy test containers oscillate their shard-cleave epoch with zero failures and are queued for a reconciliation ticket.

Leftover-resource policy

Scored runs create real buckets and objects; cleanup failures are tracked in every report (cleanup_failed, 0 in the current runs). What remains is kept deliberately:

Resource	Policy
`worm-base-*` buckets	KEEP — WORM evidence
`peregrine-s3-<date>-<hex>` + `+segments` / `+versions` companions	KEEP — run companions; the primary bucket is often already deleted
Human/FUSE tenants (`AUTH_lab`, `s3://mytest`)	never touched by automation
Rollback binaries and prior generation build dirs	KEEP — they are the rollback path

Hard boundaries (all still in force)

Do not restart HAProxy or Keepalived; do not move the VIP.
Do not enable swift-account-reaper outside an explicitly authorized wave. The expirer is live; do not put AUTH_test entries back on its queue.
Do not touch AUTH_lab, AUTH_test, or s3://mytest.
Do not edit the dual-oracle runner’s assertions or case set; do not downgrade Rust extras to 501 (why).
Do not delete evidence — reports, rollback binaries, incident trees, prior build dirs.
Do not format or wipe /srv/node devices, ever.

Lab-only divergence disclosure

The Python oracle on the lab’s swift3 node carries small, documented, reversible s3api header patches (XML Content-Type on four GET paths and 404 header preservation) used to eliminate HEADER_NOISE from the strict comparison. This is disclosed wherever scores are published: the oracle is 2.33 + lab patches, not pristine upstream, and the other nodes’ Python instances are unpatched. Details and rollback.