---
title: "Incidents & freezes"
description: "The AUTH_test routing incident, the frozen daemon set, leftover-resource policy, and the standing hard boundaries."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.myswift.rs/llms.txt
> Use this file to discover all available pages before exploring further.

# Incidents & freezes

> **Historical evidence**
>
> Incidents are retained as immutable operational lessons. They are not current
> fleet status. Read [Current status](/status-2026-08-31) before acting on a PID,
> binary generation, daemon freeze, or rollback reference below.

Production credibility comes from what a project writes down about its
failures. These are Peregrine's, with their current containment state.

## 2026-08-14 — AUTH_test invalid-bucket routing

**What happened.** A historical S3 routing hole translated requests carrying
invalid bucket names into Swift-native requests against the internal
`AUTH_test` account. The account-reaper subsequently unlinked affected
`.data` files under that account. The damaged data was test data.

**Containment and fix.** The routing hole was closed the same day by the
`887be43` hotfix generation (invalid buckets rejected before any Swift
routing; empty-key writes answered honestly) and the fix was carried
forward — superseded, not reverted — by the Wave-2 → Size generations that
are dual-oracle verified live. The account itself is fenced by the
`account_freeze` middleware ([configuration](/configuration#account-freeze)),
verified S3-side by the 2026-08-14 `s3api-frozen-accounts` run.

**Standing decisions.**

- Forensic evidence stays: `/root/work/peregrine-auth-test-incident-20260814/`
  on the build node. No database resurrection, no disk carving, no
  provider snapshot — the data was test data; the *evidence* is what is
  preserved.
- The write-side forensic loop was **not** closed to a tidy conclusion and
  is recorded that way. Unfinished is written as unfinished.

## The frozen daemons — 12/13 as of 2026-08-17

During incident response, 13 consistency daemons across the fleet were set
`disabled + inactive` (`ENABLE=NONE`) to stop background mutation while the
data path stayed up.

The dedicated Wave-6 recovery (2026-08-17) re-enabled **11 of 13** across
all four nodes in staged batches — auditors first (read-only), then
replicators and the EC reconstructor, then updaters, sharder and
reconciler — canary node first, one full pass observed per unit, fleet
after. Result: zero quarantine growth, zero object loss, zero crash-loops,
disk flat, a ~55-minute quiet window to close. Evidence:
`/root/work/peregrine-daemon-unfreeze-20260817/` (swift3, 56 files).

`swift-object-expirer` was unfrozen the same evening after the
`AUTH_test/fsuite-10720/willexpire` queue row was deleted on all three
container replicas (object hashdir untouched). One unit stays frozen:

| Unit | Why it stays frozen |
|------|---------------------|
| `swift-account-reaper` | ring majority for `AUTH_test` still carries `DELETED`; `delay_reaping` was raised to 604800 on-node as defense in depth, but starting the reaper remains a user-gated decision |

Known steady state after recovery, recorded rather than hidden: 357
incident-era async pendings pointed at deleted `AUTH_test` containers retry
flat (correct Swift semantics, zero growth); four legacy test containers
oscillate their shard-cleave epoch with zero failures and are queued for a
reconciliation ticket.

## Leftover-resource policy

Scored runs create real buckets and objects; cleanup failures are tracked in
every report (`cleanup_failed`, 0 in the current runs). What remains is
kept deliberately:

| Resource | Policy |
|----------|--------|
| `worm-base-*` buckets | KEEP — WORM evidence |
| `peregrine-s3-<date>-<hex>` + `+segments` / `+versions` companions | KEEP — run companions; the primary bucket is often already deleted |
| Human/FUSE tenants (`AUTH_lab`, `s3://mytest`) | never touched by automation |
| Rollback binaries and prior generation build dirs | KEEP — they are the rollback path |

## Hard boundaries (all still in force)

1. Do not restart HAProxy or Keepalived; do not move the VIP.
2. Do not enable `swift-account-reaper` outside an explicitly authorized
   wave. The expirer is live; do not put `AUTH_test` entries back on its
   queue.
3. Do not touch `AUTH_lab`, `AUTH_test`, or `s3://mytest`.
4. Do not edit the dual-oracle runner's assertions or case set; do not
   downgrade Rust extras to 501 ([why](/dual-oracle#the-runner-is-part-of-the-contract)).
5. Do not delete evidence — reports, rollback binaries, incident trees,
   prior build dirs.
6. Do not format or wipe `/srv/node` devices, ever.

## Lab-only divergence disclosure

The Python oracle on the lab's swift3 node carries small, documented,
reversible s3api header patches (XML `Content-Type` on four GET paths and
404 header preservation) used to eliminate HEADER_NOISE from the strict
comparison. This is disclosed wherever scores are published: the oracle is
**2.33 + lab patches**, not pristine upstream, and the other nodes'
Python instances are unpatched.
[Details and rollback](/dual-oracle#lab-only-oracle-patches).

Source: https://docs.myswift.rs/incidents/index.mdx