---
title: "Operations"
description: "Health, replication and reconstruction, the node-down HA drill, and EC self-heal."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.myswift.rs/llms.txt
> Use this file to discover all available pages before exploring further.

# Operations

> **Dated fleet record**
>
> The fleet paragraph below records the 2026-08-18 Guard deployment. It is not
> the current candidate state and does not authorize a rollout. The `17adf0b`
> line is G6 GREEN but remains isolated from production; G7 is NOT
> ACCEPTED. Read
> [Current status](/status-2026-08-31) first.

## Historical fleet state — 2026-08-18

As of 2026-08-18: proxies 4/4 + VIP run the **Guard** generation
(`3d0138d5…`), object servers are the 2026-08-18 BUG-1 build
(`471e8b73…`), drain 0, VIP on
swift2. Consistency daemons are **12/13 enabled + active** — expirer was
unfrozen 2026-08-17 after `AUTH_test` queue surgery;
`swift-account-reaper` stays frozen
([why](/incidents#the-frozen-daemons--1213-as-of-2026-08-17)). S3
dual-oracle stands at **49/8, not GREEN** (post-GUARD fail set identical to
post-GATE). Stance **IMPLEMENTED_SUBSET_ONLY**. Account-root 405 and
version-index CAS are live. Object BUG-1 (tagging/restore-only POST on a
locked object) is live. `worm_clock_max_offset_ms` stays 0.
Binary genealogy: [Releases](/releases) · upgrade/rollback recipes:
[Runbooks](/runbooks) · scoring: [Dual-oracle](/dual-oracle).

Earlier cold/restore closeout (LocalDir, not tape/Glacier, default off):
[Cold + Contabo S3 delivery](/cold-contabo-delivery).

## Alerting & DR (2026-08-17)

- **Alerting**: Prometheus rules are live on swift1 and aligned to the
  12/13 daemon set (`FrozenDaemonStarted` watches only the still-frozen
  reaper). Alertmanager + a local JSONL sink
  (`/var/log/peregrine-alerts.jsonl`) are running on loopback; a Feishu /
  human receiver is a documented swap-in, not yet connected. Rules are
  versioned in `tools/monitoring/`.
- **DR**: weekly cron is installed on all four nodes (Mondays 03:05–03:20
  UTC, 35-day retention). Config-plane bundles stay off git; encrypted
  offsite copies of the 2026-08-17 set live on Drive with the passphrase
  only in the operator Keychain. Procedure: `tools/dr/DR-RESTORE.md`.
  Disk posture after the 2026-08-17 cleanup: all four root disks ≤ 76%,
  journald capped at 500 MB fleet-wide.

## Health at a glance

A cluster health check should report services, the load balancer, rings,
capacity, replication passes, accounting, and data integrity — and print
`UNKNOWN` for anything it cannot determine rather than scoring a failed probe as
fine. The reference `health.sh` folds a per-node service census, HAProxy backend
state, ring device counts, per-device usage, the latest replicator /
reconstructor / updater pass, the account rollup vs container truth, and a
read-back integrity check into one screen.

Lab topology, EC binary requirements, and subscription cutover rules:
[Lab cluster](/lab-cluster).

## Replication & reconstruction

> **Daemon state (2026-08-17)**
>
> The consistency plane is live again: auditors, replicators, the EC
> reconstructor, updaters, sharder, reconciler, and (after queue surgery)
> the object expirer. Only `swift-account-reaper` remains frozen — see
> [Incidents & freezes](/incidents#the-frozen-daemons--1213-as-of-2026-08-17).

- **Object replicator** — suffix-hash push, handoff revert, rsync /
  rsync-over-ssh.
- **Reconstructor** (EC) — rebuilds this node's missing fragment: gathers
  `ndata` coherent peer archives, reconstructs at the node's fragment index, and
  persists it durably.

> **Reconciling mixed EC state by hand**
>
> A deleted EC object can leave stale data fragments alongside newer tombstones on
> different devices, a state the reconstructor may loop on. Do not delete a
> fragment as a first response. Capture timestamps, policy, partition, fragment
> indexes, durable markers, tombstones, and peer state; establish the authoritative
> timestamp; then follow the evidence-preserving [recovery procedure](/recovery).

## EC self-heal

In the historical 2026-08-18 fleet drill, deleting fragment `#2` of a live
object left the degraded read serving correct data
(2+1 tolerates one loss), and the reconstructor **rebuilt the exact fragment on
its owner node within ≤15 s**, with all three fragments back at the original
timestamp. This dated canary is not a current G6 or G7 acceptance result.

## Node-down HA drill

Drive a real failover from the [swift-console](/swift-console) Lab:

1. **Step 1**

   Take a node down (journaled, auto-restarts on TTL).

```sh
POST /lab/api/node/down   {"node":"swift2","ttl_secs":300}
```
2. **Step 2**

   Wait for the load balancer to detect the dead proxy, then drive load on the
   survivors.
3. **Step 3**

   Bring it back.

```sh
POST /lab/api/node/up     {"node":"swift2"}
```

**Historical 2026-08-18 result:** with one node fully down, replication and
erasure coding each served 100% of that drill's reads and writes, with EC writes
surviving via handoffs. This is a bounded dated result, not a current G7 pass or
an availability SLO.

> **Tokens on node-down**
>
> tempauth tokens cached on a downed node's memcached go invalid, so a client
> re-authenticates on `401` (standard Swift). The re-auth during the outage
> succeeds — the memcache client does not hang on the down server.

Source: https://docs.myswift.rs/operations/index.mdx