Testing & verification

Peregrine is verified in ordered layers. A change is only accepted when the required layer is green and its provenance is recorded; a unit result, a single-node SAIO result, or a partial canary cannot close a later gate.

Gate order and exact accounting

G0 provenance → G1 Linux environment → G2 reproducible build → G3 route-level async proof → G4 official Swift functional parity → G5 official S3 parity → G6 storage/probe semantics → G7 concurrency and fault behavior → G8 performance and soak. The accepted G6 record has two explicit formal lanes: 147 replication identities in W068 and 32 EC identities in W069. W070 merges them against the frozen oracle exactly once; the replication lane is never promoted by itself. This closes G6 and opens G7, but does not close G7.

0. CI — every push, the full suite

Since 2026-08-17 (.github/workflows/ci.yml) every push and pull request runs the entire workspace test suite plus the EC slice on GitHub Actions, with liberasurecode pinned ≥ 1.6.5 (why). cargo fmt and cargo clippy run as report boards over the known pre-existing debt. This gate exists because filtered, manual test runs let a broken test target rot unnoticed for two weeks — that class of debt is now structurally closed.

1. Unit + golden (format fidelity)

Every serialized format has a golden round-trip: the real Python Swift writes a fixture, swift-rust must read it and reproduce it byte-for-byte, then Rust writes and Python reads back. The workspace runs 1 400+ tests across 83 targets and grows with every fault-injection wave (2026-08-17 baseline: 1 422 passed, zero failures; the WORM and version-CAS waves added ~60 more the same day).

cargo test --workspace --exclude swift-ec
cargo test -p swift-ec -p swift-object-server -p swift-proxy-server \
  --features swift-proxy-server/ec,swift-object-server/ec
cargo clippy --workspace --exclude swift-ec --all-targets -- -D warnings

2. Functional parity vs the Python oracle

A single parameterized suite runs against both a Rust stack and a Python Swift SAIO, so the two PASS/FAIL tables are a direct parity comparison: CRUD, byte ranges (single, suffix, multipart), swob conditionals, SLO/DLO, erasure coding, copy, expiry, ACLs, cross-account denial, and negative cases.

3. Performance A/B

A concurrent driver runs a PUT/GET matrix (object size × concurrency) and reports throughput (ops/s, MB/s) and latency percentiles. Run it against a Rust SAIO and a Python SAIO on the same host for an implementation A/B, and against the cluster for production numbers. cosbench-rs + autocos provide COSBench-grade load for larger runs.

4. Production, on the HA cluster

Against the four-node Contabo HA lab (VIP 10.0.0.10:8085 — see Lab cluster for topology and cutover rules):

5. Strict S3 dual-oracle

The S3 surface has its own scored gate: the frozen 57-case runner sends identical requests to the live Rust fleet and a stock Python Swift 2.33 oracle and compares what a client sees. Methodology, the GREEN contract, and the historical 49/8 scoreboard: Dual-oracle verification.

Recent delivery — cold + Contabo S3

Lab LocalDir cold + restore + archive-on-transition units GREEN; Contabo S3 extended matrix 29/29 GREEN (bounded); canary runbook only. See Cold + Contabo S3 delivery (IMPLEMENTED_SUBSET_ONLY).

The harnesses

swift-rust/tools/ ships every harness used above, each runnable against any endpoint: