Performance

All numbers below are from a four-node Contabo HA lab against Python OpenStack Swift 2.35.0. Evidence pack: tools/test-results/contabo-deploy-20260801/ (write-path levers under perf-levers/).

Fairness redesign (2026-08-03): same-host SAIO A/Bs on swift1 (VIP MASTER + monitoring hub) are labeled NOISY / SUPERSEDED for formal performance conclusions. VIP-only cluster throughput is HA-PATH ONLY, not DIRECT-4PROXY. func 54/54 is CORE-PATH-ONLY. See docs/fairness-lab/ and tools/test-results/fairness-lab-20260803/REPORT.html.

The accept-loop latency fix

The swift-http accept loop set the listener non-blocking and slept 100 ms on WouldBlock, so every freshly-arriving connection waited up to ~100 ms to be accepted. Internal hops use Connection: close, so a GET paid it twice (proxy + object server). /healthcheck — which touches no backend — measured a flat 95 ms, proving the delay was the server’s own accept path. The fix: a 1 ms poll, plus TCP_NODELAY on the outbound backend sockets.

Rust vs Python (same-host SAIO A/B) — NOISY / SUPERSEDED

Tags: NOISY · SUPERSEDED. Both SAIOs ran on swift1 with VIP MASTER + Prometheus/console/cluster services. Useful as a directional signal only; not a formal iso-resource A/B. Formal performance requires the fairness-lab Performance mode (one implementation owns all 12 disks; DIRECT-4PROXY).

Dated evidence pack medians (perf-levers/SUMMARY.json → saio_phase0, 2026-08-02): ~3.79× @c32. Deep-verify re-check ~3.45× under the same hub noise. Pack includes workers=16 + lock-free accept; not an L1a delta.

Retired baseline (pre workers/accept work): 54 / 159 vs 32 / 255 — do not treat as current.

Production load (4-node, via HAProxy) — HA-PATH ONLY

Tag: HA-PATH. VIP / HAProxy entry. Not a DIRECT-4PROXY core-throughput conclusion (single ingress can cap aggregate bandwidth).

A 90-second mixed 80/20 run at 64 workers: 240,120 ops, 0 failures, 2,667 ops/s, 167 MiB/s, p50 9.6 ms / p99 127 ms. A autocos sweep (6 tasks) ran with 0 failures; reads reached 4,180 ops/s (4 KB), 1,346 MB/s (1 MB), and 1,920 MB/s (16 MB).

Clean-load re-measure (2026-08-02, client swift4 → VIP 10.0.0.10:8085, HA-PATH): 4KB write @128 ≈ 138 ops/s fail=0; 4KB read @128 ≈ 1055 ops/s fail=0; 1MB write/read @32 fail=0. 16MB read @8 remains ACCEPT_WITH_WARN (normal-stage success ~9% even after prepare 80/80) — see deep-verify-20260802/PERF-REMEASURE.md. Do not treat 16MB as cleared.

Write-concurrency optimization

A clean single-node A/B (an earlier multi-node 2-vs-32 try was a noisy load-host artifact and is not evidence) showed object-server workers = 2 starved write concurrency:

Shipped: worker floor 16, lock-free crossbeam MPMC accept queue (replacing Arc<Mutex<Receiver>>). Cluster 4 KB write p99 dropped from ~1017 ms to ~330 ms on that work.

Write-path lever A/B (2026-08-02 Contabo + SAIO)

Primary Contabo gate: shared load host swift1 → 10.0.0.1:8085 (noisy; second-order deltas limited).

Production knobs after the program (restored / kept):

workers = 16
container_update_mode = sync
container_update_timeout = 1.0
fsync_on_close = true
reuse_port = false

Updater concurrency = 16 / interval = 5 kept for pending drain on sync failures. Evidence: perf-levers/SUMMARY.json + per-lever COMPARE / DECISION JSON.