Lab cluster

The development lab is a four-node Contabo HA cluster used to prove the Rust Swift family end-to-end — the same environment described under Testing §4. The previous Azure PAYG topology (10.42.* + ILB) is retired. Full cutover notes live in the monorepo under tools/CONTABO-CLUSTER.md.

Fairness lab (2026-08-03): Contabo is a hybrid on swift-deploy-rs, not full ansible parity. SAIO on swift1 is NOISY for performance; VIP path is HA-PATH ONLY; formal throughput requires Performance mode + DIRECT-4PROXY. See monorepo docs/fairness-lab/ and tools/test-results/fairness-lab-20260803/REPORT.html.

Topology

Plane	Address	Role
Management / SSH	public `169.58.108.{85,86,87,121}`	SSH, deploy
Proxy / client	`10.0.0.1–4` + Keepalived VIP `10.0.0.10:8085`	HAProxy → proxy `:8080`
Storage	`10.0.4.1–4`	account / container / object data path
Replication	`10.0.8.1–4`	replicator / reconstructor

Nodes: swift1–swift4. Keepalived VIP 10.0.0.10 is currently on swift2 (eth1 secondary). Prefer the real VIP https://10.0.0.10:8085 for clients and gates. Per-node http://10.0.0.N:8085 remains valid (local HAProxy). Account / container / object daemons are peer-equivalent and do not follow the VIP.

Devices: /srv/node/{d1,d2,d3} XFS per node (12 total). Never format them from deploy tooling.

Lab policies: replication default, and erasure coding ec-2-1 via liberasurecode_rs_vand. Rings and the replication key must be identical on every node.

Install checklist

Full daemon set + HAProxy on each node; Keepalived for the VIP.
Schedule the object auditor with a timer unit.
Ship EC plugin libraries (libnullcode, liberasurecode_rs_vand) with a working liberasurecode_rs_vand.so.1.0.1 link. liberasurecode must be ≥ 1.6.5: older releases segfault when concurrent rs_vand descriptors are destroyed (reproduced deterministically in CI on Ubuntu’s 1.6.2; fixed upstream in 1.6.5, hardened in 1.8.0). The fleet loads 1.8.0 from /usr/local/lib — verified 2026-08-17 on all four nodes.
Build release binaries with the ec cargo features. Without them the proxy returns HTTP 501 for EC puts.
On swift1: Prometheus, Loki, Alloy, statsd_exporter, swift-console, cabt / autocos; node_exporter on every node (nodes_up should be 4).
Python SAIO :8090 + Rust SAIO :8081 on swift1 for Lab Shadow. The dual-oracle Python instance is a separate concern: Swift 2.33 + s3api + object versioning on swift3 :8090 (with documented lab-only header patches — see Dual-oracle).
After copying binaries under SELinux, run restorecon so labels stay executable (bin_t); haproxy_connect_any=1, ip_nonlocal_bind=1.

Auth

Harness user: test:tester. The key is intentionally omitted from Git and is loaded from the root-readable /etc/swift/peregrine-lab.env on the Linux lab controller. The user name is historical; the current cluster is Contabo.

Build with erasure coding

cd swift-rust
cargo build --release \
  --features swift-proxy-server/ec,swift-object-server/ec

Install on Linux only — never copy macOS Mach-O binaries to the cluster.

Cluster cutover

When replacing VMs or a provider:

Migrate configuration, rings, keys, binaries, systemd units, monitoring, and lab tools.
Treat /srv/node object bytes as optional — omit them for a fresh lab.
Re-run the hard gates: func-suite.sh, ha-test.sh, ec-heal-test.sh, and confirm Prometheus nodes_up=4.

Destroying the previous cluster permanently deletes any object data you did not copy. Keep a written go/no-go note before teardown.