---
title: "Lab cluster"
description: "Four-node Contabo HA lab layout, EC build requirements, and cutover rules."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.myswift.rs/llms.txt
> Use this file to discover all available pages before exploring further.

# Lab cluster

> **Isolation is part of the gate**
>
> This four-node Contabo topology is the **lab**, not a production rollout. As
> of 2026-08-31 the G6-accepted Rust Swift source is
> `17adf0bfa78b30b2a7eed9f39836e0d63c715d2c`. Formal W068 replication and W069
> EC lanes merge exactly once in W070: 179/179 identities, zero unexpected result,
> and no leaked isolated listener or process. G6 is GREEN; G7 is not accepted.
> It is not declared deployed from this page. Do not restart or replace production
> `:8080`, the VIP, HAProxy, Keepalived, rings, Swift data, or active binaries
> while G7 evidence is being prepared or collected.

The development lab is a four-node Contabo HA cluster used to prove the Rust
Swift family
end-to-end — the same environment described under [Testing](/testing) §4. The
previous Azure PAYG topology (`10.42.*` + ILB) is **retired**. Full cutover
notes live in the monorepo under `tools/CONTABO-CLUSTER.md`.

**Fairness lab (2026-08-03):** Contabo is a hybrid on `swift-deploy-rs`, not full
ansible parity. SAIO on swift1 is **NOISY** for performance; VIP path is
**HA-PATH ONLY**; formal throughput requires Performance mode +
`DIRECT-4PROXY`. See monorepo `docs/fairness-lab/` and
`tools/test-results/fairness-lab-20260803/REPORT.html`.

## Topology

| Plane | Address | Role |
|-------|---------|------|
| Management / SSH | public `169.58.108.{85,86,87,121}` | SSH, deploy |
| Proxy / client | `10.0.0.1–4` + Keepalived VIP **`10.0.0.10:8085`** | HAProxy → proxy `:8080` |
| Storage | `10.0.4.1–4` | account / container / object data path |
| Replication | `10.0.8.1–4` | replicator / reconstructor |

Nodes: `swift1`–`swift4`. Keepalived VIP `10.0.0.10` is currently on
**swift2** (`eth1` secondary). Prefer the **real VIP**
`https://10.0.0.10:8085` for clients and gates. Per-node
`http://10.0.0.N:8085` remains valid (local HAProxy). Account / container /
object daemons are peer-equivalent and do not follow the VIP.

Devices: `/srv/node/{d1,d2,d3}` XFS per node (12 total). **Never format** them
from deploy tooling.

Lab policies: replication `default`, and erasure coding `ec-2-1` via
`liberasurecode_rs_vand`. Rings and the replication key must be identical on
every node.

## Install checklist

- Full daemon set + HAProxy on each node; Keepalived for the VIP.
- Schedule the object auditor with a **timer** unit.
- Ship EC plugin libraries (`libnullcode`, `liberasurecode_rs_vand`) with a
  working `liberasurecode_rs_vand.so.1.0.1` link. **liberasurecode must be
  ≥ 1.6.5**: older releases segfault when concurrent `rs_vand` descriptors
  are destroyed (reproduced deterministically in CI on Ubuntu's 1.6.2; fixed
  upstream in 1.6.5, hardened in 1.8.0). The fleet loads 1.8.0 from
  `/usr/local/lib` — verified 2026-08-17 on all four nodes.
- Build release binaries **with the `ec` cargo features**. Without them the
  proxy returns HTTP 501 for EC puts.
- On swift1: Prometheus, Loki, Alloy, statsd_exporter, `swift-console`,
  `cabt` / `autocos`; `node_exporter` on every node (`nodes_up` should be 4).
- Python SAIO `:8090` + Rust SAIO `:8081` on swift1 for Lab Shadow. The
  **dual-oracle** Python instance is a separate concern: Swift 2.33 + s3api +
  object versioning on **swift3** `:8090` (with documented lab-only header
  patches — see [Dual-oracle](/dual-oracle#lab-only-oracle-patches)).
- After copying binaries under SELinux, run `restorecon` so labels stay
  executable (`bin_t`); `haproxy_connect_any=1`, `ip_nonlocal_bind=1`.

## Auth

Harness user: `test:tester`. The key is intentionally omitted from Git and is
loaded from the root-readable `/etc/swift/peregrine-lab.env` on the Linux lab
controller. The user name is historical; the current cluster is Contabo.

## Build with erasure coding

```sh
cd swift-rust
cargo build --release \
  --features swift-proxy-server/ec,swift-object-server/ec
```

Install on Linux only — never copy macOS Mach-O binaries to the cluster.

## Cluster cutover

When replacing VMs or a provider:

1. Migrate configuration, rings, keys, binaries, systemd units, monitoring, and
   lab tools.
2. Treat `/srv/node` object bytes as **optional** — omit them for a fresh lab.
3. Re-run the hard gates: `func-suite.sh`, `ha-test.sh`, `ec-heal-test.sh`, and
   confirm Prometheus `nodes_up=4`.

Destroying the previous cluster permanently deletes any object data you did not
copy. Keep a written go/no-go note before teardown.

## See also

- [Operations](/operations) — health, HA drill, EC heal
- [Testing](/testing) — harness levels
- [Performance](/performance) — Contabo + SAIO numbers
- [Get started](/getting-started) — local SAIO bootstrap

Source: https://docs.myswift.rs/lab-cluster/index.mdx