Two-server redundancy
The optional primary and replica pair — replication, file sync, failover, resync and the Cluster screen.
Spec: amendment v0.3.0-HA (SPEC-AMENDMENT-v0.3.0-HA.md). Run HA1 (v0.8.0) delivered replication, the write gate, the node agent with leadership lease and self-fencing, rejoin (clean / errant / gap) with quarantine and webhook replay, reseed, switchover and failback, and the node:* CLI. Run HA2 (v0.9.0) adds file sync (H4, §12), the degraded lease mode for licensing (H8, §13), the Super → Cluster screen with alerts and self-check items (H11, §14), and the Nginx fallback config (H7, docs/nginx-upstream-backup.conf, §9).
A single-server install needs none of this and behaves exactly as before.
1. What you get#
| Event | What happens |
|---|---|
| Replica fails and comes back | It catches up from the primary. It reseeds only if it was away longer than binlog retention (7 days). |
| Primary fails | The primary's lease expires (15 s). After a further 10 s the replica promotes itself with epoch + 1. Measured on the test box: 17–25 s from failure to the replica answering /health/write with 200. |
| Old primary comes back | It starts read-only and sees the newer epoch. It rejoins as a replica, catches up, and quarantines any writes the new primary never got. |
| Preferred primary is healthy again | With a witness and preferred_primary, a planned switchover moves the primary back once that node has been in sync for auto_failback_after seconds. No data is lost. |
| A cannot reach the witness | A fences itself within 15 s (MariaDB read_only=ON, and the app refuses writes). B promotes after the lease expires. When the network heals, A rejoins. |
| No witness configured | Nothing promotes automatically. Use node:promote (or Promote standby on Super → Cluster). Nodes never fence for lack of a witness. |
| Files uploaded on the primary | Logos, dispute evidence, product binaries and GeoIP files reach the replica within ~10 s (sha256-checked); a full reconcile runs on rejoin, after a reseed and daily. |
| Primary unreachable, licensing | The replica answers /api/v1/license/* in degraded mode: same signing key, TTL ≤ 1 h, "Service is running on the standby server"; its writes are spooled locally and replayed exactly once when a primary is back or it is promoted. |
2. Requirements#
- Two machines (A, B) on a low-latency private link. Semi-sync waits for B on every commit.
- The same billing version on both.
- MariaDB 10.11+ (built-in semi-sync,
READ_ONLY ADMINprivilege). - NTP on both nodes and on the witness: clocks within 2 s.
- A third vantage point for the witness: webgate (contract in amendment H7), or until then
app/tools/fake_witness.phpon a third machine. - PHP with
proc_openenabled, if you wantmariadb-dumpand decoded quarantine reports. Without it the built-in PHP dumper is used and reports list GTIDs only.
3. MariaDB settings (both nodes)#
Print the block for a node with php app/cli.php node:mycnf --node=A (or B), put it in /etc/mysql/mariadb.conf.d/60-billing-ha.cnf, and restart MariaDB:
[mariadbd]
server_id = 1 # A = 1, B = 2
bind_address = 10.0.0.11 # private address
log_bin = billing-bin
relay_log = billing-relay
relay_log_recovery = ON
log_slave_updates = ON
binlog_format = ROW
gtid_domain_id = 0
gtid_strict_mode = ON
binlog_expire_logs_seconds = 604800
sync_binlog = 1
innodb_flush_log_at_trx_commit = 1
rpl_semi_sync_master_enabled = ON
rpl_semi_sync_slave_enabled = ON
rpl_semi_sync_master_timeout = 2000
rpl_semi_sync_master_wait_point = AFTER_SYNC
replicate_ignore_db = billing_local
read_only = ON
ssl_cert = /etc/mysql/ssl/server-cert.pem
ssl_key = /etc/mysql/ssl/server-key.pem
ssl_ca = /etc/mysql/ssl/ca.pem
read_only = ONat startup is deliberate. A restarted node takes writes only after its agent has confirmed the lease (or, without a witness, that the peer is not a newer primary). Install billing first, then apply the block. If the block is already active during the install, runSET GLOBAL read_only=OFFfor the duration of the install.- Replication runs over TLS with the certificates above. Give the replica the CA with
--repl-ssl-ca, and add--repl-ssl-verifyto check the server certificate (its name must match the address the replica connects to).
4. Pairing#
Node A (installed and serving)#
sudo -u www-data php app/cli.php node:init --node=A --name=encrypted \
--peer-url=https://10.0.0.12:8443 --peer-host=10.0.0.12 --peer-db-port=3306 \
--db-root-user=root --witness=https://webgate.internal/witness --secret=<64 hex, same as on webgate> \
--preferred=A --failback-after=600 --repl-ssl-ca=/etc/mysql/ssl/ca.pem --repl-ssl-verify
sudo systemctl enable --now billing-node-agent
node:init does the following:
- creates
billing_node(the node admin: replication control and the local schema) and[email protected](REPLICATION SLAVE only); - creates the node-local schema
billing_local; - writes the
clusterblock intodata/config.php; - takes the first lease (epoch 1) and makes A primary;
- prints a one-time pairing token, valid 1 h.
node:tokenissues a new one.
All of this runs with sql_log_bin = 0: users, grants and local tables never replicate.
Node B (code deployed, not installed)#
sudo -u www-data php app/cli.php node:join --node=B --peer=https://10.0.0.11:8443 --token=<token> \
--db-root-user=root --db-host=127.0.0.1 --db-port=3306 --peer-db-host=10.0.0.11
sudo systemctl enable --now billing-node-agent
node:join does the following:
- Fetches the pairing bundle over the node channel. The request is authenticated with a key derived from the token, and the bundle is sealed with another token-derived key.
- Installs the same master key. If one already exists, its fingerprint must match, or the join stops.
- Creates the same database users with the same passwords, plus its own
billing_nodeandbilling_repl@A. - Writes the config.
- Reseeds from A (snapshot streamed over the node channel) and starts replicating.
If B reaches the witness under a different address, add --witness=….
5. Node channel (private vhost)#
Every request to /__node/<op> must come from an address in cluster.channel_allow (default: the peer's address) and carry an HMAC header (X-Billing-Ts, X-Billing-Nonce, X-Billing-Sig; ±30 s; single-use nonce).
Serve it only on the private link: nginx-node-channel.conf. On the public vhost, add location ^~ /__node/ { return 404; } (included in the example), and do not route it in webgate.
Operations:
| Operation | Purpose |
|---|---|
status | Node state for the peer |
dump | Reseed snapshot (served by the primary only) |
switchover | Planned switchover handshake |
event | Forwarded cluster history |
quarantine | Errant-transaction report |
missing / replay | Webhook comparison and replay |
pair | Pairing, authenticated by the token |
file (v0.9.0) | One synced file, streamed as stored (?path=uploads/…; only uploads/, downloads/, geo/) |
manifest (v0.9.0) | Per-directory file hashes for the reconcile (?dir=uploads) |
spool (v0.9.0) | Replay of the standby's licensing spool (writable node only) |
control (v0.9.0) | An action from the other node's Super → Cluster screen, queued for this node's agent |
6. The agent#
- Unit file:
billing-node-agent.service. It runs as the PHP-FPM user, so files indata/keep one owner. - Loop: every 2 s it observes this node, the witness and the peer, then executes one action from the decision table (
Cluster\AgentDecision, a pure function covered by unit tests). - Lease (primary): renewed every 5 s. After 15 s without a renewal the node fences:
SET GLOBAL read_only = ON;Node::writable()turns false and the app answers writes with 503 +Retry-After: 30;- jobs and pseudo-cron stop.
- Unfence: automatic, as soon as the witness grants the lease at the same epoch again.
- Promotion (replica): all of these must hold:
- the witness shows the lease free, with the previous holder expired for ≥ 10 s;
- the relay log is fully applied;
- the replica was in contact with the primary within 60 s and ≤ 5 s behind at that contact;
- the peer does not report itself writable.
Otherwise the agent holds and emails super admins how to force it.
- The application gate: every writer goes through
Db.Dbrefuses INSERT, UPDATE, DELETE and DDL unlessNode::writable(), a cached read (≤ 1 s) ofdata/cache/node.json, which the agent publishes. MariaDB'sread_onlyis the second, hard guard.tools/refcheck.phpfails the build if code outside the allowed files opens its own database connection.
7. Day-to-day#
| Command | Does |
|---|---|
node:status [--json] | Role, epoch, writable, GTIDs, replication threads, lag, semi-sync status and fallback count, witness holder and expiry, peer, last action, recent events |
node:switchover | Planned switchover. Writes pause for a few seconds, no data loss. The peer must be a healthy replica. |
node:promote [--force] [--wait=40] | Manual promotion. It refuses while the peer is a writable primary. Without --force it refuses if the replica is not current; --force prints the possible-loss window and proceeds. It waits for the witness lease to become free. |
node:reseed | Rebuild this replica from the primary |
node:failback pause|resume | Pause or resume automatic failback on this node |
node:quarantine [--handled=ID] | List errant-transaction reports, replayed webhooks and review items; mark one handled |
node:files [--run|--reconcile] [--json] | File sync on the replica: backlog, one pass now, or a full reconcile (§12) |
node:spool [--replay] [--json] | Licensing spool size; replay it now (§13) |
node:agent [--loops=N] | Run the loop in the foreground (systemd does this) |
Upgrades:
- Upgrade the replica first.
node:switchover.- Upgrade the old primary.
- Run
migrateon the primary. DDL and GRANTs replicate. On the replicamigrate(v0.9.0) only brings the node-local schema up to date and says so; the agent also does that when it starts.
v0.8.0 → v0.9.0: migration 0008 adds file_log and license_spool_applied (run on the primary); the local schema gains file_applied, node_kv, node_requests and the spool key column. Files already on both nodes are checked by the first reconcile after the upgraded replica's agent starts (nothing is fetched when they match).
8. When a node comes back#
The agent classifies the returning node against the current primary's GTID history:
- Clean. Every local GTID is on the primary:
CHANGE MASTER … MASTER_USE_GTID=slave_pos, catch up, and the node becomes a replica. - Errant. The node committed transactions the primary never received (the semi-sync fallback window right before a crash):
- A report
data/quarantine/<epoch>-<node>.sqllists every errant GTID with its tables and row counts. Withproc_openandmariadb-binlogavailable, it also holds the decoded row events. - Provider webhooks stored on the node that the primary does not have are replayed on the primary from their original payloads. This is idempotent on the provider event id.
- Everything else is listed for review:
node:quarantine, and the Super → Cluster screen (§14). - The node is reseeded. The replaced tables stay in
billing_prevuntil the next reseed.
- A report
- Gap. The primary purged binlogs this node still needs, or the node has no data: reseed.
Reseed:
- Take a consistent snapshot of the primary:
mariadb-dump --single-transaction --gtid, or the built-in dumper withSTART TRANSACTION WITH CONSISTENT SNAPSHOT+BINLOG_GTID_POS. - Stream it over the node channel. A truncated transfer is detected by a terminator line.
- Restore into
billing_reseedand swap it in with one atomicRENAME TABLE. RESET MASTER, then setgtid_binlog_state/gtid_slave_posto the snapshot, and replicate.
Both nodes down. Start the node with the newer epoch first (node:status shows it). The other one stays fenced until the primary is reachable, then rejoins.
9. Without webgate#
- Configure
--witness=none. Automatic promotion is then disabled: failover isnode:promoteon the survivor (or Promote standby on Super → Cluster). Everything else works, including the agent, rejoin, reseed, manual switchover, file sync and the degraded lease mode. Automatic failback is off. - Route traffic with plain Nginx:
nginx-upstream-backup.conf. A first, B asbackup; license calls are also retried on the other node when the first answers 503 (a fenced primary or a replica answers 503 without writing; a replica in degraded mode answers the lease). The two-node test harness builds its gate from this exact file. - Adding a witness later (webgate arrives after you ran without one, or
witnessgoes back fromnone): set it on both nodes and restart the agents, the primary's first. A fresh witness grants the primary its current epoch; a witness that knew the cluster before grants the previous holder its current, higher epoch. Any other node still has to present exactly stored + 1 (v0.9.0 rule,Cluster\WitnessRules; webgate must implement the same). - Or run the fake witness on a third machine:
WITNESS_DIR=/var/lib/billing-witness WITNESS_SECRET=<cluster secret> php -S 10.0.0.20:9100 app/tools/fake_witness.phpUse
--witness=http://10.0.0.20:9100/witnesson both nodes. It persists its state, so a restart keeps the holder.
10. Testing#
- Unit tests:
php app/cli.php test Cluster. They cover GTID comparison, HMAC replay and skew, the witness rules and their persistence, the decision table, the write gate on every writer, both dumpers, webhook replay, pairing, the node channel and trusted proxies. - Mutation proof:
php app/tools/mutate.php Cluster. - Two-node scenarios on one box (root, about 8 minutes):
sudo php app/tests/e2e/ha/e2e_ha.php. It runs two MariaDB instances (3307/3308), two PHP-FPM + Nginx stacks, the fake witness and the Nginx gate fromnginx-upstream-backup.conf. It covers scenarios 1–9 of amendment H12 plus scenario 10 (file sync through the real upload paths, replica outage, out-of-band files and reconcile, reverse direction, and the Super → Cluster actions). - Upgrade regression:
sudo php app/tests/e2e/upgrade/regress_upgrade.php --from=<previous release tree> --db-user=… --db-pass=…installs the previous release on a front-controller and a plain vhost, seeds both with its e2e suites, upgrades in place and runs this release's suites again.
11. webgate in front (trusted proxy)#
Set the addresses webgate connects from in data/config.php on both nodes:
'http' => ['verify_tls' => true, 'trusted_proxies' => ['10.0.0.5', '10.0.0.0/24']],
- Client IP: only for requests whose TCP peer is in that list,
X-Forwarded-Foris read right to left; the first address that is not itself a trusted proxy is the client. This IP is used for rate limits (including the per-IP lease limit), sessions, audit and logs. - HTTPS:
X-Forwarded-Proto: httpsmarks the request as HTTPS. - Untrusted peers: headers from anyone else are ignored.
Health checks for webgate's routing are GET /health/write, /health/lease and /health/read. With the plain vhost use /index.php?r=health/write. /health/lease is 200 on the writable primary, and on a replica in degraded lease mode whose lag at last contact was ≤ 300 s and whose data is no older than the smallest active license_lease_ttl_s / 4 (the body then carries "degraded": true and data_age_s).
12. File synchronization (H4)#
- What:
data/uploads/(logos, dispute evidence),data/downloads/(licensed binaries),data/geo/(GeoIP).data/backups/,data/logs/anddata/cache/stay per node. - How: every write or delete in those directories goes through
Billing\Core\FileStore, which writes atomically and, on a cluster node, inserts a row into the replicated tablefile_log(path as stored, sha256 and size of the stored bytes, op, epoch, node). A non-writable node refuses file writes before touching the disk. - Fetch: the replica's agent, every 10 s, takes the
file_logrows not yet inbilling_local.file_applied, keeps the newest row per path, fetches the file from the primary (/__node/file, HMAC-signed, streamed), checks the sha256, moves it into place atomically and marks the row applied. A delete row removes the file. A file whose hash does not match (the primary changed it again) stays pending until the next row arrives. - Reconcile (after a rejoin, after a reseed, after a switchover, daily, or
node:files --reconcile): the replica compares per-directory manifest digests with the primary (/__node/manifest). Where they differ it fetches every missing or different file — also files copied onto the primary by hand, such as a GeoLite2 database — and moves files the primary does not have todata/cache/file-extras/<time>/(never deletes them). Then every log row up to the watermark read before the manifests counts as applied. - Backlog:
node:status,node:files, the Cluster screen and the self-check show it; more than 100 files or an oldest one older than 5 minutes is a warning and an alert. file_logkeeps 30 days (files.prunejob); the daily reconcile covers anything older.- Both nodes must use the same data layout (
wrappedon or off): the self-check fails otherwise.
13. Licensing under HA (H8)#
- Normal operation:
/api/v1/license/*goes to the primary. A replica whose primary is reachable answers 503, which the client treats as "unreachable" (it keeps serving on its lease). - Degraded lease mode (
licensing.degraded_mode, on by default): when this node is a replica and no writable primary is reachable, its agent publisheslease_degraded, and the replica answers leases from replicated data:- the decision code is unchanged and signs with the same key (
product_signing_keysreplicates; the master key is identical), so leases from both nodes verify with the same public key; - TTL = min(product TTL,
licensing.degraded_ttl_cap_s, default 3600), andmessage= "Service is running on the standby server"; - it refuses (503) when its data is older than the product's TTL / 4 or its lag at last contact was above 300 s;
- every write — instance upsert,
last_seen_at, denials (one per license × instance × reason per hour), release — goes tobilling_local.license_spool(ULID key, timestamp); the per-IP / per-key rate limits count inbilling_local.rate_limits_local; - seats = replicated active instances (minus those released here) + instances spooled here.
- the decision code is unchanged and signs with the same key (
- Replay: when a writable primary is reachable again, the agent posts the spool to it (
/__node/spool, 200 rows per batch); after a promotion it applies the spool locally. Rows run through the normalLicenseWriterin timestamp order withlast_seen_at = max(), an older write never overwrites a newer row, and every event's detail ends with "recorded on standby". Each row's key goes into the replicatedlicense_spool_appliedin the same transaction, so a batch delivered twice applies once. Confirmed rows are deleted from the spool;cluster_eventsgetsspool-replayed. - Downloads work on either node once it is writable: the binaries are synced (§12).
- Interop: the HA scenario 8 saves one envelope signed by A and one signed by B and verifies both offline with
app/tests/interop/verify_envelope.go.
14. Super → Cluster (H11)#
- Node cards for both nodes: role and epoch, version and master-key fingerprint (with match), replication threads and lag, semi-sync status and fallback count, last GTID, file backlog, licensing spool, lease expiry, last action; the witness holder and expiry above.
- Actions, each confirmed by typing a phrase (
SWITCHOVER,PROMOTE B,RESEED B,PAUSE FAILBACK,RESUME FAILBACK) and audited:- Switchover — runs on the current primary; needs a healthy replica.
- Promote standby — only when no node is a writable primary; shows the possible-loss window; "accept data loss" =
--force. - Reseed node — never the primary.
- Pause / resume auto-failback — on both nodes.
The screen never changes roles inside the web request: it queues the action in the target node's
billing_local.node_requests(directly, or through the peer's/__node/control), and that node's agent runs it on its next loop. The result shows under Requests and in the history. On the writable node the action is written toaudit_log; on the standby to the cluster history. - History:
cluster_events(promoted, fenced, unfenced, rejoined-clean, rejoined-errant, reseeded, switchover, failback, spool-replayed, files-reconciled, ui-…), with durations. - Quarantine: reports, replayed webhooks with their results, review items with Mark handled (writable node only).
- On the standby: a super admin already signed in (sessions replicate) can open the screen and queue actions there; signing in needs the writable node.
- Alerts (email to super admins, de-duplicated, 15 min): failover, fencing, rejoin with errant transactions, replication lag > 60 s for 5 minutes, file backlog over the thresholds, witness unreachable for 30 s, version or master-key mismatch between the nodes, reseed failure.
- Self-check items (Super → Self-check,
cli.php selfcheck): replication running with lag < 30 s, semi-sync active, same version, same master key, same data layout, witness reachable, clock offset to the peer < 2 s, binlog retention ≥ 7 days, file backlog, licensing spool replayed.
15. Backups and API v1 on a pair (Run 6)#
- Backups. The archive job (
backups.nightly), the drill (backups.drill) and the freshness watch run on the writable node only, like every job. Each node keeps its owndata/backups/(H4), and archives are named with the node id (billing-…-a.bkp). Thebackupsrows replicate, so after a switchover the screen lists the other node's archives as "not on this server".cli.php backup:runon a replica refuses. - Restore drill. On a cluster the drill runs as the node admin with
sql_log_bin = 0, so<db>_restore_testnever reaches the other node and creates no errant transaction. Pairings made with 0.11 grant the node admin that schema. After upgrading an existing pair, runcli.php backup:grant --db-root-user=root [--admin-host=…]on each node. - Restore (
backup:restore) is refused on a cluster node. Restore onto one server with the cluster disabled, then pair the second server again. - API v1 is answered by the writable primary; a replica answers 503
unavailablewithRetry-After, because each request writes its rate-limit counter. Keys, webhook endpoints, events and deliveries replicate. The delivery job (webhooks.deliver) runs on the primary. - Two-node e2e scenario 11 covers these points (
app/tests/e2e/ha/e2e_ha.php --scenarios=11).
16. Help and SEO on a pair (Run 7)#
- Help pages are served by both nodes. Edits made in Super → Help & SEO are rows in
help_overridesand replicate, so the standby shows them too. Page views of the help are recorded by the writable node only (the standby serves its pages without writing). - Static SEO files (
robots.txt,sitemap.xml,llms.txt,llms-full.txt,site.webmanifest) live in each node's own webroot. Theseo.publishjob runs on the writable node, so the standby keeps the files it had (or the shipped ones) until it is promoted; its first hourly run then refreshes them. - Open Graph images are cached per node in
data/cache/og/and rendered on demand on either node. - Two-node e2e scenario 12 covers these points (
app/tests/e2e/ha/e2e_ha.php --scenarios=12).