billingHelp

Two-server redundancy

The optional primary and replica pair — replication, file sync, failover, resync and the Cluster screen.

Spec: amendment v0.3.0-HA (SPEC-AMENDMENT-v0.3.0-HA.md). Run HA1 (v0.8.0) delivered replication, the write gate, the node agent with leadership lease and self-fencing, rejoin (clean / errant / gap) with quarantine and webhook replay, reseed, switchover and failback, and the node:* CLI. Run HA2 (v0.9.0) adds file sync (H4, §12), the degraded lease mode for licensing (H8, §13), the Super → Cluster screen with alerts and self-check items (H11, §14), and the Nginx fallback config (H7, docs/nginx-upstream-backup.conf, §9).

A single-server install needs none of this and behaves exactly as before.

1. What you get#

EventWhat happens
Replica fails and comes backIt catches up from the primary. It reseeds only if it was away longer than binlog retention (7 days).
Primary failsThe primary's lease expires (15 s). After a further 10 s the replica promotes itself with epoch + 1. Measured on the test box: 17–25 s from failure to the replica answering /health/write with 200.
Old primary comes backIt starts read-only and sees the newer epoch. It rejoins as a replica, catches up, and quarantines any writes the new primary never got.
Preferred primary is healthy againWith a witness and preferred_primary, a planned switchover moves the primary back once that node has been in sync for auto_failback_after seconds. No data is lost.
A cannot reach the witnessA fences itself within 15 s (MariaDB read_only=ON, and the app refuses writes). B promotes after the lease expires. When the network heals, A rejoins.
No witness configuredNothing promotes automatically. Use node:promote (or Promote standby on Super → Cluster). Nodes never fence for lack of a witness.
Files uploaded on the primaryLogos, dispute evidence, product binaries and GeoIP files reach the replica within ~10 s (sha256-checked); a full reconcile runs on rejoin, after a reseed and daily.
Primary unreachable, licensingThe replica answers /api/v1/license/* in degraded mode: same signing key, TTL ≤ 1 h, "Service is running on the standby server"; its writes are spooled locally and replayed exactly once when a primary is back or it is promoted.

2. Requirements#

  • Two machines (A, B) on a low-latency private link. Semi-sync waits for B on every commit.
  • The same billing version on both.
  • MariaDB 10.11+ (built-in semi-sync, READ_ONLY ADMIN privilege).
  • NTP on both nodes and on the witness: clocks within 2 s.
  • A third vantage point for the witness: webgate (contract in amendment H7), or until then app/tools/fake_witness.php on a third machine.
  • PHP with proc_open enabled, if you want mariadb-dump and decoded quarantine reports. Without it the built-in PHP dumper is used and reports list GTIDs only.

3. MariaDB settings (both nodes)#

Print the block for a node with php app/cli.php node:mycnf --node=A (or B), put it in /etc/mysql/mariadb.conf.d/60-billing-ha.cnf, and restart MariaDB:

[mariadbd]
server_id                       = 1            # A = 1, B = 2
bind_address                    = 10.0.0.11    # private address
log_bin                         = billing-bin
relay_log                       = billing-relay
relay_log_recovery              = ON
log_slave_updates               = ON
binlog_format                   = ROW
gtid_domain_id                  = 0
gtid_strict_mode                = ON
binlog_expire_logs_seconds      = 604800
sync_binlog                     = 1
innodb_flush_log_at_trx_commit  = 1
rpl_semi_sync_master_enabled    = ON
rpl_semi_sync_slave_enabled     = ON
rpl_semi_sync_master_timeout    = 2000
rpl_semi_sync_master_wait_point = AFTER_SYNC
replicate_ignore_db             = billing_local
read_only                       = ON
ssl_cert                        = /etc/mysql/ssl/server-cert.pem
ssl_key                         = /etc/mysql/ssl/server-key.pem
ssl_ca                          = /etc/mysql/ssl/ca.pem
  • read_only = ON at startup is deliberate. A restarted node takes writes only after its agent has confirmed the lease (or, without a witness, that the peer is not a newer primary). Install billing first, then apply the block. If the block is already active during the install, run SET GLOBAL read_only=OFF for the duration of the install.
  • Replication runs over TLS with the certificates above. Give the replica the CA with --repl-ssl-ca, and add --repl-ssl-verify to check the server certificate (its name must match the address the replica connects to).

4. Pairing#

Node A (installed and serving)#

sudo -u www-data php app/cli.php node:init --node=A --name=encrypted \
  --peer-url=https://10.0.0.12:8443 --peer-host=10.0.0.12 --peer-db-port=3306 \
  --db-root-user=root --witness=https://webgate.internal/witness --secret=<64 hex, same as on webgate> \
  --preferred=A --failback-after=600 --repl-ssl-ca=/etc/mysql/ssl/ca.pem --repl-ssl-verify
sudo systemctl enable --now billing-node-agent

node:init does the following:

  • creates billing_node (the node admin: replication control and the local schema) and [email protected] (REPLICATION SLAVE only);
  • creates the node-local schema billing_local;
  • writes the cluster block into data/config.php;
  • takes the first lease (epoch 1) and makes A primary;
  • prints a one-time pairing token, valid 1 h. node:token issues a new one.

All of this runs with sql_log_bin = 0: users, grants and local tables never replicate.

Node B (code deployed, not installed)#

sudo -u www-data php app/cli.php node:join --node=B --peer=https://10.0.0.11:8443 --token=<token> \
  --db-root-user=root --db-host=127.0.0.1 --db-port=3306 --peer-db-host=10.0.0.11
sudo systemctl enable --now billing-node-agent

node:join does the following:

  1. Fetches the pairing bundle over the node channel. The request is authenticated with a key derived from the token, and the bundle is sealed with another token-derived key.
  2. Installs the same master key. If one already exists, its fingerprint must match, or the join stops.
  3. Creates the same database users with the same passwords, plus its own billing_node and billing_repl@A.
  4. Writes the config.
  5. Reseeds from A (snapshot streamed over the node channel) and starts replicating.

If B reaches the witness under a different address, add --witness=….

5. Node channel (private vhost)#

Every request to /__node/<op> must come from an address in cluster.channel_allow (default: the peer's address) and carry an HMAC header (X-Billing-Ts, X-Billing-Nonce, X-Billing-Sig; ±30 s; single-use nonce).

Serve it only on the private link: nginx-node-channel.conf. On the public vhost, add location ^~ /__node/ { return 404; } (included in the example), and do not route it in webgate.

Operations:

OperationPurpose
statusNode state for the peer
dumpReseed snapshot (served by the primary only)
switchoverPlanned switchover handshake
eventForwarded cluster history
quarantineErrant-transaction report
missing / replayWebhook comparison and replay
pairPairing, authenticated by the token
file (v0.9.0)One synced file, streamed as stored (?path=uploads/…; only uploads/, downloads/, geo/)
manifest (v0.9.0)Per-directory file hashes for the reconcile (?dir=uploads)
spool (v0.9.0)Replay of the standby's licensing spool (writable node only)
control (v0.9.0)An action from the other node's Super → Cluster screen, queued for this node's agent

6. The agent#

  • Unit file: billing-node-agent.service. It runs as the PHP-FPM user, so files in data/ keep one owner.
  • Loop: every 2 s it observes this node, the witness and the peer, then executes one action from the decision table (Cluster\AgentDecision, a pure function covered by unit tests).
  • Lease (primary): renewed every 5 s. After 15 s without a renewal the node fences:
    • SET GLOBAL read_only = ON;
    • Node::writable() turns false and the app answers writes with 503 + Retry-After: 30;
    • jobs and pseudo-cron stop.
  • Unfence: automatic, as soon as the witness grants the lease at the same epoch again.
  • Promotion (replica): all of these must hold:
    • the witness shows the lease free, with the previous holder expired for ≥ 10 s;
    • the relay log is fully applied;
    • the replica was in contact with the primary within 60 s and ≤ 5 s behind at that contact;
    • the peer does not report itself writable.

    Otherwise the agent holds and emails super admins how to force it.

  • The application gate: every writer goes through Db. Db refuses INSERT, UPDATE, DELETE and DDL unless Node::writable(), a cached read (≤ 1 s) of data/cache/node.json, which the agent publishes. MariaDB's read_only is the second, hard guard. tools/refcheck.php fails the build if code outside the allowed files opens its own database connection.

7. Day-to-day#

CommandDoes
node:status [--json]Role, epoch, writable, GTIDs, replication threads, lag, semi-sync status and fallback count, witness holder and expiry, peer, last action, recent events
node:switchoverPlanned switchover. Writes pause for a few seconds, no data loss. The peer must be a healthy replica.
node:promote [--force] [--wait=40]Manual promotion. It refuses while the peer is a writable primary. Without --force it refuses if the replica is not current; --force prints the possible-loss window and proceeds. It waits for the witness lease to become free.
node:reseedRebuild this replica from the primary
node:failback pause|resumePause or resume automatic failback on this node
node:quarantine [--handled=ID]List errant-transaction reports, replayed webhooks and review items; mark one handled
node:files [--run|--reconcile] [--json]File sync on the replica: backlog, one pass now, or a full reconcile (§12)
node:spool [--replay] [--json]Licensing spool size; replay it now (§13)
node:agent [--loops=N]Run the loop in the foreground (systemd does this)

Upgrades:

  1. Upgrade the replica first.
  2. node:switchover.
  3. Upgrade the old primary.
  4. Run migrate on the primary. DDL and GRANTs replicate. On the replica migrate (v0.9.0) only brings the node-local schema up to date and says so; the agent also does that when it starts.

v0.8.0 → v0.9.0: migration 0008 adds file_log and license_spool_applied (run on the primary); the local schema gains file_applied, node_kv, node_requests and the spool key column. Files already on both nodes are checked by the first reconcile after the upgraded replica's agent starts (nothing is fetched when they match).

8. When a node comes back#

The agent classifies the returning node against the current primary's GTID history:

  • Clean. Every local GTID is on the primary: CHANGE MASTER … MASTER_USE_GTID=slave_pos, catch up, and the node becomes a replica.
  • Errant. The node committed transactions the primary never received (the semi-sync fallback window right before a crash):
    1. A report data/quarantine/<epoch>-<node>.sql lists every errant GTID with its tables and row counts. With proc_open and mariadb-binlog available, it also holds the decoded row events.
    2. Provider webhooks stored on the node that the primary does not have are replayed on the primary from their original payloads. This is idempotent on the provider event id.
    3. Everything else is listed for review: node:quarantine, and the Super → Cluster screen (§14).
    4. The node is reseeded. The replaced tables stay in billing_prev until the next reseed.
  • Gap. The primary purged binlogs this node still needs, or the node has no data: reseed.

Reseed:

  1. Take a consistent snapshot of the primary: mariadb-dump --single-transaction --gtid, or the built-in dumper with START TRANSACTION WITH CONSISTENT SNAPSHOT + BINLOG_GTID_POS.
  2. Stream it over the node channel. A truncated transfer is detected by a terminator line.
  3. Restore into billing_reseed and swap it in with one atomic RENAME TABLE.
  4. RESET MASTER, then set gtid_binlog_state / gtid_slave_pos to the snapshot, and replicate.

Both nodes down. Start the node with the newer epoch first (node:status shows it). The other one stays fenced until the primary is reachable, then rejoins.

9. Without webgate#

  • Configure --witness=none. Automatic promotion is then disabled: failover is node:promote on the survivor (or Promote standby on Super → Cluster). Everything else works, including the agent, rejoin, reseed, manual switchover, file sync and the degraded lease mode. Automatic failback is off.
  • Route traffic with plain Nginx: nginx-upstream-backup.conf. A first, B as backup; license calls are also retried on the other node when the first answers 503 (a fenced primary or a replica answers 503 without writing; a replica in degraded mode answers the lease). The two-node test harness builds its gate from this exact file.
  • Adding a witness later (webgate arrives after you ran without one, or witness goes back from none): set it on both nodes and restart the agents, the primary's first. A fresh witness grants the primary its current epoch; a witness that knew the cluster before grants the previous holder its current, higher epoch. Any other node still has to present exactly stored + 1 (v0.9.0 rule, Cluster\WitnessRules; webgate must implement the same).
  • Or run the fake witness on a third machine:
    WITNESS_DIR=/var/lib/billing-witness WITNESS_SECRET=<cluster secret> php -S 10.0.0.20:9100 app/tools/fake_witness.php

    Use --witness=http://10.0.0.20:9100/witness on both nodes. It persists its state, so a restart keeps the holder.

10. Testing#

  • Unit tests: php app/cli.php test Cluster. They cover GTID comparison, HMAC replay and skew, the witness rules and their persistence, the decision table, the write gate on every writer, both dumpers, webhook replay, pairing, the node channel and trusted proxies.
  • Mutation proof: php app/tools/mutate.php Cluster.
  • Two-node scenarios on one box (root, about 8 minutes): sudo php app/tests/e2e/ha/e2e_ha.php. It runs two MariaDB instances (3307/3308), two PHP-FPM + Nginx stacks, the fake witness and the Nginx gate from nginx-upstream-backup.conf. It covers scenarios 1–9 of amendment H12 plus scenario 10 (file sync through the real upload paths, replica outage, out-of-band files and reconcile, reverse direction, and the Super → Cluster actions).
  • Upgrade regression: sudo php app/tests/e2e/upgrade/regress_upgrade.php --from=<previous release tree> --db-user=… --db-pass=… installs the previous release on a front-controller and a plain vhost, seeds both with its e2e suites, upgrades in place and runs this release's suites again.

11. webgate in front (trusted proxy)#

Set the addresses webgate connects from in data/config.php on both nodes:

'http' => ['verify_tls' => true, 'trusted_proxies' => ['10.0.0.5', '10.0.0.0/24']],
  • Client IP: only for requests whose TCP peer is in that list, X-Forwarded-For is read right to left; the first address that is not itself a trusted proxy is the client. This IP is used for rate limits (including the per-IP lease limit), sessions, audit and logs.
  • HTTPS: X-Forwarded-Proto: https marks the request as HTTPS.
  • Untrusted peers: headers from anyone else are ignored.

Health checks for webgate's routing are GET /health/write, /health/lease and /health/read. With the plain vhost use /index.php?r=health/write. /health/lease is 200 on the writable primary, and on a replica in degraded lease mode whose lag at last contact was ≤ 300 s and whose data is no older than the smallest active license_lease_ttl_s / 4 (the body then carries "degraded": true and data_age_s).

12. File synchronization (H4)#

  • What: data/uploads/ (logos, dispute evidence), data/downloads/ (licensed binaries), data/geo/ (GeoIP). data/backups/, data/logs/ and data/cache/ stay per node.
  • How: every write or delete in those directories goes through Billing\Core\FileStore, which writes atomically and, on a cluster node, inserts a row into the replicated table file_log (path as stored, sha256 and size of the stored bytes, op, epoch, node). A non-writable node refuses file writes before touching the disk.
  • Fetch: the replica's agent, every 10 s, takes the file_log rows not yet in billing_local.file_applied, keeps the newest row per path, fetches the file from the primary (/__node/file, HMAC-signed, streamed), checks the sha256, moves it into place atomically and marks the row applied. A delete row removes the file. A file whose hash does not match (the primary changed it again) stays pending until the next row arrives.
  • Reconcile (after a rejoin, after a reseed, after a switchover, daily, or node:files --reconcile): the replica compares per-directory manifest digests with the primary (/__node/manifest). Where they differ it fetches every missing or different file — also files copied onto the primary by hand, such as a GeoLite2 database — and moves files the primary does not have to data/cache/file-extras/<time>/ (never deletes them). Then every log row up to the watermark read before the manifests counts as applied.
  • Backlog: node:status, node:files, the Cluster screen and the self-check show it; more than 100 files or an oldest one older than 5 minutes is a warning and an alert.
  • file_log keeps 30 days (files.prune job); the daily reconcile covers anything older.
  • Both nodes must use the same data layout (wrapped on or off): the self-check fails otherwise.

13. Licensing under HA (H8)#

  • Normal operation: /api/v1/license/* goes to the primary. A replica whose primary is reachable answers 503, which the client treats as "unreachable" (it keeps serving on its lease).
  • Degraded lease mode (licensing.degraded_mode, on by default): when this node is a replica and no writable primary is reachable, its agent publishes lease_degraded, and the replica answers leases from replicated data:
    • the decision code is unchanged and signs with the same key (product_signing_keys replicates; the master key is identical), so leases from both nodes verify with the same public key;
    • TTL = min(product TTL, licensing.degraded_ttl_cap_s, default 3600), and message = "Service is running on the standby server";
    • it refuses (503) when its data is older than the product's TTL / 4 or its lag at last contact was above 300 s;
    • every write — instance upsert, last_seen_at, denials (one per license × instance × reason per hour), release — goes to billing_local.license_spool (ULID key, timestamp); the per-IP / per-key rate limits count in billing_local.rate_limits_local;
    • seats = replicated active instances (minus those released here) + instances spooled here.
  • Replay: when a writable primary is reachable again, the agent posts the spool to it (/__node/spool, 200 rows per batch); after a promotion it applies the spool locally. Rows run through the normal LicenseWriter in timestamp order with last_seen_at = max(), an older write never overwrites a newer row, and every event's detail ends with "recorded on standby". Each row's key goes into the replicated license_spool_applied in the same transaction, so a batch delivered twice applies once. Confirmed rows are deleted from the spool; cluster_events gets spool-replayed.
  • Downloads work on either node once it is writable: the binaries are synced (§12).
  • Interop: the HA scenario 8 saves one envelope signed by A and one signed by B and verifies both offline with app/tests/interop/verify_envelope.go.

14. Super → Cluster (H11)#

  • Node cards for both nodes: role and epoch, version and master-key fingerprint (with match), replication threads and lag, semi-sync status and fallback count, last GTID, file backlog, licensing spool, lease expiry, last action; the witness holder and expiry above.
  • Actions, each confirmed by typing a phrase (SWITCHOVER, PROMOTE B, RESEED B, PAUSE FAILBACK, RESUME FAILBACK) and audited:
    • Switchover — runs on the current primary; needs a healthy replica.
    • Promote standby — only when no node is a writable primary; shows the possible-loss window; "accept data loss" = --force.
    • Reseed node — never the primary.
    • Pause / resume auto-failback — on both nodes.

    The screen never changes roles inside the web request: it queues the action in the target node's billing_local.node_requests (directly, or through the peer's /__node/control), and that node's agent runs it on its next loop. The result shows under Requests and in the history. On the writable node the action is written to audit_log; on the standby to the cluster history.

  • History: cluster_events (promoted, fenced, unfenced, rejoined-clean, rejoined-errant, reseeded, switchover, failback, spool-replayed, files-reconciled, ui-…), with durations.
  • Quarantine: reports, replayed webhooks with their results, review items with Mark handled (writable node only).
  • On the standby: a super admin already signed in (sessions replicate) can open the screen and queue actions there; signing in needs the writable node.
  • Alerts (email to super admins, de-duplicated, 15 min): failover, fencing, rejoin with errant transactions, replication lag > 60 s for 5 minutes, file backlog over the thresholds, witness unreachable for 30 s, version or master-key mismatch between the nodes, reseed failure.
  • Self-check items (Super → Self-check, cli.php selfcheck): replication running with lag < 30 s, semi-sync active, same version, same master key, same data layout, witness reachable, clock offset to the peer < 2 s, binlog retention ≥ 7 days, file backlog, licensing spool replayed.

15. Backups and API v1 on a pair (Run 6)#

  • Backups. The archive job (backups.nightly), the drill (backups.drill) and the freshness watch run on the writable node only, like every job. Each node keeps its own data/backups/ (H4), and archives are named with the node id (billing-…-a.bkp). The backups rows replicate, so after a switchover the screen lists the other node's archives as "not on this server". cli.php backup:run on a replica refuses.
  • Restore drill. On a cluster the drill runs as the node admin with sql_log_bin = 0, so <db>_restore_test never reaches the other node and creates no errant transaction. Pairings made with 0.11 grant the node admin that schema. After upgrading an existing pair, run cli.php backup:grant --db-root-user=root [--admin-host=…] on each node.
  • Restore (backup:restore) is refused on a cluster node. Restore onto one server with the cluster disabled, then pair the second server again.
  • API v1 is answered by the writable primary; a replica answers 503 unavailable with Retry-After, because each request writes its rate-limit counter. Keys, webhook endpoints, events and deliveries replicate. The delivery job (webhooks.deliver) runs on the primary.
  • Two-node e2e scenario 11 covers these points (app/tests/e2e/ha/e2e_ha.php --scenarios=11).

16. Help and SEO on a pair (Run 7)#

  • Help pages are served by both nodes. Edits made in Super → Help & SEO are rows in help_overrides and replicate, so the standby shows them too. Page views of the help are recorded by the writable node only (the standby serves its pages without writing).
  • Static SEO files (robots.txt, sitemap.xml, llms.txt, llms-full.txt, site.webmanifest) live in each node's own webroot. The seo.publish job runs on the writable node, so the standby keeps the files it had (or the shipped ones) until it is promoted; its first hourly run then refreshes them.
  • Open Graph images are cached per node in data/cache/og/ and rendered on demand on either node.
  • Two-node e2e scenario 12 covers these points (app/tests/e2e/ha/e2e_ha.php --scenarios=12).