billing help · For platform operators
Platform manual
Running the platform as super admin: architecture, tenants, gateway defaults, backups, GeoIP, audit, self-check, two-server redundancy, licensing, upgrades and troubleshooting.
- Architecture and the platform overview
- Workspaces (tenants)
- Gateway defaults and managed credential sets
- Email settings and the outbox
- Backups and restore
- GeoIP files
- Audit log and acting as someone
- Health, jobs, self-check and system monitor
- Two-server redundancy
- Licensing across workspaces
- Platform analytics and insights
- Help, search engines and AI crawlers
- Super-admin sign-in and security
- Upgrading
- Troubleshooting
Architecture and the platform overview
Text description
Browsers reach Nginx, which serves the webroot (public/: entry files, assets, static SEO files) and passes PHP requests to PHP-FPM. PHP runs the application in app/ (core, providers, views, help content), which is never served. app/ reads and writes data/ (config.php, master key, uploads, GeoIP files, backups, logs, cache) and the MariaDB database through three database users: app (data changes only), backup (read and lock) and migrate (schema). Outbound, the application calls the payment providers' APIs and SMTP2GO for email. Inbound, the providers deliver signed webhooks to public/webhooks/. A system cron job, or the built-in pseudo-cron, runs the scheduled jobs.
Three directories#
| Directory | Contains | Served? |
|---|---|---|
public/ | Entry files (index.php, admin/, pay/, account/, super/, api/, help/, webhooks/), assets, robots.txt, sitemap.xml, llms.txt, favicon | Yes — this is the Nginx root |
app/ | All PHP code, views, help pages, fonts for images, tests, tools | Never |
data/ | config.php, keys/master.key, uploads, GeoIP files, licensed downloads, backups, logs, cache | Never |
The application needs no rewrite rules. With the hardened vhost's try_files … /index.php, links are clean (/admin/settings/users); on a plain PHP vhost they use the entry files (/admin/?r=settings/users). The application notices which one works and builds its links to match.
How a request is handled#
Text description
A request reaches a physical entry file (index.php, or admin/index.php and the other area entries on a plain vhost). The kernel boots the application and the router finds the route in one route table, for clean URLs and for the ?r= form alike. The route's rules run next: sign-in for the right area (super admin, workspace member, customer, API key), the role's capability, and the CSRF and origin check on every change. The controller does the work and renders a view. Every response leaves with the security headers (CSP with a nonce, HSTS, frame and referrer rules, robots) and, on public pages, a cookieless page-view pixel. Pseudo-cron runs after the response is sent.
Every workspace-owned row carries the workspace id. The data layer adds it to every query from the signed-in context, so no screen builds its own workspace filter. Only super admins and system jobs can open a cross-workspace handle, and each use is logged.
How a webhook is handled#
Text description
The provider posts an event to /webhooks/provider.php with the credential set in the query string (Authorize.net: no query string; the set is found by its signature key). The signature is checked on the raw body before anything is parsed or stored; a bad signature is refused. The event id is recorded once, so duplicates are ignored. The object is fetched again from the provider's API. The state machine applies the change if it is allowed; a stale or out-of-order event is kept but not applied. Then emails, outbound webhooks and analytics follow. If processing fails the endpoint answers 500 and the provider retries.
Three database users#
- app — reads and changes data, table by table; it can only add to the audit log, never change or delete it.
- backup — reads and locks, for the nightly dump.
- migrate — owns the schema, applies migrations and maintains the app user's grants.
The Platform overview#
The overview shows, at a glance:
- System — version, database, disk, scheduler and the latest self-check result, each linking to its screen;
- Backups — the newest archive's age, the latest restore drill, and whether an off-box copy is recent;
- Tenants — active, awaiting their owner, suspended, with the newest ones listed;
- Credential sets by provider across all workspaces;
- Recent activity from the audit log.
Workspaces (tenants)
Workspaces are created by invitation only. There are no limits on workspaces, products or members, and the platform charges workspaces no fees.
Text description
A workspace is created by the super admin and starts as Invited until its owner accepts the invitation. Active workspaces take payments. Suspended workspaces stop taking payments and their staff cannot sign in; they can be made active again. Offboarding winds a workspace down before it is Deleted.
Inviting a workspace#
Tenants → Invite a tenant:
- Business name and Address — the address is the workspace's slug in its pay links (
/pay/<address>/…). Choose it carefully; it appears in every link buyers see. - Owner name and Owner email.
- Create and invite owner. The owner receives a single-use link valid for 7 days. The screen also shows the link once, in case the email does not arrive.
The workspace is Invited until the owner accepts; then it becomes Active.
A workspace's page#
- Members with their role and last sign-in. Act as opens the workspace as that member, for support (Audit and impersonation).
- Invite another owner or re-send — a new owner invitation.
- Find a customer of this workspace — search by email, then act as the customer.
- Gateway credentials and Manage gateways — the workspace's credential sets; add managed sets (Gateway defaults).
- Settings — Platform-managed stops the workspace's own members from adding credential sets; sets you add work either way.
- Activity — the workspace's latest audit entries.
Suspending and reactivating#
Suspend stops the workspace at once: its pay pages answer not found, its members cannot sign in, and its API keys stop working. Subscriptions at the providers are not touched — they keep renewing into the workspace's own provider accounts. Reactivate restores everything.
Offboarding and deletion are deliberate, separate steps: export what the workspace needs first, and keep backups for your retention period.
Gateway defaults and managed credential sets
Gateway defaults. Super → Settings → Gateway defaults holds provider accounts the platform owns.
- Keys only: a default stores the API keys, sealed. Webhook values belong to each workspace, because each workspace registers its own endpoint.
- Lifecycle: Draft until Verify makes one authenticated call to the provider; then Verified.
Applying a default. On a workspace's Manage gateways page, choose From a default.
- This creates a credential set in that workspace with the default's keys, re-sealed under the workspace's binding. The set is marked managed and linked to the default.
- Paste the provider's webhook value for the workspace endpoint (the URL is shown on the set). Then verify and put the set in service, as usual.
- Enter keys creates a managed set without a default.
Managed sets in the workspace:
- Members can bind them to products.
- Only the super admin can edit, verify, activate, disable or enable them. Members see a "managed by the platform" note instead of the edit forms.
- On a set linked to a default, the super admin edits only the label and the webhook value; the keys come from the default.
Key rotation:
- Edit the default's keys. The default goes back to Draft, and linked sets keep the old keys and stay in service.
- Verify the default. The new keys are copied to every linked set (audited as
credential_set.keys_propagated), and their status is kept: no downtime.
Limits:
- Authorize.net: a default can serve one workspace only, because Authorize.net webhooks are matched to a set by the account Signature Key.
- Deleting a default: refused while workspace sets still use it.
Workspace setting. On the workspace page, Platform-managed stops the workspace's own members from adding credential sets. Sets the super admin adds work either way.
A gateway default is a provider account the platform owns: payments made through it arrive in the platform's account. To let workspaces use their own accounts without copying keys, see Connected accounts and Super → Platform apps.
Credential sets of a workspace#
On a workspace's Manage gateways page you see every set of that workspace, with its status. You can add a managed set from a default or by entering keys, and for managed sets you edit the label and webhook value, verify, put in service, disable and enable. Sets the workspace added itself are read-only for you, so a workspace's own provider accounts stay under its control.
Text description
A credential set starts as Draft. Verify makes one authenticated call to the provider; when it succeeds the set is Verified. Put in service needs the webhook value too and makes the set Test or Live, depending on its environment. A set can be disabled by hand, or automatically after three provider authentication failures in a row (the owners are emailed), and enabled again.
Email settings and the outbox
Super → Settings → Email configures how every message leaves the platform. The step-by-step setup, including SPF, DKIM and DMARC, is in Email through SMTP2GO.
On this screen#
- Sender address — the From address of every message, on a domain verified in SMTP2GO. The sender name is the workspace's name for customer emails and the platform name otherwise.
- SMTP2GO API key and API region — the key is stored sealed and shown as its last four characters.
- Send test message — sends to your own address and runs the sender at once, so you see the result immediately.
- Bounce & spam webhook — the URL to add in SMTP2GO; hard bounces and complaints then suppress the address.
- Suppressed addresses — addresses no longer mailed, with the reason.
- Outbox — recent messages with their status (queued, sent, failed, suppressed) and attempts.
How sending works#
Every message is written to the outbox first. The mail.send job sends queued messages every minute and retries failures with increasing delays. A message that keeps failing is marked failed; Health & jobs shows the last error. Nothing is lost while email is not configured: messages wait until it is.
Backups and restore
Text description
Every night the database is dumped as the backup user and packed into a tar archive together with config.php, uploads, GeoIP files and licensed downloads, plus a manifest with the size and SHA-256 of every member and the row count of every table. The archive is encrypted with AES-256-GCM in chunks, with a key derived from the master key, and written to data/backups. Retention keeps 7 daily, 4 weekly and 12 monthly archives. Every week the drill decrypts the newest archive, loads it into a scratch database, compares every table's row count and every file's checksum, and drops the scratch database. Download archives regularly to keep a copy off the server; keep the master key off the server too.
On the Backups screen#
- the newest archive's age (an alarm above 26 hours), size, the latest restore drill, and free disk space;
- the archives, with the retention slot (daily, weekly, monthly) that keeps each one;
- Back up now, Run the drill, Download (audited; counts as an off-box copy), and I made an off-box copy.
Keep copies away from the server: Keeping backups off the server.
1. What is backed up#
A nightly job (backups.nightly, 02:37 UTC) writes one archive to data/backups/. The archive holds:
| Member | Content |
|---|---|
db.sql | The whole database, one consistent snapshot. |
config.php | data/config.php. |
files/uploads/… | Logos and dispute evidence. |
files/geo/… | GeoIP databases. |
files/downloads/… | Licensed binaries. |
manifest.json | Size and sha256 of every member, plus the row count of every table in the dump. |
The archive does not contain the master key (data/keys/master.key), because the master key is what decrypts it. Keep the master key somewhere other than the server: the installer printed it once. Without it no archive can be opened.
data/logs/, data/cache/ and data/backups/ itself are not backed up.
2. Format#
billing-20261110-023700[-a].bkp the archive (node id suffix on a cluster)
billing-20261110-023700[-a].bkp.sha256 "<sha256> <name>.bkp" — check a copy with sha256sum -c
The plaintext is a standard POSIX ustar tar, written by our own tar writer. It is encrypted with AES-256-GCM in 1 MiB chunks:
- Key. HKDF-SHA256 of the master key's backup subkey, with a random 16-byte salt per archive. Every archive therefore has its own key.
- Nonce. An 8-byte random prefix plus the chunk number.
- Header. The header (magic, salt, nonce prefix, chunk size, master-key fingerprint) is authenticated with every chunk. The chunk number and a final-chunk flag are authenticated too.
These rules have consequences:
- Tampering, reordering or a truncated copy is detected before any byte is used.
- An archive made with another master key is recognised from its fingerprint.
Wrapped-blobs mode (data/ inside the webroot): the archive and its side file are named ….bkp.php and ….bkp.sha256.php and start with the PHP exit guard. The guard is not part of the archive bytes or of the sha256. Downloads and every reader skip it.
Temporary files (the plain db.sql, the partial archive) live in data/backups/.work-<random>/, are mode 0600, and are deleted when the run ends. A directory left behind by a crash is removed after a day.
3. Database dump#
The dump reuses the HA dumper:
mariadb-dump --single-transaction --hex-blobwhen PHP may run programs;- otherwise the built-in PHP dumper, with a consistent-snapshot transaction and binary columns as hex.
To force either one, set 'backups' => ['dumper' => 'php' | 'mariadb-dump'] in data/config.php.
The dump runs as the backup DB user, which has SELECT, LOCK TABLES, SHOW VIEW and TRIGGER, so no position or GTID data is taken.
The row count of every table is counted from the dump itself:
- one-line INSERTs (built-in dumper) and one-tuple-per-line INSERTs (mariadb-dump 10.11+) are both understood;
- the PHP dumper's own total is cross-checked against that count.
Because the dump is one snapshot, these counts are exactly what was live at dump time.
4. Retention (GFS)#
After each nightly run, archives of this server that no slot keeps are deleted. The slots are:
- the newest archive of each of the last 7 days;
- the newest of each of the last 4 ISO weeks;
- the newest of each of the last 12 months.
The newest archive is always kept. Failed attempts are listed for 90 days.
To change the counts, set 'backups' => ['retention' => ['daily' => 7, 'weekly' => 4, 'monthly' => 12]].
5. Restore drill#
A weekly job (backups.drill, Sundays 05:13 UTC) runs the drill on the newest archive. You can also run it from Super → Backups or with cli.php backup:drill <name>. The drill:
- decrypts the archive and authenticates every chunk;
- checks each member's size and sha256 against the manifest;
- loads
db.sqlinto the scratch database<db>_restore_test(billing_restore_testby default); - compares the row count of every table with the count at dump time, and lists missing or extra tables;
- drops the scratch database.
Nothing live is touched.
Results are stored on the backup and shown on Super → Backups. A failure e-mails the super admins, at most once a day.
Privileges. The drill runs as the migrate user, which needs CREATE/DROP on <db>_restore_test:
- New installs get this grant automatically.
- After upgrading from 0.10, run once
php app/cli.php backup:grant --db-root-user=root. - On a cluster, the drill runs as the node admin with
sql_log_bin = 0, so the scratch database never replicates. Runbackup:granton each node; pairings made with 0.11 or later already include the grant.
6. Restore#
Restoring is done on the command line, on a single server. On a cluster, restore onto one server with the cluster disabled, then pair the second server again (CLUSTER.md).
sudo -u www-data php app/cli.php backup:list
sudo -u www-data php app/cli.php backup:verify billing-20261110-023700 # read everything, change nothing
sudo -u www-data php app/cli.php backup:restore billing-20261110-023700 --yes [--db-only | --files-only] [--with-config] \
[--db-root-user=root --db-root-pass=…]
The restore runs in this order:
- Check. Everything is decrypted and checked first. Any problem stops the restore before anything changes.
- Database. The dump is loaded into
<db>_restore_tmpand swapped in with oneRENAME TABLE. The replaced tables stay in<db>_prevuntil the next restore. Grants are refreshed.- The migrate user may lack rights on
<db>_restore_tmpand<db>_prev. In that case pass a root account with--db-root-user.
- The migrate user may lack rights on
- Files.
data/uploads,data/geoanddata/downloadsare replaced. The previous directories move todata/cache/restore-prev-<time>/. - Configuration.
- Without
--with-config, the archivedconfig.phpis written next to the live one asconfig.php.restored, for you to compare. The live file can differ on purpose: new database passwords, a new base URL. - With
--with-configit replaces the live one, and the old one is kept asconfig.php.prev-<time>.
- Without
- Mode check. An archive made in wrapped-blobs mode restores files only into a wrapped install, and the reverse. Use
--db-onlyotherwise.
After a restore, run php app/cli.php selfcheck.
Restore on a new server:
- Install billing with the same database name.
- Put the old master key in
data/keys/master.key(mode 0600) before the first request. - Copy the archive into
data/backups/. - Run
backup:restore <file> --yes --with-config, then fixconfig.phpif the database users changed.
7. Off the server#
Every archive sits on the server until SFTP and S3 targets arrive in a later release. Super → Backups and the self-check (backup_offbox) show a reminder until an off-box copy is recorded. These count as an off-box copy, for 8 days:
- Downloading an archive on Super → Backups. The download is audited and the response carries the archive's sha256 in
X-Backup-Sha256. - "I made an off-box copy" on the same screen, for copies made with scp or rsync.
To open an archive anywhere, you need only PHP and the master key file:
php app/cli.php backup:decrypt billing-20261110-023700.bkp out.tar --key-file=master.key
tar -tf out.tar # db.sql, config.php, files/…, manifest.json
backup:extract <name|file> <empty dir> decrypts and unpacks in one step, checking every member against the manifest.
8. Alerts and checks#
| What | Where |
|---|---|
| Newest good backup older than 26 h | Self-check backup_freshness = fail; hourly backups.watch e-mails super admins (once a day) |
| A backup run failed | Row marked failed with the reason; e-mail; status fail until a run succeeds |
| Restore drill failed | E-mail; status warn |
| No off-box copy in 8 days | Self-check backup_offbox = warn; reminder on Super → Backups and the platform overview |
| Not enough disk space (< 2 × the estimated size) | The run refuses to start, with the numbers |
A fresh install with no backup yet shows warn, not fail.
9. CLI reference#
| Command | |
|---|---|
backup:run | Back up now, then apply retention. |
backup:list [--json] | Archives on this server, the GFS slots that keep them, the drill result, and the freshness status (exit 1 on fail). |
backup:verify <name|file> | Decrypt and check every member; no database changes. |
backup:drill [name] | Restore drill (default: the newest archive). |
backup:extract <name|file> <dir> | Decrypt and unpack. |
backup:decrypt <file> <out.tar> [--key-file=…] | Decrypt to a plain tar. Works without an install. |
backup:restore <name|file> --yes […] | §6. |
backup:grant --db-root-user=… [--admin-host=…] | Give the drill its scratch-database rights (upgrades from 0.10). |
GeoIP files
Super → GeoIP lists every .mmdb file the platform knows, which one is active for each kind (City, Country, ASN), its build date and size, and where it came from. Getting the files from MaxMind is described in GeoIP files from MaxMind.
- Upload checks a file before using it: it must open, be a City, Country or ASN database and answer test lookups. On a two-server pair the file is copied to the other server automatically.
- For each kind, the newest build wins. Older files stay listed as not used; delete them when you no longer need them.
- A file older than 60 days is marked stale, and the self-check warns.
- Look up an IP address to see what the files return for it — useful to check a new build.
Without GeoIP files everything works; payments and page views simply carry no location.
Audit log and acting as someone
The log is append-only: the app's database user can only add rows (since v0.2.0).
- Platform (Super → Audit log):
- filters: workspace slug, who (super admin, member, customer, system), action prefix, search (email, id, IP), date range, impersonation only;
- 50 entries per page;
- each entry has a page with a field-by-field before/after table;
- CSV export.
- Workspace (Settings → Audit log, owners):
- same filters, always bound to the workspace;
- entries made by super admins appear as Platform admin, without their email.
Start. On a workspace page:
- Act as next to a member;
- Find a customer → Act as next to a customer.
A reason (ticket number or note) is required and written to the workspace's audit log.
The session:
- uses the normal member or customer cookie, marked with the super admin's id;
- lasts at most 1 hour, with the usual idle limit;
- ends when the super admin signs out or is disabled;
- does not appear in the person's own device list.
While acting:
- A banner shows who you are acting as, with a Stop button. Stopping returns you to the workspace page in the super area.
- Every action is logged under the person, with
impersonator_idset to the super admin. - Blocked, because they would change the person's own sign-in:
- member password changes;
- signing out one session or every session;
- saving the customer profile;
- customer sign-out-everywhere.
Health, jobs, self-check and system monitor
Health & jobs#
- Scheduler — whether jobs are driven by system cron, by the cron URL, or by page views (pseudo-cron), and when it last ticked.
- Jobs — every job with its schedule, last run, result and next run. Run now runs one job immediately.
- Recent runs — results and errors.
- Email outbox, Database, Disk and Routing (clean URLs or entry files).
Jobs run only on the writable server of a pair.
Self-check#
Self-check runs the deployment and security checks: PHP and MariaDB versions, extensions, Argon2id, app/ and data/ outside the webroot, key and config permissions, database users and least-privilege grants, security headers, that webhook endpoints refuse forged signatures, backup freshness and off-box copies, test credentials on live products, licensing keys, GeoIP freshness, the static SEO files, the help pages and the bundled fonts.
Each check is Pass, Warn or Fail, with the detail needed to fix it. It runs daily (failures are emailed to super admins) and from the command line:
sudo -u www-data php app/cli.php selfcheck # exit code 1 when anything fails
Security checks#
| Check | What it proves | If it does not pass |
|---|---|---|
| Every route requires the right sign-in, capability and CSRF check | The route table follows the access rules: private areas need their sign-in, state-changing routes keep the form token, no write is open to the read-only role. | Only after a local code change: undo it, or copy the release again. |
| Forged form submissions are refused | A form from another site, without the token or without an Origin is rejected by the running code; signed in, only the token of that session is accepted. | Copy the release again. |
| Session cookies are HttpOnly, Secure, SameSite and host-bound | The cookie settings used over HTTPS. | Copy the release again. |
| PHP runtime settings | display_errors and expose_php are off, allow_url_include is off. Read from PHP-FPM, so run it from this page. | Set them in the FPM php.ini; see Hardening the server. |
| Application code is read-only for PHP | The PHP user cannot change app/ or the entry scripts. | sudo chown -R root:root app public, then give back the five SEO files. |
| No stray files in the webroot | No archives, database dumps, keys, logs, .git folders or phpinfo.php under public/. | Move or delete the files it names. A release zip unpacked and left in the webroot is the usual cause. |
| Shipped code matches the release manifest | Every file of app/ and public/ equals the release (app/MANIFEST.sha256), no foreign PHP file sits next to them, and the manifest carries a valid release signature (app/MANIFEST.sig). Warns when the signature file is missing. | See the list of files, copy the release again, and find out why if you did not change them. php app/cli.php security:manifest prints every difference. A bad signature means the manifest was rewritten: treat the server as tampered with. |
| Two-step sign-in for staff | Two-step sign-in is required for super admins and every active super admin has it; says how many workspaces require it from their members. | Super → Security; see Super-admin sign-in and security. |
| Scheduler key strength | The cron URL key has at least 32 random characters. | Put a longer cron.key in data/config.php. |
| Outbound connection safeguards | Certificates of the payment providers are verified; webhook endpoints must be public https addresses. | Remove the test-only settings it names from data/config.php. |
| data/ is closed to other system users | Other accounts on the server cannot list or read data/. | sudo chmod o-rwx on the directories it names. |
| Client addresses are read correctly | Requests do not arrive through a proxy that the application does not know. Read from the request, so run it from this page. | Add the proxy to http.trusted_proxies; see Hardening the server. |
| Private areas refuse anonymous requests (live) | Through the web server: the workspace and platform areas answer with the sign-in page, the API with 401, the scheduler with 403 for a wrong key. | The web server sends requests somewhere else than this install: compare the vhost with the shipped example. |
| Dispute evidence carries the wrapped-blobs guard | Only on installs with data/ inside the webroot. | php app/cli.php evidence:wrap. |
| Platform apps and connected accounts | No set in service depends on a disabled platform application; every platform webhook endpoint refuses a forged signature. Warns for a verified application without its webhook value, for a connected set whose provider could not be asked three times in a row, and for an access token within five days of running out. Skipped when no platform application exists. | Enable the application or disconnect the sets it names; add the webhook value; see Connected accounts. |
System monitor#
System monitor shows the server's vitals — CPU, memory, swap, disks, open files, time sync, TLS certificate expiry, reboot needed, automatic updates — PHP and OPcache, MariaDB (buffer pool, connections, slow queries, temporary tables on disk, largest tables) and the application (queues, open checkouts, webhooks today). Recommendations turn findings into concrete settings and the exact commands to apply them. Critical findings are emailed to super admins at most every 6 hours.
External monitoring#
Point an uptime monitor at /health (/index.php?r=health on a plain vhost). It answers 200 {"status":"ok"} when the database, data directory and scheduler are fine, and 503 otherwise.
Two-server redundancy
Spec: amendment v0.3.0-HA (SPEC-AMENDMENT-v0.3.0-HA.md). Run HA1 (v0.8.0) delivered replication, the write gate, the node agent with leadership lease and self-fencing, rejoin (clean / errant / gap) with quarantine and webhook replay, reseed, switchover and failback, and the node:* CLI. Run HA2 (v0.9.0) adds file sync (H4, §12), the degraded lease mode for licensing (H8, §13), the Super → Cluster screen with alerts and self-check items (H11, §14), and the Nginx fallback config (H7, docs/nginx-upstream-backup.conf, §9).
A single-server install needs none of this and behaves exactly as before.
1. What you get#
| Event | What happens |
|---|---|
| Replica fails and comes back | It catches up from the primary. It reseeds only if it was away longer than binlog retention (7 days). |
| Primary fails | The primary's lease expires (15 s). After a further 10 s the replica promotes itself with epoch + 1. Measured on the test box: 17–25 s from failure to the replica answering /health/write with 200. |
| Old primary comes back | It starts read-only and sees the newer epoch. It rejoins as a replica, catches up, and quarantines any writes the new primary never got. |
| Preferred primary is healthy again | With a witness and preferred_primary, a planned switchover moves the primary back once that node has been in sync for auto_failback_after seconds. No data is lost. |
| A cannot reach the witness | A fences itself within 15 s (MariaDB read_only=ON, and the app refuses writes). B promotes after the lease expires. When the network heals, A rejoins. |
| No witness configured | Nothing promotes automatically. Use node:promote (or Promote standby on Super → Cluster). Nodes never fence for lack of a witness. |
| Files uploaded on the primary | Logos, dispute evidence, product binaries and GeoIP files reach the replica within ~10 s (sha256-checked); a full reconcile runs on rejoin, after a reseed and daily. |
| Primary unreachable, licensing | The replica answers /api/v1/license/* in degraded mode: same signing key, TTL ≤ 1 h, "Service is running on the standby server"; its writes are spooled locally and replayed exactly once when a primary is back or it is promoted. |
2. Requirements#
- Two machines (A, B) on a low-latency private link. Semi-sync waits for B on every commit.
- The same billing version on both.
- MariaDB 10.11+ (built-in semi-sync,
READ_ONLY ADMINprivilege). - NTP on both nodes and on the witness: clocks within 2 s.
- A third vantage point for the witness: webgate (contract in amendment H7), or until then
app/tools/fake_witness.phpon a third machine. - PHP with
proc_openenabled, if you wantmariadb-dumpand decoded quarantine reports. Without it the built-in PHP dumper is used and reports list GTIDs only.
3. MariaDB settings (both nodes)#
Print the block for a node with php app/cli.php node:mycnf --node=A (or B), put it in /etc/mysql/mariadb.conf.d/60-billing-ha.cnf, and restart MariaDB:
[mariadbd]
server_id = 1 # A = 1, B = 2
bind_address = 10.0.0.11 # private address
log_bin = billing-bin
relay_log = billing-relay
relay_log_recovery = ON
log_slave_updates = ON
binlog_format = ROW
gtid_domain_id = 0
gtid_strict_mode = ON
binlog_expire_logs_seconds = 604800
sync_binlog = 1
innodb_flush_log_at_trx_commit = 1
rpl_semi_sync_master_enabled = ON
rpl_semi_sync_slave_enabled = ON
rpl_semi_sync_master_timeout = 2000
rpl_semi_sync_master_wait_point = AFTER_SYNC
replicate_ignore_db = billing_local
read_only = ON
ssl_cert = /etc/mysql/ssl/server-cert.pem
ssl_key = /etc/mysql/ssl/server-key.pem
ssl_ca = /etc/mysql/ssl/ca.pem
read_only = ONat startup is deliberate. A restarted node takes writes only after its agent has confirmed the lease (or, without a witness, that the peer is not a newer primary). Install billing first, then apply the block. If the block is already active during the install, runSET GLOBAL read_only=OFFfor the duration of the install.- Replication runs over TLS with the certificates above. Give the replica the CA with
--repl-ssl-ca, and add--repl-ssl-verifyto check the server certificate (its name must match the address the replica connects to).
4. Pairing#
Node A (installed and serving)#
sudo -u www-data php app/cli.php node:init --node=A --name=encrypted \
--peer-url=https://10.0.0.12:8443 --peer-host=10.0.0.12 --peer-db-port=3306 \
--db-root-user=root --witness=https://webgate.internal/witness --secret=<64 hex, same as on webgate> \
--preferred=A --failback-after=600 --repl-ssl-ca=/etc/mysql/ssl/ca.pem --repl-ssl-verify
sudo systemctl enable --now billing-node-agent
node:init does the following:
- creates
billing_node(the node admin: replication control and the local schema) and[email protected](REPLICATION SLAVE only); - creates the node-local schema
billing_local; - writes the
clusterblock intodata/config.php; - takes the first lease (epoch 1) and makes A primary;
- prints a one-time pairing token, valid 1 h.
node:tokenissues a new one.
All of this runs with sql_log_bin = 0: users, grants and local tables never replicate.
Node B (code deployed, not installed)#
sudo -u www-data php app/cli.php node:join --node=B --peer=https://10.0.0.11:8443 --token=<token> \
--db-root-user=root --db-host=127.0.0.1 --db-port=3306 --peer-db-host=10.0.0.11
sudo systemctl enable --now billing-node-agent
node:join does the following:
- Fetches the pairing bundle over the node channel. The request is authenticated with a key derived from the token, and the bundle is sealed with another token-derived key.
- Installs the same master key. If one already exists, its fingerprint must match, or the join stops.
- Creates the same database users with the same passwords, plus its own
billing_nodeandbilling_repl@A. - Writes the config.
- Reseeds from A (snapshot streamed over the node channel) and starts replicating.
If B reaches the witness under a different address, add --witness=….
5. Node channel (private vhost)#
Every request to /__node/<op> must come from an address in cluster.channel_allow (default: the peer's address) and carry an HMAC header (X-Billing-Ts, X-Billing-Nonce, X-Billing-Sig; ±30 s; single-use nonce).
Serve it only on the private link: nginx-node-channel.conf. On the public vhost, add location ^~ /__node/ { return 404; } (included in the example), and do not route it in webgate.
Operations:
| Operation | Purpose |
|---|---|
status | Node state for the peer |
dump | Reseed snapshot (served by the primary only) |
switchover | Planned switchover handshake |
event | Forwarded cluster history |
quarantine | Errant-transaction report |
missing / replay | Webhook comparison and replay |
pair | Pairing, authenticated by the token |
file (v0.9.0) | One synced file, streamed as stored (?path=uploads/…; only uploads/, downloads/, geo/) |
manifest (v0.9.0) | Per-directory file hashes for the reconcile (?dir=uploads) |
spool (v0.9.0) | Replay of the standby's licensing spool (writable node only) |
control (v0.9.0) | An action from the other node's Super → Cluster screen, queued for this node's agent |
6. The agent#
- Unit file:
billing-node-agent.service. It runs as the PHP-FPM user, so files indata/keep one owner. - Loop: every 2 s it observes this node, the witness and the peer, then executes one action from the decision table (
Cluster\AgentDecision, a pure function covered by unit tests). - Lease (primary): renewed every 5 s. After 15 s without a renewal the node fences:
SET GLOBAL read_only = ON;Node::writable()turns false and the app answers writes with 503 +Retry-After: 30;- jobs and pseudo-cron stop.
- Unfence: automatic, as soon as the witness grants the lease at the same epoch again.
- Promotion (replica): all of these must hold:
- the witness shows the lease free, with the previous holder expired for ≥ 10 s;
- the relay log is fully applied;
- the replica was in contact with the primary within 60 s and ≤ 5 s behind at that contact;
- the peer does not report itself writable.
Otherwise the agent holds and emails super admins how to force it.
- The application gate: every writer goes through
Db.Dbrefuses INSERT, UPDATE, DELETE and DDL unlessNode::writable(), a cached read (≤ 1 s) ofdata/cache/node.json, which the agent publishes. MariaDB'sread_onlyis the second, hard guard.tools/refcheck.phpfails the build if code outside the allowed files opens its own database connection.
7. Day-to-day#
| Command | Does |
|---|---|
node:status [--json] | Role, epoch, writable, GTIDs, replication threads, lag, semi-sync status and fallback count, witness holder and expiry, peer, last action, recent events |
node:switchover | Planned switchover. Writes pause for a few seconds, no data loss. The peer must be a healthy replica. |
node:promote [--force] [--wait=40] | Manual promotion. It refuses while the peer is a writable primary. Without --force it refuses if the replica is not current; --force prints the possible-loss window and proceeds. It waits for the witness lease to become free. |
node:reseed | Rebuild this replica from the primary |
node:failback pause|resume | Pause or resume automatic failback on this node |
node:quarantine [--handled=ID] | List errant-transaction reports, replayed webhooks and review items; mark one handled |
node:files [--run|--reconcile] [--json] | File sync on the replica: backlog, one pass now, or a full reconcile (§12) |
node:spool [--replay] [--json] | Licensing spool size; replay it now (§13) |
node:agent [--loops=N] | Run the loop in the foreground (systemd does this) |
Upgrades:
- Upgrade the replica first.
node:switchover.- Upgrade the old primary.
- Run
migrateon the primary. DDL and GRANTs replicate. On the replicamigrate(v0.9.0) only brings the node-local schema up to date and says so; the agent also does that when it starts.
v0.8.0 → v0.9.0: migration 0008 adds file_log and license_spool_applied (run on the primary); the local schema gains file_applied, node_kv, node_requests and the spool key column. Files already on both nodes are checked by the first reconcile after the upgraded replica's agent starts (nothing is fetched when they match).
8. When a node comes back#
The agent classifies the returning node against the current primary's GTID history:
- Clean. Every local GTID is on the primary:
CHANGE MASTER … MASTER_USE_GTID=slave_pos, catch up, and the node becomes a replica. - Errant. The node committed transactions the primary never received (the semi-sync fallback window right before a crash):
- A report
data/quarantine/<epoch>-<node>.sqllists every errant GTID with its tables and row counts. Withproc_openandmariadb-binlogavailable, it also holds the decoded row events. - Provider webhooks stored on the node that the primary does not have are replayed on the primary from their original payloads. This is idempotent on the provider event id.
- Everything else is listed for review:
node:quarantine, and the Super → Cluster screen (§14). - The node is reseeded. The replaced tables stay in
billing_prevuntil the next reseed.
- A report
- Gap. The primary purged binlogs this node still needs, or the node has no data: reseed.
Reseed:
- Take a consistent snapshot of the primary:
mariadb-dump --single-transaction --gtid, or the built-in dumper withSTART TRANSACTION WITH CONSISTENT SNAPSHOT+BINLOG_GTID_POS. - Stream it over the node channel. A truncated transfer is detected by a terminator line.
- Restore into
billing_reseedand swap it in with one atomicRENAME TABLE. RESET MASTER, then setgtid_binlog_state/gtid_slave_posto the snapshot, and replicate.
Both nodes down. Start the node with the newer epoch first (node:status shows it). The other one stays fenced until the primary is reachable, then rejoins.
9. Without webgate#
- Configure
--witness=none. Automatic promotion is then disabled: failover isnode:promoteon the survivor (or Promote standby on Super → Cluster). Everything else works, including the agent, rejoin, reseed, manual switchover, file sync and the degraded lease mode. Automatic failback is off. - Route traffic with plain Nginx:
nginx-upstream-backup.conf. A first, B asbackup; license calls are also retried on the other node when the first answers 503 (a fenced primary or a replica answers 503 without writing; a replica in degraded mode answers the lease). The two-node test harness builds its gate from this exact file. - Adding a witness later (webgate arrives after you ran without one, or
witnessgoes back fromnone): set it on both nodes and restart the agents, the primary's first. A fresh witness grants the primary its current epoch; a witness that knew the cluster before grants the previous holder its current, higher epoch. Any other node still has to present exactly stored + 1 (v0.9.0 rule,Cluster\WitnessRules; webgate must implement the same). - Or run the fake witness on a third machine:
WITNESS_DIR=/var/lib/billing-witness WITNESS_SECRET=<cluster secret> php -S 10.0.0.20:9100 app/tools/fake_witness.phpUse
--witness=http://10.0.0.20:9100/witnesson both nodes. It persists its state, so a restart keeps the holder.
10. Testing#
- Unit tests:
php app/cli.php test Cluster. They cover GTID comparison, HMAC replay and skew, the witness rules and their persistence, the decision table, the write gate on every writer, both dumpers, webhook replay, pairing, the node channel and trusted proxies. - Mutation proof:
php app/tools/mutate.php Cluster. - Two-node scenarios on one box (root, about 8 minutes):
sudo php app/tests/e2e/ha/e2e_ha.php. It runs two MariaDB instances (3307/3308), two PHP-FPM + Nginx stacks, the fake witness and the Nginx gate fromnginx-upstream-backup.conf. It covers scenarios 1–9 of amendment H12 plus scenario 10 (file sync through the real upload paths, replica outage, out-of-band files and reconcile, reverse direction, and the Super → Cluster actions). - Upgrade regression:
sudo php app/tests/e2e/upgrade/regress_upgrade.php --from=<previous release tree> --db-user=… --db-pass=…installs the previous release on a front-controller and a plain vhost, seeds both with its e2e suites, upgrades in place and runs this release's suites again.
11. webgate in front (trusted proxy)#
Set the addresses webgate connects from in data/config.php on both nodes:
'http' => ['verify_tls' => true, 'trusted_proxies' => ['10.0.0.5', '10.0.0.0/24']],
- Client IP: only for requests whose TCP peer is in that list,
X-Forwarded-Foris read right to left; the first address that is not itself a trusted proxy is the client. This IP is used for rate limits (including the per-IP lease limit), sessions, audit and logs. - HTTPS:
X-Forwarded-Proto: httpsmarks the request as HTTPS. - Untrusted peers: headers from anyone else are ignored.
Health checks for webgate's routing are GET /health/write, /health/lease and /health/read. With the plain vhost use /index.php?r=health/write. /health/lease is 200 on the writable primary, and on a replica in degraded lease mode whose lag at last contact was ≤ 300 s and whose data is no older than the smallest active license_lease_ttl_s / 4 (the body then carries "degraded": true and data_age_s).
12. File synchronization (H4)#
- What:
data/uploads/(logos, dispute evidence),data/downloads/(licensed binaries),data/geo/(GeoIP).data/backups/,data/logs/anddata/cache/stay per node. - How: every write or delete in those directories goes through
Billing\Core\FileStore, which writes atomically and, on a cluster node, inserts a row into the replicated tablefile_log(path as stored, sha256 and size of the stored bytes, op, epoch, node). A non-writable node refuses file writes before touching the disk. - Fetch: the replica's agent, every 10 s, takes the
file_logrows not yet inbilling_local.file_applied, keeps the newest row per path, fetches the file from the primary (/__node/file, HMAC-signed, streamed), checks the sha256, moves it into place atomically and marks the row applied. A delete row removes the file. A file whose hash does not match (the primary changed it again) stays pending until the next row arrives. - Reconcile (after a rejoin, after a reseed, after a switchover, daily, or
node:files --reconcile): the replica compares per-directory manifest digests with the primary (/__node/manifest). Where they differ it fetches every missing or different file — also files copied onto the primary by hand, such as a GeoLite2 database — and moves files the primary does not have todata/cache/file-extras/<time>/(never deletes them). Then every log row up to the watermark read before the manifests counts as applied. - Backlog:
node:status,node:files, the Cluster screen and the self-check show it; more than 100 files or an oldest one older than 5 minutes is a warning and an alert. file_logkeeps 30 days (files.prunejob); the daily reconcile covers anything older.- Both nodes must use the same data layout (
wrappedon or off): the self-check fails otherwise.
13. Licensing under HA (H8)#
- Normal operation:
/api/v1/license/*goes to the primary. A replica whose primary is reachable answers 503, which the client treats as "unreachable" (it keeps serving on its lease). - Degraded lease mode (
licensing.degraded_mode, on by default): when this node is a replica and no writable primary is reachable, its agent publisheslease_degraded, and the replica answers leases from replicated data:- the decision code is unchanged and signs with the same key (
product_signing_keysreplicates; the master key is identical), so leases from both nodes verify with the same public key; - TTL = min(product TTL,
licensing.degraded_ttl_cap_s, default 3600), andmessage= "Service is running on the standby server"; - it refuses (503) when its data is older than the product's TTL / 4 or its lag at last contact was above 300 s;
- every write — instance upsert,
last_seen_at, denials (one per license × instance × reason per hour), release — goes tobilling_local.license_spool(ULID key, timestamp); the per-IP / per-key rate limits count inbilling_local.rate_limits_local; - seats = replicated active instances (minus those released here) + instances spooled here.
- the decision code is unchanged and signs with the same key (
- Replay: when a writable primary is reachable again, the agent posts the spool to it (
/__node/spool, 200 rows per batch); after a promotion it applies the spool locally. Rows run through the normalLicenseWriterin timestamp order withlast_seen_at = max(), an older write never overwrites a newer row, and every event's detail ends with "recorded on standby". Each row's key goes into the replicatedlicense_spool_appliedin the same transaction, so a batch delivered twice applies once. Confirmed rows are deleted from the spool;cluster_eventsgetsspool-replayed. - Downloads work on either node once it is writable: the binaries are synced (§12).
- Interop: the HA scenario 8 saves one envelope signed by A and one signed by B and verifies both offline with
app/tests/interop/verify_envelope.go.
14. Super → Cluster (H11)#
- Node cards for both nodes: role and epoch, version and master-key fingerprint (with match), replication threads and lag, semi-sync status and fallback count, last GTID, file backlog, licensing spool, lease expiry, last action; the witness holder and expiry above.
- Actions, each confirmed by typing a phrase (
SWITCHOVER,PROMOTE B,RESEED B,PAUSE FAILBACK,RESUME FAILBACK) and audited:- Switchover — runs on the current primary; needs a healthy replica.
- Promote standby — only when no node is a writable primary; shows the possible-loss window; "accept data loss" =
--force. - Reseed node — never the primary.
- Pause / resume auto-failback — on both nodes.
The screen never changes roles inside the web request: it queues the action in the target node's
billing_local.node_requests(directly, or through the peer's/__node/control), and that node's agent runs it on its next loop. The result shows under Requests and in the history. On the writable node the action is written toaudit_log; on the standby to the cluster history. - History:
cluster_events(promoted, fenced, unfenced, rejoined-clean, rejoined-errant, reseeded, switchover, failback, spool-replayed, files-reconciled, ui-…), with durations. - Quarantine: reports, replayed webhooks with their results, review items with Mark handled (writable node only).
- On the standby: a super admin already signed in (sessions replicate) can open the screen and queue actions there; signing in needs the writable node.
- Alerts (email to super admins, de-duplicated, 15 min): failover, fencing, rejoin with errant transactions, replication lag > 60 s for 5 minutes, file backlog over the thresholds, witness unreachable for 30 s, version or master-key mismatch between the nodes, reseed failure.
- Self-check items (Super → Self-check,
cli.php selfcheck): replication running with lag < 30 s, semi-sync active, same version, same master key, same data layout, witness reachable, clock offset to the peer < 2 s, binlog retention ≥ 7 days, file backlog, licensing spool replayed.
15. Backups and API v1 on a pair (Run 6)#
- Backups. The archive job (
backups.nightly), the drill (backups.drill) and the freshness watch run on the writable node only, like every job. Each node keeps its owndata/backups/(H4), and archives are named with the node id (billing-…-a.bkp). Thebackupsrows replicate, so after a switchover the screen lists the other node's archives as "not on this server".cli.php backup:runon a replica refuses. - Restore drill. On a cluster the drill runs as the node admin with
sql_log_bin = 0, so<db>_restore_testnever reaches the other node and creates no errant transaction. Pairings made with 0.11 grant the node admin that schema. After upgrading an existing pair, runcli.php backup:grant --db-root-user=root [--admin-host=…]on each node. - Restore (
backup:restore) is refused on a cluster node. Restore onto one server with the cluster disabled, then pair the second server again. - API v1 is answered by the writable primary; a replica answers 503
unavailablewithRetry-After, because each request writes its rate-limit counter. Keys, webhook endpoints, events and deliveries replicate. The delivery job (webhooks.deliver) runs on the primary. - Two-node e2e scenario 11 covers these points (
app/tests/e2e/ha/e2e_ha.php --scenarios=11).
16. Help and SEO on a pair (Run 7)#
- Help pages are served by both nodes. Edits made in Super → Help & SEO are rows in
help_overridesand replicate, so the standby shows them too. Page views of the help are recorded by the writable node only (the standby serves its pages without writing). - Static SEO files (
robots.txt,sitemap.xml,llms.txt,llms-full.txt,site.webmanifest) live in each node's own webroot. Theseo.publishjob runs on the writable node, so the standby keeps the files it had (or the shipped ones) until it is promoted; its first hourly run then refreshes them. - Open Graph images are cached per node in
data/cache/og/and rendered on demand on either node. - Two-node e2e scenario 12 covers these points (
app/tests/e2e/ha/e2e_ha.php --scenarios=12).
Licensing across workspaces
Super → Licenses lists the licenses of every workspace, with filters for workspace, product, status and customer. A license's page shows its key state, seats, expiry, features and every machine that leased it, with the latest lease events.
The actions are the same as in a workspace — suspend, resume, revoke, change seats or expiry, regenerate the key, and remove, block or unblock a machine — and each is recorded in that workspace's audit log as a platform action.
The self-check verifies that every active leased product has an active signing key and a download. How licensing works end to end is in the workspace manual: Licensing.
Platform analytics and insights
Super → Analytics shows the same figures as a workspace's analytics, for the whole platform and per workspace: gross, net, refunds, disputes, refund and dispute rates, MRR and subscriptions, and the platform's own pages (the landing page and the help). Workspaces whose dispute rate passes 0.75% are highlighted; above 1.0% super admins receive an alert.
Super → Insights runs the same analysis across every workspace: period-over-period changes, the 30-day trend and forecast, unusual days, concentration of revenue in a few products, refund and dispute rates against their thresholds, MRR, churn, free-to-paid conversion and lifetime value. Every recommendation shows the numbers it is based on.
How each figure is counted is explained in Analytics and insights. All amounts are USD and all days are UTC.
Help, search engines and AI crawlers
Super → Help & SEO controls how the public side of the platform presents itself.
Search and social#
- Site description — shown under the landing page in search results, in link previews and at the top of
llms.txt. - AI crawlers — Allow lets assistants such as ChatGPT, Claude and Perplexity read the public pages; Block tells their crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot and others) to stay away. Search engines are not affected either way.
What every public page carries#
- a title, a description and a canonical address;
- Open Graph and Twitter tags with a generated 1200×630 image (the product, its price and the workspace's accent colour);
- structured data (JSON-LD): Organization and WebSite on the landing page, Product with an Offer per plan and breadcrumbs on product pages, FAQPage and HowTo on help pages where they apply.
Private pages — account, receipts, checkout hand-off, payment links, staff screens — are marked noindex and listed as disallowed in robots.txt.
The static files#
robots.txt, sitemap.xml, llms.txt, llms-full.txt and site.webmanifest are regenerated every hour by the seo.publish job and written into the webroot, so a plain PHP vhost serves them at the usual addresses. Publish now runs it immediately.
If the webroot belongs to root (the recommended layout), give the PHP user just these five files once:
cd /var/www/encrypted.com/public
sudo touch robots.txt sitemap.xml llms.txt llms-full.txt site.webmanifest
sudo chown www-data:www-data robots.txt sitemap.xml llms.txt llms-full.txt site.webmanifest
Until then, the routes answer at the same addresses wherever Nginx passes unknown paths to the application, and the self-check shows a warning.
Editing help pages#
Every help page ships with the release. Edit next to a page opens the editor:
- change the title, the description and the text (Markdown: headings, lists, tables,
**bold**, `code,> Note:callouts, links such astext, and diagrams with{{diagram:name}}`); - Preview shows the page as visitors will see it; Save publishes it; Revert to shipped text removes your edit.
Edits are kept in the database, so upgrades never overwrite them. When an upgrade changes the shipped version of a page you edited, the editor warns you so that you can merge the changes. Every save and revert is recorded in the audit log. HTML in the text is shown as text, never run.
Super-admin sign-in and security
Signing in#
Super admins sign in at /super/login. Passwords are hashed with Argon2id. After 5 wrong passwords in 15 minutes the account is locked and an unlock link is emailed; 50 failures from one address within 15 minutes block that address for a while.
Sessions end after 30 minutes without activity and after 12 hours in any case. The session id changes at sign-in and whenever privileges change.
Two-step sign-in#
Profile → Two-step sign-in → Set up adds a code from an authenticator app after the password, with ten single-use recovery codes. The steps are the same as for workspace members: see Two-step sign-in and security settings.
Super → Security → Require two-step sign-in from every super admin makes it compulsory. Turn it on for yourself first. Super admins without it are signed out and set it up at their next sign-in. The page also lists every super admin with their status, and how many workspaces require it from their members.
A super admin who lost phone and recovery codes is reset by another super admin on Super → Security (Reset), or on the server:
sudo -u www-data php app/cli.php staff:2fa-reset [email protected] # a super admin
sudo -u www-data php app/cli.php staff:2fa-reset [email protected] --workspace=tulix # a workspace member
sudo -u www-data php app/cli.php security:super-2fa on # require it (on | off | status)
The self-check item Two-step sign-in for staff warns until it is required for super admins and every super admin has it.
Profile#
Profile (your name at the bottom of the menu) changes your password and lists your signed-in devices; sign out one or all of them. Signing out also ends any acting as sessions you started.
Command line#
sudo -u www-data php app/cli.php super:create [email protected] "Your Name" # add a super admin
sudo -u www-data php app/cli.php super:reset-password [email protected] # new password, signs out every session
What the application enforces#
- Tenant isolation — every query on workspace data carries the workspace; an id from another workspace answers "not found".
- Roles and product access — the finance viewer cannot change anything; an admin limited to some products cannot open, list or export anything of the others; API keys follow the same limits.
- Forms — every state-changing request needs the form token and must come from this site. Signed in, the token is derived from the session, so a token planted in a cookie or taken from another session is refused.
- Customer registration — by default the form never tells whether an address already has an account; the mailbox gets the link (see How customers register).
- Version number — anonymous visitors do not see which release runs (
/health, sign-in pages, help, robots.txt, the API reference). Signed-in super admins see it at the bottom of the menu and on/health. - Webhooks — a provider event acts only on payments made with a credential set of the same provider and the same mode: an event signed by a test set never touches a live payment.
- Customer accounts — an email address counts as confirmed only when the link was opened inside that account; an unconfirmed account never receives a purchase somebody else pays for (see Customers and access).
- Password forms — the current password can be tried five times in 15 minutes on the change-password forms, for staff and customers alike.
- Receipts — looking up confirmation codes that open nothing is limited to 30 tries in 10 minutes per address.
The test suite walks the whole route table for each of these rules; docs/SECURITY.md in the release package lists the tests and the review they came from.
Checking the installed code#
php app/cli.php security:manifest # compares app/ and public/ with app/MANIFEST.sha256, checks its signature; exit 1 on a difference or a bad signature
php app/tools/verify_release.php /path/to/unpacked/release # before installing: signature, then every file
The manifest is signed (app/MANIFEST.sig, Ed25519) with the release key, which never leaves the publisher's workstation. A changed file then needs a new manifest, and a new manifest breaks the signature. verify_release.php prints the key id: compare it with the release key id published outside the package (docs/SECURITY.md §5 and the delivery report). With --key=<public key> it trusts only the key you hold.
The self-check runs the same comparison daily and fails on a bad signature.
Good practice#
- Give each operator their own super-admin account; the audit log then says who did what.
- Keep the master key (
data/keys/master.key) off the server, in a password manager or safe, and never next to the backups. - Use the hardened vhost with TLS, so cookies are
__Host-,SecureandSameSite=Lax, and HSTS applies. - Run the self-check after every change to the server.
Upgrading
The routine#
- Take a backup (Super → Backups → Back up now) and download it.
- Unpack the new release next to the old one, then replace
public/andapp/. Never replacedata/: it holds your configuration, keys, uploads and backups. - Apply migrations and refresh the grants:
cd /var/www/encrypted.com
sudo -u www-data php app/cli.php migrate --db-root-user=root # asks for the root password; or run as the migrate user
- Do the version-specific steps below, if any, for every version you skipped.
- Run the self-check, open the Platform overview, and make one test payment in a test workspace.
On a two-server pair, upgrade the replica first, then the primary; the cluster page shows both versions.
Version-specific steps#
- Replace
public/andapp/(keepdata/). There is no migration and no new setting;php app/cli.php migrateonly refreshes grants. - Confirm the copy:
php app/cli.php security:manifestcompares every file withapp/MANIFEST.sha256and must end with "match the release manifest". Files it lists asEXTRAare PHP files of older versions or of your own; remove what you do not recognise. - Run the self-check from Super → Self-check (not only from the command line: the PHP-FPM settings and the proxy check are read from the web request). New items are listed in the help under Health, jobs, self-check. Typical first-run findings:
- Application code is read-only for PHP:
sudo chown -R root:root app public, then step 3 of §9i again for the five SEO files. - Client addresses are read correctly: behind webgate or another proxy, add its address to
http.trusted_proxiesindata/config.php. - No stray files in the webroot: delete the release archive or dumps left under
public/.
- Application code is read-only for PHP:
- Wrapped-blobs installs only (
data/inside the webroot):sudo -u www-data php app/cli.php evidence:wrapre-stores dispute evidence uploaded before 1.0 with the PHP guard. - Behaviour that changes, so support knows:
- The "confirm your email" link now needs the customer to be signed in. Links already sent keep working, after sign-in.
- A customer account whose address was never confirmed loses its password when a payment for that address arrives without sign-in, when staff grant access or record an offline payment for it, or when an API key creates a checkout for it. The customer sets a new password through the emailed link. Confirmed accounts are unaffected.
- A provider event signed by a credential set of another provider, or of the other mode (test / live), no longer acts on a payment; it is answered 200 and logged as
webhook.source_mismatch. If a product was moved from a test set to a live set of a different provider account while test payments were still open, those test payments now stay as they are. - Receipt lookups that open nothing answer 429 after 30 tries in 10 minutes from one address; five wrong "current password" entries close the change-password form for 15 minutes.
- Replace
public/andapp/(keepdata/). New files:public/assets/fonts/(bundled OFL fonts and their licences),public/assets/icons/,public/favicon.ico,public/site.webmanifest, a new defaultpublic/robots.txt, andapp/help-content/,app/fonts/. sudo -u www-data php app/cli.php migrate— migration0011_seo_help(help page edits;pageviews_raw.pagegainshelp).- Let PHP keep the SEO files current while the webroot stays root-owned (once):
cd /var/www/encrypted.com/public sudo touch sitemap.xml llms.txt llms-full.txt sudo chown www-data:www-data robots.txt sitemap.xml llms.txt llms-full.txt site.webmanifestWithout this step everything works; on a plain vhost
/sitemap.xmland/llms.txtare then not served, and the self-check warns (seo_files). - Super → Help & SEO: write the site description, choose the AI-crawler policy, press Publish now.
- Self-check:
seo_filesandhelp_contentshould pass. Nginx needs no change.
sudo cp -r app public /var/www/encrypted.com/ # keep data/ untouched
sudo -u www-data php /var/www/encrypted.com/app/cli.php migrate # applies 0010_api_backups + refreshes grants
sudo php /var/www/encrypted.com/app/cli.php backup:grant --db-root-user=root # once: the restore drill's scratch database
sudo -u www-data php /var/www/encrypted.com/app/cli.php backup:run # first backup now (otherwise tonight at 02:37 UTC)
- Backups go to
data/backups/nightly. The restore drill runs on Sundays, and GFS keeps 7 daily, 4 weekly and 12 monthly archives. Copy archives off the server (Super → Backups → Download, or scp), and keep the master key (data/keys/master.key) in yet another place. The self-check warns until an off-box copy is recorded. Details:BACKUPS.md. - Disk: an archive is roughly the size of the database plus uploads, GeoIP files and downloads. A run refuses to start below twice that much free space.
- API v1 and webhooks: Workspace → Settings → API & webhooks. Reference at
/api/docs, OpenAPI at/api/v1/openapi.json. Details:API.md. Nothing changes for existing pages./api/is already disallowed inrobots.txt. - Plain vhost (no rewrites): API URLs are
/api/?r=v1/…. PATCH/DELETE have POST aliases, and also work against/api/index.php?r=v1/…. - A pair: upgrade as in §9f. Run
backup:granton each node (--admin-host=if the node admin account is not@localhost). Backups and webhook deliveries run on the writable node, and each node keeps its owndata/backups/.
Older versions: see docs/INSTALL.md in the release package.
Troubleshooting
Where to look first#
- Self-check — most configuration problems show up here with the fix.
- Health & jobs — scheduler, jobs, outbox and their last errors.
- Logs —
data/logs/(one JSON line per event). Errors shown to users carry a Reference that matches a log line. - Audit log — who changed what, and when.
Common problems#
Payments stay "Pending" or "Confirming your payment"#
The provider's webhook is not arriving or is refused. Check the endpoint in the provider's dashboard and its delivery log, and see When payments stay pending.
No email arrives#
Check Super → Settings → Email (transport and key set?), send a test message, and look at the outbox for the error. Check the recipient is not on the suppressed list. See Email through SMTP2GO.
Jobs do not run#
Health & jobs shows the scheduler's last tick. Without system cron, jobs run only after page views. Add the cron entry from the Server guide.
Links look like /admin/?r=… instead of /admin/…#
The site runs on a plain PHP vhost, or the application has not yet seen a clean URL work. Both forms are correct. For clean URLs use the hardened vhost with try_files … /index.php; the hourly routing probe notices the change.
Sign-in returns to the sign-in page#
The session cookie is not kept. With HTTPS, the base URL in data/config.php must start with https://, and the site must be opened at that exact host. Check that the server clock is right.
An upload is refused#
Logos are limited to 512 KB, dispute evidence to 5 MB, GeoIP files by PHP's upload_max_filesize. Larger GeoIP files can be copied into data/geo/ instead.
sitemap.xml or llms.txt is missing on a plain vhost#
The webroot is not writable for PHP, so the static copies were not written. See Help, search engines and AI crawlers.
A page shows "Something went wrong on our side. Reference …"#
Search data/logs/ for the reference. It names the error and where it happened.