billingHelp

Backups and restore

Nightly encrypted archives, retention, the weekly restore drill, downloading copies, restoring, and the backup alerts.

Nightly backup and weekly restore drill Every night the database is dumped as the backup user and packed into a tar archive together with config.php, uploads, GeoIP files and licensed downloads, plus a manifest with the size and SHA-256 of every member and the row count of every table. The archive is encrypted with AES-256-GCM in chunks, with a key derived from the master key, and written to data/backups. Retention keeps 7 daily, 4 weekly and 12 monthly archives. Every week the drill decrypts the newest archive, loads it into a scratch database, compares every table's row count and every file's checksum, and drops the scratch database. Download archives regularly to keep a copy off the server; keep the master key off the server too. Dumpbackup DB user Tar + manifestfiles, counts, sha256 AES-256-GCMkey from master key data/backups/GFS 7 · 4 · 12 Off-box copydownload / copy weekly Weekly restore drilldecrypt → scratch DB → compare counts and files
Nightly backup and weekly restore drill
Text description

Every night the database is dumped as the backup user and packed into a tar archive together with config.php, uploads, GeoIP files and licensed downloads, plus a manifest with the size and SHA-256 of every member and the row count of every table. The archive is encrypted with AES-256-GCM in chunks, with a key derived from the master key, and written to data/backups. Retention keeps 7 daily, 4 weekly and 12 monthly archives. Every week the drill decrypts the newest archive, loads it into a scratch database, compares every table's row count and every file's checksum, and drops the scratch database. Download archives regularly to keep a copy off the server; keep the master key off the server too.

On the Backups screen#

  • the newest archive's age (an alarm above 26 hours), size, the latest restore drill, and free disk space;
  • the archives, with the retention slot (daily, weekly, monthly) that keeps each one;
  • Back up now, Run the drill, Download (audited; counts as an off-box copy), and I made an off-box copy.

Keep copies away from the server: Keeping backups off the server.

1. What is backed up#

A nightly job (backups.nightly, 02:37 UTC) writes one archive to data/backups/. The archive holds:

MemberContent
db.sqlThe whole database, one consistent snapshot.
config.phpdata/config.php.
files/uploads/…Logos and dispute evidence.
files/geo/…GeoIP databases.
files/downloads/…Licensed binaries.
manifest.jsonSize and sha256 of every member, plus the row count of every table in the dump.

The archive does not contain the master key (data/keys/master.key), because the master key is what decrypts it. Keep the master key somewhere other than the server: the installer printed it once. Without it no archive can be opened.

data/logs/, data/cache/ and data/backups/ itself are not backed up.

2. Format#

billing-20261110-023700[-a].bkp          the archive (node id suffix on a cluster)
billing-20261110-023700[-a].bkp.sha256   "<sha256>  <name>.bkp" — check a copy with sha256sum -c

The plaintext is a standard POSIX ustar tar, written by our own tar writer. It is encrypted with AES-256-GCM in 1 MiB chunks:

  • Key. HKDF-SHA256 of the master key's backup subkey, with a random 16-byte salt per archive. Every archive therefore has its own key.
  • Nonce. An 8-byte random prefix plus the chunk number.
  • Header. The header (magic, salt, nonce prefix, chunk size, master-key fingerprint) is authenticated with every chunk. The chunk number and a final-chunk flag are authenticated too.

These rules have consequences:

  • Tampering, reordering or a truncated copy is detected before any byte is used.
  • An archive made with another master key is recognised from its fingerprint.

Wrapped-blobs mode (data/ inside the webroot): the archive and its side file are named ….bkp.php and ….bkp.sha256.php and start with the PHP exit guard. The guard is not part of the archive bytes or of the sha256. Downloads and every reader skip it.

Temporary files (the plain db.sql, the partial archive) live in data/backups/.work-<random>/, are mode 0600, and are deleted when the run ends. A directory left behind by a crash is removed after a day.

3. Database dump#

The dump reuses the HA dumper:

  • mariadb-dump --single-transaction --hex-blob when PHP may run programs;
  • otherwise the built-in PHP dumper, with a consistent-snapshot transaction and binary columns as hex.

To force either one, set 'backups' => ['dumper' => 'php' | 'mariadb-dump'] in data/config.php.

The dump runs as the backup DB user, which has SELECT, LOCK TABLES, SHOW VIEW and TRIGGER, so no position or GTID data is taken.

The row count of every table is counted from the dump itself:

  • one-line INSERTs (built-in dumper) and one-tuple-per-line INSERTs (mariadb-dump 10.11+) are both understood;
  • the PHP dumper's own total is cross-checked against that count.

Because the dump is one snapshot, these counts are exactly what was live at dump time.

4. Retention (GFS)#

After each nightly run, archives of this server that no slot keeps are deleted. The slots are:

  • the newest archive of each of the last 7 days;
  • the newest of each of the last 4 ISO weeks;
  • the newest of each of the last 12 months.

The newest archive is always kept. Failed attempts are listed for 90 days.

To change the counts, set 'backups' => ['retention' => ['daily' => 7, 'weekly' => 4, 'monthly' => 12]].

5. Restore drill#

A weekly job (backups.drill, Sundays 05:13 UTC) runs the drill on the newest archive. You can also run it from Super → Backups or with cli.php backup:drill <name>. The drill:

  1. decrypts the archive and authenticates every chunk;
  2. checks each member's size and sha256 against the manifest;
  3. loads db.sql into the scratch database <db>_restore_test (billing_restore_test by default);
  4. compares the row count of every table with the count at dump time, and lists missing or extra tables;
  5. drops the scratch database.

Nothing live is touched.

Results are stored on the backup and shown on Super → Backups. A failure e-mails the super admins, at most once a day.

Privileges. The drill runs as the migrate user, which needs CREATE/DROP on <db>_restore_test:

  • New installs get this grant automatically.
  • After upgrading from 0.10, run once php app/cli.php backup:grant --db-root-user=root.
  • On a cluster, the drill runs as the node admin with sql_log_bin = 0, so the scratch database never replicates. Run backup:grant on each node; pairings made with 0.11 or later already include the grant.

6. Restore#

Restoring is done on the command line, on a single server. On a cluster, restore onto one server with the cluster disabled, then pair the second server again (CLUSTER.md).

sudo -u www-data php app/cli.php backup:list
sudo -u www-data php app/cli.php backup:verify billing-20261110-023700        # read everything, change nothing
sudo -u www-data php app/cli.php backup:restore billing-20261110-023700 --yes [--db-only | --files-only] [--with-config] \
     [--db-root-user=root --db-root-pass=…]

The restore runs in this order:

  1. Check. Everything is decrypted and checked first. Any problem stops the restore before anything changes.
  2. Database. The dump is loaded into <db>_restore_tmp and swapped in with one RENAME TABLE. The replaced tables stay in <db>_prev until the next restore. Grants are refreshed.
    • The migrate user may lack rights on <db>_restore_tmp and <db>_prev. In that case pass a root account with --db-root-user.
  3. Files. data/uploads, data/geo and data/downloads are replaced. The previous directories move to data/cache/restore-prev-<time>/.
  4. Configuration.
    • Without --with-config, the archived config.php is written next to the live one as config.php.restored, for you to compare. The live file can differ on purpose: new database passwords, a new base URL.
    • With --with-config it replaces the live one, and the old one is kept as config.php.prev-<time>.
  5. Mode check. An archive made in wrapped-blobs mode restores files only into a wrapped install, and the reverse. Use --db-only otherwise.

After a restore, run php app/cli.php selfcheck.

Restore on a new server:

  1. Install billing with the same database name.
  2. Put the old master key in data/keys/master.key (mode 0600) before the first request.
  3. Copy the archive into data/backups/.
  4. Run backup:restore <file> --yes --with-config, then fix config.php if the database users changed.

7. Off the server#

Every archive sits on the server until SFTP and S3 targets arrive in a later release. Super → Backups and the self-check (backup_offbox) show a reminder until an off-box copy is recorded. These count as an off-box copy, for 8 days:

  • Downloading an archive on Super → Backups. The download is audited and the response carries the archive's sha256 in X-Backup-Sha256.
  • "I made an off-box copy" on the same screen, for copies made with scp or rsync.

To open an archive anywhere, you need only PHP and the master key file:

php app/cli.php backup:decrypt billing-20261110-023700.bkp out.tar --key-file=master.key
tar -tf out.tar     # db.sql, config.php, files/…, manifest.json

backup:extract <name|file> <empty dir> decrypts and unpacks in one step, checking every member against the manifest.

8. Alerts and checks#

WhatWhere
Newest good backup older than 26 hSelf-check backup_freshness = fail; hourly backups.watch e-mails super admins (once a day)
A backup run failedRow marked failed with the reason; e-mail; status fail until a run succeeds
Restore drill failedE-mail; status warn
No off-box copy in 8 daysSelf-check backup_offbox = warn; reminder on Super → Backups and the platform overview
Not enough disk space (< 2 × the estimated size)The run refuses to start, with the numbers

A fresh install with no backup yet shows warn, not fail.

9. CLI reference#

Command
backup:runBack up now, then apply retention.
backup:list [--json]Archives on this server, the GFS slots that keep them, the drill result, and the freshness status (exit 1 on fail).
backup:verify <name|file>Decrypt and check every member; no database changes.
backup:drill [name]Restore drill (default: the newest archive).
backup:extract <name|file> <dir>Decrypt and unpack.
backup:decrypt <file> <out.tar> [--key-file=…]Decrypt to a plain tar. Works without an install.
backup:restore <name|file> --yes […]§6.
backup:grant --db-root-user=… [--admin-host=…]Give the drill its scratch-database rights (upgrades from 0.10).