Backups and restore
Nightly encrypted archives, retention, the weekly restore drill, downloading copies, restoring, and the backup alerts.
Text description
Every night the database is dumped as the backup user and packed into a tar archive together with config.php, uploads, GeoIP files and licensed downloads, plus a manifest with the size and SHA-256 of every member and the row count of every table. The archive is encrypted with AES-256-GCM in chunks, with a key derived from the master key, and written to data/backups. Retention keeps 7 daily, 4 weekly and 12 monthly archives. Every week the drill decrypts the newest archive, loads it into a scratch database, compares every table's row count and every file's checksum, and drops the scratch database. Download archives regularly to keep a copy off the server; keep the master key off the server too.
On the Backups screen#
- the newest archive's age (an alarm above 26 hours), size, the latest restore drill, and free disk space;
- the archives, with the retention slot (daily, weekly, monthly) that keeps each one;
- Back up now, Run the drill, Download (audited; counts as an off-box copy), and I made an off-box copy.
Keep copies away from the server: Keeping backups off the server.
1. What is backed up#
A nightly job (backups.nightly, 02:37 UTC) writes one archive to data/backups/. The archive holds:
| Member | Content |
|---|---|
db.sql | The whole database, one consistent snapshot. |
config.php | data/config.php. |
files/uploads/… | Logos and dispute evidence. |
files/geo/… | GeoIP databases. |
files/downloads/… | Licensed binaries. |
manifest.json | Size and sha256 of every member, plus the row count of every table in the dump. |
The archive does not contain the master key (data/keys/master.key), because the master key is what decrypts it. Keep the master key somewhere other than the server: the installer printed it once. Without it no archive can be opened.
data/logs/, data/cache/ and data/backups/ itself are not backed up.
2. Format#
billing-20261110-023700[-a].bkp the archive (node id suffix on a cluster)
billing-20261110-023700[-a].bkp.sha256 "<sha256> <name>.bkp" — check a copy with sha256sum -c
The plaintext is a standard POSIX ustar tar, written by our own tar writer. It is encrypted with AES-256-GCM in 1 MiB chunks:
- Key. HKDF-SHA256 of the master key's backup subkey, with a random 16-byte salt per archive. Every archive therefore has its own key.
- Nonce. An 8-byte random prefix plus the chunk number.
- Header. The header (magic, salt, nonce prefix, chunk size, master-key fingerprint) is authenticated with every chunk. The chunk number and a final-chunk flag are authenticated too.
These rules have consequences:
- Tampering, reordering or a truncated copy is detected before any byte is used.
- An archive made with another master key is recognised from its fingerprint.
Wrapped-blobs mode (data/ inside the webroot): the archive and its side file are named ….bkp.php and ….bkp.sha256.php and start with the PHP exit guard. The guard is not part of the archive bytes or of the sha256. Downloads and every reader skip it.
Temporary files (the plain db.sql, the partial archive) live in data/backups/.work-<random>/, are mode 0600, and are deleted when the run ends. A directory left behind by a crash is removed after a day.
3. Database dump#
The dump reuses the HA dumper:
mariadb-dump --single-transaction --hex-blobwhen PHP may run programs;- otherwise the built-in PHP dumper, with a consistent-snapshot transaction and binary columns as hex.
To force either one, set 'backups' => ['dumper' => 'php' | 'mariadb-dump'] in data/config.php.
The dump runs as the backup DB user, which has SELECT, LOCK TABLES, SHOW VIEW and TRIGGER, so no position or GTID data is taken.
The row count of every table is counted from the dump itself:
- one-line INSERTs (built-in dumper) and one-tuple-per-line INSERTs (mariadb-dump 10.11+) are both understood;
- the PHP dumper's own total is cross-checked against that count.
Because the dump is one snapshot, these counts are exactly what was live at dump time.
4. Retention (GFS)#
After each nightly run, archives of this server that no slot keeps are deleted. The slots are:
- the newest archive of each of the last 7 days;
- the newest of each of the last 4 ISO weeks;
- the newest of each of the last 12 months.
The newest archive is always kept. Failed attempts are listed for 90 days.
To change the counts, set 'backups' => ['retention' => ['daily' => 7, 'weekly' => 4, 'monthly' => 12]].
5. Restore drill#
A weekly job (backups.drill, Sundays 05:13 UTC) runs the drill on the newest archive. You can also run it from Super → Backups or with cli.php backup:drill <name>. The drill:
- decrypts the archive and authenticates every chunk;
- checks each member's size and sha256 against the manifest;
- loads
db.sqlinto the scratch database<db>_restore_test(billing_restore_testby default); - compares the row count of every table with the count at dump time, and lists missing or extra tables;
- drops the scratch database.
Nothing live is touched.
Results are stored on the backup and shown on Super → Backups. A failure e-mails the super admins, at most once a day.
Privileges. The drill runs as the migrate user, which needs CREATE/DROP on <db>_restore_test:
- New installs get this grant automatically.
- After upgrading from 0.10, run once
php app/cli.php backup:grant --db-root-user=root. - On a cluster, the drill runs as the node admin with
sql_log_bin = 0, so the scratch database never replicates. Runbackup:granton each node; pairings made with 0.11 or later already include the grant.
6. Restore#
Restoring is done on the command line, on a single server. On a cluster, restore onto one server with the cluster disabled, then pair the second server again (CLUSTER.md).
sudo -u www-data php app/cli.php backup:list
sudo -u www-data php app/cli.php backup:verify billing-20261110-023700 # read everything, change nothing
sudo -u www-data php app/cli.php backup:restore billing-20261110-023700 --yes [--db-only | --files-only] [--with-config] \
[--db-root-user=root --db-root-pass=…]
The restore runs in this order:
- Check. Everything is decrypted and checked first. Any problem stops the restore before anything changes.
- Database. The dump is loaded into
<db>_restore_tmpand swapped in with oneRENAME TABLE. The replaced tables stay in<db>_prevuntil the next restore. Grants are refreshed.- The migrate user may lack rights on
<db>_restore_tmpand<db>_prev. In that case pass a root account with--db-root-user.
- The migrate user may lack rights on
- Files.
data/uploads,data/geoanddata/downloadsare replaced. The previous directories move todata/cache/restore-prev-<time>/. - Configuration.
- Without
--with-config, the archivedconfig.phpis written next to the live one asconfig.php.restored, for you to compare. The live file can differ on purpose: new database passwords, a new base URL. - With
--with-configit replaces the live one, and the old one is kept asconfig.php.prev-<time>.
- Without
- Mode check. An archive made in wrapped-blobs mode restores files only into a wrapped install, and the reverse. Use
--db-onlyotherwise.
After a restore, run php app/cli.php selfcheck.
Restore on a new server:
- Install billing with the same database name.
- Put the old master key in
data/keys/master.key(mode 0600) before the first request. - Copy the archive into
data/backups/. - Run
backup:restore <file> --yes --with-config, then fixconfig.phpif the database users changed.
7. Off the server#
Every archive sits on the server until SFTP and S3 targets arrive in a later release. Super → Backups and the self-check (backup_offbox) show a reminder until an off-box copy is recorded. These count as an off-box copy, for 8 days:
- Downloading an archive on Super → Backups. The download is audited and the response carries the archive's sha256 in
X-Backup-Sha256. - "I made an off-box copy" on the same screen, for copies made with scp or rsync.
To open an archive anywhere, you need only PHP and the master key file:
php app/cli.php backup:decrypt billing-20261110-023700.bkp out.tar --key-file=master.key
tar -tf out.tar # db.sql, config.php, files/…, manifest.json
backup:extract <name|file> <empty dir> decrypts and unpacks in one step, checking every member against the manifest.
8. Alerts and checks#
| What | Where |
|---|---|
| Newest good backup older than 26 h | Self-check backup_freshness = fail; hourly backups.watch e-mails super admins (once a day) |
| A backup run failed | Row marked failed with the reason; e-mail; status fail until a run succeeds |
| Restore drill failed | E-mail; status warn |
| No off-box copy in 8 days | Self-check backup_offbox = warn; reminder on Super → Backups and the platform overview |
| Not enough disk space (< 2 × the estimated size) | The run refuses to start, with the numbers |
A fresh install with no backup yet shows warn, not fail.
9. CLI reference#
| Command | |
|---|---|
backup:run | Back up now, then apply retention. |
backup:list [--json] | Archives on this server, the GFS slots that keep them, the drill result, and the freshness status (exit 1 on fail). |
backup:verify <name|file> | Decrypt and check every member; no database changes. |
backup:drill [name] | Restore drill (default: the newest archive). |
backup:extract <name|file> <dir> | Decrypt and unpack. |
backup:decrypt <file> <out.tar> [--key-file=…] | Decrypt to a plain tar. Works without an install. |
backup:restore <name|file> --yes […] | §6. |
backup:grant --db-root-user=… [--admin-host=…] | Give the drill its scratch-database rights (upgrades from 0.10). |