Backups

Harper can back up and restore its databases natively. A RocksDB managed backup or snapshot download is a whole-database copy of authoritative state — all tables, the audit/transaction log, and file-backed blobs unless exclude_blobs is requested — so a restored database is a consistent point-in-time image. Local derived state, including full-text index files, is excluded and rebuilt after restore. LMDB get_backup is the exception: it streams only the .mdb file, can stream a subset of tables (including the audit store only when requested), and leaves external blob files to a separate, coordinated backup.

Two complementary mechanisms are available:

• Managed backups Added in: v5.2.0 Engine: RocksDB — incremental, verifiable backups kept in a server-side repository. Harper creates, lists, verifies, deletes, purges, and restores them through the backup operations.
• Snapshot download Engines: RocksDB, LMDB — get_backup streams a RocksDB archive or an LMDB .mdb file over HTTP for storage off-host, with no server-side artifact.

Every backup operation is available through the Operations API and as a CLI command under the same operation name.

How managed backups work

Managed backups use the RocksDB backup engine. Each backup is a directory under the configured backup root (storage.backupPath, default <rootPath>/backup), with one subdirectory per database. Because RocksDB data files are immutable once written, backups in the same location share unchanged files: the first backup copies the whole database, and each subsequent backup only copies what changed since the last one. Shared files are reference-counted, so deleting a backup only removes files no remaining backup references.

Every managed backup includes the database's transaction log — so a restored database keeps its read_audit_log history as of the backup point — and, for a database with file-backed blobs, a snapshot of those blobs (skip it with exclude_blobs).

Each backup copies the transaction-log snapshot. Blob snapshots use hard links when supported on the same filesystem, and copy the files otherwise. Measure physical repository growth over successive backups, including the log copies and blob snapshots. Exports that do not preserve file sharing require space for the full materialized copies.

The long-running operations (create_backup, verify_backup, restore_backup) run as background jobs when invoked through a running server: the operation returns a job_id immediately, and get_job reports the outcome.

Online and offline

Backup operations work whether or not Harper is running. Invoked from the CLI while the server is running, the operation is forwarded to the server; while the server is stopped, the command operates directly on the database and backup files. get_backup is the exception — it streams from a running server and has no offline form. A CLI target, set with target=, an environment variable or harper login, changes this; see running backup operations from the CLI.

The offline path matters for restore. RocksDB is single-writer, so an in-place restore requires the database to be fully closed first. A running Harper server can close its own worker threads' handles, but it cannot close a handle held by a loaded component, nor stop the system database it depends on to run — those databases can only be restored with the server stopped. See "when can a database be restored" below.

Limitations

• Managed backups require the RocksDB storage engine. For LMDB databases, use get_backup or volume snapshots.
• Whole-database granularity. There is no per-table backup or restore. The exception is get_backup on LMDB, which can stream selected tables; a table-filtered download includes the audit store only with include_audit.
• One storage root per database. Backup is rejected if the database's tables resolve to multiple storage roots. A custom database-level storage.path, or per-table paths that still resolve to a single root, are supported.
• Backups live on the node that created them. The backup repository is a local directory. RocksDB shares files across backup IDs (a backup ID is not a self-contained folder), so disaster-recovery copies must take the entire per-database repository — <backupPath>/<database> — not an individual backup, and must do so while no backup operation is running, or from an atomic filesystem snapshot. A live recursive copy can race create_backup/delete_backup/purge_backups and produce an unrestorable copy. Alternatively, use get_backup to pull a snapshot from a running server.
• get_backup always streams the current state. It cannot download a historical managed backup; to move a retained backup off-host, copy its whole per-database repository as above.
• A restore is a point-in-time rollback. In a replicated cluster, coordinate a restore with replication before bringing the node back.
• Full-text indexes rebuild locally after restore. Ordinary table reads remain available. Writes can receive retryable DERIVED_INDEX_LAGGING responses while replay catches up after a long rebuild; see write backpressure. While the rebuild is active, full-text requests return 503 with code: "INDEX_REBUILDING" and retryable: true; branch on the code because other 503 responses are not necessarily rebuilds.
• A restore checks the backup before it deletes anything. Changed in: v5.4.0 A backup with a corrupt database file, or one this version of Harper cannot read, is refused and the database is left as it was (a database an earlier restore left incomplete stays blocked). The check reads the whole backup, which adds to how long an online restore keeps the database unavailable. Blob file contents are not checked. See restore_backup.
• An interrupted restore leaves the database unloadable. If restore_backup is interrupted after it has started replacing the database's files (crash, power loss), Harper marks the database as incompletely restored and skips loading it on the next start, logging an incomplete-restore error. Rerun restore_backup for the same database and backup_id to recover; do not load or hand-repair the directory.

When can a database be restored?

An in-place restore purges and rewrites the database's files, which requires the database to be fully closed first. Whether a restore can run online depends on what is holding the database open:

Database	Online restore_backup (server running)	Offline harper restore_backup (server stopped)
A user database not opened by any loaded component	Yes — restored in place	Yes
A user database that a loaded component keeps open	No — the job ends in ERROR	Yes
The system database	No — rejected up front	Yes

• Online works only when no loaded component is holding the database open. Because restore_backup runs as a background job, the request returns a job_id first; if a component is holding the database open, the check fails inside the job, so the job ends in ERROR (surfaced by get_job) with a message telling you to restore offline — Harper cannot force a component's handles closed while the server is running, and restoring under an open instance would corrupt it. (Harper does not track which component uses which database, so it cannot selectively stop one; it can only detect that the database is still open.) The system database is rejected up front, before a job is created.
• Offline — the same restore_backup command run from the CLI with the server stopped — works for any database, because no components are loaded and nothing holds it open. This is the required path for the system database and for any database a component keeps open.

Online restore always restores in place. To restore into a new database instead of overwriting the source, run restore_backup with target_database from the CLI with the server stopped.

Example: incremental backups and restore

Create a managed backup of the data database (the default):

harper create_backup database=data

Later backups share unchanged RocksDB data files, while transaction-log copies and blob snapshots have their own storage cost; see how managed backups work. Schedule create_backup as often as your recovery point requires, and manage retention with purge_backups:

harper purge_backups database=data keep_count=7

List the backups to find the one to restore:

harper list_backups database=data

Restore the latest backup, or pass backup_id=<id> for an earlier one:

harper restore_backup database=data

With the server running, this restores the database in place — Harper closes the database across its worker threads, checks the backup, restores it, and reloads it — as long as nothing is holding the database open (see when can a database be restored?). With the server stopped, the same command restores the files directly and works for any database.

Example: download a snapshot and restore it manually

get_backup streams a full snapshot of a database from a running server — local or remote — which makes it the simplest way to keep backups off-host. Because it can stream from a remote target, it also doubles as a way to clone a database from another node onto this one: pull the snapshot with target=<url>, then restore it locally as shown below. What it streams depends on the storage engine, so the manual restore differs too. The examples below use the data database and the default storage path (storage.path, default <rootPath>/database).

RocksDB Engine: RocksDB

For a RocksDB database the stream is a tar archive, gzipped by default. It contains the database's current state — all tables, the transaction log, and, for a database with file-backed blobs, a blobs/<rootIndex>/... tree for each blob root (pass exclude_blobs=true to omit them):

harper get_backup database=data out=./data.tar.gz
# or read the snapshot from another node, still writing it to a file here
harper get_backup database=data target=https://node-2.example.com:9925 out=./data.tar.gz

If the database has no file-backed blobs, the archive extracts into a directory that opens directly as a RocksDB database. Stop Harper, replace the database's directory under the storage path with the extracted archive, and start Harper again:

Before v5.4.0, harper stop returns before Harper finishes closing databases, so the procedures below capture its PID and wait for the process to exit before changing files. From v5.4.0, harper stop waits for the exit itself, and the loop still covers the cases where it gives up while Harper is running or finds no process to stop.

HARPER_PID=$(cat ~/hdb/hdb.pid)
harper stop
while kill -0 "$HARPER_PID" 2>/dev/null; do sleep 1; done
mv ~/hdb/database/data ~/hdb/database/data.old # optional: keep a copy of the previous database
mkdir -p ~/hdb/database/data
tar -xzf data.tar.gz -C ~/hdb/database/data
harper start

If the database has file-backed blobs, the RocksDB files and the blobs/ tree restore to different locations: the RocksDB files into the database directory, and each blobs/<rootIndex>/... tree into its blob root (by default <rootPath>/blobs/<database>, or the paths configured in storage.blobPaths; the archive's blobs/README.md lists the index-to-path mapping). Extracting the whole archive into the database directory would bury the blobs where the engine can't find them, silently losing blob data. Restore the two parts separately:

HARPER_PID=$(cat ~/hdb/hdb.pid)
harper stop
while kill -0 "$HARPER_PID" 2>/dev/null; do sleep 1; done
# 1. RocksDB files -> database directory (exclude the blobs tree)
mv ~/hdb/database/data ~/hdb/database/data.old # optional: keep a copy of the previous database
mkdir -p ~/hdb/database/data
tar -xzf data.tar.gz -C ~/hdb/database/data --exclude='blobs' --exclude='blobs/*'
# 2. each blobs/<rootIndex>/ -> its blob root (index 0 shown; repeat per root)
mkdir -p ~/hdb/blobs/data
tar -xzf data.tar.gz -C ~/hdb/blobs/data --strip-components=2 blobs/0
harper start

The target node must have at least as many blob roots configured as the snapshot captured — blob references store their root index, so a missing root leaves those blobs unresolved. For anything beyond the simplest single-root case, prefer managed backups: restore_backup restores the RocksDB files and every blob root automatically.

LMDB Engine: LMDB

warning

This stream does not include external blob files. For LMDB databases with file-backed blobs, use a coordinated filesystem or volume backup of the .mdb file and every configured blob root with Harper stopped. Restore that matching set and validate blob reads before serving traffic; restoring the .mdb file alone leaves its external blob references unresolved if the source files are lost.

For an LMDB database the stream is the database's single .mdb file:

harper get_backup database=data out=./data.mdb
# or pull from another instance
harper get_backup database=data target=https://node-2.example.com:9925 out=./data.mdb

To restore it, stop Harper, replace the database's .mdb file under the storage path (<storage.path>/<database>.mdb) with the downloaded file, and start Harper again:

HARPER_PID=$(cat ~/hdb/hdb.pid)
harper stop
while kill -0 "$HARPER_PID" 2>/dev/null; do sleep 1; done
mv ~/hdb/database/data.mdb ~/hdb/database/data.mdb.old # optional: keep a copy of the previous database
cp data.mdb ~/hdb/database/data.mdb
harper start