Version: v5

Transaction Logging

Harper maintains a transaction log for every database: a record of every data change, capturing the operation type, the user who made the change, the timestamp, and both the new and original record values. There is one transaction log per database, shared by all tables.

Audit log and transaction log are the same thing

Some operations and settings still carry the older "audit log" name — read_audit_log, delete_audit_logs_before, the logging.auditLog setting, and the @table(audit:) directive — but they all act on this single transaction log. The "audit log" is not a separate mechanism; the distinct-log terminology is a historical artifact.

Available since: v4.1.0

On RocksDB (the default storage engine) the transaction log is physically a single database-wide log shared by all tables: history is read per table, while deletion operates on the whole database's log.

Changed in: v5.4.0 By default on RocksDB, a transaction that fails after its entry was written to the log is marked aborted and left out of the log's history, including read_audit_log results. The storage.rocks.transactionLogAbortMarkers setting controls this.

The transaction log is enabled by default. To disable it, set logging.auditLog to false in harper-config.yaml and restart Harper.

The transaction log is required for real-time messaging (WebSocket and MQTT subscriptions), replication and full-text indexing. Do not disable it while any of these features are in use.

Operations

`read_audit_log`

Queries the transaction log for a specific table. Supports filtering by timestamp, username, or primary key value.

By timestamp:

{
	"operation": "read_audit_log",
	"schema": "dev",
	"table": "dog",
	"search_type": "timestamp",
	"search_values": [1660585740558]
}

Timestamp behavior:

`search_values`	Result
`[]`	All records for the table
`[timestamp]`	All records after the provided timestamp
`[from, to]`	Records between the two timestamps

By username:

{
	"operation": "read_audit_log",
	"schema": "dev",
	"table": "dog",
	"search_type": "username",
	"search_values": ["admin"]
}

By primary key:

{
	"operation": "read_audit_log",
	"schema": "dev",
	"table": "dog",
	"search_type": "hash_value",
	"search_values": [318]
}

Response example:

{
	"operation": "update",
	"user_name": "HDB_ADMIN",
	"timestamp": 1607035559122.277,
	"hash_values": [1, 2],
	"records": [
		{
			"id": 1,
			"breed": "Muttzilla",
			"age": 6,
			"__updatedtime__": 1607035559122
		}
	],
	"original_records": [
		{
			"__createdtime__": 1607035556801,
			"__updatedtime__": 1607035556801,
			"age": 5,
			"breed": "Mutt",
			"id": 1,
			"name": "Harper"
		}
	]
}

The original_records field contains the record state before the operation was applied.

`delete_audit_logs_before`

Deletes transaction log entries older than the specified timestamp. Deprecated in favor of delete_transaction_logs_before.

Changed in: v4.3.0 — Audit log cleanup improved to reduce resource consumption during scheduled cleanups

Changed in: v4.5.0 — Storage reclamation: Harper automatically evicts older audit log entries when free storage drops below a configurable threshold

Changed in: v5.2.0 — This operation is unsupported on the RocksDB storage engine (the default): it requires table, but history cannot be deleted for a single table because all tables in a database share one transaction log. For an existing table the job fails with an error directing you to delete_transaction_logs_before; for a nonexistent table the job fails with a not-found error. The operation remains usable on LMDB.

{
	"operation": "delete_audit_logs_before",
	"schema": "dev",
	"table": "dog",
	"timestamp": 1598290282817
}

`delete_transaction_logs_before`

Engines: RocksDB, LMDB

Changed in: v5.2.0 — On RocksDB, a request that includes table now fails; previously the table scope was silently ignored and the entire database's transaction log was purged. On either engine, a table that does not exist now fails with a not-found error (previously a typo'd table fell through to the database-wide purge on RocksDB, and was a silent no-op on LMDB). On LMDB, a valid table continues to scope the deletion to that table's history, unchanged.

Deletes transaction log entries older than the specified timestamp.

Database-wide and irreversible

On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit table and pass only database (or schema) and timestamp. This purges whole log files whose entries predate the timestamp, removing read_audit_log history for every table in the database — there is no per-table survivor and no undo. The only recovery route is a backup, which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica: the sender forces a full base copy instead of incremental catch-up. That happens for any replica whose position is below the point the purge reached (at most the purge timestamp), even if the database had no writes after that position (see when a peer forces a base copy). A full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. The same history can be required to bridge a full-text index rebuild from its table scan to current writes. Cleanup that removes the bridge during a rebuild can fail that attempt; stop aggressive cleanup before retrying so the next attempt can scan from a fresh committed-log boundary. Size automatic retention for the workload.

Parameters:

database (or the deprecated schema alias): string (required) — a request naming neither fails validation before a job starts; a database that does not exist fails the job with a not-found error.
timestamp: number (required) — epoch milliseconds; entries older than this are deleted.
table: string (LMDB only) — scopes deletion to that table's history. On RocksDB the job fails (see the warning above).
cleanup_deleted_records: boolean (optional) — LMDB only; additionally removes leftover tombstone entries for records deleted before the timestamp, a repair step for tombstones that normal audit log cleanup should already have removed. Ignored on RocksDB.

On LMDB, the table-scoped deletion scans the database's full audit history (and cleanup_deleted_records: true adds a second full scan of the table's records), so the cost grows with total history depth — schedule accordingly on databases with deep audit history.

Request validation runs first and synchronously: a request that omits both database and schema is rejected immediately with an error and no job ID (there is nothing to poll). Once accepted, the operation runs as a background job that returns 200 with a job ID, and operation-time failures surface through get_job — the job ends with status ERROR and a message describing the failure (for example, the RocksDB table-scope rejection, or a table/database that does not exist).

RocksDB (database-wide — omit table):

{
	"operation": "delete_transaction_logs_before",
	"database": "dev",
	"timestamp": 1598290282817
}

LMDB (table is required — omitting it deletes nothing and reports entries_deleted: 0 with job status COMPLETE):

{
	"operation": "delete_transaction_logs_before",
	"database": "dev",
	"table": "dog",
	"timestamp": 1598290282817
}

Response:

{
	"message": "Starting job with id 2fe25039-566e-4670-8bb3-2db3d4e07e69",
	"job_id": "2fe25039-566e-4670-8bb3-2db3d4e07e69"
}

get_job reports the outcome. A successful job's result carries entries_deleted and log_files_deleted (the count of purged log files on RocksDB; 0 on LMDB, which has no separate log files) — the record of how much was deleted. A failed job looks like:

[
	{
		"id": "2fe25039-566e-4670-8bb3-2db3d4e07e69",
		"type": "delete_transaction_logs_before",
		"status": "ERROR",
		"message": "There was an error running deleteTransactionLogsBefore job with id 2fe25039-566e-4670-8bb3-2db3d4e07e69 - Table-level transaction log deletion is not supported for RocksDB tables because all tables in a database share one transaction log; to delete the transaction logs for the entire 'dev' database, use delete_transaction_logs_before with only 'database' and 'timestamp'"
	}
]

Enabling the Transaction Log Per Table

You can enable or disable the transaction log for individual tables using the @table directive's audit argument in your schema:

type Dog @table(audit: true) {
	id: Long @primaryKey
	name: String
}

This overrides the logging.auditLog global configuration for that specific table. New schema tables with @fullText must explicitly declare @table(audit: true), even when global audit logging is enabled. Existing audited tables can add an index without restating audit. Keep auditing enabled while any full-text index is declared. Setting @table(audit: false) while a full-text index is declared is rejected during schema registration; this is a per-table prerequisite, independent of the global default.

Logging — Application and system logging (separate from the transaction log)
Replication — Replication and clustering, which consume the transaction log
Logging Configuration — Global transaction log configuration (logging.auditLog)
Operations API — Sending operations to Harper

Operations​

read_audit_log​

delete_audit_logs_before​

delete_transaction_logs_before​

Enabling the Transaction Log Per Table​

Related Documentation​

Operations

`read_audit_log`

`delete_audit_logs_before`

`delete_transaction_logs_before`

Enabling the Transaction Log Per Table

Related Documentation