Production Readiness Checklist

Use this checklist to review a service before launch. It brings together the checks from the Administration guides, with links to the procedures and explanations.

Copy it into your runbook and adapt it to your deployment.

How to use this

Complete each item with evidence: a dated measurement, a written procedure, a configured alert, or a completed drill.

Mark items that do not apply with a reason, so the reviewer can see which decisions were deliberate.

Before you start, agree on:

• Who signs off. Name the person responsible for accepting the evidence.
• What "outstanding" means. Some items can be accepted as gaps with an owner and a date. Decide in advance which ones cannot.
• When you run it again. This is not a one-time launch artifact. Re-run it after a Harper version upgrade, a topology change, or significant data growth, since most of the measured numbers expire.

Service definition

From How Harper Runs in Production.

• Critical user journeys named, with an owner for each
• Ports confirmed from get_configuration rather than assumed
• Operations API not reachable from the public internet
• Harper version recorded per node, with an alert on drift
• Databases listed, with replication scope confirmed from cluster_status
• Storage engine recorded per database
• Components and versions recorded from get_components
• Downstream dependencies named, with their failure behavior understood
• Fabric or self-managed responsibilities understood and divided

Capacity and topology

From Sizing a Harper Cluster.

• F, the tolerated simultaneous node loss, declared explicitly and visible to change operators
• Per-node throughput measured with your own application code, data shape, and query mix
• Measurement taken with a peer synchronizing, not on an idle cluster
• Target utilization derived from latency at your SLO percentile
• (N - F) x per-node x utilization >= peak verified with real numbers
• Measurement conditions recorded alongside the number, and dated
• Topology's "what must be proven" satisfied, including independence of failure domains
• Maintenance policy states what happens when a drain would breach F
• Storage growth thresholds set, separately from request-rate thresholds

Health and traffic admission

From Health Checks and Traffic Admission.

• Liveness, availability flag, application readiness, and journey synthetic all exist as separate signals
• Fabric's existing status component retained, or @harperdb/status-check configured for a self-managed cluster
• Traffic layer health check points at GET /status on the application port
• Availability flag storage still declares replicate: false, if you forked the component or persist it yourself
• Readiness route scoped to one journey's dependencies, with a timeout on every downstream call
• Readiness response includes the component version
• Drain sequence documented, with measured time to zero traffic
• Return sequence documented, including convergence verification and a stability window before full weight
• Measured node re-entry time recorded

Replication

From Operating Replication.

• Replication scope read off the running cluster, from every node
• Whether the system database is in scope is a deliberate, documented decision
• Identity provisioning procedure matches that decision
• Tables that must commit together confirmed to be in one database
• Durable topology constraints live in harper-config.yaml routes, not only in add_node calls
• Measured convergence time recorded under load, per critical database
• Alerts on missing peers, sustained latency growth, stalled commit progress during writes, and stale sentinel values
• Non-commutative operations identified and given an owner, a serialization point, or an external coordinator
• Replication certificate expiry tracked with an alert
• Long-absent node policy decided in advance: isolate replication and rebuild this node from empty when peers are authoritative, or rebuild the peers from it when it is
• Replication interruption and recovery exercise completed and dated

Deployment and rollback

From Safe Deployments and Rollback.

• Artifacts immutable, referenced by pinned version rather than by branch
• Deploy and expose are separate actions in the pipeline
• Previous known-good artifact addressable, and redeploying it rehearsed
• On v5.3.0 and later, the previous deployment_id still retained on every target node and activation rehearsed
• Cohort ladder defined, from smallest useful sample to full exposure
• Stop thresholds and hold times declared before a rollout begins
• Named decision owner for advance, hold, stop, and roll back
• Pipeline checks deployment details and waits for the chosen restart mode to finish, polling the job when required, then verifies the serving version on every node
• Nodes that must not serve while they change are drained first, since neither restart mode takes nodes out of rotation
• deployment_timeout and ignore_replication_errors set deliberately
• Configuration changes go through the same change record as code
• get_configuration read back after every configuration change
• Node-local parameters never replicated
• Schema changes follow expand then contract, with destructive operations documented and verified per node
• Forward repair defined for changes that alter persisted meaning
• Measured release reversal time recorded from a rehearsal

Observability

From Monitoring and Triage.

• A dashboard exists that can place a symptom at edge, traffic layer, node, replication, data, or change
• Alerts on journey success and latency at the SLO percentile, not the mean
• Alerts on worker utilization and task queue latency, with thresholds derived from measured capacity
• Alerts on replication convergence lag
• Alerts on storage headroom and the age of the newest fully verified recovery point
• Metrics exported off the node
• analytics.replicate setting known and deliberate
• Structured logs include component, version, node, request id, operation, duration, and outcome
• Logs centralized off the node
• Deployments annotated onto dashboards
• Triage sequence written where on-call can find it
• Fault injection drill completed, with time to detection recorded

Backup and recovery

From Backup and Recovery.

• Failure classes enumerated, each with the mechanism that addresses it
• Database recovery points and journey recovery times recorded using the objective definitions
• Mechanism chosen per database according to its storage engine
• No database in scope has tables spanning multiple storage roots, or the exclusion is known and accepted
• Backup cadence matches the stated recovery point
• Repository capacity measured across successive backups, including transaction-log copies and blob hard links
• Export capacity accounts for full copies where the destination does not preserve file sharing
• Off-host copy exists in a separate failure and administrative boundary
• Backup artifacts have immutable identities, manifests, and checksums
• Destination encryption, versioning, and retention checked after upload where applicable
• Backup writer cannot weaken the destination's retention policy
• Managed repository copies take the whole per-database directory, quiesced or from an atomic snapshot
• verify_backup runs on a schedule
• Blob integrity understood to be unverified by verify_backup
• Restore constraints documented for user databases, component-held databases, and system
• Off-host get_backup file restore procedure written down and rehearsed with Harper stopped, including manual extraction and blob-root recovery; restore_backup cannot read the file
• Restore authority named, and restore execution logged
• Restore procedure isolates the node from replication, not just from user traffic
• Restore tested in a disposable, isolated environment using a compatible Harper version
• Restored inventory, counts, business invariants, and critical journeys validated
• Artifact ID, backup time, validation results, operator, approver, achieved RPO, and achieved RTO recorded
• system database restore rehearsed offline
• Preservation export planned separately where records require long-term, software-independent access

Objectives

From Engineering RPO, RTO, and Uptime.

• Objectives recorded using the definitions, with each journey's measurement boundary and dependent databases named
• Exclusions documented
• Every scenario in the scenario map has a named mechanism
• Availability target converted to minutes per month
• Error budget policy states what changes when the budget runs low
• Backup cadence reconciled against recovery point
• storage.writeAsync confirmed off, or the databases it is on recorded beside their recovery point
• Restore time plus detection time reconciled against recovery time
• Convergence time reconciled against the admission gate
• Every unreconciled gap has an owner and a date
• Degraded modes declared per journey, with authority to invoke them

Security and access

• Least privilege enforced for operators, applications, and automation
• super_user credentials inventoried, with a rotation procedure
• Operations API restricted to administrative networks
• TLS material inventoried, with expiry alerts for both application and replication certificates
• Deploy credentials held in a secret store, not in pipeline configuration
• Administrative access logged, including CLI access on the hosts
• Filesystem access on nodes understood to permit offline restore without an API credential

Ownership and runbooks

• On-call rotation and escalation path defined
• Runbooks exist for drain, return, deploy, reverse, restore, and cluster expansion
• Every runbook has been executed by someone other than its author
• Change record location known, and used
• Incident and postmortem process defined
• A named owner for this checklist, and a date for the next review

Exercises completed

Record the date of the most recent run of each. An undated exercise is an undocumented one.

Exercise	Guide	Last run	Result
Service boundary inventory	How Harper Runs in Production
Drain and return under load	Health Checks
Peak-load drain and rejoin	Sizing
Replication interruption and recovery	Operating Replication
Fault injection and time to detection	Monitoring and Triage
Stopped rollout and reversal	Safe Deployments
Restore drill, user database	Backup and Recovery
Restore drill, system database, offline	Backup and Recovery
Scenario tabletop	Engineering RPO, RTO, and Uptime

Additional Resources

• Reliability Plan Template for the document that holds the answers this checklist asks for
• Every guide in this section, linked per group above
• Security overview and certificate management
• Configuration options for the settings referenced throughout