Reliability Plan Template
The Production Readiness Checklist asks the questions. This is the document that holds the answers.
Keep one plan per service, somewhere your on-call team can reach during an incident. Replace the example values with your deployment's measurements and procedures.
Example values are included in italics to show the intended level of specificity. Replace them.
1. Service identity
| Field | Value |
|---|---|
| Service name | orders-api |
| Business owner | name, team |
| Technical owner | name, team |
| On-call rotation | link to schedule |
| Escalation path | first, second, and decision authority for a stop |
| Harper deployment | Fabric cluster, or self-managed, with cluster identifier |
| Harper version | 5.2.x, per node |
| Plan last reviewed | date |
| Next review due | date |
2. Critical journeys
List the user-facing paths that must work, including the databases and dependencies each one needs.
| Journey | Description | Databases and tables involved | Downstream dependencies | Owner |
|---|---|---|---|---|
| submit order | POST through the public API | orders.orders, orders.items | pricing, payments | |
| browse catalog | cached read | catalog.products | none |
3. Objectives
Use the objective definitions: availability and recovery time per journey, recovery point per database.
| Journey | Availability target | Measurement boundary | Recovery time objective | Recovery point objective by database | Exclusions |
|---|---|---|---|---|---|
| 99.95% | at the CDN edge | 15 min from start of failure | orders: 5 min; catalog: 1 hour | planned maintenance, third-party outages |
Monthly unavailability budget: 21 min 55 sec at 99.95%
Error budget policy: what changes when the remaining budget falls below a threshold, and who decides
4. Architecture and topology
- Node count and
F, the tolerated simultaneous loss: 5 nodes, F = 1 - Failure domain layout: how nodes are distributed, and the proof that domains are independent
- Traffic layer and routing: what fronts Harper, and how it selects nodes
- Replication scope: which databases, which tables excluded, and whether
systemis in scope - Sharding in use: yes or no, and where
- Storage engine per database: per database
5. Capacity
| Measurement | Value | Conditions | Date measured |
|---|---|---|---|
| Peak required throughput | per journey | ||
| Tested per-node throughput | data volume, query mix, peer synchronizing y/n | ||
| Target utilization | derived from latency at the SLO percentile | ||
Surviving capacity at F | measured in the drain drill, not calculated |
Maintenance policy when a drain would breach F: pause, add capacity, or accept with documented duration
6. Traffic admission
- Liveness probe: endpoint, interval, threshold
- Availability flag:
status-checkdeployed how, and who may set it - Readiness route: path, and the dependencies it checks
- Journey synthetic: what it does, from where, how often
- Drain sequence: link to runbook
- Return sequence, including stability window: link to runbook
- Measured time to zero traffic after drain: value, date
- Measured node re-entry time including convergence: value, date
7. Replication
- Measured convergence time under load, per critical database: value, date
- Measured full synchronization time for a new node: value, date
- Admission gate threshold, and how it compares to convergence: value
- Non-commutative operations and their handling: operation, and the owner, serialization point, or coordinator
- Certificate expiry dates and alert lead time: dates
8. Deployment and change
- Artifact source and reference format: registry or git, pinned version format
- Cohort ladder: one node, internal, 5 percent, 25 percent, full
- Stop thresholds and hold times: stated numerically
- Decision authority for advance, hold, stop, roll back: name or role
- Restart mode used, and how completion is confirmed:
rolling, polling the job - Rollback reference and retention check: pinned package, or retained
deployment_idon every node for v5.3.0 and later - Measured release reversal time: value, date
- Forward repair procedures for data-affecting releases: link
- Change record location: link
9. Observability
- Dashboard location: link
- Alerts configured, with thresholds and destinations: table or link
- Metric export path: Grafana on Fabric, or Prometheus scrape
- Log destination and retention: system, retention period
- Deployment annotations enabled: yes or no
- Triage runbook: link
- Measured time to detection, per drill: value, date
10. Backup and recovery
| Database | Mechanism | Cadence | Retention | Off-host destination | Last verified | Last restore drill | Measured restore time | Measured data loss |
|---|---|---|---|---|---|---|---|---|
- Restore authority: who may execute a restore
- Restore constraints per database: online, or offline only because a component holds it open
systemdatabase restore procedure: offline only, link to runbook- Failure classes assigned to restore versus forward repair: table
- Artifact record: immutable ID, backup timestamp, manifest and checksum locations
- Destination controls: failure and administrative boundaries, encryption, versioning, retention, read-back evidence
- Backup writer permissions: how retention changes are prevented
- Newest fully verified recovery point: artifact ID, timestamp, validation results, operator, approver
- Restore validation: database/table inventory, counts, business invariants, critical journeys, compatible Harper version
- Achieved RPO and RTO: measured data loss and time from failure to verified service
- Preservation export, if required: format, reader, and retention independent of Harper
11. Scenario responses
| Scenario | Mechanism | Runbook | What proves it works | Last exercised |
|---|---|---|---|---|
| single node loss | ||||
| node return after restart | ||||
| node replacement | ||||
| failure domain loss | ||||
| region loss | ||||
| bad release, no data change | ||||
| bad release that changed data | ||||
| accidental destructive operation | ||||
| storage loss on one node | ||||
| certificate expiry | ||||
| downstream dependency outage |
12. Degraded modes
| Journey | Condition | Declared behavior | Who may invoke | Automatic or manual |
|---|---|---|---|---|
| pricing service unavailable | serve last known price, flag the response | |||
| node behind on replication | remove from rotation | |||
| cluster below capacity budget | shed non-critical traffic |
13. Security and access
- Operator roles and privileges: who has what
super_usercredential inventory and rotation: link, schedule- Operations API network restriction: how enforced
- Certificate inventory and expiry alerts: link
- Deploy credential custody: secret store, rotation method
- Administrative access logging, including host-level: system
14. Open risks
Record accepted gaps with their impact, owner, and review date.
| Risk | Impact if realized | Why accepted | Owner | Review date |
|---|---|---|---|---|
15. Revision history
| Date | Author | Change | Trigger |
|---|---|---|---|
| launch, version upgrade, topology change, incident |
Keeping this current
Most of the numbers in this plan are properties of a Harper version, a data volume, a topology, and a set of components. All four change, so the numbers expire. Re-measure and revise after:
- A Harper version upgrade
- A topology change, including adding or removing nodes
- Significant data growth
- Any incident that produced a surprise
Date each measurement so operators can judge whether it still describes the current service.
Additional Resources
- Production Readiness Checklist for the gate that populates this plan
- Engineering RPO, RTO, and Uptime for how to derive sections 3 and 11
- Backup and Recovery for section 10
- Safe Deployments and Rollback for section 8
- Monitoring and Triage for section 9