Skip to main content

Sizing a Harper Cluster

Size a Harper cluster so that, after losing F nodes, the survivors can carry peak demand within your utilization limit.

This guide works through the capacity calculation, the load tests it depends on, and the headroom needed for maintenance and recovery.

What You Will Learn​

  • The capacity invariant that sizes for the failure state, with a worked example
  • What workload and recovery conditions to include in a per-node throughput measurement
  • Why maintenance is a planned failure, and what that means for change windows
  • How to choose a topology from your objectives rather than from a diagram
  • How to convert an availability SLO into minutes per month, and what each tier demands of your automation

Prerequisites​

  • How Harper Runs in Production, and your service boundary inventory
  • Health Checks and Traffic Admission, because capacity headroom is meaningless if traffic cannot be moved off a node
  • A load generation tool and a non-production cluster you can push to saturation
  • A stated peak throughput requirement for your critical journeys

Size for the failure state​

The rule:

(N - F) x tested per-node throughput x target utilization >= required peak throughput

Where N is your node count, F is the largest simultaneous node loss the design tolerates, and target utilization is the fraction of measured capacity you are willing to run at, which should leave room for latency to stay acceptable rather than merely for requests to complete.

Worked example. Suppose your critical journey needs 12,000 requests per second at peak, a node sustains 5,000 requests per second in your own test with acceptable latency, you will run at 70 percent utilization, and you want to survive losing one node:

Required surviving capacity = 12,000 / 0.7        = 17,143 req/s
Surviving nodes needed      = 17,143 / 5,000      = 3.43 -> 4
N                           = 4 + F (1)           = 5 nodes

This requires five nodes. Three would handle the peak only while all remain available, without the chosen utilization headroom.

In this example, tolerating two simultaneous failures requires six nodes. The 70 percent utilization limit also matters: it leaves the surviving nodes room to handle variable load while keeping latency acceptable.

Measure per-node capacity​

Use production-like conditions when measuring throughput:

  • Representative application code, data shape, and query mix. A Harper node runs your component logic in the same process as the data access, so your code is part of the capacity measurement in a way it would not be for a standalone database. A synthetic key-value benchmark tells you very little about the node's capacity to serve your journey.
  • Realistic downstream latency. If your resource calls an upstream pricing service, its latency occupies worker capacity on the node.
  • Replication catch-up load. A node that is feeding a recovering peer is doing work that does not appear in its own request metrics. Measure with a peer synchronizing, because that is precisely the state you will be in when you are already down a node.
  • Traffic imbalance. Configured weights express intent. Actual request distribution is what consumes capacity, and it is rarely even.

Measure each node separately rather than dividing a cluster total by node count. The CLI saves a login per URL. Log in once to each node's direct Operations API URL, for example harper login https://node-1.example.com:9925. Replace the example URLs with your own; a load-balanced URL answers from an arbitrary node.

for node in https://node-1.example.com:9925 https://node-2.example.com:9925 https://node-3.example.com:9925; do
  echo "== $node"
  harper system_information attributes='["cpu","memory","disk","network","threads","harperdb_processes"]' target="$node" json=true
done

Watch CPU, memory, event loop and worker pressure from the threads attribute, disk, and network, and pair them with request rate and latency from analytics. The number you want is not the point where requests start failing. It is the point where your journey's latency leaves your SLO, which arrives earlier.

tip

Record the conditions alongside the number: Harper version, node size, data volume, query mix, and whether a peer was synchronizing. A per-node capacity figure without its conditions is not reusable, and six months later nobody will remember whether the test included replication load.

Maintenance is a planned failure​

Draining a node for a deploy, an upgrade, or an investigation consumes exactly the same headroom that a node failure does. So a cluster sized for F = 1 is running at F = 0 for the duration of every maintenance window, with no tolerance left.

The operating rule that follows: if draining one node means the service can no longer tolerate its declared F, pause the change. Either wait for lower traffic, or add capacity for the window, or accept and document the reduced tolerance for a bounded period. What you should not do is treat the maintenance window as free because nothing is technically broken.

Sizing for F = 2 can let you drain one node for maintenance and still tolerate another node's failure at peak load.

Choose a topology from objectives​

Pick the pattern that matches the failures you actually need to survive, and know what you have to prove for it to count.

PatternWhen it fitsWhat you must prove
Single region, multiple peersNode and host failures dominate, and low inter-node latency mattersN - F capacity under load, peer catch-up time, traffic removal, backup independence
Multiple failure domains, one regionZone or rack isolation is available from your providerNo shared power or network fault, acceptable replication latency across domains, balanced traffic
Multiple regionsRegional impairment or user locality is a real requirementData authority per region, route propagation time, convergence, degraded-mode behavior
Edge or on-premises peersData locality or residency constrains where data can liveReplication scope per location, identity provisioning, backup path from each site

Test the failures your topology is meant to survive. Confirm that supposedly independent domains do not share a power supply, network path, or other dependency that could take them down together.

Note that adding a node is a data movement event as well as a capacity event, since a Harper node carries data along with request handling. A node whose databases have never synchronized downloads them in full before it is useful, so scaling out is not instantaneous and cannot be your response to an unexpected traffic spike. Size ahead of demand.

Turn the SLO into a budget​

An availability target is a quantity of unavailability you are permitted to spend per month. Written that way it becomes an operating constraint rather than an aspiration.

Monthly SLOApproximate maximum unavailabilityWhat it demands
99.9%43 min 50 secManual diagnosis can fit inside some incidents, but one untested maintenance window can consume the month
99.95%21 min 55 secAutomated node removal and a rehearsed release reversal become necessary
99.99%4 min 23 secNode failure must be close to transparent. Detection, traffic removal, and validation have to be automated, because no human response fits in the budget

Figures assume a 30.44 day month. Your contractual definition, exclusions, and measurement boundary may differ, and the boundary matters more than the number: availability measured at your CDN edge and availability measured at the Harper node are different quantities.

Compare your measured detection and recovery times with the budget. A manual response that fits a 99.9 percent target may consume the entire budget at 99.99 percent.

Prove it​

Run the failure state rather than calculating it. On a cluster carrying representative load at your expected peak:

  1. Record per-node request rate and latency from analytics, and saturation with the system_information loop above. Repeat those measurements during the drain and rejoin.
  2. Drain one node using the sequence from Health Checks and Traffic Admission.
  3. Measure what the survivors do: request rate, latency at your SLO percentile, CPU and worker pressure, and whether any user-visible errors appeared during the transition.
  4. Keep the drained node out of user traffic while it rejoins replication. Measure the survivors while they carry peak traffic and feed the recovering peer.
  5. Compare measured surviving capacity against your invariant. If step 4 pushed latency out of SLO, your target utilization is too high or your N is too low.

Operational notes​

  • Utilization targets belong per journey, not per cluster. A write-heavy journey and a cached read journey consume very different resources on the same node, so a single cluster-wide utilization figure will be wrong for both.
  • Re-measure after a version upgrade. Per-node capacity is a property of a Harper version, your component code, and your data volume. All three change.
  • Watch measured distribution, not configured weights. Per-node request counts from analytics are the ground truth. A misconfigured weight, a sticky session policy, or DNS caching can leave one node doing far more work than the topology claims.
  • Storage growth is a capacity dimension too. Disk headroom, compaction behavior, and backup space all scale with data volume rather than with request rate, so they need their own thresholds. See compaction.
  • On Fabric, cluster shape is managed but the invariant is unchanged. You are still choosing N and living with F.

Readiness checklist​

  • F declared explicitly, and written down where change operators will see it
  • Per-node throughput measured with your own application code and data shape
  • Measurement taken with a peer synchronizing, not on an idle cluster
  • Target utilization set from latency at your SLO percentile, not from request success
  • (N - F) x per-node x utilization >= peak verified with real numbers
  • Measurement conditions recorded alongside the capacity figure
  • Maintenance policy states what happens when a drain would breach F
  • Topology's "what you must prove" column exercised, not assumed
  • SLO converted to minutes per month, with the measurement boundary named
  • Peak-load drain and rejoin exercise completed and dated

Additional Resources​