Skip to main content

Monitoring and Triage

Monitor the parts of the service you can investigate or change: the public route, individual Harper nodes, replication, downstream dependencies, and recent deployments.

This guide identifies useful signals at each boundary and gives you a procedure for narrowing an incident.

What You Will Learn​

  • The layers a Harper symptom can live in, and the minimum signal for each
  • Which Harper metrics are worth an alert, by name, and which are only worth a dashboard
  • How to get metrics out of Harper, on Fabric and self-managed
  • How to preload an APM or tracing agent so it can instrument your application
  • How to write log entries that are still useful during an incident
  • A triage sequence that narrows a Harper incident in about five minutes

Prerequisites​

  • A cluster with your application deployed and taking traffic
  • A super_user credential for the Operations API
  • Somewhere to send metrics: Grafana on Fabric, or a Prometheus-compatible system for self-managed
  • Operating Replication, since replication signals are half of what you will watch

Monitor at the boundaries you operate​

LayerMinimum signalsWhere they come from
User and edgeSuccess, latency, correctness, broken out by geography and cohortYour synthetic journey, real-user telemetry, CDN and traffic layer logs
HTTP and componentRequest rate, error rate, latency percentiles, active component version, availability flagsuccess and duration metrics, GET /status, your readiness route
Node resourcesCPU, memory, worker utilization, task queue latency, disk, restartssystem_information, utilization, main-thread-utilization, host or container metrics
ReplicationPeer connections, per-database sockets, latency, convergence lagcluster_status, replication-latency, system.hdb_nodes
DataRead and write rate, transaction commit time, queue depth, freshnesstransaction-commit-time, write and read transaction queue depth, your own sentinel
StorageDatabase size, volume free space, table growthdatabase-size, storage-volume, table-size
ChangeComponent and config version, cohort, operator, start and end, outcomelist_deployments, get_deployment, get_components, get_configuration read-back
RecoveryBackup age, job state, verification result, last restore drilllist_backups, verify_backup, get_job

Show deployments and configuration changes alongside request errors and latency. Their timing helps you identify whether a release affected one node, one version, or the whole cluster.

The signals worth an alert​

Harper records a large standard metric set automatically. Most of it belongs on a dashboard. This much belongs on a pager:

Alert onMetric or sourceWhy this one
Journey success rate below SLOsuccess, by resource path and methodThe only signal that maps directly to user impact
Journey latency at your SLO percentileduration, by resource path and methodAlert on the percentile in your SLO, never on the mean
Worker saturationutilization, per worker threadPercentage of time the worker was processing requests. This is your real headroom signal
Event loop backpressuremain-thread-utilization, the taskQueueLatency attributeRises before throughput drops, so it is an early warning rather than a postmortem input
Replication convergence lagreplication-latency, source timeline, and sentinel checksConnection health alone does not establish that a node's data is current
Write commit timetransaction-commit-time Added in: v5.2.0Storage-level degradation shows here before it shows in request latency
Write queue depth growingwrite-transaction-queue-depth Added in: v5.2.0A growing queue means the node is accepting work faster than it can commit it
Storage headroomstorage-volume free, and database-sizeDisk exhaustion is an outage with no graceful degradation
Missing peercluster_statusCovered in Operating Replication
Verified recovery point older than RPOBackup manifest and destination/restore validation recordsA successful job alone does not establish that the copy can restore service

Alert on the latency percentile in your SLO. Set saturation thresholds from your measurements in Sizing a Harper Cluster, where you determine the load at which latency becomes unacceptable.

Get the metrics out​

The CLI examples use direct Operations API URLs. Log in once to each URL, for example harper login https://node-1.example.com:9925. Replace the example URLs with your own; a load-balanced URL answers from an arbitrary node.

Harper stores raw analytics in hdb_raw_analytics and aggregates them into hdb_analytics. Query these tables or export metrics to your monitoring system.

Use the Grafana integration. It ships dashboards over Harper's analytics without you building a pipeline, which is the fastest path to the alert list above.

To find out what is actually available on your version rather than guessing from documentation:

harper list_metrics metric_types='["builtin","custom"]' target=https://node-1.example.com:9925 json=true

Use describe_metric for one metric's shape:

harper describe_metric metric=resource-usage target=https://node-1.example.com:9925 json=true

Your application can add its own metrics with server.recordAnalytics(), which is how you get business-level signals such as checkout completion onto the same dashboard as node saturation. That correlation is usually what tells you whether a technical symptom matters.

tip

Check analytics.replicate in the analytics configuration before building dashboards. Whether metrics stay node-local or replicate changes what a cluster-wide query means, and it is easier to decide that deliberately than to discover it after building panels on the wrong assumption.

Run an APM or tracing agent​

Harper's own metrics tell you how the node is behaving. They do not give you distributed traces across your application code and its downstream calls, which is what you want when a journey is slow and you need to know where the time went. That comes from an instrumentation agent, and an agent has to load before the code it instruments.

Two configuration keys put a module on each worker thread's startup, ahead of Harper's own modules and yours:

threads:
  preloadRequire: dd-trace/init # the entry that calls init()
  preload: dd-trace/register.js # ESM loader hooks for automatic instrumentation

threads.preload Added in: v5.2.0 loads a module via Node's --import, which is how an agent installs the loader hooks that let it instrument modules imported later. threads.preloadRequire Added in: v5.2.0 uses --require, which runs the module body and is typically how an agent's initialization entry actually starts it. Both are documented under threads configuration.

Agent entry points vary: some install loader hooks and start the agent together; others need separate preload and preloadRequire entries. Loading hooks without starting the agent can produce no-op spans and no exports.

Both keys apply only to worker threads and require Node.js. Bare specifiers resolve against installed components' node_modules, so the agent can be a component dependency.

Whatever agent you use, follow its own worker-thread documentation, and finish by confirming spans actually arrive at your collector rather than assuming the configuration took.

Logs you can use during an incident​

Harper's logger global takes a message plus a context object from component code. Call this helper after a successful submission, passing the request context and measured duration in milliseconds:

import { server } from 'harper';

function logOrderSubmitted(context, elapsed) {
	logger.info('order submitted', {
		component: 'orders-api',
		version: process.env.APP_VERSION,
		node: server.hostname,
		requestId: context.requestId,
		operation: 'submit',
		durationMs: elapsed,
		outcome: 'ok',
	});
}

Include the component and its version, the node, a request or trace identifier, the operation, its duration, the outcome, and a safe error class. That field set is what lets you answer "was this only the new version" and "was this only one node" without guessing, and those are the first two questions in almost every incident.

Three details in that example need care:

  • The node name comes from server.hostname, not an environment variable. Harper does not set a node-name variable in the process environment, and server.hostname is the same identity analytics uses, so it is what correlates with your metrics.
  • context.requestId is only populated when http.logging.id is enabled, and HTTP request logging is off by default. Without it the field is undefined and you silently lose your correlation id. Either enable it in logging configuration or generate an id in your own code.
  • APP_VERSION is yours to inject. Harper does not provide it. Set it in your deployment so the log line can name the build.
warning

Harper's logger is built on Node's Console, so this renders as a formatted text line, not JSON. The context object is inspected into the message rather than emitted as separate fields:

[main/3] [info]: order submitted { component: 'orders-api', version: '2.4.1', ... }

That is fine for reading during an incident, but a log pipeline cannot key on component or outcome without parsing the line. If you need queryable fields, serialize the context yourself and log a single JSON string.

Levels run trace, debug, info, warn, error, fatal, and notify. The default is warn, which drops the example's info message. Set logging.external.level to info to retain it, and check the resulting volume. notify is always logged.

console.log output does not reach the log files unless logging.console is enabled, so unstructured console output is not a production record. Centralize logs off the node, because the node you most need logs from is the one you are about to restart.

Read logs through the API when you need them from a specific node:

harper read_log limit=200 level=error target=https://node-1.example.com:9925 json=true

The five-minute triage sequence​

Use these checks to locate the affected layer, then contain the failure before investigating its cause.

  1. Confirm and bound the impact. What journey, starting exactly when, in which geography or cohort, and which component version is live. Without a start time you cannot correlate anything.

  2. Compare the public route against a direct node call. Request the same journey through your traffic layer, then directly against each node on 9926. If direct calls succeed and the public route fails, you are looking at the traffic layer or the availability flag, not at Harper.

  3. Check admission state on every node. Use each node's configured application URL for /status and your readiness route. An unavailable node is out of rotation.

    for node in https://node-1.example.com https://node-2.example.com https://node-3.example.com; do
      echo "== $node"
      curl -s -o /dev/null -w '%{http_code}\n' "$node/status"
      curl -s -w '\n%{http_code}\n' "$node/Readiness"
    done

  4. Compare a suspect node against a healthy peer. Same call, both nodes, then diff:

    for node in node-1 node-2; do
      harper system_information target="https://$node.example.com:9925" \
        attributes='["cpu","memory","threads","harperdb_processes"]' json=true > "$node.json"
    done
    diff -u node-1.json node-2.json

    Differences between peers are more informative than absolute values, because they tell you whether this is one node or the whole cluster.

  5. Check replication. Inspect the expected peer/database sockets in cluster_status, sustained latency, and lastCommitConfirmed progress while writes continue. Compare lastReceivedRemoteTime with the expected source timeline and verify freshness with a sentinel or journey test. Timestamp differences alone do not prove convergence.

    for node in https://node-1.example.com:9925 https://node-2.example.com:9925 https://node-3.example.com:9925; do
      echo "== $node"
      harper cluster_status target="$node" json=true
    done

  6. Correlate with the last change. list_deployments for what shipped and when, and get_configuration read back against what you believe is configured. Remember that a configuration change applied without a restart leaves a node running something other than its stated configuration.

    for node in https://node-1.example.com:9925 https://node-2.example.com:9925 https://node-3.example.com:9925; do
      echo "== $node"
      harper list_deployments limit=5 target="$node" json=true
      harper get_configuration target="$node" json=true
    done

  7. Contain with the smallest reversible action. Stop a ramp, drain one node, deactivate a feature, restore prior traffic weights, or isolate suspected data. Record the decision, who made it, and the next decision deadline.

If required telemetry is missing, investigate the gap before declaring the service healthy. Record containment actions so the next operator can see what changed.

Prove it​

Pick a fault and inject it on a non-production cluster, then time yourself. Good candidates: saturate one node's workers, block the replication port on one peer, make a downstream dependency return errors, or deploy a component that fails on one route.

Record time to alert, time to identify the affected layer, and any misleading or missing signals. Update the procedure before repeating the drill.

Operational notes​

  • Node-level dashboards, not cluster averages. A cluster average conceals the single node doing twice the work, which is the most common cause of a latency complaint that looks like nothing on a dashboard.
  • Watch measured request distribution, not configured weights. Per-node request counts are the ground truth. Sticky sessions, DNS caching, and connection reuse all skew actual distribution away from intent.
  • Annotate deployments onto your graphs. If your monitoring supports annotations, feed list_deployments into them. This single change resolves more incidents faster than any additional metric.
  • read_audit_log needs transaction logging enabled and is a heavier tool for reconstructing what changed in a table. Know before an incident whether you have it on, because turning it on afterwards does not help.
  • Protect your telemetry surfaces. read_log and get_components can expose configuration and source detail, so they are super_user operations for a reason. Restrict and log their use.

Readiness checklist​

  • A dashboard exists that can place a symptom at edge, traffic layer, node, replication, data, or change
  • Alerts on journey success and latency at the SLO percentile, not the mean
  • Alerts on worker utilization and taskQueueLatency, thresholds derived from measured per-node capacity
  • Alerts on replication convergence lag and missing peers
  • Alerts on storage headroom and the newest fully verified recovery point's age
  • Metrics exported off the node, via Grafana on Fabric or the Prometheus exporter self-managed
  • If you run an APM agent, both threads.preload and threads.preloadRequire set as that agent requires, with spans confirmed at the collector
  • analytics.replicate setting known and deliberate
  • Structured logging includes component, version, node, request id, operation, duration, and outcome
  • Logs centralized off the node
  • Deployments annotated onto dashboards
  • Triage sequence written down where on-call can find it
  • Fault injection drill completed, with time-to-detection recorded

Additional Resources​