Safe Deployments and Rollback

This guide deploys a Harper application to a cluster, waits until every node runs it, and rolls it back. It then covers limiting a release's first traffic and deciding when to expand or reverse it. The commands use the Harper CLI and match projects created with npm create harper@latest. Deploying from a CI/CD Pipeline covers pipeline authentication, private sources, and registry artifacts in more depth.

What You Will Learn

• How to deploy a Harper backend, or a frontend built once, from your machine or from GitHub Actions
• How to choose between restart=true and restart=rolling, and when to poll a job
• How to roll back from the CLI or from Studio
• How to limit a release's first traffic, and decide when to advance, hold, or reverse
• How to classify a change so you reverse the right thing, and why reversing the wrong thing can cause data loss

Prerequisites

• A Harper Fabric cluster, or a self-managed Harper Pro cluster. Rolling back to a kept release needs v5.3.0 or later.
• The Harper CLI on your machine: npm install -g harper
• An application, for example one created with npm create harper@latest
• Health Checks and Traffic Admission, for taking a node out of rotation
• Monitoring and Triage, for comparing versions during a rollout

Deploy a release

Log in

Set the cluster's Operations API URL in your project's .env:

# .env
HARPER_CLI_TARGET='https://my-cluster.example.com:9925'

npm create harper@latest asks for your cluster URL and writes it here, adding https:// if you omit the scheme and port 9925 if you omit the port. If you skipped the prompt, replace the placeholder. From the project directory, log in:

harper login

Press Enter to accept the .env value at the Cluster Target URL prompt, then enter the cluster's admin username and password. The CLI saves a token in ~/.harperdb/credentials.json, not in .env.

Added in: v5.1.0

The CLI loads .env from the directory you run it in. Commands such as harper deploy, harper list_deployments, and harper get_job then need no target=. Pinning the target here keeps this project's commands from falling back to the last cluster you logged in to. An existing environment variable wins over .env, so CI can still set HARPER_CLI_TARGET from a repository variable. See authentication for the full precedence rules.

The target URL is not a secret. To share it with your team, remove .env from the template's .gitignore and commit the file, or force-add it. Templates ignore .env and .env.*, except .env.example. Keep credentials such as CLI_TARGET_USERNAME and CLI_TARGET_PASSWORD out of it: the deploy upload includes .env, so its contents reach every node.

A Harper backend

An application that runs from source, such as a create-harper project without a frontend build, deploys straight from its directory:

harper deploy project=my-app restart=true

The CLI packages the directory and uploads it, and Harper installs its dependencies. On Fabric and Harper Pro, a deploy reaches every node unless you pass replicated=false. The create-harper templates' npm run deploy script uses restart=rolling and polls the job; see Choose a restart mode.

The upload includes everything in the directory except node_modules, including .git and any .env file. Deploy from a clean checkout, or stage only the files you want to ship, as the next section does.

A frontend built once

The React and Vue templates load the @harperfast/vite plugin, which builds the frontend on each node when the build output is missing, so every node also carries the build toolchain. To build once and ship only the output, keep a second configuration file for the release. Copy your config.yaml to deploy.config.yaml, keep its rest, graphqlSchema, jsResource, and static entries as they are, and remove the @harperfast/vite and @harperfast/schema-codegen entries, which load packages the release does not install. For the JavaScript React template, that leaves:

# deploy.config.yaml
rest: true
graphqlSchema:
  files: 'schemas/*.graphql'
jsResource:
  files: 'resources/*.js' # the TypeScript templates use 'resources/*.ts'
static:
  files: 'dist/**'

Build, copy the release into its own directory, and deploy that directory:

npm ci
npm run build
rm -rf deploy && mkdir deploy
cp -R dist schemas resources deploy/
cp deploy.config.yaml deploy/config.yaml
cp .env deploy/.env
(cd deploy && harper deploy project=my-app restart=true)

The CLI reads .env from the directory it runs in, so the staged directory needs this copy to find the target. It holds only the target URL, so shipping it is harmless. The staged directory has no package.json, so the nodes skip the install step. If your resources import npm packages, add a package.json to deploy/ that lists only those runtime dependencies. Harper Studio deploys itself this way.

Deploy from GitHub Actions

create-harper projects include .github/workflows/deploy.yaml. It tests every pull request and, when a change merges to main, deploys with restart=rolling and waits for the job. It authenticates with GitHub's OIDC token, so the repository stores no Harper credential. Set up the cluster side once from your machine, logged in as a super_user:

npm run deploy:setup-ci

Added in: v5.4.0

The script runs harper deploy setup=true provider=github-actions. It creates a deploy-only user and a trust policy pinned to main and the production environment, and sets the repository's HARPER_CLI_TARGET variable. With an earlier CLI, create the user and policy by hand, as Let the workflow authenticate with OIDC describes, pinning workflow_ref to main. Trust policies need a v5.3.0 or later cluster.

For a frontend built once, replace the template's workflow with one that builds in one job and deploys in another:

# .github/workflows/deploy.yaml
name: Deploy to Harper
on:
  push:
    branches: [main]

permissions:
  contents: read

concurrency:
  group: deploy-${{ github.ref }}

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
      - uses: actions/setup-node@v7
        with:
          node-version: 26
      - run: npm ci
      - run: npm test
      - run: npm run build
      - name: Stage the release
        run: |
          mkdir deploy
          cp -R dist schemas resources deploy/
          cp deploy.config.yaml deploy/config.yaml
      - uses: actions/upload-artifact@v7
        with:
          name: release
          path: deploy/

  deploy:
    needs: build
    runs-on: ubuntu-latest
    environment: production
    permissions:
      id-token: write
    env:
      HARPER_CLI_TARGET: ${{ vars.HARPER_CLI_TARGET }}
    steps:
      - uses: actions/download-artifact@v8
        with:
          name: release
          path: deploy
      - uses: actions/setup-node@v7
        with:
          node-version: 26
      - run: npm install --global harper@^5.3
      - name: Deploy the release
        working-directory: deploy
        run: harper deploy project=my-app restart=true

The build job runs your dependencies' install scripts and build tools, and cannot request an identity token. The deploy job can, and it runs none of your repository's code: it installs only the Harper CLI and uploads the built release. Path B explains this split, and publishes the build to a registry instead, which gives the release a version you can redeploy from any machine.

Choose a restart mode

Use restart=true for the deployments above. The node you called installs the release, then sends it to every other node at the same time. Each node installs it and replaces its HTTP workers a few at a time while the others keep serving. The node you called restarts last. Peer restarts overlap, so several nodes are short some workers at the same time.

Changed in: v5.3.0

From v5.3.0, each node waits for its restart before answering, and the command returns once every node has answered. There is no job to poll. If a node fails to take the deploy, the command fails with a non-zero exit and names it, unless you set ignore_replication_errors. The release stays deployed on the node you called. A failed restart does not fail the deploy, so check the serving version on every node through your readiness route or a journey test. Before v5.3.0, the command returns as soon as the restart is requested.

Use restart=rolling when you would rather change one node at a time. A restart_service job changes the node you called, then each other node in turn.

Changed in: v5.4.0

From v5.4.0, unless the result's certification is unavailable, the other nodes only stage the release and the job activates it on each in turn. The CLI waits until every node has taken it, exiting non-zero and naming any node that did not. wait=false exits early; wait_timeout=<ms> bounds the wait, which defaults to ten minutes per node. See waiting for a rolling deploy.

Added in: v5.4.0

For both modes, the node you called certifies the release in a canary worker before any other node receives it. A release that fails to load is rejected and replicated nowhere.

Neither mode takes a node out of rotation, drains its connections, or runs your checks between nodes. harper#2975 tracks load-balancer-aware deployments. To keep traffic off a node while it changes, drain it first, as in isolated node validation or the drain and return sequence.

Wait for every node

Before v5.4.0, or when certification is unavailable, a rolling deploy returns once the node you called has the release. Log in to one node's own URL, deploy to it, and poll its restartJobId there until COMPLETE or ERROR:

harper login https://node-1.example.com:9925
harper deploy project=my-app restart=rolling target=https://node-1.example.com:9925 json=true > deploy.json
(
  JOB_ID=$(jq -r '.restartJobId // empty' deploy.json)
  [ -n "$JOB_ID" ] || { echo "The rolling deploy returned no restartJobId"; exit 1; }
  for attempt in $(seq 60); do
    sleep 10
    harper get_job id="$JOB_ID" target=https://node-1.example.com:9925 json=true > job.json || continue
    case $(jq -r '.[0].status // "MISSING"' job.json) in
      COMPLETE) echo COMPLETE; exit 0 ;;
      ERROR) jq -r '.[0].message' job.json; exit 1 ;;
      MISSING) echo "Job is not on the node that answered" ;;
    esac
  done
  echo "Job $JOB_ID did not complete within 60 polls"
  exit 1
)

Jobs are not replicated, so the script deploys to and polls one node rather than the load-balanced cluster URL in .env, where a poll can reach a node that never saw the job. See the job lives on one node. Without certification, including before v5.4.0, a COMPLETE job can still leave a node it could not reach on the old release. Check the serving version on every node afterwards.

Two deploy parameters control replication failures:

• deployment_timeout Added in: v5.1.4 is how long a peer waits for the replicated payload before failing, defaulting to 120000 ms. Raise it for large components or slow links.
• ignore_replication_errors Added in: v5.1.4 treats a peer that fails to receive the deploy as non-fatal. By default a failed peer makes the whole operation return a non-2xx status, while the component is still deployed on the origin node. Decide which behavior you want before you need it.

Roll back

From the CLI

Added in: v5.3.0

When a deploy replaces a release, each node keeps the replaced one under its deployment_id. List recent releases, newest first:

harper list_deployments project=my-app status=success limit=5 json=true \
  | jq -r '.deployments[] | "\(.deployment_id)  \(.started_at / 1000 | floor | todate)"'

Activate the release you want back:

harper deploy project=my-app deployment_id=<deployment_id> restart=true

Nothing is rebuilt or reinstalled. The command waits for every node to answer; check the serving version on every node afterwards. A node that no longer keeps the release refuses it: deployment.stagingRetention.maxCount sets how many replaced releases each node keeps, defaulting to 5. See going back to a previous release for which releases are kept.

A kept release that loaded a component through a package: entry, such as the templates' @harperfast/schema-codegen, cannot be activated: the deploy fails with HTTP 409 and a message saying a node_modules/.../node_modules/harper link "links outside the build". Releases without package: entries, such as the built-once frontend above, activate normally.

Before v5.3.0, when no node keeps the release, or when a package: component prevents activation, deploy the previous release again: check out its commit and repeat the deploy, or pass the previous version as package.

From Studio

Added in: v5.3.0

In Harper Studio, open the cluster and go to Config > Deployments. Open the release you want back and choose Roll back to this release. Studio activates the same kept release on every node, one at a time, and shows the rollout's progress; the same package: component limit applies. Check the serving version on every node afterwards.

Limit a release's first traffic

A replicated deploy changes every node. To test a release on part of your traffic first, separate installing it from exposing it, and decide at each step whether to expand.

1. Preflight. Confirm the target version, component inventory, replication convergence, peer capacity with one node held out, and backups if data is at risk. Check that the prior release can still be activated or redeployed. Name the decision owner and set stop thresholds before starting.
2. Limit. Start with one drained node, an internal group, selected tenants, a region, or a small traffic percentage. Choose enough requests to make the comparison meaningful.
3. Observe. Compare old and new versions on request success, SLO latency, worker saturation, data correctness, replication, and downstream errors.
4. Decide. Advance, hold, or roll back against the agreed thresholds. Hold if required telemetry is missing.
5. Expand. Increase exposure only after minimum sample and hold conditions pass, and keep enough healthy capacity on the old version to reverse.
6. Close. Verify the same release and configuration on every node, restore intended traffic, and record the outcome with the measurements that supported each decision.

Pattern	How Harper is used	Fits when, and watch out for
Isolated node validation	Drain one node, deploy with replicated=false, validate, re-admit a group	Strongest validation before any user sees the release. Needs spare capacity and a direct URL for each node
Replicated deployment	harper deploy with restart=true or restart=rolling	Use true for overlapping peer restarts, or rolling to change one node at a time. Both keep nodes in rotation
Feature-targeted release	Deploy compatible code broadly, enable it by flag, header, or tenant	Best when behavior can be separated from deployment. Assign an owner and a removal date to each flag
Parallel environment	Deploy to a separate node pool, shift traffic after validation	Strong isolation and fast reversal. Write ownership and replication between pools need real design
Regional progression	Deploy the same release region by region, exposing each after its checks	Limits the blast radius. One region does not predict every traffic shape

Isolated node validation needs at least three nodes, so the others can carry the traffic of the node you hold out. Take one node out of rotation with the drain sequence, log in to its own URL, and deploy to it alone:

1. Validate on one node

harper login https://node-a.example.com:9925
harper deploy project=orders-api package=@my-org/orders-api@2.4.1 \
  replicated=false restart=true target=https://node-a.example.com:9925

Run your readiness route and journey tests against that node directly, then admit a small group of users to it.

2. Deploy to the rest

harper deploy project=orders-api package=@my-org/orders-api@2.4.1 \
  restart=true

On v5.3.0 and later, the command waits for every node to answer. Check the serving version on every node afterwards. To change one node at a time, use the rolling mode.

Deploy a pinned version or commit rather than a moving branch, so you can tell what ran yesterday and deploy it again. A private registry needs a credentials reference.

danger

Component deployment replicates by default. Pass replicated=false when validating on one node. Other operations have different replication defaults; check the operation's reference before an isolated rollout.

To activate a release for part of your traffic without a second deploy, mount it at a path or hostname. urlPath mounts a component at an HTTP path and host serves it on a virtual hostname. Routing by either has been available since v5.1.0. What arrived in v5.2.0 is declaring that routing on the application's entry in the root harper-config.yaml, and the matching host parameter on deploy_component. Declared that way, the routing is persisted deployment state rather than a runtime toggle. See HTTP middleware routing.

Configuration changes are deployments too

Apply the same review, rollout, and rollback checks to configuration changes.

set_configuration supports replicated=true Added in: v5.2.0 to apply a change across the cluster in one call, with per-node outcomes in the response. For settings that apply on worker restart, follow with restart_service, which restarts workers on each node in turn:

harper set_configuration logging_level=info replicated=true
harper restart_service service=http replicated=true

Check the setting's restart requirements first. Process-level settings such as threads.v8Flags Added in: v5.4.0 require a full process restart; restart_service does not apply them. Drain and restart each node in turn, using the restart operation or your process or container manager, and verify it before advancing. The restart operation does not replicate.

danger

Only replicate cluster-appropriate parameters. Node-local values such as ports, node.hostname, file paths, TLS material, and replication.hostname, url, or routes would overwrite every peer's own values. Replicating one of these is a cluster-wide outage delivered in a single API call.

get_status.restartRequired tracks component and code restarts, so it does not identify pending set_configuration changes. Record those changes yourself, use the required restart type, and read back harper get_configuration. Read-back confirms the configured value; verify the running behavior too.

Classify the change before reversing it

Different changes need different reversal paths, with different compatibility requirements and blast radii:

Change type	Reversal	Watch out for
Component release	Activate the previous deployment_id, or redeploy it	Each node must still keep the release, or the artifact must exist
Feature behavior	Turn the flag off, no deploy needed	Fastest reversal available. Requires the flag to have been built in
Traffic exposure	Restore prior weights or set the node unavailable	Fastest containment. Does not undo anything already written
Configuration	set_configuration back, then restart	Needs a restart, so it is not instant
Harper runtime version	Version-specific procedure	Mixed versions change replication and deploy behavior
Schema or data	Forward repair or restore	See the warning below

• Rehearse rolling back before launch, not during an incident.
• Prefer expand-then-contract schema evolution. Add the new shape, migrate, then remove the old shape in a separate change. Destructive schema behavior depends on the operation and version, so each destructive change needs an exact written procedure and verification on every node.
• Use the same checks for rollback as for forward movement: availability, journey tests, peer stability, data validation, and traffic. A rollback is a deployment and can fail like one.

warning

If a release changed the meaning of persisted data, code rollback alone will not fix it, and restoring a database to undo application code is usually the wrong move. A restore rolls back every write in the window, including all the valid ones, so it can violate your RPO in order to fix a code bug. Define forward repair or replay for these cases instead. See Backup and Recovery.

Prove it

Rehearse a stopped rollout end to end on a non-production cluster:

1. Declare a threshold before you start, for example "stop if journey success on the new group is more than 0.5 percent below the control group over five minutes."
2. Deploy a release that fails that threshold deliberately, to a limited group.
3. Detect it through your dashboards rather than because you know what you did.
4. Roll it back, and record both the time from the decision to verified service and the total time from failure to verified service. Compare the total, including detection and decision time, with your RTO.
5. Verify every node afterwards: harper get_components against each node's URL, and harper list_deployments project=<name> showing the rollback.

Operational notes

• Keep Harper versions the same on every node. Mixed versions change replication and deployment behavior, so detect and finish a rollout that stalls halfway.
• Keep deploy credentials in your delivery platform's secret store, or use the OIDC setup above so there is none. Use TLS and least privilege, and keep the deploy result as change evidence. See secrets.
• Keep a single-node deployment path. Every deploy replicates by default, so without replicated=false and a direct node URL you cannot test anything on one node.

Readiness checklist

• Each release is a pinned version, a commit, or a staged build, never a moving branch
• The rollback path is rehearsed: on v5.3.0 and later the previous deployment_id is kept on every node; otherwise the previous artifact is still available
• Deploy and expose are separate steps for changes that need a limited first audience
• Cohort ladder defined, from smallest useful sample to full exposure
• Stop thresholds and hold times declared before the rollout starts
• A named decision owner for advance, hold, and roll back
• Pipeline waits for the chosen restart mode to finish, polling the job when required, then checks the serving version on every node
• Nodes that must not serve while they change are drained first, since neither restart mode takes nodes out of rotation
• replicated=false used deliberately wherever a change is meant to reach one node only, since replication is the default
• Restart mode chosen deliberately: overlapping peer restarts with restart=true, or one node at a time with restart=rolling
• deployment_timeout and ignore_replication_errors set deliberately
• Configuration changes go through the same change record as code, and get_configuration is read back after each one
• Schema changes follow expand-then-contract, with destructive operations documented per node
• Forward repair defined for changes that alter persisted meaning
• Rollback duration and total recovery time recorded from a rehearsal, with the total compared against RTO

Additional Resources

• Deploying from a CI/CD Pipeline for OIDC setup, private sources, and registry artifacts
• harper deploy for every CLI deploy parameter
• deploy_component reference for the server-side behavior, including urlPath, host, and replication controls
• Deployment records for list_deployments and get_deployment
• Static files for serving a built frontend
• HTTP middleware routing for urlPath and host activation
• Configuration operations for set_configuration and restart requirements
• Database schema for schema evolution
• Multiple Applications on One Cluster for running more than one component side by side