# Harper Docs > Harper is a single in-process runtime that unifies your database, application logic, APIs, caching, real-time messaging, and agent loops, replicated across a multi-region data application fabric. This site is the official technical documentation: guides, API and configuration reference, and Fabric operations. ## search - [Search the documentation](/search.md) ## fabric - [Fabric Studio](/fabric.md): Fabric Studio is the web-based GUI for Harper. Studio enables you to administer, navigate, and monitor all of your Harper clusters in a simple, user-friendly interface without any knowledge of the underlying Harper API. It’s free to sign up, get started today! - [API Documentation](/fabric/api-documentation.md): Harper provides automatically generated API documentation page via Swagger UI integration. This page allows developers to explore and test the various API endpoints available in your Harper instance/cluster. - [Cluster Creation & Management](/fabric/cluster-creation-management.md): What is a Cluster? - [Create a Fabric Studio Account](/fabric/create-account.md): Start at the Harper Fabric Studio sign up page. - [Create an Organization](/fabric/create-organization.md): What is an organization? - [Custom Domains](/fabric/custom-domains.md): What are Custom Domains? - [Database Management](/fabric/database-management.md): Import CSV Data - [Setup Grafana Integration with Harper](/fabric/grafana-integration.md): Grafana is an observability platform for monitoring and visualizing metrics, logs, and traces. Harper provides a plugin to integrate with Grafana for enhanced analytics and visualization capabilities. - [Logging](/fabric/logging.md): Log Filtering - [Managing Applications](/fabric/managing-applications.md): After setting a username/password for cluster, you will automatically be directed to the applications page for the cluster. From here, users can either import or create an application to run on Harper. - [Organization Management](/fabric/organization-management.md): Organizations can be managed in a variety of ways, including: roles and user permissions, adding/removing users, creating new environments (clusters) and updating billing information. ## learn - [Welcome to Harper Learn](/learn.md): This documentation section contains thorough guides for learning how to develop and manage applications with Harper. The guides are present in a logical order to build up knowledge across Harper's vast feature set. The guides are example based and provide a hands-on approach to teaching and demonstrating key features. Guides can be referenced independently, but assume the reader is familiar with concepts presented in previous guides. - [Coming Soon](/learn/administration/coming-soon.md) - [Active Caching with Subscriptions](/learn/developers/active-caching-subscriptions.md): The passive caching pattern — fetching from the source on demand, expiring on a timer — works well for data that changes infrequently and where brief staleness is acceptable. But for data that changes often, a TTL is a blunt instrument: too short and you're making unnecessary upstream calls; too long and you're serving stale data. - [Caching AI Generations with Harper](/learn/developers/caching-ai-generations.md): AI API calls are expensive. Generating a product description, summarizing an article, or personalizing a recommendation with a large language model can cost fractions of a cent per call — but those fractions add up fast at scale. And in most applications, the same content gets generated over and over: the same product page viewed by thousands of users, the same document summarized dozens of times. - [Caching with Harper](/learn/developers/caching-with-harper.md): Every production application hits the same wall eventually: an external API you depend on is slow, rate-limited, or expensive to call. Harper's caching system lets you wrap any external data source — a REST API, a microservice, a database — and serve responses from a fast local cache while transparently fetching fresh data only when needed. - [Harper Applications in Depth](/learn/developers/harper-applications-in-depth.md): In the Getting Started guides you successfully installed Harper and created your first application. You experienced Harper's schema and database system, and the automatic REST API feature too. This guide dives deeper into Harper's component architecture differentiating applications from plugins, and introduces multiple new ways to interact with Harper. By the end of this guide you will be a confident Harper application developer capable of creating just about anything with Harper! - [Writing quality MCP and OpenAPI descriptions](/learn/developers/mcp-and-openapi-metadata.md): When an MCP client connects to Harper, the LLM on the other side sees your application as a list of tools. The text it reads to pick the right tool — the tool description, the per-attribute property descriptions, the output schema shape — is the dominant signal for tool selection. The same metadata also drives Harper's OpenAPI document, which any HTTP API consumer (Swagger UI, Redoc, generated SDKs, machine clients) reads. - [Semantic Caching with Vector Indexing](/learn/developers/semantic-caching-vector-indexing.md): Caching LLM responses by exact prompt text catches almost nothing useful. "What shoes are good for hiking?" and "Which shoes work best for hiking?" are semantically identical but would be cache misses for each other with a string key. The result is that you pay for every generation even when a good answer already exists in the cache. - [Write-Through Caching](/learn/developers/write-through-caching.md): Read-through caches — where Harper fetches from the source on cache misses — only cover half the story. When clients write data, those writes still go directly to the origin. The cache only learns about the change the next time the TTL expires or someone calls invalidate. - [Create Your First Application](/learn/getting-started/create-your-first-application.md): With Harper successfully installed and setup, let's dive into building your first Harper Application, a simple REST API. Harper lets you build powerful APIs with minimal effort. - [Install and Connect Harper](/learn/getting-started/install-and-connect-harper.md): One of Harper's primary goals since day one was to be easy to install and get started with. The core Harper application itself is just a Node.js application with some native module dependencies. The simplest and easiest way to get started using Harper is by installing it using npm (or any npm compatible Node.js package manager). In addition to installing Harper directly to your local development environment, the Harper team provides a Docker image (harperfast/harper), and most recently a platform service called Harper Fabric. - [Using AI Agents](/learn/getting-started/using-agents.md): AI-powered development tools can significantly accelerate your workflow when building Harper applications. Whether you want a dedicated assistant or prefer using your favorite LLM, Harper provides the tools and context needed to make AI an effective part of your development process. ## reference - [Harper v4 Reference](/reference/v4.md): Complete technical reference for Harper v4. Each section covers a core feature or subsystem — configuration options, APIs, and operational details. - [Analytics Operations](/reference/v4/analytics/operations.md): Operations for querying Harper analytics data. All operations require superuser permission. - [Analytics](/reference/v4/analytics/overview.md): (resource and storage analytics) - [CLI Authentication](/reference/v4/cli/authentication.md): The Harper CLI handles authentication differently for local and remote operations. - [CLI Commands](/reference/v4/cli/commands.md): This page documents the core Harper CLI commands for managing Harper instances. For Operations API commands available through the CLI, see Operations API Commands. - [Operations API Commands](/reference/v4/cli/operations-api-commands.md): The Harper CLI supports executing operations from the Operations API directly from the command line. This enables powerful automation and scripting capabilities. - [Harper CLI Overview](/reference/v4/cli/overview.md): The Harper command line interface (CLI) is used to administer self-installed Harper instances. - [Applications](/reference/v4/components/applications.md): The contents of this page primarily relate to application components. The term "components" in the Operations API and CLI generally refers to applications specifically. See the Components Overview for a full explanation of terminology. - [Extension API](/reference/v4/components/extension-api.md): As of Harper v4.6, a new iteration of the extension system called Plugins was released. Plugins simplify the API and are recommended for new extension development. See the Plugin API reference. Both extensions and plugins are supported; extensions are not yet deprecated. - [JavaScript Environment](/reference/v4/components/javascript-environment.md): Harper executes component JavaScript inside Node.js VM contexts — isolated module environments that share the same Node.js runtime but have their own global scope. This means each component runs in its own module context while still being able to access Harper's global APIs without any imports. - [Components](/reference/v4/components/overview.md): Components are the high-level concept for modules that extend the Harper core platform with additional functionality. Components encapsulate both applications and extensions. - [Plugin API](/reference/v4/components/plugin-api.md): (experimental) - [Configuration Operations](/reference/v4/configuration/operations.md): Operations API endpoints for reading and modifying Harper configuration. - [Configuration Options](/reference/v4/configuration/options.md): Quick reference for all harperdb-config.yaml top-level sections. - [Configuration](/reference/v4/configuration/overview.md): Harper is configured through a YAML file called harperdb-config.yaml located in the Harper root directory. By default the root directory is a folder named hdb in the home directory of the current user. - [API](/reference/v4/database/api.md): Harper exposes a set of global variables and functions that JavaScript code (in components, applications, and plugins) can use to interact with the database system. - [Compaction](/reference/v4/database/compaction.md): Database files grow over time as records are inserted, updated, and deleted. Deleted records and updated values leave behind free space (fragmentation) in the database file, which can increase file size and potentially affect performance. Compaction eliminates this free space, creating a smaller, contiguous database file. - [Data Loader](/reference/v4/database/data-loader.md): The Data Loader is a built-in component that loads data from JSON or YAML files into Harper tables as part of component deployment. It is designed for seeding tables with initial records — configuration data, reference data, default users, or other records that should exist when a component is first deployed or updated. - [Jobs](/reference/v4/database/jobs.md): Harper uses an asynchronous job system for long-running data operations. When a bulk operation is initiated — such as loading a large CSV file or exporting millions of records — Harper starts a background job and immediately returns a job ID. Use the job ID to check progress and status. - [Database](/reference/v4/database/overview.md): Harper's database system is the foundation of its data storage and retrieval capabilities. It is built on top of LMDB (Lightning Memory-Mapped Database) and is designed to provide high performance, ACID-compliant storage with automatic indexing and flexible schema support. - [Schema](/reference/v4/database/schema.md): Harper uses GraphQL Schema Definition Language (SDL) to declaratively define table structure. Schema definitions are loaded from .graphql files in a component directory and control table creation, attribute types, indexing, and relationships. - [Storage Algorithm](/reference/v4/database/storage-algorithm.md): Harper's storage algorithm is the foundation of all database functionality. It is built on top of LMDB (Lightning Memory-Mapped Database), a high-performance key-value store, and extends it with automatic indexing, query-language-agnostic data access, and ACID compliance. - [System Tables](/reference/v4/database/system-tables.md): Harper maintains a set of internal system tables in the system database. These tables store analytics, job tracking, replication configuration, and other internal state. Most are read-only from the application perspective; some can be queried for observability or management purposes. - [Transaction Logging](/reference/v4/database/transaction.md): Harper provides two complementary mechanisms for recording a history of data changes on a table: the audit log and the transaction log. Both are available at the table level and serve different use cases. - [Environment Variables](/reference/v4/environment-variables/overview.md): Harper supports loading environment variables in Harper applications process.env using the built-in loadEnv plugin. This is the standard way to supply secrets and configuration to your Harper components without hardcoding values. loadEnv does not need to be installed as it is built into Harper and only needs to be declared in your config.yaml. - [Define Fastify Routes](/reference/v4/fastify-routes/overview.md): Fastify routes are discouraged in favor of modern routing with Custom Resources, but remain a supported feature for backwards compatibility and specific use cases. - [GraphQL Querying](/reference/v4/graphql-querying/overview.md): (provisional) - [HTTP API](/reference/v4/http/api.md): The server global object is available in all Harper component code. It provides access to the HTTP server middleware chain, WebSocket server, authentication, resource registry, and cluster information. - [HTTP Configuration](/reference/v4/http/configuration.md): The http section in harperdb-config.yaml controls the built-in HTTP server that serves REST, WebSocket, component, and Operations API traffic. - [HTTP Server](/reference/v4/http/overview.md): Harper includes a built-in HTTP server that serves as the primary interface for REST, WebSocket, MQTT-over-WebSocket, and component-defined endpoints. The same server handles all application traffic on a configurable port (default 9926). - [TLS Configuration](/reference/v4/http/tls.md): Harper uses a top-level tls section in harperdb-config.yaml to configure Transport Layer Security. This configuration is shared by the HTTP server (HTTPS), the MQTT broker (secure MQTT), and any TLS socket servers created via the HTTP API. - [Harper Cloud](/reference/v4/legacy/cloud.md): Harper Cloud (also sometimes referred to as Harper Studio) was Harper's original PaaS offering. - [Custom Functions](/reference/v4/legacy/custom-functions.md): Custom Functions were Harper's original mechanism for adding custom API endpoints and application logic to a Harper instance. They allowed developers to define Fastify-based HTTP routes that ran inside Harper with direct access to the database, and could be deployed across instances via Studio. - [Logging API](/reference/v4/logging/api.md): logger - [Logging Configuration](/reference/v4/logging/configuration.md): The logging section in harperdb-config.yaml controls standard log output. Many logging settings are applied dynamically without a restart (added in v4.6.0). - [Logging Operations](/reference/v4/logging/operations.md): Operations for reading the standard Harper log (hdb.log). All operations are restricted to super_user roles only. - [Logging](/reference/v4/logging/overview.md): Harper's core logging system is used for diagnostics, monitoring, and observability. It has an extensive configuration system, and even supports feature-specific (per-component) configurations in latest versions. Furthermore, the logger global API is available for creating custom logs from any JavaScript application or plugin code. - [MQTT Configuration](/reference/v4/mqtt/configuration.md): The mqtt section in harperdb-config.yaml controls Harper's built-in MQTT broker. MQTT is enabled by default. - [MQTT](/reference/v4/mqtt/overview.md): Harper includes a built-in MQTT broker that provides real-time pub/sub messaging deeply integrated with the database. Unlike a generic MQTT broker, Harper's MQTT implementation connects topics directly to database records — publishing to a topic writes to the database, and subscribing to a topic delivers live updates for the corresponding record. - [Operations Reference](/reference/v4/operations-api/operations.md): This page lists all available Operations API operations, grouped by category. Each entry links to the feature section where the full documentation lives. - [Operations API](/reference/v4/operations-api/overview.md): The Operations API provides a comprehensive set of capabilities for configuring, deploying, administering, and controlling Harper. It is the primary programmatic interface for all administrative and operational tasks that are not handled through the REST interface. - [SQL](/reference/v4/operations-api/sql.md): SQL querying is not recommended for production use or on large tables. SQL queries often do not utilize indexes and are not optimized for performance. Use the REST interface for production data access — it provides a more stable, secure, and performant interface. SQL is intended for ad-hoc data investigation and administrative queries. - [Clustering](/reference/v4/replication/clustering.md): Operations API for managing Harper's replication system. For an overview of how replication works, see Replication Overview. For sharding configuration, see Sharding. - [Replication Overview](/reference/v4/replication/overview.md): Harper's replication system is designed to make distributed data replication fast and reliable across multiple nodes. You can build a distributed database that ensures high availability, disaster recovery, and data localization — all without complex setup. Nodes can be added or removed dynamically, you can choose which data to replicate, and you can monitor cluster health without jumping through hoops. - [Sharding](/reference/v4/replication/sharding.md): (provisional) - [Resources](/reference/v4/resources/overview.md): Harper's Resource API is the foundation for building custom data access logic and connecting data sources. Resources are JavaScript classes that define how data is accessed, modified, subscribed to, and served over HTTP, MQTT, and WebSocket protocols. - [Query Optimization](/reference/v4/resources/query-optimization.md): (query planning and execution improvements) - [Resource API](/reference/v4/resources/resource-api.md): The Resource API provides a unified JavaScript interface for accessing, querying, modifying, and subscribing to data resources in Harper. Tables extend the base Resource class, and all resource interactions — whether from HTTP requests, MQTT messages, or application code — flow through this interface. - [Content Types](/reference/v4/rest/content-types.md): Harper supports multiple content types (MIME types) for both HTTP request bodies and response bodies. Harper follows HTTP standards: use the Content-Type request header to specify the encoding of the request body, and use the Accept header to request a specific response format. - [REST Headers](/reference/v4/rest/headers.md): Harper's REST interface uses standard HTTP headers for content negotiation, caching, and performance instrumentation. - [REST Overview](/reference/v4/rest/overview.md): Harper provides a powerful, efficient, and standard-compliant HTTP REST interface for interacting with tables and other resources. The REST interface is the recommended interface for data access, querying, and manipulation over HTTP, providing the best performance and HTTP interoperability with different clients. - [REST Querying](/reference/v4/rest/querying.md): Harper's REST interface supports a rich URL-based query language for filtering, sorting, selecting, and limiting records. Queries are expressed as URL query parameters on collection paths. - [Server-Sent Events](/reference/v4/rest/server-sent-events.md): Harper supports Server-Sent Events (SSE), a simple and efficient mechanism for browser-based applications to receive real-time updates from the server over a standard HTTP connection. SSE is a one-directional transport — the server pushes events to the client, and the client has no way to send messages back on the same connection. - [WebSockets](/reference/v4/rest/websockets.md): Harper supports WebSocket connections through the REST interface, enabling real-time bidirectional communication with resources. WebSocket connections target a resource URL path — by default, connecting to a resource subscribes to changes for that resource. - [Security API](/reference/v4/security/api.md): Harper exposes security-related globals accessible in all component JavaScript modules without needing to import them. - [Basic Authentication](/reference/v4/security/basic-authentication.md): Available since: v4.1.0 - [Certificate Management](/reference/v4/security/certificate-management.md): This page covers certificate management for Harper's external-facing HTTP and Operations APIs. For replication certificate management, see Replication Certificate Management. - [Certificate Verification](/reference/v4/security/certificate-verification.md): (OCSP support added) - [Authentication Configuration](/reference/v4/security/configuration.md): Harper's authentication system is configured via the top-level authentication section of harperdb-config.yaml. - [JWT Authentication](/reference/v4/security/jwt-authentication.md): s - [mTLS Authentication](/reference/v4/security/mtls-authentication.md): Harper supports Mutual TLS (mTLS) authentication for incoming HTTP connections. When enabled, the client must present a certificate signed by a trusted Certificate Authority (CA). If the certificate is valid and trusted, the connection is authenticated using the user whose username matches the CN (Common Name) from the client certificate's subject. - [Security](/reference/v4/security/overview.md): Harper uses role-based, attribute-level security to ensure that users can only gain access to the data they are supposed to be able to access. Granular permissions allow for unparalleled flexibility and control, and can lower the total cost of ownership compared to other database solutions, since you no longer need to replicate subsets of data to isolate use cases. - [Static Files](/reference/v4/static-files/overview.md): - Added in: v4.5.0 - [Local Studio](/reference/v4/studio/overview.md): - Added in: v4.1.0 - [Configuration](/reference/v4/users-and-roles/configuration.md): Managing Roles with Config Files - [Operations](/reference/v4/users-and-roles/operations.md): Roles - [Users & Roles](/reference/v4/users-and-roles/overview.md): Harper uses a Role-Based Access Control (RBAC) framework to manage access to Harper instances. Each user is assigned a role that determines their permissions to access database resources and run operations. - [Harper v5 Reference](/reference/v5.md): Complete technical reference for Harper v5. Each section covers a core feature or subsystem — configuration options, APIs, and operational details. - [Analytics Operations](/reference/v5/analytics/operations.md): Operations for querying Harper analytics data. All operations require superuser permission. - [Analytics](/reference/v5/analytics/overview.md): (resource and storage analytics) - [CLI Authentication](/reference/v5/cli/authentication.md): The Harper CLI handles authentication differently for local and remote operations. - [CLI Commands](/reference/v5/cli/commands.md): This page documents the core Harper CLI commands for managing Harper instances. For Operations API commands available through the CLI, see Operations API Commands. - [Operations API Commands](/reference/v5/cli/operations-api-commands.md): The Harper CLI supports executing operations from the Operations API directly from the command line. This enables powerful automation and scripting capabilities. - [Harper CLI Overview](/reference/v5/cli/overview.md): The Harper command line interface (CLI) is used to administer self-installed Harper instances. - [Applications](/reference/v5/components/applications.md): The contents of this page primarily relate to application components. The term "components" in the Operations API and CLI generally refers to applications specifically. See the Components Overview for a full explanation of terminology. - [Extension API](/reference/v5/components/extension-api.md): As of Harper v4.6 and now stable in v5.0, a new iteration of the extension system called plugins was released. Plugins simplify the API and are recommended for new extension development. See the Plugin API reference. Extensions are still supported as of v5.0 but are considered deprecated in favor of plugins. - [JavaScript Environment](/reference/v5/components/javascript-environment.md): Harper executes component JavaScript in distinct module caches, using Node.js's VM module loader. This provides contextualized module environments that share the same Node.js runtime but have their own set of modules isolated from other applications. This means each application runs in its own module context while still being able to access Harper's full set of APIs. - [Next.js Plugin (@harperfast/nextjs)](/reference/v5/components/nextjs.md): @harperfast/nextjs is a Harper Plugin for running Next.js applications directly on Harper. It supports Next.js v14, v15, and v16. - [Components](/reference/v5/components/overview.md): Components are the high-level concept for modules that extend the Harper core platform with additional functionality. Components encapsulate both applications and extensions. - [Plugin API](/reference/v5/components/plugin-api.md): Plugins are the building blocks of the Harper component system. Applications depend on plugins to provide the functionality they implement. For example, the built-in graphqlSchema plugin enables applications to define databases and tables using GraphQL schemas. The @harperfast/nextjs plugin provides the functionality to build a Next.js application on Harper. Plugins export a single handleApplication method and are always executed on worker threads. - [Worker Thread Debugging](/reference/v5/configuration/debugging.md): Harper runs as a main thread plus a pool of worker threads (configurable via threads.count). The threads.debug option exposes the Node.js inspector on each thread so you can attach Chrome DevTools, VS Code, or any Chrome DevTools Protocol (CDP) client to step through component code, inspect heap state, or capture CPU profiles. - [Configuration Operations](/reference/v5/configuration/operations.md): Operations API endpoints for reading and modifying Harper configuration. - [Configuration Options](/reference/v5/configuration/options.md): Quick reference for all harper-config.yaml top-level sections. - [Configuration](/reference/v5/configuration/overview.md): Harper is configured through a YAML file called harper-config.yaml located in the Harper root directory. By default the root directory is a folder named hdb in the home directory of the current user. - [API](/reference/v5/database/api.md): Harper exposes a set of global variables and functions that JavaScript code (in components, applications, and plugins) can use to interact with the database system. - [Compaction](/reference/v5/database/compaction.md): Database files grow over time as records are inserted, updated, and deleted. Deleted records and updated values leave behind free space (fragmentation) in the database file, which can increase file size and potentially affect performance. Compaction eliminates this free space, creating a smaller, contiguous database file. - [Data Loader](/reference/v5/database/data-loader.md): The Data Loader is a built-in component that loads data from JSON or YAML files into Harper tables as part of component deployment. It is designed for seeding tables with initial records — configuration data, reference data, default users, or other records that should exist when a component is first deployed or updated. - [Jobs](/reference/v5/database/jobs.md): Harper uses an asynchronous job system for long-running data operations. When a bulk operation is initiated — such as loading a large CSV file or exporting millions of records — Harper starts a background job and immediately returns a job ID. Use the job ID to check progress and status. - [Database](/reference/v5/database/overview.md): Harper's database system is the foundation of its data storage and retrieval capabilities. Harper supports two storage enginers, RocksDB and LMDB (Lightning Memory-Mapped Database) and is designed to provide high performance, ACID-compliant storage with indexing and flexible schema support. - [Schema](/reference/v5/database/schema.md): Harper uses GraphQL Schema Definition Language (SDL) to declaratively define table structure. Schema definitions are loaded from .graphql files in a component directory and control table creation, attribute types, indexing, and relationships. - [Storage Algorithm](/reference/v5/database/storage-algorithm.md): Harper's storage algorithm is the foundation of all database functionality. It is built on top of RocksDB (the default) or LMDB (legacy), both high-performance key-value stores, and extends them with automatic indexing, query-language-agnostic data access, and ACID compliance. - [Storage Tuning](/reference/v5/database/storage-tuning.md): Harper's storage configuration section controls how database files are written, cached, and reclaimed on disk. Defaults are tuned for safety and balanced throughput; this page covers the knobs that matter for production deployments with specific workload profiles. - [System Tables](/reference/v5/database/system-tables.md): Harper maintains a set of internal system tables in the system database. These tables store analytics, job tracking, replication configuration, and other internal state. Most are read-only from the application perspective; some can be queried for observability or management purposes. - [Transaction Logging](/reference/v5/database/transaction.md): Harper provides two complementary mechanisms for recording a history of data changes on a table: the audit log and the transaction log. Both are available at the table level and serve different use cases. - [Environment Variables](/reference/v5/environment-variables/overview.md): Harper supports loading environment variables in Harper applications process.env using the built-in loadEnv plugin. This is the standard way to supply secrets and configuration to your Harper components without hardcoding values. loadEnv does not need to be installed as it is built into Harper and only needs to be declared in your config.yaml. - [Define Fastify Routes](/reference/v5/fastify-routes/overview.md): Fastify routes are discouraged in favor of modern routing with Custom Resources, but remain a supported feature for backwards compatibility and specific use cases. - [GraphQL Querying](/reference/v5/graphql-querying/overview.md): (provisional) - [HTTP API](/reference/v5/http/api.md): The server global object is available in all Harper component code. It provides access to the HTTP server middleware chain, WebSocket server, authentication, resource registry, and cluster information. - [HTTP Configuration](/reference/v5/http/configuration.md): The http section in harper-config.yaml controls the built-in HTTP server that serves REST, WebSocket, component, and Operations API traffic. - [HTTP Server](/reference/v5/http/overview.md): Harper includes a built-in HTTP server that serves as the primary interface for REST, WebSocket, MQTT-over-WebSocket, and component-defined endpoints. The same server handles all application traffic on a configurable port (default 9926). - [TLS Configuration](/reference/v5/http/tls.md): Harper uses a top-level tls section in harper-config.yaml to configure Transport Layer Security. This configuration is shared by the HTTP server (HTTPS), the MQTT broker (secure MQTT), and any TLS socket servers created via the HTTP API. - [Harper Cloud](/reference/v5/legacy/cloud.md): Harper Cloud (also sometimes referred to as Harper Studio) was Harper's original PaaS offering. - [Custom Functions](/reference/v5/legacy/custom-functions.md): Custom Functions were Harper's original mechanism for adding custom API endpoints and application logic to a Harper instance. They allowed developers to define Fastify-based HTTP routes that ran inside Harper with direct access to the database, and could be deployed across instances via Studio. - [Logging API](/reference/v5/logging/api.md): logger - [Logging Configuration](/reference/v5/logging/configuration.md): The logging section in harper-config.yaml controls standard log output. Many logging settings are applied dynamically without a restart (added in v4.6.0). - [Logging Operations](/reference/v5/logging/operations.md): Operations for reading the standard Harper log (hdb.log). All operations are restricted to super_user roles only. - [Logging](/reference/v5/logging/overview.md): Harper's core logging system is used for diagnostics, monitoring, and observability. It has an extensive configuration system, and even supports feature-specific (per-component) configurations in latest versions. Furthermore, the logger global API is available for creating custom logs from any JavaScript application or plugin code. - [Harper MCP CLI](/reference/v5/mcp/cli.md): harper mcp is a stdio bridge that lets MCP hosts (Claude Desktop, Cursor, Zed, or any client that speaks the stdio MCP transport) talk to a running Harper instance over Harper's Streamable HTTP MCP endpoint. - [MCP Configuration](/reference/v5/mcp/configuration.md): All MCP configuration lives under the top-level mcp - [Migrating from the External MCP Server](/reference/v5/mcp/migration.md): Earlier deployments of Harper used the standalone HarperFast/mcp-server addon — a separate Node.js process that wrapped Harper's Operations API and exposed MCP over stdio. With v5.1, MCP is now a first-class built-in server-side surface; the external addon is deprecated and will be archived alongside this release. - [MCP Overview](/reference/v5/mcp/overview.md): Harper implements the Model Context Protocol (MCP) as a first-class server-side surface, letting large-language-model hosts (Claude Desktop, Cursor, Zed, custom agents) discover and invoke Harper operations, resources, and tables through a standard wire protocol. The MCP server runs in-process inside Harper — there is no separate addon, no sidecar, and no out-of-process broker. - [MCP Resource Subscriptions](/reference/v5/mcp/subscriptions.md): Beyond the notifications/*/listchanged events that tell a client what tools and resources exist, Harper can push a notification whenever the contents_ of a specific application resource change. A client subscribes to a resource URI with resources/subscribe; Harper watches that record (or table) through its audit-log change feed and emits notifications/resources/updated on every commit. - [MCP tool payload sourcing](/reference/v5/mcp/tool-metadata.md): Harper's MCP server publishes tools via the Model Context Protocol (rev 2025-06-18, Streamable HTTP transport). This page is a reference for what fields appear on each generated tool descriptor and where the data comes from. For authoring guidance, see the Writing quality MCP and OpenAPI descriptions how-to. - [MCP Tools and Resources](/reference/v5/mcp/tools-and-resources.md): This page documents what the MCP server actually exposes — which tools land on tools/list for which user, how their input schemas are built, and what shows up in resources/list for each profile. Configuration knobs that gate this surface are documented in MCP Configuration. - [Analytics](/reference/v5/models/analytics.md): Every model call is recorded for observability and usage accounting, at two levels of granularity: a per-call log table for forensics, and aggregate counters in Harper's general analytics for dashboards and trends. - [API](/reference/v5/models/api.md): The models object exposes three methods. All of them accept an optional model option naming the configured logical model to use; when omitted, the logical name default is used. Calling a logical name with no configured backend, or asking a backend for a capability it does not support (for example, embeddings from a generation-only backend), throws an error — capability checks run up front, before any request is made. - [Backends](/reference/v5/models/backends.md): Four model backends ship with Harper. Each model entry in the models configuration selects one with its backend field. - [Models](/reference/v5/models/overview.md): Harper provides a unified API for calling AI models — text embeddings and text generation — from application code. Models are configured by an operator under logical names; application code requests a model by its logical name and Harper routes the call to the configured backend (Ollama, OpenAI, Anthropic, or Amazon Bedrock) — or to a custom backend a component registers. Swapping providers is a configuration change, not a code change. A logical name can also name an ordered group of backends to try, and calls can require specific capabilities — see Routing & Fallback. - [Routing & Fallback](/reference/v5/models/routing.md): Every models call resolves through a router that returns an ordered list of candidate backends for the requested logical name. The facade uses the first candidate whose capabilities satisfy the call and falls through to the next on failure — capability-aware selection and fallback are the same mechanism. - [Tool Calling](/reference/v5/models/tool-calling.md): Generative models can be given tools — functions the model may request calls to while producing a response. Tools are declared on the input object (they are model-facing content, like messages), and the toolMode option selects who resolves them: - [MQTT Configuration](/reference/v5/mqtt/configuration.md): The mqtt section in harper-config.yaml controls Harper's built-in MQTT broker. MQTT is enabled by default. - [MQTT](/reference/v5/mqtt/overview.md): Harper includes a built-in MQTT broker that provides real-time pub/sub messaging deeply integrated with the database. Unlike a generic MQTT broker, Harper's MQTT implementation connects topics directly to database records — publishing to a topic writes to the database, and subscribing to a topic delivers live updates for the corresponding record. - [Operations Reference](/reference/v5/operations-api/operations.md): This page lists all available Operations API operations, grouped by category. Each entry links to the feature section where the full documentation lives. - [Operations API](/reference/v5/operations-api/overview.md): The Operations API provides a comprehensive set of capabilities for configuring, deploying, administering, and controlling Harper. It is the primary programmatic interface for all administrative and operational tasks that are not handled through the REST interface. - [SQL](/reference/v5/operations-api/sql.md): SQL querying is not recommended for production use or on large tables. SQL queries often do not utilize indexes and are not optimized for performance. Use the REST interface for production data access — it provides a more stable, secure, and performant interface. SQL is intended for ad-hoc data investigation and administrative queries. - [Clustering](/reference/v5/replication/clustering.md): Operations API for managing Harper's replication system. For an overview of how replication works, see Replication Overview. For sharding configuration, see Sharding. - [Replication Overview](/reference/v5/replication/overview.md): Harper's replication system is designed to make distributed data replication fast and reliable across multiple nodes. You can build a distributed database that ensures high availability, disaster recovery, and data localization — all without complex setup. Nodes can be added or removed dynamically, you can choose which data to replicate, and you can monitor cluster health without jumping through hoops. - [Sharding](/reference/v5/replication/sharding.md): (provisional) - [Resources](/reference/v5/resources/overview.md): Harper's Resource API is the foundation for building custom data access logic and connecting data sources. Resources are JavaScript classes that define how data is accessed, modified, subscribed to, and served over HTTP, MQTT, and WebSocket protocols. - [Query Optimization](/reference/v5/resources/query-optimization.md): (query planning and execution improvements) - [Resource API](/reference/v5/resources/resource-api.md): The Resource API provides a unified JavaScript interface for accessing, querying, modifying, and subscribing to data resources in Harper. Tables extend the base Resource class, and all resource interactions — whether from HTTP requests, MQTT messages, or application code — flow through this interface. - [Content Types](/reference/v5/rest/content-types.md): Harper supports multiple content types (MIME types) for both HTTP request bodies and response bodies. Harper follows HTTP standards: use the Content-Type request header to specify the encoding of the request body, and use the Accept header to request a specific response format. - [REST Headers](/reference/v5/rest/headers.md): Harper's REST interface uses standard HTTP headers for content negotiation, caching, and performance instrumentation. - [REST Overview](/reference/v5/rest/overview.md): Harper provides a powerful, efficient, and standard-compliant HTTP REST interface for interacting with tables and other resources. The REST interface is the recommended interface for data access, querying, and manipulation over HTTP, providing the best performance and HTTP interoperability with different clients. - [REST Querying](/reference/v5/rest/querying.md): Harper's REST interface supports a rich URL-based query language for filtering, sorting, selecting, and limiting records. Queries are expressed as URL query parameters on collection paths. - [Server-Sent Events](/reference/v5/rest/server-sent-events.md): Harper supports Server-Sent Events (SSE), a simple and efficient mechanism for browser-based applications to receive real-time updates from the server over a standard HTTP connection. SSE is a one-directional transport — the server pushes events to the client, and the client has no way to send messages back on the same connection. - [WebSockets](/reference/v5/rest/websockets.md): Harper supports WebSocket connections through the REST interface, enabling real-time bidirectional communication with resources. WebSocket connections target a resource URL path — by default, connecting to a resource subscribes to changes for that resource. - [Security API](/reference/v5/security/api.md): Harper exposes security-related globals accessible in all component JavaScript modules without needing to import them. - [Basic Authentication](/reference/v5/security/basic-authentication.md): Available since: v4.1.0 - [Certificate Management](/reference/v5/security/certificate-management.md): This page covers certificate management for Harper's external-facing HTTP and Operations APIs. For replication certificate management, see Replication Certificate Management. - [Certificate Verification](/reference/v5/security/certificate-verification.md): (OCSP support added) - [Authentication Configuration](/reference/v5/security/configuration.md): Harper's authentication system is configured via the top-level authentication section of harper-config.yaml. - [Impersonation](/reference/v5/security/impersonation.md): Impersonation allows a super_user to execute operations API requests as if they were a different, less-privileged user. This is useful for testing permissions, debugging access issues, and building admin tools that preview what a given user or role can see and do — all without needing that user's credentials. - [JWT Authentication](/reference/v5/security/jwt-authentication.md): s - [mTLS Authentication](/reference/v5/security/mtls-authentication.md): Harper supports Mutual TLS (mTLS) authentication for incoming HTTP connections. When enabled, the client must present a certificate signed by a trusted Certificate Authority (CA). If the certificate is valid and trusted, the connection is authenticated using the user whose username matches the CN (Common Name) from the client certificate's subject. - [Security](/reference/v5/security/overview.md): Harper uses role-based, attribute-level security to ensure that users can only gain access to the data they are supposed to be able to access. Granular permissions allow for unparalleled flexibility and control, and can lower the total cost of ownership compared to other database solutions, since you no longer need to replicate subsets of data to isolate use cases. - [Static Files](/reference/v5/static-files/overview.md): - Added in: v4.5.0 - [Local Studio](/reference/v5/studio/overview.md): - Added in: v4.1.0 - [Configuration](/reference/v5/users-and-roles/configuration.md): Managing Roles with Config Files - [Operations](/reference/v5/users-and-roles/operations.md): Roles - [Users & Roles](/reference/v5/users-and-roles/overview.md): Harper uses a Role-Based Access Control (RBAC) framework to manage access to Harper instances. Each user is assigned a role that determines their permissions to access database resources and run operations. ## release-notes - [Release Notes](/release-notes.md): Complete version history and changelog for Harper - [HarperDB Alby (Version 1)](/release-notes/v1-alby.md): Did you know our release names are dedicated to employee pups? For our first release, Alby was our pup. - [1.1.0](/release-notes/v1-alby/1.1.0.md): HarperDB 1.1.0, Alby Release - [1.2.0](/release-notes/v1-alby/1.2.0.md): HarperDB 1.2.0, Alby Release - [1.3.0](/release-notes/v1-alby/1.3.0.md): HarperDB 1.3.0, Alby Release - [1.3.1](/release-notes/v1-alby/1.3.1.md): HarperDB 1.3.1, Alby Release - [HarperDB Penny (Version 2)](/release-notes/v2-penny.md): Did you know our release names are dedicated to employee pups? For our second release, Penny was the star. - [2.1.1](/release-notes/v2-penny/2.1.1.md): HarperDB 2.1.1, Penny Release - [2.2.0](/release-notes/v2-penny/2.2.0.md): HarperDB 2.2.0, Penny Release - [2.2.2](/release-notes/v2-penny/2.2.2.md): HarperDB 2.2.2, Penny Release - [2.2.3](/release-notes/v2-penny/2.2.3.md): HarperDB 2.2.3, Penny Release - [2.3.0](/release-notes/v2-penny/2.3.0.md): HarperDB 2.3.0, Penny Release - [2.3.1](/release-notes/v2-penny/2.3.1.md): HarperDB 2.3.1, Penny Release - [HarperDB Monkey (Version 3)](/release-notes/v3-monkey.md): Did you know our release names are dedicated to employee pups? For our third release, we have Monkey. - [3.0.0](/release-notes/v3-monkey/3.0.0.md): HarperDB 3.0, Monkey Release - [3.1.0](/release-notes/v3-monkey/3.1.0.md): HarperDB 3.1.0, Monkey Release - [3.1.1](/release-notes/v3-monkey/3.1.1.md): HarperDB 3.1.1, Monkey Release - [3.1.2](/release-notes/v3-monkey/3.1.2.md): HarperDB 3.1.2, Monkey Release - [3.1.3](/release-notes/v3-monkey/3.1.3.md): HarperDB 3.1.3, Monkey Release - [3.1.4](/release-notes/v3-monkey/3.1.4.md): HarperDB 3.1.4, Monkey Release - [3.1.5](/release-notes/v3-monkey/3.1.5.md): HarperDB 3.1.5, Monkey Release - [3.2.0](/release-notes/v3-monkey/3.2.0.md): HarperDB 3.2.0, Monkey Release - [3.2.1](/release-notes/v3-monkey/3.2.1.md): HarperDB 3.2.1, Monkey Release - [3.3.0](/release-notes/v3-monkey/3.3.0.md): HarperDB 3.3.0 - Monkey - [Harper Tucker (Version 4)](/release-notes/v4-tucker.md): HarperDB version 4 (Tucker release) represents major step forward in database technology. This release line has ground-breaking architectural advancements including: - [4.0.0](/release-notes/v4-tucker/4.0.0.md): HarperDB 4.0.0, Tucker Release - [4.0.1](/release-notes/v4-tucker/4.0.1.md): HarperDB 4.0.1, Tucker Release - [4.0.2](/release-notes/v4-tucker/4.0.2.md): HarperDB 4.0.2, Tucker Release - [4.0.3](/release-notes/v4-tucker/4.0.3.md): HarperDB 4.0.3, Tucker Release - [4.0.4](/release-notes/v4-tucker/4.0.4.md): HarperDB 4.0.4, Tucker Release - [4.0.5](/release-notes/v4-tucker/4.0.5.md): HarperDB 4.0.5, Tucker Release - [4.0.6](/release-notes/v4-tucker/4.0.6.md): HarperDB 4.0.6, Tucker Release - [4.0.7](/release-notes/v4-tucker/4.0.7.md): HarperDB 4.0.7, Tucker Release - [4.1.0](/release-notes/v4-tucker/4.1.0.md): HarperDB 4.1 introduces the ability to use worker threads for concurrently handling HTTP requests. Previously this was handled by processes. This shift provides important benefits in terms of better control of traffic delegation with support for optimized load tracking and session affinity, better debuggability, and reduced memory footprint. - [4.1.1](/release-notes/v4-tucker/4.1.1.md): 06/16/2023 - [4.1.2](/release-notes/v4-tucker/4.1.2.md): HarperDB 4.1.2, Tucker Release - [4.2.0](/release-notes/v4-tucker/4.2.0.md): HarperDB 4.2.0 - [4.2.1](/release-notes/v4-tucker/4.2.1.md): HarperDB 4.2.1, Tucker Release - [4.2.2](/release-notes/v4-tucker/4.2.2.md): HarperDB 4.2.2, Tucker Release - [4.2.3](/release-notes/v4-tucker/4.2.3.md): HarperDB 4.2.3, Tucker Release - [4.2.4](/release-notes/v4-tucker/4.2.4.md): HarperDB 4.2.4, Tucker Release - [4.2.5](/release-notes/v4-tucker/4.2.5.md): HarperDB 4.2.5, Tucker Release - [4.2.6](/release-notes/v4-tucker/4.2.6.md): HarperDB 4.2.6, Tucker Release - [4.2.7](/release-notes/v4-tucker/4.2.7.md): HarperDB 4.2.7 - [4.2.8](/release-notes/v4-tucker/4.2.8.md): HarperDB 4.2.8 - [4.3.0](/release-notes/v4-tucker/4.3.0.md): HarperDB 4.3.0, Tucker Release - [4.3.1](/release-notes/v4-tucker/4.3.1.md): HarperDB 4.3.1 - [4.3.10](/release-notes/v4-tucker/4.3.10.md): HarperDB 4.3.10 - [4.3.11](/release-notes/v4-tucker/4.3.11.md): HarperDB 4.3.11 - [4.3.12](/release-notes/v4-tucker/4.3.12.md): HarperDB 4.3.12 - [4.3.13](/release-notes/v4-tucker/4.3.13.md): HarperDB 4.3.13 - [4.3.14](/release-notes/v4-tucker/4.3.14.md): HarperDB 4.3.14 - [4.3.15](/release-notes/v4-tucker/4.3.15.md): HarperDB 4.3.15 - [4.3.16](/release-notes/v4-tucker/4.3.16.md): HarperDB 4.3.16 - [4.3.17](/release-notes/v4-tucker/4.3.17.md): HarperDB 4.3.17 - [4.3.18](/release-notes/v4-tucker/4.3.18.md): HarperDB 4.3.18 - [4.3.19](/release-notes/v4-tucker/4.3.19.md): HarperDB 4.3.19 - [4.3.2](/release-notes/v4-tucker/4.3.2.md): HarperDB 4.3.2 - [4.3.20](/release-notes/v4-tucker/4.3.20.md): HarperDB 4.3.20 - [4.3.21](/release-notes/v4-tucker/4.3.21.md): HarperDB 4.3.21 - [4.3.22](/release-notes/v4-tucker/4.3.22.md): HarperDB 4.3.22 - [4.3.23](/release-notes/v4-tucker/4.3.23.md): HarperDB 4.3.23 - [4.3.24](/release-notes/v4-tucker/4.3.24.md): HarperDB 4.3.24 - [4.3.25](/release-notes/v4-tucker/4.3.25.md): HarperDB 4.3.25 - [4.3.26](/release-notes/v4-tucker/4.3.26.md): HarperDB 4.3.26 - [4.3.27](/release-notes/v4-tucker/4.3.27.md): HarperDB 4.3.27 - [4.3.28](/release-notes/v4-tucker/4.3.28.md): HarperDB 4.3.28 - [4.3.29](/release-notes/v4-tucker/4.3.29.md): HarperDB 4.3.29 - [4.3.3](/release-notes/v4-tucker/4.3.3.md): HarperDB 4.3.3 - [4.3.30](/release-notes/v4-tucker/4.3.30.md): HarperDB 4.3.30 - [4.3.31](/release-notes/v4-tucker/4.3.31.md): HarperDB 4.3.31 - [4.3.32](/release-notes/v4-tucker/4.3.32.md): HarperDB 4.3.32 - [4.3.33](/release-notes/v4-tucker/4.3.33.md): HarperDB 4.3.33 - [4.3.34](/release-notes/v4-tucker/4.3.34.md): HarperDB 4.3.34 - [4.3.35](/release-notes/v4-tucker/4.3.35.md): HarperDB 4.3.35 - [4.3.36](/release-notes/v4-tucker/4.3.36.md): HarperDB 4.3.36 - [4.3.37](/release-notes/v4-tucker/4.3.37.md): HarperDB 4.3.37 - [4.3.38](/release-notes/v4-tucker/4.3.38.md): HarperDB 4.3.38 - [4.3.4](/release-notes/v4-tucker/4.3.4.md): HarperDB 4.3.4 - [4.3.5](/release-notes/v4-tucker/4.3.5.md): HarperDB 4.3.5 - [4.3.6](/release-notes/v4-tucker/4.3.6.md): HarperDB 4.3.6 - [4.3.7](/release-notes/v4-tucker/4.3.7.md): HarperDB 4.3.7 - [4.3.8](/release-notes/v4-tucker/4.3.8.md): HarperDB 4.3.8 - [4.3.9](/release-notes/v4-tucker/4.3.9.md): HarperDB 4.3.9 - [4.4.0](/release-notes/v4-tucker/4.4.0.md): HarperDB 4.4.0 - [4.4.1](/release-notes/v4-tucker/4.4.1.md): HarperDB 4.4.1 - [4.4.10](/release-notes/v4-tucker/4.4.10.md): HarperDB 4.4.10 - [4.4.11](/release-notes/v4-tucker/4.4.11.md): HarperDB 4.4.11 - [4.4.12](/release-notes/v4-tucker/4.4.12.md): HarperDB 4.4.12 - [4.4.13](/release-notes/v4-tucker/4.4.13.md): HarperDB 4.4.13 - [4.4.14](/release-notes/v4-tucker/4.4.14.md): HarperDB 4.4.14 - [4.4.15](/release-notes/v4-tucker/4.4.15.md): HarperDB 4.4.15 - [4.4.16](/release-notes/v4-tucker/4.4.16.md): HarperDB 4.4.16 - [4.4.17](/release-notes/v4-tucker/4.4.17.md): HarperDB 4.4.17 - [4.4.18](/release-notes/v4-tucker/4.4.18.md): HarperDB 4.4.18 - [4.4.19](/release-notes/v4-tucker/4.4.19.md): HarperDB 4.4.19 - [4.4.2](/release-notes/v4-tucker/4.4.2.md): HarperDB 4.4.2 - [4.4.20](/release-notes/v4-tucker/4.4.20.md): HarperDB 4.4.20 - [4.4.21](/release-notes/v4-tucker/4.4.21.md): HarperDB 4.4.21 - [4.4.22](/release-notes/v4-tucker/4.4.22.md): HarperDB 4.4.22 - [4.4.23](/release-notes/v4-tucker/4.4.23.md): HarperDB 4.4.23 - [4.4.24](/release-notes/v4-tucker/4.4.24.md): HarperDB 4.4.24 - [4.4.3](/release-notes/v4-tucker/4.4.3.md): HarperDB 4.4.3 - [4.4.4](/release-notes/v4-tucker/4.4.4.md): HarperDB 4.4.4 - [4.4.5](/release-notes/v4-tucker/4.4.5.md): HarperDB 4.4.5 - [4.4.6](/release-notes/v4-tucker/4.4.6.md): HarperDB 4.4.6 - [4.4.7](/release-notes/v4-tucker/4.4.7.md): HarperDB 4.4.7 - [4.4.8](/release-notes/v4-tucker/4.4.8.md): HarperDB 4.4.8 - [4.4.9](/release-notes/v4-tucker/4.4.9.md): HarperDB 4.4.9 - [4.5.0](/release-notes/v4-tucker/4.5.0.md): HarperDB 4.5.0 - [4.5.1](/release-notes/v4-tucker/4.5.1.md): HarperDB 4.5.1 - [4.5.10](/release-notes/v4-tucker/4.5.10.md): HarperDB 4.5.10 - [4.5.11](/release-notes/v4-tucker/4.5.11.md): HarperDB 4.5.11 - [4.5.12](/release-notes/v4-tucker/4.5.12.md): HarperDB 4.5.12 - [4.5.13](/release-notes/v4-tucker/4.5.13.md): HarperDB 4.5.13 - [4.5.14](/release-notes/v4-tucker/4.5.14.md): HarperDB 4.5.14 - [4.5.15](/release-notes/v4-tucker/4.5.15.md): HarperDB 4.5.15 - [4.5.16](/release-notes/v4-tucker/4.5.16.md): HarperDB 4.5.16 - [4.5.17](/release-notes/v4-tucker/4.5.17.md): HarperDB 4.5.17 - [4.5.18](/release-notes/v4-tucker/4.5.18.md): HarperDB 4.5.18 - [4.5.19](/release-notes/v4-tucker/4.5.19.md): HarperDB 4.5.19 - [4.5.2](/release-notes/v4-tucker/4.5.2.md): HarperDB 4.5.2 - [4.5.20](/release-notes/v4-tucker/4.5.20.md): HarperDB 4.5.20 - [4.5.21](/release-notes/v4-tucker/4.5.21.md): HarperDB 4.5.21 - [4.5.22](/release-notes/v4-tucker/4.5.22.md): HarperDB 4.5.22 - [4.5.23](/release-notes/v4-tucker/4.5.23.md): HarperDB 4.5.23 - [4.5.24](/release-notes/v4-tucker/4.5.24.md): HarperDB 4.5.24 - [4.5.25](/release-notes/v4-tucker/4.5.25.md): HarperDB 4.5.25 - [4.5.26](/release-notes/v4-tucker/4.5.26.md): HarperDB 4.5.26 - [4.5.27](/release-notes/v4-tucker/4.5.27.md): HarperDB 4.5.27 - [4.5.28](/release-notes/v4-tucker/4.5.28.md): HarperDB 4.5.28 - [4.5.29](/release-notes/v4-tucker/4.5.29.md): HarperDB 4.5.29 - [4.5.3](/release-notes/v4-tucker/4.5.3.md): HarperDB 4.5.3 - [4.5.30](/release-notes/v4-tucker/4.5.30.md): HarperDB 4.5.30 - [4.5.31](/release-notes/v4-tucker/4.5.31.md): HarperDB 4.5.31 - [4.5.32](/release-notes/v4-tucker/4.5.32.md): HarperDB 4.5.32 - [4.5.33](/release-notes/v4-tucker/4.5.33.md): HarperDB 4.5.33 - [4.5.34](/release-notes/v4-tucker/4.5.34.md): HarperDB 4.5.34 - [4.5.35](/release-notes/v4-tucker/4.5.35.md): HarperDB 4.5.35 - [4.5.36](/release-notes/v4-tucker/4.5.36.md): HarperDB 4.5.36 - [4.5.37](/release-notes/v4-tucker/4.5.37.md): HarperDB 4.5.37 - [4.5.38](/release-notes/v4-tucker/4.5.38.md): HarperDB 4.5.38 - [4.5.39](/release-notes/v4-tucker/4.5.39.md): HarperDB 4.5.39 - [4.5.4](/release-notes/v4-tucker/4.5.4.md): HarperDB 4.5.4 - [4.5.40](/release-notes/v4-tucker/4.5.40.md): HarperDB 4.5.40 - [4.5.41](/release-notes/v4-tucker/4.5.41.md): HarperDB 4.5.41 - [4.5.42](/release-notes/v4-tucker/4.5.42.md): HarperDB 4.5.42 - [4.5.5](/release-notes/v4-tucker/4.5.5.md): HarperDB 4.5.5 - [4.5.6](/release-notes/v4-tucker/4.5.6.md): HarperDB 4.5.6 - [4.5.7](/release-notes/v4-tucker/4.5.7.md): HarperDB 4.5.7 - [4.5.8](/release-notes/v4-tucker/4.5.8.md): HarperDB 4.5.8 - [4.5.9](/release-notes/v4-tucker/4.5.9.md): HarperDB 4.5.9 - [4.6.0](/release-notes/v4-tucker/4.6.0.md): HarperDB 4.6.0 - [4.6.1](/release-notes/v4-tucker/4.6.1.md): 7/10/2025 - [4.6.10](/release-notes/v4-tucker/4.6.10.md): 9/22/2025 - [4.6.11](/release-notes/v4-tucker/4.6.11.md): 9/26/2025 - [4.6.12](/release-notes/v4-tucker/4.6.12.md): 10/1/2025 - [4.6.13](/release-notes/v4-tucker/4.6.13.md): 10/2/2025 - [4.6.14](/release-notes/v4-tucker/4.6.14.md): 10/21/2025 - [4.6.15](/release-notes/v4-tucker/4.6.15.md): 11/5/2025 - [4.6.16](/release-notes/v4-tucker/4.6.16.md): 11/8/2025 - [4.6.17](/release-notes/v4-tucker/4.6.17.md): 11/11/2025 - [4.6.18](/release-notes/v4-tucker/4.6.18.md): HarperDB 4.6.18 - [4.6.19](/release-notes/v4-tucker/4.6.19.md): HarperDB 4.6.19 - [4.6.2](/release-notes/v4-tucker/4.6.2.md): 7/15/2025 - [4.6.20](/release-notes/v4-tucker/4.6.20.md): HarperDB 4.6.20 - [4.6.21](/release-notes/v4-tucker/4.6.21.md): HarperDB 4.6.21 - [4.6.22](/release-notes/v4-tucker/4.6.22.md): HarperDB 4.6.22 - [4.6.23](/release-notes/v4-tucker/4.6.23.md): HarperDB 4.6.23 - [4.6.24](/release-notes/v4-tucker/4.6.24.md): HarperDB 4.6.24 - [4.6.25](/release-notes/v4-tucker/4.6.25.md): HarperDB 4.6.25 - [4.6.26](/release-notes/v4-tucker/4.6.26.md): HarperDB 4.6.26 - [4.6.3](/release-notes/v4-tucker/4.6.3.md): 7/30/2025 - [4.6.4](/release-notes/v4-tucker/4.6.4.md): 8/1/2025 - [4.6.5](/release-notes/v4-tucker/4.6.5.md): 8/7/2025 - [4.6.6](/release-notes/v4-tucker/4.6.6.md): 8/15/2025 - [4.6.7](/release-notes/v4-tucker/4.6.7.md): 8/18/2025 - [4.6.8](/release-notes/v4-tucker/4.6.8.md): 9/3/2025 - [4.6.9](/release-notes/v4-tucker/4.6.9.md): 9/9/2025 - [4.7.0](/release-notes/v4-tucker/4.7.0.md): 10/16/2025 - [4.7.1](/release-notes/v4-tucker/4.7.1.md): 10/16/2025 - [4.7.10](/release-notes/v4-tucker/4.7.10.md): 11/26/2025 - [4.7.11](/release-notes/v4-tucker/4.7.11.md): 12/02/2025 - [4.7.12](/release-notes/v4-tucker/4.7.12.md): 12/08/2025 - [4.7.13](/release-notes/v4-tucker/4.7.13.md): 1/2/2026 - [4.7.14](/release-notes/v4-tucker/4.7.14.md): 1/6/2026 - [4.7.15](/release-notes/v4-tucker/4.7.15.md): 1/9/2026 - [4.7.16](/release-notes/v4-tucker/4.7.16.md): 1/23/2026 - [4.7.17](/release-notes/v4-tucker/4.7.17.md): 1/27/2026 - [4.7.18](/release-notes/v4-tucker/4.7.18.md): 2/6/2026 - [4.7.19](/release-notes/v4-tucker/4.7.19.md): 2/7/2026 - [4.7.2](/release-notes/v4-tucker/4.7.2.md): 10/21/2025 - [4.7.20](/release-notes/v4-tucker/4.7.20.md): 2/27/2026 - [4.7.21](/release-notes/v4-tucker/4.7.21.md): 3/6/2026 - [4.7.22](/release-notes/v4-tucker/4.7.22.md): 3/10/2026 - [4.7.23](/release-notes/v4-tucker/4.7.23.md): 3/16/2026 - [4.7.24](/release-notes/v4-tucker/4.7.24.md): 3/19/2026 - [4.7.25](/release-notes/v4-tucker/4.7.25.md): 3/30/2026 - [4.7.26](/release-notes/v4-tucker/4.7.26.md): HarperDB 4.7.26 - [4.7.27](/release-notes/v4-tucker/4.7.27.md): HarperDB 4.7.27 - [4.7.28](/release-notes/v4-tucker/4.7.28.md): HarperDB 4.7.28 - [4.7.29](/release-notes/v4-tucker/4.7.29.md): HarperDB 4.7.29 - [4.7.3](/release-notes/v4-tucker/4.7.3.md): 10/28/2025 - [4.7.30](/release-notes/v4-tucker/4.7.30.md): HarperDB 4.7.30 - [4.7.31](/release-notes/v4-tucker/4.7.31.md): HarperDB 4.7.31 - [4.7.32](/release-notes/v4-tucker/4.7.32.md): HarperDB 4.7.32 - [4.7.33](/release-notes/v4-tucker/4.7.33.md): HarperDB 4.7.33 - [4.7.4](/release-notes/v4-tucker/4.7.4.md): 11/07/2025 - [4.7.5](/release-notes/v4-tucker/4.7.5.md): 11/11/2025 - [4.7.6](/release-notes/v4-tucker/4.7.6.md): 11/12/2025 - [4.7.7](/release-notes/v4-tucker/4.7.7.md): 11/17/2025 - [4.7.8](/release-notes/v4-tucker/4.7.8.md): 11/21/2025 - [4.7.9](/release-notes/v4-tucker/4.7.9.md): 11/25/2025 - [Harper Tucker (Version 4)](/release-notes/v4-tucker/tucker.md): Did you know our release names are dedicated to employee pups? For our fourth release, we have Tucker. - [Harper Lincoln (Version 5)](/release-notes/v5-lincoln.md): All specific full patch version release notes for 5.x.x are available on the releases page. - [5.0 Release Notes](/release-notes/v5-lincoln/5.0.md): Patch Releases - [5.1 Release Notes](/release-notes/v5-lincoln/5.1.md): Patch Releases - [5.2 Release Notes](/release-notes/v5-lincoln/5.2.md): Patch Releases - [Harper Lincoln (Version 5)](/release-notes/v5-lincoln/lincoln.md): In honor of Lincoln. - [v5-migration](/release-notes/v5-lincoln/v5-migration.md): Harper version 5.0 includes many updates to provide a cleaner, more consistent and secure environment. However, there are some breaking changes, and users should review the migration guide for details on how to update their applications. Note that applications that have race conditions that are prone to timing or rely on undocumented features or bugs are always prone to breakage at any point, including major version upgrades. This document describes the important changes to make for applications correctly built in documented APIs. ## Optional - [Harper product overview (harper.fast)](https://www.harper.fast): Marketing site with positioning and use cases; companion llms.txt at https://www.harper.fast/llms.txt --- # Full Documentation Content [Skip to main content](#__docusaurus_skipToContent_fallback) [![Harper Logo](/img/HarperPrimaryBlk.svg)![Harper Logo](/img/HarperPrimaryWht.svg)](/) [Learn](/learn.md)[Reference](/reference/v5.md)[Release Notes](/release-notes/v5-lincoln.md)[Fabric](/fabric.md) [GitHub](https://github.com/HarperFast/documentation) Search # Search the documentation Type your search here v5 (current) Powered by[](https://www.algolia.com/) Documentation * [Learn](/learn.md) * [Reference](/reference) * [Release Notes](/release-notes/v5-lincoln.md) * [Fabric](/fabric.md) Community * [Discord](https://harper.fast/discord) * [LinkedIn](https://www.linkedin.com/company/harperfast/) * [X (Twitter)](https://twitter.com/harper_fast) * [Bluesky](https://bsky.app/profile/harper.fast) More * [Harper Fast](https://harper.fast) * [Blog](https://harper.fast/resources) * [GitHub](https://github.com/HarperFast) * [Contact](mailto:opensource@harperdb.io) Copyright © 2026 HarperDB, Inc. --- # Fabric Studio Fabric Studio is the web-based GUI for Harper. Studio enables you to administer, navigate, and monitor all of your Harper clusters in a simple, user-friendly interface without any knowledge of the underlying Harper API. It’s free to sign up, get started today! [Sign up for free!](https://fabric.harper.fast/#/sign-up) Harper includes a simplified local Studio that is packaged with all Harper installations and served directly from the cluster. It can be enabled in the [configuration file](/reference/v4/configuration/options.md#localstudio). This section is dedicated to the hosted Studio accessed at [studio.harperdb.io](https://fabric.harper.fast/). *** ## How does Studio Work? While Fabric Studio is web based and hosted by us, all database interactions are performed on the Harper cluster the studio is connected to. The Harper Studio loads in your browser, at which point you login to your Harper clusters. Credentials are stored in your browser cache and are not transmitted back to Harper. All database interactions are made via the Harper Operations API directly from your browser to your cluster. ## What can I manage? Fabric Studio enables users to manage both Harper Cloud clusters and privately hosted clusters all from a single UI. All Harper clusters feature identical behavior whether they are hosted by us or by you. --- # API Documentation Harper provides automatically generated API documentation page via [Swagger UI](https://github.com/swagger-api/swagger-ui) integration. This page allows developers to explore and test the various API endpoints available in your Harper instance/cluster. ## Accessing the API Documentation To access the API documentation, navigate to the following URL in your web browser: 1. Sign in to your Harper Fabric Studio. 2. Select your Organization and Cluster. 3. Sign in to the Cluster and navigate to the `APIs` tab in the sub-menu. ## API Execution In order to execute API calls directly from the Swagger UI: 1. Click an endpoint to expand it. 2. Enter the required information in the provided fields under the "Parameters" tab. 3. Scroll down and click the "Execute" button. 4. The server response from the API call will be displayed below, including HTTP status code, and response body. ## Authorize To authorize API requests, you need to include a valid basic authentication header or bearer authentication token. 1. Click on the "Authorize" button in the Swagger Documentation UI. 2. Choose your preferred authentication method (Basic or Bearer) and enter your credentials or token. 3. Click the "Authorize" button to apply the authentication key to all subsequent requests. 4. Execute an API call to verify that the authorization was successful. --- # Cluster Creation & Management ## What is a Cluster? A cluster is a group of instances managed together to run applications and services within a Harper organization; it is the deployable environment where your workloads live. Clusters can be created and managed directly from the Fabric Studio UI (no DevOps required!) ## Creating a Cluster 1. Navigate to your organization page. 2. Click the "+ New Cluster" button in the upper right of the sub-menu. 3. Enter the required fields: * **Cluster Name**: A unique name for your cluster within the organization. * **Harper Deployment**: Choose between Colocated, Dedicated, or Self-hosted (see below for details). * **Performance & Usage**: Select the cluster size that best fits your needs. Examples include Free, Medium, High, Very High. * **Host Name (Full Host Name)**: Enter the host name for your cluster. This will be part of the URL used to access your cluster (e.g., `..harperfabric.com`). * **Region**: Select the geographic region where you want your cluster to be hosted. Examples include US, Global, Europe. * **Estimated P90 Latency, Distribution**: Displays estimated latency based on your selected region and instance size. 4. Click the "Confirm Payment Details" or "Create New Cluster"(if you chose the free tier) button on the bottom right of the page. 5. \*Confirm or replace the preferred payment method. Add a new card if necessary 6. Cluster will begin provisioning as soon as you complete your selections. ## Editing a Cluster To edit an existing cluster: 1. Navigate to your organization page. 2. Locate the cluster you want to edit in the list of clusters. 3. Click the three dots menu next to the cluster name and select "Edit" from the dropdown. 4. Make the necessary changes to the cluster configuration such as: * Performance & Usage * Modifying the Region location and Estimated P90 Latency * Adding or removing additional regions. 5. Click the "Save Changes" or "Confirm Payment Details" button to summarize and apply your modifications. ## Harper Deployment Types: ### Colocated: Multi-tenant clusters are deployed on shared hosts alongside clusters from other organizations, but data and workloads remain completely isolated. Colocated deployments are optimal for organizations seeking excellent performance across available regions. ### Dedicated: Dedicated clusters run on hosts reserved for a single organization. These environments are available in ten more specialized regions, and offer performance isolation and higher resource limits. Dedicated deployments are ideal for organizations with stricter compliance or performance requirements ### Self-hosted: Self-hosted clusters are provisioned entirely outside of Harper’s infrastructure, on an organization's owned and operated servers or cloud accounts. Please follow the cluster configuration menu for more information on estimated performance and cost. Clusters will begin provisioning as soon as you complete your selections. Clusters are provisioned in real time, as soon as selections are complete ### Additional Information: * Cannot guarantee any provisioning time for self-hosted. (**Note**: All performance metrics are estimates unless otherwise noted.) * Once a cluster is created, you will be prompted to set a username and password for each cluster. ## Connecting the Harper CLI to a Cluster The cluster's **Config → Overview** page exposes its **Application URL** — the hostname the Harper CLI and SDKs target. Pass it to `harper login` to authenticate; the CLI stores the token (and writes `HARPER_CLI_TARGET` to a local `.env`) so subsequent commands don't need credentials repeated. ``` harper login # Provide cluster username and password when prompted ``` See [CLI Authentication](/reference/v5/cli/authentication.md) for the full set of authentication methods — including environment variables for CI/CD pipelines. --- # Create a Fabric Studio Account Start at the [Harper Fabric Studio sign up page](https://fabric.harper.fast/#/sign-up). 1. Provide the following information: * First Name * Last Name * Email Address * Password 2. Review the Privacy Policy and Terms of Service. 3. Click the "Sign Up For Free" button. 4. Once you complete the sign-up form, you will receive a verification email. Click the link in the email to verify your account. 5. After verifying your email, you can log in to Fabric Studio using your email address and password. Note: Your email address will be used as your username and cannot be changed. --- # Create an Organization What is an organization? An organization is a way to group and manage multiple clusters and users within Fabric Studio. Organizations help you: * Organize your clusters and resources * Control access based on roles and permissions * Collaborate with team members effectively. * Manage billing and subscriptions for multiple clusters under a single entity. To create a new organization in Fabric Studio, follow these steps: 1. Log in to your Fabric Studio account. 2. Click on the "New Organization" button in the right of the sub-menu. 3. Enter Organization Details: * **Name**: The name of your organization. * **Subdomain**: This will be part of the host name URL for accessing your organization's clusters (e.g., with subdomain "acme" the cluster full host name would be: `.acme.harperfabric.com`). 4. Click the "Create New Organization" button to finalize the creation of your new organization. Once your organization is created, you can invite team members and manage your organization's settings. --- # Custom Domains ## What are Custom Domains? Custom domains allow you to serve your Harper Fabric cluster from your own domain (e.g., `api.yourcompany.com`) instead of the default `..harperfabric.com` URL. Harper Fabric handles domain verification, DNS validation, and TLS certificate provisioning — all from within the Fabric Studio UI. ## Prerequisites Before adding a custom domain, make sure you have: * An existing Harper Fabric cluster (see [Cluster Creation & Management](/fabric/cluster-creation-management.md) if you need to create one) * A domain or subdomain that you own (e.g., `api.example.com`, `data.mycompany.io`) * Access to your DNS registrar or DNS management provider (e.g., Cloudflare, Route 53, GoDaddy, Namecheap) ## Accessing Domain Configuration Domain configuration is accessible through two primary paths: 1. **From the Fabric Dashboard:** * Log in to . * Click on your organization name. * Locate the cluster you want to configure and click the context menu (three dots "**...**"). * Select the **Domains** configuration option. 2. **From Cluster Configuration:** * Navigate to your cluster’s page within your organization. * Select **Config** from the top navigation menu. * Select **Domains** from the left sidebar. ## Adding a Custom Domain 1. Access domain configuration using one of the paths above. 2. Enter your domain in the **New Domain Name** field — for example, `api.example.com` or `example.com`. 3. Click the **+ Add** button. Harper Fabric will register the domain and display the DNS records you need to configure. A confirmation notification will appear: *“Domain added! Please add the TXT record above to your domain registrar.”* ## Configuring DNS Records After adding your domain, Fabric displays two DNS records in the **Next Steps** column of the domains table. You will need to add both of these records at your DNS registrar. ### TXT Record (Domain Verification) This record proves that you own the domain. Add the following to your DNS registrar: * **Type**: `TXT` * **Name**: `_fabric.` (e.g., `_fabric.api.example.com`) * **TTL**: Auto * **Content**: The unique verification string displayed in the Fabric UI (Note: Use the copy buttons next to each value in the Fabric UI to avoid typos.) ### CNAME Record (Traffic Routing) After ownership is verified, you will be instructed to add a CNAME record to your domain registrar pointing to the Harper Fabric load balancer. * **Type**: `CNAME` * **Name**: Your subdomain prefix (e.g., `api` for `api.example.com`) * **TTL**: Auto * **Target**: Your cluster’s Fabric hostname (e.g., `my-cluster.my-org.harperfabric.com`) Apex Domains Some registrars do not support CNAME records for apex domains (e.g., `yourdomain.com`). In this case, we recommend: 1. Registering a subdomain like `www.yourdomain.com`. 2. Redirecting the apex domain (`@` or `yourdomain.com`) to the `www` subdomain. Alternatively, check if your provider supports ALIAS or CNAME flattening. ## Validating Your Domain After adding both DNS records at your registrar, you’ll need to wait for DNS propagation. This typically takes a few minutes to an hour, depending on your TTL settings. Once your DNS records have propagated (this may take **10 minutes** or more): 1. Return to **Config** → **Domains** in your cluster. 2. Click the **Validate** button at the top of the page. Fabric will verify that the required TXT record exists and confirm domain ownership. If validation fails, double-check that the TXT record **Name** and **Content** values match exactly what Fabric provided, then wait a few more minutes and try again. You can verify DNS propagation yourself using tools like [dnschecker.org](https://dnschecker.org) or the command line: ``` dig _fabric.api.example.com TXT ``` ## Binding the Domain After successful validation, use the **Bind** column in the domains table to bind the domain to your cluster. Once bound, Harper Fabric will automatically begin generating an SSL certificate for your domain: * This process typically takes **5-10 minutes**. * The interface will show you the progress as it goes. Everything should be working once the SSL certificate is successfully generated! Traffic to your custom domain will then be routed to your Harper Fabric cluster. ## Managing Domains From the **Config** → **Domains** page, you can manage all custom domains associated with your cluster: * **Add additional domains**: Repeat the process above for each domain you want to connect. * **Remove a domain**: Click **Remove Domain** next to the domain entry in the table. (Note: Don’t forget to also clean up the corresponding DNS records at your registrar.) * **Refresh status**: Click the **Refresh** button to update the validation status of all domains. Each domain entry also displays a unique **Domain ID** (e.g., `dom-xxxxxxxxxxxx`) for reference. ## Additional Information * TLS certificates are managed automatically by Harper Fabric once a domain is validated and bound. You can also manage custom certificates under the **Certificates** section in Config. * You can add multiple custom domains to the same cluster. * Your cluster’s default Fabric hostname can be found on the **Config** → **Overview** page under **Application URL**. * DNS propagation can take up to 24–48 hours in rare cases, though it typically completes within minutes. --- # Database Management ## Import CSV Data 1. Navigate to the "Database" tab in the sub-menu 2. Select the desired database and table. If no tables exist, create a new table first. 3. Click the "Import CSV" button located above the table view. 4. Select/Drag-and-drop the CSV file from your local machine. 5. Click the "Upload CSV" button to import the data into the selected table. 6. The data will be uploaded and displayed in the table view. This may take a few moments depending on the size of the file. 7. Refresh the table view by clicking the "Refresh" button if necessary. ## Add a New Record 1. Navigate to the "Database" tab in the sub-menu 2. Select the desired database and table. 3. Click the "Add New Record" button located above the table view. 4. Fill in the necessary fields in the form that appears.(These fields will correspond to the columns/schema in the selected table.) 5. Click the "Save" button to add the new record to the table. ## Create a New Table 1. Navigate to the "Database" tab in the sub-menu 2. Select the desired database. 3. Click the "Create a Table" button located to the left of the table view. 4. Fill in the: * Table Name: The name of the new table. * Primary Key: The primary key field for the table. * Database Name: The database where the table will be created.(Will automatically fill based upon the selected database where the table is being created.) 5. Click the "Create New Table" button to create the new table. 6. The new table will appear in the table list on the left side of the screen. ## Delete a Table 1. Navigate to the "Database" tab in the sub-menu 2. Select the desired database and table. 3. Click the three dots button located above the table view on the top right. 4. Select "Drop Table" from the dropdown menu. 5. Confirm the deletion by clicking the "Drop" button in the confirmation dialog. 6. The table will be deleted from the database. ## Delete a Record 1. Navigate to the "Database" tab in the sub-menu 2. Select the desired database and table. 3. Locate the record you want to delete in the table view. 4. Click the record and a modal will appear. 5. Click the "Delete Row" button in the modal to confirm the deletion. 6. The record will be deleted from the table. ## Edit a Record 1. Navigate to the "Database" tab in the sub-menu 2. Select the desired database and table. 3. Locate the record you want to edit in the table view. 4. Click the record and a modal will appear. 5. Edit the fields as necessary in the modal. 6. Click the "Save Changes" button to save the changes to the record. 7. The updated record will be displayed in the table view. --- # Setup Grafana Integration with Harper Grafana is an observability platform for monitoring and visualizing metrics, logs, and traces. Harper provides a plugin to integrate with Grafana for enhanced analytics and visualization capabilities. To install the Harper Grafana integration: 1. Navigate to Harper's plug-in inside the [Grafana marketplace](https://grafana.com/grafana/plugins/harperfast-harper-datasource/). 2. Sign-in to your Grafana account. If you do not have an account, you will need to create one. 3. Click the "Get plugin" button. ## Installing on a Local/Self-Hosted Grafana Instance 4. Follow the steps under "[Installing on a local Grafana](https://grafana.com/grafana/plugins/harperfast-harper-datasource/?tab=installation)" ## Connect to Harper 4. Navigate to your Grafana instance URL specified under `Installing Harper on Grafana Cloud`. 5. On the left sidebar, click the `Connections` navigation link and select `Add new connection` 6. In the search bar, type `Harper` to filter the list of available data source plugins. 7. On the top right corner, click the `Add new data source` button. 8. You will be directed to the `Settings` page for the new data source. Configure the following settings: * **Name**: Provide a name for the data source (e.g., `My Fabric Cluster Analytics`). * **Operations API URL**: Enter the URL to your Harper Fabric cluster's operations API * Found in Harper Fabric Studio navigating to the cluster: * Clicking the three dots and selecting `Copy API Url`. * Add `:9925` to the end of the URL if not already present. * **Username**: Enter a username with permission to use the analytics ops in the Operations API. * **Password**: Enter the password for the specified username. 9. Click the `Save & Test` button to save the data source configuration and test the connection. You should see a message indicating that the data source is working. ## Building Dashboards Once the Harper data source is configured, you can start building dashboards in Grafana. ## Explore 1. Click on the `Explore` navigation link in the left sidebar. 2. You can now create queries using the Harper data source to visualize your Harper Fabric cluster metrics and logs. Reference the [Harper Analytics Operations](/reference/v4/analytics/overview.md) for more details on available metrics and query options. --- # Logging ## Log Filtering Log filtering allows users to customize the log output based on specific criteria. This can help in isolating relevant information and reducing noise in the logs list. To filter logs: 1. Navigate to the "Logs" section in Fabric Studio. 2. Use the filter options available at the left side menu logs list to specify criteria such as: * Log Limit: The maximum number of log entries to display(e.g. 10, 100, 250, 500, 1000) (default is 100) * Log Level (e.g., Error, Warn, Info, Debug, Trace, Notify, All) (default is All) * Start Date: The beginning date and time for the log entries to display * End Date: The ending date and time for the log entries to display 3. Click the "Apply Filters" button to update the log list based on the selected criteria. 4. The log list will refresh to show only the entries that match the specified filters. ## Log Details To view detailed information about a specific log entry: 1. Navigate to the "Logs" section in Fabric Studio. 2. Click on a log entry in the logs list to open the log details modal. 3. The log details modal will display comprehensive information about the selected log entry, including: * Timestamp: The date and time when the log entry was created * Level: The severity level of the log entry (e.g., Error, Warn) * Thread: The thread identifier where the log entry originated * Tags: Any associated tags for categorizing the log entry * Message: The main content of the log entry 4. Review the information in the modal to gain insights into the specific event or issue recorded in the log entry. --- # Managing Applications After setting a username/password for cluster, you will automatically be directed to the applications page for the cluster. From here, users can either import or create an application to run on Harper. ## Creating a New Application To create a new application: 1. Select "Applications" from the menu if not already there 2. Click the "+ New Application" button in the upper right of the sub-menu 3. Enter the "New Application Name" (max 75 characters) 4. Click "Create"( **Note**: this will prompt the cluster to restart) 5. Your new application will appear in the applications list ## Importing an Application To import an existing application: 1. Select "Applications" from the menu if not already there 2. Click the "Import Application" button in the center of the page 3. Enter the "Application Name" (max 75 characters) 4. Enter the "Package Reference URL" (must be a valid URL) 5. Click "Create"( **Note**: this will prompt the cluster to restart) 6. Your imported application will appear in the applications list ## Updating an application To update an existing application: 1. Select "Applications" from the menu if not already there 2. Click the top level name of the application in the applications list 3. Select "Redeploy Application" 4. Enter the new "Package Reference URL" (must be a valid URL) 5. Click "Redeploy"( **Note**: this will prompt the cluster to restart) 6. Your application will be updated and redeployed. --- # Organization Management Organizations can be managed in a variety of ways, including: roles and user permissions, adding/removing users, creating new environments (clusters) and updating billing information. ## Role Management Organizations can be made up of many users, each with different roles and permissions. Roles and permissions can be created and managed by the organization admin. ### Creating a new role 1. Navigate to the organization page 2. Click "Roles" in the menu on top of screen 3. User will be navigated to role table 4. Note: “admin” will appear in role table as default role, with 1 user (creator) assigned. Admins can update and delete organizations 5. Click on “+ Add” button in top right corner and a modal will appear 6. Enter role information: * Role Name: name of the new role * Can update organization: toggle on/off * Can delete organization: toggle on/off * JSON Permissions: enter custom JSON permissions 7. Click "Save Changes" button to create new role Create and customize as many roles as appropriate for organization ### User management 1. Select “Users” from menu on top of screen 2. List of all active users from organization will appear 3. Note: organization creator will default as active admin, with a precreated user ID 4. To add new users, click “+ Add” icon in top right corner 5. Enter new user’s email 6. Select desired role from drop down. (**Note**: roles and associated permissions can be created and managed by organization admins) 7. Click "Add User" button to finalize adding new user **Note**: If they don't yet have a Fabric account, you will be prompted to invite them. They will be sent a verification email with instructions on how to activate their account. Once accepted, the new user will be added to the organization, with all the privileges of their shiny new role. --- # Welcome to Harper Learn This documentation section contains thorough guides for learning how to develop and manage applications with Harper. The guides are present in a logical order to build up knowledge across Harper's vast feature set. The guides are example based and provide a hands-on approach to teaching and demonstrating key features. Guides can be referenced independently, but assume the reader is familiar with concepts presented in previous guides. Most guides present both local-based and [Harper Fabric](https://fabric.harper.fast) cloud-based examples and instructions. Regardless, in order to properly complete all examples, we recommend the following prerequisite tools installed and configured on your local machine: * Command line access and general file-system and networking permissions * `sudo` is not required * For local development, Harper requires permission to read/write files and localhost networking permissions * HTTP client of choice * Most examples will present both curl and fetch-based HTTP requests * GUI-based HTTP clients will also work fine * Node.js Current, Active LTS, or Maintenance LTS version * For updated Node.js installation instructions refer to the official [Download Node.js](https://nodejs.org/en/download) page * For more information on valid versions refer to [Node.js Releases](https://nodejs.org/en/about/previous-releases) * Verify your Node.js version by running `node -v` in the command line * Code editor of choice * Everything from `vim` to Visual Studio Code to WebStorm IDE will work fine for the purposes of these guides If you ever have questions, join our official community [Discord](https://harper.fast/discord). Harper documentation is open source. If you notice anything out-of-place with the guide content, please [open an issue](https://github.com/HarperFast/documentation/issues) or submit changes directly using the "Edit this page" link at the bottom of every page. info Eagle-eye developers may notice some things still reference Harper's previous name, HarperDB. The "database" part is not gone, Harper has simply evolved to become so much more than *just* a database. Harper is one-and-the-same with HarperDB, so please bare with us as we chip away at some renames. When you're ready to get started, click the "Next" tab below to begin your Harper adventure! --- # Coming Soon --- # Active Caching with Subscriptions The passive caching pattern — fetching from the source on demand, expiring on a timer — works well for data that changes infrequently and where brief staleness is acceptable. But for data that changes often, a TTL is a blunt instrument: too short and you're making unnecessary upstream calls; too long and you're serving stale data. Active caching solves this by inverting the flow. Instead of polling the source, your cache *subscribes* to it. When the source changes, it pushes the update directly into the Harper cache — instantly, without waiting for a TTL to expire. Records stay fresh until they actually change, and there's no background polling overhead. In this guide you will implement an active cache for a live sports scoreboard feed. The source streams score updates as server-sent events; the Harper cache receives each update immediately and serves the current score to any number of downstream clients. ## What You Will Learn * How passive and active caching differ in architecture and trade-offs * How to implement a `subscribe` method on a source Resource * How to yield events from an async generator * How to push events from a callback-based source using the subscription stream * When to use `put` vs. `invalidate` events * How to control which threads run the subscription ## Prerequisites * Completed [Caching with Harper](/learn/developers/caching-with-harper.md) * Familiarity with async generators in JavaScript ## Passive vs. Active Caching In the passive pattern, Harper drives the flow: ``` Client → Harper (cache miss or stale) → Source → Harper stores result → Client ``` The cache only knows data is stale when a client asks for it and the TTL has elapsed. Between TTL resets, the source can change any number of times and the cache has no idea. In the active pattern, the source drives the flow: ``` Source changes → Source pushes event → Harper updates cache proactively Client → Harper (always fresh) → Client ``` Harper receives every change the moment it happens. No TTL is needed — records stay cached indefinitely and are only replaced when the source says they changed. | Aspect | Passive | Active | | ------------------ | ------------------------------- | ----------------------------- | | TTL required | Yes | No (optional as a fallback) | | Staleness window | Up to TTL duration | Near-zero | | Upstream calls | One per record per TTL interval | Only on actual changes | | Source requirement | Simple `get` endpoint | Streaming or push-capable API | | Complexity | Low | Moderate | ## Setting Up the Application Clone the example repository and open it in your editor. ``` git clone https://github.com/HarperFast/active-caching-example.git harper-active-caching ``` The repository has the following structure: ``` harper-active-caching/ ├── config.yaml ├── schema.graphql └── resources.js ``` Start Harper in dev mode from inside the directory: ``` harper dev . ``` ## Defining the Cache Table Open `schema.graphql`. The scoreboard cache table has no `expiration` — it stays valid until the source pushes an update: ``` type GameScore @table @export { id: ID @primaryKey # game ID, e.g. "game-001" homeTeam: String @indexed awayTeam: String @indexed homeScore: Int awayScore: Int status: String @indexed # "live", "final", "upcoming" lastUpdated: Long } ``` Without `expiration`, records never go stale passively. The only way they update is when the source pushes a `put` or `invalidate` event — or when Harper calls `get()` on a cache miss for a record that hasn't been loaded yet. ## Implementing the Active Source Open `resources.js`. The `ScoreboardFeed` class connects to an imaginary streaming API and yields score updates as Harper cache events. ``` // resources.js const SCORES_API_BASE = process.env.SCORES_API_BASE ?? 'https://scores.example.com'; const scoreboardFeed = { async get(id) { // Called on cache miss — fetch the initial state for a specific game const response = await fetch(`${SCORES_API_BASE}/games/${id}`); if (!response.ok) { const error = new Error('Game not found'); error.statusCode = 404; throw error; } return response.json(); }, async *subscribe() { // Called once to stream all ongoing updates into the cache const response = await fetch(`${SCORES_API_BASE}/stream`, { headers: { Accept: 'text/event-stream' }, }); for await (const chunk of response.body) { const lines = chunk.toString().split('\n'); for (const line of lines) { if (!line.startsWith('data: ')) continue; const event = JSON.parse(line.slice(6)); yield { type: 'put', id: event.gameId, value: event.score, timestamp: event.ts, }; } } }, }; tables.GameScore.sourcedFrom(scoreboardFeed); ``` `get()` and `subscribe()` have distinct roles: * **`get()`** — handles cache misses. If a client asks for `game-001` before the subscription has delivered it, Harper calls `get()` to fetch the initial state. * **`subscribe()`** — streams all future updates. Harper calls this once at startup and propagates every yielded event into the cache automatically. ### How Harper calls `subscribe` Harper calls `subscribe()` once per process immediately after `sourcedFrom` is registered. The method should return (or be) an async iterable that yields events indefinitely. Harper does not call `subscribe()` per record — a single subscription covers the entire table. ## Event Types The `type` field on each yielded event controls how Harper applies the update: ``` // Replace the entire cached record with the new value yield { type: 'put', id: 'game-001', value: { homeScore: 3, awayScore: 1, ... } }; // Tell Harper the record changed without sending the new value. // Harper will evict the record; the next client request triggers a get() call. yield { type: 'invalidate', id: 'game-001' }; // Remove the record from the cache yield { type: 'delete', id: 'game-001' }; ``` Use `put` when the event stream includes full record values — this is the most efficient path because Harper stores the value immediately without a follow-up `get()` call. Use `invalidate` when the stream only signals that something changed, and you want Harper to lazy-load the new value on demand. ## Using a Callback-Based Source Not all sources use async iterables. If your upstream uses a callback or event-emitter API, use the default subscription stream instead of an async generator: ``` const scoreboardFeed = { subscribe() { const subscription = super.subscribe(); // default stream const socket = new WebSocket(`wss://scores.example.com/ws`); socket.on('message', (raw) => { const event = JSON.parse(raw); subscription.send({ type: 'put', id: event.gameId, value: event.score, timestamp: event.ts, }); }); socket.on('error', (err) => { subscription.error(err); // surfaces to Harper's error handling }); return subscription; }, }; ``` ## Configuring the Application Open `config.yaml`: ``` graphqlSchema: files: 'schema.graphql' rest: true jsResource: files: 'resources.js' ``` * `graphqlSchema` loads `schema.graphql` and creates the `GameScore` table. * `rest` exposes `GameScore` as an HTTP endpoint. * `jsResource` loads `resources.js`, registers `ScoreboardFeed`, and starts the subscription on startup. ## Observing Active Updates With Harper running, open two terminals. In the first, poll a game score every second: #### curl ``` watch -n1 'curl -s http://localhost:9926/GameScore/game-001 | jq .' ``` #### fetch ``` setInterval(async () => { const data = await fetch('http://localhost:9926/GameScore/game-001').then((r) => r.json()); console.log(data.homeScore, data.awayScore, data.status); }, 1000); ``` In the second terminal, simulate a score update being pushed by the source (bypassing the stream for testing): #### curl ``` curl -X PUT 'http://localhost:9926/GameScore/game-001' \ -H 'Content-Type: application/json' \ -d '{"homeTeam":"Rangers","awayTeam":"Hawks","homeScore":3,"awayScore":2,"status":"live","lastUpdated":1712500000000}' ``` #### fetch ``` await fetch('http://localhost:9926/GameScore/game-001', { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ homeTeam: 'Rangers', awayTeam: 'Hawks', homeScore: 3, awayScore: 2, status: 'live', lastUpdated: 1712500000000, }), }); ``` The first terminal will reflect the new score immediately — no TTL expiry needed, no cache miss, no upstream call. The cache was updated in-place by the `put` event. ## Controlling Subscription Threads Harper runs multiple worker threads. By default, `subscribe()` runs on exactly one thread to prevent duplicate events and race conditions — if every thread opened its own connection to the source, every event would be processed multiple times. In rare cases you may want subscriptions on multiple threads — for example, if your source shards data and each thread should subscribe to a different shard. Use `subscribeOnThisThread` to control this: ``` const scoreboardFeed = { subscribeOnThisThread(threadIndex) { return threadIndex === 0; // default: only thread 0 } async *subscribe() { ... } } ``` ## Adding a TTL Fallback Even with an active subscription, network interruptions can cause the connection to drop. You can add `expiration` to the table as a safety net — if the subscription fails and a record becomes stale, Harper will fall back to calling `get()`: ``` type GameScore @table(expiration: 60) @export { id: ID @primaryKey ... } ``` With this in place, records are guaranteed to be at most 60 seconds stale even if the subscription connection drops. ## Putting It All Together Here is the complete `resources.js`: ``` // resources.js const SCORES_API_BASE = process.env.SCORES_API_BASE ?? 'https://scores.example.com'; const scoreboardFeed = { async get(id) { const response = await fetch(`${SCORES_API_BASE}/games/${id}`); if (!response.ok) { const error = new Error('Game not found'); error.statusCode = 404; throw error; } return response.json(); }, async *subscribe() { const response = await fetch(`${SCORES_API_BASE}/stream`, { headers: { Accept: 'text/event-stream' }, }); for await (const chunk of response.body) { const lines = chunk.toString().split('\n'); for (const line of lines) { if (!line.startsWith('data: ')) continue; const event = JSON.parse(line.slice(6)); yield { type: 'put', id: event.gameId, value: event.score, timestamp: event.ts, }; } } }, }; tables.GameScore.sourcedFrom(scoreboardFeed); ``` ## What Comes Next This guide covered active caching with a push-based subscription. The [Semantic Caching with Vector Indexing](/learn/developers/semantic-caching-vector-indexing.md) guide applies caching to AI-powered search — instead of keying the cache by exact ID, Harper finds semantically similar cached answers using vector similarity, so equivalent questions never hit the LLM twice. ## Additional Resources * [Caching with Harper](/learn/developers/caching-with-harper.md) — foundational passive caching guide * [Resource API](/reference/v5/resources/resource-api.md) — `sourcedFrom`, `subscribe`, event types * [Database Schema](/reference/v5/database/schema.md) — `@table(expiration:)` and eviction configuration --- # Caching AI Generations with Harper AI API calls are expensive. Generating a product description, summarizing an article, or personalizing a recommendation with a large language model can cost fractions of a cent per call — but those fractions add up fast at scale. And in most applications, the same content gets generated over and over: the same product page viewed by thousands of users, the same document summarized dozens of times. Harper's caching system is a natural fit for this problem. You can generate AI content once, cache it close to your users, and serve it instantly on every subsequent request. When the underlying data changes you invalidate the cached generation and let it be regenerated on the next access. In this guide you will build a product description endpoint backed by an LLM, wrap it in a Harper cache table, and implement an invalidation strategy so descriptions stay fresh when product data changes. ## What You Will Learn * How to build a Resource class that calls an AI API and returns generated content * How to cache AI generations in a Harper table with an appropriate TTL * How to expose an invalidation endpoint so descriptions can be refreshed on demand * How to use ETags and conditional requests to avoid redundant content delivery downstream ## Prerequisites * Completed [Caching with Harper](/learn/developers/caching-with-harper.md) (recommended — this guide builds directly on those concepts) * Working Harper installation (local or Fabric) * An OpenAI API key (or another compatible LLM provider) * A command-line HTTP client (`curl` recommended) or familiarity with `fetch` ## The Problem This Solves Consider a product catalog with hundreds of items. Each product needs a compelling description tailored to its attributes. Generating those descriptions with an LLM produces higher-quality copy than static text, and you can re-generate them easily as your brand voice evolves. The challenge: you cannot call the LLM on every product page load. A single generation might take 1–3 seconds and cost money. Instead, you generate on first access, cache the result, and regenerate only when needed. Harper makes this straightforward: 1. A `ProductDescription` cache table stores generated descriptions with a long TTL (24 hours). 2. A `DescriptionGenerator` Resource class calls the LLM and returns the generated text. 3. `sourcedFrom` connects the generator to the cache — Harper handles all the caching logic. 4. An exported `ProductDescription` Resource class provides the REST endpoint and handles invalidation requests. ## Setting Up the Application Clone the example repository: ``` git clone https://github.com/HarperFast/ai-cache-example.git harper-ai-cache ``` Create a `.env` file with your API key: ``` OPENAI_API_KEY=sk-... ``` Then start Harper: ``` harper dev . ``` ## Defining the Schema Open `schema.graphql`. There are two tables: `Product` holds the product catalog, and `ProductDescription` is the cache table for AI-generated descriptions. ``` type Product @table @export { id: ID @primaryKey name: String @indexed category: String @indexed price: Float features: [String] } type ProductDescription @table(expiration: 86400) @export { id: ID @primaryKey description: String generatedAt: Long product: Product @relationship(from: id) } ``` The `product` field on `ProductDescription` uses `@relationship(from: id)` to declare that its own primary key (`id`) is a foreign key pointing to `Product`. Since `ProductDescription` records share the same ID as the product they describe, this gives consumers a way to fetch the full product details alongside the description in a single request. Both tables use `@export` — Harper's auto-generated REST endpoints are sufficient for reading and writing. Invalidation will be handled automatically inside a custom `Product` Resource class rather than through a separate endpoint. `ProductDescription` has a 24-hour TTL (`expiration: 86400`). A one-day window is reasonable for marketing copy — it's long enough to avoid redundant generation, and short enough that descriptions stay reasonably current even without explicit invalidation. ## Configuring the Application Open `config.yaml` and enable the required plugins: ``` graphqlSchema: files: 'schema.graphql' rest: true jsResource: files: 'resources.js' ``` * `graphqlSchema` loads `schema.graphql` and creates both tables. * `rest` enables Harper's REST API on port `9926`. * `jsResource` loads `resources.js`, registering the generator source, the `sourcedFrom` connection, and the exported `ProductDescription` endpoint. ## Seeding the Product Catalog Rather than seeding data from `resources.js`, this application uses Harper's built-in [Data Loader](/reference/v5/database/data-loader.md) to populate the `Product` table from a JSON file on startup. Add `dataLoader` to `config.yaml`: ``` graphqlSchema: files: 'schema.graphql' rest: true jsResource: files: 'resources.js' dataLoader: files: 'data/products.json' ``` Then create `data/products.json`: ``` { "table": "Product", "records": [ { "id": "prod-001", "name": "Titanium Water Bottle", "category": "Outdoors", "price": 39.99, "features": ["BPA-free", "Keeps cold 24h", "Lightweight 200g"] }, { "id": "prod-002", "name": "Noise-Cancelling Headphones", "category": "Electronics", "price": 199.99, "features": ["40h battery", "Foldable", "USB-C charging"] } ] } ``` The Data Loader runs on every startup and deployment. It uses content hashing to skip records that haven't changed, so it's safe to redeploy without duplicating data or overwriting any manual edits. ## Building the Description Generator The `DescriptionGenerator` class is the upstream source for the `ProductDescription` cache. Its `get` method fetches the product from the `Product` table (which it extends), builds a prompt, and calls the OpenAI API. ``` // resources.js const OPENAI_API_KEY = process.env.OPENAI_API_KEY; class DescriptionGenerator extends tables.Product { static async get(productId) { // Load the product data from our base class, the Product table const product = await super.get(productId); if (!product) { const error = new Error('Product not found'); error.statusCode = 404; throw error; } // Build a prompt from the product's attributes const prompt = `Write a compelling, two-sentence product description for the following item. Product name: ${product.name} Category: ${product.category} Price: $${product.price} Features: ${product.features.join(', ')} Keep the tone enthusiastic but professional.`; // Call the OpenAI chat completions API const response = await fetch('https://api.openai.com/v1/chat/completions', { method: 'POST', headers: { 'Content-Type': 'application/json', 'Authorization': `Bearer ${OPENAI_API_KEY}`, }, body: JSON.stringify({ model: 'gpt-4o-mini', messages: [{ role: 'user', content: prompt }], max_tokens: 120, }), }); const result = await response.json(); const description = result.choices[0].message.content.trim(); return { id: productId, description, generatedAt: Date.now(), }; } } tables.ProductDescription.sourcedFrom(DescriptionGenerator); ``` With `sourcedFrom` in place, Harper now handles all caching behavior automatically: * **Cache miss**: the first request for `/ProductDescription/prod-001` calls `DescriptionGenerator.get()`, stores the result, and returns it. * **Cache hit**: subsequent requests within 24 hours return the stored description instantly — no LLM call. * **Cache expiry**: after 24 hours, the next request regenerates the description. * **Cache invalidation**: Because we are extending a table, Harper automatically invalidates the cache when the source table is updated. ## Requesting a Generated Description With Harper running, request a description for the first product: #### curl ``` curl -i 'http://localhost:9926/ProductDescription/prod-001' ``` #### fetch ``` const response = await fetch('http://localhost:9926/ProductDescription/prod-001'); const etag = response.headers.get('etag'); const data = await response.json(); console.log(data.description); console.log('ETag:', etag); ``` The first request will take a moment — Harper is calling the OpenAI API. You will get back something like: #### curl ``` HTTP/1.1 200 OK content-type: application/json etag: "abCDefGHij" ... { "id": "prod-001", "description": "Stay hydrated on every adventure with this ultralight titanium bottle that keeps your drinks cold for a full 24 hours. BPA-free and weighing just 200g, it's built for those who refuse to compromise between performance and sustainability.", "generatedAt": 1712500000000 } ``` #### fetch ``` Stay hydrated on every adventure with this ultralight titanium bottle... ETag: "abCDefGHij" ``` Make the same request again immediately: #### curl ``` curl -i 'http://localhost:9926/ProductDescription/prod-001' ``` #### fetch ``` const second = await fetch('http://localhost:9926/ProductDescription/prod-001'); console.log(second.status); ``` #### curl ``` HTTP/1.1 200 OK content-type: application/json etag: "abCDefGHij" server-timing: db;dur=1.2 ... { ... same description ... } ``` #### fetch ``` 200 ``` The response is instant. Check the `server-timing` header — the database read time will be in single-digit milliseconds rather than the seconds the LLM call took. ## Using ETags to Avoid Redundant Transfers Harper automatically includes an `ETag` header with every record response. You can use `If-None-Match` to avoid re-transferring a description your client already has: #### curl ``` # Use the etag value from the previous response, double quotes included curl -i 'http://localhost:9926/ProductDescription/prod-001' \ -H 'If-None-Match: "abCDefGHij"' ``` #### fetch ``` const first = await fetch('http://localhost:9926/ProductDescription/prod-001'); const etag = first.headers.get('etag'); // e.g. "abCDefGHij" const second = await fetch('http://localhost:9926/ProductDescription/prod-001', { headers: { 'If-None-Match': etag }, }); console.log(second.status); ``` #### curl ``` HTTP/1.1 304 Not Modified etag: "abCDefGHij" ``` #### fetch ``` 304 ``` A `304 Not Modified` response means the cached description your client holds is still current. No data is serialized or transmitted. This layering — Harper's internal cache plus HTTP conditional requests for downstream caches — means a regeneration event only propagates to clients when their cached copy actually becomes stale. ## Querying a Description with Its Product Because `ProductDescription` declares a `@relationship` to `Product`, you can fetch the description and the full product record together using the `select` query parameter: #### curl ``` curl -s 'http://localhost:9926/ProductDescription/prod-001?select(description,generatedAt,product(name,price,features))' ``` #### fetch ``` const response = await fetch( 'http://localhost:9926/ProductDescription/prod-001?select(description,generatedAt,product(name,price,features))' ); const data = await response.json(); console.log(data); ``` #### curl ``` { "description": "Stay hydrated on every adventure...", "generatedAt": 1712500000000, "product": { "name": "Titanium Water Bottle", "price": 39.99, "features": ["BPA-free", "Keeps cold 24h", "Lightweight 200g"] } } ``` #### fetch ``` { description: 'Stay hydrated on every adventure...', generatedAt: 1712500000000, product: { name: 'Titanium Water Bottle', price: 39.99, features: [ ... ] } } ``` Harper resolves the relationship and joins the `Product` record in a single database read — no extra round-trips. ## Invalidating Descriptions When Products Change With the custom `Product` class in place, a single `PATCH` to the product is all it takes. Harper calls your `patch` override, saves the update, and invalidates the cached description in the same operation — no second request needed. #### curl ``` curl -X PATCH 'http://localhost:9926/Product/prod-001' \ -H 'Content-Type: application/json' \ -d '{"features": ["BPA-free", "Keeps cold 48h", "Lightweight 180g"]}' ``` #### fetch ``` await fetch('http://localhost:9926/Product/prod-001', { method: 'PATCH', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ features: ['BPA-free', 'Keeps cold 48h', 'Lightweight 180g'] }), }); ``` #### curl ``` HTTP/1.1 204 No Content ``` #### fetch ``` 204 ``` The next `GET /ProductDescription/prod-001` triggers a new LLM call with the updated features. Every subsequent request serves the new cached description instantly. ## Handling Errors Gracefully LLM APIs can fail — rate limits, network errors, service outages. By default, Harper will surface a 500 error to the client if `DescriptionGenerator.get()` throws. For a production cache you may want to serve the stale description if the LLM is temporarily unavailable. Add the `stale-if-error` Cache-Control directive to your request to accept a stale cached response when the source returns an error: #### curl ``` curl -i 'http://localhost:9926/ProductDescription/prod-001' \ -H 'Cache-Control: stale-if-error' ``` #### fetch ``` const response = await fetch('http://localhost:9926/ProductDescription/prod-001', { headers: { 'Cache-Control': 'stale-if-error' }, }); ``` With `stale-if-error`, Harper returns the most recently cached description rather than propagating the upstream error — a sensible default for AI-generated marketing copy where slightly stale content is better than a broken page. ## Going Further This guide used the *passive* caching pattern: Harper fetches from the source on demand. For high-traffic applications, you may want to *proactively* populate the cache — for example, pre-generating descriptions for all products at startup or on a schedule. This is covered in the [Active Caching and Subscriptions](#) guide (coming soon). You might also consider: * **Category-wide invalidation**: extend the `patch` handler to iterate `tables.ProductDescription` and call `invalidate` on each record when a category-level change affects many products at once. * **Version-aware ETags**: include a version field in `DescriptionGenerator.get()` so clients can detect stale descriptions proactively rather than waiting for a server-side invalidation. * **Cost tracking**: log `generatedAt` changes to measure how often you are actually hitting the LLM versus serving from cache. ## Additional Resources * [Caching with Harper](/learn/developers/caching-with-harper.md) — foundational guide covering `sourcedFrom`, ETags, and TTL expiration * [Resource API](/reference/v5/resources/resource-api.md) — `sourcedFrom`, `invalidate`, `getContext`, static methods * [Database Schema](/reference/v5/database/schema.md) — `@table(expiration:)` directive reference * [REST Headers](/reference/v5/rest/headers.md) — ETag, `If-None-Match`, `Cache-Control` directives * [harper-ecommerce-template](https://github.com/HarperFast/harper-ecommerce-template) — Full ecommerce application using Harper with AI-generated content --- # Caching with Harper Every production application hits the same wall eventually: an external API you depend on is slow, rate-limited, or expensive to call. Harper's caching system lets you wrap any external data source — a REST API, a microservice, a database — and serve responses from a fast local cache while transparently fetching fresh data only when needed. In this guide you will build a Harper application that caches responses from a public API, observe caching behavior using ETags and HTTP status codes, and learn how to invalidate entries on demand. ## What You Will Learn * How to define a cache table using the `@table(expiration:)` schema directive * How to wrap an external data source with a custom Resource class * How to connect a data source to a cache table using `sourcedFrom` * How to observe caching behavior through ETag and `304 Not Modified` responses * How to manually invalidate a cached entry ## Prerequisites * Completed [Install and Connect Harper](/learn/getting-started/install-and-connect-harper.md) * Completed [Create Your First Application](/learn/getting-started/create-your-first-application.md) * Working Harper installation (local or Fabric) * A command-line HTTP client (`curl` recommended) or familiarity with `fetch` ## Setting Up the Application Clone the example repository and open it in your editor. If you are using a container install, clone into the mounted `dev/` directory. ``` git clone https://github.com/HarperFast/caching-guide-example.git harper-caching ``` The repository has the following structure: ``` harper-caching/ ├── config.yaml ├── schema.graphql └── resources.js ``` Start Harper in dev mode from inside the directory: ``` harper dev . ``` ## Defining a Cache Table Open `schema.graphql`. The cache table is defined with a single addition to the familiar `@table` directive: an `expiration` argument. ``` type JokeCache @table(expiration: 60) @export { id: ID @primaryKey setup: String punchline: String } ``` The `expiration: 60` argument tells Harper that any record in this table is considered *stale* after 60 seconds. When a stale record is requested, Harper fetches a fresh copy from the source resource and stores it before returning the response. info A table's `expiration` is measured in seconds. Harper also supports separate `eviction` and `scanInterval` arguments if you need fine-grained control over when records are physically removed from the table. See the [Schema reference](/reference/v5/database/schema.md) for details. ## Wrapping an External Data Source The source for the cache is a simple object in `resources.js`. The [public jokeAPI](https://official-joke-api.appspot.com/) returns a joke by ID as a JSON object — a perfect stand-in for any real external API. ``` // resources.js const jokeAPI = { async get(id) { const response = await fetch(`https://official-joke-api.appspot.com/jokes/${id}`); return response.json(); }, }; tables.JokeCache.sourcedFrom(jokeAPI); ``` `sourcedFrom` registers `jokeAPI` as the upstream source for `JokeCache`. Harper's caching behavior now works as follows: 1. A request arrives for `/JokeCache/1`. 2. Harper checks if the record with id `1` exists in `JokeCache` and is not stale. 3. If it is fresh, Harper returns it immediately. 4. If it is missing or stale, Harper calls `jokeAPI.get()` to fetch the data, stores it in `JokeCache`, and returns the result. Multiple simultaneous requests for the same missing or stale record will all wait on a single upstream call — Harper prevents cache stampedes automatically. ## Configuring the Application With the schema and resource in place, open `config.yaml` and enable the two plugins this application needs: ``` graphqlSchema: files: 'schema.graphql' rest: true jsResource: files: 'resources.js' ``` * `graphqlSchema` loads `schema.graphql` and creates the `JokeCache` table. * `rest` enables Harper's REST API on port `9926`, exposing any `@export`-ed tables and resources as HTTP endpoints. * `jsResource` loads `resources.js`, registering the `jokeAPI` source and the `sourcedFrom` connection — as well as any exported Resource classes as endpoints. Restart Harper (or let `harper dev` pick up the change automatically), then continue. note If you need to check your work, checkout the [`01-cached-api`](https://github.com/HarperFast/caching-guide-example/tree/01-cached-api) branch. ## Making Your First Cached Request With Harper running, fetch a joke: #### curl ``` curl -i 'http://localhost:9926/JokeCache/1' ``` #### fetch ``` const response = await fetch('http://localhost:9926/JokeCache/1'); console.log(response.status); // 200 console.log(response.headers.get('etag')); const joke = await response.json(); console.log(joke); ``` You should see a `200` response: #### curl ``` HTTP/1.1 200 OK content-type: application/json etag: "abCDefGHij" ... { "id": 1, "type": "general", "setup": "What did the ocean say to the beach?", "punchline": "Nothing, it just waved." } ``` #### fetch ``` 200 "abCDefGHij" { id: 1, type: 'general', setup: 'What did the ocean say to the beach?', punchline: 'Nothing, it just waved.' } ``` Note the double quotes on the ETag — they are part of the value itself, not just string delimiters. You will need to include them when passing the ETag back in a request header. Harper automatically computes an ETag from the record's last-modified timestamp. This is the key to downstream caching. ## Observing Caching Behavior with ETags Make the same request again, this time passing the ETag back in the `If-None-Match` header: #### curl ``` # Use the etag value from the previous response, double quotes included curl -i 'http://localhost:9926/JokeCache/1' \ -H 'If-None-Match: "abCDefGHij"' ``` #### fetch ``` // Store the etag from the first request const first = await fetch('http://localhost:9926/JokeCache/1'); const etag = first.headers.get('etag'); // e.g. "abCDefGHij" // Second request using the etag const second = await fetch('http://localhost:9926/JokeCache/1', { headers: { 'If-None-Match': etag }, }); console.log(second.status); // 304 ``` #### curl ``` HTTP/1.1 304 Not Modified etag: "abCDefGHij" ``` #### fetch ``` 304 ``` The response status will be `304 Not Modified` with an empty body. Harper compared the record's current ETag to the one you sent and found them identical — the data hasn't changed, so there's nothing to transfer. This is standard HTTP conditional request behavior. Any HTTP cache layer between your client and Harper — a CDN, a service worker, or a browser cache — can use this same mechanism to avoid redundant data transfers. info The `ETag` / `If-None-Match` pattern is documented in detail in the [REST Headers reference](/reference/v5/rest/headers.md). ## Watching Cache Expiration The `JokeCache` table has a 60-second expiration. After 60 seconds, the cached record becomes stale and the next request will fetch a fresh copy from `jokeAPI`. You can force this behavior immediately by passing the `no-cache` directive in the `Cache-Control` request header, which tells Harper to bypass the local cache and always go to the source: #### curl ``` curl -i 'http://localhost:9926/JokeCache/1' \ -H 'Cache-Control: no-cache' ``` #### fetch ``` const response = await fetch('http://localhost:9926/JokeCache/1', { headers: { 'Cache-Control': 'no-cache' }, }); ``` You will see a `200` response, and if you check the Harper logs you will see an outbound request to `jokeAPI`. ## Invalidating a Cache Entry Sometimes you know the source data has changed and you do not want to wait for the TTL to expire. Harper's Resource API exposes an `invalidate` method that marks a cached record as stale immediately, so it will be reloaded from the source on the next access. First, remove the `@export` directive from the `JokeCache` schema: ``` type JokeCache @table(expiration: 60) { id: ID @primaryKey setup: String punchline: String } ``` Then, create an exported class of the same name in `resources.js` with a custom `POST` handler: ``` export class JokeCache extends tables.JokeCache { static async post(target, data) { const body = await data; if (body?.action === 'invalidate') { this.invalidate(target); return { status: 200, data: { message: 'invalidated' } }; } } } ``` By exporting this class, Harper registers it as the endpoint for `/JokeCache`. The `@export` directive in the schema is no longer required separately because the export is provided by this class. Now you can trigger invalidation with a `POST` request: #### curl ``` curl -X POST 'http://localhost:9926/JokeCache/1' \ -H 'Content-Type: application/json' \ -d '{"action": "invalidate"}' ``` #### fetch ``` await fetch('http://localhost:9926/JokeCache/1', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ action: 'invalidate' }), }); ``` The next `GET /JokeCache/1` will trigger a fresh fetch from `jokeAPI` regardless of whether the TTL has expired. note If you need to check your work, checkout the [`02-invalidate-example`](https://github.com/HarperFast/caching-guide-example/tree/02-invalidate-example) branch. ## Putting It All Together Here is the complete `resources.js` for this guide: ``` // resources.js const jokeAPI = { async get(id) { const response = await fetch(`https://official-joke-api.appspot.com/jokes/${id}`); return response.json(); }, }; tables.JokeCache.sourcedFrom(jokeAPI); export class JokeCache extends tables.JokeCache { static async post(target, data) { const body = await data; if (body?.action === 'invalidate') { this.invalidate(target); return { status: 200, data: { message: 'invalidated' } }; } } } ``` And the complete `schema.graphql`: ``` type JokeCache @table(expiration: 60) { id: ID @primaryKey setup: String punchline: String } ``` ## What Comes Next This guide covered the passive caching pattern: Harper fetches from the source on demand and serves the cached copy until the TTL expires. The next guide, [Caching AI Generations with Harper](/learn/developers/caching-ai-generations.md), applies these same techniques to a real-world problem — caching expensive AI-generated content so that you don't pay for the same generation twice. ## Additional Resources * [Database Schema](/reference/v5/database/schema.md) — `@table` directive and `expiration` argument * [Resource API](/reference/v5/resources/resource-api.md) — `sourcedFrom`, `invalidate`, static and instance methods * [REST Headers](/reference/v5/rest/headers.md) — ETag and `If-None-Match` conditional requests * [REST Overview](/reference/v5/rest/overview.md) — HTTP methods and URL structure * [react-ssr-example](https://github.com/HarperFast/react-ssr-example) — A full example using `sourcedFrom` to cache server-rendered HTML pages --- # Harper Applications in Depth In the Getting Started guides you successfully installed Harper and created your first application. You experienced Harper's schema and database system, and the automatic REST API feature too. This guide dives deeper into Harper's component architecture differentiating applications from plugins, and introduces multiple new ways to interact with Harper. By the end of this guide you will be a confident Harper application developer capable of creating just about anything with Harper! ## What You Will Learn * The fundamental concepts and architecture of Harper applications * How applications and plugins work together in the Harper ecosystem * The distinction between applications and plugins * The structure of the Harper installation * How to interact with Harper using the Operations API and CLI * Introduction to the Harper Resource API for custom endpoint development * Essential debugging techniques for Harper application development ## Prerequisites * Completed [Install and Connect Harper](/learn/getting-started/install-and-connect-harper.md) * Completed [Create Your First Application](/learn/getting-started/create-your-first-application.md) * Working Harper installation (local or Fabric) ## The Harper Stack in more Detail In the previous guide we introduced a high-level Harper architecture diagram: ``` ┏━━━━━━━━━━━━━━━━━━┓ ┃ Applications ┃ ┠──────────────────┨ ┃ Plugins ┃ ┃ - rest ┃ ┃ - graphqlSchema ┃ ┃ - ... ┃ ┠──────────────────┨ ┃ Core Services: ┃ ┃ - database ┃ ┃ - networking ┃ ┃ - component ┃ ┃ management ┃ ┗━━━━━━━━━━━━━━━━━━┛ ``` And defined some key Harper concepts: **Components** are extensions of the core Harper systems, and are further classified as **plugins** and **applications**. **Plugins** have access to APIs exposing many of Harper's core services, and are capable of implementing more advanced features than what the core services provide. **Applications** use plugins to implement user-facing functionality and business logic, such as implementing database schemas and creating web applications. The most important thing to remember is that plugins enable the functionality and applications implement it. Similar to that of a front-end framework. React is like a plugin; on its own it doesn't actually do anything. You actually need to build an application with React for it do anything meaningful. Harper itself is a Node.js application. It runs in a single process, and uses worker threads for parallelization. Harper is meant to be a long-running process. Plugins run exclusively on worker threads. Some of Harper's core services, such as the database and the networking socket router, are implemented directly within the main process. However, a majority of Harper's core functionality is implemented as built-in plugins. Some application changes require restarting the Harper in order to take affect. As you've experienced already, the `harper dev` command automatically restarts threads on detected file changes. Plugins are sometimes capable to dynamically responding to application changes (without a restart), but not all of Harper's built-in or custom plugins take full advantage of that API capability yet. If you are not using the harper `dev` command, make sure to restart Harper when updating applications. Later in this guide you will learn other methods for restarting Harper beyond just the `dev` command. Beyond the component system, Harper also includes some other important subsystems. Harper includes the Operations API and CLI for system administration purposes. Furthermore, Harper exposes a number of code-based APIs such as the Resource API and other useful globals such as `logger` as the core building blocks for implementing custom application functionality. You'll learn about all of these throughout this guide. ### Component Classification: Built-in vs Custom Harper further classifies components (plugins and applications) as either built-in or custom. **Built-in** components are internal to Harper, require no additional installation steps, and are immediately accessible for use. The `graphqlSchema` and `rest` plugins are great examples of built-in plugins. **Custom** components are external to Harper, generally available as an npm package or git repository, and do require additional installation steps in order to be used. Custom components can be authored by anyone, including Harper. Any of Harper's official custom components are published using the `@harperdb` and `@harperfast` package scopes, such as the [`@harperdb/nextjs`](https://github.com/HarperDB/nextjs) plugin for developing Next.js applications or the [`@harperdb/status-check`](https://github.com/HarperDB/status-check) application. Harper's reference documentation contains detailed documentation for all [built-in components](/reference/v5/components/overview.md#built-in-extensions-reference). Custom components are documented within their respective repositories. Harper does not currently include any built-in applications, making "custom applications" a bit redundant. Generally, we just refer to them as "applications". However, there is a multitude of both built-in and custom plugins, and so the documentation tends to specify whenever relevant. ### Harper Installation File Structure One of the founding principles of Harper is its simplicity. When you install Harper locally or are using a Fabric instance, we wanted it to be effortless to introspect your installation and understand how it works. Refer back to the previous installation guide and determine what path you installed Harper to. For many users, this path is likely within your home directory. Local, container installs likely mounted to a similar path. Fabric users should follow along using the browser. Within every Harper installation are these core files and directories: ``` ~/hdb ┠─ backup/ ┠─ components/ ┠─ database/ ┠─ keys/ ┠─ log/ ┠─ harper-application-lock.json ┠─ harper-config.yaml ┠─ hdb.pid ┠─ operations-server ┗━ README.md ``` Some of these files are runtime-only such as `hdb.pid` and `operations-server`. The `README.md` contains some relevant information about the installation files and directories, as well as additional links to the documentation site. The directories themselves are fairly self-explanatory: * `backup/` is for backups of the configuration file * `components/` is where your application code goes (when you deploy it) * `database/` is the database files * `keys/` is any security keys for the purpose of authentication; more on this in a future guide * `log/` is where log files are stored The `harper-application-lock.json` file is similar to any sort of lockfile. Its purpose is to ensure Harper is installing the correct versions of applications and plugins. This is an internal file and you shouldn't ever have to modify it yourself. Finally, and most importantly is the `harper-config.yaml`. This is the main configuration file for your Harper installation. Lets open this file and inspect its contents. Local users should open it in their editor of choice, Fabric users should navigate to the "Config" tab from their main organization view. You should see a number of top-level properties such as `http`, `threads`, `authentication`, and more. Each of these corresponds to one of Harper's built-in core features. This file is the source of truth for Harper configuration values. You can make changes directly to the file and then restart Harper for them to take affect. ## Working with the Operations API In the first guide, we introduced you to the `/health` endpoint. This is provided by Harper's built-in Operations API. The **Operations API** provides a full set of capabilities for configuring, deploying, administering, and controlling Harper. It is configured on port `9925` by default, and primarily functions through JSON-based, POST requests to the root path `/`. It has some additional functionalities too such as the `/health` endpoint and an OpenAPI endpoint `/api/openapi/rest`. The operations API root path POST requests must be authenticated. Harper provides an `authentication.authorizeLocal` configuration option for automatically authorizing any requests from the loopback IP address as the superuser (the one created during Harper installation). This option is enabled automatically when Harper is installed using the `dev` default config (as was instructed in the getting started guide). Thus, local installation users may make unauthenticated requests. Container based installation users must use `--network host` when running the container in order to make use of this option. Fabric or any other remote host installations generally must authenticate all requests. note The `authentication.authorizeLocal` option should be disabled for any Harper servers that may be accessed by untrusted users from the same instance. For example, it should be disabled if you are using a local proxy, or for general server hardening. This and future learn guides omit `Authorization` headers from any request examples. The assumption is that all local installation readers have `authorizeLocal` enabled, local container installation users are running the container with a shared network host, and Fabric users are using the UI or an authenticated HTTP client to make requests. Nonetheless, we've included the following section on how to setup Basic Authentication in case it is necessary. Most readers may skip ahead to the [First Operation API Request](#first-operations-api-request) section. Basic Authentication The simplest authorization scheme is [Basic Authentication](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Authentication#basic_authentication_scheme) which transmits credentials as username/password pairs encoded using base64. Importantly, this scheme does not encrypt credentials. If used over an insecure connection, such as HTTP, they are susceptible to being compromised. Only ever use Basic Authentication over secured connections, such as HTTPS. Even then, its better to upgrade to an encryption based authentication scheme or certificates. Harper supports many different authentication mechanisms, and they will all be covered in later Learn guides. Use the username and password values from the previous [Install and Connect](/learn/getting-started/install-and-connect-harper.md) guide to generate an authorization value. The important part is to combine the `username` and `password` using a colon `:` character, encode that using base64, and then append the result to `"Basic "`. ``` const username = 'HDB_ADMIN'; const password = 'abc123!'; const authorizationValue = `Basic ${btoa(`${username}:${password}`)}`; ``` You would use the `authorizationValue` as the value for the `Authorization` header such as: ``` fetch('/', { // ... headers: { Authorization: authorizationValue, }, // ... }); ``` ### First Operations API Request There are many great operations to chose from, but to get started, lets try the `get_status` operation. note All `operation` values will be in `snake_case`; all lowercase and underscores in-place of spaces. First, ensure Harper is running (refer to the previous guide if you need a quick refresher). Then, using your HTTP client of choice, create a POST request to your running Harper instance with `Content-Type: application/json` header, and a JSON body containing `{ "operation": "get_status" }`. note Fabric users, remember to replace `http://localhost` with your Fabric instance URL, and include an authorization header. #### curl ``` curl -s 'http://localhost:9925/' \ -X POST \ -H "Content-Type: application/json" \ -d '{ "operation": "get_status" }' \ | jq ``` #### fetch ``` const response = await fetch('http://localhost:9925/', { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify({ operation: 'get_status', }), }); const data = await response.json(); console.log(data); ``` This operation returns a JSON object with three top-level properties: `restartRequired`, `systemStatus`, and `componentStatus`. ``` { "systemStatus": [ // ... ], "componentStatus": [ { "name": "http", "componentName": "http", "status": "healthy", "lastChecked": { "workers": { "0": 1770141380945 }, "main": 1770141380484 } } // ... ], "restartRequired": false } ``` The `restartRequired` property is a mechanism for Harper plugins to indicate they require a restart for some changes to take effect. The other two properties are lists containing status objects corresponding to different parts of Harper. These should all read `"status": "healthy"` right now, and you may recognize some of the `"name"` and `"componentName"` fields as they correspond to Harper's built-in subsystems (such as `"http"`, `"threads"`, and `"authentication"`). ### More with Operations API The Operations API is mainly intended to be used for system administration purposes. This API runs on a separate port than the main application port serving user traffic, providing a distinct interface for clear differentiation between secure system administration and the application interface designed for high-load, performance and application defined actions. (It does have the ability to do data management, which may overlap with application capabilities, but this is part of a full system administration API). Harper keeps a [reference of all operations](/reference/v5/operations-api/overview.md) in the Operations API reference documentation, but here a few more you can try immediately: `user_info`, `read_log`, and `describe_all`. For `describe_all` to work, ensure that you are still running the Harper application you created in the previous guide. If you need to, checkout the [`02-rest-api`](https://github.com/HarperFast/create-your-first-application/tree/02-rest-api) branch of the `HarperFast/create-your-first-application` repository to ensure you have the necessary application files for this example. You should see a JSON object with a top-level property `"data"`. This operation returns a map of all databases and tables. The `"data"` is the default database in Harper. Within that object, there should be a `"Dog"` key. This is the table you defined with `graphqlSchema` in the previous guide. The entire JSON response should look something like this: ``` { "data": { "Dog": { "schema": "data", "name": "Dog", "hash_attribute": "id", "audit": true, "schema_defined": true, "attributes": [ { "attribute": "id", "type": "ID", "is_primary_key": true }, { "attribute": "name", "type": "String" }, { "attribute": "breed", "type": "String" }, { "attribute": "age", "type": "Int" } ], "db_size": 212992, "sources": [], "record_count": 1, "table_size": 16384, "db_audit_size": 16384 } } } ``` Now lets keep drilling down in specificity by using the `describe_database` and then the `describe_table` operations. The difference this time is that these operations require additional properties. For `describe_database`, you can specify `"database": "data"`. The entire request body would look something like this: ``` { "operation": "describe_database", "database": "data" } ``` The response this time should omit the top-level `"data"` key, and instead be just an object containing `"Dog"` (the singular table defined in the `data` database so far). And for `describe_table`, you would specify both `"database": "data"` and `"table": "Dog"`, ``` { "operation": "describe_database", "database": "data", "table": "Dog" } ``` Now there is yet another way to get information about the `Dog` table; with the REST interface! Create a `GET` request to `http://localhost:9926/Dog` and its important that you omit any trailing forward slash `/`, this request should return a slightly different JSON object describing the `Dog` table. #### curl ``` curl -s 'http://localhost:9926/Dog' | jq ``` #### fetch ``` const response = await fetch('http://localhost:9926/Dog'); const data = await response.json(); console.log(data); ``` Expected result: ``` { "records": "./", "name": "Dog", "database": "data", "auditSize": 3, "attributes": [ { "type": "ID", "name": "id", "isPrimaryKey": true, "attribute": "id" }, { "type": "String", "name": "name", "attribute": "name" }, { "type": "String", "name": "breed", "attribute": "breed" }, { "type": "Int", "name": "age", "attribute": "age" } ] } ``` All in all, the Operations API is a fundamental tool for managing and introspecting your Harper instance. We'll cover more operations throughout the Learn guides. ## The Harper CLI So far you've only used the Harper CLI to run Harper itself, but it can do so much more than that! In previous guides we demonstrated how to use the `harper` and `harper dev` commands to run Harper; with the later automatically restarting threads on application changes. There are a few more ways to manage a Harper instance using the CLI. * `harper run ` is an alias for the default `harper` command. These commands run Harper in the current process * `harper start` will start Harper in a background process * `harper stop` command will gracefully shutdown Harper * `harper restart` will restart the main process and all threads (different than the thread-only restart from the `dev` command) * `harper status` displays the status of the process including the PID There are a few more commands not listed here (check out the [CLI reference](/reference/v5/cli/overview.md) if you're interested), and there is one more fun trick with the CLI. Certain operations from the Operations API are available as CLI commands! They follow the convention: `harper =`, and return YAML by default. You can always pass `json=true` to see the result in JSON instead. We'll dive deeper in the CLI operations later for the purpose of deploying and managing your application, but for now, try out some of the operations you've already learned. Don't forget that you can append `json=true` and `| jq` to get nicely formatted JSON output. ``` harper get_status harper describe_all harper describe_database database=data harper describe_table database=data table=Dog ``` ## Expanding your Harper Application with custom Resources If you're following along from getting started, you should have a basic Harper application running containing a `schema.graphql` and `config.yaml` files defining a simple `Dog` table and REST endpoint. Lets expand on this example while also exploring more of Harper's lifecycle and application development capabilities. note If you want to ensure you're application code is at the right starting point, checkout the [`02-rest-api`](https://github.com/HarperFast/create-your-first-application/tree/02-rest-api) branch of the `HarperFast/create-your-first-application` repository. Create a new file `resources.js` within your Harper application; here we are going to define custom Resources. **Resources** are the mechanism for defining custom functionality in your Harper application. This gives you tremendous flexibility and control over how data is accessed and modified in Harper. The corresponding Resource API is a unified API for modeling different data sources within Harper as JavaScript classes. Generally, this is where the core business logic of your application lives. Database tables (the ones defined by `graphqlSchema` entries) are `Resource` classes, and so extending the function of a table is as simple as extending their class. Resource classes have methods that correspond to standard HTTP/REST methods, like `get`, `post`, `patch`, and `put` to implement specific handling for any of these methods (for tables they all have default implementations). Furthermore, by simply `export` 'ing a resource class, Harper will generate REST API endpoints for it just like the `@export` directive did in `graphqlSchema`. The [Resource API](/reference/v5/resources/overview.md) is quite powerful, and we'll dive into different aspects throughout future Learn guides, but for now lets start with a simple example extending the existing `Dog` table that already exists in the application. Inside of `resources.js` add the following code for defining a `DogWithHumanAge` custom resource: ``` // Fun fact, the 7:1 ratio is a misconception // https://www.akc.org/expert-advice/health/how-to-calculate-dog-years-to-human-years/ function calculateHumanAge(dogAge) { if (dogAge === 1) { return 15; } else if (dogAge === 2) { return 24; } else { return 24 + 5 * (dogAge - 2); } } export class DogWithHumanAge extends tables.Dog { static async get(target) { const dogRecord = await super.get(target); return { ...dogRecord, humanAge: calculateHumanAge(dogRecord.age), }; } } ``` Then open `config.yaml` and add the `jsResource` plugin: ``` # Harper application configuration graphqlSchema: files: 'schema.graphql' jsResource: files: 'resources.js' rest: true ``` Ensure Harper has restarted (automatically in `dev` mode or by manually starting/stopping it), and then prepare to query the new resource endpoint. Just like with the `Dog` table, the automatically generated endpoint matches the name of the exported class, in this case `DogWithHumanAge/`. In the getting started guide we created a singular dog record with an id of `001`. Create a `GET` request to `/DogWithHumanAge/001` and display the resulting JSON: #### curl ``` curl -s 'http://localhost:9926/DogWithHumanAge/001' | jq ``` #### fetch ``` const response = await fetch('http://localhost:9926/DogWithHumanAge/001'); const dog = await response.json(); console.log(dog); ``` The resulting JSON object should look similar to the original `Dog/001` entry, except this time there is a new property `humanAge`. ``` { "name": "Harper", "breed": "Black Labrador / Chow Mix", "age": 5, "id": "001", "humanAge": 39 } ``` Notably, did you see how we were able to use the `001` id with the new resource immediately? And it was able to derive the underlying `Dog` record? Lets take a closer look at the custom resource implementation to understand how this works: ``` export class DogWithHumanAge extends tables.Dog { static async get(target) { // ... } } ``` The `DogWithHumanAge` class extends from `tables.Dog`. The `tables` reference is a global added by Harper that is a map of all tables, such as the ones defined by `graphqlSchema`. A table class represents the collection of all the records in the table. Its an interface for querying and accessing records from the table and even creating/updating records too. The purpose of the `static loadAsInstance = false;` line is to ensure that we can `get` a direct enumerable record of the dog (rather than an class instance referencing the record). The `DogWithHumanAge` class extends the functionality of `tables.Dog`. The `export` keyword instructs Harper to automatically generate a REST API endpoint for the custom resource using the same name (`DogWithHumanAge/`). ``` export class DogWithHumanAge extends tables.Dog { static async get(target) { const dogRecord = await super.get(target); return { ...dogRecord, humanAge: calculateHumanAge(dogRecord.age), }; } } ``` As we mentioned before, the `tables.Dog` class represents the entire collection of records in that table. Thus, the `get()` method uses the `super` keyword to reference the `tables.Dog` class its extended from in order to retrieve the `target` record. By passing through the `target` object to `super.get(target)`, we are querying the original `Dog` table defined in `graphqlSchema`. The `dogRecord` instance corresponds to whatever the request `/` portion specified. The rest of the `get()` method returns a new object with a copy of `dogRecord` and a newly computed `humanAge` field. The `dogRecord` isn't just a plain JSON object; it has its own set of methods including comprehensive getters and setters. However, Harper does make all the defined properties available as enumerable properties so by using the `...` spread operator, you can easily copy all of the relevant properties of the `dogRecord` instance. Now if you perhaps tried to use a query string selector, like `GET /DogWithHumanAge/?age=5`, this `get()` method implementation can't quite handle it yet; but this will be covered soon! For now, celebrate that you've successfully implemented your first custom resource! note If you need to check your work, checkout the [`03-custom-resource`](https://github.com/HarperFast/create-your-first-application/tree/03-custom-resource) branch. ## Debugging Harper Applications Now that your Harper application is actually executing some custom logic; lets learn how to efficiently debug your code. Harper provides the ability to launch a proper debugger as part of the Harper process; which you can connect to from any debugger tool such as your browser or IDE. This will let you properly debug your application code, particularly the custom code you implemented in `resources.js` with `jsResource` plugin. note Harper v4 ships as a built and minified package, so debugging the Harper source may be confusing; you generally will only want to debug your custom code. However, in the near future, as Harper v5 is released using our new open source core, you will be able to debug Harper's core too! Before getting started, either open the `harper-config.yaml` file or use the Operations API to inspect how Harper is currently configured (using the `get_configuration` operation). We are looking for the `"threads"` part of the configuration object in particular. #### CLI ``` harper get_configuration json=true | jq .'threads' ``` #### curl ``` curl -s 'http://localhost:9925/' \ -X POST \ -H "Content-Type: application/json" \ -d '{ "operation": "get_configuration" }' | jq .'threads' ``` #### fetch ``` const response = await fetch('http://localhost:9925/', { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify({ operation: 'get_configuration', }), }); const data = await response.json(); console.log(data.threads); ``` The result should contain two properties, `count` and `debug`. With the default development config, these configuration properties should have the values: ``` { "count": 1, "debug": true } ``` If your values aren't the same, modify them so that `debug: true` is set. This as another great opportunity to try out another operation, `set_configuration`. You'll need to specify the nested property as snake\_case, so to set `debug: true`, you would specify: `harper set_configuration threads_debug=true`, or in JSON `{ "operation": "set_configuration", "threads_debug": true }`. Don't forget to restart Harper after making configuration changes. If you look closely at your Harper output, you should see a line `Debugger listening on` followed by a WebSocket URL (starts with `ws://`). Using a debugger of your choice, attach to this debug process. Now set a breakpoint somewhere in `resources.js` such as the last return statement of `calculateHumanAge()`. Then run the `GET /DogWithHumanAge/001` query from earlier and watch as your debugger breaks in your custom code! Don't forget to *continue* the process using the debugger to let the request complete. note When using the debugger, and breaking on application code for too long, particularly in a custom resource, you may see log lines such as: ``` [http/1] [warn]: JavaScript execution has taken too long and is not allowing proper event queue cycling, consider using 'await new Promise(setImmediate)' in code that will execute for a long duration ``` You can disconnect the debugger and resume using Harper as usual. ## Logging with Harper Logger Harper comes with a built-in logger. It is what powers the `harper` CLI output and throughout building your first application and trying some operations, you've likely seen many additional lines in your terminal output. You may have even tried the `read_log` operation too. The logger is a very fundamental piece to Harper applications. However, since this is *just JavaScript* you can use the `Console` API (`console.log()`) too! Harper's logger is available as a global `logger` API. It has methods for each log level, and some additional utilities too. The available log levels (in hierarchical order) are: 1. trace 2. debug 3. info 4. warn 5. error 6. fatal 7. notify The main logger is configured at the top-level of the configuration object. Try using the operations API to view your instance's current logging configuration. The `logger` configuration object has a lot to it, but the most important fields for application *development* (don't worry, we'll discuss running applications in production in another guide) are `level`, `console`, and `stdStreams`. For example, `level` is defaulted to `info` in `dev` mode. This means that all logs from `info` to `notify` levels will be created by default. If you want to see additional levels, change the log level in the configuration. The best way to conceptualize Harper's logger is that its primary function is to output structure logs to specified log files. By default this is `/logs/hdb.log`. Now the `stdStreams` configuration option is what instructs the logger to *also* log Harper logs to the standard output and error streams (aka `stdout` and `stderr`). Furthermore, the `console` option instructs Harper to forward `Console` API logs (aka `console.log()`) to the log file. Ensure that both of these settings are enabled. Lets quickly practice creating some logs in the custom resources code. Add a `logger.info()` and a `console.log()` anywhere in `resources.js`, such as within the `get()` method: ``` // ... export class DogWithHumanAge extends tables.Dog { static async get(target) { logger.info('Hello from inside DogWithHumanAge!'); const dogRecord = await super.get(target); console.log('dogRecord', dogRecord); // ... } } ``` Ensure Harper restarts and then execute the `GET /DogWithHumanAge/001` query again. You should see: ``` [http/1] [info]: Hello from inside DogWithHumanAge! dogRecord RecordObject { name: 'Harper', breed: 'Black Labrador / Chow Mix', age: 5, id: '001' } ``` As you can see, the `logger.info()` message has the structured `[http/1] [info]:` piece, and the `console.log()` does not. We'll cover logs more in depth in a later guide about running your Harper app in production. note If you need to check your work, checkout the [`04-logger`](https://github.com/HarperFast/create-your-first-application/tree/04-logger) branch. ## What You've Accomplished This guide started off with a deep dive into Harper's component architecture and differentiating applications from plugins. You learned about Harper's file structure, how to use Harper's Operations API for system management, and learned some new CLI commands too. You got your first taste of Harper's Resource API and implemented your first custom endpoint. We'll be using the Resource API a lot more throughout later guides. Finally, you learned how to use a debugger with your Harper application and how to use Harper's built-in logger too. At this point, you should confident to start tinkering with your own ideas for a Harper application. In the next guide we'll be exploring more of Harper's Resource API, exploring more schema directives, and diving deeper into what Harper applications are really capable of. ## Additional Resources * [Operations API](/reference/v5/operations-api/overview.md) * [`logger` global reference](/reference/v5/logging/api.md) * [Resources](/reference/v5/resources/overview.md) * [Components](/reference/v5/components/overview.md) --- # Writing quality MCP and OpenAPI descriptions When an MCP client connects to Harper, the LLM on the other side sees your application as a list of tools. The text it reads to pick the right tool — the tool description, the per-attribute property descriptions, the output schema shape — is the dominant signal for tool selection. The same metadata also drives Harper's OpenAPI document, which any HTTP API consumer (Swagger UI, Redoc, generated SDKs, machine clients) reads. This guide shows how to author that metadata once and have it flow to both surfaces — via GraphQL docstrings for `@table @export` Resources, and via class-level statics for programmatic Resource subclasses. ## Why descriptions matter Harper auto-generates MCP tools for every exported Resource. Without descriptions, every tool gets a generic template: `"get on resource '/Product' (table Product). Runtime RBAC enforces per-record access at call time."` An LLM picking between `get_Product`, `get_Order`, `get_Customer` sees three near-identical descriptions. Tool selection becomes guesswork. Add a one-line docstring to your `@table @export` type and the picture changes: each tool's description includes a sentence about what the resource actually represents, and every searchable attribute has a per-attribute description the LLM can use to form queries. ## Path A: `@table @export` Resources via GraphQL docstrings For table-backed Resources, the natural authoring locus is the GraphQL schema. Triple-quoted docstrings on types and fields are picked up by Harper's parser and flow through to both MCP and OpenAPI automatically — no JavaScript code changes required. ### Before ``` type Product @table @export { sku: String! @primaryKey name: String! priceCents: Int! inStock: Int! } ``` MCP `tools/list` returns: ``` { "name": "get_Product", "description": "get on resource '/Product' (table Product). Runtime RBAC (allowGet) enforces per-record access at call time.", "inputSchema": { "type": "object", "properties": { "id": { "type": "string", "description": "Primary key (sku)." } }, "required": ["id"] } } ``` ### After ``` """ Product catalog row — what shows up in the storefront listing, search, and inventory feeds. One row per SKU. """ type Product @table @export { """ Stock keeping unit — globally unique across catalogs. """ sku: String! @primaryKey """ Display name shown in the storefront. 100 chars max. """ name: String! """ Retail price in cents (USD). """ priceCents: Int! """ Current inventory level. Decremented by orders; reconciled nightly. """ inStock: Int! } ``` MCP `tools/list` now returns: ``` { "name": "get_Product", "description": "Product catalog row — what shows up in the storefront listing, search, and inventory feeds. One row per SKU.\n\nFetches a single Product record by sku. Runtime RBAC (allowGet) enforces per-record access at call time.", "inputSchema": { "type": "object", "properties": { "id": { "type": "string", "description": "Primary key (sku)." } }, "required": ["id"] }, "outputSchema": { "type": "object", "properties": { "sku": { "type": "string", "description": "Stock keeping unit — globally unique across catalogs." }, "name": { "type": "string", "description": "Display name shown in the storefront. 100 chars max." }, "priceCents": { "type": "integer", "description": "Retail price in cents (USD)." }, "inStock": { "type": "integer", "description": "Current inventory level. Decremented by orders; reconciled nightly." } }, "required": ["sku", "name", "priceCents", "inStock"], "additionalProperties": false } } ``` And `/openapi.json` picks up the same data: schema-level `description`, per-property `description`, and prepended path-level descriptions for every verb on `/Product`. ### `search_*` gets typed and described too For `search_Product`, the `conditions[].attribute` field becomes a closed `enum` of the readable attributes, and each per-property description threads through. The LLM goes from "an attribute name (string)" to "one of these specific attribute names, with this meaning each." ### Authoring rubric * **Lead with a verb-led sentence on the type:** "Product catalog row…", "Customer profile and order history…". Skip the trivia ("This is the Product table"); the LLM already knows it's a table. * **Field docstrings should explain meaning, not type.** Saying "Integer." adds nothing — the schema already says `Int!`. Saying "Retail price in cents (USD)" lets the LLM construct sensible queries. * **Mention units, formats, and edge cases.** "ISO 8601 timestamp", "cents not dollars", "null for SKUs that have never been counted". * **Keep docstrings short.** Long descriptions waste LLM context and clutter the OpenAPI UI. ## Path B: Programmatic Resources via class-level statics For Resources without `@table @export` backing — Resource subclasses that override `get`/`post`/`put`/`delete` directly, or that aggregate across multiple tables — there's no GraphQL schema to derive from. Declare the same metadata directly on the class as JSON-Schema-shaped statics. The MCP and OpenAPI layers read both surfaces uniformly. ``` import { Resource } from 'harperdb'; export class ProductInventory extends Resource { static description = 'Aggregate inventory analytics computed over the Product catalog. ' + 'Read-only; the underlying Product table is the system of record.'; static properties = { sku: { type: 'string', primaryKey: true, description: 'Stock keeping unit; matches Product.sku.' }, onHand: { type: 'integer', description: 'Current warehouse count.' }, reserved: { type: 'integer', description: 'Units allocated to open orders but not yet shipped.' }, stockStatus: { type: 'string', enum: ['in_stock', 'out_of_stock', 'backorder'], description: 'Derived from onHand vs reserved.', }, }; async get(id) { /* returns { sku, onHand, reserved, stockStatus } */ } async search(query) { /* ... */ } } ``` See the [Resource API reference](/reference/v5/resources/resource-api.md#class-level-metadata-for-mcp-and-openapi) for the full surface, including `static outputSchemas` for per-verb projection overrides, `static hidden` for full suppression, and `static mcp` for narrow MCP-only annotation overrides. ## Inheritance: extending a table Resources extending a `@table @export` Resource inherit the auto-derived metadata. Override individual entries with spread: ``` const { Product } = tables; class CustomProduct extends Product { static properties = { ...Product.properties, priceCents: { ...Product.properties.priceCents, description: 'Retail price in cents, including any per-customer adjustments.', }, }; } ``` The author writes against the canonical `properties` API. Internal code paths that need ordered iteration continue to read `Class.attributes` (the Array form), preserved through inheritance. ## Hiding sensitive fields with `@hidden` OpenAPI is typically exposed to anyone reachable on the HTTP port — there's no per-user filtering on `/openapi.json`. A docstring on a sensitive field publishes that text to anyone who can hit the endpoint. The `@hidden` directive suppresses a field (or an entire type) from both MCP and OpenAPI without affecting data access: ``` type Customer @table @export { id: Long @primaryKey name: String """ Internal — used by the pricing engine; not for external consumers. """ creditScore: Int @hidden } ``` `creditScore` is still queryable via direct Harper interfaces under the caller's `attribute_permissions` — `@hidden` is a metadata-visibility directive, not access control. For programmatic Resources, the equivalent is `static hidden = true` on the class (or `hidden: true` on a per-property entry in `static properties`). > **Trust model.** Docstrings reach LLMs and public OpenAPI consumers verbatim. Treat them as code: don't put secrets, internal-only commentary, or speculative prose in them. Use `@hidden` to suppress fields that shouldn't surface publicly. ## RBAC and per-user filtering For MCP tool descriptors, `attribute_permissions` already filters the schema per-user — an attribute the caller cannot read is dropped from that user's view of the tool descriptor, along with its description. The new metadata flows through the existing pipeline. For OpenAPI, the document is global and not per-user filtered. Use `@hidden` (or `static hidden`) to control what surfaces there. ## Verifying the end-to-end flow 1. Add `"""docstrings"""` to a `@table @export` type and save your component. 2. Hit MCP `tools/list` for the application profile — confirm `get_*`, `search_*`, etc. descriptions include the type docstring and per-attribute descriptions are present in the `inputSchema` and `outputSchema`. 3. Hit `/openapi.json` on the application HTTP port — confirm the path-level descriptions and per-property descriptions show up in Swagger UI / Redoc. 4. Add `@hidden` to an attribute — confirm it disappears from both surfaces while remaining queryable via direct REST/SQL. --- # Semantic Caching with Vector Indexing Caching LLM responses by exact prompt text catches almost nothing useful. "What shoes are good for hiking?" and "Which shoes work best for hiking?" are semantically identical but would be cache misses for each other with a string key. The result is that you pay for every generation even when a good answer already exists in the cache. Semantic caching solves this by storing a vector embedding alongside each cached response. When a new question arrives, Harper computes its embedding, searches the cache for a response that is *close enough* in meaning, and returns it if the similarity distance is below a threshold. Only questions with no sufficiently similar answer in the cache hit the LLM. In this guide you will build a product assistant that answers customer questions using OpenAI. Semantically equivalent questions share a single cached answer — you only pay for a generation once, regardless of how the question is phrased. ## What You Will Learn * How to store vector embeddings alongside text in a Harper table * How to define a vector index using `@indexed(type: "HNSW")` * How to query by cosine similarity using `table.search({ sort: { attribute, target } })` * How to implement a semantic cache lookup before calling the LLM * How to set a similarity/distance threshold to control cache hit quality ## Prerequisites * Completed [Caching with Harper](/learn/developers/caching-with-harper.md) * An OpenAI API key (`OPENAI_API_KEY` environment variable) * Familiarity with the concept of embeddings (vectors of floats representing meaning) ## The Architecture This guide uses a deliberate architecture that separates concerns cleanly: ``` Client → /QuestionAnswer/ ↓ Harper checks semantic cache ↓ near miss? ↓ no similar cached answer? Return cached answer Call LLM → store answer + embedding ``` The cache is keyed by a content-addressed ID (hash of the normalized question text). On each request, Harper: 1. Embeds the incoming question 2. Searches the HNSW index for any cached answer with cosine similarity above the threshold 3. If found, returns the cached answer immediately 4. If not, generates a new answer, stores it with its embedding, and returns it Subsequent questions that are phrased differently but mean the same thing will land within the similarity threshold and return the cached answer — no LLM call needed. ## Defining the Schema Open `schema.graphql`: ``` type QuestionAnswer @table(expiration: 604800) @export { id: ID @primaryKey # SHA-256 of normalized question text question: String answer: String embedding: [Float] @indexed(type: "HNSW", distance: "cosine") generatedAt: Long } ``` The key field is `embedding: [Float] @indexed(type: "HNSW", distance: "cosine")`. This creates an HNSW vector index on the embedding vectors, enabling approximate nearest-neighbor search by cosine similarity. `expiration: 604800` sets a 7-day TTL. LLM answers are not infinitely fresh — product details change, pricing shifts — so a week is a reasonable window. After 7 days the record is evicted and the next identical question generates a fresh answer. ## Configuring the Application Open `config.yaml`: ``` graphqlSchema: files: 'schema.graphql' rest: true jsResource: files: 'resources.js' ``` ## Building the Semantic Cache Resource The core logic lives in `resources.js`. The `ProductAssistant` class overrides `get` to implement the semantic cache lookup and generation pipeline. ``` // resources.js const OPENAI_API_KEY = process.env.OPENAI_API_KEY; const SIMILARITY_THRESHOLD = 0.92; // cosine similarity; tune for your use case // --- Embedding helper --- async function embed(text) { const response = await fetch('https://api.openai.com/v1/embeddings', { method: 'POST', headers: { 'Content-Type': 'application/json', 'Authorization': `Bearer ${OPENAI_API_KEY}`, }, body: JSON.stringify({ model: 'text-embedding-3-small', input: text, }), }); const result = await response.json(); return result.data[0].embedding; // [Float] array } // --- Answer generation helper --- async function generateAnswer(question) { const response = await fetch('https://api.openai.com/v1/chat/completions', { method: 'POST', headers: { 'Content-Type': 'application/json', 'Authorization': `Bearer ${OPENAI_API_KEY}`, }, body: JSON.stringify({ model: 'gpt-4o-mini', messages: [ { role: 'system', content: 'You are a helpful product assistant. Answer customer questions clearly and concisely.', }, { role: 'user', content: question }, ], max_tokens: 200, }), }); const result = await response.json(); return result.choices[0].message.content.trim(); } // --- Content-addressed ID --- async function questionId(text) { const normalized = text.trim().toLowerCase(); const buf = await crypto.subtle.digest('SHA-256', new TextEncoder().encode(normalized)); return Array.from(new Uint8Array(buf)) .map((b) => b.toString(16).padStart(2, '0')) .join('') .slice(0, 16); } // --- Semantic cache resource --- export class QuestionAnswer extends Resource { static async get(target) { const rawQuestion = target.get('q'); if (!rawQuestion) { const error = new Error('Missing required query parameter: q'); error.statusCode = 400; throw error; } const question = rawQuestion.trim(); // 1. Embed the incoming question const queryEmbedding = await embed(question); // 2. Search the HNSW index for the nearest cached answer const results = tables.QuestionAnswer.search({ sort: { attribute: 'embedding', target: queryEmbedding }, limit: 1, select: ['id', 'question', 'answer', 'generatedAt', 'embedding', '$distance'], }); for await (const cached of results) { const similarity = 1 - cached.$distance; if (similarity >= SIMILARITY_THRESHOLD) { // Cache hit — return the stored answer return { answer: cached.answer, cachedQuestion: cached.question, generatedAt: cached.generatedAt, cacheHit: true, similarity: Math.round(similarity * 1000) / 1000, }; } } // 3. Cache miss — generate a new answer const answer = await generateAnswer(question); const id = await questionId(question); await tables.QuestionAnswer.put({ id, question, answer, embedding: queryEmbedding, generatedAt: Date.now(), }); return { answer, question, generatedAt: Date.now(), cacheHit: false, }; } } ``` ### The semantic cache flow The `get` handler implements the full pipeline in sequence: 1. **Embed** the incoming question using OpenAI's `text-embedding-3-small` model. 2. **Search** the HNSW index for the nearest stored embedding. HNSW returns approximate nearest neighbors extremely quickly — even with thousands of cached answers, the search takes microseconds. 3. **Check similarity** against `SIMILARITY_THRESHOLD` (similarity is 1 - distance). A score of `1.0` is a perfect semantic match; `0.92` is a reasonable default for product Q\&A (questions that mean the same thing typically score above `0.95`; genuinely different questions typically score below `0.85`). 4. **Return the cached answer** if above threshold — no LLM call needed. 5. **Generate and cache** a new answer if below threshold, storing the embedding for future lookups. note The similarity threshold is the most important tuning knob. Set it too high and you miss cache hits for slight rephrasing. Set it too low and you return irrelevant cached answers. Start at `0.92` and adjust based on your domain. ## Querying the Assistant With Harper running, ask a question: #### curl ``` curl -s 'http://localhost:9926/QuestionAnswer?q=What+shoes+are+best+for+hiking' ``` #### fetch ``` const response = await fetch( 'http://localhost:9926/QuestionAnswer?' + new URLSearchParams({ q: 'What shoes are best for hiking' }) ); const data = await response.json(); console.log(data); ``` First request — cache miss, LLM called: #### curl ``` { "answer": "For hiking, look for boots with ankle support, a grippy rubber sole...", "question": "what shoes are best for hiking", "generatedAt": 1712500000000, "cacheHit": false } ``` #### fetch ``` { answer: 'For hiking, look for boots with ankle support, a grippy rubber sole...', question: 'what shoes are best for hiking', generatedAt: 1712500000000, cacheHit: false } ``` Now ask a semantically equivalent question phrased differently: #### curl ``` curl -s 'http://localhost:9926/QuestionAnswer?q=Which+footwear+is+recommended+for+trail+hiking' ``` #### fetch ``` const response = await fetch( 'http://localhost:9926/QuestionAnswer?' + new URLSearchParams({ q: 'Which footwear is recommended for trail hiking' }) ); const data = await response.json(); console.log(data); ``` Cache hit — same answer returned instantly: #### curl ``` { "answer": "For hiking, look for boots with ankle support, a grippy rubber sole...", "cachedQuestion": "what shoes are best for hiking", "generatedAt": 1712500000000, "cacheHit": true, "similarity": 0.961 } ``` #### fetch ``` { answer: 'For hiking, look for boots with ankle support, a grippy rubber sole...', cachedQuestion: 'what shoes are best for hiking', generatedAt: 1712500000000, cacheHit: true, similarity: 0.961 } ``` The second question scored `0.961` cosine similarity — above the `0.92` threshold — so it returned the cached answer without calling the LLM. ## Cache Expiration and Freshness The `QuestionAnswer` table has a 7-day TTL (`expiration: 604800`). After 7 days, a record is evicted and the next request for a similar question generates a fresh answer. You can bypass the TTL and force a fresh generation by passing `Cache-Control: no-cache`: #### curl ``` curl -s 'http://localhost:9926/QuestionAnswer?q=What+shoes+are+best+for+hiking' \ -H 'Cache-Control: no-cache' ``` #### fetch ``` const response = await fetch( 'http://localhost:9926/QuestionAnswer?' + new URLSearchParams({ q: 'What shoes are best for hiking' }), { headers: { 'Cache-Control': 'no-cache' } } ); ``` ## Going Further * **Domain-specific system prompts**: pass product catalog context in the system prompt so answers are grounded in your actual inventory. * **Fine-tuning the threshold**: log `similarity` values for hits and misses to find the ideal threshold for your query distribution. * **Multi-table semantic caches**: maintain separate caches for different question domains (support, sales, returns) with different system prompts and TTLs. * **Embedding model selection**: `text-embedding-3-small` is fast and cheap; `text-embedding-3-large` offers higher accuracy for ambiguous queries. ## Additional Resources * [Caching with Harper](/learn/developers/caching-with-harper.md) — foundational passive caching guide * [Database Schema](/reference/v5/database/schema.md) — `@indexed(type: "HNSW")` vector index configuration and parameters * [Resource API](/reference/v5/resources/resource-api.md) — `search`, `sort`, Query object --- # Write-Through Caching Read-through caches — where Harper fetches from the source on cache misses — only cover half the story. When clients write data, those writes still go directly to the origin. The cache only learns about the change the next time the TTL expires or someone calls `invalidate`. Write-through caching closes this loop. Writes to a Harper cache table are forwarded to the origin source before being committed locally. The cache stays consistent with the origin at all times: reads come from Harper (fast), writes go through Harper to the origin (transactional), and Harper stores the result. In this guide you will build a write-through cache for a product inventory API. Reads are served from Harper. Writes go through Harper to the upstream REST API, and the updated value is immediately available in the local cache — no invalidation step needed. ## What You Will Learn * How to implement `put` and `delete` on a source Resource to enable write-through * How write-through works transactionally (two-phase commit) * How to load existing record data inside a write method using `ensureLoaded` * When write-through caching is the right pattern ## Prerequisites * Completed [Caching with Harper](/learn/developers/caching-with-harper.md) ## When to Use Write-Through Caching Write-through is a good fit when: * Clients write data that must be durably stored in an external system * You want reads to always come from Harper (fast) without a separate invalidation step * The origin supports idempotent `PUT`/`DELETE` operations * Consistency between the cache and origin is more important than write latency Write-through adds latency to writes because Harper waits for the origin to confirm before committing. If write throughput is a priority and brief inconsistency is acceptable, consider using invalidation-based caching instead. ## Defining the Schema ``` type InventoryItem @table(expiration: 300) @export { id: ID @primaryKey sku: String @indexed name: String quantity: Int warehouseId: String @indexed } ``` A 5-minute TTL (`expiration: 300`) ensures items don't stay stale forever if something goes wrong with the write-through path. In normal operation, writes keep the cache current and the TTL is rarely relevant. ## Implementing the Source Resource The `inventoryAPI` implements `get`, `put`, and `delete` methods. When write-through is active, Harper calls these methods in the appropriate order. ``` // resources.js const INVENTORY_API = process.env.INVENTORY_API ?? 'https://inventory.example.com'; const API_KEY = process.env.INVENTORY_API_KEY; const inventoryAPI = { get headers() { return { 'Content-Type': 'application/json', 'Authorization': `Bearer ${API_KEY}`, }; }, async get(id) { const response = await fetch(`${INVENTORY_API}/items/${id}`, { headers: this.headers, }); if (response.status === 404) { const error = new Error('Item not found'); error.statusCode = 404; throw error; } return response.json(); }, async put(id, data) { const body = await data; const response = await fetch(`${INVENTORY_API}/items/${id}`, { method: 'PUT', headers: this.headers, body: JSON.stringify(body), }); if (!response.ok) { const error = new Error(`Upstream PUT failed: ${response.status}`); error.statusCode = response.status; throw error; } // Return the confirmed record from the origin (may include server-generated fields) return response.json(); }, async delete(id) { const response = await fetch(`${INVENTORY_API}/items/${id}`, { method: 'DELETE', headers: this.headers, }); if (!response.ok) { const error = new Error(`Upstream DELETE failed: ${response.status}`); error.statusCode = response.status; throw error; } }, }; tables.InventoryItem.sourcedFrom(inventoryAPI); ``` With `put` and `delete` implemented on the source, Harper's write-through behavior activates automatically: * A `PUT /InventoryItem/item-001` request calls `inventoryAPI.put()` first * If the upstream call succeeds, Harper commits the result to the local cache * If the upstream call throws, Harper does not write to the cache — the local and remote data remain consistent * The updated record is immediately available for reads from Harper, no invalidation needed ## How the Two-Phase Write Works Write-through uses a two-phase commit to keep the cache and origin in sync: ``` Client → PUT /InventoryItem/item-001 ↓ Harper calls inventoryAPI.put() ↓ success ↓ error Harper commits to local Harper does not write; cache; 200 returned error propagated to client ``` Harper waits for the upstream `put()` to resolve before writing locally. This means write latency includes the round-trip to the origin, but reads from the local cache are always authoritative. ## Loading Existing Data in Write Methods By default, `put` and `delete` methods do not automatically load the existing cached record. If you need the current record value inside a write method — for example, to validate a field transition or merge partial data — call `get()` first: ``` const inventoryAPI = { async put(data) { // Load the existing record from cache or origin before writing const existing = await this.get(target); const incoming = await data; // Example: prevent setting quantity below zero const current = this.quantity ?? 0; if (incoming.quantity !== undefined && incoming.quantity < 0) { const error = new Error('Quantity cannot be negative'); error.statusCode = 400; throw error; } const response = await fetch(`${INVENTORY_API}/items/${this.getId()}`, { method: 'PUT', headers: this.headers, body: JSON.stringify(incoming), }); return response.json(); }, get, }; ``` ## Configuring the Application Open `config.yaml`: ``` graphqlSchema: files: 'schema.graphql' rest: true jsResource: files: 'resources.js' ``` ## Making Writes With Harper running, write an inventory item: #### curl ``` curl -i -X PUT 'http://localhost:9926/InventoryItem/item-001' \ -H 'Content-Type: application/json' \ -d '{"sku":"WB-100","name":"Titanium Water Bottle","quantity":150,"warehouseId":"WH-A"}' ``` #### fetch ``` const response = await fetch('http://localhost:9926/InventoryItem/item-001', { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ sku: 'WB-100', name: 'Titanium Water Bottle', quantity: 150, warehouseId: 'WH-A', }), }); console.log(response.status); ``` #### curl ``` HTTP/1.1 200 OK ``` #### fetch ``` 200 ``` The write went to `inventoryAPI.put()` first, was confirmed by the origin, and is now cached in Harper. A subsequent read comes directly from Harper without an upstream call: #### curl ``` curl -s 'http://localhost:9926/InventoryItem/item-001' ``` #### fetch ``` const item = await fetch('http://localhost:9926/InventoryItem/item-001').then((r) => r.json()); console.log(item.quantity); // 150 ``` #### curl ``` { "id": "item-001", "sku": "WB-100", "name": "Titanium Water Bottle", "quantity": 150, "warehouseId": "WH-A" } ``` #### fetch ``` 150 ``` ## Deleting a Record Deletes also flow through the source: #### curl ``` curl -i -X DELETE 'http://localhost:9926/InventoryItem/item-001' ``` #### fetch ``` await fetch('http://localhost:9926/InventoryItem/item-001', { method: 'DELETE' }); ``` #### curl ``` HTTP/1.1 204 No Content ``` #### fetch ``` 204 ``` Harper calls `inventoryAPI.delete()`, waits for confirmation, then removes the record from the local cache. A subsequent read triggers a `get()` call to the origin (or returns 404 if it's gone). ## Putting It All Together Complete `resources.js`: ``` // resources.js const INVENTORY_API = process.env.INVENTORY_API ?? 'https://inventory.example.com'; const API_KEY = process.env.INVENTORY_API_KEY; const inventoryAPI = { get headers() { return { 'Content-Type': 'application/json', 'Authorization': `Bearer ${API_KEY}`, }; } async get() { const response = await fetch(`${INVENTORY_API}/items/${this.getId()}`, { headers: this.headers, }); if (response.status === 404) { const error = new Error('Item not found'); error.statusCode = 404; throw error; } return response.json(); } async put(data) { const response = await fetch(`${INVENTORY_API}/items/${this.getId()}`, { method: 'PUT', headers: this.headers, body: JSON.stringify(await data), }); if (!response.ok) { const error = new Error(`Upstream PUT failed: ${response.status}`); error.statusCode = response.status; throw error; } return response.json(); } async delete() { const response = await fetch(`${INVENTORY_API}/items/${this.getId()}`, { method: 'DELETE', headers: this.headers, }); if (!response.ok) { const error = new Error(`Upstream DELETE failed: ${response.status}`); error.statusCode = response.status; throw error; } } } tables.InventoryItem.sourcedFrom(inventoryAPI); ``` ## Additional Resources * [Caching with Harper](/learn/developers/caching-with-harper.md) — foundational passive caching guide * [Resource API](/reference/v5/resources/resource-api.md) — `sourcedFrom`, `put`, `delete`, `ensureLoaded` * [Database Schema](/reference/v5/database/schema.md) — `@table(expiration:)` directive reference --- # Create Your First Application With Harper successfully installed and setup, let's dive into building your first Harper Application, a simple REST API. Harper lets you build powerful APIs with minimal effort. ## What You Will Learn * Overview of Harper architecture * What are Harper core services, plugins, and applications * How to define a table using schemas * How to run a Harper application * How to automatically create a REST API from a table schema * How to interact with the table using the generated REST API ## Prerequisites * Working Harper Installation (previous guide [Install and Connect Harper](/learn/getting-started/install-and-connect-harper.md)) ## Harper Architecture Before diving into building your first Harper application, it is important to understand a bit about Harper's architecture. The simplest way to think about Harper is as a stack. ``` ┏━━━━━━━━━━━━━━━━━━┓ ┃ Applications ┃ ┠──────────────────┨ ┃ Plugins ┃ ┃ - rest ┃ ┃ - graphqlSchema ┃ ┃ - ... ┃ ┠──────────────────┨ ┃ Core Services: ┃ ┃ - database ┃ ┃ - networking ┃ ┃ - component ┃ ┃ management ┃ ┗━━━━━━━━━━━━━━━━━━┛ ``` At the bottom are the **core services** that make up the foundation of the Harper platform. This includes the high-performance **database**, extensible **networking** middleware, and **component** management system. Components are extensions of the core Harper system, and are further classified as **plugins** and **applications**. **Plugins** come next in the stack. Plugins have access to APIs exposing many of Harper's core services, and are capable of implementing more advanced features than what the core services provide. Many of Harper's features are implemented as **built-in plugins**. Additional features can be implemented as **custom plugins**. In this guide, we'll be demonstrating some of Harper's built-in plugins `graphqlSchema` and `rest`. Later guides will demonstrate many more. And finally, the top of the stack are **applications**. This is where any user-facing functionality is implemented. Applications use plugins to implement their business logic, everything from database table schemas to web applications. The key difference between plugins and applications is that plugins enable the functionality and applications implement it. It's similar to that of a front-end framework. React on its own doesn't actually do anything, you actually need to build a "React App" for it to do anything meaningful. ## Initializing the Harper Application Let's get started building your first Harper application! #### Local Installation Get started by cloning the [`HarperFast/create-your-first-application`](https://github.com/HarperFast/create-your-first-application) repo and opening it your editor of choice. If you have installed Harper using a container, make sure to clone into the `dev/` directory that the container was mounted to. ``` git clone https://github.com/HarperFast/create-your-first-application.git first-harper-app ``` #### Fabric From the "Cluster" page, navigate to the "Applications" tab and click on "New Application" on the left-hand sidebar. Give the application a name such as "first-harper-app", then click on the "Import" tab. Specify `https://github.com/HarperFast/create-your-first-application` in the "Git Repository URL" field. Keep the "Install Command" empty and the "Authorization" as "Public Access". Finally, click the "Import Application" button and wait for the application to be instantiated. ## Creating a Table The core of most Harper applications is the data. Harper's data system is made up of databases and tables. There are many ways to create them, and the primary method is to use a GraphQL-like syntax to define table schemas. Even though you use GraphQL to define your table schemas, you do not need to use GraphQL for querying. #### Local Installation Open `schema.graphql` in your text editor. #### Fabric Navigate to the Files tab for your new application and open the `schema.graphql` file. Within `schema.graphql`, add: ``` type Dog @table { id: ID @primaryKey } ``` Harper has defined custom directives, such as `@table` and `@primaryKey`, to specify special behavior for the table schema. The `@table` directive is what instructs Harper that this is in fact a table schema versus an arbitrary type. The `@primaryKey` directive specifies which attribute is meant to be the primary key for indexing. Next, lets add some more properties to the schema. ``` type Dog @table { id: ID @primaryKey name: String breed: String age: Int } ``` Harper's schema system piggybacks off of the standard GraphQL field types such as `String`, `Int`, and many more. info The Harper schema system has a lot of great features for making it effortless to define tables. We'll demonstrate more custom directives later. Now you have a schema for a `Dog` table with four attributes `id`, `name`, `breed`, and `age`. The next step is to tell Harper about your schema file. Open the `config.yaml` file and add the following: ``` graphqlSchema: files: 'schema.graphql' ``` The `config.yaml` file is how Harper applications configure plugins. The `graphqlSchema` plugin is built-in to Harper so there is no additional steps needed to configure it, but custom plugins do require installing dependencies (more on that in another guide). The `files` property allows you to specify a file glob pattern for the plugin. In this case, we are only specifying a single file, but you can specify any glob pattern here too. With the `schema.graphql` and `config.yaml` in place, now its time to run your application for the first time. note If you need to check your work, checkout the [`01-create-table`](https://github.com/HarperFast/create-your-first-application/tree/01-create-table) branch. ## Running the Application #### Local Installation If Harper is still running, shut it down using `CTRL/CMD + C`. Within your application directory, open a command line and run `harper dev .` The `dev` command will watch all files (except for `node_modules` directory) within your application directory and restart Harper when changes are detected. #### Fabric Click "Restart Cluster" to apply the new file changes. ## Verifying the Table Was Created Once Harper has started, confirm your `Dog` table was picked up from `schema.graphql` by running: ``` harper describe_database database=data ``` You should see YAML output describing the `Dog` table and its attributes: ``` Dog: schema: data name: Dog primary_key: id attributes: - attribute: id type: ID is_primary_key: true - attribute: name type: String - attribute: breed type: String - attribute: age type: Int ... ``` If the table is missing, double-check that `config.yaml` references the correct schema file and that Harper restarted after the file was saved. ## Enabling the REST API Navigate back to the `schema.graphql` file and add `@export` directive to the table schema: ``` type Dog @table @export { id: ID @primaryKey name: String breed: String age: Int } ``` Then in `config.yaml` enable the REST API plugin: ``` graphqlSchema: files: 'schema.graphql' rest: true ``` #### Local Installation If Harper is still running with the `dev` command, it should have automatically restarted. If you look closely at the Harper logs, a new line should be added to the system configuration details: ``` REST: HTTP: 9926 ``` This line tells you that the Harper REST API is configured on port `9926` (this is configurable and this value is the default). #### Fabric Click "Restart Cluster" to apply the new file changes. note If you need to check your work, checkout the [`02-rest-api`](https://github.com/HarperFast/create-your-first-application/tree/02-rest-api) branch. ## Create a Record With everything in place, now its time to create your first record for the `Dog` table. The goal is to create a new `Dog` record with an ID of `001` and the properties: ``` { "name": "Harper", "breed": "Black Labrador / Chow Mix", "age": 5 } ``` With the automatic REST API generation you enabled in the previous step, you now have a plethora of options for interacting with the `Dog` table. Fabric users should use the UI directly, but can optionally follow along with the following request snippets if they want. Importantly, Fabric users must do things differently: 1. Replace `http://localhost` with their Fabric instance's URL 2. Include an Authorization header in the requests. Here is some additional information on how to create a Basic Authentication token: Basic Authentication The simplest authorization scheme is [Basic Authentication](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Authentication#basic_authentication_scheme) which transmits credentials as username/password pairs encoded using base64. Importantly, this scheme does not encrypt credentials. If used over an insecure connection, such as HTTP, they are susceptible to being compromised. Only ever use Basic Authentication over secured connections, such as HTTPS. Even then, its better to upgrade to an encryption based authentication scheme or certificates. Harper supports many different authentication mechanisms, and they will all be covered in later Learn guides. Use the username and password values from the previous [Install and Connect](/learn/getting-started/install-and-connect-harper.md) guide to generate an authorization value. The important part is to combine the `username` and `password` using a colon `:` character, encode that using base64, and then append the result to `"Basic "`. ``` const username = 'HDB_ADMIN'; const password = 'abc123!'; const authorizationValue = `Basic ${btoa(`${username}:${password}`)}`; ``` You would use the `authorizationValue` as the value for the `Authorization` header such as: ``` fetch('/', { // ... headers: { Authorization: authorizationValue, }, // ... }); ``` Create a `PUT` request using the REST API port and the path `/Dog/001`. Include a JSON body with the specified attributes except for `id`. The `001` in the URL will be used as the ID for this entry. #### curl ``` curl 'http://localhost:9926/Dog/001' \ -X PUT \ -H "Content-Type: application/json" \ -d '{ "name": "Harper", "breed": "Black Labrador / Chow Mix", "age": 5 }' \ -w "%{http_code}" ``` #### fetch ``` const response = await fetch('http://localhost:9926/Dog/001', { method: 'PUT', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify({ name: 'Harper', breed: 'Black Labrador / Chow Mix', age: 5, }), }); console.log(response.status); ``` If you see a `204` status code, then the record was successfully created! ## Read a Record Now, with the returned ID, create a `GET` request to the endpoint `/Dog/001`: #### curl ``` curl -s 'http://localhost:9926/Dog/001' | jq ``` #### fetch ``` const response = await fetch('http://localhost:9926/Dog/001'); const dog = await response.json(); console.log(dog); ``` You should see the record we just created returned as JSON: ``` { "name": "Harper", "breed": "Black Labrador / Chow Mix", "age": 5, "id": "001" } ``` ## Query a Record The REST API isn't just for basic CRUD operations. It can also be used to create search queries using URL query strings. Start by creating another `GET` query, but this time do not include the ID (`001`) part. Make sure to include a trailing slash. Then, include a query string containing an attribute and a value such as `?age=5`. #### curl ``` curl -s 'http://localhost:9926/Dog/?age=5' | jq ``` #### fetch ``` const response = await fetch('http://localhost:9926/Dog/?age=5'); const dog = await response.json(); console.log(dog); ``` Search queries return a list of records that match the specified conditions, in this example there is only one record in the table, so the result will be a list containing just that one record: ``` [ { "name": "Harper", "breed": "Black Labrador / Chow Mix", "age": 5, "id": "001" } ] ``` Fantastic work! You've successfully created your first Harper application. There is so much more that the Harper platform has to offer. Continue on to more guide content to learn more about building with Harper. ## Bonus: Deploy your Application to Fabric If you have been developing your Harper application locally, this section will walk you through deploying it to Fabric. Before continuing, if you haven't already set up a Fabric cluster, go back to the previous article and complete the [Getting started with Fabric](/learn/getting-started/install-and-connect-harper.md#getting-started-with-fabric) step and make sure to keep track of the cluster URL, as well as the admin username and password. Harper supports both **pull** and **push** based deployment workflows. Pull-based deployments is generally powered by a Git repository where Harper will *pull* your application from the repository. Push-based deployments is powered by the Harper CLI and is where your the user will *push* your application to the Harper instance. For a true production application, Harper recommends using pull-based deployments so that you can deploy tagged versions of your application repository. But for development and experimentation, push-based is perfectly fine. Later guides will explore pull-based deployment workflows in more detail. To get started with push-based deployments, open a command line and set the current directory to the application directory. First, log in to your Fabric cluster: ``` harper login ``` Type in your cluster's username and password. Then, run the `harper deploy` command: ``` harper deploy \ target= \ project=first-harper-app \ restart=true \ replicated=true ``` ## Additional Resources * [Using AI Agents](/learn/getting-started/using-agents.md) - Build Harper applications faster with AI. * [Table Schema](/reference/v5/database/schema.md) reference * [REST](/reference/v5/rest/overview.md) reference --- # Install and Connect Harper One of Harper's primary goals since day one was to be easy to install and get started with. The core Harper application itself is just a Node.js application with some native module dependencies. The simplest and easiest way to get started using Harper is by installing it using npm (or any npm compatible Node.js package manager). In addition to installing Harper directly to your local development environment, the Harper team provides a Docker image ([`harperfast/harper`](https://hub.docker.com/r/harper/harper)), and most recently a platform service called [Harper Fabric](https://fabric.harper.fast). This guide will demonstrate all three ways to get started as well as introduce some basic Harper features such as the CLI and our built in health endpoint. ## What You Will Learn * How to install Harper locally using a Node.js package manager or a container * How to use the `harper` CLI * How to get setup using Harper Fabric * How to perform a health check using the built-in Harper Operations API health endpoint ## Prerequisites Like the [Welcome](/learn.md) page stated, all guide pages require a set of system prerequisites such as command line access, HTTP client, and an up-to-date Node.js version. This guide is no different, and uniquely *only* requires those prerequisites. They are repeated here for your convenience, but future guides will not include them. * Command line access and general file-system and networking permissions * `sudo` is not required * For local development, Harper requires permission to read/write files and localhost networking permissions * HTTP client of choice * Most examples will present both curl and fetch-based HTTP requests * GUI-based HTTP clients will also work fine * Node.js Current, Active LTS, or Maintenance LTS version * For updated Node.js installation instructions refer to the official [Download Node.js](https://nodejs.org/en/download) page * For more information on valid versions refer to [Node.js Releases](https://nodejs.org/en/about/previous-releases) * Verify your Node.js version by running `node -v` in the command line * Code editor of choice * Everything from `vim` to Visual Studio Code to WebStorm IDE will work fine for the purposes of these guides ## Local installation and setup note If you want to use the cloud-hosted, platform service Harper Fabric instead of a local installation, skip ahead to the [Getting started with Fabric](#getting-started-with-fabric) section. #### npm Harper is published to the npm registry as [`harper`](https://www.npmjs.com/package/harper) and requires Node.js current, active LTS, or maintenance LTS versions to run. The fastest way to get started is by installing Harper globally using an npm compatible package manager: ``` npm install -g harper ``` Then, execute the Harper CLI: ``` harper ``` Harper also has a "pro" version, which includes full enterprise scale capabilities, such as replication, certificate management, and more advanced analytics. This is also the version running on Fabric. However, this is licensed with Elastic 2.0, and you need a usage license through Harper if you want to use this for a managed service. The oper and pro versions offer the same APIs for development, so the pro version is not necessary to start building applications. If you do want to install, it is available with at `@harperfast/harper-pro`: ``` npm install -g @harperfast/harper-pro ``` (Harper pro can be started with the `harper` command as well). When installing locally on your machine, you can specify any destination for Harper. We recommend using something within your home directory such as `$HOME/harper`. Keep note of what you specify for the username and password. Make sure you select the `dev` default config and set the hostname to `localhost`. important Do not actually enter `$HOME` in the destination prompt; it should automatically fill in the value of your home directory. Otherwise, specify the absolute path for the Harper installation. ``` Starting HarperDB install... Terms & Conditions can be found at https://harperdb.io/legal/end-user-license-agreement and can be viewed by typing or copying and pasting the URL into your web browser. I agree to the HarperDB Terms and Conditions: (yes/no) yes Please enter a destination for HarperDB: $HOME/harper Please enter a username for the administrative user: HDB_ADMIN Please enter a password for the administrative user: [hidden] Default Config - dev (easy access/debugging) or prod (security/performance): (dev/prod) dev Please enter the hostname for this server: localhost HarperDB installation was successful. [main/0] [notify]: HarperDB installation was successful. ``` #### Docker Harper is readily available as a Docker image [`harperfast/harper`](https://hub.docker.com/r/harper/harper). The image is based off of a Node.js image and the default tag is always published using the latest Harper version and latest Node.js Active LTS version. The image uses sensible default environment variables, agreeing to the terms and conditions, setting a rootpath ensured by the image itself, creating a default admin user with username `HDB_ADMIN` and password `password`, and enabled standard streams logging. Using a Docker compatible container manager of choice, the simplest way to get started is using: ``` docker pull harperfast/harper docker run -it \ --name harper \ -v $HOME/harper:/home/harper/harper \ -v $HOME/dev:/home/harper/dev \ -e DEFAULTS_MODE=dev \ -e REPLICATION_HOSTNAME=localhost \ -p 9925:9925 \ -p 9926:9926 \ harperfast/harper ``` The `-v` options will mount the Harper installation (`harper/`) as well as a development directory (`dev/`) to the container host which is useful for development purposes. The `harper/` path will contain the Harper installation parts, and the `dev/` directory can be used to create projects (which future guides will require). The additional environment variables specified by `-e` options ensures the installation is setup for local development. The image is configured to automatically run the `harper` command upon startup. During container creation, this will automatically complete the Harper installation step. The Harper installation process is normally an interactive prompt; however, it also supports overrides using environment variables or CLI arguments. Since the image contains preset environment variables and additional environment variables `DEFAULTS_MODE` and `REPLICATION_HOSTNAME` are included in the `docker run` command, Harper will complete its installation step completely non-interactively. After completing the installation step, Harper should now be running in the active process. As long as the `logging.stdStream` configuration option is set to `true` (which is the default when using the `dev` default mode), Harper will also stream all logs to the `stdout` and `stderr` streams too. If all is working correctly, you should see the following output in your command line: ``` Starting HarperDB... ▒▒▒▓▓▓▓▓▓▓▓▓▓▓▒▒ ▒▒▒▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒ ▒▒▒▓▓▓▓▓▓▓▓▓▓▓▓▓▓▒▒▒▒▒▒▓▓▓▓▓▓▓▓▓▓▓▒▒ ▒▒▒▒▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒▓▓▓▓▓▓▓▓▒ ▒▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▓▓▓▓▒ ▒▒▓▓▓▓▓▓▓▓▓▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▓▓▓▓▓▒▒ ▒▒▒▒▓▓▓▓▓▓▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▓▓▓▓▓▓▓▓▓▓▒▒▒▒▒▒ ▒▒▒▒▒▒▒▓▓▓▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒ ▒▒▒▒▒▒▒▒▒▒▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒▒▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒▒▒ ▒▒▒▒▒▒▒▒▒▒▒▒▒▓▓▒▒▒▒▒▒▒▒▒▒▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒▒ ▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒ ▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▒▒▒▒▒▒ ▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ ▒▒▒▓▓▓▓▓▓▓▓▓▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒ ▒▒▒▓▓▓▓▓▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒ ▒▒▒▒▓▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒ ▒▒▒▓▓▒▒▒▒▒▒▒▒▒▒▒ ▒▒▒▒▒▒▒ HarperDB, Inc. Denver, CO. [main/0] [info]: HarperDB PID [main/0] [info]: Checking if HDB software has been updated Debugger listening on ws://127.0.0.1:9229/ For help, see: https://nodejs.org/en/docs/inspector [main/0] [info]: All root applications loaded [http/1] [info]: Domain socket listening on /harper/operations-server HarperDB 4.y.z successfully started [main/0] [notify]: HarperDB successfully started. Hostname: localhost Worker Threads: 1 Root Path: /harper Debugging: enabled: true Logging: level: info, location: /harper/log/hdb.log, stdout/err Default: HTTP (and WS): 9926, CORS: enabled for * Operations API: HTTP: 9925, CORS: enabled for *, unix socket: /harper/operations-server MQTT: TCP: 1883, TLS: 8883, WS: 9926 Replication: WS: 9925, WSS: 9933 Note that log messages are being sent to the console (stdout and stderr) in addition to the log file /harper/log/hdb.log. This can be disabled by setting logging.stdStreams to false, and the log file can be directly monitored/tailed. This server does not have valid usage licenses, this should only be used for educational and development purposes. ``` This initial output contains a lot of helpful information. After the ASCII logo, there are a number of important log lines displaying the application process ID, the debugger endpoint, the domain socket path, and the Harper application version. Log lines are always prepended with the thread name, number, and log level. `[main/0]` is the main thread. `[http/1]` is the singular, additional worker thread. `[info]` is the default log level. After the log lines is specific application configuration information. It shows the application hostname, the number of worker threads, where Harper was installed, is the debugger enabled, logging level and location followed by ports, CORS, and socket path details for various networking protocols. We'll explain all of these in due time. Interrupting the process (`CTRL/CMD + C`) will shut down Harper. With Harper successfully running, skip ahead to [Performing a health check](#performing-a-health-check) to learn how to verify your local Harper instance is running and complete this getting started guide. Or continue reading for more information on getting started with Harper Fabric. ## Getting started with Fabric Fabric is our service for managing and deploying Harper on a distributed network. Fabric makes it easy to create new Harper clusters, the Harper application platform running on distributed nodes, and deploy your application to this service. Fabric has a management interface, and provides a UI for managing your deployments and even your local instance that you just installed. You can sign up for Fabric for free and create a free Harper cluster to deploy your application: * Go to [Fabric](https://fabric.harper.fast) and sign-up for a new account. * You will need to agree to the terms of service and verify your email address. * Once you have created an account, you can create an organization. This will allow you to collaboratively manage your Harper services with others. This will also define the host domain that will be used. * You can now create a new Harper cluster or instance: * Create a free Harper cluster for trying out Harper. * Purchase a Harper cluster with higher performance, scalability, and limits. * Add your own local instance to manage everything in one place. After successfully creating a Harper Fabric cluster, take note of the cluster URL and continue to the [Performing a health check](#performing-a-health-check) section to verify your instance. If you have any issues getting started with Fabric, consult the dedicated [Fabric documentation](/fabric.md). If you still need help, join the official Harper community [Discord](https://harper.fast/discord) and get help from the Harper engineering team. ## Performing a health check To check if everything is configured correctly and ready to go for development, run a health check using the built-in `/health` endpoint from the Harper Operations API server. The Operations API provides a full set of capabilities for configuring, deploying, administering, and controlling Harper. We'll learn more about it throughout all of the guide pages. note If you are using a Harper Fabric cluster, replace the `http://localhost` with the cluster URL. #### curl ``` curl 'http://localhost:9925/health' ``` #### fetch ``` const response = await fetch('http://localhost:9925/health'); const text = await response.text(); console.log(text); ``` If you see `HarperDB is running.`, fantastic work! You've successfully installed and setup Harper. Continue on to the next part of the Getting Started section, [creating your first Harper application](/learn/getting-started/create-your-first-application.md). ## Additional Resources * [Harper CLI](/reference/v5/cli/overview.md) reference documentation * [Harper Fabric](/fabric.md) documentation --- # Using AI Agents AI-powered development tools can significantly accelerate your workflow when building Harper applications. Whether you want a dedicated assistant or prefer using your favorite LLM, Harper provides the tools and context needed to make AI an effective part of your development process. ## What You Will Learn * How to install and run `harper-agent`, a purpose-built AI assistant for Harper development * How to configure `harper-agent` with your preferred AI provider (Claude, ChatGPT, Gemini, or Ollama) * How to use Harper Skills to improve general-purpose AI tools like Claude, ChatGPT, Cursor, or GitHub Copilot * How to provide your AI tools with Harper-specific context for better code generation ## Prerequisites * An API key for your preferred AI provider. See the [harper-agent repository](https://github.com/HarperFast/harper-agent) for a list of supported providers and instructions for obtaining credentials. * Node.js and npm installed on your system ## Harper Agent The `harper-agent` is an open source, purpose-built AI assistant designed specifically for Harper development. It understands the Harper ecosystem and can help you with tasks ranging from project setup to debugging. See its source code in the [`HarperFast/harper-agent`](https://github.com/HarperFast/harper-agent) repository. ### Features * **Application Creation**: Scaffold full Harper applications from natural language descriptions. * **Code Generation**: Write schema definitions, custom functions, and integration code. * **Diagnosis and Running**: Run your application and let the agent diagnose and fix errors. * **Browser Control**: The agent can even interact with a browser to help you test and manage your applications. ### Installation Install the Harper Agent globally using npm: ``` npm install -g @harperfast/agent ``` ### Usage To start the agent, simply run: ``` harper-agent ``` On first run, it will help you configure your preferred AI model (Gemini, Claude, ChatGPT, or Ollama). Once configured, you can interact with it directly from your terminal. ## Skills: Empowering General-Purpose Agents If you prefer using general-purpose AI tools like **Claude**, **ChatGPT**, **GitHub Copilot**, or **Cursor**, you can provide them with Harper-specific "Skills" to improve their accuracy and performance. ### What are Skills? Skills are a collection of Harper-specific context, documentation, and best practices. When an AI agent has access to these skills, it is much more likely to generate high-quality, idiomatic Harper code and follow current best practices. ### Getting Skills The easiest way to get Harper skills is by using the `create-harper` bootstrapper. When you create a new Harper project, a `skills/` directory is automatically included. ``` npm create harper@latest ``` You can also browse our ever evolving library of skills in the [`HarperFast/skills`](https://github.com/HarperFast/skills) repository. ### How to use Skills Once you have a `skills/` directory in your project, you can use it with your favorite AI tools: * **Chat-based AI (Claude/ChatGPT)**: Reference anything in your `skills/` directory to your conversation to provide the AI with immediate context. Some agents allow you to reference skills with `/` commands, like `/harper-best-practices`. * **IDE Extensions (Cursor/Copilot)**: Ensure these tools are indexing your project. They will automatically pick up the context from the `skills/` directory to provide better completions and chat responses. * **Custom Agents**: If you are building your own AI-powered workflows, you can point your agent to these skills to give it specialized knowledge of Harper. By leveraging these AI tools, you can move from idea to a running Harper application faster than ever before. ## Additional Resources * [`HarperFast/harper-agent`](https://github.com/HarperFast/harper-agent) — Source code, documentation, and supported AI providers for the Harper Agent * [`HarperFast/create-harper`](https://github.com/HarperFast/create-harper) — Bootstrap a new Harper project with skills included * [`HarperFast/skills`](https://github.com/HarperFast/skills) — Browse the full collection of Harper Skills --- # Harper v4 Reference Complete technical reference for Harper v4. Each section covers a core feature or subsystem — configuration options, APIs, and operational details. For concept introductions, tutorials, and guides, see the [Learn](/learn.md) section. ## Sections ### Data & Application | Section | Description | | -------------------------------------------------- | ----------------------------------------------------------------------- | | [Database](/reference/v4/database/overview.md) | Schema system, storage, indexing, transactions, and the database JS API | | [Resources](/reference/v4/resources/overview.md) | Custom resource classes, the Resource API, and query optimization | | [Components](/reference/v4/components/overview.md) | Applications, extensions, the Plugin API, and the JS environment | ### Access & Security | Section | Description | | ---------------------------------------------------------- | -------------------------------------------------------------------------- | | [REST](/reference/v4/rest/overview.md) | Auto-REST interface, querying, content types, headers, WebSockets, and SSE | | [HTTP](/reference/v4/http/overview.md) | HTTP server configuration, TLS, and the `server` API | | [Security](/reference/v4/security/overview.md) | Authentication mechanisms, certificates, and CORS/SSL configuration | | [Users & Roles](/reference/v4/users-and-roles/overview.md) | RBAC, roles configuration, and user management operations | ### Setup & Operation | Section | Description | | ---------------------------------------------------------- | ------------------------------------------------------------- | | [CLI](/reference/v4/cli/overview.md) | All CLI commands, Operations API commands, and authentication | | [Configuration](/reference/v4/configuration/overview.md) | `harperdb-config.yaml` options and configuration operations | | [Operations API](/reference/v4/operations-api/overview.md) | Full index of all Operations API operations | ### Features | Section | Description | | ------------------------------------------------------------------------ | ------------------------------------------------------------------ | | [Logging](/reference/v4/logging/overview.md) | Log configuration, the `logger` API, and log management operations | | [Analytics](/reference/v4/analytics/overview.md) | Resource and storage analytics, system tables | | [MQTT](/reference/v4/mqtt/overview.md) | MQTT broker configuration and usage | | [Static Files](/reference/v4/static-files/overview.md) | Static file serving via the `static` plugin | | [Environment Variables](/reference/v4/environment-variables/overview.md) | Environment variable loading via the `loadEnv` plugin | | [Replication](/reference/v4/replication/overview.md) | Native replication, clustering, and sharding | | [GraphQL Querying](/reference/v4/graphql-querying/overview.md) | Experimental GraphQL support | | [Studio](/reference/v4/studio/overview.md) | Local Studio UI configuration and access | | [Fastify Routes](/reference/v4/fastify-routes/overview.md) | Fastify route definitions (discouraged in favor of components) | ### Legacy | Section | Description | | ------------------------------------------------------------ | ------------------------------------------------------------------------ | | [Harper Cloud](/reference/v4/legacy/cloud.md) | Legacy Harper Cloud documentation — see Fabric for current cloud hosting | | [Custom Functions](/reference/v4/legacy/custom-functions.md) | Deprecated predecessor to Components | --- # Analytics Operations Operations for querying Harper analytics data. All operations require `superuser` permission. Analytics data can also be queried directly via `search_by_conditions` on the `hdb_raw_analytics` and `hdb_analytics` tables in the `system` database — see [Analytics Overview](/reference/v4/analytics/overview.md) for details on the table structure. *** ## `list_metrics` Returns the list of available metric names that can be queried with `get_analytics`. ### Parameters | Parameter | Required | Type | Description | | -------------- | -------- | --------- | ------------------------------------------------------------------------ | | `operation` | Yes | string | Must be `"list_metrics"` | | `metric_types` | No | string\[] | Filter by type: `"builtin"`, `"custom"`, or both. Default: `["builtin"]` | ### Request ``` { "operation": "list_metrics", "metric_types": ["custom", "builtin"] } ``` ### Response ``` ["resource-usage", "table-size", "database-size", "main-thread-utilization", "utilization", "storage-volume"] ``` *** ## `describe_metric` Returns the structure and available attributes for a specific metric. ### Parameters | Parameter | Required | Type | Description | | ----------- | -------- | ------ | ------------------------------ | | `operation` | Yes | string | Must be `"describe_metric"` | | `metric` | Yes | string | Name of the metric to describe | ### Request ``` { "operation": "describe_metric", "metric": "resource-usage" } ``` ### Response ``` { "attributes": [ { "name": "id", "type": "number" }, { "name": "metric", "type": "string" }, { "name": "userCPUTime", "type": "number" }, { "name": "systemCPUTime", "type": "number" }, { "name": "node", "type": "string" } ] } ``` *** ## `get_analytics` Queries analytics data for a specific metric over a time range. ### Parameters | Parameter | Required | Type | Description | | ---------------- | -------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `operation` | Yes | string | Must be `"get_analytics"` | | `metric` | Yes | string | Metric name — use `list_metrics` to get valid values | | `start_time` | No | number | Start of time range as Unix timestamp in milliseconds | | `end_time` | No | number | End of time range as Unix timestamp in milliseconds | | `get_attributes` | No | string\[] | Attributes to include in each result. If omitted, all attributes are returned | | `conditions` | No | object\[] | Additional filter conditions. Same format as [`search_by_conditions`](/reference/v4/operations-api/operations.md#nosql-operations) | ### Request ``` { "operation": "get_analytics", "metric": "resource-usage", "start_time": 1769198332754, "end_time": 1769198532754, "get_attributes": ["id", "metric", "userCPUTime", "systemCPUTime"], "conditions": [ { "attribute": "node", "operator": "equals", "value": "node1.example.com" } ] } ``` ### Response ``` [ { "id": "12345", "metric": "resource-usage", "userCPUTime": 100, "systemCPUTime": 50 }, { "id": "67890", "metric": "resource-usage", "userCPUTime": 150, "systemCPUTime": 75 } ] ``` ## Related * [Analytics Overview](/reference/v4/analytics/overview.md) * [Operations API Overview](/reference/v4/operations-api/overview.md) --- # Analytics *Added in: v4.5.0* (resource and storage analytics) Harper collects real-time telemetry and statistics across all operations, URL endpoints, and messaging topics. This data can be used to monitor server health, understand traffic and usage patterns, identify resource-intensive queries, and inform scaling decisions. ## Storage Tables Analytics data is stored in two system tables in the `system` database: | Table | Description | | ------------------- | ------------------------------------------------------------------------------------------- | | `hdb_raw_analytics` | Per-second raw entries recorded by each thread. One record per second per active thread. | | `hdb_analytics` | Aggregate entries recorded once per minute, summarizing all per-second data across threads. | Both tables require `superuser` permission to query. ## Raw Analytics (`hdb_raw_analytics`) Raw entries are recorded once per second (when there is activity) by each thread. Each record captures all activity in the last second along with system resource information. Records use the timestamp in milliseconds since epoch as the primary key. Query raw analytics using `search_by_conditions` on the `hdb_raw_analytics` table. The example below fetches 10 seconds of raw entries: ``` POST http://localhost:9925 Content-Type: application/json { "operation": "search_by_conditions", "schema": "system", "table": "hdb_raw_analytics", "conditions": [{ "search_attribute": "id", "search_type": "between", "search_value": [1688594000000, 1688594010000] }] } ``` Example raw entry: ``` { "time": 1688594390708, "period": 1000.8336279988289, "metrics": [ { "metric": "bytes-sent", "path": "search_by_conditions", "type": "operation", "median": 202, "mean": 202, "p95": 202, "p90": 202, "count": 1 }, { "metric": "memory", "threadId": 2, "rss": 1492664320, "heapTotal": 124596224, "heapUsed": 119563120, "external": 3469790, "arrayBuffers": 798721 }, { "metric": "utilization", "idle": 138227.52767700003, "active": 70.5066209952347, "utilization": 0.0005098165086230495 } ], "threadId": 2, "totalBytesProcessed": 12182820, "id": 1688594390708.6853 } ``` ## Aggregate Analytics (`hdb_analytics`) Aggregate entries are recorded once per minute, combining per-second raw entries from all threads into a single summary record. Use `search_by_conditions` on the `hdb_analytics` table with a broader time range: ``` POST http://localhost:9925 Content-Type: application/json { "operation": "search_by_conditions", "schema": "system", "table": "hdb_analytics", "conditions": [{ "search_attribute": "id", "search_type": "between", "search_value": [1688194100000, 1688594990000] }] } ``` Example aggregate entry: ``` { "period": 60000, "metric": "bytes-sent", "method": "connack", "type": "mqtt", "median": 4, "mean": 4, "p95": 4, "p90": 4, "count": 1, "id": 1688589569646, "time": 1688589569646 } ``` ## Standard Metrics Harper automatically tracks the following metrics for all services. Applications can also define custom metrics via [`server.recordAnalytics()`](/reference/v4/http/api.md#serverrecordanalyticsvalue-metric-path-method-type). ### HTTP Metrics | `metric` | `path` | `method` | `type` | Unit | Description | | ------------------ | ------------- | -------------- | --------------------------- | ----- | ---------------------------------------- | | `duration` | resource path | request method | `cache-hit` or `cache-miss` | ms | Duration of request handler | | `duration` | route path | request method | `fastify-route` | ms | Duration of Fastify route handler | | `duration` | operation | | `operation` | ms | Duration of Operations API operation | | `success` | resource path | request method | | % | Percentage of successful requests | | `success` | route path | request method | `fastify-route` | % | | | `success` | operation | | `operation` | % | | | `bytes-sent` | resource path | request method | | bytes | Response bytes sent | | `bytes-sent` | route path | request method | `fastify-route` | bytes | | | `bytes-sent` | operation | | `operation` | bytes | | | `transfer` | resource path | request method | `operation` | ms | Duration of response transfer | | `transfer` | route path | request method | `fastify-route` | ms | | | `transfer` | operation | | `operation` | ms | | | `socket-routed` | | | | % | Percentage of sockets immediately routed | | `tls-handshake` | | | | ms | TLS handshake duration | | `tls-reused` | | | | % | Percentage of TLS sessions reused | | `cache-hit` | table name | | | % | Percentage of cache hits | | `cache-resolution` | table name | | | ms | Duration of resolving uncached entries | ### MQTT / WebSocket Metrics | `metric` | `path` | `method` | `type` | Unit | Description | | ------------------ | ------ | ------------ | ------ | ----- | ------------------------------------------------ | | `mqtt-connections` | | | | count | Number of open direct MQTT connections | | `ws-connections` | | | | count | Number of open WebSocket connections | | `connection` | `mqtt` | `connect` | | % | Percentage of successful direct MQTT connections | | `connection` | `mqtt` | `disconnect` | | % | Percentage of explicit direct MQTT disconnects | | `connection` | `ws` | `connect` | | % | Percentage of successful WebSocket connections | | `connection` | `ws` | `disconnect` | | % | Percentage of explicit WebSocket disconnects | | `bytes-sent` | topic | mqtt command | `mqtt` | bytes | Bytes sent for a given MQTT command and topic | ### Replication Metrics | `metric` | `path` | `method` | `type` | Unit | Description | | ---------------- | ------------- | ------------- | --------- | ----- | ----------------------------------- | | `bytes-sent` | node.database | `replication` | `egress` | bytes | Bytes sent for replication | | `bytes-sent` | node.database | `replication` | `blob` | bytes | Bytes sent for blob replication | | `bytes-received` | node.database | `replication` | `ingress` | bytes | Bytes received for replication | | `bytes-received` | node.database | `replication` | `blob` | bytes | Bytes received for blob replication | ### Resource Usage Metrics | `metric` | Key attributes | Other | Unit | Description | | ------------------------- | ------------------------------------------------------------------------------------------------ | ------------------- | ------- | --------------------------------------------------------------------------------- | | `database-size` | `size`, `used`, `free`, `audit` | `database` | bytes | Database file size breakdown | | `main-thread-utilization` | `idle`, `active`, `taskQueueLatency`, `rss`, `heapTotal`, `heapUsed`, `external`, `arrayBuffers` | `time` | various | Main thread resource usage: idle/active time, queue latency, and memory breakdown | | `resource-usage` | (see below) | | various | Node.js process resource usage (see [resource-usage](#resource-usage-metric)) | | `storage-volume` | `available`, `free`, `size` | `database` | bytes | Storage volume size breakdown | | `table-size` | `size` | `database`, `table` | bytes | Table file size | | `utilization` | | | % | Percentage of time the worker thread was processing requests | #### `resource-usage` Metric Includes everything returned by Node.js [`process.resourceUsage()`](https://nodejs.org/api/process.html#processresourceusage) (with `userCPUTime` and `systemCPUTime` converted to milliseconds), plus: | Field | Unit | Description | | ---------------- | ---- | ------------------------------------------- | | `time` | ms | Unix timestamp when the metric was recorded | | `period` | ms | Duration of the measurement period | | `cpuUtilization` | % | CPU utilization (user + system combined) | ## Custom Metrics Applications can record custom metrics using the `server.recordAnalytics()` API. See [HTTP API](/reference/v4/http/api.md) for details. ## Analytics Configuration The `analytics.aggregatePeriod` configuration option controls how frequently aggregate summaries are written. See [Configuration Overview](/reference/v4/configuration/overview.md) for details. Per-component analytics logging can be configured via `analytics.logging`. See [Logging Configuration](/reference/v4/logging/configuration.md) for details. ## Related * [Analytics Operations](/reference/v4/analytics/operations.md) * [HTTP API](/reference/v4/http/api.md) * [Logging Configuration](/reference/v4/logging/configuration.md) * [Configuration Overview](/reference/v4/configuration/overview.md) --- # CLI Authentication The Harper CLI handles authentication differently for local and remote operations. ## Local Operations Available since: v4.1.0 For local operations (operations executed on the same machine where Harper is installed), the CLI communicates with Harper via Unix domain sockets instead of HTTP. Domain socket requests are automatically authenticated as the superuser, so no additional authentication parameters are required. **Example**: ``` # No authentication needed for local operations harper describe_database database=dev harper get_components harper set_configuration logging_level=info ``` When no `target` parameter is specified, the CLI defaults to using the local domain socket connection, providing secure, authenticated access to the local Harper instance. ## Remote Operations Available since: v4.1.0; expanded in: v4.3.0 For remote operations (operations executed on a remote Harper instance via the `target` parameter), you must provide authentication credentials. ### Authentication Methods #### Method 1: Environment Variables (Recommended) Set the following environment variables to avoid exposing credentials in command history: ``` export CLI_TARGET_USERNAME=HDB_ADMIN export CLI_TARGET_PASSWORD=password ``` Then execute remote operations without including credentials in the command: ``` harper describe_database database=dev target=https://server.com:9925 harper get_components target=https://remote-instance.example.com:9925 ``` **Benefits**: * Credentials not visible in command history * More secure for scripting * Can be set once per session * Supported by most CI/CD systems **Example Script**: ``` #!/bin/bash # Set credentials from secure environment export CLI_TARGET_USERNAME=HDB_ADMIN export CLI_TARGET_PASSWORD=$SECURE_PASSWORD # from secret manager # Execute operations harper deploy target=https://prod-server.com:9925 replicated=true harper restart target=https://prod-server.com:9925 replicated=true ``` #### Method 2: Command Parameters Provide credentials directly as command parameters: ``` harper describe_database \ database=dev \ target=https://server.com:9925 \ username=HDB_ADMIN \ password=password ``` **Parameters**: * `username=` - Harper admin username * `password=` - Harper admin password **Cautions**: * Credentials visible in command history * Less secure for production environments * Exposed in process listings * Not recommended for scripts ### Target Parameter The `target` parameter specifies the full HTTP/HTTPS URL of the remote Harper instance: **Format**: `target=://:` **Examples**: ``` # HTTPS on default operations API port target=https://server.example.com:9925 # HTTP (not recommended for production) target=http://localhost:9925 # Custom port target=https://server.example.com:8080 ``` ## Security Best Practices ### 1. Use Environment Variables Always use environment variables for credentials in scripts and automation: ``` export CLI_TARGET_USERNAME=HDB_ADMIN export CLI_TARGET_PASSWORD=$SECURE_PASSWORD ``` ### 2. Use HTTPS Always use HTTPS for remote operations to encrypt credentials in transit: ``` # Good target=https://server.com:9925 # Bad - credentials sent in plain text target=http://server.com:9925 ``` ### 3. Manage Secrets Securely Store credentials in secure secret management systems: * Environment variables from secret managers (AWS Secrets Manager, HashiCorp Vault, etc.) * CI/CD secret storage (GitHub Secrets, GitLab CI Variables, etc.) * Operating system credential stores **Example with AWS Secrets Manager**: ``` #!/bin/bash # Retrieve credentials from AWS Secrets Manager export CLI_TARGET_USERNAME=$(aws secretsmanager get-secret-value \ --secret-id harper-admin-user \ --query SecretString \ --output text) export CLI_TARGET_PASSWORD=$(aws secretsmanager get-secret-value \ --secret-id harper-admin-password \ --query SecretString \ --output text) # Execute operations harper deploy target=https://prod.example.com:9925 ``` ### 4. Use Least Privilege Create dedicated users with minimal required permissions for CLI operations instead of using the main admin account. See [Users and Roles](/reference/v4/users-and-roles/overview.md) for more information. ### 5. Rotate Credentials Regularly rotate credentials, especially for automated systems and CI/CD pipelines. ### 6. Audit Access Monitor and audit CLI operations, especially for production environments. See [Logging](/reference/v4/logging/overview.md) for more information on logging. ## Troubleshooting ### Authentication Failures If you receive authentication errors: 1. **Verify credentials are correct**: * Check username and password * Ensure no extra whitespace 2. **Verify the target URL**: * Ensure the URL is correct and reachable * Check the port number * Verify HTTPS/HTTP protocol 3. **Check network connectivity**: ``` curl https://server.com:9925 ``` 4. **Verify user permissions**: * Ensure the user has permission to execute the operation * Check user roles and permissions ### Environment Variable Issues If environment variables aren't working: 1. **Verify variables are set**: ``` echo $CLI_TARGET_USERNAME echo $CLI_TARGET_PASSWORD ``` 2. **Export variables**: Ensure you used `export`, not just assignment: ``` # Wrong - variable only available in current shell CLI_TARGET_USERNAME=admin # Correct - variable available to child processes export CLI_TARGET_USERNAME=admin ``` 3. **Check variable scope**: * Variables must be exported before running commands * Variables set in one terminal don't affect other terminals ## See Also * [CLI Overview](/reference/v4/cli/overview.md) - General CLI information * [CLI Commands](/reference/v4/cli/commands.md) - Core CLI commands * [Operations API Commands](/reference/v4/cli/operations-api-commands.md) - Operations available through CLI * [Security Overview](/reference/v4/security/overview.md) - Harper security features * [Users and Roles](/reference/v4/users-and-roles/overview.md) - User management --- # CLI Commands This page documents the core Harper CLI commands for managing Harper instances. For Operations API commands available through the CLI, see [Operations API Commands](/reference/v4/cli/operations-api-commands.md). ## Process Management Commands ### `harper` *Added in: v4.1.0* Run Harper in the foreground as a standard process. This is the recommended way to run Harper. ``` harper ``` When you run `harper`: * If Harper is not installed, it will guide you through the installation process * Once installed, it runs Harper in the foreground as a standard process, compatible with systemd, Docker, and other process management tools **First-Time Installation**: If Harper is not installed, you can provide configuration parameters via environment variables or command line arguments: **Using Environment Variables**: ``` # Minimum required parameters for no additional CLI prompts export TC_AGREEMENT=yes export HDB_ADMIN_USERNAME=HDB_ADMIN export HDB_ADMIN_PASSWORD=password export ROOTPATH=/hdb/ harper ``` note If you specify `DEFAULT_MODE=dev` you will also need to specify the `REPLICATION_HOSTNAME=localhost` **Using Command Line Arguments**: ``` # Minimum required parameters for no additional CLI prompts harper \ --TC_AGREEMENT=yes \ --HDB_ADMIN_USERNAME=HDB_ADMIN \ --HDB_ADMIN_PASSWORD=password \ --ROOTPATH='/hdb' ``` **Note**: When used in conjunction, command line arguments override environment variables. See [Configuration](/reference/v4/configuration/overview.md) for a full list of configuration parameters. info For more information on installation, see [Getting Started / Install and Connect Harper](/learn/getting-started/install-and-connect-harper.md). ### `harper run` *Added in: v4.2.0* Run a Harper application from any location as a foreground, standard process (similar to `harper`). ``` harper run /path/to/app ``` This command runs Harper with the specified application directory without automatic reloading or dev-specific features. ### `harper dev` *Added in: v4.2.0* Run Harper in development mode from a specified directory with automatic reloading. Recommended for local application development. Operates similar to `harper` and `harper run`. ``` harper dev /path/to/app ``` **Features**: * Pushes logs to standard streams automatically * Uses a single thread for simpler debugging * Auto-restart on file changes ### `harper restart` Available since: v4.1.0 Restart a running Harper instance regardless if its a foreground (`harper`, `harper run`, or `harper dev`) or background (`harper start`) process. ``` harper restart ``` ### `harper start` Available since: v4.1.0 Start Harper in background (daemon mode). ``` harper start ``` After installation, this command launches Harper as a background process. Remember that the Harper PID is available in a `hdb.pid` file within the installation directory. ### `harper stop` Available since: v4.1.0 Stop a running Harper instance. ``` harper stop ``` ## Installation Commands ### `harper install` Available since: v4.1.0 Install Harper with interactive prompts or automated configuration. ``` harper install ``` The `harper install` command operates exactly like the [`harper`](#harper) command, but exits as soon as the installation completes. See the [`harper`](#harper) command documentation above for details on providing configuration parameters via environment variables or command line arguments. **Note**: We recommend using `harper` instead of `harper install` as it provides a consistent workflow for both installation and running Harper. ## Information Commands ### `harper version` Available since: v4.1.0 Display the installed Harper version. ``` harper version ``` **Example Output**: ``` 4.7.0 ``` ### `harper status` Available since: v4.1.0 Display the status of Harper and clustering. ``` harper status ``` Shows: * Harper process status * Clustering network status * Replication statuses In Harper versions where NATS is supported, this command also shows the clustering hub and leaf processes too. ### `harper help` Available since: v4.1.0 Display all available Harper CLI commands with brief descriptions. ``` harper help ``` ## Maintenance Commands ### `harper renew-certs` Available since: v4.1.0 Renew Harper-generated self-signed certificates. ``` harper renew-certs ``` This command regenerates the self-signed SSL/TLS certificates used by Harper. ### `harper copy-db` Available since: v4.1.0 Copy a Harper database with compaction to eliminate free-space and fragmentation. ``` harper copy-db ``` **Parameters**: * `` - Name of the source database * `` - Full path to the target database file **Example**: ``` harper copy-db data /home/user/hdb/database/copy.mdb ``` This copies the default `data` database to a new location with compaction applied. **Use Cases**: * Database optimization * Eliminating fragmentation * Creating compacted backups * Reclaiming free space See also: [Database Compaction](/reference/v4/database/compaction.md) for more information. #### How Backups Work Harper uses a transactional commit process that ensures data on disk is always transactionally consistent with storage. This means Harper maintains database integrity in the event of a crash and allows you to use standard volume snapshot tools to make backups. **Backup Process**: Database files are stored in the `hdb/database` directory. As long as the snapshot is an atomic snapshot of these database files, the data can be copied/moved back into the database directory to restore a previous backup (with Harper shut down), and database integrity will be preserved. **Important Notes**: * **Atomic Snapshots**: Use volume snapshot tools that create atomic snapshots * **Not Safe**: Simply copying an in-use database file using `cp` is **not reliable** * Progressive reads occur at different points in time * Results in an unreliable copy that likely won't be usable * **Safe Copying**: Standard file copying is only reliable for database files that are **not in use** **Recommended Backup Tools**: * LVM snapshots * ZFS snapshots * BTRFS snapshots * Cloud provider volume snapshots (AWS EBS, Azure Disk, GCP Persistent Disk) * Enterprise backup solutions with snapshot capabilities ## Remote Operations The CLI supports executing commands on remote Harper instances. For details, see [CLI Overview - Remote Operations](/reference/v4/cli/overview.md#remote-operations). ## See Also * [CLI Overview](/reference/v4/cli/overview.md) - General CLI information * [Operations API Commands](/reference/v4/cli/operations-api-commands.md) - Operations available through CLI * [CLI Authentication](/reference/v4/cli/authentication.md) - Authentication mechanisms * [Configuration](/reference/v4/configuration/overview.md) - Configuration parameters for installation * [Database Compaction](/reference/v4/database/compaction.md) - More on database compaction --- # Operations API Commands *Added in: v4.3.0* The Harper CLI supports executing operations from the [Operations API](/reference/v4/operations-api/overview.md) directly from the command line. This enables powerful automation and scripting capabilities. ## General Syntax ``` harper = ``` **Output Format**: * Default: YAML * JSON: Pass `json=true` as a parameter ## Supported Operations The following operations are available through the CLI. Operations that require complex nested parameters or object structures are not supported via CLI and must be executed through the HTTP API. ### Complete Operations List note This is just a brief overview of all operations available as CLI commands. Review the respective operation documentation for more information on available arguments and expected behavior. Keep in mind that all operations options are converted to CLI arguments in the same way (using `snake_case`). | Operation | Description | Category | Available Since | | -------------------------------- | ------------------------------------- | --------------------------------------------------------------------------------- | --------------- | | `describe_table` | Describe table structure and metadata | [Database](/reference/v4/operations-api/operations.md#databases--tables) | v4.3.0 | | `describe_all` | Describe all databases and tables | [Database](/reference/v4/operations-api/operations.md#databases--tables) | v4.3.0 | | `describe_database` | Describe database structure | [Database](/reference/v4/operations-api/operations.md#databases--tables) | v4.3.0 | | `create_database` | Create a new database | [Database](/reference/v4/operations-api/operations.md#databases--tables) | v4.3.0 | | `drop_database` | Delete a database | [Database](/reference/v4/operations-api/operations.md#databases--tables) | v4.3.0 | | `create_table` | Create a new table | [Database](/reference/v4/operations-api/operations.md#databases--tables) | v4.3.0 | | `drop_table` | Delete a table | [Database](/reference/v4/operations-api/operations.md#databases--tables) | v4.3.0 | | `create_attribute` | Create a table attribute | [Database](/reference/v4/operations-api/operations.md#databases--tables) | v4.3.0 | | `drop_attribute` | Delete a table attribute | [Database](/reference/v4/operations-api/operations.md#databases--tables) | v4.3.0 | | `search_by_id` | Search records by ID | [Data](/reference/v4/operations-api/operations.md#nosql-operations) | v4.3.0 | | `search_by_value` | Search records by attribute value | [Data](/reference/v4/operations-api/operations.md#nosql-operations) | v4.3.0 | | `insert` | Insert new records | [Data](/reference/v4/operations-api/operations.md#nosql-operations) | v4.4.9 | | `update` | Update existing records | [Data](/reference/v4/operations-api/operations.md#nosql-operations) | v4.4.9 | | `upsert` | Insert or update records | [Data](/reference/v4/operations-api/operations.md#nosql-operations) | v4.4.9 | | `delete` | Delete records | [Data](/reference/v4/operations-api/operations.md#nosql-operations) | v4.3.0 | | `sql` | Execute SQL queries | [Data](/reference/v4/operations-api/operations.md#nosql-operations) | v4.3.0 | | `csv_file_load` | Load data from CSV file | [Data](/reference/v4/operations-api/operations.md#nosql-operations) | v4.3.0 | | `csv_url_load` | Load data from CSV URL | [Data](/reference/v4/operations-api/operations.md#nosql-operations) | v4.3.0 | | `list_users` | List all users | [Security](/reference/v4/operations-api/operations.md#certificate-management) | v4.3.0 | | `add_user` | Create a new user | [Security](/reference/v4/operations-api/operations.md#certificate-management) | v4.3.0 | | `alter_user` | Modify user properties | [Security](/reference/v4/operations-api/operations.md#certificate-management) | v4.3.0 | | `drop_user` | Delete a user | [Security](/reference/v4/operations-api/operations.md#certificate-management) | v4.3.0 | | `list_roles` | List all roles | [Security](/reference/v4/operations-api/operations.md#certificate-management) | v4.3.0 | | `drop_role` | Delete a role | [Security](/reference/v4/operations-api/operations.md#certificate-management) | v4.3.0 | | `create_csr` | Create certificate signing request | [Security](/reference/v4/operations-api/operations.md#certificate-management) | v4.4.0 | | `sign_certificate` | Sign a certificate | [Security](/reference/v4/operations-api/operations.md#certificate-management) | v4.4.0 | | `list_certificates` | List SSL/TLS certificates | [Security](/reference/v4/operations-api/operations.md#certificate-management) | v4.4.0 | | `add_certificate` | Add SSL/TLS certificate | [Security](/reference/v4/operations-api/operations.md#certificate-management) | v4.4.0 | | `remove_certificate` | Remove SSL/TLS certificate | [Security](/reference/v4/operations-api/operations.md#certificate-management) | v4.4.0 | | `add_ssh_key` | Add SSH key | [Security](/reference/v4/operations-api/operations.md#certificate-management) | v4.4.0 | | `get_ssh_key` | Get SSH key | [Security](/reference/v4/operations-api/operations.md#certificate-management) | v4.7.2 | | `update_ssh_key` | Update SSH key | [Security](/reference/v4/operations-api/operations.md#certificate-management) | v4.4.0 | | `delete_ssh_key` | Delete SSH key | [Security](/reference/v4/operations-api/operations.md#certificate-management) | v4.4.0 | | `list_ssh_keys` | List all SSH keys | [Security](/reference/v4/operations-api/operations.md#certificate-management) | v4.4.0 | | `set_ssh_known_hosts` | Set SSH known hosts | [Security](/reference/v4/operations-api/operations.md#certificate-management) | v4.4.0 | | `get_ssh_known_hosts` | Get SSH known hosts | [Security](/reference/v4/operations-api/operations.md#certificate-management) | v4.4.0 | | `cluster_get_routes` | Get cluster routing information | [Clustering](/reference/v4/operations-api/operations.md#replication--clustering) | v4.3.0 | | `cluster_network` | Get cluster network status | [Clustering](/reference/v4/operations-api/operations.md#replication--clustering) | v4.3.0 | | `cluster_status` | Get cluster status | [Clustering](/reference/v4/operations-api/operations.md#replication--clustering) | v4.3.0 | | `remove_node` | Remove node from cluster | [Clustering](/reference/v4/operations-api/operations.md#replication--clustering) | v4.3.0 | | `add_component` | Add a component | [Components](/reference/v4/operations-api/operations.md#components) | v4.3.0 | | `deploy_component` | Deploy a component | [Components](/reference/v4/operations-api/operations.md#components) | v4.3.0 | | `deploy` (alias) | Alias for `deploy_component` | [Components](/reference/v4/operations-api/operations.md#components) | v4.3.0 | | `package_component` | Package a component | [Components](/reference/v4/operations-api/operations.md#components) | v4.3.0 | | `package` (alias) | Alias for `package_component` | [Components](/reference/v4/operations-api/operations.md#components) | v4.3.0 | | `drop_component` | Remove a component | [Components](/reference/v4/operations-api/operations.md#components) | v4.3.0 | | `get_components` | List all components | [Components](/reference/v4/operations-api/operations.md#components) | v4.3.0 | | `get_component_file` | Get component file contents | [Components](/reference/v4/operations-api/operations.md#components) | v4.3.0 | | `set_component_file` | Set component file contents | [Components](/reference/v4/operations-api/operations.md#components) | v4.3.0 | | `install_node_modules` | Install Node.js dependencies | [Components](/reference/v4/operations-api/operations.md#components) | v4.3.0 | | `set_configuration` | Update configuration settings | [Configuration](/reference/v4/operations-api/operations.md#configuration) | v4.3.0 | | `get_configuration` | Get current configuration | [Configuration](/reference/v4/operations-api/operations.md#configuration) | v4.3.0 | | `create_authentication_tokens` | Create authentication tokens | [Authentication](/reference/v4/operations-api/operations.md#token-authentication) | v4.3.0 | | `refresh_operation_token` | Refresh operation token | [Authentication](/reference/v4/operations-api/operations.md#token-authentication) | v4.3.0 | | `restart_service` | Restart Harper service | [System](/reference/v4/operations-api/operations.md#registration--licensing) | v4.3.0 | | `restart` | Restart Harper instance | [System](/reference/v4/operations-api/operations.md#registration--licensing) | v4.3.0 | | `system_information` | Get system information | [System](/reference/v4/operations-api/operations.md#registration--licensing) | v4.3.0 | | `registration_info` | Get registration information | [Licensing](/reference/v4/operations-api/operations.md#registration--licensing) | v4.3.0 | | `get_fingerprint` | Get instance fingerprint | [Licensing](/reference/v4/operations-api/operations.md#registration--licensing) | v4.3.0 | | `set_license` | Set license key | [Licensing](/reference/v4/operations-api/operations.md#registration--licensing) | v4.3.0 | | `get_usage_licenses` | Get usage and license info | [Licensing](/reference/v4/operations-api/operations.md#registration--licensing) | v4.7.3 | | `get_job` | Get job status | [Jobs](/reference/v4/operations-api/operations.md#jobs) | v4.3.0 | | `search_jobs_by_start_date` | Search jobs by start date | [Jobs](/reference/v4/operations-api/operations.md#jobs) | v4.3.0 | | `read_log` | Read application logs | [Logging](/reference/v4/operations-api/operations.md#logs) | v4.3.0 | | `read_transaction_log` | Read transaction logs | [Logging](/reference/v4/operations-api/operations.md#logs) | v4.3.0 | | `read_audit_log` | Read audit logs | [Logging](/reference/v4/operations-api/operations.md#logs) | v4.3.0 | | `delete_transaction_logs_before` | Delete old transaction logs | [Logging](/reference/v4/operations-api/operations.md#logs) | v4.3.0 | | `purge_stream` | Purge streaming data | [Maintenance](/reference/v4/operations-api/operations.md#jobs) | v4.3.0 | | `delete_records_before` | Delete old records | [Maintenance](/reference/v4/operations-api/operations.md#jobs) | v4.3.0 | | `get_status` | Get custom status information | [Status](/reference/v4/operations-api/operations.md#registration--licensing) | v4.6.0 | | `set_status` | Set custom status information | [Status](/reference/v4/operations-api/operations.md#registration--licensing) | v4.6.0 | | `clear_status` | Clear custom status information | [Status](/reference/v4/operations-api/operations.md#registration--licensing) | v4.6.0 | ### Command Aliases The following aliases are available for convenience: * `deploy` → `deploy_component` * `package` → `package_component` For detailed parameter information for each operation, see the [Operations API documentation](/reference/v4/operations-api/operations.md). ## Command Examples ### Database Operations **Describe a database**: ``` harper describe_database database=dev ``` **Describe a table** (with YAML output): ``` harper describe_table database=dev table=dog ``` **Example Output**: ``` schema: dev name: dog hash_attribute: id audit: true schema_defined: false attributes: - attribute: id is_primary_key: true - attribute: name indexed: true clustering_stream_name: 3307bb542e0081253klnfd3f1cf551b record_count: 10 last_updated_record: 1724483231970.9949 ``` tip For detailed information on database and table structures, see the [Database Reference](/reference/v4/database/overview.md). ### Data Operations **Search by ID** (with JSON output): ``` harper search_by_id database=dev table=dog ids='["1"]' get_attributes='["*"]' json=true ``` **Search by value**: ``` harper search_by_value table=dog search_attribute=name search_value=harper get_attributes='["id", "name"]' ``` tip For more information on querying data, see the [REST Reference](/reference/v4/rest/overview.md) and [GraphQL Querying](/reference/v4/graphql-querying/overview.md). ### Configuration Operations **Set configuration**: ``` harper set_configuration logging_level=error ``` **Get configuration**: ``` harper get_configuration ``` tip For comprehensive configuration options, see the [Configuration Reference](/reference/v4/configuration/overview.md). ### Component Operations **Deploy a component**: ``` harper deploy_component project=my-cool-app package=https://github.com/HarperDB/application-template ``` **Get all components**: ``` harper get_components ``` **Note**: `deploy` is an alias for `deploy_component`: ``` harper deploy project=my-app package=https://github.com/user/repo ``` tip For more information on components and applications, see the [Components Reference](/reference/v4/components/overview.md). ### User and Role Operations **List users**: ``` harper list_users ``` **List roles**: ``` harper list_roles ``` tip For detailed information on users, roles, and authentication, see the [Security Reference](/reference/v4/security/overview.md). ## Remote Operations All CLI operations can be executed on remote Harper instances. See [CLI Overview - Remote Operations](/reference/v4/cli/overview.md#remote-operations) for details on authentication and remote execution. ### Remote Component Deployment When using remote operations, you can deploy a local component or application to the remote instance. **Deploy current directory**: If you omit the `package` parameter, the current directory will be packaged and deployed: ``` harper deploy target=https://server.com:9925 ``` **Note**: `deploy` is an alias for `deploy_component`. **Deploy to clustered environment**: For clustered environments, use the `replicated=true` parameter to ensure the deployment is replicated to all nodes: ``` harper deploy target=https://server.com:9925 replicated=true ``` **Restart after deployment** (with replication): After deploying to a clustered environment, restart all nodes to apply changes: ``` harper restart target=https://server.com:9925 replicated=true ``` For more information on Harper applications and components, see: * [Components](/reference/v4/components/overview.md) - Application architecture and structure * [Deploying Harper Applications](/learn/getting-started/install-and-connect-harper.md) - Step-by-step deployment guide ## Parameter Formatting ### String Parameters Simple string values can be passed directly: ``` harper describe_table database=dev table=dog ``` ### Array Parameters Array parameters must be quoted and formatted as JSON: ``` harper search_by_id database=dev table=dog ids='["1","2","3"]' ``` ### Object Parameters Object parameters are not supported via CLI. For operations requiring complex nested objects, use: * The [Operations API](/reference/v4/operations-api/overview.md) via HTTP * A custom script or tool ### Boolean Parameters Boolean values can be passed as strings: ``` harper get_configuration json=true harper deploy target=https://server.com:9925 replicated=true ``` ## Output Formatting ### YAML (Default) By default, CLI operation results are formatted as YAML for readability: ``` harper describe_table database=dev table=dog ``` ### JSON Pass `json=true` to get JSON output (useful for scripting): ``` harper describe_table database=dev table=dog json=true ``` ## Scripting and Automation The Operations API commands through the CLI are ideal for: * Build and deployment scripts * Automation workflows * CI/CD pipelines * Administrative tasks * Monitoring and health checks **Example Script**: ``` #!/bin/bash # Deploy component to remote cluster export CLI_TARGET_USERNAME=HDB_ADMIN export CLI_TARGET_PASSWORD=$SECURE_PASSWORD harper deploy \ target=https://cluster-node-1.example.com:9925 \ replicated=true \ package=https://github.com/myorg/my-component # Restart the cluster harper restart \ target=https://cluster-node-1.example.com:9925 \ replicated=true # Check status harper get_components \ target=https://cluster-node-1.example.com:9925 \ json=true ``` ## Limitations The following operation types are **not supported** via CLI: * Operations requiring complex nested JSON structures * Operations with array-of-objects parameters * File upload operations * Streaming operations For these operations, use the [Operations API](/reference/v4/operations-api/overview.md) directly via HTTP. ## See Also * [CLI Overview](/reference/v4/cli/overview.md) - General CLI information * [CLI Commands](/reference/v4/cli/commands.md) - Core CLI commands * [Operations API Overview](/reference/v4/operations-api/overview.md) - Operations API documentation * [Operations API Reference](/reference/v4/operations-api/operations.md) - Complete operations list * [CLI Authentication](/reference/v4/cli/authentication.md) - Authentication details --- # Harper CLI Overview The Harper command line interface (CLI) is used to administer self-installed Harper instances. ## Installation Available since: v4.1.0 Harper is typically installed globally via npm: ``` npm i -g harperdb ``` The installation includes the Harper CLI, which provides comprehensive management capabilities for local and remote Harper instances. For detailed installation instructions, see the [Getting Started / Install And Connect Harper](https://docs.harperdb.io/docs/getting-started/install-and-connect-harper) guide. ## Command Name *Changed in: v4.7.0* The CLI command is `harper`. From v4.1.0 to v4.6.x, the command was only available as `harperdb`. Starting in v4.7.0, the preferred command is `harper`, though `harperdb` continues to work as an alias for backward compatibility. **Examples**: ``` # Modern usage (v4.7.0+) harper harper describe_table database=dev table=dog # Legacy usage (still supported) harperdb harperdb describe_table database=dev table=dog ``` All examples in this documentation use `harper`. ## General Usage The primary way to use Harper is to run the `harper` command. When you run `harper`: * If Harper is not installed, it will guide you through the installation process * Once installed, it runs Harper in the foreground as a standard process * This makes it compatible with systemd, Docker, and other process management tools * Output logs directly to the console for easy monitoring The CLI supports two main categories of commands: 1. **System Commands** - Core Harper management commands (start, stop, restart, status, etc.) 2. **Operations API Commands** - Execute operations from the Operations API directly via the CLI Both system and operations commands can be executed on local or remote Harper instances. For remote operations, authentication credentials can be provided via command parameters or environment variables. ### CLI Installation Targeting By default, the CLI targets the Harper installation path stored in `~/.harperdb/hdb_boot_properties.file`. You can override this to target a specific Harper installation by specifying the `--ROOTPATH` command line argument or the `ROOTPATH` environment variable. **Example: Target a specific installation**: ``` # Using command line argument harper status --ROOTPATH /custom/path/to/hdb # Using environment variable export ROOTPATH=/custom/path/to/hdb harper status ``` ### Process ID File When Harper is running, the process identifier (PID) is stored in a file named `hdb.pid` located in the Harper installation directory. This file can be used by external process management tools or scripts to monitor or manage the Harper process. **Location**: `/hdb.pid` **Example**: ``` # Read the PID cat /path/to/hdb/hdb.pid # Use with external tools kill -0 $(cat /path/to/hdb/hdb.pid) # Check if process is running ``` ## System Management Commands | Command | Description | Available Since | | ---------------------------------- | ------------------------------------------------------------ | --------------- | | `harper` | Run Harper in foreground mode (default behavior) | v4.1.0 | | `harper run ` | Run Harper application from any directory | v4.2.0 | | `harper dev ` | Run Harper in dev mode with auto-restart and console logging | v4.2.0 | | `harper restart` | Restart Harper | v4.1.0 | | `harper start` | Start Harper in background (daemon mode) | v4.1.0 | | `harper stop` | Stop a running Harper instance | v4.1.0 | | `harper status` | Display Harper and clustering status | v4.1.0 | | `harper version` | Show installed Harper version | v4.1.0 | | `harper renew-certs` | Renew Harper-generated self-signed certificates | v4.1.0 | | `harper copy-db ` | Copy a database with compaction | v4.1.0 | | `harper help` | Display all available CLI commands | v4.1.0 | See [CLI Commands](/reference/v4/cli/commands.md) for detailed documentation on each command. ## Operations API Commands *Added in: v4.3.0* The Harper CLI supports executing most operations from the [Operations API](/reference/v4/operations-api/overview.md) directly from the command line. This includes operations that do not require complex nested parameters. **Syntax**: `harper =` **Output Format**: Results are formatted as YAML by default. Pass `json=true` for JSON output. **Examples**: ``` # Describe a table harper describe_table database=dev table=dog # Set configuration harper set_configuration logging_level=error # Deploy a component harper deploy_component project=my-app package=https://github.com/user/repo # Get all components harper get_components # Search by ID (JSON output) harper search_by_id database=dev table=dog ids='["1"]' json=true # SQL query harper sql sql='select * from dev.dog where id="1"' ``` See [Operations API Commands](/reference/v4/cli/operations-api-commands.md) for the complete list of available operations. ## Remote Operations *Changed in: v4.3.0* (expanded remote operations support) The CLI can execute operations on remote Harper instances by passing the `target` parameter with the HTTP address of the remote instance. **Authentication**: Provide credentials via: * Parameters: `username= password=` * Environment variables: `CLI_TARGET_USERNAME` and `CLI_TARGET_PASSWORD` See [CLI Authentication](/reference/v4/cli/authentication.md) for detailed information on authentication methods and best practices. **Example: CLI Target Environment Variables**: ``` export CLI_TARGET_USERNAME=HDB_ADMIN export CLI_TARGET_PASSWORD=password harper describe_database database=dev target=https://server.com:9925 ``` **Example: CLI Options**: ``` harper describe_database database=dev target=https://server.com:9925 username=HDB_ADMIN password=password ``` ## Development Mode *Added in: v4.2.0* For local application and component development, use `harper dev`: ``` harper dev /path/to/app ``` **Features**: * Console logging for immediate feedback * Debugging enabled * Auto-restart on file changes * Ideal for rapid iteration during development See [CLI Commands](/reference/v4/cli/commands.md) for detailed information on `harper dev` and other development commands. ## See Also * [CLI Commands](/reference/v4/cli/commands.md) - Detailed reference for each CLI command * [Operations API Commands](/reference/v4/cli/operations-api-commands.md) - Operations available through CLI * [CLI Authentication](/reference/v4/cli/authentication.md) - Authentication mechanisms * [Configuration](/reference/v4/configuration/overview.md) - Harper configuration options * [Operations API](/reference/v4/operations-api/overview.md) - Full operations API reference --- # Applications > The contents of this page primarily relate to **application** components. The term "components" in the Operations API and CLI generally refers to applications specifically. See the [Components Overview](/reference/v4/components/overview.md) for a full explanation of terminology. Harper offers several approaches to managing applications that differ between local development and remote Harper instances. ## Local Development ### `dev` and `run` Commands *Added in: v4.2.0* The quickest way to run an application locally is with the `dev` command inside the application directory: ``` harperdb dev . ``` The `dev` command watches for file changes and restarts Harper worker threads automatically. The `run` command is similar but does not watch for changes. Use `run` when the main thread needs to be restarted (the `dev` command does not restart the main thread). Stop either process with SIGINT (Ctrl+C). ### Deploying to a Local Harper Instance To mimic interaction with a hosted Harper instance locally: 1. Start Harper: `harperdb` 2. Deploy the application: ``` harperdb deploy \ project= \ package= \ restart=true ``` * Omit `target` to deploy to the locally running instance. * Setting `package=` creates a symlink so file changes are picked up automatically between restarts. * `restart=true` restarts worker threads after deploy. Use `restart=rolling` for a rolling restart. 3. Use `harperdb restart` in another terminal to restart threads at any time. 4. Remove an application: `harperdb drop_component project=` > Not all [component operations](#operations-api) are available via CLI. When in doubt, use the Operations API via direct HTTP requests to the local Harper instance. Example: ``` harperdb deploy \ project=test-application \ package=/Users/dev/test-application \ restart=true ``` > Use `package=$(pwd)` if your current directory is the application directory. ## Remote Management Managing applications on a remote Harper instance uses the same operations as local management. The key difference is specifying a `target` along with credentials: ``` harperdb deploy \ project= \ package= \ username= \ password= \ target= \ restart=true \ replicated=true ``` Credentials can also be provided via environment variables: ``` export CLI_TARGET_USERNAME= export CLI_TARGET_PASSWORD= harperdb deploy \ project= \ package= \ target= \ restart=true \ replicated=true ``` ### Package Sources When deploying remotely, the `package` field can be any valid npm dependency value: * **Omit** `package` to package and deploy the current local directory * **npm package**: `package="@harperdb/status-check"` * **GitHub**: `package="HarperDB/status-check"` or `package="https://github.com/HarperDB/status-check"` * **Private repo (SSH)**: `package="git+ssh://git@github.com:HarperDB/secret-app.git"` * **Tarball**: `package="https://example.com/application.tar.gz"` When using git tags, use the `semver` directive for reliable versioning: ``` HarperDB/application-template#semver:v1.0.0 ``` Harper generates a `package.json` from component configurations and uses a form of `npm install` to resolve them. This is why specifying a local file path creates a symlink (changes are picked up between restarts without redeploying). For SSH-based private repos, use the [Add SSH Key](#add_ssh_key) operation to register keys first. ## Dependency Management Harper uses `npm` and `package.json` for dependency management. During application loading, Harper follows this resolution order to determine how to install dependencies: 1. If `node_modules` exists, or if `package.json` is absent — skip installation 2. Check the application's `harperdb-config.yaml` for `install: { command, timeout }` fields 3. Derive the package manager from [`package.json#devEngines#packageManager`](https://docs.npmjs.com/cli/v10/configuring-npm/package-json#devengines) 4. Default to `npm install` The `add_component` and `deploy_component` operations support `install_command` and `install_timeout` fields for customizing this behavior. ### Example `harperdb-config.yaml` with Custom Install ``` myApp: package: ./my-app install: command: yarn install timeout: 600000 # 10 minutes ``` ### Example `package.json` with `devEngines` ``` { "name": "my-app", "version": "1.0.0", "devEngines": { "packageManager": { "name": "pnpm", "onFail": "error" } } } ``` > If you plan to use an alternative package manager, ensure it is installed on the host machine. Harper does not support the `"onFail": "download"` option and falls back to `"onFail": "error"` behavior. ## Advanced: Direct `harperdb-config.yaml` Configuration Applications can be added to Harper by adding them directly to `harperdb-config.yaml` (located in the Harper `rootPath`, typically `~/hdb`). ``` status-check: package: '@harperdb/status-check' ``` The entry name does not need to match a `package.json` dependency. Harper transforms these entries into a `package.json` and runs `npm install`. Any valid npm dependency specifier works: ``` myGithubComponent: package: HarperDB-Add-Ons/package#v2.2.0 myNPMComponent: package: harperdb myTarBall: package: /Users/harper/cool-component.tar myLocal: package: /Users/harper/local myWebsite: package: https://harperdb-component ``` Harper generates a `package.json` and installs all components into `` (default: `~/hdb/components`). A symlink back to `/node_modules` is created for dependency resolution. > Use `harperdb get_configuration` to find the `rootPath` and `componentsRoot` values on your instance. ## Operations API Component operations are restricted to `super_user` roles. ### `add_component` Creates a new component project in the component root directory using a template. * `project` *(required)* — Name of the project to create * `template` *(optional)* — Git URL of a template repository. Defaults to `https://github.com/HarperFast/application-template` * `install_command` *(optional)* — Install command. Defaults to `npm install` * `install_timeout` *(optional)* — Install timeout in milliseconds. Defaults to `300000` (5 minutes) * `replicated` *(optional)* — Replicate to all cluster nodes ``` { "operation": "add_component", "project": "my-component" } ``` ### `deploy_component` Deploys a component using a package reference or a base64-encoded `.tar` payload. * `project` *(required)* — Name of the project * `package` *(optional)* — Any valid npm reference (GitHub, npm, tarball, local path, URL) * `payload` *(optional)* — Base64-encoded `.tar` file content * `force` *(optional)* — Allow deploying over protected core components. Defaults to `false` * `restart` *(optional)* — `true` for immediate restart, `'rolling'` for sequential cluster restart * `replicated` *(optional)* — Replicate to all cluster nodes * `install_command` *(optional)* — Install command override * `install_timeout` *(optional)* — Install timeout override in milliseconds ``` { "operation": "deploy_component", "project": "my-component", "package": "HarperDB/application-template#semver:v1.0.0", "replicated": true, "restart": "rolling" } ``` ### `drop_component` Deletes a component project or a specific file within it. * `project` *(required)* — Project name * `file` *(optional)* — Path relative to project folder. If omitted, deletes the entire project * `replicated` *(optional)* — Replicate deletion to all cluster nodes * `restart` *(optional)* — Restart Harper after dropping ``` { "operation": "drop_component", "project": "my-component" } ``` ### `package_component` Packages a project folder as a base64-encoded `.tar` string. * `project` *(required)* — Project name * `skip_node_modules` *(optional)* — Exclude `node_modules` from the package ``` { "operation": "package_component", "project": "my-component", "skip_node_modules": true } ``` ### `get_components` Returns all local component files, folders, and configuration from `harperdb-config.yaml`. ``` { "operation": "get_components" } ``` ### `get_component_file` Returns the contents of a file within a component project. * `project` *(required)* — Project name * `file` *(required)* — Path relative to project folder * `encoding` *(optional)* — File encoding. Defaults to `utf8` ``` { "operation": "get_component_file", "project": "my-component", "file": "resources.js" } ``` ### `set_component_file` Creates or updates a file within a component project. * `project` *(required)* — Project name * `file` *(required)* — Path relative to project folder * `payload` *(required)* — File content to write * `encoding` *(optional)* — File encoding. Defaults to `utf8` * `replicated` *(optional)* — Replicate update to all cluster nodes ``` { "operation": "set_component_file", "project": "my-component", "file": "test.js", "payload": "console.log('hello world')" } ``` ### SSH Key Management For deploying from private repositories, SSH keys must be registered on the Harper instance. #### `add_ssh_key` * `name` *(required)* — Key name * `key` *(required)* — Private key contents (must be ed25519; use `\n` for line breaks with trailing `\n`) * `host` *(required)* — Host alias for SSH config (used in `package` URL) * `hostname` *(required)* — Actual domain (e.g., `github.com`) * `known_hosts` *(optional)* — Public SSH keys of the host. Auto-retrieved for `github.com` * `replicated` *(optional)* — Replicate to all cluster nodes ``` { "operation": "add_ssh_key", "name": "my-key", "key": "-----BEGIN OPENSSH PRIVATE KEY-----\n...\n-----END OPENSSH PRIVATE KEY-----\n", "host": "my-key.github.com", "hostname": "github.com" } ``` After adding a key, use the configured host in deploy package URLs: ``` "package": "git+ssh://git@my-key.github.com:my-org/my-repo.git#semver:v1.0.0" ``` Additional SSH key operations: `update_ssh_key`, `delete_ssh_key`, `list_ssh_keys`, `set_ssh_known_hosts`, `get_ssh_known_hosts`. --- # Extension API > As of Harper v4.6, a new iteration of the extension system called **Plugins** was released. Plugins simplify the API and are recommended for new extension development. See the [Plugin API](/reference/v4/components/plugin-api.md) reference. Both extensions and plugins are supported; extensions are not yet deprecated. Extensions are components that provide reusable building blocks for applications. There are two key types: * **Resource Extensions** — Handle specific files or directories * **Protocol Extensions** — More advanced extensions that can return a Resource Extension; primarily used for implementing higher-level protocols and custom networking handlers An extension is distinguished from a plain component by implementing one or more of the Resource Extension or Protocol Extension API methods. ## Declaring an Extension All extensions must define a `config.yaml` with an `extensionModule` option pointing to the extension source code (path resolves from the module root directory): ``` extensionModule: ./extension.js ``` If written in TypeScript or another compiled language, point to the built output: ``` extensionModule: ./dist/index.js ``` ## Resource Extension A Resource Extension processes specific files or directories. It is comprised of four function exports: | Method | Thread | Timing | | ------------------- | ------------------ | ------------------------- | | `handleFile()` | All worker threads | Executed on every restart | | `handleDirectory()` | All worker threads | Executed on every restart | | `setupFile()` | Main thread only | Once, at initial start | | `setupDirectory()` | Main thread only | Once, at initial start | > **Important**: `harperdb restart` only restarts worker threads. Code in `setupFile()` and `setupDirectory()` runs only when Harper fully shuts down and starts again—not on `deploy` or `restart`. `handleFile()` and `setupFile()` have identical signatures. `handleDirectory()` and `setupDirectory()` have identical signatures. ### Resource Extension Configuration Resource Extensions can be configured with `files` and `urlPath` options in `config.yaml`: * `files` — `string | string[] | FilesOptionObject` *(required)* — Glob pattern(s) determining which files and directories are resolved. Harper uses [fast-glob](https://github.com/mrmlnc/fast-glob) for matching. * `source` — `string | string[]` *(required when object form)* — Glob pattern string(s) * `only` — `'all' | 'files' | 'directories'` *(optional)* — Restrict matching to a single entry type. Defaults to `'all'` * `ignore` — `string[]` *(optional)* — Patterns to exclude from matches * `urlPath` — `string` *(optional)* — Base URL path prepended to resolved entries * Starting with `./` (e.g., `'./static/'`) prepends the component name to the URL path * Value of `.` uses the component name as the base path * `..` is invalid and causes an error * Leading/trailing slashes are handled automatically (`/static/`, `static/`, and `/static` are equivalent) Examples: ``` # Serve HTML files from web/ at the /static/ URL path static: files: 'web/*.html' urlPath: 'static' # Load all GraphQL schemas from src/schema/ graphqlSchema: files: 'src/schema/*.graphql' # Match files in web/, excluding web/images/ static: files: source: 'web/**/*' ignore: ['web/images'] # Match only files (not directories) myExtension: files: source: 'dir/**/*' only: 'files' ``` ### Resource Extension API At minimum, a Resource Extension must implement one of the four methods. As a standalone extension, export them directly: ``` // ESM export function handleFile() {} export function setupDirectory() {} // CJS function handleDirectory() {} function setupFile() {} module.exports = { handleDirectory, setupFile }; ``` When returned by a Protocol Extension, define them on the returned object: ``` export function start() { return { handleFile() {}, }; } ``` #### `handleFile(contents, urlPath, absolutePath, resources): void | Promise` #### `setupFile(contents, urlPath, absolutePath, resources): void | Promise` Process individual files. Can be async. Parameters: * `contents` — `Buffer` — File contents * `urlPath` — `string` — Recommended URL path for the file * `absolutePath` — `string` — Absolute filesystem path * `resources` — `Object` — Currently loaded resources #### `handleDirectory(urlPath, absolutePath, resources): boolean | void | Promise` #### `setupDirectory(urlPath, absolutePath, resources): boolean | void | Promise` Process directories. Can be async. If the function returns a truthy value, the component loading sequence ends and no other entries in the directory are processed. Parameters: * `urlPath` — `string` — Recommended URL path for the directory * `absolutePath` — `string` — Absolute filesystem path * `resources` — `Object` — Currently loaded resources ## Protocol Extension A Protocol Extension is a more advanced form of Resource Extension, primarily used for implementing higher-level protocols (e.g., building and running a Next.js project) or adding custom networking handlers. Protocol Extensions use the [`server`](/reference/v4/http/api.md) global API for custom networking. ### Protocol Extension Configuration In addition to the `files`, `urlPath`, and `package` options, Protocol Extensions accept any additional configuration options defined under the extension name in `config.yaml`. These options are passed through to the `options` object of `start()` and `startOnMainThread()`. Many protocol extensions accept `port` and `securePort` options for configuring networking handlers. Example using `@harperdb/nextjs`: ``` '@harperdb/nextjs': package: '@harperdb/nextjs' files: './' prebuilt: true dev: false ``` ### Protocol Extension API A Protocol Extension defines up to two methods: | Method | Thread | Timing | | --------------------- | ------------------ | ------------------------- | | `start()` | All worker threads | Executed on every restart | | `startOnMainThread()` | Main thread only | Once, at initial start | Both methods receive the same `options` object and can return a Resource Extension (an object with any of the Resource Extension methods). #### `start(options): ResourceExtension | Promise` #### `startOnMainThread(options): ResourceExtension | Promise` Parameters: * `options` — `Object` — Extension configuration options from `config.yaml` Returns: An object implementing any of the Resource Extension methods ## Version History * **v4.2.0** — Extension system introduced as part of the component architecture * **v4.6.0** — New extension API with support for dynamic reloading; Plugin API introduced as the recommended alternative --- # JavaScript Environment Harper executes component JavaScript inside Node.js VM contexts — isolated module environments that share the same Node.js runtime but have their own global scope. This means each component runs in its own module context while still being able to access Harper's global APIs without any imports. ## Module Loading Harper supports both ESM and CommonJS module formats. All Harper globals are available directly as global variables in any component module. They are also accessible by importing from the `harperdb` package, which can provide better TypeScript typing: ``` import { tables, Resource } from 'harperdb'; ``` ``` const { tables, Resource } = require('harperdb'); ``` For components in their own directory, link the package to your local `harperdb` installation: ``` npm link harperdb ``` All installed components have `harperdb` automatically linked. ## Global APIs ### `tables` An object whose properties are the tables in the default database (`data`). Each table defined in `schema.graphql` is accessible as a property and implements the Resource API. See [Database API](/reference/v4/database/api.md) for full reference. ### `databases` An object containing all databases defined in Harper. Each database is an object of its tables — `databases.data` is always equivalent to `tables`. See [Database API](/reference/v4/database/api.md) for full reference. ### `transaction(fn)` Executes a function inside a database transaction. Changes made within the function are committed atomically, or rolled back if an error is thrown. See [Transactions](/reference/v4/database/transaction.md) for full reference. ### `createBlob(data, options?)` *Added in: v4.5.0* Creates a [`Blob`](https://developer.mozilla.org/en-US/docs/Web/API/Blob) backed by Harper's storage engine. Use it to store large binary content (images, audio, video, etc.) in `Blob`-typed schema fields. See [Database API](/reference/v4/database/api.md) for full reference. ### `Resource` The base class for all Harper resources, including tables and custom data sources. Extend `Resource` to implement custom data providers. See [Resource API](/reference/v4/resources/resource-api.md) for full reference. ### `server` Provides access to Harper's HTTP server middleware chain, WebSocket server, authentication helpers, resource registry, and cluster information. Also exposes `server.contentTypes` as an alias for the `contentTypes` global. See [HTTP API](/reference/v4/http/api.md) for full reference. ### `contentTypes` A `Map` of MIME type strings to content type handler objects. Harper uses this map for content negotiation — deserializing incoming request bodies and serializing outgoing responses. You can register custom handlers to support additional formats. See [HTTP API](/reference/v4/http/api.md) for full reference. ### `logger` Provides structured logging methods (`trace`, `debug`, `info`, `warn`, `error`, `fatal`, `notify`) that write to Harper's log file. Available without any imports in all component code. See [Logging API](/reference/v4/logging/api.md) for full reference. --- # Components **Components** are the high-level concept for modules that extend the Harper core platform with additional functionality. Components encapsulate both applications and extensions. > Harper is actively working to disambiguate component terminology. When you see "component" in the Operations API or CLI, it generally refers to an application. Documentation does its best to clarify which classification of component is meant wherever possible. ## Concepts ### Applications *Added in: v4.2.0* **Applications** implement specific user-facing features or functionality. Applications are built on top of extensions and represent the end product that users interact with. For example, a Next.js application serving a web interface or an Apollo GraphQL server providing a GraphQL API are both applications. Also, a collection of Harper Schemas and/or custom Resources is also an application. ### Extensions *Added in: v4.2.0* **Extensions** are the building blocks of the Harper component system. Applications depend on extensions to provide the functionality they implement. For example, the built-in `graphqlSchema` extension enables applications to define databases and tables using GraphQL schemas. The `@harperdb/nextjs` and `@harperdb/apollo` extensions provide building blocks for Next.js and Apollo applications respectively. Extensions can also depend on other extensions. For example, `@harperdb/apollo` depends on the built-in `graphqlSchema` extension to create a cache table for Apollo queries. ### Plugins (Experimental) *Added in: v4.6.0* (experimental) **Plugins** are a new iteration of the extension system introduced in v4.6. They are simultaneously a simplification and extensibility upgrade over extensions. Instead of defining multiple methods (`start` vs `startOnMainThread`, `handleFile` vs `setupFile`, `handleDirectory` vs `setupDirectory`), plugins only export a single `handleApplication` method. Plugins are **experimental**. In time extensions will be deprecated in favor of plugins, but both are currently supported. See the [Plugin API](/reference/v4/components/plugin-api.md) reference for complete documentation. ### Built-In vs. Custom Components **Built-in** components are included with Harper by default and referenced directly by name. Examples include `graphqlSchema`, `rest`, `jsResource`, `static`, and `loadEnv`. **Custom** components use external references—npm packages, GitHub repositories, or local directories—and are typically included as `package.json` dependencies. Harper does not currently include built-in applications. All applications are custom. ## Architecture The relationship between applications, extensions, and Harper core: ``` Applications ├── Next.js App → @harperdb/nextjs extension ├── Apollo App → @harperdb/apollo extension └── Custom Resource → jsResource + graphqlSchema + rest extensions Extensions ├── Custom: @harperdb/nextjs, @harperdb/apollo, @harperdb/astro └── Built-In: graphqlSchema, jsResource, rest, static, loadEnv, ... Core └── database, file-system, networking ``` ## Configuration Harper components are configured with a `config.yaml` file in the root of the component module directory. This file is how a component configures other components it depends on. Each entry starts with a component name, with configuration values indented below: ``` componentName: option-1: value option-2: value ``` ### Default Configuration Components without a `config.yaml` get this default configuration automatically: ``` rest: true graphqlSchema: files: '*.graphql' roles: files: 'roles.yaml' jsResource: files: 'resources.js' fastifyRoutes: files: 'routes/*.js' urlPath: '.' static: files: 'web/**' ``` If a `config.yaml` is provided, it **replaces** the default config entirely (no merging). ### Custom Component Configuration Any custom component must be configured with a `package` option for Harper to load it. The component name must match a `package.json` dependency: ``` { "dependencies": { "@harperdb/nextjs": "1.0.0" } } ``` ``` '@harperdb/nextjs': package: '@harperdb/nextjs' files: './' ``` The `package` value supports any valid npm dependency specifier: npm packages, GitHub repos, tarballs, local paths, and URLs. This is because Harper generates a `package.json` from component configurations and uses `npm install` to resolve them. ### Extension and Plugin Configuration Extensions require an `extensionModule` option pointing to the extension source. Plugins require a `pluginModule` option. See [Extension API](/reference/v4/components/extension-api.md) and [Plugin API](/reference/v4/components/plugin-api.md) for details. ## Built-In Extensions Reference | Name | Description | | ------------------------------------------------------------ | ------------------------------------------------- | | [`dataLoader`](/reference/v4/database/data-loader.md) | Load data from JSON/YAML files into Harper tables | | [`fastifyRoutes`](/reference/v4/fastify-routes/overview.md) | Define custom endpoints with Fastify | | [`graphql`](/reference/v4/graphql-querying/overview.md) | Enable GraphQL querying (experimental) | | [`graphqlSchema`](/reference/v4/database/schema.md) | Define table schemas with GraphQL syntax | | [`jsResource`](/reference/v4/resources/overview.md) | Define custom JavaScript-based resources | | [`loadEnv`](/reference/v4/environment-variables/overview.md) | Load environment variables from `.env` files | | [`rest`](/reference/v4/rest/overview.md) | Enable automatic REST endpoint generation | | [`roles`](/reference/v4/users-and-roles/overview.md) | Define role-based access control from YAML files | | [`static`](/reference/v4/static-files/overview.md) | Serve static files via HTTP | ## Known Custom Components ### Applications * [`@harperdb/status-check`](https://github.com/HarperDB/status-check) * [`@harperdb/prometheus-exporter`](https://github.com/HarperDB/prometheus-exporter) * [`@harperdb/acl-connect`](https://github.com/HarperDB/acl-connect) ### Extensions * [`@harperdb/nextjs`](https://github.com/HarperDB/nextjs) * [`@harperdb/apollo`](https://github.com/HarperDB/apollo) * [`@harperdb/astro`](https://github.com/HarperDB/astro) ## Component Status Monitoring *Added in: v4.7.0* Harper collects status from each component at load time and tracks any registered status change notifications. This provides visibility into the health and state of running components. ## Evolution History * **v4.1.0** — Custom functions with worker threads (predecessor to components) * **v4.2.0** — Component architecture introduced; Resource API, REST interface, MQTT, WebSockets, SSE, configurable schemas * **v4.3.0** — Component configuration improvements * **v4.6.0** — New extension API with dynamic reloading; Plugin API introduced (experimental) * **v4.7.0** — Component status monitoring; further plugin API improvements ## See Also * [Applications](/reference/v4/components/applications.md) — Managing and deploying applications * [Extension API](/reference/v4/components/extension-api.md) — Building custom extensions * [Plugin API](/reference/v4/components/plugin-api.md) — Building plugins (experimental, recommended for new extensions) * [Resource API](/reference/v4/resources/resource-api.md) — Resource class interface * [Database Schema](/reference/v4/database/schema.md) — Defining schemas with graphqlSchema --- # Plugin API *Added in: v4.6.0* (experimental) > The Plugin API is **experimental**. It is the recommended approach for building new extensions, and is intended to replace the [Extension API](/reference/v4/components/extension-api.md) in the future. Both systems are supported simultaneously. The Plugin API is a new iteration of the extension system that simplifies the interface. Instead of defining multiple methods (`start`, `startOnMainThread`, `handleFile`, `setupFile`, `handleDirectory`, `setupDirectory`), a plugin exports a single `handleApplication` method. ## Declaring a Plugin A plugin must specify a `pluginModule` option in `config.yaml` pointing to the plugin source: ``` pluginModule: plugin.js ``` For TypeScript or other compiled languages, point to the built output: ``` pluginModule: ./dist/index.js ``` It is recommended that plugins have a `package.json` with standard JavaScript package metadata (name, version, type, etc.). Plugins are standard JavaScript packages and can be published to npm, written in TypeScript, or export executables. ## Configuration General plugin configuration options: * `files` — `string | string[] | FilesOptionObject` *(optional)* — Glob pattern(s) for files and directories handled by the plugin's default `EntryHandler`. Pattern rules: * Cannot contain `..` or start with `/` * `.` or `./` is transformed to `**/*` automatically * `urlPath` — `string` *(optional)* — Base URL path prepended to resolved `files` entries. Cannot contain `..`. If starts with `./` or is `.`, the plugin name is automatically prepended * `timeout` — `number` *(optional)* — Timeout in milliseconds for plugin operations. Takes precedence over the plugin's `defaultTimeout` and the system default (30 seconds) ### File Entries ``` # Serve files from web/ at /static/ static: files: 'web/**/*' urlPath: '/static/' # Load only *.graphql files from src/schema/ graphqlSchema: files: 'src/schema/*.graphql' # Exclude a subdirectory static: files: source: 'web/**/*' ignore: 'web/images/**' ``` > Note: Unlike the Extension API, the Plugin API `files` object does **not** support an `only` field. Use `entryEvent.entryType` or `entryEvent.eventType` in your handler instead. ### Timeouts The system default timeout is **30 seconds**. If `handleApplication()` does not complete within this time, the component loader throws an error to prevent indefinite hanging. Plugins can override the system default by exporting a `defaultTimeout`: ``` export const defaultTimeout = 60_000; // 60 seconds ``` Users can override at the application level in `config.yaml`: ``` customPlugin: package: '@org/custom-plugin' files: 'foo.js' timeout: 45_000 # 45 seconds ``` ## TypeScript Support All classes and types are exported from the `harperdb` package: ``` import type { Scope, Config } from 'harperdb'; ``` ## API Reference ### Function: `handleApplication(scope: Scope): void | Promise` The only required export from a plugin module. The component loader executes it sequentially across all worker threads. It can be async and is awaited. Avoid event-loop-blocking operations within `handleApplication()`. ``` export function handleApplication(scope: Scope) { // Use scope to access config, resources, server, etc. } ``` Parameters: * `scope` — [`Scope`](#class-scope) — Access to the application's configuration, resources, and APIs The `handleApplication()` method cannot coexist with Extension API methods (`start`, `handleFile`, etc.). Defining both will throw an error. ### Class: `Scope` Extends [`EventEmitter`](https://nodejs.org/docs/latest/api/events.html#class-eventemitter) The central object passed to `handleApplication()`. Provides access to configuration, file entries, server APIs, and logging. #### Events * **`'close'`** — Emitted after `scope.close()` is called * **`'error'`** — `error: unknown` — An error occurred * **`'ready'`** — Emitted when the Scope is ready after loading the config file #### `scope.handleEntry([files][, handler])` Returns an [`EntryHandler`](#class-entryhandler) for watching and processing file system entries. Overloads: * `scope.handleEntry()` — Returns the default `EntryHandler` based on `files`/`urlPath` in `config.yaml` * `scope.handleEntry(handler)` — Returns default `EntryHandler`, registers `handler` for the `'all'` event * `scope.handleEntry(files)` — Returns a new `EntryHandler` for custom `files` config * `scope.handleEntry(files, handler)` — Returns a new `EntryHandler` with a custom `'all'` event handler Example: ``` export function handleApplication(scope) { // Default handler with inline callback scope.handleEntry((entry) => { switch (entry.eventType) { case 'add': case 'change': // handle file add/change break; case 'unlink': // handle file deletion break; } }); // Custom handler for specific files const tsHandler = scope.handleEntry({ files: 'src/**/*.ts' }); } ``` #### `scope.requestRestart()` Request a Harper restart. Does not restart immediately—indicates to the user that a restart is required. Called automatically if no `scope.options.on('change')` handler is defined or if a required handler is missing. #### `scope.resources` Returns: `Map` — Currently loaded [Resource](/reference/v4/resources/resource-api.md) instances. #### `scope.server` Returns: `server` — Reference to the [server](/reference/v4/http/api.md) global API. Use for registering HTTP middleware, custom networking, etc. #### `scope.options` Returns: [`OptionsWatcher`](#class-optionswatcher) — Access to the application's configuration options. Emits `'change'` events when the plugin's section of the config file is modified. #### `scope.logger` Returns: `logger` — Scoped logger instance. Recommended over the global `logger`. #### `scope.name` Returns: `string` — The plugin name as configured in `config.yaml`. #### `scope.directory` Returns: `string` — Root directory of the application component (where `config.yaml` lives). #### `scope.close()` Closes all associated entry handlers and the `scope.options` instance, emits `'close'`, and removes all listeners. ### Class: `OptionsWatcher` Extends [`EventEmitter`](https://nodejs.org/docs/latest/api/events.html#class-eventemitter) Provides reactive access to plugin configuration options, scoped to the specific plugin within the application's `config.yaml`. #### Events * **`'change'`** — `key: string[], value: ConfigValue, config: ConfigValue` — Emitted when a config option changes * `key` — Option key split into parts (e.g., `foo.bar` → `['foo', 'bar']`) * `value` — New value * `config` — Entire plugin configuration object * **`'close'`** — Emitted when the watcher is closed * **`'error'`** — `error: unknown` — An error occurred * **`'ready'`** — `config: ConfigValue | undefined` — Emitted on initial load and after `'remove'` recovery * **`'remove'`** — Config was removed (file deleted, config key deleted, or parse failure) Example: ``` export function handleApplication(scope) { scope.options.on('change', (key, value, config) => { if (key[0] === 'files') { scope.logger.info(`Files option changed to: ${value}`); } }); } ``` #### `options.get(key: string[]): ConfigValue | undefined` Get the value at a specific config key path. #### `options.getAll(): ConfigValue | undefined` Get the entire plugin configuration object. #### `options.getRoot(): Config | undefined` Get the root `config.yaml` object (all plugins and options). #### `options.close()` Close the watcher, preventing further events. ### Class: `EntryHandler` Extends [`EventEmitter`](https://nodejs.org/docs/latest/api/events.html#class-eventemitter) Created by [`scope.handleEntry()`](#scopehandleentry). Watches file system entries matching a `files` glob pattern and emits events as files are added, changed, or removed. #### Events * **`'all'`** — `entry: FileEntryEvent | DirectoryEntryEvent` — Emitted for all entry events (add, change, unlink, addDir, unlinkDir). This is the event registered by the `scope.handleEntry(handler)` shorthand. * **`'add'`** — `entry: AddFileEvent` — File created or first seen * **`'addDir'`** — `entry: AddDirectoryEvent` — Directory created or first seen * **`'change'`** — `entry: ChangeFileEvent` — File modified * **`'close'`** — Entry handler closed * **`'error'`** — `error: unknown` — An error occurred * **`'ready'`** — Handler ready and watching * **`'unlink'`** — `entry: UnlinkFileEvent` — File deleted * **`'unlinkDir'`** — `entry: UnlinkDirectoryEvent` — Directory deleted Recommended pattern for handling all events: ``` scope.handleEntry((entry) => { switch (entry.eventType) { case 'add': break; case 'change': break; case 'unlink': break; case 'addDir': break; case 'unlinkDir': break; } }); ``` #### `entryHandler.name` Returns: `string` — Plugin name. #### `entryHandler.directory` Returns: `string` — Application root directory. #### `entryHandler.close()` Closes the entry handler, removing all listeners. Can be restarted with `update()`. #### `entryHandler.update(config: FilesOption | FileAndURLPathConfig)` Update the handler to watch new entries. Closes and recreates the underlying watcher while preserving existing listeners. Returns a Promise that resolves when the updated handler is ready. ### Interfaces #### `FilesOption` `string | string[] | FilesOptionObject` #### `FilesOptionObject` * `source` — `string | string[]` *(required)* — Glob pattern(s) * `ignore` — `string | string[]` *(optional)* — Patterns to exclude #### `FileAndURLPathConfig` * `files` — `FilesOption` *(required)* * `urlPath` — `string` *(optional)* #### `BaseEntry` * `stats` — `fs.Stats | undefined` — File system stats (may be absent depending on event, entry type, and platform) * `urlPath` — `string` — URL path of the entry, resolved from `files` + `urlPath` options * `absolutePath` — `string` — Absolute filesystem path #### `FileEntry` Extends `BaseEntry` * `contents` — `Buffer` — File contents (automatically read) #### `EntryEvent` Extends `BaseEntry` * `eventType` — `string` — Type of event * `entryType` — `'file' | 'directory'` — Entry type #### `AddFileEvent` * `eventType: 'add'` * `entryType: 'file'` * Extends `EntryEvent`, `FileEntry` #### `ChangeFileEvent` * `eventType: 'change'` * `entryType: 'file'` * Extends `EntryEvent`, `FileEntry` #### `UnlinkFileEvent` * `eventType: 'unlink'` * `entryType: 'file'` * Extends `EntryEvent`, `FileEntry` #### `FileEntryEvent` `AddFileEvent | ChangeFileEvent | UnlinkFileEvent` #### `AddDirectoryEvent` * `eventType: 'addDir'` * `entryType: 'directory'` * Extends `EntryEvent` #### `UnlinkDirectoryEvent` * `eventType: 'unlinkDir'` * `entryType: 'directory'` * Extends `EntryEvent` #### `DirectoryEntryEvent` `AddDirectoryEvent | UnlinkDirectoryEvent` #### `Config` `{ [key: string]: ConfigValue }` Parsed representation of `config.yaml`. #### `ConfigValue` `string | number | boolean | null | undefined | ConfigValue[] | Config` #### `onEntryEventHandler` `(entryEvent: FileEntryEvent | DirectoryEntryEvent): void` Function signature for the `'all'` event handler passed to `scope.handleEntry()`. ## Example: Static File Server Plugin A simplified form of the built-in `static` extension demonstrating key Plugin API patterns: ``` export function handleApplication(scope) { const staticFiles = new Map(); // React to config changes scope.options.on('change', (key, value, config) => { if (key[0] === 'files' || key[0] === 'urlPath') { staticFiles.clear(); scope.logger.info(`Static files reset due to change in ${key.join('.')}`); } }); // Handle file entry events scope.handleEntry((entry) => { if (entry.entryType === 'directory') return; switch (entry.eventType) { case 'add': case 'change': staticFiles.set(entry.urlPath, entry.contents); break; case 'unlink': staticFiles.delete(entry.urlPath); break; } }); // Register HTTP middleware scope.server.http( (req, next) => { if (req.method !== 'GET') return next(req); const file = staticFiles.get(req.pathname); return file ? { statusCode: 200, body: file } : { statusCode: 404, body: 'File not found' }; }, { runFirst: true } ); } ``` ## Version History * **v4.6.0** — Plugin API introduced (experimental) * **v4.7.0** — Further improvements to the Plugin API --- # Configuration Operations Operations API endpoints for reading and modifying Harper configuration. *All operations in this section are restricted to `super_user` roles.* For the full list of configurable options, see [Configuration Options](/reference/v4/configuration/options.md). *** ## Set Configuration Modifies one or more Harper configuration parameters. **Requires a [restart](/reference/v4/operations-api/operations.md#restart) or [restart\_service](/reference/v4/operations-api/operations.md#restart_service) to take effect.** `operation` *(required)* — must be `set_configuration` Additional properties correspond to configuration keys in underscore-separated format (e.g. `logging_level` for `logging.level`, `clustering_enabled` for `clustering.enabled`). ### Body ``` { "operation": "set_configuration", "logging_level": "trace", "clustering_enabled": true } ``` ### Response: 200 ``` { "message": "Configuration successfully set. You must restart HarperDB for new config settings to take effect." } ``` *** ## Get Configuration Returns the current Harper configuration. `operation` *(required)* — must be `get_configuration` ### Body ``` { "operation": "get_configuration" } ``` ### Response: 200 ``` { "http": { "compressionThreshold": 1200, "cors": false, "corsAccessList": [null], "keepAliveTimeout": 30000, "port": 9926, "securePort": null, "timeout": 120000 }, "threads": 11, "authentication": { "cacheTTL": 30000, "enableSessions": true, "operationTokenTimeout": "1d", "refreshTokenTimeout": "30d" }, "analytics": { "aggregatePeriod": 60 }, "replication": { "hostname": "node1", "databases": "*", "routes": null, "url": "wss://127.0.0.1:9925" }, "componentsRoot": "/Users/hdb/components", "localStudio": { "enabled": false }, "logging": { "auditAuthEvents": { "logFailed": false, "logSuccessful": false }, "auditLog": true, "auditRetention": "3d", "file": true, "level": "error", "root": "/Users/hdb/log", "rotation": { "enabled": false, "compress": false, "interval": null, "maxSize": null, "path": "/Users/hdb/log" }, "stdStreams": false }, "mqtt": { "network": { "port": 1883, "securePort": 8883 }, "webSocket": true, "requireAuthentication": true }, "operationsApi": { "network": { "cors": true, "corsAccessList": ["*"], "domainSocket": "/Users/hdb/operations-server", "port": 9925, "securePort": null } }, "rootPath": "/Users/hdb", "storage": { "writeAsync": false, "caching": true, "compression": false, "noReadAhead": true, "path": "/Users/hdb/database", "prefetchWrites": true }, "tls": { "privateKey": "/Users/hdb/keys/privateKey.pem" } } ``` --- # Configuration Options Quick reference for all `harperdb-config.yaml` top-level sections. For how to apply configuration (YAML file, environment variables, CLI, Operations API), see [Configuration Overview](/reference/v4/configuration/overview.md). *** ## `http` Configures the Harper component server (HTTP, REST API, WebSocket). See [HTTP Configuration](/reference/v4/http/configuration.md) for full details. ``` http: port: 9926 securePort: 4443 cors: true timeout: 120000 mtls: false logging: level: info path: ~/hdb/log/http.log ``` * `sessionAffinity` — Route requests from same client to same worker thread (`ip` or header name) * `compressionThreshold` — Response size threshold for Brotli compression; *Default*: `1200` (bytes) * `cors` — Enable CORS; *Default*: `true` * `corsAccessList` — Allowed domains for CORS requests * `corsAccessControlAllowHeaders` — `Access-Control-Allow-Headers` value for OPTIONS preflight * `headersTimeout` — Max wait for complete HTTP headers (ms); *Default*: `60000` * `maxHeaderSize` — Max HTTP header size (bytes); *Default*: `16394` * `requestQueueLimit` — Max estimated request queue time (ms) before 503; *Default*: `20000` * `keepAliveTimeout` — Inactivity before closing keep-alive connection (ms); *Default*: `30000` * `port` — HTTP port; *Default*: `9926` * `securePort` — HTTPS port; requires [TLS configuration](/reference/v4/http/tls.md); *Default*: `null` * `http2` — Enable HTTP/2; *Default*: `false` (Added in: v4.5.0) * `timeout` — Request timeout (ms); *Default*: `120000` * `mtls` — Enable [mTLS authentication](/reference/v4/security/mtls-authentication.md) for incoming connections; sub-options: `user`, `required`, `certificateVerification` (see [Certificate Verification](/reference/v4/security/certificate-verification.md)) * `logging` — HTTP request logging (disabled by default, Added in: v4.6.0); sub-options: `level`, `path`, `timing`, `headers`, `id`. See [Logging Configuration](/reference/v4/logging/configuration.md) *** ## `threads` Worker thread pool configuration. ``` threads: count: 11 maxHeapMemory: 300 ``` * `count` — Number of worker threads; *Default*: CPU count minus one * `maxHeapMemory` — Heap limit per thread (MB) * `heapSnapshotNearLimit` — Take heap snapshot when approaching limit * `debug` — Enable debugging; sub-options: `port`, `startingPort`, `host`, `waitForDebugger` *** ## `authentication` Authentication and session configuration. Added in: v4.1.0; `enableSessions` added in v4.2.0. See [Authentication Configuration](/reference/v4/security/configuration.md). ``` authentication: authorizeLocal: true cacheTTL: 30000 enableSessions: true operationTokenTimeout: 1d refreshTokenTimeout: 30d ``` * `authorizeLocal` — Auto-authorize loopback requests as superuser; *Default*: `true` * `cacheTTL` — Session cache duration (ms); *Default*: `30000` * `enableSessions` — Cookie-based sessions; *Default*: `true` * `operationTokenTimeout` — Access token lifetime; *Default*: `1d` * `refreshTokenTimeout` — Refresh token lifetime; *Default*: `1d` * `logging` — Authentication event logging (Added in: v4.6.0); sub-options: `path`, `level`, `tag`, `stdStreams`. See [Logging Configuration](/reference/v4/logging/configuration.md) *** ## `operationsApi` Harper Operations API endpoint configuration. See [Operations API Overview](/reference/v4/operations-api/overview.md). ``` operationsApi: network: port: 9925 cors: true tls: certificate: ~/hdb/keys/certificate.pem privateKey: ~/hdb/keys/privateKey.pem ``` * `network.cors` / `network.corsAccessList` — CORS settings * `network.domainSocket` — Unix socket path for CLI communication; *Default*: `/hdb/operations-server` * `network.headersTimeout` / `network.keepAliveTimeout` / `network.timeout` — Timeout settings (ms) * `network.port` — Operations API port; *Default*: `9925` * `network.securePort` — HTTPS port; *Default*: `null` * `tls` — TLS override for the Operations API; sub-options: `certificate`, `certificateAuthority`, `privateKey`. See [`tls`](#tls) *** ## `tls` Global TLS configuration for HTTPS and TLS sockets (used by HTTP and MQTT). Can be a single object or an array for SNI. See [TLS](/reference/v4/http/tls.md) and [Certificate Management](/reference/v4/security/certificate-management.md). ``` tls: certificate: ~/hdb/keys/certificate.pem certificateAuthority: ~/hdb/keys/ca.pem privateKey: ~/hdb/keys/privateKey.pem ``` * `certificate` — Path to TLS certificate; *Default*: `/keys/certificate.pem` * `certificateAuthority` — Path to CA file; *Default*: `/keys/ca.pem` * `privateKey` — Path to private key; *Default*: `/keys/privateKey.pem` * `ciphers` — Allowed TLS cipher suites *** ## `mqtt` MQTT protocol configuration. Added in: v4.2.0. See [MQTT Configuration](/reference/v4/mqtt/configuration.md). ``` mqtt: network: port: 1883 securePort: 8883 webSocket: true requireAuthentication: true ``` * `network.port` — Insecure MQTT port; *Default*: `1883` * `network.securePort` — Secure MQTT port; *Default*: `8883` * `network.mtls` — Enable [mTLS](/reference/v4/security/mtls-authentication.md) for MQTT connections; sub-options: `user`, `required`, `certificateAuthority`, `certificateVerification` * `webSocket` — Enable MQTT over WebSocket on HTTP port; *Default*: `true` * `requireAuthentication` — Require credentials or mTLS; *Default*: `true` * `logging` — MQTT event logging (Added in: v4.6.0); sub-options: `path`, `level`, `tag`, `stdStreams`. See [Logging Configuration](/reference/v4/logging/configuration.md) *** ## `logging` Application logging. Added in: v4.1.0; per-component logging added in v4.6.0. See [Logging Configuration](/reference/v4/logging/configuration.md). ``` logging: level: warn root: ~/hdb/log stdStreams: false auditLog: false rotation: interval: 1D maxSize: 100M ``` * `level` — Log verbosity (`trace` → `debug` → `info` → `warn` → `error` → `fatal` → `notify`); *Default*: `warn` * `file` — Write to file; *Default*: `true` * `root` — Log directory; *Default*: `/log` * `path` — Explicit log file path (overrides `root`) * `stdStreams` — Write to stdout/stderr; *Default*: `false` * `console` — Include `console.*` output; *Default*: `true` * `auditLog` — Enable table transaction audit logging; *Default*: `false` * `auditRetention` — Audit log retention duration; *Default*: `3d` * `external` — Logging for components using the logger API; sub-options: `level`, `path` * `rotation.enabled` / `rotation.compress` / `rotation.interval` / `rotation.maxSize` / `rotation.path` — Log file rotation (activates when `interval` or `maxSize` is set) * `auditAuthEvents.logFailed` / `auditAuthEvents.logSuccessful` — Log failed/successful authentication events; *Default*: `false` *** ## `replication` Native WebSocket-based replication (Plexus). Added in: v4.4.0. See [Replication](/reference/v4/replication/overview.md) and [Clustering](/reference/v4/replication/clustering.md). ``` replication: hostname: server-one url: wss://server-one:9933 databases: '*' routes: - wss://server-two:9933 ``` * `hostname` — This instance's hostname within the cluster * `url` — WebSocket URL peers use to connect to this instance * `databases` — Databases to replicate; *Default*: `"*"` (all). Each entry supports `name` and `sharded` * `routes` — Peer nodes; URL strings or `{hostname, port, startTime, revokedCertificates}` objects * `port` — Replication port * `securePort` — Secure replication port; *Default*: `9933` (changed from `9925` in v4.5.0) * `enableRootCAs` — Verify against Node.js Mozilla CA store; *Default*: `true` * `blobTimeout` — Blob transfer timeout (ms); *Default*: `120000` * `failOver` — Failover to alternate node if peer unreachable; *Default*: `true` * `shard` — Shard ID for traffic routing; see [Sharding](/reference/v4/replication/sharding.md) * `mtls.certificateVerification` — Certificate revocation checking (CRL/OCSP) for replication connections; see [Certificate Verification](/reference/v4/security/certificate-verification.md) * `logging` — Replication event logging; sub-options: `path`, `level`, `tag`, `stdStreams`. See [Logging Configuration](/reference/v4/logging/configuration.md) *** ## `storage` Database storage configuration. See [Database Overview](/reference/v4/database/overview.md) and [Compaction](/reference/v4/database/compaction.md). ``` storage: path: ~/hdb/database caching: true compression: true compactOnStart: false ``` * `writeAsync` — Disable disk sync for higher throughput (**disables durability guarantees**); *Default*: `false` * `caching` — In-memory record caching; *Default*: `true` * `compression` — LZ4 record compression; *Default*: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold` * `compactOnStart` — Compact all non-system databases on startup; *Default*: `false` (Added in: v4.3.0) * `compactOnStartKeepBackup` — Retain compaction backups; *Default*: `false` * `maxTransactionQueueTime` — Max write queue time before 503; *Default*: `45s` * `noReadAhead` — Advise OS against read-ahead; *Default*: `false` * `prefetchWrites` — Prefetch before write transactions; *Default*: `true` * `path` — Database files directory; *Default*: `/database` * `blobPaths` — Blob storage directory or directories; *Default*: `/blobs` (Added in: v4.5.0) * `pageSize` — Database page size (bytes); *Default*: OS default * `reclamation.threshold` / `reclamation.interval` / `reclamation.evictionFactor` — Background storage reclamation settings (Added in: v4.5.0) *** ## `databases` Per-database and per-table file path overrides. Must be set before the database/table is created. See [Database Overview](/reference/v4/database/overview.md). ``` databases: myDatabase: path: /data/myDatabase auditPath: /data/myDatabase-audit tables: myTable: path: /data/myTable ``` * `.path` — Database files directory * `.auditPath` — Audit log directory for this database * `.tables..path` — Table files directory *** ## `analytics` Analytics aggregation configuration. See [Analytics Overview](/reference/v4/analytics/overview.md). ``` analytics: aggregatePeriod: 60 replicate: false ``` * `aggregatePeriod` — Aggregation interval (seconds); *Default*: `60` (Added in: v4.5.0) * `replicate` — Replicate analytics data across cluster; *Default*: `false` *** ## `localStudio` Local Harper Studio GUI. See [Studio](/reference/v4/studio/overview.md). ``` localStudio: enabled: true ``` * `enabled` — Enable local Studio at `http://localhost:`; *Default*: `false` *** ## `componentsRoot` Path to local component files. Added in: v4.2.0 (previously `customFunctionsRoot`). See [Components](/reference/v4/components/overview.md). ``` componentsRoot: ~/hdb/components ``` *** ## `rootPath` Root directory for all Harper persistent data, config, logs, and components. ``` rootPath: /var/lib/harper ``` *** ## Component Configuration Installed components are configured directly at the root of `harperdb-config.yaml` using the component name as the key — not nested under a `components:` section. See [Components](/reference/v4/components/overview.md). ``` my-component: package: 'HarperDB-Add-Ons/my-component' port: 4321 ``` * `.package` — NPM package name, GitHub repo (`user/repo`), or local path * `.port` — Port for the component; *Default*: value of `http.port` --- # Configuration Harper is configured through a [YAML](https://yaml.org/) file called `harperdb-config.yaml` located in the Harper root directory. By default the root directory is a folder named `hdb` in the home directory of the current user. Some configuration values are pre-populated in the config file on install, regardless of whether they are used. For a complete reference of all available configuration options, see [Configuration Options](/reference/v4/configuration/options.md). *** ## The Configuration File To change a configuration value, edit `harperdb-config.yaml` and save. **Harper must be restarted for changes to take effect.** Configuration keys use camelCase (e.g. `operationsApi`). Nested keys use dot notation conceptually (e.g. `operationsApi.network.port`). *** ## Setting Configuration Values All configuration values can be set through four mechanisms: ### 1. YAML File (direct edit) Edit `harperdb-config.yaml` directly: ``` http: port: 9926 logging: level: warn ``` ### 2. Environment Variables Map YAML keys to `SCREAMING_SNAKE_CASE`. Use underscores for nesting. Keys are case-insensitive. Examples: * `http.port` → `HTTP_PORT=9926` * `logging.rotation.enabled` → `LOGGING_ROTATION_ENABLED=false` * `operationsApi.network.port` → `OPERATIONSAPI_NETWORK_PORT=9925` ``` HTTP_PORT=9926 harperdb ``` > **Note:** Component configuration cannot be set via environment variables or CLI arguments. ### 3. CLI Arguments Same naming convention as environment variables, prefixed with `--`: ``` harperdb --HTTP_PORT 9926 --LOGGING_LEVEL warn ``` ### 4. Operations API Use `set_configuration` with underscore-separated key paths: ``` { "operation": "set_configuration", "http_port": 9926, "logging_level": "warn" } ``` See [Configuration Operations](/reference/v4/configuration/operations.md) for the full `set_configuration` and `get_configuration` API reference. *** ## Custom Config File Path To specify a custom config file location at install time, use the `HDB_CONFIG` variable: ``` # Use a custom config file path HDB_CONFIG=/path/to/custom-config.yaml harperdb # Install over an existing config HDB_CONFIG=/existing/rootpath/harperdb-config.yaml harperdb ``` *** ## Environment Variable-Based Configuration *Added in: v4.7.2* Harper provides two special environment variables for managing configuration across deployments: `HARPER_DEFAULT_CONFIG` and `HARPER_SET_CONFIG`. Both accept JSON-formatted configuration that mirrors the structure of `harperdb-config.yaml`. ``` export HARPER_DEFAULT_CONFIG='{"http":{"port":8080},"logging":{"level":"info"}}' export HARPER_SET_CONFIG='{"authentication":{"enabled":true}}' ``` ### HARPER\_DEFAULT\_CONFIG Provides default configuration values while respecting user modifications. Ideal for supplying sensible defaults without preventing administrators from customizing their instances. **At installation time:** * Overrides template default values * Respects values set by `HARPER_SET_CONFIG` * Respects values from existing config files (when using `HDB_CONFIG`) **At runtime:** * Only updates values it originally set * Detects and respects manual user edits to the config file * When a key is removed from the variable, the original value is restored **Example:** ``` export HARPER_DEFAULT_CONFIG='{"http":{"port":8080},"logging":{"level":"info"}}' harperdb # If an administrator manually changes the port to 9000, Harper will # detect this edit and respect it on subsequent restarts. # If http.port is removed from HARPER_DEFAULT_CONFIG later, # the port reverts to the original template default (9926). ``` ### HARPER\_SET\_CONFIG Forces configuration values that cannot be overridden by user edits. Designed for security policies, compliance requirements, or critical operational settings. **At runtime:** * Always overrides all other configuration sources * Takes precedence over user edits, file values, and `HARPER_DEFAULT_CONFIG` * When a key is removed from the variable, it is deleted from the config (not restored) **Example:** ``` export HARPER_SET_CONFIG='{"authentication":{"enabled":true},"logging":{"level":"error","stdStreams":true}}' harperdb # Any change to these values in harperdb-config.yaml will be # overridden on the next restart. ``` ### Combining Both Variables ``` # Provide sensible defaults (can be overridden by admins) export HARPER_DEFAULT_CONFIG='{"http":{"port":8080,"cors":true},"logging":{"level":"info"}}' # Enforce critical settings (cannot be changed) export HARPER_SET_CONFIG='{"authentication":{"enabled":true}}' ``` ### Configuration Precedence From highest to lowest: 1. **`HARPER_SET_CONFIG`** — Always wins 2. **User manual edits** — Detected via drift detection 3. **`HARPER_DEFAULT_CONFIG`** — Applied if no user edits detected 4. **File defaults** — Original template values ### State Tracking Harper maintains a state file at `{rootPath}/backup/.harper-config-state.json` to track the source of each configuration value. This enables: * **Drift detection**: Identifying when users manually edit values set by `HARPER_DEFAULT_CONFIG` * **Restoration**: Restoring original values when keys are removed from `HARPER_DEFAULT_CONFIG` * **Conflict resolution**: Determining which source should take precedence ### Format Reference The JSON structure mirrors the YAML config file: **YAML:** ``` http: port: 8080 cors: true logging: level: info rotation: enabled: true ``` **Environment variable (JSON):** ``` { "http": { "port": 8080, "cors": true }, "logging": { "level": "info", "rotation": { "enabled": true } } } ``` ### Important Notes * Both variables must contain valid JSON matching the structure of `harperdb-config.yaml` * Invalid values are caught by Harper's configuration validator at startup * Changes to these variables require a Harper restart to take effect * The state file is per-instance (stored in the root path) --- # API Harper exposes a set of global variables and functions that JavaScript code (in components, applications, and plugins) can use to interact with the database system. ## `tables` `tables` is an object whose properties are the tables in the default database (`data`). Each table defined in your `schema.graphql` file is available as a property, and the value is the table class that implements the [Resource API](/reference/v4/resources/resource-api.md). ``` # schema.graphql type Product @table { id: Long @primaryKey name: String price: Float } ``` ``` const { Product } = tables; // same as: databases.data.Product ``` ### Example ``` // Create a new record (id auto-generated) const created = await Product.create({ name: 'Shirt', price: 9.5 }); // Modify the record await Product.patch(created.id, { price: Math.round(created.price * 0.8 * 100) / 100 }); // Retrieve by primary key const record = await Product.get(created.id); // Query with conditions const query = { conditions: [{ attribute: 'price', comparator: 'less_than', value: 8.0 }], }; for await (const record of Product.search(query)) { // ... } ``` For the full set of methods available on table classes, see the [Resource API](/reference/v4/resources/resource-api.md). ## `databases` `databases` is an object whose properties are Harper databases. Each database contains its tables as properties, the same way `tables` does for the default database. In fact, `databases.data === tables` is always true. Use `databases` when you need to access tables from a non-default database. ### Example ``` const { Product } = databases.data; // default database const Events = databases.analytics.Events; // another database // Create an event record const event = await Events.create({ eventType: 'login', timestamp: Date.now() }); // Query events for await (const e of Events.search({ conditions: [{ attribute: 'eventType', value: 'login' }] })) { // handle each event } ``` To define tables in a non-default database, use the `database` argument on the `@table` directive in your schema: ``` type Events @table(database: "analytics") { id: Long @primaryKey eventType: String @indexed } ``` See [Schema](/reference/v4/database/schema.md) for full schema definition syntax. ## `transaction(context?, callback)` `transaction` executes a callback within a database transaction and returns a promise that resolves when the transaction commits. The callback may be async. ``` transaction(context?: object, callback: (txn: Transaction) => any | Promise): Promise ``` For most operations — HTTP request handlers, for example — Harper automatically starts a transaction. Use `transaction()` explicitly when your code runs outside of a natural transaction context, such as in timers or background jobs. ### Basic Usage ``` import { tables } from 'harperdb'; const { MyTable } = tables; if (isMainThread) { setInterval(async () => { let data = await (await fetch('https://example.com/data')).json(); transaction(async (txn) => { for (let item of data) { await MyTable.put(item, txn); } }); }, 3600000); // every hour } ``` ### Nesting If `transaction()` is called with a context that already has an active transaction, it reuses that transaction, executes the callback immediately, and returns. This makes `transaction()` safe to call defensively to ensure a transaction is always active. ### Transaction Object The callback receives a `txn` object with the following members: | Member | Type | Description | | --------------------- | --------------- | ------------------------------------------------------ | | `commit()` | `() => Promise` | Commits the current transaction | | `abort()` | `() => void` | Aborts the transaction and resets it | | `resetReadSnapshot()` | `() => void` | Resets the read snapshot to the latest committed state | | `timestamp` | `number` | Timestamp associated with the current transaction | On normal callback completion the transaction is committed automatically. If the callback throws, the transaction is aborted. ### Transaction Scope and Atomicity Transactions span a single database. All tables within the same database share a single transactional context: reads return a consistent snapshot, and writes across multiple tables are committed atomically. If code accesses tables in different databases, each database gets its own transaction with no cross-database atomicity guarantee. For deeper background on Harper's transaction model, see [Storage Algorithm](/reference/v4/database/storage-algorithm.md). ## `createBlob(data, options?)` *Added in: v4.5.0* `createBlob` creates a [`Blob`](https://developer.mozilla.org/en-US/docs/Web/API/Blob) backed by Harper's storage engine. Use it to store large binary content (images, audio, video, large HTML, etc.) in a `Blob`-typed schema field. ``` createBlob(data: Buffer | Uint8Array | ReadableStream | string, options?: BlobOptions): Blob ``` Harper's `Blob` extends the [Web API `Blob`](https://developer.mozilla.org/en-US/docs/Web/API/Blob), so standard methods (`.text()`, `.arrayBuffer()`, `.stream()`, `.slice()`, `.bytes()`) are all available. Unlike `Bytes`, blobs are stored separately from the record, support streaming, and do not need to fit in memory. ### Basic Usage Declare a blob field in your schema (see [Schema — Blob Type](/reference/v4/database/schema.md#blob-type)): ``` type MyTable @table { id: Any! @primaryKey data: Blob } ``` Create and store a blob: ``` let blob = createBlob(largeBuffer); await MyTable.put({ id: 'my-record', data: blob }); ``` Retrieve blob data using standard `Blob` methods: ``` let record = await MyTable.get('my-record'); let buffer = await record.data.bytes(); // ArrayBuffer let text = await record.data.text(); // string let stream = record.data.stream(); // ReadableStream ``` ### Streaming `createBlob` supports streaming data in as data is streamed out — useful for large media where low-latency transmission from origin is critical: ``` let blob = createBlob(incomingStream); // blob exists, but data is still streaming to storage await MyTable.put({ id: 'my-record', data: blob }); let record = await MyTable.get('my-record'); // blob data is accessible as it arrives let outgoingStream = record.data.stream(); ``` Because blobs can be referenced before they are fully written, they are **not** ACID-compliant by default. Use `saveBeforeCommit: true` to wait for the full write before committing: ``` let blob = createBlob(stream, { saveBeforeCommit: true }); await MyTable.put({ id: 'my-record', data: blob }); // put() resolves only after blob is fully written and record is committed ``` ### `BlobOptions` | Option | Type | Default | Description | | ------------------ | --------- | ------- | ----------------------------------------------------------------------- | | `saveBeforeCommit` | `boolean` | `false` | Wait for the blob to be fully written before committing the transaction | ### Error Handling Blobs written from a stream can fail mid-stream after the record is committed. Register an error handler to respond to interrupted writes: ``` export class MyEndpoint extends MyTable { async get(target) { const record = await super.get(target); let blob = record.data; blob.on('error', () => { MyTable.invalidate(target); }); return { status: 200, headers: {}, body: blob }; } } ``` ### `size` Property Blobs created from a stream may not have `size` available immediately. Listen for the `size` event if you need it: ``` let blob = record.data; if (blob.size === undefined) { blob.on('size', (size) => { // called once size is determined }); } ``` ### Blob Coercion When a field is typed as `Blob` in the schema, any string or buffer assigned via `put`, `patch`, or `publish` is automatically coerced to a `Blob`. This means plain JSON HTTP bodies and MQTT messages work without manual `createBlob()` calls in most cases. ## Related Documentation * [Schema](/reference/v4/database/schema.md) — Defining tables and blob fields * [Resource API](/reference/v4/resources/resource-api.md) — Full table class method reference * [Transaction Logging](/reference/v4/database/transaction.md) — Audit log and transaction log for data change history * [Configuration](/reference/v4/configuration/options.md) — Blob storage path configuration --- # Compaction *Added in: v4.3.0* Database files grow over time as records are inserted, updated, and deleted. Deleted records and updated values leave behind free space (fragmentation) in the database file, which can increase file size and potentially affect performance. Compaction eliminates this free space, creating a smaller, contiguous database file. > **Note:** Compaction does not compress your data. It removes internal fragmentation to make the file smaller. To enable compression on a database, use compaction to copy the database with updated storage configuration applied. Compaction is also the mechanism to apply storage configuration changes (such as enabling compression) to existing databases, since some storage settings cannot be changed in-place. ## Copy Compaction Creates a compacted copy of a database file. The original database is left unchanged. > **Recommendation:** Stop Harper before performing copy compaction to prevent any record loss during the copy operation. Run using the [CLI](/reference/v4/cli/commands.md): ``` harperdb copy-db ``` The `source-database` is the database name (not a file path). The target is the full file path where the compacted copy will be written. To replace the original database with the compacted copy, move or rename the output file to the original database path after Harper is stopped. **Example — compact the default `data` database:** ``` harperdb copy-db data /home/user/hdb/database/copy.mdb ``` ## Compact on Start Automatically compacts all non-system databases when Harper starts. Harper will not start until compaction is complete. Under the hood, it loops through all user databases, creates a backup of each, compacts it, replaces the original with the compacted copy, and removes the backup. Configure in `harperdb-config.yaml`: ``` storage: compactOnStart: true compactOnStartKeepBackup: false ``` Using CLI environment variables: ``` STORAGE_COMPACTONSTART=true STORAGE_COMPACTONSTARTKEEPBACKUP=true harperdb ``` ### Options | Option | Type | Default | Description | | -------------------------- | ------- | ------- | ------------------------------------------------------------------------------- | | `compactOnStart` | Boolean | `false` | Compact all databases at startup. Automatically reset to `false` after running. | | `compactOnStartKeepBackup` | Boolean | `false` | Retain the backup copy created during compact on start | > **Note:** `compactOnStart` is automatically set back to `false` after it runs, so compaction only happens on the next start if you explicitly re-enable it. ## Related Documentation * [Storage Algorithm](/reference/v4/database/storage-algorithm.md) — How Harper stores data using LMDB * [CLI Commands](/reference/v4/cli/commands.md) — `copy-db` CLI command reference * [Configuration](/reference/v4/configuration/options.md) — Full storage configuration options including compression settings --- # Data Loader *Added in: v4.6.0* The Data Loader is a built-in component that loads data from JSON or YAML files into Harper tables as part of component deployment. It is designed for seeding tables with initial records — configuration data, reference data, default users, or other records that should exist when a component is first deployed or updated. ## Configuration In your component's `config.yaml`, use the `dataLoader` key to specify the data files to load: ``` dataLoader: files: 'data/*.json' ``` `dataLoader` is an [Extension](/reference/v4/components/extension-api.md) and supports the standard `files` configuration option, including glob patterns. ## Data File Format Each data file loads records into a single table. The file specifies the target database, table, and an array of records. ### JSON Example ``` { "database": "myapp", "table": "users", "records": [ { "id": 1, "username": "admin", "email": "admin@example.com", "role": "administrator" }, { "id": 2, "username": "user1", "email": "user1@example.com", "role": "standard" } ] } ``` ### YAML Example ``` database: myapp table: settings records: - id: 1 setting_name: app_name setting_value: My Application - id: 2 setting_name: version setting_value: '1.0.0' ``` One table per file. To load data into multiple tables, create a separate file for each table. ## File Patterns The `files` option accepts a single path, a list of paths, or a glob pattern: ``` # Single file dataLoader: files: 'data/seed-data.json' # Multiple specific files dataLoader: files: - 'data/users.json' - 'data/settings.yaml' - 'data/initial-products.json' # Glob pattern dataLoader: files: 'data/**/*.{json,yaml,yml}' ``` ## Loading Behavior The Data Loader runs on every full system start and every component deployment — this includes fresh installs, restarts of the Harper process or threads, and redeployments of the component. Because the Data Loader runs on every startup and deployment, change detection is central to how it works safely. On each run: 1. All specified data files are read (JSON or YAML) 2. Each file is validated to reference a single table 3. Records are inserted or updated based on content hash comparison: * New records are inserted if they don't exist * Existing records are updated only if the data file content has changed * Records created outside the Data Loader (via Operations API, REST, etc.) are never overwritten * Records modified by users after being loaded are preserved and not overwritten * Extra fields added by users to data-loaded records are preserved during updates 4. SHA-256 content hashes are stored in the [`hdb_dataloader_hash`](/reference/v4/database/system-tables.md#hdb_dataloader_hash) system table to track which records have been loaded and detect changes ### Change Detection | Scenario | Behavior | | -------------------------------------------------- | ------------------------------------------------ | | New record | Inserted; content hash stored | | Unchanged record | Skipped (no writes) | | Changed data file | Updated via `patch`, preserving any extra fields | | Record created by user (not data loader) | Never overwritten | | Record modified by user after load | Preserved, not overwritten | | Extra fields added by user to a data-loaded record | Preserved during updates | This design makes data files safe to redeploy repeatedly — across deployments, node scaling, and system restarts — without losing manual modifications or causing unnecessary writes. ## Best Practices **Define schemas first.** While the Data Loader can infer schemas from the records it loads, it is strongly recommended to define table schemas explicitly using the [graphqlSchema component](/reference/v4/database/schema.md) before loading data. This ensures proper types, constraints, and relationships. **One table per file.** Each data file must target a single table. Organize files accordingly. **Idempotent data.** Design files to be safe to load multiple times without creating duplicate or conflicting records. **Version control.** Include data files in version control for consistency across deployments and environments. **Environment-specific data.** Consider using different data files for different environments (development, staging, production) to avoid loading inappropriate records. **Validate before deploying.** Ensure data files are valid JSON or YAML and match your table schemas before deployment to catch type mismatches early. **No sensitive data.** Do not include passwords, API keys, or secrets directly in data files. Use environment variables or secure configuration management instead. ## Example Component Structure A common production use case is shipping reference data — lookup tables like countries and regions — as part of a component. The records are version-controlled alongside the code, consistent across every environment, and the data loader keeps them in sync on every deployment without touching any user-modified fields. ``` my-component/ ├── config.yaml ├── schemas.graphql ├── roles.yaml └── data/ ├── countries.json # ISO country codes — reference data, ships with component └── regions.json # region/subdivision codes ``` **`config.yaml`**: ``` graphqlSchema: files: 'schemas.graphql' roles: files: 'roles.yaml' dataLoader: files: 'data/*.json' rest: true ``` **`schemas.graphql`**: ``` type Country @table(database: "myapp") @export { id: String @primaryKey # ISO 3166-1 alpha-2, e.g. "US" name: String @indexed region: String @indexed } type Region @table(database: "myapp") @export { id: String @primaryKey # ISO 3166-2, e.g. "US-CA" name: String @indexed countryId: String @indexed country: Country @relationship(from: countryId) } ``` **`data/countries.json`**: ``` { "database": "myapp", "table": "Country", "records": [ { "id": "US", "name": "United States", "region": "Americas" }, { "id": "GB", "name": "United Kingdom", "region": "Europe" }, { "id": "DE", "name": "Germany", "region": "Europe" } // ... all ~250 ISO countries ] } ``` **`data/regions.json`**: ``` { "database": "myapp", "table": "Region", "records": [ { "id": "US-CA", "name": "California", "countryId": "US" }, { "id": "US-NY", "name": "New York", "countryId": "US" }, { "id": "GB-ENG", "name": "England", "countryId": "GB" } // ... ] } ``` Because the data loader uses content hashing, adding new countries or correcting a name in the file will update only the changed records on the next deployment — existing records that haven't changed are skipped entirely. ## Related Documentation * [Schema](/reference/v4/database/schema.md) — Defining table structure before loading data * [Jobs](/reference/v4/database/jobs.md) — Bulk data operations via the Operations API (CSV/JSON import from file, URL, or S3) * [Components](/reference/v4/components/overview.md) — Extension and plugin system that the data loader is built on --- # Jobs Harper uses an asynchronous job system for long-running data operations. When a bulk operation is initiated — such as loading a large CSV file or exporting millions of records — Harper starts a background job and immediately returns a job ID. Use the job ID to check progress and status. Job status values: * `IN_PROGRESS` — the job is currently running * `COMPLETE` — the job finished successfully ## Bulk Operations The following operations create jobs. All bulk operations are sent to the [Operations API](/reference/v4/operations-api/overview.md). ### CSV Data Load Ingests CSV data provided directly in the request body. * `operation` *(required)* — `csv_data_load` * `database` *(optional)* — target database; defaults to `data` * `table` *(required)* — target table * `action` *(optional)* — `insert`, `update`, or `upsert`; defaults to `insert` * `data` *(required)* — CSV content as a string ``` { "operation": "csv_data_load", "database": "dev", "action": "insert", "table": "breed", "data": "id,name,country\n1,Labrador,Canada\n2,Poodle,France\n" } ``` Response: ``` { "message": "Starting job with id 2fe25039-566e-4670-8bb3-2db3d4e07e69", "job_id": "2fe25039-566e-4670-8bb3-2db3d4e07e69" } ``` *** ### CSV File Load Ingests CSV data from a file on the server's local filesystem. > The CSV file must reside on the same machine running Harper. * `operation` *(required)* — `csv_file_load` * `database` *(optional)* — target database; defaults to `data` * `table` *(required)* — target table * `action` *(optional)* — `insert`, `update`, or `upsert`; defaults to `insert` * `file_path` *(required)* — absolute path to the CSV file on the host ``` { "operation": "csv_file_load", "action": "insert", "database": "dev", "table": "breed", "file_path": "/home/user/imports/breeds.csv" } ``` *** ### CSV URL Load Ingests CSV data from a URL. * `operation` *(required)* — `csv_url_load` * `database` *(optional)* — target database; defaults to `data` * `table` *(required)* — target table * `action` *(optional)* — `insert`, `update`, or `upsert`; defaults to `insert` * `csv_url` *(required)* — URL pointing to the CSV file ``` { "operation": "csv_url_load", "action": "insert", "database": "dev", "table": "breed", "csv_url": "https://s3.amazonaws.com/mydata/breeds.csv" } ``` *** ### Import from S3 Imports CSV or JSON files from an AWS S3 bucket. * `operation` *(required)* — `import_from_s3` * `database` *(optional)* — target database; defaults to `data` * `table` *(required)* — target table * `action` *(optional)* — `insert`, `update`, or `upsert`; defaults to `insert` * `s3` *(required)* — S3 connection details: * `aws_access_key_id` * `aws_secret_access_key` * `bucket` * `key` — filename including extension (`.csv` or `.json`) * `region` ``` { "operation": "import_from_s3", "action": "insert", "database": "dev", "table": "dog", "s3": { "aws_access_key_id": "YOUR_KEY", "aws_secret_access_key": "YOUR_SECRET_KEY", "bucket": "BUCKET_NAME", "key": "dogs.json", "region": "us-east-1" } } ``` *** ### Export Local Exports table data to a local file in JSON or CSV format. * `operation` *(required)* — `export_local` * `format` *(required)* — `json` or `csv` * `path` *(required)* — local directory path where the export file will be written * `search_operation` *(required)* — query to select records: `search_by_hash`, `search_by_value`, `search_by_conditions`, or `sql` *Changed in: v4.3.0* — `search_by_conditions` added as a supported search operation for exports * `filename` *(optional)* — filename without extension; auto-generated from epoch timestamp if omitted ``` { "operation": "export_local", "format": "json", "path": "/data/exports/", "search_operation": { "operation": "sql", "sql": "SELECT * FROM dev.breed" } } ``` *** ### Export to S3 Exports table data to an AWS S3 bucket in JSON or CSV format. *Changed in: v4.3.0* — `search_by_conditions` added as a supported search operation * `operation` *(required)* — `export_to_s3` * `format` *(required)* — `json` or `csv` * `s3` *(required)* — S3 connection details (same fields as Import from S3, plus `key` for the output object name) * `search_operation` *(required)* — `search_by_hash`, `search_by_value`, `search_by_conditions`, or `sql` ``` { "operation": "export_to_s3", "format": "json", "s3": { "aws_access_key_id": "YOUR_KEY", "aws_secret_access_key": "YOUR_SECRET_KEY", "bucket": "BUCKET_NAME", "key": "exports/dogs.json", "region": "us-east-1" }, "search_operation": { "operation": "sql", "sql": "SELECT * FROM dev.dog" } } ``` *** ### Delete Records Before Deletes records older than a given timestamp from a table. Operates only on the local node — clustered replicas retain their data. *Restricted to `super_user` roles.* * `operation` *(required)* — `delete_records_before` * `schema` *(required)* — database name * `table` *(required)* — table name * `date` *(required)* — records with `__createdtime__` before this timestamp are deleted. Format: `YYYY-MM-DDThh:mm:ss.sZ` ``` { "operation": "delete_records_before", "date": "2024-01-01T00:00:00.000Z", "schema": "dev", "table": "breed" } ``` ## Managing Jobs ### Get Job Returns status, metrics, and messages for a specific job by ID. * `operation` *(required)* — `get_job` * `id` *(required)* — job ID ``` { "operation": "get_job", "id": "4a982782-929a-4507-8794-26dae1132def" } ``` Response: ``` [ { "__createdtime__": 1611615798782, "__updatedtime__": 1611615801207, "created_datetime": 1611615798774, "end_datetime": 1611615801206, "id": "4a982782-929a-4507-8794-26dae1132def", "job_body": null, "message": "successfully loaded 350 of 350 records", "start_datetime": 1611615798805, "status": "COMPLETE", "type": "csv_url_load", "user": "HDB_ADMIN", "start_datetime_converted": "2021-01-25T23:03:18.805Z", "end_datetime_converted": "2021-01-25T23:03:21.206Z" } ] ``` *** ### Search Jobs by Start Date Returns all jobs started within a time window. *Restricted to `super_user` roles.* * `operation` *(required)* — `search_jobs_by_start_date` * `from_date` *(required)* — start of the search window (ISO 8601 format) * `to_date` *(required)* — end of the search window (ISO 8601 format) ``` { "operation": "search_jobs_by_start_date", "from_date": "2024-01-01T00:00:00.000+0000", "to_date": "2024-01-02T00:00:00.000+0000" } ``` ## Related Documentation * [Data Loader](/reference/v4/database/data-loader.md) — Component-based data loading as part of deployment * [Operations API](/reference/v4/operations-api/overview.md) — Sending operations to Harper * [Transaction Logging](/reference/v4/database/transaction.md) — Recording a history of changes made to tables --- # Database Harper's database system is the foundation of its data storage and retrieval capabilities. It is built on top of [LMDB](https://www.symas.com/lmdb) (Lightning Memory-Mapped Database) and is designed to provide high performance, ACID-compliant storage with automatic indexing and flexible schema support. ## How Harper Stores Data Harper organizes data in a three-tier hierarchy: * **Databases** — containers that group related tables together in a single transactional file * **Tables** — collections of records with a common data pattern * **Records** — individual data objects with a primary key and any number of attributes All tables within a database share the same transaction context, meaning reads and writes across tables in the same database can be performed atomically. ### The Schema System and Auto-REST The most common way to use Harper's database is through the **schema system**. By defining a [GraphQL schema](/reference/v4/database/schema.md), you can: * Declare tables and their attribute types * Control which attributes are indexed * Define relationships between tables * Automatically expose data via REST, MQTT, and other interfaces You do not need to build custom application code to use the database. A schema definition alone is enough to create fully functional, queryable REST endpoints for your data. For more advanced use cases, you can extend table behavior using the [Resource API](/reference/v4/resources/resource-api.md). ### Architecture Overview ``` ┌──────────┐ ┌──────────┐ │ Clients │ │ Clients │ └────┬─────┘ └────┬─────┘ │ │ ▼ ▼ ┌────────────────────────────────────────┐ │ │ │ Socket routing/management │ ├───────────────────────┬────────────────┤ │ │ │ │ Server Interfaces ─►│ Authentication │ │ RESTful HTTP, MQTT │ Authorization │ │ ◄─┤ │ │ ▲ └────────────────┤ │ │ │ │ ├───┼──────────┼─────────────────────────┤ │ │ │ ▲ │ │ ▼ Resources ▲ │ ┌───────────┐ │ │ │ └─┤ │ │ ├─────────────────┴────┐ │ App │ │ │ ├─►│ resources │ │ │ Database tables │ └───────────┘ │ │ │ ▲ │ ├──────────────────────┘ │ │ │ ▲ ▼ │ │ │ ┌────────────────┐ │ │ │ │ External │ │ │ │ │ data sources ├────┘ │ │ │ │ │ │ └────────────────┘ │ │ │ └────────────────────────────────────────┘ ``` ## Databases Harper databases hold a collection of tables in a single transactionally-consistent file. This means reads and writes can be performed atomically across all tables in the same database, and multi-table transactions are replicated as a single atomic unit. The default database is named `data`. Most applications will use this default. Additional databases can be created for namespace separation — this is particularly useful for components designed for reuse across multiple applications, where a unique database name avoids naming collisions. > **Note:** Transactions do not preserve atomicity across different databases, only across tables within the same database. ## Tables Tables group records with a common data pattern. A table must have: * **Table name** — used to identify the table * **Primary key** — the unique identifier for each record (also referred to as `hash_attribute` in the Operations API) Primary keys must be unique. If a primary key is not provided on insert, Harper auto-generates one: * A **UUID string** for primary keys typed as `String` or `ID` * An **auto-incrementing integer** for primary keys typed as `Int`, `Long`, or `Any` Numeric primary keys are more efficient than UUIDs for large tables. ## Dynamic vs. Defined Schemas Harper tables can operate in two modes: **Defined schemas** (recommended): Tables with schemas explicitly declared using [GraphQL schema syntax](/reference/v4/database/schema.md). This provides predictable structure, precise control over indexing, and data integrity. Schemas are declared in a component's `schema.graphql` file. **Dynamic schemas**: Tables created through the Operations API or Studio without a schema definition. Attributes are reflexively added as data is ingested. All top-level attributes are automatically indexed. Dynamic schema tables automatically maintain `__createdtime__` and `__updatedtime__` audit attributes on every record. It is best practice to define schemas for production tables. Dynamic schemas are convenient for experimentation and prototyping. ## Key Concepts For deeper coverage of each database feature, see the dedicated pages in this section: * **[Schema](/reference/v4/database/schema.md)** — Defining table structure, types, indexes, relationships, and computed properties using GraphQL schema syntax * **[API](/reference/v4/database/api.md)** — The `tables`, `databases`, `transaction()`, and `createBlob()` globals for interacting with the database from code * **[Data Loader](/reference/v4/database/data-loader.md)** — Loading seed or initial data into tables as part of component deployment * **[Storage Algorithm](/reference/v4/database/storage-algorithm.md)** — How Harper stores data using LMDB with universal indexing and ACID compliance * **[Jobs](/reference/v4/database/jobs.md)** — Asynchronous bulk data operations (CSV import/export, S3 import/export) * **[System Tables](/reference/v4/database/system-tables.md)** — Harper internal tables for analytics, data loader state, and other system features * **[Compaction](/reference/v4/database/compaction.md)** — Reducing database file size by eliminating fragmentation and free space * **[Transaction Logging](/reference/v4/database/transaction.md)** — Recording and querying a history of data changes via audit log and transaction log ## Related Documentation * [REST](/reference/v4/rest/overview.md) — HTTP interface built on top of the database resource system * [Resources](/reference/v4/resources/overview.md) — Custom application logic extending database tables * [Operations API](/reference/v4/operations-api/overview.md) — Direct database management operations (create/drop databases and tables, insert/update/delete records) * [Configuration](/reference/v4/configuration/overview.md) — Storage configuration options (compression, blob paths, compaction) --- # Schema Harper uses GraphQL Schema Definition Language (SDL) to declaratively define table structure. Schema definitions are loaded from `.graphql` files in a component directory and control table creation, attribute types, indexing, and relationships. ## Overview *Added in: v4.2.0* Schemas are defined using standard [GraphQL type definitions](https://graphql.org/learn/schema/) with Harper-specific directives. A schema definition: * Ensures required tables exist when a component is deployed * Enforces attribute types and required constraints * Controls which attributes are indexed * Defines relationships between tables * Configures computed properties, expiration, and audit behavior Schemas are flexible by default — records may include additional properties beyond those declared in the schema. Use the `@sealed` directive to prevent this. A minimal example: ``` type Dog @table { id: Long @primaryKey name: String breed: String age: Int } type Breed @table { id: Long @primaryKey name: String @indexed } ``` ### Loading Schemas In a component's `config.yaml`, specify the schema file with the `graphqlSchema` plugin: ``` graphqlSchema: files: 'schema.graphql' ``` Keep in mind that both plugins and applications can specify schemas. ## Type Directives Type directives apply to the entire table type definition. ### `@table` Marks a GraphQL type as a Harper database table. The type name becomes the table name by default. ``` type MyTable @table { id: Long @primaryKey } ``` Optional arguments: | Argument | Type | Default | Description | | ------------ | --------- | -------------- | ----------------------------------------------------------------------- | | `table` | `String` | type name | Override the table name | | `database` | `String` | `"data"` | Database to place the table in | | `expiration` | `Int` | — | Auto-expire records after this many seconds (useful for caching tables) | | `audit` | `Boolean` | config default | Enable audit log for this table | **Examples:** ``` # Override table name type Product @table(table: "products") { id: Long @primaryKey } # Place in a specific database type Order @table(database: "commerce") { id: Long @primaryKey } # Auto-expire records after 1 hour (e.g., a session cache) type Session @table(expiration: 3600) { id: Long @primaryKey userId: String } # Enable audit log for this table explicitly type AuditedRecord @table(audit: true) { id: Long @primaryKey value: String } # Combine multiple arguments type Event @table(database: "analytics", expiration: 86400) { id: Long @primaryKey name: String @indexed } ``` **Database naming:** Since all tables default to the `data` database, when designing plugins or applications, consider using unique database names to avoid table naming collisions. ### `@export` Exposes the table as an externally accessible resource endpoint, available via REST, MQTT, and other interfaces. ``` type MyTable @table @export(name: "my-table") { id: Long @primaryKey } ``` The optional `name` parameter specifies the URL path segment (e.g., `/my-table/`). Without `name`, the type name is used. ### `@sealed` Prevents records from including any properties beyond those explicitly declared in the type. By default, Harper allows records to have additional properties. ``` type StrictRecord @table @sealed { id: Long @primaryKey name: String } ``` ## Field Directives Field directives apply to individual attributes in a type definition. ### `@primaryKey` Designates the attribute as the table's primary key. Primary keys must be unique; inserts with a duplicate primary key are rejected. ``` type Product @table { id: Long @primaryKey name: String } ``` If no primary key is provided on insert, Harper auto-generates one: * **UUID string** — when type is `String` or `ID` * **Auto-incrementing integer** — when type is `Int`, `Long`, or `Any` *Changed in: v4.4.0* Auto-incrementing integer primary keys were added. Previously only UUID generation was supported for `ID` and `String` types. Using `Long` or `Any` is recommended for auto-generated numeric keys. `Int` is limited to 32-bit and may be insufficient for large tables. ### `@indexed` Creates a secondary index on the attribute for fast querying. Required for filtering by this attribute in REST queries, SQL, or NoSQL operations. ``` type Product @table { id: Long @primaryKey category: String @indexed price: Float @indexed } ``` If the field value is an array, each element in the array is individually indexed, enabling queries by any individual value. Null values are indexed by default (added in v4.3.0), enabling queries like `GET /Product/?category=null`. ### `@createdTime` Automatically assigns a creation timestamp (Unix epoch milliseconds) to the attribute when a record is created. ``` type Event @table { id: Long @primaryKey createdAt: Long @createdTime } ``` ### `@updatedTime` Automatically assigns a timestamp (Unix epoch milliseconds) each time the record is updated. ``` type Event @table { id: Long @primaryKey updatedAt: Long @updatedTime } ``` ## Relationships *Added in: v4.3.0* The `@relationship` directive defines how one table relates to another through a foreign key. Relationships enable join queries and allow related records to be selected as nested properties in query results. ### `@relationship(from: attribute)` — many-to-one or many-to-many The foreign key is in this table, referencing the primary key of the target table. ``` type RealityShow @table @export { id: Long @primaryKey networkId: Long @indexed # foreign key network: Network @relationship(from: networkId) # many-to-one title: String @indexed } type Network @table @export { id: Long @primaryKey name: String @indexed # e.g. "Bravo", "Peacock", "Netflix" } ``` Query shows by network name: ``` GET /RealityShow?network.name=Bravo ``` If the foreign key is an array, this establishes a many-to-many relationship (e.g., a show with multiple streaming homes): ``` type RealityShow @table @export { id: Long @primaryKey networkIds: [Long] @indexed networks: [Network] @relationship(from: networkIds) } ``` ### `@relationship(to: attribute)` — one-to-many or many-to-many The foreign key is in the target table, referencing the primary key of this table. The result type must be an array. ``` type Network @table @export { id: Long @primaryKey name: String @indexed # e.g. "Bravo", "Peacock", "Netflix" shows: [RealityShow] @relationship(to: networkId) # one-to-many # shows like "Real Housewives of Atlanta", "The Traitors", "Vanderpump Rules" } ``` ### `@relationship(from: attribute, to: attribute)` — foreign key to foreign key Both `from` and `to` can be specified together to define a relationship where neither side uses the primary key — a foreign key to foreign key join. This is useful for many-to-many relationships that join on non-primary-key attributes. ``` type OrderItem @table @export { id: Long @primaryKey orderId: Long @indexed productSku: Long @indexed product: Product @relationship(from: productSku, to: sku) # join on sku, not primary key } type Product @table @export { id: Long @primaryKey sku: Long @indexed name: String } ``` Schemas can also define self-referential relationships, enabling parent-child hierarchies within a single table. ## Computed Properties *Added in: v4.4.0* The `@computed` directive marks a field as derived from other fields at query time. Computed properties are not stored in the database but are evaluated when the field is accessed. ``` type Product @table { id: Long @primaryKey price: Float taxRate: Float totalPrice: Float @computed(from: "price + (price * taxRate)") } ``` The `from` argument is a JavaScript expression that can reference other record fields. Computed properties can also be defined in JavaScript for complex logic: ``` type Product @table { id: Long @primaryKey totalPrice: Float @computed } ``` ``` tables.Product.setComputedAttribute('totalPrice', (record) => { return record.price + record.price * record.taxRate; }); ``` Computed properties are not included in query results by default — use `select` to include them explicitly. ### Computed Indexes Computed properties can be indexed with `@indexed`, enabling custom indexing strategies such as composite indexes, full-text search, or vector indexing: ``` type Product @table { id: Long @primaryKey tags: String tagsSeparated: String[] @computed(from: "tags.split(/\\s*,\\s*/)") @indexed } ``` When using a JavaScript function for an indexed computed property, use the `version` argument to ensure re-indexing when the function changes: ``` type Product @table { id: Long @primaryKey totalPrice: Float @computed(version: 1) @indexed } ``` Increment `version` whenever the computation function changes. Failing to do so can result in an inconsistent index. ## Vector Indexing *Added in: v4.6.0* Use `@indexed(type: "HNSW")` to create a vector index using the Hierarchical Navigable Small World algorithm, designed for fast approximate nearest-neighbor search on high-dimensional vectors. ``` type Document @table { id: Long @primaryKey textEmbeddings: [Float] @indexed(type: "HNSW") } ``` Query by nearest neighbors using the `sort` parameter: ``` let results = Document.search({ sort: { attribute: 'textEmbeddings', target: searchVector }, limit: 5, }); ``` HNSW can be combined with filter conditions: ``` let results = Document.search({ conditions: [{ attribute: 'price', comparator: 'lt', value: 50 }], sort: { attribute: 'textEmbeddings', target: searchVector }, limit: 5, }); ``` ### HNSW Parameters | Parameter | Default | Description | | ---------------------- | ----------------- | --------------------------------------------------------------------------------------------------- | | `distance` | `"cosine"` | Distance function: `"euclidean"` or `"cosine"` (negative cosine similarity) | | `efConstruction` | `100` | Max nodes explored during index construction. Higher = better recall, lower = better performance | | `M` | `16` | Preferred connections per graph layer. Higher = more space, better recall for high-dimensional data | | `optimizeRouting` | `0.5` | Heuristic aggressiveness for omitting redundant connections (0 = off, 1 = most aggressive) | | `mL` | computed from `M` | Normalization factor for level generation | | `efSearchConstruction` | `50` | Max nodes explored during search | Example with custom parameters: ``` type Document @table { id: Long @primaryKey textEmbeddings: [Float] @indexed(type: "HNSW", distance: "euclidean", optimizeRouting: 0, efSearchConstruction: 100) } ``` ## Field Types Harper supports the following field types: | Type | Description | | --------- | ----------------------------------------------------------------------------------------------- | | `String` | Unicode text, UTF-8 encoded | | `Int` | 32-bit signed integer (−2,147,483,648 to 2,147,483,647) | | `Long` | 54-bit signed integer (−9,007,199,254,740,992 to 9,007,199,254,740,992) | | `Float` | 64-bit double precision floating point | | `BigInt` | Integer up to \~300 digits. Note: distinct JavaScript type; handle appropriately in custom code | | `Boolean` | `true` or `false` | | `ID` | String; indicates a non-human-readable identifier | | `Any` | Any primitive, object, or array | | `Date` | JavaScript `Date` object | | `Bytes` | Binary data as `Buffer` or `Uint8Array` | | `Blob` | Binary large object; designed for streaming content >20KB | Added `BigInt` in v4.3.0 Added `Blob` in v4.5.0 Arrays of a type are expressed with `[Type]` syntax (e.g., `[Float]` for a vector). ### Blob Type *Added in: v4.5.0* `Blob` fields are designed for large binary content. Harper's `Blob` type implements the [Web API `Blob` interface](https://developer.mozilla.org/en-US/docs/Web/API/Blob), so all standard `Blob` methods (`.text()`, `.arrayBuffer()`, `.stream()`, `.slice()`) are available. Unlike `Bytes`, blobs are stored separately from the record, support streaming, and do not need to be held entirely in memory. Use `Blob` for content typically larger than 20KB (images, video, audio, large HTML, etc.). See [Blob usage details](#blob-usage) below. #### Blob Usage Declare a blob field: ``` type MyTable @table { id: Any! @primaryKey data: Blob } ``` Create and store a blob using [`createBlob()`](/reference/v4/database/api.md#createblobdata-options): ``` let blob = createBlob(largeBuffer); await MyTable.put({ id: 'my-record', data: blob }); ``` Retrieve blob data using standard Web API `Blob` methods: ``` let record = await MyTable.get('my-record'); let buffer = await record.data.bytes(); // ArrayBuffer let text = await record.data.text(); // string let stream = record.data.stream(); // ReadableStream ``` Blobs support asynchronous streaming, meaning a record can reference a blob before it is fully written to storage. Use `saveBeforeCommit: true` to wait for full write before committing: ``` let blob = createBlob(stream, { saveBeforeCommit: true }); await MyTable.put({ id: 'my-record', data: blob }); ``` Any string or buffer assigned to a `Blob` field in a `put`, `patch`, or `publish` is automatically coerced to a `Blob`. When returning a blob via REST, register an error handler to handle interrupted streams: ``` export class MyEndpoint extends MyTable { async get(target) { const record = super.get(target); let blob = record.data; blob.on('error', () => { MyTable.invalidate(target); }); return { status: 200, headers: {}, body: blob }; } } ``` ## Dynamic Schema Behavior When a table is created through the Operations API or Studio without a schema definition, it follows dynamic schema behavior: * Attributes are reflexively created as data is ingested * All top-level attributes are automatically indexed * Records automatically get `__createdtime__` and `__updatedtime__` audit attributes Dynamic schema tables are additive — new attributes are added as new data arrives. Existing records will have `null` for any newly added attributes. Use `create_attribute` and `drop_attribute` operations to manually manage attributes on dynamic schema tables. See the [Operations API](/reference/v4/operations-api/operations.md#databases--tables) for details. ## OpenAPI Specification Tables exported with `@export` are described via an `/openapi` endpoint on the main HTTP server associated with the REST service (default port 9926). ``` GET http://localhost:9926/openapi ``` This provides an OpenAPI 3.x description of all exported resource endpoints. The endpoint is a starting guide and may not cover every edge case. ## Renaming Tables Harper does **not** support renaming tables. Changing a type name in a schema definition creates a new, empty table — the original table and its data are unaffected. ## Related Documentation * [JavaScript API](/reference/v4/database/api.md) — `tables`, `databases`, `transaction()`, and `createBlob()` globals for working with schema-defined tables in code * [Data Loader](/reference/v4/database/data-loader.md) — Seed tables with initial data alongside schema deployment * [REST Querying](/reference/v4/rest/querying.md) — Querying tables via HTTP using schema-defined attributes and relationships * [Resources](/reference/v4/resources/resource-api.md) — Extending table behavior with custom application logic * [Storage Algorithm](/reference/v4/database/storage-algorithm.md) — How Harper indexes and stores schema-defined data * [Configuration](/reference/v4/configuration/options.md) — Component configuration for schemas --- # Storage Algorithm Harper's storage algorithm is the foundation of all database functionality. It is built on top of [LMDB](https://www.symas.com/lmdb) (Lightning Memory-Mapped Database), a high-performance key-value store, and extends it with automatic indexing, query-language-agnostic data access, and ACID compliance. ## Query Language Agnostic Harper's storage layer is decoupled from any specific query language. Data inserted via NoSQL operations can be read via SQL, REST, or the Resource API — all accessing the same underlying storage. This architecture allows Harper to add new query interfaces without changing how data is stored. ## ACID Compliance Harper provides full ACID compliance on each node using Multi-Version Concurrency Control (MVCC) through LMDB: * **Atomicity**: All writes in a transaction either fully commit or fully roll back * **Consistency**: Each transaction moves data from one valid state to another * **Isolation**: Readers and writers operate independently — readers do not block writers and writers do not block readers * **Durability**: Committed transactions are persisted to disk Each Harper table has a single writer process, eliminating deadlocks and ensuring writes are executed in the order received. Multiple reader processes can operate concurrently for high-throughput reads. ## Universally Indexed *Changed in: v4.3.0* — Storage performance improvements including better free-space management For [dynamic schema tables](/reference/v4/database/overview.md#dynamic-vs-defined-schemas), all top-level attributes are automatically indexed immediately upon ingestion — Harper reflexively creates the attribute and its index as new data arrives. For [schema-defined tables](/reference/v4/database/schema.md), indexes are created for all attributes marked with `@indexed`. Indexes are type-agnostic, ordering values as follows: 1. Booleans 2. Numbers (ordered numerically) 3. Strings (ordered lexically) ### LMDB Storage Layout Within the LMDB implementation, table records are grouped into a single LMDB environment file. Each attribute index is stored as a sub-database (`dbi`) within that environment. ## Compression *Changed in: v4.3.0* — Compression is now enabled by default for all records over 4KB Harper compresses record data automatically for records over 4KB. Compression settings can be configured in the [storage configuration](/reference/v4/configuration/options.md). Note that compression settings cannot be changed on existing databases without creating a new compacted copy — see [Compaction](/reference/v4/database/compaction.md). ## Performance Characteristics Harper inherits the following performance properties from LMDB: * **Memory-mapped I/O**: Data is accessed via memory mapping, enabling fast reads without data duplication between disk and memory * **Buffer cache integration**: Fully exploits the OS buffer cache for reduced I/O * **CPU cache optimization**: Built to maximize data locality within CPU caches * **Deadlock-free writes**: Full serialization of writers guarantees write ordering without deadlocks * **Zero-copy reads**: Readers access data directly from the memory map without copying ## Indexing Example Given a table with records like this: ``` ┌────┬────────┬────────┐ │ id │ field1 │ field2 │ ├────┼────────┼────────┤ │ 1 │ A │ X │ │ 2 │ 25 │ X │ │ 3 │ -1 │ Y │ │ 4 │ A │ │ │ 5 │ true │ 2 │ └────┴────────┴────────┘ ``` Harper maintains three separate LMDB sub-databases for that table: ``` Table (LMDB environment file) │ ├── primary index: id │ ┌─────┬──────────────────────────────────────┐ │ │ Key │ Value (full record) │ │ ├─────┼──────────────────────────────────────┤ │ │ 1 │ { id:1, field1:"A", field2:"X" } │ │ │ 2 │ { id:2, field1:25, field2:"X" } │ │ │ 3 │ { id:3, field1:-1, field2:"Y" } │ │ │ 4 │ { id:4, field1:"A" } │ │ │ 5 │ { id:5, field1:true, field2:2 } │ │ └─────┴──────────────────────────────────────┘ │ ├── secondary index: field1 secondary index: field2 │ ┌────────┬───────┐ ┌────────┬───────┐ │ │ Key │ Value │ │ Key │ Value │ │ ├────────┼───────┤ ├────────┼───────┤ │ │ -1 │ 3 │ │ 2 │ 5 │ │ │ 25 │ 2 │ │ X │ 1 │ │ │ A │ 1 │ │ X │ 2 │ │ │ A │ 4 │ │ Y │ 3 │ │ │ true │ 5 │ └────────┴───────┘ │ └────────┴───────┘ ``` Secondary indexes store the attribute value as the key and the record's primary key (`id`) as the value. To resolve a query result, Harper looks up the matching ids in the secondary index, then fetches the full records from the primary index. Indexes are ordered — booleans first, then numbers (numerically), then strings (lexically) — enabling efficient range queries across all types. ## Related Documentation * [Schema](/reference/v4/database/schema.md) — Defining indexed attributes and vector indexes * [Compaction](/reference/v4/database/compaction.md) — Reclaiming free space and applying new storage configuration to existing databases * [Configuration](/reference/v4/configuration/options.md) — Storage configuration options (compression, memory maps, blob paths) --- # System Tables Harper maintains a set of internal system tables in the `system` database. These tables store analytics, job tracking, replication configuration, and other internal state. Most are read-only from the application perspective; some can be queried for observability or management purposes. System tables are prefixed with `hdb_` and reside in the `system` database. ## Analytics Tables *Added in: v4.5.0* (resource and storage analytics expansion) ### `hdb_raw_analytics` Stores per-second, per-thread performance metrics. Records are written once per second (when there is activity) and include metrics for all operations, URL endpoints, and messaging topics, plus system resource information such as memory and CPU utilization. Records have a primary key equal to the timestamp in milliseconds since Unix epoch. Query with `search_by_conditions` (requires `superuser` permission): ``` { "operation": "search_by_conditions", "schema": "system", "table": "hdb_raw_analytics", "conditions": [ { "search_attribute": "id", "search_type": "between", "search_value": [1688594000000, 1688594010000] } ] } ``` A typical record: ``` { "time": 1688594390708, "period": 1000.8336279988289, "metrics": [ { "metric": "bytes-sent", "path": "search_by_conditions", "type": "operation", "median": 202, "mean": 202, "p95": 202, "p90": 202, "count": 1 }, { "metric": "memory", "threadId": 2, "rss": 1492664320, "heapTotal": 124596224, "heapUsed": 119563120, "external": 3469790, "arrayBuffers": 798721 }, { "metric": "utilization", "idle": 138227.52767700003, "active": 70.5066209952347, "utilization": 0.0005098165086230495 } ], "threadId": 2, "totalBytesProcessed": 12182820, "id": 1688594390708.6853 } ``` ### `hdb_analytics` Stores per-minute aggregate analytics. Once per minute, Harper aggregates all per-second raw entries from all threads into summary records in this table. Query it for longer-term performance trends. ``` { "operation": "search_by_conditions", "schema": "system", "table": "hdb_analytics", "conditions": [ { "search_attribute": "id", "search_type": "between", "search_value": [1688194100000, 1688594990000] } ] } ``` A typical aggregate record: ``` { "period": 60000, "metric": "bytes-sent", "method": "connack", "type": "mqtt", "median": 4, "mean": 4, "p95": 4, "p90": 4, "count": 1, "id": 1688589569646, "time": 1688589569646 } ``` For a full reference of available metrics and their fields, see [Analytics](/reference/v4/analytics/overview.md "Complete analytics metrics reference"). ## Data Loader Table ### `hdb_dataloader_hash` *Added in: v4.6.0* Used internally by the [Data Loader](/reference/v4/database/data-loader.md) to track which records have been loaded and detect changes. Stores SHA-256 content hashes of data file records so that unchanged records are not re-written on subsequent deployments. This table is managed automatically by the Data Loader. No direct interaction is required. ## Replication Tables ### `hdb_nodes` Stores the configuration and state of known nodes in a cluster, including connection details, replication settings, and revoked certificate serial numbers. Can be queried to inspect the current replication topology: ``` { "operation": "search_by_hash", "schema": "system", "table": "hdb_nodes", "hash_values": ["node-id"] } ``` Used by the `add_node`, `update_node`, and related clustering operations. See [Replication](/reference/v4/replication/clustering.md) for details. ### `hdb_certificate` Stores TLS certificates used in replication. Can be queried to inspect the certificates currently known to the cluster. ## Related Documentation * [Analytics](/reference/v4/analytics/overview.md) — Full reference for analytics metrics tracked in `hdb_analytics` and `hdb_raw_analytics` * [Data Loader](/reference/v4/database/data-loader.md) — Component that writes to `hdb_dataloader_hash` * [Replication](/reference/v4/replication/overview.md) — Clustering and replication system that uses `hdb_nodes` and `hdb_certificate` * [Operations API](/reference/v4/operations-api/overview.md) — Querying system tables using `search_by_conditions` --- # Transaction Logging Harper provides two complementary mechanisms for recording a history of data changes on a table: the **audit log** and the **transaction log**. Both are available at the table level and serve different use cases. | Feature | Audit Log | Transaction Log | | ----------------------------- | --------------------------------- | ------------------------------ | | Storage | Standard Harper table (per-table) | Clustering streams (per-table) | | Requires clustering | No | Yes | | Available since | v4.1.0 | v4.1.0 | | Stores original record values | Yes | No | | Query by username | Yes | No | | Query by primary key | Yes | No | | Used for real-time messaging | Yes (required) | No | ## Audit Log Available since: v4.1.0 The audit log is a data store that tracks every transaction across all tables in a database. Harper automatically creates and maintains a single audit log per database. The audit log captures the operation type, the user who made the change, the timestamp, and both the new and original record values. The audit log is **enabled by default**. To disable it, set [`logging.auditLog`](/reference/v4/logging/configuration.md) to `false` in `harperdb-config.yaml` and restart Harper. > The audit log is required for real-time messaging (WebSocket and MQTT subscriptions) and replication. Do not disable it if real-time features or replication are in use. ### Audit Log Operations #### `read_audit_log` Queries the audit log for a specific table. Supports filtering by timestamp, username, or primary key value. **By timestamp:** ``` { "operation": "read_audit_log", "schema": "dev", "table": "dog", "search_type": "timestamp", "search_values": [1660585740558] } ``` Timestamp behavior: | `search_values` | Result | | --------------- | ---------------------------------------- | | `[]` | All records for the table | | `[timestamp]` | All records after the provided timestamp | | `[from, to]` | Records between the two timestamps | **By username:** ``` { "operation": "read_audit_log", "schema": "dev", "table": "dog", "search_type": "username", "search_values": ["admin"] } ``` **By primary key:** ``` { "operation": "read_audit_log", "schema": "dev", "table": "dog", "search_type": "hash_value", "search_values": [318] } ``` **Response example:** ``` { "operation": "update", "user_name": "HDB_ADMIN", "timestamp": 1607035559122.277, "hash_values": [1, 2], "records": [ { "id": 1, "breed": "Muttzilla", "age": 6, "__updatedtime__": 1607035559122 } ], "original_records": [ { "__createdtime__": 1607035556801, "__updatedtime__": 1607035556801, "age": 5, "breed": "Mutt", "id": 1, "name": "Harper" } ] } ``` The `original_records` field contains the record state before the operation was applied. #### `delete_audit_logs_before` Deletes audit log entries older than the specified timestamp. *Changed in: v4.3.0* — Audit log cleanup improved to reduce resource consumption during scheduled cleanups *Changed in: v4.5.0* — Storage reclamation: Harper automatically evicts older audit log entries when free storage drops below a configurable threshold ``` { "operation": "delete_audit_logs_before", "schema": "dev", "table": "dog", "timestamp": 1598290282817 } ``` *** ## Enabling Audit Log Per Table You can enable or disable the audit log for individual tables using the `@table` directive's `audit` argument in your schema: ``` type Dog @table(audit: true) { id: Long @primaryKey name: String } ``` This overrides the [`logging.auditLog`](/reference/v4/logging/configuration.md) global configuration for that specific table. ## Related Documentation * [Logging](/reference/v4/logging/overview.md) — Application and system logging (separate from transaction/audit logging) * [Replication](/reference/v4/replication/overview.md) — Clustering setup required for transaction logs * [Logging Configuration](/reference/v4/logging/configuration.md) — Global audit log configuration (`logging.auditLog`) * [Operations API](/reference/v4/operations-api/overview.md) — Sending operations to Harper --- # Environment Variables Harper supports loading environment variables in Harper applications `process.env` using the built-in `loadEnv` plugin. This is the standard way to supply secrets and configuration to your Harper components without hardcoding values. `loadEnv` does **not** need to be installed as it is built into Harper and only needs to be declared in your `config.yaml`. note If you are looking for information on how to configure your Harper installation using environment variables, see [Configuration](/reference/v4/configuration/overview.md) section for more information. ## Configuration | Option | Type | Required | Description | | ---------- | -------------------- | -------- | -------------------------------------------------------------------------------------- | | `files` | `string \| string[]` | **Yes** | Path(s) or glob pattern(s) to the env file(s) to load. | | `override` | `boolean` | No | If `true`, loaded values override existing environment variables. Defaults to `false`. | ## Basic Usage ``` loadEnv: files: '.env' ``` This loads the `.env` file from the root of your component directory into `process.env`. ## Load Order > **Important:** Specify `loadEnv` first in your `config.yaml` so that environment variables are loaded before any other components start. ``` # config.yaml — loadEnv must come first loadEnv: files: '.env' rest: true myApp: files: './src/*.js' ``` Because Harper is a single-process application, environment variables are loaded onto `process.env` and are shared across all components. As long as `loadEnv` is listed before dependent components, those components will have access to the loaded variables. ## Override Behavior By default, `loadEnv` follows the standard dotenv convention: **existing environment variables take precedence** over values in `.env` files. This means variables already set in the shell or container environment will not be overwritten. To override existing environment variables, use the `override` option: ``` loadEnv: files: '.env' override: true ``` ## Multiple Files As a Harper plugin, `loadEnv` supports multiple files using either glob patterns or a list of files in the configuration: ``` loadEnv: files: - '.env' - '.env.local' ``` or ``` loadEnv: files: 'env-vars/*' ``` Files are loaded in the order specified. ## Related * [Components Overview](/reference/v4/components/overview.md) * [Configuration](/reference/v4/configuration/overview.md) --- # Define Fastify Routes note Fastify routes are discouraged in favor of modern routing with [Custom Resources](/reference/v4/resources/overview.md), but remain a supported feature for backwards compatibility and specific use cases. Harper provides a build-in plugin for loading [Fastify](https://www.fastify.io/) routes as a way to define custom endpoints for your Harper application. While we generally recommend building your endpoints/APIs with Harper's [REST interface](/reference/v4/rest/overview.md) for better performance and standards compliance, Fastify routes can provide an extensive API for highly customized path handling. Below is a very simple example of a route declaration. The fastify route handler can be configured in your application's config.yaml (this is the default config if you used the [application template](https://github.com/HarperDB/application-template)): ``` fastifyRoutes: files: routes/*.js # specify the location of route definition modules ``` By default, route URLs are configured to be: ``` :// ``` However, you can specify the path to be `/` if you wish to have your routes handling the root path of incoming URLs. * The route below, using the default config, within the **dogs** project, with a route of **breeds** would be available at ****. In effect, this route is just a pass-through to Harper. The same result could have been achieved by hitting the core Harper API, since it uses **hdbCore.preValidation** and **hdbCore.request**, which are defined in the "helper methods" section, below. ``` export default async (server, { hdbCore, logger }) => { server.route({ url: '/', method: 'POST', preValidation: hdbCore.preValidation, handler: hdbCore.request, }); }; ``` ## Custom Handlers For endpoints where you want to execute multiple operations against Harper, or perform additional processing (like an ML classification, or an aggregation, or a call to a 3rd party API), you can define your own logic in the handler. The function below will execute a query against the dogs table, and filter the results to only return those dogs over 4 years in age. **IMPORTANT: This route has NO preValidation and uses hdbCore.requestWithoutAuthentication, which- as the name implies- bypasses all user authentication. See the security concerns and mitigations in the "helper methods" section, below.** ``` export default async (server, { hdbCore, logger }) => { server.route({ url: '/:id', method: 'GET', handler: (request) => { request.body= { operation: 'sql', sql: `SELECT * FROM dev.dog WHERE id = ${request.params.id}` }; const result = await hdbCore.requestWithoutAuthentication(request); return result.filter((dog) => dog.age > 4); } }); } ``` ## Custom preValidation Hooks The simple example above was just a pass-through to Harper- the exact same result could have been achieved by hitting the core Harper API. But for many applications, you may want to authenticate the user using custom logic you write, or by conferring with a 3rd party service. Custom preValidation hooks let you do just that. Below is an example of a route that uses a custom validation hook: ``` import customValidation from '../helpers/customValidation'; export default async (server, { hdbCore, logger }) => { server.route({ url: '/:id', method: 'GET', preValidation: (request) => customValidation(request, logger), handler: (request) => { request.body = { operation: 'sql', sql: `SELECT * FROM dev.dog WHERE id = ${request.params.id}`, }; return hdbCore.requestWithoutAuthentication(request); }, }); }; ``` Notice we imported customValidation from the **helpers** directory. To include a helper, and to see the actual code within customValidation, see [Helper Methods](#helper-methods). ## Helper Methods When declaring routes, you are given access to 2 helper methods: hdbCore and logger. ### hdbCore hdbCore contains three functions that allow you to authenticate an inbound request, and execute operations against Harper directly, by passing the standard Operations API. #### preValidation This is an array of functions used for fastify authentication. The second function takes the authorization header from the inbound request and executes the same authentication as the standard Harper Operations API (for example, `hdbCore.preValidation[1](req, resp, callback)`). It will determine if the user exists, and if they are allowed to perform this operation. **If you use the request method, you have to use preValidation to get the authenticated user**. #### request This will execute a request with Harper using the operations API. The `request.body` should contain a standard Harper operation and must also include the `hdb_user` property that was in `request.body` provided in the callback. #### requestWithoutAuthentication Executes a request against Harper without any security checks around whether the inbound user is allowed to make this request. For security purposes, you should always take the following precautions when using this method: * Properly handle user-submitted values, including url params. User-submitted values should only be used for `search_value` and for defining values in records. Special care should be taken to properly escape any values if user-submitted values are used for SQL. ### logger This helper allows you to write directly to the log file, hdb.log. It's useful for debugging during development, although you may also use the console logger. There are 5 functions contained within logger, each of which pertains to a different **logging.level** configuration in your harperdb-config.yaml file. * logger.trace('Starting the handler for /dogs') * logger.debug('This should only fire once') * logger.warn('This should never ever fire') * logger.error('This did not go well') * logger.fatal('This did not go very well at all') --- # GraphQL Querying *Added in: v4.4.0* (provisional) *Changed in: v4.5.0* (disabled by default, configuration options) Harper supports GraphQL in a variety of ways. It can be used for [defining schemas](/reference/v4/components/applications.md), and for querying [Resources](/reference/v4/resources/overview.md). Get started by setting `graphql: true` in `config.yaml`. This configuration option was added in v4.5.0 to allow more granular control over the GraphQL endpoint. This automatically enables a `/graphql` endpoint that can be used for GraphQL queries. > Harper's GraphQL component is inspired by the [GraphQL Over HTTP](https://graphql.github.io/graphql-over-http/draft/#) specification; however, it does not fully implement neither that specification nor the [GraphQL](https://spec.graphql.org/) specification. Queries can either be `GET` or `POST` requests, and both follow essentially the same request format. `GET` requests must use search parameters, and `POST` requests use the request body. For example, to request the GraphQL Query: ``` query GetDogs { Dog { id name } } ``` The `GET` request would look like: ``` GET /graphql?query=query+GetDogs+%7B+Dog+%7B+id+name+%7D+%7D+%7D Accept: application/graphql-response+json ``` And the `POST` request would look like: ``` POST /graphql/ Content-Type: application/json Accept: application/graphql-response+json { "query": "query GetDogs { Dog { id name } } }" } ``` > Tip: For the best user experience, include the `Accept: application/graphql-response+json` header in your request. This provides better status codes for errors. The Harper GraphQL querying system is strictly limited to exported Harper Resources. This will typically be a table that uses the `@exported` directive in its schema or `export`'ed custom resources. Queries can only specify Harper Resources and their attributes in the selection set. Queries can filter using [arguments](https://graphql.org/learn/queries/#arguments) on the top-level Resource field. Harper provides a short form pattern for simple queries, and a long form pattern based off of the [Resource Query API](/reference/v4/rest/querying.md) for more complex queries. Unlike REST queries, GraphQL queries can specify multiple resources simultaneously: ``` query GetDogsAndOwners { Dog { id name breed } Owner { id name occupation } } ``` This will return all dogs and owners in the database. And is equivalent to executing two REST queries: ``` GET /Dog/?select(id,name,breed) # and GET /Owner/?select(id,name,occupation) ``` ## Request Parameters There are three request parameters for GraphQL queries: `query`, `operationName`, and `variables` 1. `query` - *Required* - The string representation of the GraphQL document. 1. Limited to [Executable Definitions](https://spec.graphql.org/October2021/#executabledefinition) only. 2. i.e. GraphQL [`query`](https://graphql.org/learn/queries/#fields) or `mutation` (coming soon) operations, and [fragments](https://graphql.org/learn/queries/#fragments). 3. If an shorthand, unnamed, or singular named query is provided, they will be executed by default. Otherwise, if there are multiple queries, the `operationName` parameter must be used. 2. `operationName` - *Optional* - The name of the query operation to execute if multiple queries are provided in the `query` parameter 3. `variables` - *Optional* - A map of variable values to be used for the specified query ## Type Checking The Harper GraphQL Querying system is designed to handle GraphQL queries and map them directly to Harper's tables, schemas, fields, and relationships to easily query with GraphQL syntax with minimal configuration, code, and overhead. However, the "GraphQL", as a technology has come to encompass an entire model of resolvers and a type checking system, which is outside of the scope of using GraphQL as a *query* language for data retrieval from Harper. Therefore, the querying system generally does **not** type check, and type checking behavior is outside the scope of resolving queries and is only loosely defined in Harper. In variable definitions, the querying system will ensure non-null values exist (and error appropriately), but it will not do any type checking of the value itself. For example, the variable `$name: String!` states that `name` should be a non-null, string value. * If the request does not contain the `name` variable, an error will be returned * If the request provides `null` for the `name` variable, an error will be returned * If the request provides any non-string value for the `name` variable, i.e. `1`, `true`, `{ foo: "bar" }`, the behavior is undefined and an error may or may not be returned. * If the variable definition is changed to include a default value, `$name: String! = "John"`, then when omitted, `"John"` will be used. * If `null` is provided as the variable value, an error will still be returned. * If the default value does not match the type specified (i.e. `$name: String! = 0`), this is also considered undefined behavior. It may or may not fail in a variety of ways. * Fragments will generally extend non-specified types, and the querying system will do no validity checking on them. For example, `fragment Fields on Any { ... }` is just as valid as `fragment Fields on MadeUpTypeName { ... }`. See the Fragments sections for more details. The only notable place the querying system will do some level of type analysis is the transformation of arguments into a query. * Objects will be transformed into properly nested attributes * Strings and Boolean values are passed through as their AST values * Float and Int values will be parsed using the JavaScript `parseFloat` and `parseInt` methods respectively. * List and Enums are not supported. ## Fragments The querying system loosely supports fragments. Both fragment definitions and inline fragments are supported, and are entirely a composition utility. Since this system does very little type checking, the `on Type` part of fragments is entirely pointless. Any value can be used for `Type` and it will have the same effect. For example, in the query ``` query Get { Dog { ...DogFields } } fragment DogFields on Dog { name breed } ``` The `Dog` type in the fragment has no correlation to the `Dog` resource in the query (that correlates to the Harper `Dog` resource). You can literally specify anything in the fragment and it will behave the same way: ``` fragment DogFields on Any { ... } # this is recommended fragment DogFields on Cat { ... } fragment DogFields on Animal { ... } fragment DogFields on LiterallyAnything { ... } ``` As an actual example, fragments should be used for composition: ``` query Get { Dog { ...sharedFields breed } Owner { ...sharedFields occupation } } fragment sharedFields on Any { id name } ``` ## Short Form Querying Any attribute can be used as an argument for a query. In this short form, multiple arguments is treated as multiple equivalency conditions with the default `and` operation. For example, the following query requires an `id` variable to be provided, and the system will search for a `Dog` record matching that id. ``` query GetDog($id: ID!) { Dog(id: $id) { name breed owner { name } } } ``` And as a properly formed request: ``` POST /graphql/ Content-Type: application/json Accept: application/graphql-response+json { "query": "query GetDog($id: ID!) { Dog(id: $id) { name breed owner {name}}", "variables": { "id": "0" } } ``` The REST equivalent would be: ``` GET /Dog/?id==0&select(name,breed,owner{name}) # or GET /Dog/0?select(name,breed,owner{name}) ``` Short form queries can handle nested attributes as well. For example, return all dogs who have an owner with the name `"John"` ``` query GetDog { Dog(owner: { name: "John" }) { name breed owner { name } } } ``` Would be equivalent to ``` GET /Dog/?owner.name==John&select(name,breed,owner{name}) ``` And finally, we can put all of these together to create semi-complex, equality based queries! The following query has two variables and will return all dogs who have the specified name as well as the specified owner name. ``` query GetDog($dogName: String!, $ownerName: String!) { Dog(name: $dogName, owner: { name: $ownerName }) { name breed owner { name } } } ``` --- # HTTP API The `server` global object is available in all Harper component code. It provides access to the HTTP server middleware chain, WebSocket server, authentication, resource registry, and cluster information. ## `server.http(listener, options)` Add a handler to the HTTP request middleware chain. ``` server.http(listener: RequestListener, options?: HttpOptions): HttpServer[] ``` Returns an array of `HttpServer` instances based on the `options.port` and `options.securePort` values. **Example:** ``` server.http( (request, next) => { if (request.url === '/graphql') return handleGraphQLRequest(request); return next(request); }, { runFirst: true } ); ``` ### `RequestListener` ``` type RequestListener = (request: Request, next: RequestListener) => Promise; ``` To continue the middleware chain, call `next(request)`. To short-circuit, return a `Response` (or `Response`-like object) directly. ### `HttpOptions` | Property | Type | Default | Description | | ------------ | ------- | ----------------- | --------------------------------------------- | | `runFirst` | boolean | `false` | Insert this handler at the front of the chain | | `port` | number | `http.port` | Target the HTTP server on this port | | `securePort` | number | `http.securePort` | Target the HTTPS server on this port | ### `HttpServer` A Node.js [`http.Server`](https://nodejs.org/api/http.html#class-httpserver) or [`https.SecureServer`](https://nodejs.org/api/https.html#class-httpsserver) instance. *** ## `Request` A `Request` object is passed to HTTP middleware handlers and direct static REST handlers. It follows the [WHATWG `Request`](https://developer.mozilla.org/en-US/docs/Web/API/Request) API with additional Harper-specific properties. ### Properties | Property | Type | Description | | ---------- | --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `url` | string | The request target (path + query string), e.g. `/path?query=string` | | `method` | string | HTTP method: `GET`, `POST`, `PUT`, `DELETE`, etc. | | `headers` | [`Headers`](https://developer.mozilla.org/en-US/docs/Web/API/Headers) | Request headers | | `pathname` | string | Path portion of the URL, without query string | | `protocol` | string | `http` or `https` | | `data` | any | Deserialized body, based on `Content-Type` header | | `ip` | string | Remote IP address of the client (or last proxy) | | `host` | string | Host from the request headers | | `session` | object | Current cookie-based session (a `Table` record instance). Update with `request.session.update({ key: value })`. A cookie is set automatically the first time a session is updated or a login occurs. | ### Methods #### `request.login(username, password)` ``` login(username: string, password: string): Promise ``` Authenticates the user by username and password. On success, creates a session and sets a cookie on the response. Rejects if authentication fails. #### `request.sendEarlyHints(link, headers?)` ``` sendEarlyHints(link: string, headers?: object): void ``` Sends an [Early Hints](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/103) (HTTP 103) response before the final response. Useful in cache resolution functions to hint at preloadable resources: ``` class Origin { async get(request) { this.getContext().requestContext.sendEarlyHints(''); return fetch(request); } } Cache.sourcedFrom(Origin); ``` ### Low-Level Node.js Access caution These properties expose the raw Node.js request/response objects and should be used with caution. Using them can break other middleware handlers that depend on the layered `Request`/`Response` pattern. | Property | Description | | --------------- | ----------------------------------------------------------------------------------------------------- | | `_nodeRequest` | Underlying [`http.IncomingMessage`](https://nodejs.org/api/http.html#http_class_http_incomingmessage) | | `_nodeResponse` | Underlying [`http.ServerResponse`](https://nodejs.org/api/http.html#http_class_http_serverresponse) | *** ## `Response` REST method handlers can return: * **Data directly** — Serialized using Harper's content negotiation * **A `Response` object** — The WHATWG [`Response`](https://developer.mozilla.org/en-US/docs/Web/API/Response) * **A `Response`-like object** — A plain object with the following properties: | Property | Type | Description | | --------- | --------------------------------------------------------------------- | ------------------------------------------------- | | `status` | number | HTTP status code (e.g. `200`, `404`) | | `headers` | [`Headers`](https://developer.mozilla.org/en-US/docs/Web/API/Headers) | Response headers | | `data` | any | Response data, serialized via content negotiation | | `body` | Buffer \| string \| ReadableStream \| Blob | Raw response body (alternative to `data`) | *** ## `server.ws(listener, options)` Add a handler to the WebSocket connection middleware chain. ``` server.ws(listener: WsListener, options?: WsOptions): HttpServer[] ``` **Example:** ``` server.ws((ws, request, chainCompletion) => { chainCompletion.then(() => { ws.on('message', (data) => console.log('received:', data)); ws.send('hello'); }); }); ``` ### `WsListener` ``` type WsListener = (ws: WebSocket, request: Request, chainCompletion: Promise, next: WsListener) => Promise; ``` | Parameter | Description | | ----------------- | ------------------------------------------------------------------------------------------------------------------------- | | `ws` | [`WebSocket`](https://github.com/websockets/ws/blob/main/doc/ws.md#class-websocket) instance | | `request` | Harper `Request` object from the upgrade event | | `chainCompletion` | `Promise` that resolves when the HTTP request chain finishes. Await before sending to ensure the HTTP request is handled. | | `next` | Continue chain: `next(ws, request, chainCompletion)` | ### `WsOptions` | Property | Type | Default | Description | | ------------ | ------- | ----------------- | ----------------------------------------------- | | `maxPayload` | number | 100 MB | Maximum WebSocket payload size | | `runFirst` | boolean | `false` | Insert this handler at the front of the chain | | `port` | number | `http.port` | Target the WebSocket server on this port | | `securePort` | number | `http.securePort` | Target the secure WebSocket server on this port | *** ## `server.upgrade(listener, options)` Add a handler to the HTTP server `upgrade` event. Use this to delegate upgrade events to an external WebSocket server. ``` server.upgrade(listener: UpgradeListener, options?: UpgradeOptions): void ``` **Example** (from the Harper Next.js component): ``` server.upgrade( (request, socket, head, next) => { if (request.url === '/_next/webpack-hmr') { return upgradeHandler(request, socket, head).then(() => { request.__harperdb_request_upgraded = true; next(request, socket, head); }); } return next(request, socket, head); }, { runFirst: true } ); ``` When `server.ws()` is registered, Harper adds a default upgrade handler. The default handler sets `request.__harperdb_request_upgraded = true` after upgrading, and checks for this flag before upgrading again (so external upgrade handlers can detect whether Harper has already handled the upgrade). ### `UpgradeListener` ``` type UpgradeListener = (request: IncomingMessage, socket: Socket, head: Buffer, next: UpgradeListener) => void; ``` ### `UpgradeOptions` | Property | Type | Default | Description | | ------------ | ------- | ----------------- | ------------------------------------ | | `runFirst` | boolean | `false` | Insert at the front of the chain | | `port` | number | `http.port` | Target the HTTP server on this port | | `securePort` | number | `http.securePort` | Target the HTTPS server on this port | *** ## `server.socket(listener, options)` Create a raw TCP or TLS socket server. ``` server.socket(listener: ConnectionListener, options: SocketOptions): SocketServer ``` Only one socket server is created per call. A `securePort` takes precedence over `port`. ### `ConnectionListener` Node.js connection listener as in [`net.createServer`](https://nodejs.org/api/net.html#netcreateserveroptions-connectionlistener) or [`tls.createServer`](https://nodejs.org/api/tls.html#tlscreateserveroptions-secureconnectionlistener). ### `SocketOptions` | Property | Type | Description | | ------------ | ------ | -------------------------------------------------------------------------- | | `port` | number | Port for a [`net.Server`](https://nodejs.org/api/net.html#class-netserver) | | `securePort` | number | Port for a [`tls.Server`](https://nodejs.org/api/tls.html#class-tlsserver) | ### `SocketServer` A Node.js [`net.Server`](https://nodejs.org/api/net.html#class-netserver) or [`tls.Server`](https://nodejs.org/api/tls.html#class-tlsserver) instance. *** ## `server.authenticateUser(username, password)` *Added in: v4.5.0* ``` server.authenticateUser(username: string, password: string): Promise ``` Returns the user object for the given username after verifying the password. Throws if the password is incorrect. Use this when you need to explicitly verify a user's credentials (e.g., in a custom login endpoint). For lookup without password verification, use [`server.getUser()`](#servergetuserusername). *** ## `server.getUser(username)` ``` server.getUser(username: string): Promise ``` Returns the user object for the given username without verifying credentials. Use for authorization checks when the user is already authenticated. *** ## `server.resources` The central registry of all resources exported for REST, MQTT, and other protocols. ### `server.resources.set(name, resource, exportTypes?)` Register a resource: ``` class NewResource extends Resource {} server.resources.set('NewResource', NewResource); // Limit to specific protocols: server.resources.set('NewResource', NewResource, { rest: true, mqtt: false }); ``` ### `server.resources.getMatch(path, exportType?)` Find a resource matching a path: ``` server.resources.getMatch('/NewResource/some-id'); server.resources.getMatch('/NewResource/some-id', 'rest'); ``` *** ## `server.operation(operation, context?, authorize?)` Execute an [Operations API](/reference/v4/operations-api/overview.md) operation programmatically. ``` server.operation(operation: object, context?: { username: string }, authorize?: boolean): Promise ``` | Parameter | Type | Description | | ----------- | ---------------------- | ---------------------------------------------------- | | `operation` | object | Operations API request body | | `context` | `{ username: string }` | Optional: execute as this user | | `authorize` | boolean | Whether to apply authorization. Defaults to `false`. | *** ## `server.recordAnalytics(value, metric, path?, method?, type?)` Record a metric into Harper's analytics system. ``` server.recordAnalytics(value: number, metric: string, path?: string, method?: string, type?: string): void ``` | Parameter | Description | | --------- | ---------------------------------------------------------------------------- | | `value` | Numeric value (e.g. duration in ms, bytes) | | `metric` | Metric name | | `path` | Optional URL path for grouping (omit per-record IDs — use the resource name) | | `method` | Optional HTTP method for grouping | | `type` | Optional type for grouping | Metrics are aggregated and available via the [analytics API](/reference/v4/analytics/overview.md). *** ## `server.config` The parsed `harperdb-config.yaml` configuration object. Read-only access to Harper's current runtime configuration. *** ## `server.nodes` Returns an array of node objects registered in the cluster. ## `server.shards` Returns a map of shard number to an array of associated nodes. ## `server.hostname` Returns the hostname of the current node. ## `server.contentTypes` Returns the `Map` of registered content type handlers. Same as the global [`contentTypes`](#contenttypes) object. *** ## `contentTypes` A [`Map`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map) of content type handlers for HTTP request/response serialization. Harper uses content negotiation: the `Content-Type` header selects the deserializer for incoming requests, and the `Accept` header selects the serializer for responses. ### Built-in Content Types | MIME type | Description | | --------------------- | ------------------ | | `application/json` | JSON | | `application/cbor` | CBOR | | `application/msgpack` | MessagePack | | `text/csv` | CSV | | `text/event-stream` | Server-Sent Events | ### Custom Content Type Handlers Register or replace a handler by setting it on the `contentTypes` map: ``` import { contentTypes } from 'harperdb'; contentTypes.set('text/xml', { serialize(data) { return '' + serialize(data) + ''; }, q: 0.8, // quality: lower = less preferred during content negotiation }); ``` ### Handler Interface | Property | Type | Description | | --------------------------- | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `serialize(data)` | `(any) => Buffer \| Uint8Array \| string` | Serialize data for a response | | `serializeStream(data)` | `(any) => ReadableStream` | Serialize as a stream (for async iterables or large data) | | `deserialize(buffer)` | `(Buffer \| string) => any` | Deserialize an incoming request body. Used when `deserializeStream` is absent. String for `text/*` types, Buffer for binary types. | | `deserializeStream(stream)` | `(ReadableStream) => any` | Deserialize an incoming request stream | | `q` | number (0–1) | Quality indicator for content negotiation. Defaults to `1`. | *** ## Related * [HTTP Overview](/reference/v4/http/overview.md) * [HTTP Configuration](/reference/v4/http/configuration.md) * [REST Overview](/reference/v4/rest/overview.md) * [Security API](/reference/v4/security/api.md) --- # HTTP Configuration The `http` section in `harperdb-config.yaml` controls the built-in HTTP server that serves REST, WebSocket, component, and Operations API traffic. Harper must be restarted for configuration changes to take effect. ## Ports ### `http.port` Type: `integer` Default: `9926` The port the HTTP server listens on. This is the primary port for REST, WebSocket, MQTT-over-WebSocket, and component traffic. ### `http.securePort` Type: `integer` Default: `null` The port for HTTPS connections. Requires a valid `tls` section configured with certificate and key. When set, Harper accepts both plaintext (`http.port`) and TLS connections (`http.securePort`) simultaneously. ## TLS TLS is configured in its own top-level `tls` section in `harperdb-config.yaml`, separate from the `http` section. It is shared by the HTTP server (HTTPS), the MQTT broker (secure MQTT), and any TLS socket servers. See [TLS Configuration](/reference/v4/http/tls.md) for all options including multi-domain (SNI) certificates and the Operations API override. To enable HTTPS, set `http.securePort` and add a `tls` block: ``` http: securePort: 9927 tls: certificate: ~/hdb/keys/certificate.pem certificateAuthority: ~/hdb/keys/ca.pem privateKey: ~/hdb/keys/privateKey.pem ``` ## HTTP/2 ### `http.http2` *Added in: v4.5.0* Type: `boolean` Default: `false` Enables HTTP/2 for all API endpoints. HTTP/2 requires TLS, so `http.securePort` must also be set. ``` http: http2: true securePort: 9927 ``` ## Timeouts and Limits ### `http.headersTimeout` Type: `integer` Default: `60000` (ms) Maximum time in milliseconds the server waits to receive the complete HTTP headers for a request. ### `http.keepAliveTimeout` Type: `integer` Default: `30000` (ms) Milliseconds of inactivity after which the server closes an idle keep-alive connection. ### `http.timeout` Type: `integer` Default: `120000` (ms) Maximum time in milliseconds before a request times out. ### `http.maxHeaderSize` Type: `integer` Default: `16394` (bytes) Maximum allowed size of HTTP request headers. ### `http.requestQueueLimit` Type: `integer` Default: `20000` (ms) The maximum estimated request queue time in milliseconds. When the queue exceeds this limit, requests are rejected with HTTP 503. ## Compression ### `http.compressionThreshold` *Added in: v4.2.0* Type: `number` Default: `1200` (bytes) For clients that support Brotli encoding (`Accept-Encoding: br`), responses larger than this threshold are compressed. Streaming query responses are always compressed for supporting clients, regardless of this setting (since their size is unknown upfront). ``` http: compressionThreshold: 1200 ``` ## CORS ### `http.cors` Type: `boolean` Default: `true` Enables Cross-Origin Resource Sharing, allowing requests from different origins. ### `http.corsAccessList` Type: `string[]` Default: `null` An array of allowed origin domains when CORS is enabled. When `null`, all origins are allowed. ``` http: cors: true corsAccessList: - https://example.com - https://app.example.com ``` ### `http.corsAccessControlAllowHeaders` *Added in: v4.5.0* Type: `string` Default: `"Accept, Content-Type, Authorization"` Comma-separated list of headers allowed in the [`Access-Control-Allow-Headers`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Headers) response header for OPTIONS (preflight) requests. ## Session Affinity ### `http.sessionAffinity` *Added in: v4.1.0* Type: `string` Default: `null` Routes repeated requests from the same client to the same worker thread. This can improve caching locality and provide fairness in request handling. Accepted values: * `ip` — Route by the remote IP address. Use this when Harper is the public-facing server and each client has a distinct IP. * `` — Route by the value of any HTTP header (e.g., `Authorization`). Use this when Harper is behind a proxy where all requests share the same source IP. ``` http: sessionAffinity: ip ``` caution If Harper is behind a reverse proxy and you use `ip`, all requests will share the proxy's IP and will be routed to a single thread. Use a header-based value instead. ## mTLS ### `http.mtls` *Added in: v4.3.0* Type: `boolean | object` Default: `false` Enables mutual TLS (mTLS) authentication for HTTP connections. When set to `true`, client certificates are verified against the CA specified in `tls.certificateAuthority`. Authenticated connections use the `CN` (common name) from the certificate subject as the Harper username. ``` http: mtls: true ``` For granular control, specify an object: | Property | Type | Default | Description | | ------------------------- | ----------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `user` | string | (CN from cert) | Authenticate all mTLS connections as this specific user. Set to `null` to skip credential-based authentication (requires combining with `required: true`). | | `required` | boolean | `false` | Reject any connection that does not provide a valid client certificate. | | `certificateVerification` | boolean \| object | `false` | Enable CRL/OCSP certificate revocation checking. See below. | ### `http.mtls.certificateVerification` *Added in: v4.7.0* (OCSP support) Type: `boolean | object` Default: `false` When mTLS is enabled, Harper can verify the revocation status of client certificates using CRL (Certificate Revocation List) and/or OCSP (Online Certificate Status Protocol). Disabled by default; must be explicitly enabled for environments that require certificate revocation checking. Set to `true` to enable with all defaults, or configure as an object: **Global:** * `failureMode` — `'fail-closed'` (default) | `'fail-open'`. Whether to reject or allow connections when revocation checking fails. **CRL** (enabled by default when `certificateVerification` is enabled): * `crl.enabled` — boolean, default `true` * `crl.timeout` — ms to wait for CRL download, default `10000` * `crl.cacheTtl` — ms to cache CRL, default `86400000` (24h) * `crl.gracePeriod` — ms grace period after CRL `nextUpdate`, default `86400000` (24h) * `crl.failureMode` — CRL-specific failure mode **OCSP** (enabled by default as CRL fallback): * `ocsp.enabled` — boolean, default `true` * `ocsp.timeout` — ms to wait for OCSP response, default `5000` * `ocsp.cacheTtl` — ms to cache successful responses, default `3600000` (1h) * `ocsp.errorCacheTtl` — ms to cache errors, default `300000` (5m) * `ocsp.failureMode` — OCSP-specific failure mode Harper uses a CRL-first strategy with OCSP fallback. If both fail, the configured `failureMode` is applied. **Examples:** ``` # Basic mTLS, no revocation checking http: mtls: true # mTLS with revocation checking (recommended for production) http: mtls: certificateVerification: true # Require mTLS for all connections + revocation checking http: mtls: required: true certificateVerification: true # Custom verification settings http: mtls: certificateVerification: failureMode: fail-closed crl: timeout: 15000 cacheTtl: 43200000 ocsp: timeout: 8000 cacheTtl: 7200000 ``` ## Logging HTTP request logging is disabled by default. Enabling the `http.logging` block turns on request logging. ### `http.logging` *Added in: v4.6.0* Type: `object` Default: disabled ``` http: logging: level: info # info = all requests, warn = 4xx+, error = 5xx path: ~/hdb/log/http.log timing: true # log request timing headers: false # log request headers (verbose) id: true # assign and log a unique request ID ``` The `level` controls which requests are logged: * `info` (or more verbose) — All HTTP requests * `warn` — Requests with status 400 or above * `error` — Requests with status 500 or above ## Complete Example ``` http: port: 9926 securePort: 9927 http2: true cors: true corsAccessList: - null compressionThreshold: 1200 headersTimeout: 60000 keepAliveTimeout: 30000 timeout: 120000 maxHeaderSize: 16384 requestQueueLimit: 20000 sessionAffinity: null mtls: false logging: level: warn path: ~/hdb/log/http.log timing: true # tls is a top-level section — see TLS Configuration tls: certificate: ~/hdb/keys/certificate.pem certificateAuthority: ~/hdb/keys/ca.pem privateKey: ~/hdb/keys/privateKey.pem ``` ## Related * [HTTP Overview](/reference/v4/http/overview.md) * [HTTP API](/reference/v4/http/api.md) * [TLS Configuration](/reference/v4/http/tls.md) * [Security Overview](/reference/v4/security/overview.md) * [Configuration Overview](/reference/v4/configuration/overview.md) --- # HTTP Server Harper includes a built-in HTTP server that serves as the primary interface for REST, WebSocket, MQTT-over-WebSocket, and component-defined endpoints. The same server handles all application traffic on a configurable port (default `9926`). ## Architecture Harper's HTTP server is multi-threaded. Each thread runs an independent copy of the HTTP stack, and incoming connections are distributed across threads using `SO_REUSEPORT` socket sharing — the most performant mechanism available for multi-threaded socket handling. *Added in: v4.1.0* (worker threads for HTTP requests) *Changed in: v4.2.0* (switched from process-per-thread model with session-affinity delegation to `SO_REUSEPORT` socket sharing) In previous versions: Session-affinity based socket delegation was used to route requests. This has been deprecated in favor of `SO_REUSEPORT`. ## Request Handling Harper uses a layered middleware chain for HTTP request processing. Components and applications can add handlers to this chain using the [`server.http()`](/reference/v4/http/api.md#serverhttplistener-options) API. Handlers are called in order; each handler can either process the request and return a `Response`, or pass it along to the next handler with `next(request)`. Request and response objects follow the [WHATWG Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) conventions (`Request` and `Response` classes), providing good composability for layered middleware and clean mapping to REST resource handlers. ## Protocols Served The HTTP server handles multiple protocols on the same port: * **REST** — CRUD operations on Harper resources via standard HTTP methods * **WebSockets** — Real-time bidirectional connections (via `server.ws()`) * **MQTT over WebSocket** — MQTT clients connecting over WebSocket (sub-protocol `mqtt`) * **Server-Sent Events** — Streaming updates to browser clients * **Operations API** — Management API (configurable to share or use separate port) ## TLS / HTTPS HTTPS support is enabled by setting `http.securePort` in `harperdb-config.yaml` and configuring the `tls` section with a certificate and private key. The same `tls` configuration is shared by HTTPS and MQTT secure connections. See [Configuration](/reference/v4/http/configuration.md) for TLS options and [Security](/reference/v4/security/overview.md) for certificate management details. ## HTTP/2 *Added in: v4.5.0* HTTP/2 can be enabled with the `http2: true` option in `harperdb-config.yaml`. When enabled, HTTP/2 applies to all API endpoints served on `http.securePort` (HTTP/2 requires TLS). ## Compression Harper automatically compresses HTTP responses using Brotli for clients that advertise `Accept-Encoding: br`. Compression applies when the response body exceeds the configured `compressionThreshold` (default 1200 bytes). Streaming query responses are always compressed for clients that support it (since their size is not known upfront). ## Logging HTTP request logging is not enabled by default. To enable it, add an `http.logging` block to your configuration. See [Configuration](/reference/v4/http/configuration.md#logging) for details. ## Related * [HTTP Configuration](/reference/v4/http/configuration.md) * [HTTP API](/reference/v4/http/api.md) * [REST Overview](/reference/v4/rest/overview.md) * [Security Overview](/reference/v4/security/overview.md) --- # TLS Configuration Harper uses a top-level `tls` section in `harperdb-config.yaml` to configure Transport Layer Security. This configuration is shared by the HTTP server (HTTPS), the MQTT broker (secure MQTT), and any TLS socket servers created via the [HTTP API](/reference/v4/http/api.md#serversocketlistener-options). The `operationsApi` section can optionally define its own `tls` block, which overrides the root `tls` for Operations API traffic only. See the [Operations API Configuration](/reference/v4/configuration/operations.md) for more details. Harper must be restarted for TLS configuration changes to take effect. ## TLS Configuration ``` tls: certificate: ~/hdb/keys/certificate.pem certificateAuthority: ~/hdb/keys/ca.pem privateKey: ~/hdb/keys/privateKey.pem ``` ### `tls.certificate` Type: `string` Default: `"/keys/certificate.pem"` Path to the PEM-encoded certificate file. ### `tls.certificateAuthority` Type: `string` Default: `"/keys/ca.pem"` Path to the PEM-encoded certificate authority (CA) file. Used to verify client certificates when mTLS is enabled. ### `tls.privateKey` Type: `string` Default: `"/keys/privateKey.pem"` Path to the PEM-encoded private key file. ### `tls.host` Type: `string | undefined` The domain name this certificate entry applies to, used for SNI (Server Name Indication) matching. Only relevant when `tls` is defined as an array. When omitted, the certificate's common name (CN) is used as the host name. ### `tls.ciphers` Type: `string | undefined` Default: `crypto.defaultCipherList` Colon-separated list of allowed TLS cipher suites. When omitted, Node.js [default ciphers](https://nodejs.org/api/crypto.html#nodejs-crypto-constants) are used. See Node.js [Modifying the default TLS cipher suite](https://nodejs.org/api/tls.html#modifying-the-default-tls-cipher-suite) for more information. ## Enabling HTTPS To enable HTTPS, set `http.securePort` in addition to the `tls` section: ``` http: securePort: 9927 tls: certificate: ~/hdb/keys/certificate.pem certificateAuthority: ~/hdb/keys/ca.pem privateKey: ~/hdb/keys/privateKey.pem ``` When `http.securePort` is set, Harper accepts plaintext connections on `http.port` and TLS connections on `http.securePort` simultaneously. ## Multi-Domain Certificates (SNI) To serve different certificates for different domains using Server Name Indication (SNI), define `tls` as an array of configuration objects. Each entry can optionally include a `host` property specifying which domain it applies to. If `host` is omitted, the certificate's common name and subject alternate names (SANs) are used. ``` tls: - certificate: ~/hdb/keys/certificate1.pem certificateAuthority: ~/hdb/keys/ca1.pem privateKey: ~/hdb/keys/privateKey1.pem host: example.com - certificate: ~/hdb/keys/certificate2.pem certificateAuthority: ~/hdb/keys/ca2.pem privateKey: ~/hdb/keys/privateKey2.pem # host omitted: certificate's CN is used ``` ## Operations API Override The `operationsApi` section can define its own `tls` block to use a separate certificate for the Operations API: ``` tls: certificate: ~/hdb/keys/certificate.pem certificateAuthority: ~/hdb/keys/ca.pem privateKey: ~/hdb/keys/privateKey.pem operationsApi: network: securePort: 9924 tls: certificate: ~/hdb/keys/ops-certificate.pem certificateAuthority: ~/hdb/keys/ops-ca.pem privateKey: ~/hdb/keys/ops-privateKey.pem ``` See the [Operations API Configuration](/reference/v4/configuration/operations.md) for more details. ## Related * [HTTP Configuration](/reference/v4/http/configuration.md) — `http.securePort`, `http.http2`, `http.mtls` * [HTTP Overview](/reference/v4/http/overview.md) * [Security mTLS Authentication](/reference/v4/security/mtls-authentication.md) --- # Harper Cloud Harper Cloud (also sometimes referred to as Harper Studio) was Harper's original PaaS offering. It has been fully replaced by [Harper Fabric](https://fabric.harper.fast). All users are encouraged to migrate or get started using Harper Fabric immediately. [Local Studio](/reference/v4/studio/overview.md) is still an available feature, and now uses the same client as Harper Fabric. Reach out to or join our community [Discord](https://harper.fast/discord) if you have questions. --- # Custom Functions Custom Functions were Harper's original mechanism for adding custom API endpoints and application logic to a Harper instance. They allowed developers to define Fastify-based HTTP routes that ran inside Harper with direct access to the database, and could be deployed across instances via Studio. Custom Functions were superseded by the [Components](/reference/v4/components/overview.md) system introduced in v4.2.0. Components provide the same capabilities with a more robust architecture, better tooling, and support for extensions and plugins. All users are encouraged to migrate Custom Functions to Components. See the [Components](/reference/v4/components/overview.md) documentation for the modern approach. Reach out to or join our community [Discord](https://harper.fast/discord) if you have questions. --- # Logging API ## `logger` The `logger` global is available in all JavaScript components without any imports. It writes structured log entries to the standard Harper log file (`hdb.log`) at the configured `logging.external` level and path. See [Logging Configuration](/reference/v4/logging/configuration.md#loggingexternal) for per-component log configuration. The `logger` global is a `MainLogger`. Calling `logger.withTag(tag)` returns a `TaggedLogger` scoped to that tag. ### `MainLogger` `MainLogger` always has all log-level methods defined. It also exposes `withTag()` to create a `TaggedLogger`. ``` interface MainLogger { trace(...messages: any[]): void; debug(...messages: any[]): void; info(...messages: any[]): void; warn(...messages: any[]): void; error(...messages: any[]): void; fatal(...messages: any[]): void; notify(...messages: any[]): void; withTag(tag: string): TaggedLogger; } ``` Each method corresponds to a log level. Only entries at or above the configured `logging.level` (or `logging.external.level`) are written. See [Log Levels](/reference/v4/logging/overview.md#log-levels) for the full hierarchy. ### `TaggedLogger` `TaggedLogger` is returned by `logger.withTag(tag)`. It prefixes every log entry with the given tag, making it easy to filter log output by component or context. Because `TaggedLogger` is bound to the configured log level at creation time, methods for levels that are currently disabled are `null`. Always use optional chaining (`?.`) when calling methods on a `TaggedLogger`. ``` interface TaggedLogger { trace: ((...messages: any[]) => void) | null; debug: ((...messages: any[]) => void) | null; info: ((...messages: any[]) => void) | null; warn: ((...messages: any[]) => void) | null; error: ((...messages: any[]) => void) | null; fatal: ((...messages: any[]) => void) | null; notify: ((...messages: any[]) => void) | null; } ``` `TaggedLogger` does not have a `withTag()` method. ### Usage #### Basic logging with `logger` ``` export class MyResource extends Resource { async get(id) { logger.debug('Fetching record', { id }); const record = await super.get(id); if (!record) { logger.warn('Record not found', { id }); } return record; } async put(record) { logger.info('Updating record', { id: record.id }); try { return await super.put(record); } catch (err) { logger.error('Failed to update record', err); throw err; } } } ``` #### Tagged logging with `withTag()` Create a tagged logger once per module or class and reuse it. Always use `?.` when calling methods since a given level may be `null` if it is below the configured log level. ``` const log = logger.withTag('my-resource'); export class MyResource extends Resource { async get(id) { log.debug?.('Fetching record', { id }); const record = await super.get(id); if (!record) { log.warn?.('Record not found', { id }); } return record; } async put(record) { log.info?.('Updating record', { id: record.id }); try { return await super.put(record); } catch (err) { log.error?.('Failed to update record', err); throw err; } } } ``` Tagged entries appear in the log with the tag included in the entry header: ``` 2023-03-09T14:25:05.269Z [info] [my-resource]: Updating record ``` ### Log Entry Format Entries written via `logger` appear in `hdb.log` with the standard format: ``` [] [/]: ``` Entries written via a `TaggedLogger` include the tag: ``` [] []: ``` For external components, the thread context is set automatically based on which worker thread executes the code. ## Related * [Logging Overview](/reference/v4/logging/overview.md) * [Logging Configuration](/reference/v4/logging/configuration.md) * [Logging Operations](/reference/v4/logging/operations.md) --- # Logging Configuration The `logging` section in `harperdb-config.yaml` controls standard log output. Many logging settings are applied dynamically without a restart (added in v4.6.0). ## Main Logger ### `logging.level` Type: `string` Default: `warn` Controls the verbosity of logs. Levels from least to most severe: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `notify`. Setting a level includes that level and all more-severe levels. ``` logging: level: warn ``` For example, `level: warn` results in `warn`, `error`, `fatal`, and `notify` logs. ### `logging.path` Type: `string` Default: `/log/hdb.log` Full file path for the log file. ``` logging: path: ~/hdb/log/hdb.log ``` ### `logging.root` Type: `string` Default: `/log` Directory path where log files are written. Use `path` to specify the full filename; use `root` to specify only the directory (Harper determines the filename). ``` logging: root: ~/hdb/log ``` ### `logging.file` Type: `boolean` Default: `true` Whether to write logs to a file. Disable if you want to use only standard streams. ``` logging: file: true ``` ### `logging.stdStreams` Type: `boolean` Default: `false` Log to `stdout`/`stderr` in addition to (or instead of) the log file. When enabled, run Harper in the foreground (`harper`, not `harper start`). ``` logging: stdStreams: true ``` ### `logging.console` Type: `boolean` Default: `false` Controls whether `console.log` and other `console.*` calls (and anything writing to `process.stdout`/`process.stderr` from JS components) are captured to the log file. ``` logging: console: true ``` ### `logging.auditLog` Type: `boolean` Default: `false` Enables audit (table transaction) logging. When enabled, Harper records every insert, update, and delete to a corresponding audit table. Audit log data is accessed via the `read_audit_log` operation. See [Database / Transaction Logging](/reference/v4/database/transaction.md) for details on using audit logs. ``` logging: auditLog: false ``` ### `logging.auditRetention` Type: `string | number` Default: `3d` How long audit log entries are retained before automatic eviction. Accepts duration strings (e.g., `3d`, `12h`) or milliseconds. ``` logging: auditRetention: 3d ``` ## Log Rotation Rotation provides systematic management of the `hdb.log` file — compressing, archiving, and replacing it on a schedule or size threshold. Rotation is triggered when either `interval` or `maxSize` is set. > `interval` and `maxSize` are approximates only. The log file may exceed these values slightly before rotation occurs. ### `logging.rotation.enabled` Type: `boolean` Default: `true` Enables log rotation. Rotation only activates when `interval` or `maxSize` is also set. ### `logging.rotation.compress` Type: `boolean` Default: `false` Compress rotated log files with gzip. ### `logging.rotation.interval` Type: `string` Default: `null` Time between rotations. Accepted units: `D` (days), `H` (hours), `M` (minutes). Example: `1D`, `12H`. ### `logging.rotation.maxSize` Type: `string` Default: `null` Maximum log file size before rotation. Accepted units: `K` (kilobytes), `M` (megabytes), `G` (gigabytes). Example: `100M`, `1G`. ### `logging.rotation.path` Type: `string` Default: `/log` Directory for storing rotated log files. Rotated files are named: `HDB-YYYY-MM-DDT-HH-MM-SSSZ.log`. ``` logging: rotation: enabled: true compress: false interval: 1D maxSize: 100M path: ~/hdb/log ``` ## Authentication Logging ### `logging.auditAuthEvents.logFailed` *Added in: v4.2.0* Type: `boolean` Default: `false` Log all failed authentication attempts. Example log entry: ``` [error] [auth-event]: {"username":"admin","status":"failure","type":"authentication","originating_ip":"127.0.0.1","request_method":"POST","path":"/","auth_strategy":"Basic"} ``` ### `logging.auditAuthEvents.logSuccessful` *Added in: v4.2.0* Type: `boolean` Default: `false` Log all successful authentication events. Example log entry: ``` [notify] [auth-event]: {"username":"admin","status":"success","type":"authentication","originating_ip":"127.0.0.1","request_method":"POST","path":"/","auth_strategy":"Basic"} ``` ``` logging: auditAuthEvents: logFailed: false logSuccessful: false ``` ## Per-Component Logging *Added in: v4.6.0* Harper supports independent logging configurations for different components. Each component logger can have its own `path`, `root`, `level`, `tag`, and `stdStreams` settings. All components default to the main `logging` configuration unless overridden. ### `logging.external` Logging configuration for all external components that use the [`logger` API](/reference/v4/logging/api.md). ``` logging: external: level: warn path: ~/hdb/log/apps.log ``` ### `http.logging` HTTP request logging. Disabled by default — defining this section enables it. ``` http: logging: level: info # info = all requests, warn = 4xx+, error = 5xx path: ~/hdb/log/http.log timing: true # log request duration headers: false # log request headers (verbose) id: true # assign and log a unique request ID per request ``` See [HTTP Configuration](/reference/v4/http/configuration.md) for full details. ### `mqtt.logging` MQTT logging configuration. Accepts standard logging options. ``` mqtt: logging: level: warn path: ~/hdb/log/mqtt.log stdStreams: false ``` ### `authentication.logging` Authentication subsystem logging. Accepts standard logging options. ``` authentication: logging: level: warn path: ~/hdb/log/auth.log ``` ### `replication.logging` Replication subsystem logging. Accepts standard logging options. ``` replication: logging: level: warn path: ~/hdb/log/replication.log ``` ### `tls.logging` TLS subsystem logging. Accepts standard logging options. ``` tls: logging: level: warn path: ~/hdb/log/tls.log ``` ### `storage.logging` Database storage subsystem logging. Accepts standard logging options. ``` storage: logging: level: warn path: ~/hdb/log/storage.log ``` ### `analytics.logging` Analytics subsystem logging. Accepts standard logging options. ``` analytics: logging: level: warn path: ~/hdb/log/analytics.log ``` ## Clustering Log Level Clustering has a separate log level due to its verbosity. Configure with `clustering.logLevel`. Valid levels from least verbose: `error`, `warn`, `info`, `debug`, `trace`. ``` clustering: logLevel: warn ``` ## Complete Example ``` logging: level: warn path: ~/hdb/log/hdb.log file: true stdStreams: false console: false auditLog: false auditRetention: 3d rotation: enabled: true compress: false interval: 1D maxSize: 100M path: ~/hdb/log auditAuthEvents: logFailed: false logSuccessful: false external: level: warn path: ~/hdb/log/apps.log http: logging: level: warn path: ~/hdb/log/http.log timing: true ``` ## Related * [Logging Overview](/reference/v4/logging/overview.md) * [Logging API](/reference/v4/logging/api.md) * [Logging Operations](/reference/v4/logging/operations.md) * [Database / Transaction Logging](/reference/v4/database/transaction.md) * [Configuration Overview](/reference/v4/configuration/overview.md) --- # Logging Operations Operations for reading the standard Harper log (`hdb.log`). All operations are restricted to `super_user` roles only. > Audit log and transaction log operations (`read_audit_log`, `read_transaction_log`, `delete_audit_logs_before`, `delete_transaction_logs_before`) are documented in [Database / Transaction Logging](/reference/v4/database/transaction.md). *** ## `read_log` Returns log entries from the primary Harper log (`hdb.log`) matching the provided criteria. *Restricted to super\_user roles only.* ### Parameters | Parameter | Required | Type | Description | | ----------- | -------- | ------ | ------------------------------------------------------------------------------------------------------ | | `operation` | Yes | string | Must be `"read_log"` | | `start` | No | number | Result offset to start from. Default: `0` (first entry in `hdb.log`). | | `limit` | No | number | Maximum number of entries to return. Default: `1000`. | | `level` | No | string | Filter by log level. One of: `notify`, `error`, `warn`, `info`, `debug`, `trace`. Default: all levels. | | `from` | No | string | Start of time window. Format: `YYYY-MM-DD` or `YYYY-MM-DD hh:mm:ss`. Default: first entry in log. | | `until` | No | string | End of time window. Format: `YYYY-MM-DD` or `YYYY-MM-DD hh:mm:ss`. Default: last entry in log. | | `order` | No | string | Sort order: `asc` or `desc` by timestamp. Default: maintains `hdb.log` order. | | `filter` | No | string | A substring that must appear in each returned log line. | ### Request ``` { "operation": "read_log", "start": 0, "limit": 1000, "level": "error", "from": "2021-01-25T22:05:27.464+0000", "until": "2021-01-25T23:05:27.464+0000", "order": "desc" } ``` ### Response ``` [ { "level": "notify", "message": "Connected to cluster server.", "timestamp": "2021-01-25T23:03:20.710Z", "thread": "main/0", "tags": [] }, { "level": "warn", "message": "Login failed", "timestamp": "2021-01-25T22:24:45.113Z", "thread": "http/9", "tags": [] }, { "level": "error", "message": "unknown attribute 'name and breed'", "timestamp": "2021-01-25T22:23:24.167Z", "thread": "http/9", "tags": [] } ] ``` ### Response Fields | Field | Type | Description | | ----------- | ------ | ------------------------------------------------------------------------------------------------------- | | `level` | string | Log level of the entry. | | `message` | string | Log message. | | `timestamp` | string | ISO 8601 timestamp when the event occurred. | | `thread` | string | Thread name and ID (e.g., `main/0`, `http/3`). | | `tags` | array | Additional context tags. Entries from components may include `custom-function` or other component tags. | ## Related * [Logging Overview](/reference/v4/logging/overview.md) * [Logging Configuration](/reference/v4/logging/configuration.md) * [Database / Transaction Logging](/reference/v4/database/transaction.md) * [Operations API Overview](/reference/v4/operations-api/overview.md) --- # Logging Harper's core logging system is used for diagnostics, monitoring, and observability. It has an extensive configuration system, and even supports feature-specific (per-component) configurations in latest versions. Furthermore, the `logger` global API is available for creating custom logs from any JavaScript application or plugin code. > If you are looking for information on Harper's Audit and Transaction logging system, refer to the [Database](/reference/v4/database/transaction.md) section. ## Log File *Changed in: v4.1.0* — All logs consolidated into a single `hdb.log` file All standard log output is written to `/log/hdb.log` (default: `~/hdb/log/hdb.log`). ## Log Entry Format Each log entry follows this structure: ``` [/] [] ...[]: ``` Example: ``` 2023-03-09T14:25:05.269Z [main/0] [notify]: HarperDB successfully started. ``` Fields: | Field | Description | | ----------- | -------------------------------------------------------------------------------------------- | | `timestamp` | ISO 8601 date/time when the event occurred. | | `level` | Severity level. See [Log Levels](#log-levels) below. | | `thread/id` | Name and ID of the thread that produced the log entry (generally, `main`, `http`, or `job`). | | `tags` | Additional context tags (e.g., `custom-function`, `auth-event`). Most entries have no tags. | | `message` | The log message. | ### Log Levels From least to most severe (most verbose to least verbose): | Level | Description | | -------- | --------------------------------------------------------------------------------------------- | | `trace` | Highly detailed internal execution tracing. | | `debug` | Diagnostic information useful during development. | | `info` | General operational events. | | `warn` | Potential issues that don't prevent normal operation. | | `error` | Errors that affect specific operations. | | `fatal` | Critical errors causing process termination. | | `notify` | Important operational milestones (e.g., "server started"). Always logged regardless of level. | The default log level is `warn`. Setting a level includes that level and all more-severe levels. For example, `warn` logs `warn`, `error`, `fatal`, and `notify`. ## Standard Streams *Changed in: v4.6.0* By default, logs are written only to the log file. To also log to `stdout`/`stderr`, set [`logging.stdStreams: true`](/reference/v4/logging/configuration.md#loggingstdstreams) (this is automatically enabled by the `DEFAULT_MODE=dev` configuration during installation). When logging to standard streams, run Harper in the foreground (i.e. `harper`, not `harper start`). As of v4.6.0, logging to standard streams does **not** include timestamps, and console logging (`console.log`, etc.) does not get forwarded to log files unless the [`logging.console: true`](/reference/v4/logging/configuration.md#loggingconsole) option is enabled. ## Logger API JavaScript components can use the `logger` global to write structured log entries: ``` logger.trace('detailed trace message'); logger.debug('debug info', { someContext: 'value' }); logger.info('informational message'); logger.warn('potential issue'); logger.error('error occurred', error); logger.fatal('fatal error'); logger.notify('server is ready'); ``` The `logger` global provides `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `notify` methods. The logger is based on the Node.js Console API. See [Logging API](/reference/v4/logging/api.md) for full details. ## Related * [Logging Configuration](/reference/v4/logging/configuration.md) * [Logging API](/reference/v4/logging/api.md) * [Logging Operations](/reference/v4/logging/operations.md) * [Database / Transaction Logging](/reference/v4/database/transaction.md) --- # MQTT Configuration The `mqtt` section in `harperdb-config.yaml` controls Harper's built-in MQTT broker. MQTT is enabled by default. Harper must be restarted for configuration changes to take effect. ## Minimal Example ``` mqtt: network: port: 1883 securePort: 8883 webSocket: true requireAuthentication: true ``` ## Ports ### `mqtt.network.port` Type: `integer` Default: `1883` The port for plaintext (non-TLS) MQTT connections. ### `mqtt.network.securePort` Type: `integer` Default: `8883` The port for secure MQTT connections (MQTTS). Uses the `tls` configuration for certificates. See [TLS Configuration](/reference/v4/http/tls.md) for certificate setup. ## WebSocket ### `mqtt.webSocket` Type: `boolean` Default: `true` Enables MQTT over WebSockets. When enabled, Harper handles WebSocket connections on the HTTP port (default `9926`) that specify the `mqtt` sub-protocol (`Sec-WebSocket-Protocol: mqtt`). This is required by the MQTT specification and should be set by any conformant MQTT-over-WebSocket client. ``` mqtt: webSocket: true ``` ## Authentication ### `mqtt.requireAuthentication` Type: `boolean` Default: `true` Controls whether credentials are required to establish an MQTT connection. When `true`, clients must authenticate with either a username/password or a valid mTLS client certificate. When set to `false`, unauthenticated connections are allowed. Unauthenticated clients are still subject to authorization on each publish and subscribe operation — by default, tables and resources do not grant access to unauthenticated users, but this can be configured at the resource level. ``` mqtt: requireAuthentication: true ``` ## mTLS ### `mqtt.network.mtls` *Added in: v4.3.0* Type: `boolean | object` Default: `false` Enables mutual TLS (mTLS) authentication for MQTT connections. When set to `true`, client certificates are verified against the CA specified in the root `tls.certificateAuthority` section. Authenticated connections use the `CN` (common name) from the client certificate's subject as the Harper username by default. ``` mqtt: network: mtls: true ``` For granular control, specify an object with the following optional properties: ### `mqtt.network.mtls.user` Type: `string | null` Default: Common Name from client certificate Specifies a fixed username to authenticate all mTLS connections as. When set, any connection that passes certificate verification authenticates as this user regardless of the certificate's CN. Setting to `null` disables credential-based authentication for mTLS connections. When combined with `required: true`, this enforces that clients must have a valid certificate AND provide separate credential-based authentication. ### `mqtt.network.mtls.required` Type: `boolean` Default: `false` When `true`, all incoming MQTT connections must provide a valid client certificate. Connections without a valid certificate are rejected. By default, clients can authenticate with either mTLS or standard username/password credentials. ### `mqtt.network.mtls.certificateAuthority` Type: `string` Default: Path from `tls.certificateAuthority` Path to the certificate authority (CA) file used to verify MQTT client certificates. By default, uses the CA configured in the root `tls` section. Set this if MQTT clients should be verified against a different CA than the one used for HTTP/TLS. ### `mqtt.network.mtls.certificateVerification` Type: `boolean | object` Default: `true` When mTLS is enabled, Harper verifies the revocation status of client certificates using OCSP (Online Certificate Status Protocol). This ensures revoked certificates cannot be used for authentication. Set to `false` to disable revocation checking, or configure as an object: | Property | Type | Default | Description | | ------------- | ------- | ------------- | ------------------------------------------------------------------------------------------------------ | | `timeout` | integer | `5000` | Maximum milliseconds to wait for an OCSP response. | | `cacheTtl` | integer | `3600000` | Milliseconds to cache successful verification results (default 1h). | | `failureMode` | string | `'fail-open'` | Behavior when OCSP verification fails: `'fail-open'` (allow, log warning) or `'fail-closed'` (reject). | ``` mqtt: network: mtls: required: true certificateVerification: failureMode: fail-closed timeout: 5000 cacheTtl: 3600000 ``` ## mTLS Examples ``` # Require client certificate + standard credentials (combined auth) mqtt: network: mtls: user: null required: true # Authenticate all mTLS connections as a fixed user mqtt: network: mtls: user: mqtt-service-account required: true # mTLS optional — clients can use mTLS or credentials mqtt: network: mtls: true ``` ## Logging ### `mqtt.logging` Type: `object` Default: disabled Configures logging for MQTT activity. Accepts the standard logging configuration options. ``` mqtt: logging: path: ~/hdb/log/mqtt.log level: warn stdStreams: false ``` | Option | Description | | ------------ | ------------------------------------------------- | | `path` | File path for the MQTT log output. | | `root` | Alternative to `path` — sets the log directory. | | `level` | Log level: `error`, `warn`, `info`, `debug`, etc. | | `tag` | Custom tag to prefix log entries. | | `stdStreams` | When `true`, also logs to stdout/stderr. | ## Complete Example ``` mqtt: network: port: 1883 securePort: 8883 mtls: required: false certificateAuthority: ~/hdb/keys/ca.pem certificateVerification: failureMode: fail-open timeout: 5000 cacheTtl: 3600000 webSocket: true requireAuthentication: true logging: level: warn path: ~/hdb/log/mqtt.log # TLS is a top-level section, shared with HTTP tls: certificate: ~/hdb/keys/certificate.pem certificateAuthority: ~/hdb/keys/ca.pem privateKey: ~/hdb/keys/privateKey.pem ``` ## Related * [MQTT Overview](/reference/v4/mqtt/overview.md) * [TLS Configuration](/reference/v4/http/tls.md) * [Security Overview](/reference/v4/security/overview.md) * [Configuration Overview](/reference/v4/configuration/overview.md) --- # MQTT *Added in: v4.2.0* Harper includes a built-in MQTT broker that provides real-time pub/sub messaging deeply integrated with the database. Unlike a generic MQTT broker, Harper's MQTT implementation connects topics directly to database records — publishing to a topic writes to the database, and subscribing to a topic delivers live updates for the corresponding record. ## How Topics Map to Database Records MQTT topics in Harper follow the same path convention as REST endpoints. If you define a table or resource with an endpoint path of `my-resource`, the corresponding MQTT topic namespace is `my-resource`. A topic of `my-resource/some-id` corresponds to the record with id `some-id` in the `my-resource` table (or custom resource). This means: * **Subscribing** to `my-resource/some-id` delivers notification messages whenever that record is updated or deleted. * The **current value** of the record is treated as the retained message for that topic. On subscription, the subscriber immediately receives the current record as the initial retained message — no separate GET request needed. * **Publishing** with the `retain` flag set replaces the record in the database (equivalent to a PUT operation). * **Publishing without** the `retain` flag delivers the message to current subscribers without writing to the database. Defining a table that creates a topic can be as simple as adding a table with no attributes to your [schema.graphql](/reference/v4/database/schema.md) in a Harper application: ``` type MyTopic @table @export ``` ## Protocol Support Harper supports MQTT versions **v3.1.1** and **v5**, with standard publish/subscribe capabilities. ### Topics and Wildcards Harper supports multi-level topics for both publishing and subscribing: * **Multi-level wildcard (`#`)** — Subscribe to `my-resource/#` to receive notifications for all records in that resource, including nested paths (`my-resource/some-id`, `my-resource/nested/id`). * **Single-level wildcard (`+`)** — Added in v4.3.0. Subscribe to `my-resource/+/status` to match any single path segment. ### QoS Levels * **QoS 0** — At most once delivery (fire and forget). * **QoS 1** — At least once delivery (acknowledged delivery). * **QoS 2** — Harper can perform the QoS 2 conversation but does not guarantee exactly-once delivery. ### Sessions * **Clean sessions** — Subscriptions and queued messages are discarded on disconnect. * **Durable sessions** — Subscriptions and queued messages are persisted across reconnects. ### Last Will *Added in: v4.3.0* Harper supports the MQTT Last Will and Testament feature. If a client disconnects unexpectedly, the broker publishes the configured will message on its behalf. ## Content Negotiation Harper handles structured data natively. Messages can be published and received in any supported structured format — JSON, CBOR, or MessagePack — and Harper stores and delivers them as structured objects. Different clients can independently choose their preferred format: one client may publish in JSON while another subscribes and receives in CBOR. ## Ordering and Distributed Delivery Harper is designed for distributed, low-latency message delivery. Messages are delivered to subscribers immediately on arrival — Harper does not delay delivery to coordinate consensus across nodes. In a distributed cluster, messages may arrive out of order due to network topology. The behavior depends on whether the message is retained or non-retained: * **Retained messages** (published with `retain: true`, or written via PUT/upsert) maintain eventual consistency across the cluster. Harper keeps the message with the latest timestamp as the winning record state. An out-of-order earlier message will not be re-delivered to clients; the cluster converges to the most recent state. * **Non-retained messages** are always delivered to local subscribers when received, even if they arrive out of order. Every message is delivered, prioritizing completeness over strict ordering. **Non-retained messages** are suited for applications like chat where every message must be delivered. **Retained messages** are suited for sensor readings or state updates where only the latest value matters. ## Authentication MQTT connections support two authentication methods: * **Credential-based** — Standard MQTT username/password in the CONNECT packet. * **mTLS** — Added in v4.3.0. Mutual TLS authentication using client certificates. The `CN` (common name) from the client certificate subject is used as the Harper username by default. Authentication is required by default (`requireAuthentication: true`). See [MQTT Configuration](/reference/v4/mqtt/configuration.md) for details on disabling authentication or configuring mTLS options. ## Server Events API JavaScript components can listen for MQTT connection events via `server.mqtt.events`: ``` server.mqtt.events.on('connected', (session, socket) => { console.log('client connected with id', session.clientId); }); ``` Available events: | Event | Description | | -------------- | ---------------------------------------------------- | | `connection` | Client establishes a TCP or WebSocket connection | | `connected` | Client completes MQTT handshake and is authenticated | | `auth-failed` | Client fails to authenticate | | `disconnected` | Client disconnects | ## Feature Support Matrix | Feature | Support | | --------------------------------------------- | ------------------------------------------------------------ | | MQTT v3.1.1 connections | ✅ | | MQTT v5 connections | ✅ | | Secure MQTTS (TLS) | ✅ | | MQTT over WebSockets | ✅ | | Authentication via username/password | ✅ | | Authentication via mTLS | ✅ (added v4.3.0) | | Publish | ✅ | | Subscribe | ✅ | | Multi-level wildcard (`#`) | ✅ | | Single-level wildcard (`+`) | ✅ (added v4.3.0) | | QoS 0 | ✅ | | QoS 1 | ✅ | | QoS 2 | Not fully supported — conversation supported, not guaranteed | | Keep-Alive monitoring | ✅ | | Clean session | ✅ | | Durable session | ✅ | | Distributed durable session | Not supported | | Last Will | ✅ | | MQTT V5 Subscribe retain handling | ✅ (added v4.3.0) | | MQTT V5 User properties | Not supported | | MQTT V5 Will properties | Not supported | | MQTT V5 Connection properties | Not supported | | MQTT V5 Connection acknowledgement properties | Not supported | | MQTT V5 Publish properties | Not supported | | MQTT V5 Subscribe properties (general) | Not supported | | MQTT V5 Ack properties | Not supported | | MQTT V5 AUTH command | Not supported | | MQTT V5 Shared subscriptions | Not supported | ## Related * [MQTT Configuration](/reference/v4/mqtt/configuration.md) * [HTTP Overview](/reference/v4/http/overview.md) * [Security Overview](/reference/v4/security/overview.md) * [Database Schema](/reference/v4/database/schema.md) * [REST Overview](/reference/v4/rest/overview.md) --- # Operations Reference This page lists all available Operations API operations, grouped by category. Each entry links to the feature section where the full documentation lives. For endpoint and authentication setup, see the [Operations API Overview](/reference/v4/operations-api/overview.md). *** ## Databases & Tables Operations for managing databases, tables, and attributes. Detailed documentation: [Database Overview](/reference/v4/database/overview.md) | Operation | Description | Role Required | | ------------------- | ------------------------------------------------------------------- | ------------- | | `describe_all` | Returns definitions of all databases and tables, with record counts | any | | `describe_database` | Returns all table definitions for a specified database | any | | `describe_table` | Returns the definition of a specified table | any | | `create_database` | Creates a new database | super\_user | | `drop_database` | Drops a database and all its tables/records | super\_user | | `create_table` | Creates a new table with optional schema and expiration | super\_user | | `drop_table` | Drops a table and all its records | super\_user | | `create_attribute` | Adds a new attribute to a table | super\_user | | `drop_attribute` | Removes an attribute and all its values from a table | super\_user | | `get_backup` | Returns a binary snapshot of a database for backup purposes | super\_user | ### `describe_all` Returns the definitions of all databases and tables within the database. Record counts above 5000 records are estimated; the response includes `estimated_record_range` when estimated. To force an exact count (requires full table scan), include `"exact_count": true`. ``` { "operation": "describe_all" } ``` ### `describe_database` Returns all table definitions within the specified database. ``` { "operation": "describe_database", "database": "dev" } ``` ### `describe_table` Returns the definition of a specific table. ``` { "operation": "describe_table", "table": "dog", "database": "dev" } ``` ### `create_database` Creates a new database. ``` { "operation": "create_database", "database": "dev" } ``` ### `drop_database` Drops a database and all its tables/records. Supports `"replicated": true` to propagate to all cluster nodes. ``` { "operation": "drop_database", "database": "dev" } ``` ### `create_table` Creates a new table. Optional fields: `database` (defaults to `data`), `attributes` (array defining schema), `expiration` (TTL in seconds). ``` { "operation": "create_table", "database": "dev", "table": "dog", "primary_key": "id" } ``` ### `drop_table` Drops a table and all associated records. Supports `"replicated": true`. ``` { "operation": "drop_table", "database": "dev", "table": "dog" } ``` ### `create_attribute` Creates a new attribute within a table. Harper auto-creates attributes on insert/update, but this can be used to pre-define them (e.g., for role-based permission setup). ``` { "operation": "create_attribute", "database": "dev", "table": "dog", "attribute": "is_adorable" } ``` ### `drop_attribute` Drops an attribute and all its values from the specified table. ``` { "operation": "drop_attribute", "database": "dev", "table": "dog", "attribute": "is_adorable" } ``` ### `get_backup` Returns a binary snapshot of the specified database (or individual table). Safe for backup while Harper is running. Specify `"table"` for a single table or `"tables"` for a set. ``` { "operation": "get_backup", "database": "dev" } ``` *** ## NoSQL Operations Operations for inserting, updating, deleting, and querying records using NoSQL. Detailed documentation: [REST Querying Reference](/reference/v4/rest/querying.md) | Operation | Description | Role Required | | ---------------------- | ------------------------------------------------------------------------- | ------------- | | `insert` | Inserts one or more records | any | | `update` | Updates one or more records by primary key | any | | `upsert` | Inserts or updates records | any | | `delete` | Deletes records by primary key | any | | `search_by_id` | Retrieves records by primary key | any | | `search_by_value` | Retrieves records matching a value on any attribute | any | | `search_by_conditions` | Retrieves records matching complex conditions with sorting and pagination | any | ### `insert` Inserts one or more records. If a primary key is not provided, a GUID or auto-increment value is generated. ``` { "operation": "insert", "database": "dev", "table": "dog", "records": [{ "id": 1, "dog_name": "Penny" }] } ``` ### `update` Updates one or more records. Primary key must be supplied for each record. ``` { "operation": "update", "database": "dev", "table": "dog", "records": [{ "id": 1, "weight_lbs": 38 }] } ``` ### `upsert` Updates existing records and inserts new ones. Matches on primary key if provided. ``` { "operation": "upsert", "database": "dev", "table": "dog", "records": [{ "id": 1, "weight_lbs": 40 }] } ``` ### `delete` Deletes records by primary key values. ``` { "operation": "delete", "database": "dev", "table": "dog", "ids": [1, 2] } ``` ### `search_by_id` Returns records matching the given primary key values. Use `"get_attributes": ["*"]` to return all attributes. ``` { "operation": "search_by_id", "database": "dev", "table": "dog", "ids": [1, 2], "get_attributes": ["dog_name", "breed_id"] } ``` ### `search_by_value` Returns records with a matching value on any attribute. Supports wildcards (e.g., `"Ky*"`). ``` { "operation": "search_by_value", "database": "dev", "table": "dog", "attribute": "owner_name", "value": "Ky*", "get_attributes": ["id", "dog_name"] } ``` ### `search_by_conditions` Returns records matching one or more conditions. Supports `operator` (`and`/`or`), `offset`, `limit`, nested `conditions` groups, and `sort` with multi-level tie-breaking. ``` { "operation": "search_by_conditions", "database": "dev", "table": "dog", "operator": "and", "limit": 10, "get_attributes": ["*"], "conditions": [{ "attribute": "age", "comparator": "between", "value": [5, 8] }] } ``` *** ## Bulk Operations Operations for bulk import/export of data. Detailed documentation: [Database Jobs](/reference/v4/database/jobs.md) | Operation | Description | Role Required | | ----------------------- | -------------------------------------------------------------- | ------------- | | `export_local` | Exports query results to a local file in JSON or CSV | super\_user | | `csv_data_load` | Ingests CSV data provided inline | any | | `csv_file_load` | Ingests CSV data from a server-local file path | any | | `csv_url_load` | Ingests CSV data from a URL | any | | `export_to_s3` | Exports query results to AWS S3 | super\_user | | `import_from_s3` | Imports CSV or JSON data from AWS S3 | any | | `delete_records_before` | Deletes records older than a given timestamp (local node only) | super\_user | All bulk import/export operations are asynchronous and return a job ID. Use [`get_job`](#get_job) to check status. ### `export_local` Exports query results to a local path on the server. Formats: `json` or `csv`. ``` { "operation": "export_local", "format": "json", "path": "/data/", "search_operation": { "operation": "sql", "sql": "SELECT * FROM dev.dog" } } ``` ### `csv_data_load` Ingests inline CSV data. Actions: `insert` (default), `update`, `upsert`. ``` { "operation": "csv_data_load", "database": "dev", "table": "dog", "action": "insert", "data": "id,name\n1,Penny\n" } ``` ### `csv_file_load` Ingests CSV from a file path on the server running Harper. ``` { "operation": "csv_file_load", "database": "dev", "table": "dog", "file_path": "/home/user/imports/dogs.csv" } ``` ### `csv_url_load` Ingests CSV from a URL. ``` { "operation": "csv_url_load", "database": "dev", "table": "dog", "csv_url": "https://example.com/dogs.csv" } ``` ### `export_to_s3` Exports query results to an AWS S3 bucket as JSON or CSV. ``` { "operation": "export_to_s3", "format": "json", "s3": { "aws_access_key_id": "YOUR_KEY", "aws_secret_access_key": "YOUR_SECRET", "bucket": "my-bucket", "key": "dogs.json", "region": "us-east-1" }, "search_operation": { "operation": "sql", "sql": "SELECT * FROM dev.dog" } } ``` ### `import_from_s3` Imports CSV or JSON from an AWS S3 bucket. File must include a valid `.csv` or `.json` extension. ``` { "operation": "import_from_s3", "database": "dev", "table": "dog", "s3": { "aws_access_key_id": "YOUR_KEY", "aws_secret_access_key": "YOUR_SECRET", "bucket": "my-bucket", "key": "dogs.csv", "region": "us-east-1" } } ``` ### `delete_records_before` Deletes records older than the specified timestamp from the local node only. Clustered nodes retain their data. ``` { "operation": "delete_records_before", "date": "2021-01-25T23:05:27.464", "schema": "dev", "table": "dog" } ``` *** ## SQL Operations Operations for executing SQL statements. warning Harper SQL is intended for data investigation and use cases where performance is not a priority. For production workloads, use NoSQL or REST operations. SQL performance optimizations are on the roadmap. Detailed documentation: [SQL Reference](/reference/v4/operations-api/sql.md) | Operation | Description | Role Required | | --------- | ------------------------------------------------------------------ | ------------- | | `sql` | Executes a SQL `SELECT`, `INSERT`, `UPDATE`, or `DELETE` statement | any | ### `sql` Executes a standard SQL statement. ``` { "operation": "sql", "sql": "SELECT * FROM dev.dog WHERE id = 1" } ``` *** ## Users & Roles Operations for managing users and role-based access control (RBAC). Detailed documentation: [Users & Roles Operations](/reference/v4/users-and-roles/operations.md) | Operation | Description | Role Required | | ------------ | --------------------------------------------------- | ------------- | | `list_roles` | Returns all roles | super\_user | | `add_role` | Creates a new role with permissions | super\_user | | `alter_role` | Modifies an existing role's permissions | super\_user | | `drop_role` | Deletes a role (role must have no associated users) | super\_user | | `list_users` | Returns all users | super\_user | | `user_info` | Returns data for the authenticated user | any | | `add_user` | Creates a new user | super\_user | | `alter_user` | Modifies an existing user's credentials or role | super\_user | | `drop_user` | Deletes a user | super\_user | ### `list_roles` Returns all roles defined in the instance. ``` { "operation": "list_roles" } ``` ### `add_role` Creates a new role with the specified permissions. The `permission` object maps database names to table-level access rules (`read`, `insert`, `update`, `delete`). Set `super_user: true` to grant full access. ``` { "operation": "add_role", "role": "developer", "permission": { "super_user": false, "dev": { "tables": { "dog": { "read": true, "insert": true, "update": true, "delete": false } } } } } ``` ### `alter_role` Modifies an existing role's name or permissions. Requires the role's `id` (returned by `list_roles`). ``` { "operation": "alter_role", "id": "f92162e2-cd17-450c-aae0-372a76859038", "role": "senior_developer", "permission": { "super_user": false, "dev": { "tables": { "dog": { "read": true, "insert": true, "update": true, "delete": true } } } } } ``` ### `drop_role` Deletes a role. The role must have no associated users before it can be dropped. ``` { "operation": "drop_role", "id": "f92162e2-cd17-450c-aae0-372a76859038" } ``` ### `list_users` Returns all users. ``` { "operation": "list_users" } ``` ### `user_info` Returns data for the currently authenticated user. ``` { "operation": "user_info" } ``` ### `add_user` Creates a new user. `username` cannot be changed after creation. `password` is stored encrypted. ``` { "operation": "add_user", "role": "developer", "username": "hdb_user", "password": "password", "active": true } ``` ### `alter_user` Modifies an existing user's password, role, or active status. All fields except `username` are optional. ``` { "operation": "alter_user", "username": "hdb_user", "password": "new_password", "role": "senior_developer", "active": true } ``` ### `drop_user` Deletes a user by username. ``` { "operation": "drop_user", "username": "hdb_user" } ``` See [Users & Roles Operations](/reference/v4/users-and-roles/operations.md) for full documentation including permission object structure. *** ## Token Authentication Operations for JWT token creation and refresh. Detailed documentation: [JWT Authentication](/reference/v4/security/jwt-authentication.md) | Operation | Description | Role Required | | ------------------------------ | ------------------------------------------------------- | ---------------------- | | `create_authentication_tokens` | Creates an operation token and refresh token for a user | none (unauthenticated) | | `refresh_operation_token` | Creates a new operation token from a refresh token | any | ### `create_authentication_tokens` Does not require prior authentication. Returns `operation_token` (short-lived JWT) and `refresh_token` (long-lived JWT). ``` { "operation": "create_authentication_tokens", "username": "my-user", "password": "my-password" } ``` ### `refresh_operation_token` Creates a new operation token from an existing refresh token. ``` { "operation": "refresh_operation_token", "refresh_token": "EXISTING_REFRESH_TOKEN" } ``` *** ## Components Operations for deploying and managing Harper components (applications, extensions, plugins). Detailed documentation: [Components Overview](/reference/v4/components/overview.md) | Operation | Description | Role Required | | ---------------------- | ----------------------------------------------------------------------- | ------------- | | `add_component` | Creates a new component project from a template | super\_user | | `deploy_component` | Deploys a component via payload (tar) or package reference (NPM/GitHub) | super\_user | | `package_component` | Packages a component project into a base64-encoded tar | super\_user | | `drop_component` | Deletes a component or a file within a component | super\_user | | `get_components` | Lists all component files and config | super\_user | | `get_component_file` | Returns the contents of a file within a component | super\_user | | `set_component_file` | Creates or updates a file within a component | super\_user | | `add_ssh_key` | Adds an SSH key for deploying from private repositories | super\_user | | `update_ssh_key` | Updates an existing SSH key | super\_user | | `delete_ssh_key` | Deletes an SSH key | super\_user | | `list_ssh_keys` | Lists all configured SSH key names | super\_user | | `set_ssh_known_hosts` | Overwrites the SSH known\_hosts file | super\_user | | `get_ssh_known_hosts` | Returns the contents of the SSH known\_hosts file | super\_user | | `install_node_modules` | *(Deprecated)* Run npm install on component projects | super\_user | ### `deploy_component` Deploys a component. The `package` option accepts any valid NPM reference including GitHub repos (`HarperDB/app#semver:v1.0.0`), tarballs, or NPM packages. The `payload` option accepts a base64-encoded tar string from `package_component`. Supports `"replicated": true` and `"restart": true` or `"restart": "rolling"`. ``` { "operation": "deploy_component", "project": "my-app", "package": "my-org/my-app#semver:v1.2.3", "replicated": true, "restart": "rolling" } ``` ### `add_ssh_key` Adds an SSH key (must be ed25519) for authenticating deployments from private repositories. ``` { "operation": "add_ssh_key", "name": "my-key", "key": "-----BEGIN OPENSSH PRIVATE KEY-----\n...\n-----END OPENSSH PRIVATE KEY-----\n", "host": "my-key.github.com", "hostname": "github.com" } ``` *** ## Replication & Clustering Operations for configuring and managing Harper cluster replication. Detailed documentation: [Replication & Clustering](/reference/v4/replication/clustering.md) | Operation | Description | Role Required | | ----------------------- | --------------------------------------------------------------- | ------------- | | `add_node` | Adds a Harper instance to the cluster | super\_user | | `update_node` | Modifies an existing node's subscriptions | super\_user | | `remove_node` | Removes a node from the cluster | super\_user | | `cluster_status` | Returns current cluster connection status | super\_user | | `configure_cluster` | Bulk-creates/resets cluster subscriptions across multiple nodes | super\_user | | `cluster_set_routes` | Adds routes to the replication routes config (PATCH/upsert) | super\_user | | `cluster_get_routes` | Returns the current replication routes config | super\_user | | `cluster_delete_routes` | Removes routes from the replication routes config | super\_user | ### `add_node` Adds a remote Harper node to the cluster. If `subscriptions` are not provided, a fully replicating cluster is created. Optional fields: `verify_tls`, `authorization`, `retain_authorization`, `revoked_certificates`, `shard`. ``` { "operation": "add_node", "hostname": "server-two", "verify_tls": false, "authorization": { "username": "admin", "password": "password" } } ``` ### `cluster_status` Returns connection state for all cluster nodes, including per-database socket status and replication timing statistics (`lastCommitConfirmed`, `lastReceivedRemoteTime`, `lastReceivedLocalTime`). ``` { "operation": "cluster_status" } ``` ### `configure_cluster` Resets and replaces the entire clustering configuration. Each entry follows the `add_node` schema. ``` { "operation": "configure_cluster", "connections": [ { "hostname": "server-two", "subscriptions": [{ "database": "dev", "table": "dog", "subscribe": true, "publish": true }] } ] } ``` *** ## Configuration Operations for reading and updating Harper configuration. Detailed documentation: [Configuration Overview](/reference/v4/configuration/overview.md) | Operation | Description | Role Required | | ------------------- | ---------------------------------------------------------------- | ------------- | | `set_configuration` | Modifies Harper configuration file parameters (requires restart) | super\_user | | `get_configuration` | Returns the current Harper configuration | super\_user | ### `set_configuration` Updates configuration parameters in `harperdb-config.yaml`. A restart (`restart` or `restart_service`) is required for changes to take effect. ``` { "operation": "set_configuration", "logging_level": "trace", "clustering_enabled": true } ``` ### `get_configuration` Returns the full current configuration object. ``` { "operation": "get_configuration" } ``` *** ## System Operations for restarting Harper and managing system state. | Operation | Description | Role Required | | -------------------- | ----------------------------------------------------- | ------------- | | `restart` | Restarts the Harper instance | super\_user | | `restart_service` | Restarts a specific Harper service | super\_user | | `system_information` | Returns detailed host system metrics | super\_user | | `set_status` | Sets an application-specific status value (in-memory) | super\_user | | `get_status` | Returns a previously set status value | super\_user | | `clear_status` | Removes a status entry | super\_user | ### `restart` Restarts all Harper processes. May take up to 60 seconds. ``` { "operation": "restart" } ``` ### `restart_service` Restarts a specific service. `service` must be one of: `http_workers`, `clustering_config`, `clustering`. Supports `"replicated": true` for a rolling cluster restart. ``` { "operation": "restart_service", "service": "http_workers" } ``` ### `system_information` Returns system metrics including CPU, memory, disk, network, and Harper process info. Optionally filter by `attributes` array (e.g., `["cpu", "memory", "replication"]`). ``` { "operation": "system_information" } ``` ### `set_status` / `get_status` / `clear_status` Manage in-memory application status values. Status types: `primary`, `maintenance`, `availability` (availability only accepts `'Available'` or `'Unavailable'`). Status is not persisted across restarts. ``` { "operation": "set_status", "id": "primary", "status": "active" } ``` *** ## Jobs Operations for querying background job status. Detailed documentation: [Database Jobs](/reference/v4/database/jobs.md) | Operation | Description | Role Required | | --------------------------- | ------------------------------------------------ | ------------- | | `get_job` | Returns status and results for a specific job ID | any | | `search_jobs_by_start_date` | Returns jobs within a specified time window | super\_user | ### `get_job` Returns job status (`COMPLETE`, `IN_PROGRESS`, `ERROR`), timing, and result message for the specified job ID. Bulk import/export operations return a job ID on initiation. ``` { "operation": "get_job", "id": "4a982782-929a-4507-8794-26dae1132def" } ``` ### `search_jobs_by_start_date` Returns all jobs started within the specified datetime range. ``` { "operation": "search_jobs_by_start_date", "from_date": "2021-01-25T22:05:27.464+0000", "to_date": "2021-01-25T23:05:27.464+0000" } ``` *** ## Logs Operations for reading Harper logs. Detailed documentation: [Logging Operations](/reference/v4/logging/operations.md) | Operation | Description | Role Required | | -------------------------------- | ---------------------------------------------------------------------- | ------------- | | `read_log` | Returns entries from the primary `hdb.log` | super\_user | | `read_transaction_log` | Returns transaction history for a table | super\_user | | `delete_transaction_logs_before` | Deletes transaction log entries older than a timestamp | super\_user | | `read_audit_log` | Returns verbose audit history for a table (requires audit log enabled) | super\_user | | `delete_audit_logs_before` | Deletes audit log entries older than a timestamp | super\_user | ### `read_log` Returns entries from `hdb.log`. Filter by `level` (`notify`, `error`, `warn`, `info`, `debug`, `trace`), date range (`from`, `until`), and text `filter`. ``` { "operation": "read_log", "start": 0, "limit": 100, "level": "error" } ``` ### `read_transaction_log` Returns transaction history for a specific table. Optionally filter by `from`/`to` (millisecond epoch) and `limit`. ``` { "operation": "read_transaction_log", "schema": "dev", "table": "dog", "limit": 10 } ``` ### `read_audit_log` Returns verbose audit history including original record state. Requires `logging.auditLog: true` in configuration. Filter by `search_type`: `hash_value`, `timestamp`, or `username`. ``` { "operation": "read_audit_log", "schema": "dev", "table": "dog", "search_type": "username", "search_values": ["admin"] } ``` *** ## Certificate Management Operations for managing TLS certificates in the `hdb_certificate` system table. Detailed documentation: [Certificate Management](/reference/v4/security/certificate-management.md) | Operation | Description | Role Required | | -------------------- | ---------------------------------------------- | ------------- | | `add_certificate` | Adds or updates a certificate | super\_user | | `remove_certificate` | Removes a certificate and its private key file | super\_user | | `list_certificates` | Lists all certificates | super\_user | ### `add_certificate` Adds a certificate to `hdb_certificate`. If a `private_key` is provided, it is written to `/keys/` (not stored in the table). If no private key is provided, the operation searches for a matching one on disk. ``` { "operation": "add_certificate", "name": "my-cert", "certificate": "-----BEGIN CERTIFICATE-----...", "is_authority": false, "private_key": "-----BEGIN RSA PRIVATE KEY-----..." } ``` *** ## Analytics Operations for querying analytics metrics. Detailed documentation: [Analytics Operations](/reference/v4/analytics/operations.md) | Operation | Description | Role Required | | ----------------- | ----------------------------------------------- | ------------- | | `get_analytics` | Retrieves analytics data for a specified metric | any | | `list_metrics` | Lists available analytics metrics | any | | `describe_metric` | Returns the schema of a specific metric | any | ### `get_analytics` Retrieves analytics data. Supports `start_time`/`end_time` (Unix ms), `get_attributes`, and `conditions` (same format as `search_by_conditions`). ``` { "operation": "get_analytics", "metric": "resource-usage", "start_time": 1769198332754, "end_time": 1769198532754 } ``` ### `list_metrics` Returns available metric names. Filter by `metric_types`: `custom`, `builtin` (default: `builtin`). ``` { "operation": "list_metrics" } ``` *** ## Registration & Licensing Operations for license management. | Operation | Description | Role Required | | ----------------------- | -------------------------------------------------- | ------------- | | `registration_info` | Returns registration and version information | any | | `install_usage_license` | Installs a Harper usage license block | super\_user | | `get_usage_licenses` | Returns all usage licenses with consumption counts | super\_user | | `get_fingerprint` | *(Deprecated)* Returns the machine fingerprint | super\_user | | `set_license` | *(Deprecated)* Sets a license key | super\_user | ### `registration_info` Returns the instance registration status, version, RAM allocation, and license expiration. ``` { "operation": "registration_info" } ``` ### `install_usage_license` Installs a usage license block. A license is a JWT-like structure (`header.payload.signature`) signed by Harper. Multiple blocks may be installed; earliest blocks are consumed first. ``` { "operation": "install_usage_license", "license": "abc...0123.abc...0123.abc...0123" } ``` ### `get_usage_licenses` Returns all usage licenses (including expired/exhausted) with current consumption counts. Optionally filter by `region`. ``` { "operation": "get_usage_licenses" } ``` *** ## Deprecated Operations The following operations are deprecated and should not be used in new code. ### Custom Functions (Deprecated) Custom Functions were the precursor to the Component architecture introduced in v4.2.0. These operations are preserved for backward compatibility. *Deprecated in: v4.2.0* (moved to legacy in v4.7+) For modern equivalents, see [Components Overview](/reference/v4/components/overview.md). | Operation | Description | | --------------------------------- | ------------------------------------------------ | | `custom_functions_status` | Returns Custom Functions server status | | `get_custom_functions` | Lists all Custom Function projects | | `get_custom_function` | Returns a Custom Function file's content | | `set_custom_function` | Creates or updates a Custom Function file | | `drop_custom_function` | Deletes a Custom Function file | | `add_custom_function_project` | Creates a new Custom Function project | | `drop_custom_function_project` | Deletes a Custom Function project | | `package_custom_function_project` | Packages a Custom Function project as base64 tar | | `deploy_custom_function_project` | Deploys a packaged Custom Function project | ### Other Deprecated Operations | Operation | Replaced By | | ---------------------- | ------------------------------------------------------------------- | | `install_node_modules` | Handled automatically by `deploy_component` and `restart` | | `get_fingerprint` | Use `registration_info` | | `set_license` | Use `install_usage_license` | | `search_by_hash` | Use `search_by_id` | | `search_attribute` | Use `attribute` field in `search_by_value` / `search_by_conditions` | | `search_value` | Use `value` field in `search_by_value` / `search_by_conditions` | | `search_type` | Use `comparator` field in `search_by_conditions` | --- # Operations API The Operations API provides a comprehensive set of capabilities for configuring, deploying, administering, and controlling Harper. It is the primary programmatic interface for all administrative and operational tasks that are not handled through the REST interface. ## Endpoint All Operations API requests are sent as HTTP POST requests to the Operations API endpoint. By default, this listens on port `9925` on the root path: ``` POST http://:9925/ ``` See [Configuration Overview](/reference/v4/configuration/overview.md) for how to change the port and other network settings (`operationsApi.network.port`, `operationsApi.network.securePort`). ## Request Format Each request body must be a JSON object with an `operation` field that identifies the operation to perform: ``` POST https://my-harper-server:9925/ Authorization: Basic YourBase64EncodedUser:Pass Content-Type: application/json { "operation": "create_table", "table": "my-table" } ``` ## Authentication Operations API requests must be authenticated. Harper supports two authentication methods: * **Basic Auth**: Base64-encoded `username:password` in the `Authorization` header. See [Basic Authentication](/reference/v4/security/basic-authentication.md). * **JWT**: A Bearer token in the `Authorization` header, obtained via `create_authentication_tokens`. See [JWT Authentication](/reference/v4/security/jwt-authentication.md). The `create_authentication_tokens` operation itself does not require prior authentication — it accepts a username and password and returns an operation token and refresh token. ## Example with curl ``` curl --location --request POST 'https://my-harper-server:9925/' \ --header 'Authorization: Basic YourBase64EncodedUser:Pass' \ --header 'Content-Type: application/json' \ --data-raw '{ "operation": "create_table", "table": "my-table" }' ``` ## Authorization Most operations are restricted to `super_user` roles. This is noted in the documentation for each operation. Some operations (such as `user_info`, `get_job`, and `create_authentication_tokens`) are available to all authenticated users. ## Operations Reference Operations are grouped by topic. See [Operations](/reference/v4/operations-api/operations.md) for the complete reference list. **Topic categories:** | Category | Description | Detailed Docs | | ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------- | | [Databases & Tables](/reference/v4/operations-api/operations.md#databases--tables) | Create and manage databases, tables, and attributes | [Database Overview](/reference/v4/database/overview.md) | | [NoSQL Operations](/reference/v4/operations-api/operations.md#nosql-operations) | Insert, update, upsert, delete, and query records | [REST Querying Reference](/reference/v4/rest/querying.md) | | [Bulk Operations](/reference/v4/operations-api/operations.md#bulk-operations) | CSV/S3 import and export, batch delete | [Database Jobs](/reference/v4/database/jobs.md) | | [SQL Operations](/reference/v4/operations-api/operations.md#sql-operations) | Execute SQL statements (use for investigation, not production) | — | | [Users & Roles](/reference/v4/operations-api/operations.md#users--roles) | Manage users and role-based access control | [Users & Roles Operations](/reference/v4/users-and-roles/operations.md) | | [Token Authentication](/reference/v4/operations-api/operations.md#token-authentication) | Create and refresh JWT tokens | [JWT Authentication](/reference/v4/security/jwt-authentication.md) | | [Components](/reference/v4/operations-api/operations.md#components) | Deploy and manage Harper components | [Components Overview](/reference/v4/components/overview.md) | | [Replication & Clustering](/reference/v4/operations-api/operations.md#replication--clustering) | Configure cluster topology and replication | [Replication & Clustering](/reference/v4/replication/clustering.md) | | [Configuration](/reference/v4/operations-api/operations.md#configuration) | Read and update Harper configuration | — | | [System](/reference/v4/operations-api/operations.md#system) | Restart, system information, status management | — | | [Jobs](/reference/v4/operations-api/operations.md#jobs) | Query background job status | [Database Jobs](/reference/v4/database/jobs.md) | | [Logs](/reference/v4/operations-api/operations.md#logs) | Read standard, transaction, and audit logs | [Logging Operations](/reference/v4/logging/operations.md) | | [Certificate Management](/reference/v4/operations-api/operations.md#certificate-management) | Manage TLS certificates | [Certificate Management](/reference/v4/security/certificate-management.md) | | [Analytics](/reference/v4/operations-api/operations.md#analytics) | Query analytics metrics | [Analytics Operations](/reference/v4/analytics/operations.md) | | [Registration & Licensing](/reference/v4/operations-api/operations.md#registration--licensing) | License management | — | ## Past Release API Documentation For API documentation prior to v4.0, see [olddocs.harperdb.io](https://olddocs.harperdb.io). --- # SQL warning SQL querying is not recommended for production use or on large tables. SQL queries often do not utilize indexes and are not optimized for performance. Use the [REST interface](/reference/v4/rest/overview.md) for production data access — it provides a more stable, secure, and performant interface. SQL is intended for ad-hoc data investigation and administrative queries. Harper includes a SQL interface supporting SELECT, INSERT, UPDATE, and DELETE operations. Tables are referenced using `database.table` notation (e.g., `dev.dog`). ## Operations API SQL queries are executed via the Operations API using the `sql` operation: * `operation` *(required)* — must be `sql` * `sql` *(required)* — the SQL statement to execute ### Select ``` { "operation": "sql", "sql": "SELECT * FROM dev.dog WHERE id = 1" } ``` ### Insert ``` { "operation": "sql", "sql": "INSERT INTO dev.dog (id, dog_name) VALUE (22, 'Simon')" } ``` Response: ``` { "message": "inserted 1 of 1 records", "inserted_hashes": [22], "skipped_hashes": [] } ``` ### Update ``` { "operation": "sql", "sql": "UPDATE dev.dog SET dog_name = 'penelope' WHERE id = 1" } ``` ### Delete ``` { "operation": "sql", "sql": "DELETE FROM dev.dog WHERE id = 1" } ``` *** ## SELECT Syntax ``` SELECT * FROM dev.dog SELECT id, dog_name, age FROM dev.dog SELECT * FROM dev.dog ORDER BY age SELECT * FROM dev.dog ORDER BY age DESC SELECT DISTINCT breed_id FROM dev.dog SELECT COUNT(*) FROM dev.dog WHERE age > 3 ``` ### Joins Supported join types: `INNER JOIN`, `LEFT [OUTER] JOIN`, `RIGHT [OUTER] JOIN`, `FULL OUTER JOIN`, `CROSS JOIN`. ``` SELECT d.id, d.dog_name, b.name FROM dev.dog AS d INNER JOIN dev.breed AS b ON d.breed_id = b.id WHERE d.owner_name IN ('Kyle', 'Zach') ORDER BY d.dog_name ``` *** ## Features Matrix | INSERT | | | ---------------------------------- | - | | Values — multiple values supported | ✔ | | Sub-SELECT | ✗ | | UPDATE | | | -------------- | - | | SET | ✔ | | Sub-SELECT | ✗ | | Conditions | ✔ | | Date Functions | ✔ | | Math Functions | ✔ | | DELETE | | | ---------- | - | | FROM | ✔ | | Sub-SELECT | ✗ | | Conditions | ✔ | | SELECT | | | ------------------- | - | | Column SELECT | ✔ | | Aliases | ✔ | | Aggregate Functions | ✔ | | Date Functions | ✔ | | Math Functions | ✔ | | Constant Values | ✔ | | DISTINCT | ✔ | | Sub-SELECT | ✗ | | FROM | | | ---------------- | - | | Multi-table JOIN | ✔ | | INNER JOIN | ✔ | | LEFT OUTER JOIN | ✔ | | LEFT INNER JOIN | ✔ | | RIGHT OUTER JOIN | ✔ | | RIGHT INNER JOIN | ✔ | | FULL JOIN | ✔ | | UNION | ✗ | | Sub-SELECT | ✗ | | TOP | ✔ | | WHERE | | | ---------------- | - | | Multi-Conditions | ✔ | | Wildcards | ✔ | | IN | ✔ | | LIKE | ✔ | | AND, OR, NOT | ✔ | | NULL | ✔ | | BETWEEN | ✔ | | EXISTS, ANY, ALL | ✔ | | Compare columns | ✔ | | Date Functions | ✔ | | Sub-SELECT | ✗ | | GROUP BY | | | --------------------- | - | | Multi-Column GROUP BY | ✔ | | HAVING | | | ----------------------------- | - | | Aggregate function conditions | ✔ | | ORDER BY | | | --------------------- | - | | Multi-Column ORDER BY | ✔ | | Aliases | ✔ | *** ## Functions ### Aggregate | Function | Description | | ---------------------- | ----------------------------------------------------- | | `AVG(expr)` | Average of a numeric expression. | | `COUNT(col)` | Count of rows matching the criteria (nulls excluded). | | `MAX(col)` | Largest value in a column. | | `MIN(col)` | Smallest value in a column. | | `SUM(col)` | Sum of numeric values. | | `GROUP_CONCAT(expr)` | Comma-separated string of non-null values. | | `ARRAY(expr)` | Returns a list of data as a field. | | `DISTINCT_ARRAY(expr)` | Returns a deduplicated list. | ### Conversion | Function | Description | | ---------------------------------- | ------------------------------------------ | | `CAST(expr AS datatype)` | Converts a value to the specified type. | | `CONVERT(datatype, expr[, style])` | Converts a value from one type to another. | ### String | Function | Description | | ----------------------------- | ------------------------------------------------------- | | `CONCAT(s1, s2, ...)` | Joins strings together. | | `CONCAT_WS(sep, s1, s2, ...)` | Joins strings with a separator. | | `INSTR(s1, s2)` | Position of s2 within s1. | | `LEN(s)` | Length of a string. | | `LOWER(s)` | Converts to lower-case. | | `UPPER(s)` | Converts to upper-case. | | `REPLACE(s, old, new)` | Replaces all instances of old with new. | | `SUBSTRING(s, pos, len)` | Extracts a substring. | | `TRIM([chars FROM] s)` | Removes leading and trailing spaces or specified chars. | | `REGEXP pattern` | Matches a regular expression pattern. | | `REGEXP_LIKE(col, pattern)` | Matches a regular expression pattern (function form). | ### Mathematical | Function | Description | | ------------------ | --------------------------------------- | | `ABS(expr)` | Absolute value. | | `CEIL(n)` | Smallest integer ≥ n. | | `FLOOR(n)` | Largest integer ≤ n. | | `EXP(n)` | e to the power of n. | | `ROUND(n, places)` | Rounds to the specified decimal places. | | `SQRT(expr)` | Square root. | | `RANDOM(seed)` | Pseudo-random number. | ### Logical | Function | Description | | -------------------------------- | ------------------------------------------------------- | | `IF(cond, true_val, false_val)` | Returns one of two values based on a condition. | | `IIF(cond, true_val, false_val)` | Alias for IF. | | `IFNULL(expr, alt)` | Returns alt if expr is null. | | `NULLIF(expr1, expr2)` | Returns null if expr1 = expr2, otherwise returns expr1. | *** ## Date & Time Functions All SQL date operations use UTC internally. Dates are parsed as [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601), then [RFC 2822](https://tools.ietf.org/html/rfc2822#section-3.3), then `new Date(string)`. | Function | Returns | | ------------------------------------- | ------------------------------------------------------------------------------------------------ | | `CURRENT_DATE()` | Current date as `YYYY-MM-DD`. | | `CURRENT_TIME()` | Current time as `HH:mm:ss.SSS`. | | `CURRENT_TIMESTAMP` | Current Unix timestamp in milliseconds. | | `NOW()` | Current Unix timestamp in milliseconds. | | `GETDATE()` | Current Unix timestamp in milliseconds. | | `GET_SERVER_TIME()` | Current date/time in server's timezone as `YYYY-MM-DDTHH:mm:ss.SSSZZ`. | | `DATE([date_string])` | Date formatted as `YYYY-MM-DDTHH:mm:ss.SSSZZ`. | | `DATE_ADD(date, value, interval)` | Adds time to a date; returns Unix ms. | | `DATE_SUB(date, value, interval)` | Subtracts time from a date; returns Unix ms. | | `DATE_DIFF(date1, date2[, interval])` | Difference between two dates. | | `DATE_FORMAT(date, format)` | Formats a date using [moment.js format strings](https://momentjs.com/docs/#/displaying/format/). | | `EXTRACT(date, date_part)` | Extracts a part (year, month, day, hour, minute, second, millisecond). | | `OFFSET_UTC(date, offset)` | Returns the date adjusted by offset minutes (or hours if < 16). | | `DAY(date)` | Day of the month. | | `DAYOFWEEK(date)` | Day of the week (0=Sunday … 6=Saturday). | | `HOUR(datetime)` | Hour part (0–838). | | `MINUTE(datetime)` | Minute part (0–59). | | `MONTH(date)` | Month (1–12). | | `SECOND(datetime)` | Seconds part (0–59). | | `YEAR(date)` | Year. | `DATE_ADD` and `DATE_SUB` accept these interval values: | Key | Shorthand | | ------------ | --------- | | years | y | | quarters | Q | | months | M | | weeks | w | | days | d | | hours | h | | minutes | m | | seconds | s | | milliseconds | ms | *** ## JSON Search `SEARCH_JSON(expression, attribute)` queries nested JSON data that is not indexed by Harper. It uses the [JSONata](https://docs.jsonata.org/overview.html) library and works in both SELECT and WHERE clauses. ``` -- Find records where the name array contains "Harper" SELECT * FROM dev.dog WHERE SEARCH_JSON('"Harper" in *', name) ``` ``` -- Select and filter nested JSON in one query SELECT m.title, SEARCH_JSON($[name in ["Actor A", "Actor B"]].{"actor": name}, c.`cast`) AS cast FROM movies.credits c INNER JOIN movies.movie m ON c.movie_id = m.id WHERE SEARCH_JSON($count($[name in ["Actor A", "Actor B"]]), c.`cast`) >= 2 ``` *** ## Geospatial Functions Geospatial data must be stored using the [GeoJSON standard](https://geojson.org/) in a single column. All coordinates are in `[longitude, latitude]` format. | Function | Description | | -------------------------------------------- | ------------------------------------------------------------------ | | `geoArea(geoJSON)` | Area of features in square meters. | | `geoLength(geoJSON[, units])` | Length in km (default), or degrees/radians/miles. | | `geoDistance(point1, point2[, units])` | Distance between two points. | | `geoNear(point1, point2, distance[, units])` | Returns boolean: true if points are within the specified distance. | | `geoContains(geo1, geo2)` | Returns boolean: true if geo2 is completely contained by geo1. | | `geoDifference(polygon1, polygon2)` | Returns a new polygon with polygon2 clipped from polygon1. | | `geoEqual(geo1, geo2)` | Returns boolean: true if two GeoJSON features are identical. | | `geoCrosses(geo1, geo2)` | Returns boolean: true if the geometries cross each other. | | `geoConvert(coordinates, geo_type[, props])` | Converts coordinates into a GeoJSON of the specified type. | `units` options: `'degrees'`, `'radians'`, `'miles'`, `'kilometers'` (default). `geo_type` options for `geoConvert`: `'point'`, `'lineString'`, `'multiLineString'`, `'multiPoint'`, `'multiPolygon'`, `'polygon'`. *** ## Logical Operators | Keyword | Description | | --------- | ------------------------------------------------ | | `BETWEEN` | Returns values within a given range (inclusive). | | `IN` | Specifies multiple values in a WHERE clause. | | `LIKE` | Searches for a pattern. | *** ## Reserved Words If a database, table, or attribute name conflicts with a reserved word, wrap it in backticks or brackets: ``` SELECT * FROM data.`ASSERT` SELECT * FROM data.[ASSERT] ``` Full reserved word list ABSOLUTE, ACTION, ADD, AGGR, ALL, ALTER, AND, ANTI, ANY, APPLY, ARRAY, AS, ASSERT, ASC, ATTACH, AUTOINCREMENT, AUTO\_INCREMENT, AVG, BEGIN, BETWEEN, BREAK, BY, CALL, CASE, CAST, CHECK, CLASS, CLOSE, COLLATE, COLUMN, COLUMNS, COMMIT, CONSTRAINT, CONTENT, CONTINUE, CONVERT, CORRESPONDING, COUNT, CREATE, CROSS, CUBE, CURRENT\_TIMESTAMP, CURSOR, DATABASE, DECLARE, DEFAULT, DELETE, DELETED, DESC, DETACH, DISTINCT, DOUBLEPRECISION, DROP, ECHO, EDGE, END, ENUM, ELSE, EXCEPT, EXISTS, EXPLAIN, FALSE, FETCH, FIRST, FOREIGN, FROM, GO, GRAPH, GROUP, GROUPING, HAVING, HDB\_HASH, HELP, IF, IDENTITY, IS, IN, INDEX, INNER, INSERT, INSERTED, INTERSECT, INTO, JOIN, KEY, LAST, LET, LEFT, LIKE, LIMIT, LOOP, MATCHED, MATRIX, MAX, MERGE, MIN, MINUS, MODIFY, NATURAL, NEXT, NEW, NOCASE, NO, NOT, NULL, OFF, ON, ONLY, OFFSET, OPEN, OPTION, OR, ORDER, OUTER, OVER, PATH, PARTITION, PERCENT, PLAN, PRIMARY, PRINT, PRIOR, QUERY, READ, RECORDSET, REDUCE, REFERENCES, RELATIVE, REPLACE, REMOVE, RENAME, REQUIRE, RESTORE, RETURN, RETURNS, RIGHT, ROLLBACK, ROLLUP, ROW, SCHEMA, SCHEMAS, SEARCH, SELECT, SEMI, SET, SETS, SHOW, SOME, SOURCE, STRATEGY, STORE, SYSTEM, SUM, TABLE, TABLES, TARGET, TEMP, TEMPORARY, TEXTSTRING, THEN, TIMEOUT, TO, TOP, TRAN, TRANSACTION, TRIGGER, TRUE, TRUNCATE, UNION, UNIQUE, UPDATE, USE, USING, VALUE, VERTEX, VIEW, WHEN, WHERE, WHILE, WITH, WORK --- # Clustering Operations API for managing Harper's replication system. For an overview of how replication works, see [Replication Overview](/reference/v4/replication/overview.md). For sharding configuration, see [Sharding](/reference/v4/replication/sharding.md). All clustering operations require `super_user` role. *** ### Add Node Adds a new Harper instance to the cluster. If `subscriptions` are provided, it creates the specified replication relationships between the nodes. Without `subscriptions`, a fully replicating system is created (all data in all databases). **Parameters**: * `operation` *(required)* — must be `add_node` * `hostname` or `url` *(required)* — the hostname or URL of the node to add * `verify_tls` *(optional)* — whether to verify the TLS certificate. Set to `false` temporarily on fresh installs with self-signed certificates. Defaults to `true` * `authorization` *(optional)* — credentials for the node being added. Either an object with `username` and `password`, or an HTTP `Authorization` style string * `retain_authorization` *(optional)* — if `true`, stores credentials and uses them on every reconnect. Generally not recommended; prefer certificate-based authentication. Defaults to `false` * `revoked_certificates` *(optional)* — array of revoked certificate serial numbers that will not be accepted for any connections * `shard` *(optional)* — shard number for this node. Only needed when using sharding * `start_time` *(optional)* — ISO 8601 UTC datetime. If set, only data after this time is downloaded during initial synchronization instead of the entire database * `subscriptions` *(optional)* — explicit table-level replication relationships. This is optional (and discouraged). Each subscription is an object with: * `database` — database name * `table` — table name * `subscribe` — if `true`, transactions on the remote table are replicated locally * `publish` — if `true`, transactions on the local table are replicated to the remote node **Request**: ``` { "operation": "add_node", "hostname": "server-two", "verify_tls": false, "authorization": { "username": "admin", "password": "password" } } ``` **Response**: ``` { "message": "Successfully added 'server-two' to cluster" } ``` > **Note**: `set_node` is an alias for `add_node`. *** ### Update Node Modifies an existing Harper instance in the cluster. Will attempt to add the node if it does not exist. **Parameters**: * `operation` *(required)* — must be `update_node` * `hostname` *(required)* — hostname of the remote node to update * `revoked_certificates` *(optional)* — array of revoked certificate serial numbers * `shard` *(optional)* — shard number to assign to this node * `subscriptions` *(required)* — array of subscription objects (same structure as `add_node`) **Request**: ``` { "operation": "update_node", "hostname": "server-two" } ``` **Response**: ``` { "message": "Successfully updated 'server-two'" } ``` *** ### Remove Node Removes a Harper node from the cluster and stops all replication to and from that node. **Parameters**: * `operation` *(required)* — must be `remove_node` * `hostname` *(required)* — hostname of the node to remove **Request**: ``` { "operation": "remove_node", "hostname": "server-two" } ``` **Response**: ``` { "message": "Successfully removed 'server-two' from cluster" } ``` *** ### Cluster Status Returns an array of status objects from the cluster, including active WebSocket connections and replication timing statistics. *Added in: v4.4.0* ; timing statistics added in v4.5.0 **Parameters**: * `operation` *(required)* — must be `cluster_status` **Request**: ``` { "operation": "cluster_status" } ``` **Response**: ``` { "type": "cluster-status", "connections": [ { "replicateByDefault": true, "replicates": true, "url": "wss://server-2.domain.com:9933", "name": "server-2.domain.com", "subscriptions": null, "database_sockets": [ { "database": "data", "connected": true, "latency": 0.7, "thread_id": 1, "nodes": ["server-2.domain.com"], "lastCommitConfirmed": "Wed, 12 Feb 2025 19:09:34 GMT", "lastReceivedRemoteTime": "Wed, 12 Feb 2025 16:49:29 GMT", "lastReceivedLocalTime": "Wed, 12 Feb 2025 16:50:59 GMT", "lastSendTime": "Wed, 12 Feb 2025 16:50:59 GMT" } ] } ], "node_name": "server-1.domain.com", "is_enabled": true } ``` `database_sockets` shows the actual WebSocket connections between nodes — one socket per database per node. Timing fields: | Field | Description | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------- | | `lastCommitConfirmed` | Last time a receipt of confirmation was received for an outgoing commit | | `lastReceivedRemoteTime` | Timestamp (from the originating node) of the last received transaction | | `lastReceivedLocalTime` | Local time when the last transaction was received. A gap between this and `lastReceivedRemoteTime` suggests the node is catching up | | `sendingMessage` | Timestamp of the transaction actively being sent. Absent when waiting for the next transaction | *** ### Configure Cluster Bulk creates or resets subscriptions for any number of remote nodes. **Resets and replaces any existing clustering setup.** **Parameters**: * `operation` *(required)* — must be `configure_cluster` * `connections` *(required)* — array of node objects following the `add_node` schema **Request**: ``` { "operation": "configure_cluster", "connections": [ { "hostname": "server-two", "verify_tls": false, "authorization": { "username": "admin", "password": "password2" } }, { "hostname": "server-three", "verify_tls": false, "authorization": { "username": "admin", "password": "password3" } } ] } ``` **Response**: ``` { "message": "Cluster successfully configured." } ``` *** ### Cluster Set Routes Adds routes to the `replication.routes` configuration. Behaves as a PATCH/upsert — adds new routes while leaving existing routes untouched. **Parameters**: * `operation` *(required)* — must be `cluster_set_routes` * `routes` *(required)* — array of route strings (`wss://host:port`) or objects with `hostname` and `port` properties **Request**: ``` { "operation": "cluster_set_routes", "routes": [ "wss://server-two:9925", { "hostname": "server-three", "port": 9930 } ] } ``` **Response**: ``` { "message": "cluster routes successfully set", "set": ["wss://server-two:9925", { "hostname": "server-three", "port": 9930 }], "skipped": [] } ``` *** ### Cluster Get Routes Returns the replication routes from the Harper config file. **Parameters**: * `operation` *(required)* — must be `cluster_get_routes` **Request**: ``` { "operation": "cluster_get_routes" } ``` **Response**: ``` ["wss://server-two:9925", { "hostname": "server-three", "port": 9930 }] ``` *** ### Cluster Delete Routes Removes routes from the Harper config file. **Parameters**: * `operation` *(required)* — must be `cluster_delete_routes` * `routes` *(required)* — array of route objects to remove **Request**: ``` { "operation": "cluster_delete_routes", "routes": [ { "hostname": "server-three", "port": 9930 } ] } ``` **Response**: ``` { "message": "cluster routes successfully deleted", "deleted": [{ "hostname": "server-three", "port": 9930 }], "skipped": [] } ``` --- # Replication Overview Harper's replication system is designed to make distributed data replication fast and reliable across multiple nodes. You can build a distributed database that ensures high availability, disaster recovery, and data localization — all without complex setup. Nodes can be added or removed dynamically, you can choose which data to replicate, and you can monitor cluster health without jumping through hoops. ## Peer-to-Peer Model Harper replication uses a peer-to-peer model where every node in your cluster can send data to and receive data from other nodes. Nodes communicate over WebSockets, allowing data to flow in both directions. Harper automatically manages these connections and subscriptions, so you don't need to manually track data consistency. Connections between nodes are secured and reliable by default. ## Configuration ### Connecting Nodes To connect nodes to each other, provide hostnames or URLs in the `replication` section of `harperdb-config.yaml`. Each node specifies its own hostname and the routes (other nodes) it should connect to: ``` replication: hostname: server-one routes: - server-two - server-three ``` Routes can also be specified as URLs or with explicit port numbers: ``` replication: hostname: server-one routes: - wss://server-two:9933 - hostname: server-three port: 9933 ``` By default, replication connects on the secure port `9933`. ``` replication: securePort: 9933 ``` You can also manage nodes dynamically through the [Operations API](/reference/v4/replication/clustering.md) without editing the config file. ### Gossip Discovery Harper automatically replicates node information to other nodes in the cluster using [gossip-style discovery](https://highscalability.com/gossip-protocol-explained/). This means you only need to connect to one existing node in a cluster, and Harper will automatically detect and connect to all other nodes bidirectionally. ### Data Selection By default, Harper replicates all data in all databases. You can narrow replication to specific databases: ``` replication: databases: - data - system ``` All tables within a replicated database are replicated by default. To exclude a specific table from replication, set `replicate: false` in the table definition: ``` type LocalTableForNode @table(replicate: false) { id: ID! name: String! } ``` Transactions are replicated atomically, which may span multiple tables. You can also control how many nodes data is replicated to using [sharding configuration](/reference/v4/replication/sharding.md). ## Securing Connections Harper supports PKI-based security and authorization for replication connections. Two authentication methods are supported: * **Certificate-based authentication** (recommended for production): Nodes are identified by the certificate's common name (CN) or Subject Alternative Names (SANs). * **IP-based authentication** (for development/testing): Nodes are identified by IP address when using insecure connections. Harper can automatically perform CRL (Certificate Revocation List) and OCSP (Online Certificate Status Protocol) verification to ensure revoked certificates cannot be used. OCSP and CRL work automatically with certificates from public CAs when `enableRootCAs` is enabled. For self-signed certificates or private CAs without OCSP/CRL support, use Harper's manual certificate revocation feature. Certificate verification settings follow the same configuration as HTTP mTLS connections (see [Certificate Verification](/reference/v4/security/certificate-verification.md)). ### Providing Your Own Certificates If you have certificates from a public or corporate CA, enable `enableRootCAs` so nodes validate against the standard root CA list: ``` replication: enableRootCAs: true ``` Ensure the certificate's CN matches the node's hostname. ### Setting Up Custom Certificates There are two ways to configure Harper with your own certificates: 1. Use the `add_certificate` operation to upload them. 2. Specify certificate paths directly in `harperdb-config.yaml`: ``` tls: certificate: /path/to/certificate.pem certificateAuthority: /path/to/ca.pem privateKey: /path/to/privateKey.pem ``` Harper will load the provided certificates into the certificate table and use them to secure and authenticate connections. If you have a publicly-signed certificate, you can omit the `certificateAuthority` and enable `enableRootCAs` to use the bundled Mozilla CA store instead. ### Cross-Generated Certificates Harper can generate its own certificates for secure connections — useful when no existing certificates are available. When you run `add_node` over SSL with temporary credentials, Harper automatically handles certificate generation and signing: ``` { "operation": "add_node", "hostname": "server-two", "verify_tls": false, "authorization": { "username": "admin", "password": "password" } } ``` On a fresh install, set `verify_tls: false` temporarily to accept the self-signed certificate. Harper then: 1. Creates a certificate signing request (CSR) and sends it to `server-two`. 2. `server-two` signs the CSR and returns the signed certificate and CA. 3. The signed certificate is stored for all future connections. Credentials are not stored — they are discarded immediately after use. You can also provide credentials in HTTP Authorization format (Basic, Token, or JWT). ### Revoking Certificates *Added in: v4.5.0* Certificates used in replication can be revoked using the certificate serial number. Use either the `revoked_certificates` attribute in the `hdb_nodes` system table or route config: Via the operations API: ``` { "operation": "update_node", "hostname": "server-two", "revoked_certificates": ["1769F7D6A"] } ``` Via `harperdb-config.yaml`: ``` replication: routes: - hostname: server-three port: 9930 revokedCertificates: - 1769F7D6A - QA69C7E2S ``` ### Insecure IP-Based Authentication For development, testing, or secure private networks, you can disable TLS and use IP addresses to authenticate nodes. Configure replication on an insecure port and set up IP-based routes: ``` replication: port: 9933 routes: - 127.0.0.2 - 127.0.0.3 ``` > **Warning**: Never use insecure connections for production systems accessible from the public internet. Loopback addresses (`127.0.0.X`) are a convenient way to run multiple nodes on a single machine for local development. ## Controlling Replication Flow By default, Harper replicates all data in all databases with symmetric bidirectional flow. To restrict replication to one direction between certain nodes, set `sends` and `receives` on the route configuration: ``` replication: databases: - data routes: - host: node-two replicates: sends: false receives: true - host: node-three replicates: sends: true receives: false ``` In this example, the local node only receives from `node-two` (one-way inbound) and only sends to `node-three` (one-way outbound). > **Note**: When using controlled flow replication, avoid replicating the `system` database. The `system` database contains node configurations, so replicating it would cause all nodes to have identical (and incorrect) route configurations. ### Explicit Subscriptions By default, Harper automatically manages connections and subscriptions between nodes. Explicit subscriptions exist only for testing, debugging, and legacy migration — they should not be used for production replication and will likely be removed in v5. With explicit subscriptions, Harper no longer guarantees data consistency. If you want unidirectional replication, use [controlled replication flow](#controlling-replication-flow) instead. To explicitly subscribe, use `add_node` with subscription definitions: ``` { "operation": "add_node", "hostname": "server-two", "subscriptions": [ { "database": "dev", "table": "my-table", "publish": true, "subscribe": false } ] } ``` Update a subscription with `update_node`: ``` { "operation": "update_node", "hostname": "server-two", "subscriptions": [ { "database": "dev", "table": "my-table", "publish": true, "subscribe": true } ] } ``` ## Monitoring Replication *Added in: v4.5.0* (cluster status timing statistics) Use `cluster_status` to monitor the state of replication: ``` { "operation": "cluster_status" } ``` See [Clustering Operations](/reference/v4/replication/clustering.md#cluster-status) for the full response schema and field descriptions. ## Initial Synchronization and Resynchronization When a new node is added and its database has not been previously synced, Harper downloads the full database from the first node it connects to. After the initial sync completes, the node enters replication mode and receives incremental updates. If a node goes offline and comes back, it resynchronizes automatically to catch up on missed transactions. You can also specify a `start_time` in the `add_node` operation to limit the initial download to data since a given point in time: ``` { "operation": "add_node", "hostname": "server-two", "start_time": "2024-01-01T00:00:00.000Z" } ``` ## Replicated Transactions The following data operations are replicated across the cluster: * Insert * Update * Upsert * Delete * Bulk loads (CSV data load, CSV file load, CSV URL load, import from S3) **Destructive schema operations are not replicated**: `drop_database`, `drop_table`, and `drop_attribute` must be run on each node independently. Users and roles are not replicated across the cluster. Certain management operations — including component deployment and rolling restarts — can also be replicated across the cluster. ## Inspecting Cluster Configuration Query the `hdb_nodes` system table to inspect the current known nodes and their configuration: ``` { "operation": "search_by_value", "database": "system", "table": "hdb_nodes", "attribute": "name", "value": "*" } ``` The `hdb_certificate` table contains the certificates used for replication connections. ## See Also * [Clustering Operations](/reference/v4/replication/clustering.md) — Operations API for managing cluster nodes and subscriptions * [Sharding](/reference/v4/replication/sharding.md) — Distributing data across a subset of nodes * [Certificate Management](/reference/v4/security/certificate-management.md) --- # Sharding *Added in: v4.4.0* (provisional) *Changed in: v4.5.0* — expanded sharding functionality: Harper now honors write requests with residency information that will not be stored on the local node, and nodes can be declaratively configured as part of a shard. Harper's replication system supports sharding — storing different data across different subsets of nodes — while still allowing data to be accessed from any node in the cluster. This enables horizontal scalability for storage and write performance, while maintaining optimal data locality and consistency. When sharding is configured, requests for records that don't reside on the handling node are automatically forwarded to the appropriate node transparently. Clients do not need to know where data is stored. By default (without sharding), Harper replicates all data to all nodes. ## Approaches to Sharding There are two main approaches: **Dynamic sharding** — the location (residency) of records is determined dynamically based on where the record was written, the record's data, or a custom function. Records can be relocated dynamically based on where they are accessed. Residency information is specific to each record. **Static sharding** — each node is assigned to a specific numbered shard, and each record is replicated to the nodes in that shard based on the primary key, regardless of where the data was written or accessed. More predictable than dynamic sharding: data location is always determinable from the primary key. ## Dynamic Sharding ### Replication Count The simplest way to limit replication is to configure a replication count. Set `replicateTo` in the `replication` section of `harperdb-config.yaml` to specify how many additional nodes data should be replicated to: ``` replication: replicateTo: 2 ``` This ensures each record is stored on three nodes total (the node that first stored it, plus two others). ### Replication Control via REST Header With the REST interface, you can specify replication targets and confirmation requirements per request using the `X-Replicate-To` header: ``` PUT /MyTable/3 X-Replicate-To: 2;confirm=1 ``` * `2` — replicate to two additional nodes * `confirm=1` — wait for confirmation from one additional node before responding Specify exact destination nodes by hostname: ``` PUT /MyTable/3 X-Replicate-To: node1,node2 ``` The `confirm` parameter can be combined with explicit node lists. ### Replication Control via Operations API Specify `replicateTo` and `replicatedConfirmation` in the operation body: ``` { "operation": "update", "schema": "dev", "table": "MyTable", "hashValues": [3], "record": { "name": "John Doe" }, "replicateTo": 2, "replicatedConfirmation": 1 } ``` Or specify explicit nodes: ``` { // ... "replicateTo": ["node-1", "node-2"], // ... } ``` ### Programmatic Replication Control Set `replicateTo` and `replicatedConfirmation` programmatically in a resource method: ``` class MyTable extends tables.MyTable { put(record) { const context = this.getContext(); context.replicateTo = 2; // or an array of node names context.replicatedConfirmation = 1; return super.put(record); } } ``` ## Static Sharding ### Basic Static Shard Configuration Assign a node to a numbered shard in `harperdb-config.yaml`: ``` replication: shard: 1 ``` Or assign shards per route: ``` replication: routes: - hostname: node1 shard: 1 - hostname: node2 shard: 2 ``` Or dynamically via the operations API by including `shard` in an `add_node` or `set_node` operation: ``` { "operation": "add_node", "hostname": "node1", "shard": 1 } ``` Once shards are configured, use `setResidency` or `setResidencyById` (described below) to assign records to specific shards. ## Custom Sharding ### By Record Content (`setResidency`) Define a custom residency function that is called with the full record. Return an array of node hostnames or a shard number. With this approach, record metadata (including residency information) and indexed properties are replicated to all nodes, but the full record is only stored on the specified nodes. Return node hostnames: ``` MyTable.setResidency((record) => { return record.id % 2 === 0 ? ['node1'] : ['node2']; }); ``` Return a shard number (replicates to all nodes in that shard): ``` MyTable.setResidency((record) => { return record.id % 2 === 0 ? 1 : 2; }); ``` ### By Primary Key Only (`setResidencyById`) Define a residency function based solely on the primary key. Records (including metadata) are only replicated to the specified nodes — metadata does not need to be replicated everywhere, which allows data to be retrieved without needing access to record data or metadata on the requesting node. Return a shard number: ``` MyTable.setResidencyById((id) => { return id % 2 === 0 ? 1 : 2; }); ``` Return node hostnames: ``` MyTable.setResidencyById((id) => { return id % 2 === 0 ? ['node1'] : ['node2']; }); ``` ## Disabling Cross-Node Access By default, sharding allows data stored on specific nodes to be accessed from any node — requests are forwarded transparently. To disable this and only return data if it is stored on the local node, set `replicateFrom` to `false`. Via the operations API: ``` { "operation": "search_by_id", "table": "MyTable", "ids": [3], "replicateFrom": false } ``` Via the REST API: ``` GET /MyTable/3 X-Replicate-From: none ``` ## See Also * [Replication Overview](/reference/v4/replication/overview.md) — How Harper's replication system works * [Clustering Operations](/reference/v4/replication/clustering.md) — Operations API for managing cluster nodes --- # Resources Harper's Resource API is the foundation for building custom data access logic and connecting data sources. Resources are JavaScript classes that define how data is accessed, modified, subscribed to, and served over HTTP, MQTT, and WebSocket protocols. ## What Is a Resource? A **Resource** is a class that provides a unified interface for a set of records or entities. Harper's built-in tables extend the base `Resource` class, and you can extend either `Resource` or a table class to implement custom behavior for any data source — internal or external. *Added in: v4.2.0* The Resource API is designed to mirror REST/HTTP semantics: methods map directly to HTTP verbs (`get`, `put`, `patch`, `post`, `delete`), making it straightforward to build API endpoints alongside custom data logic. ## Relationship to Other Features * **Database tables** extend `Resource` automatically. You can use tables through the Resource API without writing any custom code. * The **REST plugin** maps incoming HTTP requests to Resource methods. See [REST Overview](/reference/v4/rest/overview.md). * The **MQTT plugin** routes publish/subscribe messages to `publish` and `subscribe` Resource methods. See [MQTT Overview](/reference/v4/mqtt/overview.md). * **Global APIs** (`tables`, `databases`, `transaction`) provide access to resources from JavaScript code. * The **`jsResource` plugin** (configured in `config.yaml`) registers a JavaScript file's exported Resource classes as endpoints. ## Resource API Versions The Resource API has two behavioral modes controlled by the `loadAsInstance` static property: * **V2 (recommended, `loadAsInstance = false`)**: Instance methods receive a `RequestTarget` as the first argument; no record is preloaded onto `this`. Recommended for all new code. * **V1 (legacy, `loadAsInstance = true`)**: Instance methods are called with `this` pre-bound to the matching record. Preserved for backwards compatibility. The [Resource API reference](/reference/v4/resources/resource-api.md) is written against V2. For V1 behavior and migration guidance, see the legacy instance binding section of that page. ## Extending a Table The most common use case is extending an existing table to add custom logic. Starting with a table definition in a `schema.graphql`: ``` # Omit the `@export` directive type MyTable @table { id: Long @primaryKey # ... } ``` > For more info on the schema API see [`Database / Schema`](/reference/v4/database/schema.md) Then, in a `resources.js` extend from the `tables.MyTable` global: ``` export class MyTable extends tables.MyTable { static loadAsInstance = false; // use V2 API async get(target) { // add a computed property before returning const record = await super.get(target); return { ...record, computedField: 'value' }; } post(target, data) { // custom action on POST this.create({ ...data, status: 'pending' }); } } ``` Finally, ensure everything is configured appropriately: ``` rest: true graphqlSchema: files: schema.graphql jsResource: files: resources.js ``` ## Custom External Data Source You can also extend the base `Resource` class directly to implement custom endpoints, or even wrap an external API or service as a custom caching layer: ``` export class CustomEndpoint extends Resource { static loadAsInstance = false; get(target) { return { data: doSomething(), }; } } export class MyExternalData extends Resource { static loadAsInstance = false; async get(target) { const response = await fetch(`https://api.example.com/${target.id}`); return response.json(); } put(target, data) { return fetch(`https://api.example.com/${target.id}`, { method: 'PUT', body: JSON.stringify(data), }); } } // Use as a cache source for a local table tables.MyCache.sourcedFrom(MyExternalData); ``` Resources are the true customization point for Harper. This is where the business logic of a Harper application really lives. There is a lot more to this API than these examples show. Ensure you fully review the [Resource API](/reference/v4/resources/resource-api.md) documentation, and consider exploring the Learn guides for more information. ## Exporting Resources as Endpoints Resources become HTTP/MQTT endpoints when they are exported. As the examples demonstrated if a Resource extends an existing table, make sure to not have conflicting exports between the schema and the JavaScript implementation. Alternatively, you can register resources programmatically using `server.resources.set()`. See [HTTP API](/reference/v4/http/api.md) for server extension documentation. ## Pages in This Section | Page | Description | | ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | [Resource API](/reference/v4/resources/resource-api.md) | Complete reference for instance methods, static methods, the Query object, RequestTarget, and response handling | | [Query Optimization](/reference/v4/resources/query-optimization.md) | How Harper executes queries and how to write performant conditions | --- # Query Optimization *Added in: v4.3.0* (query planning and execution improvements) Harper has powerful query functionality with excellent performance characteristics. Like any database, different queries can vary significantly in performance. Understanding how querying works helps you write queries that perform well as your dataset grows. ## Query Execution At a fundamental level, querying involves defining conditions to find matching data and then executing those conditions against the database. Harper supports indexed fields, and these indexes are used to speed up query execution. When conditions are specified in a query, Harper attempts to utilize indexes to optimize the speed of query execution. When a field is not indexed, Harper checks each potential record to determine if it matches the condition — this is a full table scan and degrades as data grows (`O(n)`). When a query has multiple conditions, Harper attempts to optimize their execution order. For intersecting conditions (the default `and` operator), Harper applies the most selective and performant condition first. If one condition can use an index and is more selective than another, it is used first to narrow the candidate set before filtering on the remaining conditions. The `search` method supports an `explain` flag that returns the query execution order Harper determined, useful for debugging and optimization: ``` const result = await MyTable.search({ conditions: [...], explain: true, }); ``` For union queries (`or` operator), each condition is executed separately and the results are merged. ## Conditions, Operators, and Indexing When a query is executed, conditions are evaluated against the database. Indexed fields significantly improve query performance. ### Index Performance Characteristics | Operator | Uses index | Notes | | -------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------ | | `equals` | Yes | Fast lookup in sorted index | | `greater_than`, `greater_than_equal`, `less_than`, `less_than_equal` | Yes | Range scan in sorted index; narrower range = faster | | `starts_with` | Yes | Prefix search in sorted index | | `not_equal` | No | Full scan required (unless combined with selective indexed condition) | | `contains` | No | Full scan required | | `ends_with` | No | Full scan required | | `!= null` | Yes (special case) | Can use indexes to find non-null records; only helpful for sparse fields | **Rule of thumb**: Use `equals`, range operators, and `starts_with` on indexed fields. Avoid `contains`, `ends_with`, and `not_equal` as the sole or first condition in large datasets. ### Indexed vs. Non-Indexed Fields Indexed fields provide `O(log n)` lookup — fast even as the dataset grows. Non-indexed fields require `O(n)` full table scans. Trade-off: indexes speed up reads but add overhead to writes (insert/update/delete must update the index). This is usually worth it for frequently queried fields. ### Primary Key vs. Secondary Index Querying on a **primary key** is faster than querying on a secondary (non-primary) index, because the primary key directly addresses the record without cross-referencing. Secondary indexes are still valuable for query conditions on other fields, but expect slightly more overhead than primary key lookups. ### Cardinality More unique values (higher cardinality) = more efficient indexed lookups. For example, an index on a boolean field has very low cardinality (only two possible values) and is less efficient than an index on a `UUID` field. High-cardinality fields benefit most from indexing. ## Relationships and Joins Harper supports relationship-based queries that join data across tables. See [Schema documentation](/reference/v4/database/schema.md) for how to define relationships. Join queries involve more lookups and naturally carry more overhead. The same indexing principles apply: * Conditions on joined table fields should use indexed columns for best performance. * If a relationship uses a foreign key, that foreign key should be indexed in both tables. * Higher cardinality foreign keys make joins more efficient. Example of an indexed foreign key that enables efficient join queries: ``` type Product @table { id: Long @primaryKey brandId: Long @indexed # foreign key — index this brand: Related @relationship(from: "brandId") } type Brand @table { id: Long @primaryKey name: String @indexed # indexed — enables efficient brand.name queries products: Product @relationship(to: "brandId") } ``` *Added in: v4.3.0* ## Sorting Sorting can significantly impact query performance. * **Aligned sort and index**: If the sort attribute is the same indexed field used in the primary condition, Harper can use the index to retrieve results already in order — very fast. * **Unaligned sort**: If the sort is on a different field than the condition, or the sort field is not indexed, Harper must retrieve and sort all matching records. For large result sets this can be slow, and it also **defeats streaming** (see below). Best practice: sort on the same indexed field you are filtering on, or sort on a secondary indexed field with a narrow enough condition to produce a manageable result set. ## Streaming Harper can stream query results — returning records as they are found rather than waiting for the entire query to complete. This improves time-to-first-byte for large queries and reduces peak memory usage. **Streaming is defeated** when: * A sort order is specified that is not aligned with the condition's index * The full result set must be materialized to perform sorting When streaming is possible, results are returned as an `AsyncIterable`: ``` for await (const record of MyTable.search({ conditions: [...] })) { // process each record as it arrives } ``` Failing to iterate the `AsyncIterable` to completion keeps a read transaction open, degrading performance. Always ensure you either fully iterate or explicitly release the query. ### Draining or Releasing a Query An open query holds an active read transaction. While that transaction is open, the underlying data pages and internal state for the query cannot be freed — they remain pinned in memory until the transaction closes. In long-running processes or under high concurrency, accumulating unreleased transactions degrades throughput and increases memory pressure. The transaction closes automatically once the `AsyncIterable` is fully iterated. If you need to stop early, you must explicitly signal that iteration is complete so Harper can release the transaction. **Breaking out of a `for await...of` loop** is the most natural way. The JavaScript runtime automatically calls `.return()` on the iterator when a `break`, `return`, or `throw` exits the loop: ``` for await (const record of MyTable.search({ conditions: [...] })) { if (meetsStopCriteria(record)) { break; // iterator.return() is called automatically — transaction is released } process(record); } ``` **Calling `.return()` manually** is useful when you hold an iterator reference directly: ``` const iterator = MyTable.search({ conditions: [...] })[Symbol.asyncIterator](); try { const { value } = await iterator.next(); process(value); } finally { await iterator.return(); // explicitly closes the iterator and releases the transaction } ``` Avoid storing an iterator and abandoning it (e.g. never calling `.next()` again without calling `.return()`), as the transaction will remain open until the iterator is garbage collected — which is non-deterministic. ## Practical Guidance ### Index fields you query on frequently ``` type Product @table { id: Long @primaryKey name: String @indexed # queried frequently category: String @indexed # queried frequently description: String # not indexed (rarely in conditions) } ``` ### Use `explain` to diagnose slow queries ``` const result = await Product.search({ conditions: [ { attribute: 'category', value: 'electronics' }, { attribute: 'price', comparator: 'less_than', value: 100 }, ], explain: true, }); // result shows the actual execution order Harper selected ``` ### Prefer selective conditions first When Harper cannot auto-reorder (e.g. with `enforceExecutionOrder`), put the most selective condition first: ``` // Better: indexed, selective condition first Product.search({ conditions: [ { attribute: 'sku', value: 'ABC-001' }, // exact match on indexed unique field { attribute: 'active', value: true }, // low cardinality filter ], }); ``` ### Use `limit` and `offset` for pagination ``` Product.search({ conditions: [...], sort: { attribute: 'createdAt', descending: true }, limit: 20, offset: page * 20, }); ``` ### Avoid wide range queries on non-indexed fields ``` // Slow: non-indexed field with range condition Product.search({ conditions: [{ attribute: 'description', comparator: 'contains', value: 'sale' }], }); // Better: use an indexed field condition to narrow first Product.search({ conditions: [ { attribute: 'category', value: 'clothing' }, // indexed — narrows to subset { attribute: 'description', comparator: 'contains', value: 'sale' }, // non-indexed, applied to smaller set ], }); ``` --- # Resource API *Added in: v4.2.0* The Resource API provides a unified JavaScript interface for accessing, querying, modifying, and subscribing to data resources in Harper. Tables extend the base `Resource` class, and all resource interactions — whether from HTTP requests, MQTT messages, or application code — flow through this interface. ## API Versions The Resource API has two behavioral modes selected by the `loadAsInstance` static property: | Version | `loadAsInstance` | Status | | ------------ | ---------------- | ------------------------------------- | | V2 (current) | `false` | Recommended for new code | | V1 (legacy) | `true` (default) | Preserved for backwards compatibility | The default value of `loadAsInstance` is `true` (V1 behavior). To opt in to V2, you must explicitly set `static loadAsInstance = false` on your resource class. This page documents V2 behavior (`loadAsInstance = false`). For V1 (legacy instance binding) behavior and migration examples, see [Legacy Instance Binding](#legacy-instance-binding-v1). ### V2 Behavioral Differences from V1 *Changed in: v4.6.0* (Resource API upgrades that formalized V2) When `loadAsInstance = false`: * Instance methods receive a `RequestTarget` as their first argument; no record is preloaded onto `this`. * The `get` method returns the record as a plain (frozen) object rather than a Resource instance. * `put`, `post`, and `patch` receive `(target, data)` — **arguments are reversed from V1**. * Authorization is handled via `target.checkPermission` rather than `allowRead`/`allowUpdate`/etc. methods. Set it to `false` to bypass permission checks entirely (e.g. for a public read endpoint), or leave it at its default to require superuser access for write operations: ``` // Public read — no auth required get(target) { target.checkPermission = false; return super.get(target); } // POST is superuser-only by default — no change needed post(target, data) { return super.post(target, data); } ``` `checkPermission` can also be set to a non-boolean value to delegate to role-based or schema-defined permissions — see the authorization documentation for details. * The `update` method returns an `Updatable` object instead of a Resource instance. * Context is tracked automatically via async context tracking; set `static explicitContext = true` to disable (improves performance). * `getId()` is not used and returns `undefined`. *** ## Resource Instance Methods These methods are defined on a Resource class and called when requests are routed to the resource. Override them to define custom behavior. ### `get(target: RequestTarget): Promise | AsyncIterable` Called for HTTP GET requests. When the request targets a single record (e.g. `/Table/some-id`), returns a single record object. When the request targets a collection (e.g. `/Table/?name=value`), the `target.isCollection` property is `true` and the default behavior calls `search()`, returning an `AsyncIterable`. ``` class MyResource extends Resource { static loadAsInstance = false; get(target) { const id = target.id; // primary key from URL path const param = target.get('param1'); // query string param const path = target.pathname; // path relative to resource return super.get(target); // default: return the record } } ``` The default `super.get(target)` returns a `RecordObject` — a frozen plain object with the record's properties plus `getUpdatedTime()` and `getExpiresAt()`. Common gotchas * **`/Table` vs `/Table/`** — `GET /Table` returns metadata about the table resource itself. `GET /Table/` (trailing slash) targets the collection and invokes `get()` as a collection request. These are distinct endpoints. * **Case sensitivity** — The URL path must match the exact casing of the exported resource or table name. `/Table/` works; `/table/` returns a 404. ### `search(query: RequestTarget): AsyncIterable` Performs a query on the resource or table. Called by `get()` on collection requests. Can be overridden to define custom query behavior. The default implementation on tables queries by the `conditions`, `limit`, `offset`, `select`, and `sort` properties parsed from the URL. ### `put(target: RequestTarget | Id, data: object): void | Response` Called for HTTP PUT requests. Writes the full record to the table, creating or replacing the existing record. ``` put(target, data) { // validate or transform before saving super.put(target, { ...data, status: data.status ?? 'active' }); } ``` ### `patch(target: RequestTarget | Id, data: object): void | Response` Called for HTTP PATCH requests. Merges `data` into the existing record, preserving any properties not included in `data`. *Added in: v4.3.0* (CRDT support for individual property updates via PATCH) ### `post(target: RequestTarget | Id, data: object): void | Response` Called for HTTP POST requests. Default behavior creates a new record. Override to implement custom actions. ### `delete(target: RequestTarget | Id): void | Response` Called for HTTP DELETE requests. Default behavior deletes the record identified by `target`. ### `update(target: RequestTarget, updates?: object): Updatable` Returns an `Updatable` instance providing mutable property access to a record. Any property changes on the `Updatable` are written to the database when the transaction commits. ``` post(target, data) { const record = this.update(target.id); record.quantity = record.quantity - 1; // saved automatically on transaction commit } ``` #### `Updatable` class The `Updatable` class provides direct property access plus: ##### `addTo(property: string, value: number)` Adds `value` to `property` using CRDT incrementation — safe for concurrent updates across threads and nodes. *Added in: v4.3.0* ``` post(target, data) { const record = this.update(target.id); record.addTo('quantity', -1); // decrement safely across nodes } ``` ##### `subtractFrom(property: string, value: number)` Subtracts `value` from `property` using CRDT incrementation. ##### `set(property: string, value: any): void` Sets a property to `value`. Equivalent to direct property assignment (`record.property = value`), but useful when the property name is dynamic. ``` const record = this.update(target.id); record.set('status', 'active'); ``` ##### `getProperty(property: string): any` Returns the current value of `property` from the record. Useful when the property name is dynamic or when you want an explicit read rather than direct property access. ``` const record = this.update(target.id); const current = record.getProperty('status'); ``` ##### `getUpdatedTime(): number` Returns the last updated time as milliseconds since epoch. ##### `getExpiresAt(): number` Returns the expiration time, if one is set. ### `publish(target: RequestTarget, message: object): void | Response` Called for MQTT publish commands. Default behavior records the message and notifies subscribers without changing the record's stored data. ### `subscribe(subscriptionRequest?: SubscriptionRequest): Promise` Called for MQTT subscribe commands. Returns a `Subscription` — an `AsyncIterable` of messages/changes. #### `SubscriptionRequest` options All properties are optional: | Property | Description | | -------------------- | ---------------------------------------------------------------------------------------------- | | `includeDescendants` | Include all updates with an id prefixed by the subscribed id (e.g. `sub/*`) | | `startTime` | Start from a past time (catch-up of historical messages). Cannot be used with `previousCount`. | | `previousCount` | Return the last N updates/messages. Cannot be used with `startTime`. | | `omitCurrent` | Do not send the current/retained record as the first update. | ### `connect(target: RequestTarget, incomingMessages?: AsyncIterable): AsyncIterable` Called for WebSocket and Server-Sent Events connections. `incomingMessages` is provided for WebSocket connections (not SSE). Returns an `AsyncIterable` of messages to send to the client. ### `invalidate(target: RequestTarget)` Marks the specified record as invalid in a caching table, so it will be reloaded from the source on next access. ### `allowStaleWhileRevalidate(entry, id): boolean` For caching tables: return `true` to serve the stale entry while revalidation happens concurrently; `false` to wait for the fresh value. Entry properties: * `version` — Timestamp/version from the source * `localTime` — When the resource was last refreshed locally * `expiresAt` — When the entry became stale * `value` — The stale record value ### `getUpdatedTime(): number` Returns the last updated time of the resource (milliseconds since epoch). ### `wasLoadedFromSource(): boolean` For caching tables, indicates that this request was a cache miss and the data was loaded from the source resource. ### `getContext(): Context` Returns the current context, which includes: * `user` — User object with username, role, and authorization information * `transaction` — The current transaction When triggered by HTTP, the context is the `Request` object with these additional properties: * `url` — Full local path including query string * `method` — HTTP method * `headers` — Request headers (access with `context.headers.get(name)`) * `responseHeaders` — Response headers (set with `context.responseHeaders.set(name, value)`) * `pathname` — Path without query string * `host` — Host from the `Host` header * `ip` — Client IP address * `body` — Raw Node.js `Readable` stream (if a request body exists) * `data` — Promise resolving to the deserialized request body * `lastModified` — Controls the `ETag`/`Last-Modified` response header * `requestContext` — (For source resources only) Context of the upstream resource making the data request ### `operation(operationObject: object, authorize?: boolean): Promise` Executes a Harper operations API call using this table as the target. Set `authorize` to `true` to enforce current-user authorization. *** ## Resource Static Methods Static methods are the preferred way to interact with tables and resources from application code. They handle transaction setup, access checks, and request parsing automatically. All instance methods have static equivalents that accept an `id` or `RequestTarget` as the first argument: ### `get(target: RequestTarget | Id | Query, context?: Resource | Context)` Retrieve a record by primary key, or query for records. ``` // By primary key const product = await Product.get(34); // By query object const product = await Product.get({ id: 34, select: ['name', 'price'] }); // Iterate a collection query for await (const record of Product.get({ conditions: [{ attribute: 'inStock', value: true }] })) { // ... } ``` ### `put(target: RequestTarget | Id, record: object, context?): Promise` ### `put(record: object, context?): Promise` Save a record (create or replace). The second form reads the primary key from the `record` object. ### `create(record: object, context?): Promise` Create a new record with an auto-generated primary key. Returns the created record. Do not include a primary key in the `record` argument. *Added in: v4.2.0* ### `patch(target: RequestTarget | Id, updates: object, context?): Promise` Apply partial updates to an existing record. ### `post(target: RequestTarget | Id, data: object, context?): Promise` Call the `post` instance method. Defaults to creating a new record. ### `delete(target: RequestTarget | Id, context?): Promise` Delete a record. ### `publish(target: RequestTarget | Id, message: object, context?): Promise` Publish a message to a record/topic. ### `subscribe(subscriptionRequest?, context?): Promise` Subscribe to record changes or messages. ### `search(query: RequestTarget | Query, context?): AsyncIterable` Query the table. See [Query Object](#query-object) below for available query options. ### `setComputedAttribute(name: string, computeFunction: (record) => any)` Define the compute function for a `@computed` schema attribute. *Added in: v4.4.0* ``` MyTable.setComputedAttribute('fullName', (record) => `${record.firstName} ${record.lastName}`); ``` ### `getRecordCount({ exactCount?: boolean }): Promise<{ recordCount: number, estimatedRange?: [number, number] }>` Returns the number of records in the table. By default returns an approximate (fast) count. Pass `{ exactCount: true }` for a precise count. *Added in: v4.5.0* ### `sourcedFrom(Resource, options?)` Configure a table to use another resource as its data source (caching behavior). When a record is not found locally, it is fetched from the source and cached. Writes are delegated to the source. Options: * `expiration` — Default TTL in seconds * `eviction` — Eviction time in seconds * `scanInterval` — Period for scanning expired records ### `parsePath(path, context, query)` Called by static methods when processing a URL path. Can be overridden to preserve the path directly as the primary key: ``` static parsePath(path) { return path; // use full path as id, no parsing } ``` ### `directURLMapping` Set this static property to `true` to map the full URL (including query string) as the primary key, bypassing query parsing. *Added in: v4.5.0* (documented in improved URL path parsing) ``` export class MyTable extends tables.MyTable { static directURLMapping = true; } // GET /MyTable/test?foo=bar → primary key is 'test?foo=bar' ``` ### `primaryKey` The name of the primary key attribute for the table. ``` const record = await Table.get(34); record[Table.primaryKey]; // → 34 ``` ### `isCollection(resource): boolean` Returns `true` if the resource instance represents a collection (query result) rather than a single record. *** ## Query Object The `Query` object is accepted by `search()` and the static `get()` method. ### `conditions` Array of condition objects to filter records. Each condition: | Property | Description | | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `attribute` | Property name, or an array for chained/joined properties (e.g. `['brand', 'name']`) | | `value` | The value to match | | `comparator` | `equals` (default), `greater_than`, `greater_than_equal`, `less_than`, `less_than_equal`, `starts_with`, `contains`, `ends_with`, `between`, `not_equal` | | `conditions` | Nested conditions array | | `operator` | `and` (default) or `or` for the nested `conditions` | Example with nested conditions: ``` Product.search({ conditions: [ { attribute: 'price', comparator: 'less_than', value: 100 }, { operator: 'or', conditions: [ { attribute: 'rating', comparator: 'greater_than', value: 4 }, { attribute: 'featured', value: true }, ], }, ], }); ``` **Chained attribute references** (for relationships/joins): Use an array to traverse relationship properties: ``` Product.search({ conditions: [{ attribute: ['brand', 'name'], value: 'Harper' }] }); ``` *Added in: v4.3.0* ### `operator` Top-level `and` (default) or `or` for the `conditions` array. ### `limit` Maximum number of records to return. ### `offset` Number of records to skip (for pagination). ### `select` Properties to include in each returned record. Can be: * Array of property names: `['name', 'price']` * Nested select for related records: `[{ name: 'brand', select: ['id', 'name'] }]` * String to return a single property per record: `'id'` Special properties: * `$id` — Returns the primary key regardless of its name * `$updatedtime` — Returns the last-updated timestamp ### `sort` Sort order object: | Property | Description | | ------------ | ---------------------------------------------------------- | | `attribute` | Property name (or array for chained relationship property) | | `descending` | Sort descending if `true` (default: `false`) | | `next` | Secondary sort to resolve ties (same structure) | ### `explain` If `true`, returns conditions reordered as Harper will execute them (for debugging and optimization). ### `enforceExecutionOrder` If `true`, forces conditions to execute in the order supplied, disabling Harper's automatic re-ordering optimization. *** ## RequestTarget `RequestTarget` represents a URL path mapped to a resource. It is a subclass of `URLSearchParams`. Properties: * `pathname` — Path relative to the resource, without query string * `search` — The query/search string portion of the URL * `id` — Primary key derived from the path * `isCollection` — `true` when the request targets a collection * `checkPermission` — Set to indicate authorization should be performed; has `action`, `resource`, and `user` sub-properties Standard `URLSearchParams` methods are available: * `get(name)`, `getAll(name)`, `set(name, value)`, `append(name, value)`, `delete(name)`, `has(name)` * Iterable: `for (const [name, value] of target) { ... }` When a URL uses Harper's extended query syntax, these are parsed onto the target: * `conditions`, `limit`, `offset`, `sort`, `select` *** ## RecordObject The `get()` method returns a `RecordObject` — a frozen plain object with all record properties, plus: * `getUpdatedTime(): number` — Last updated time (milliseconds since epoch) * `getExpiresAt(): number` — Expiration time, if set *** ## Response Object Resource methods can return: 1. **Plain data** — serialized using content negotiation 2. **`Response`-like object** with `status`, `headers`, and `data` or `body`: ``` // Redirect return { status: 302, headers: { Location: '/new-location' } }; // Custom header with data return { status: 200, headers: { 'X-Custom-Header': 'value' }, data: { message: 'ok' } }; ``` `body` must be a string, `Buffer`, Node.js stream, or `ReadableStream`. `data` is an object that will be serialized. *Added in: v4.4.0* ### Throwing Errors Uncaught errors are caught by the protocol handler. For REST, they produce error responses. Set `error.statusCode` to control the HTTP status: ``` if (!authorized) { const error = new Error('Forbidden'); error.statusCode = 403; throw error; } ``` *** ## Context and Transactions Whenever you call other resources from within a resource method, pass `this` as the context argument to share the transaction and ensure atomicity: ``` export class BlogPost extends tables.BlogPost { static loadAsInstance = false; post(target, data) { // both writes share the same transaction tables.Comment.put(data, this); const post = this.update(target.id); post.commentCount = (post.commentCount ?? 0) + 1; } } ``` See [JavaScript Environment — transaction](/reference/v4/components/javascript-environment.md#transactionfn) for explicitly starting transactions outside of request handlers. *** ## Legacy Instance Binding (V1) This documents the legacy `loadAsInstance = true` (or default pre-V2) behavior. The V2 API is recommended for all new code. When `loadAsInstance` is not `false` (or is explicitly `true`): * `this` is pre-bound to the matching record when instance methods are called. * `this.getId()` returns the current record's primary key. * Instance properties map directly to the record's fields. * `get(query)` and `put(data, query)` have arguments in the older order (no `target` first). * `allowRead()`, `allowUpdate()`, `allowCreate()`, `allowDelete()` methods are used for authorization. ``` export class MyExternalData extends Resource { static loadAsInstance = true; async get() { const response = await this.fetch(this.id); return response; } put(data) { // write to external source } delete() { // delete from external source } } tables.MyCache.sourcedFrom(MyExternalData); ``` ### Migration from V1 to V2 Updated `get`: ``` // V1 async get(query) { let id = this.getId(); this.newProperty = 'value'; return super.get(query); } // V2 static loadAsInstance = false; async get(target) { let id = target.id; let record = await super.get(target); return { ...record, newProperty: 'value' }; // record is frozen; spread to add properties } ``` Updated authorization: ``` // V1 allowRead(user) { return !!user; } // V2 static loadAsInstance = false; async get(target) { if (!this.getContext().user) { const error = new Error('Unauthorized'); error.statusCode = 401; throw error; } target.checkPermission = false; return super.get(target); } ``` Updated `post` (note reversed argument order): ``` // V1 async post(data, query) { ... } // V2 static loadAsInstance = false; async post(target, data) { ... } // target is first ``` --- # Content Types Harper supports multiple content types (MIME types) for both HTTP request bodies and response bodies. Harper follows HTTP standards: use the `Content-Type` request header to specify the encoding of the request body, and use the `Accept` header to request a specific response format. ``` Content-Type: application/cbor Accept: application/cbor ``` All content types work with any standard Harper operation. ## Supported Formats ### JSON — `application/json` JSON is the most widely used format, readable and easy to work with. It is well-supported across all HTTP tooling. **Limitations**: JSON does not natively support all Harper data types — binary data, `Date`, `Map`, and `Set` values require special handling. JSON also produces larger payloads than binary formats. **When to use**: Web development, debugging, interoperability with third-party clients, or when the standard JSON type set is sufficient. Pairing JSON with compression (`Accept-Encoding: gzip, br`) often yields compact network transfers due to favorable Huffman coding characteristics. ### CBOR — `application/cbor` CBOR is the recommended format for most production use cases. It is a highly efficient binary format with native support for the full range of Harper data types, including binary data, typed dates, and explicit Maps/Sets. **Advantages**: Very compact encoding, fast serialization, native streaming support (indefinite-length arrays for optimal time-to-first-byte on query results). Well-standardized with growing ecosystem support. **When to use**: Production APIs, performance-sensitive applications, or any scenario requiring rich data types. ### MessagePack — `application/x-msgpack` MessagePack is another efficient binary format similar to CBOR, with broader adoption in some ecosystems. It supports all Harper data types. **Limitations**: MessagePack does not natively support streaming arrays, so query results are returned as a concatenated sequence of MessagePack objects. Decoders must be prepared to handle a sequence of values rather than a single document. **When to use**: Systems with existing MessagePack support that don't have CBOR available, or when interoperability with MessagePack clients is required. CBOR is generally preferred when both are available. ### CSV — `text/csv` Comma-separated values format, suitable for data export and spreadsheet import/export. CSV lacks hierarchical structure and explicit typing. **When to use**: Ad-hoc data export, spreadsheet workflows, batch data processing. Not recommended for frequent or production API use. ## Content Type via URL Extension As an alternative to the `Accept` header, responses can be requested in a specific format using file-style URL extensions: ``` GET /product/some-id.csv GET /product/.msgpack?category=software ``` Using the `Accept` header is the recommended approach for clean, standard HTTP interactions. ## Custom Content Types Harper's content type system is extensible. Custom handlers for any serialization format (XML, YAML, proprietary formats, etc.) can be registered in the [`contentTypes`](/reference/v4/components/javascript-environment.md#contenttypes) global Map. ## Storing Arbitrary Content Types When a `PUT` or `POST` is made with a non-standard content type (e.g., `text/calendar`, `image/gif`), Harper stores the content as a record with `contentType` and `data` properties: ``` PUT /my-resource/33 Content-Type: text/calendar BEGIN:VCALENDAR VERSION:2.0 ... ``` This stores a record equivalent to: ``` { "contentType": "text/calendar", "data": "BEGIN:VCALENDAR\nVERSION:2.0\n..." } ``` Retrieving a record that has `contentType` and `data` properties returns the response with the specified `Content-Type` and body. If the content type is not from the `text` family, the data is treated as binary (a Node.js `Buffer`). Use `application/octet-stream` for binary data or for uploading to a specific property: ``` PUT /my-resource/33/image Content-Type: image/gif ...image data... ``` ## See Also * [REST Overview](/reference/v4/rest/overview.md) — HTTP methods and URL structure * [Headers](/reference/v4/rest/headers.md) — Content negotiation headers * [Querying](/reference/v4/rest/querying.md) — URL query syntax --- # REST Headers Harper's REST interface uses standard HTTP headers for content negotiation, caching, and performance instrumentation. ## Response Headers These headers are included in all Harper REST API responses: | Header | Example Value | Description | | --------------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `server-timing` | `db;dur=7.165` | Duration of the operation in milliseconds. Follows the [Server-Timing](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Server-Timing) standard and can be consumed by network monitoring tools. | | `content-type` | `application/json` | MIME type of the returned content, negotiated based on the `Accept` request header. | | `etag` | `"abc123"` | Encoded version/last-modification time of the returned record. Used for conditional requests. | | `location` | `/MyTable/new-id` | Returned on `POST` responses. Contains the path to the newly created record. | ## Request Headers ### Content-Type Specifies the format of the request body (for `PUT`, `PATCH`, `POST`): ``` Content-Type: application/json Content-Type: application/cbor Content-Type: application/x-msgpack Content-Type: text/csv ``` See [Content Types](/reference/v4/rest/content-types.md) for the full list of supported formats. ### Accept Specifies the preferred response format: ``` Accept: application/json Accept: application/cbor Accept: application/x-msgpack Accept: text/csv ``` ### If-None-Match Used for conditional GET requests. Provide the `ETag` value from a previous response to avoid re-fetching unchanged data: ``` GET /MyTable/123 If-None-Match: "abc123" ``` If the record has not changed, Harper returns `304 Not Modified` with no body. This avoids serialization and network transfer overhead and works seamlessly with browser caches and external HTTP caches. ### Accept-Encoding Harper supports standard HTTP compression. Including this header enables compressed responses: ``` Accept-Encoding: gzip, br ``` Compression is particularly effective for JSON responses. For binary formats like CBOR, compression provides diminishing returns compared to the already-compact encoding. ### Authorization Credentials for authenticating requests. See [Security Overview](/reference/v4/security/overview.md) for details on supported authentication mechanisms (Basic, JWT, mTLS). ### Sec-WebSocket-Protocol When connecting via WebSocket for MQTT, the sub-protocol must be set to `mqtt` as required by the MQTT specification: ``` Sec-WebSocket-Protocol: mqtt ``` ## Content Type via URL Extension As an alternative to the `Accept` header, content types can be specified using file-style extensions in the URL path: ``` GET /product/some-id.csv GET /product/.msgpack?category=software ``` This is not recommended for production use — prefer the `Accept` header for clean, standard HTTP interactions. ## See Also * [REST Overview](/reference/v4/rest/overview.md) — HTTP methods and URL structure * [Content Types](/reference/v4/rest/content-types.md) — Supported encoding formats * [Security Overview](/reference/v4/security/overview.md) — Authentication headers and mechanisms --- # REST Overview *Added in: v4.2.0* Harper provides a powerful, efficient, and standard-compliant HTTP REST interface for interacting with tables and other resources. The REST interface is the recommended interface for data access, querying, and manipulation over HTTP, providing the best performance and HTTP interoperability with different clients. ## How the REST Interface Works Harper's REST interface exposes database tables and custom resources as RESTful endpoints. Tables are **not** exported by default; they must be explicitly exported in a schema definition. The name of the exported resource defines the base of the endpoint path, served on the application HTTP server port (default `9926`). For more on defining schemas and exporting resources, see [Database / Schema](/reference/v4/database/schema.md). ## Configuration Enable the REST interface by adding the `rest` plugin to your application's `config.yaml`: ``` rest: true ``` **Options**: ``` rest: lastModified: true # enables Last-Modified response header support webSocket: false # disables automatic WebSocket support (enabled by default) ``` ## URL Structure The REST interface follows a consistent URL structure: | Path | Description | | -------------------------------------------- | ------------------------------------------------------------------------------------------------- | | `/my-resource` | Root path — returns a description of the resource (e.g., table metadata) | | `/my-resource/` | Trailing slash indicates a collection — represents all records; append query parameters to search | | `/my-resource/record-id` | A specific record identified by its primary key | | `/my-resource/record-id/` | Trailing slash — the collection of records with the given id prefix | | `/my-resource/record-id/with/multiple/parts` | Record id with multiple path segments | *Changed in: v4.5.0* — Resources can be defined with nested paths and accessed by exact path without a trailing slash. The `id.property` dot syntax for accessing properties via URL is only applied to properties declared in a schema. ## HTTP Methods REST operations map to HTTP methods following uniform interface principles: ### GET Retrieve a record or perform a search. Handled by the resource's `get()` method. ``` GET /MyTable/123 ``` Returns the record with primary key `123`. ``` GET /MyTable/?name=Harper ``` Returns records matching `name=Harper`. See [Querying](/reference/v4/rest/querying.md) for the full query syntax. ``` GET /MyTable/123.propertyName ``` Returns a single property of a record. Only works for properties declared in the schema. #### Conditional Requests and Caching GET responses include an `ETag` header encoding the record's version/last-modification time. Clients with a cached copy can include `If-None-Match` on subsequent requests. If the record hasn't changed, Harper returns `304 Not Modified` with no body — avoiding serialization and network transfer overhead. ### PUT Create or replace a record with a specified primary key (upsert semantics). Handled by the resource's `put(record)` method. The stored record will exactly match the submitted body — any properties not included in the body are removed from the previous record. ``` PUT /MyTable/123 Content-Type: application/json { "name": "some data" } ``` Creates or replaces the record with primary key `123`. ### POST Create a new record without specifying a primary key, or trigger a custom action. Handled by the resource's `post(data)` method. The auto-assigned primary key is returned in the `Location` response header. ``` POST /MyTable/ Content-Type: application/json { "name": "some data" } ``` ### PATCH Partially update a record, merging only the provided properties (CRDT-style update). Unspecified properties are preserved. *Added in: v4.3.0* ``` PATCH /MyTable/123 Content-Type: application/json { "status": "active" } ``` ### DELETE Delete a specific record or all records matching a query. ``` DELETE /MyTable/123 ``` Deletes the record with primary key `123`. ``` DELETE /MyTable/?status=archived ``` Deletes all records matching `status=archived`. ## Content Types Harper supports multiple content types for both request bodies and responses. Use the `Content-Type` header for request bodies and the `Accept` header to request a specific response format. See [Content Types](/reference/v4/rest/content-types.md) for the full list of supported formats and encoding recommendations. ## OpenAPI *Added in: v4.3.0* Harper automatically generates an OpenAPI specification for all resources exported via a schema. This endpoint is available at: ``` GET /openapi ``` ## See Also * [Querying](/reference/v4/rest/querying.md) — Full URL query syntax, operators, and examples * [Headers](/reference/v4/rest/headers.md) — HTTP headers used by the REST interface * [Content Types](/reference/v4/rest/content-types.md) — Supported formats (JSON, CBOR, MessagePack, CSV) * [WebSockets](/reference/v4/rest/websockets.md) — Real-time connections via WebSocket * [Server-Sent Events](/reference/v4/rest/server-sent-events.md) — One-way streaming via SSE * [HTTP Server](/reference/v4/http/overview.md) — Underlying HTTP server configuration * [Database / Schema](/reference/v4/database/schema.md) — How to define and export resources --- # REST Querying Harper's REST interface supports a rich URL-based query language for filtering, sorting, selecting, and limiting records. Queries are expressed as URL query parameters on collection paths. ## Basic Attribute Filtering Search by attribute name and value using query parameters. The queried attribute must be indexed. ``` GET /Product/?category=software ``` Multiple attributes can be combined — only one needs to be indexed for the query to execute: ``` GET /Product/?category=software&inStock=true ``` ### Null Queries *Added in: v4.3.0* Query for null values or non-null values: ``` GET /Product/?discount=null ``` Note: Only indexes created in v4.3.0 or later support null indexing. Existing indexes must be rebuilt (removed and re-added) to support null queries. ## Comparison Operators (FIQL) Harper uses [FIQL](https://datatracker.ietf.org/doc/html/draft-nottingham-atompub-fiql-00) syntax for comparison operators: | Operator | Meaning | | -------------------- | -------------------------------------- | | `==` | Equal | | `=lt=` | Less than | | `=le=` | Less than or equal | | `=gt=` | Greater than | | `=ge=` | Greater than or equal | | `=ne=`, `!=` | Not equal | | `=ct=` | Contains (strings) | | `=sw=`, `==*` | Starts with (strings) | | `=ew=` | Ends with (strings) | | `=`, `===` | Strict equality (no type conversion) | | `!==` | Strict inequality (no type conversion) | **Examples**: ``` GET /Product/?price=gt=100 GET /Product/?price=le=20 GET /Product/?name==Keyboard* GET /Product/?category=software&price=gt=100&price=lt=200 ``` For date fields, colons must be URL-encoded as `%3A`: ``` GET /Product/?listDate=gt=2017-03-08T09%3A30%3A00.000Z ``` ### Chained Conditions (Range) Omit the attribute name on the second condition to chain it against the same attribute: ``` GET /Product/?price=gt=100<=200 ``` Chaining supports `gt`/`ge` combined with `lt`/`le` for range queries. No other chaining combinations are currently supported. ### Type Conversion For FIQL comparators (`==`, `!=`, `=gt=`, etc.), Harper applies automatic type conversion: | Syntax | Behavior | | ----------------------------------------- | ------------------------------------------- | | `name==null` | Converts to `null` | | `name==123` | Converts to number if attribute is untyped | | `name==true` | Converts to boolean if attribute is untyped | | `name==number:123` | Explicit number conversion | | `name==boolean:true` | Explicit boolean conversion | | `name==string:some%20text` | Keep as string with URL decode | | `name==date:2024-01-05T20%3A07%3A27.955Z` | Explicit Date conversion | If the attribute specifies a type in the schema (e.g., `Float`), values are always converted to that type before searching. For strict operators (`=`, `===`, `!==`), no automatic type conversion is applied — the value is decoded as a URL-encoded string, and the attribute type (if declared in the schema) dictates type conversion. ## Unions (OR Logic) Use `|` instead of `&` to combine conditions with OR logic: ``` GET /Product/?rating=5|featured=true ``` ## Grouping Use parentheses or square brackets to control order of operations: ``` GET /Product/?rating=5|(price=gt=100&price=lt=200) ``` Square brackets are recommended when constructing queries from user input because standard URI encoding safely encodes `[` and `]` (but not `(`): ``` GET /Product/?rating=5&[tag=fast|tag=scalable|tag=efficient] ``` Constructing from JavaScript: ``` let url = `/Product/?rating=5&[${tags.map(encodeURIComponent).join('|')}]`; ``` Groups can be nested for complex conditions: ``` GET /Product/?price=lt=100|[rating=5&[tag=fast|tag=scalable|tag=efficient]&inStock=true] ``` ## Query Functions Harper supports special query functions using call syntax, included in the query string separated by `&`. ### `select(properties)` Specify which properties to include in the response. | Syntax | Returns | | -------------------------------------- | ------------------------------------------- | | `?select(property)` | Values of a single property directly | | `?select(property1,property2)` | Objects with only the specified properties | | `?select([property1,property2])` | Arrays of property values | | `?select(property1,)` | Objects with a single specified property | | `?select(property{subProp1,subProp2})` | Nested objects with specific sub-properties | **Examples**: ``` GET /Product/?category=software&select(name) GET /Product/?brand.name=Microsoft&select(name,brand{name}) ``` ### `limit(end)` or `limit(start,end)` Limit the number of results returned, with an optional starting offset. ``` GET /Product/?rating=gt=3&inStock=true&select(rating,name)&limit(20) GET /Product/?rating=gt=3&limit(10,30) ``` ### `sort(property)` or `sort(+property,-property,...)` Sort results by one or more properties. Prefix `+` or no prefix = ascending; `-` = descending. Multiple properties break ties in order. ``` GET /Product/?rating=gt=3&sort(+name) GET /Product/?sort(+rating,-price) ``` *Added in: v4.3.0* ## Relationships and Joins *Added in: v4.3.0* Harper supports querying across related tables through dot-syntax chained attributes. Relationships must be defined in the schema using `@relationship`. **Schema example**: ``` type Product @table @export { id: Long @primaryKey name: String brandId: Long @indexed brand: Brand @relationship(from: "brandId") } type Brand @table @export { id: Long @primaryKey name: String products: [Product] @relationship(to: "brandId") } ``` **Query by related attribute** (INNER JOIN behavior): ``` GET /Product/?brand.name=Microsoft GET /Brand/?products.name=Keyboard ``` ### Nested Select with Joins Relationship attributes are not included by default. Use `select()` to include them: ``` GET /Product/?brand.name=Microsoft&select(name,brand) GET /Product/?brand.name=Microsoft&select(name,brand{name}) GET /Product/?name=Keyboard&select(name,brand{name,id}) ``` When selecting without a filter on the related table, this acts as a LEFT JOIN — the relationship property is omitted if the foreign key is null or references a non-existent record. ### Many-to-Many Relationships Many-to-many relationships can be modeled with an array of foreign key values, without a junction table: ``` type Product @table @export { id: Long @primaryKey name: String resellerIds: [Long] @indexed resellers: [Reseller] @relationship(from: "resellerId") } ``` ``` GET /Product/?resellers.name=Cool Shop&select(id,name,resellers{name,id}) ``` The array order of `resellerIds` is preserved when resolving the relationship. ## Property Access via URL *Changed in: v4.5.0* Access a specific property of a record by appending it with dot syntax to the record id: ``` GET /MyTable/123.propertyName ``` This only works for properties declared in the schema. As of v4.5.0, dots in URL paths are no longer interpreted as property access for undeclared properties, allowing URLs to generally include dots without being misinterpreted. ## `directURLMapping` Option *Added in: v4.5.0* Resources can be configured with `directURLMapping: true` for more direct URL path handling. When enabled, the URL path is mapped more directly to the resource without the default query parameter parsing semantics. See [Database / Schema](/reference/v4/database/schema.md) for configuration details. Common gotchas * **`/Table` vs `/Table/`** — `GET /Table` returns metadata about the table resource itself. `GET /Table/` (trailing slash) targets the collection and invokes `get()` as a collection request. These are distinct endpoints. * **Case sensitivity** — The URL path must match the exact casing of the exported resource or table name. `/Table/` works; `/table/` returns a 404. ## See Also * [REST Overview](/reference/v4/rest/overview.md) — HTTP methods, URL structure, and caching * [Headers](/reference/v4/rest/headers.md) — Request and response headers * [Content Types](/reference/v4/rest/content-types.md) — Encoding formats * [Database / Schema](/reference/v4/database/schema.md) — Defining schemas, relationships, and indexes --- # Server-Sent Events *Added in: v4.2.0* Harper supports Server-Sent Events (SSE), a simple and efficient mechanism for browser-based applications to receive real-time updates from the server over a standard HTTP connection. SSE is a one-directional transport — the server pushes events to the client, and the client has no way to send messages back on the same connection. ## Connecting SSE connections are made by targeting a resource URL. By default, connecting to a resource path subscribes to changes for that resource and streams events as they occur. ``` let eventSource = new EventSource('https://server/my-resource/341', { withCredentials: true, }); eventSource.onmessage = (event) => { let data = JSON.parse(event.data); }; ``` The URL path maps to the resource in the same way as REST and WebSocket connections. Connecting to `/my-resource/341` subscribes to updates for the record with id `341` in the `my-resource` table (or custom resource). ## `connect()` Handler SSE connections use the same `connect()` method as WebSockets on resource classes, with one key difference: since SSE is one-directional, `connect()` is called without an `incomingMessages` argument. ``` export class MyResource extends Resource { async *connect() { // yield messages to send to the client while (true) { await someCondition(); yield { event: 'update', data: { value: 42 } }; } } } ``` The default `connect()` behavior subscribes to the resource and streams changes automatically. ## When to Use SSE vs WebSockets | | SSE | WebSockets | | --------------- | ------------------------------------- | -------------------------------- | | Direction | Server → Client only | Bidirectional | | Transport | Standard HTTP | HTTP upgrade | | Browser support | Native `EventSource` API | Native `WebSocket` API | | Use case | Live feeds, dashboards, notifications | Interactive real-time apps, MQTT | SSE is simpler to implement and has built-in reconnection in browsers. For scenarios requiring bidirectional communication, use [WebSockets](/reference/v4/rest/websockets.md). ## See Also * [WebSockets](/reference/v4/rest/websockets.md) — Bidirectional real-time connections * [MQTT Overview](/reference/v4/mqtt/overview.md) — Full MQTT pub/sub documentation * [REST Overview](/reference/v4/rest/overview.md) — HTTP methods and URL structure * [Resources](/reference/v4/resources/overview.md) — Custom resource API including `connect()` --- # WebSockets *Added in: v4.2.0* Harper supports WebSocket connections through the REST interface, enabling real-time bidirectional communication with resources. WebSocket connections target a resource URL path — by default, connecting to a resource subscribes to changes for that resource. ## Configuration WebSocket support is enabled automatically when the `rest` plugin is enabled. To disable it: ``` rest: webSocket: false ``` ## Connecting A WebSocket connection to a resource URL subscribes to that resource and streams change events: ``` let ws = new WebSocket('wss://server/my-resource/341'); ws.onmessage = (event) => { let data = JSON.parse(event.data); }; ``` By default, `new WebSocket('wss://server/my-resource/341')` accesses the resource defined for `my-resource` with record id `341` and subscribes to it. When the record changes or a message is published to it, the WebSocket connection receives the update. ## Custom `connect()` Handler WebSocket behavior is driven by the `connect(incomingMessages)` method on a resource class. The method must return an async iterable (or generator) that produces messages to send to the client. For more on implementing custom resources, see [Resource API](/reference/v4/resources/resource-api.md). **Simple echo server**: ``` export class Echo extends Resource { async *connect(incomingMessages) { for await (let message of incomingMessages) { yield message; // echo each message back } } } ``` **Using the default connect with event-style access**: The default `connect()` returns a convenient streaming iterable with: * A `send(message)` method for pushing outgoing messages * A `close` event for cleanup on disconnect ``` export class Example extends Resource { connect(incomingMessages) { let outgoingMessages = super.connect(); let timer = setInterval(() => { outgoingMessages.send({ greeting: 'hi again!' }); }, 1000); incomingMessages.on('data', (message) => { outgoingMessages.send(message); // echo incoming messages }); outgoingMessages.on('close', () => { clearInterval(timer); }); return outgoingMessages; } } ``` ## MQTT over WebSockets Harper also supports MQTT over WebSockets. The sub-protocol must be set to `mqtt` as required by the MQTT specification: ``` Sec-WebSocket-Protocol: mqtt ``` See [MQTT Overview](/reference/v4/mqtt/overview.md) for full MQTT documentation. ## Message Ordering in Distributed Environments Harper prioritizes low-latency delivery in distributed (multi-node) environments. Messages are delivered to local subscribers immediately upon arrival — Harper does not delay messages for inter-node coordination. In a scenario where messages arrive out-of-order across nodes: * **Non-retained messages** (published without a `retain` flag): Every message is delivered to subscribers in the order received, even if out-of-order relative to other nodes. Good for use cases like chat where every message must be delivered. * **Retained messages** (published with `retain`, or PUT/updated in the database): Only the message with the latest timestamp is kept as the "winning" record. Out-of-order older messages are not re-delivered. This ensures eventual consistency of the most recent record state across the cluster. Good for use cases like sensor readings where only the latest value matters. ## See Also * [Server-Sent Events](/reference/v4/rest/server-sent-events.md) — One-way real-time streaming * [MQTT Overview](/reference/v4/mqtt/overview.md) — Full MQTT pub/sub documentation * [REST Overview](/reference/v4/rest/overview.md) — HTTP methods and URL structure * [Resources](/reference/v4/resources/overview.md) — Custom resource API including `connect()` --- # Security API Harper exposes security-related globals accessible in all component JavaScript modules without needing to import them. *** ## `auth(username, password?): Promise` Returns the user object for the given username. If `password` is provided, it is verified before returning the user (throws on incorrect password). ``` const user = await auth('admin', 'secret'); // user.role, user.username, etc. ``` This is useful for implementing custom authentication flows or verifying credentials in component code. For HTTP-level authentication configuration, see [Security Overview](/reference/v4/security/overview.md). --- # Basic Authentication Available since: v4.1.0 Harper supports HTTP Basic Authentication. In the context of an HTTP transaction, [Basic Authentication](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Authentication#basic_authentication_scheme) is the simplest authorization scheme which transmits credentials as username/password pairs encoded using base64. Importantly, this scheme does not encrypt credentials. If used over an insecure connection, such as HTTP, they are susceptible to being compromised. Only ever use Basic Authentication over secured connections, such as HTTPS. Even then, its better to upgrade to an encryption based authentication scheme or certificates. See [HTTP / TLS](/reference/v4/http/tls.md) for more information. ## How It Works Each request must contain the `Authorization` header with a value if `Basic `, where `` is the Base64 encoding of the string `username:password`. ``` Authorization: Basic ``` ## Example The following example shows how to construct the Authorization header using `btoa()`: ``` const username = 'HDB_ADMIN'; const password = 'abc123!'; const authorizationValue = `Basic ${btoa(`${username}:${password}`)}`; ``` Then use the `authorizationValue` as the value for the `Authorization` header such as: ``` fetch('/', { // ... headers: { Authorization: authorizationValue, }, // ... }); ``` ## cURL Example With cURL you can use the `--user` (`-u`) command-line option to automatically handle the Base64 encoding: ``` curl -u "username:password" [URL] ``` ## When to Use Basic Auth Basic authentication is the simplest option and is appropriate for: * Server-to-server requests in trusted environments * Development and testing * Scenarios where token management overhead is undesirable For user-facing applications or when tokens are preferred for performance reasons, see [JWT Authentication](/reference/v4/security/jwt-authentication.md). --- # Certificate Management This page covers certificate management for Harper's external-facing HTTP and Operations APIs. For replication certificate management, see [Replication Certificate Management](/reference/v4/replication/clustering.md). ## Default Behavior On first run, Harper automatically generates self-signed TLS certificates at `/keys/`: * `certificate.pem` — The server certificate * `privateKey.pem` — The server private key * `ca.pem` — A self-signed Certificate Authority These certificates have a valid Common Name (CN), but they are not signed by a root authority. HTTPS can be used with them, but clients must be configured to accept the invalid certificate. ## Development Setup By default, HTTPS is disabled. HTTP is suitable for local development and trusted private networks. If you are developing on a remote server with requests traversing the Internet, enable HTTPS. To enable HTTPS, set `http.securePort` in `harperdb-config.yaml` and restart Harper: ``` http: securePort: 9926 ``` Harper will use the auto-generated certificates from `/keys/`. ## Production Setup For production, use certificates from your own CA or a public CA, with CNs that match the Fully Qualified Domain Name (FQDN) of your Harper node. ### Option 1: Replace Harper Certificates Enable HTTPS and replace the certificate files: ``` http: securePort: 9926 tls: certificate: ~/hdb/keys/certificate.pem privateKey: ~/hdb/keys/privateKey.pem ``` Either replace the files at `/keys/` in place, or update `tls.certificate` and `tls.privateKey` to point to your new files and restart Harper. The `operationsApi.tls` section is optional. If not set, Harper uses the values from the top-level `tls` section. You can specify different certificates for the Operations API: ``` operationsApi: tls: certificate: ~/hdb/keys/certificate.pem privateKey: ~/hdb/keys/privateKey.pem ``` ### Option 2: Nginx Reverse Proxy Instead of enabling HTTPS directly on Harper, use Nginx as a reverse proxy. Configure Nginx to handle HTTPS with certificates from your own CA or a public CA, then forward HTTP requests to Harper. This approach keeps Harper's HTTP interface internal while Nginx handles TLS termination. ### Option 3: External Reverse Proxy / Load Balancer External services such as an AWS Elastic Load Balancer or Google Cloud Load Balancing can act as TLS-terminating reverse proxies. Configure the service to accept HTTPS connections and forward over a private network to Harper as HTTP. These services typically include integrated certificate management. ## mTLS Setup Mutual TLS (mTLS) requires both client and server to present certificates. To enable mTLS, provide a CA certificate that Harper will use to verify client certificates: ``` http: mtls: required: true tls: certificateAuthority: ~/hdb/keys/ca.pem ``` For full mTLS authentication details, see [mTLS Authentication](/reference/v4/security/mtls-authentication.md). ## Certificate Verification *Added in: v4.5.0* (certificate revocation); v4.7.0 (OCSP support) When using mTLS, enable certificate verification to ensure revoked certificates cannot authenticate even if still within their validity period: ``` http: mtls: required: true certificateVerification: true ``` Harper supports two industry-standard methods: **CRL (Certificate Revocation List)** * Downloaded and cached locally (24 hours by default) * Fast verification after first download (no network requests) * Best for high-volume verification and offline scenarios **OCSP (Online Certificate Status Protocol)** * Real-time query to the CA's OCSP responder * Best for certificates without CRL distribution points * Responses cached (1 hour by default) **Harper's approach: CRL-first with OCSP fallback** 1. Checks CRL if available (fast, cached locally) 2. Falls back to OCSP if CRL is unavailable or fails 3. Applies the configured failure mode if both methods fail For full configuration options and troubleshooting, see [Certificate Verification](/reference/v4/security/certificate-verification.md). ## Dynamic Certificate Management *Added in: v4.4.0* Certificates — including CAs and private keys — can be dynamically managed without restarting Harper. ## Multiple Certificate Authorities It is possible to use different certificates for the Operations API and the HTTP (custom application) API. For example, in scenarios where only your application endpoints need to be exposed to the Internet and the Operations API is reserved for administration, you may use a private CA for the Operations API and a public CA for your application certificates. Configure each separately: ``` # Top-level tls: used by HTTP/application endpoints tls: certificate: ~/hdb/keys/app-certificate.pem privateKey: ~/hdb/keys/app-privateKey.pem # Operations API can use a separate cert operationsApi: tls: certificate: ~/hdb/keys/ops-certificate.pem privateKey: ~/hdb/keys/ops-privateKey.pem ``` ## Renewing Certificates The `harper renew-certs` CLI command renews the auto-generated Harper certificates. See [CLI Commands](/reference/v4/cli/commands.md) for details. **Changes to TLS settings require a restart**, except where dynamic certificate management is used. --- # Certificate Verification *Added in: v4.5.0* *Changed in: v4.7.0* (OCSP support added) Certificate verification (also called certificate revocation checking) ensures that revoked certificates cannot be used for mTLS authentication, even if they are otherwise valid and trusted. This is a critical security control for environments where certificates may need to be revoked before their expiration date — due to compromise, employee departure, or other security concerns. ## Overview When a client presents a certificate for mTLS authentication, Harper performs two levels of checks: 1. **Certificate Validation** (always performed by Node.js TLS): * Certificate signature is valid * Certificate is issued by a trusted CA * Certificate is within its validity period * Certificate chain is properly formed 2. **Certificate Revocation Checking** (optional, must be explicitly enabled): * Certificate has not been revoked by the issuing CA * Uses CRL and/or OCSP Revocation checking is **disabled by default**. ## Revocation Checking Methods ### CRL (Certificate Revocation List) A CRL is a digitally signed list of revoked certificates published by a Certificate Authority. **Advantages:** * Fast verification (cached locally) * Works offline once downloaded * Predictable bandwidth usage * Good for high-volume verification * No privacy concerns (no per-certificate queries) **How it works:** 1. Harper downloads the CRL from the distribution point specified in the certificate. 2. The CRL is cached locally (24 hours by default). 3. Subsequent verifications check the cached CRL — very fast, no network requests. 4. The CRL is refreshed in the background before expiration. **Configuration:** ``` http: mtls: certificateVerification: crl: timeout: 10000 # 10 seconds to download CRL cacheTtl: 86400000 # Cache for 24 hours gracePeriod: 86400000 # 24 hour grace period after nextUpdate failureMode: fail-closed # Reject on CRL check failure ``` ### OCSP (Online Certificate Status Protocol) OCSP provides real-time certificate status checking by querying the CA's OCSP responder. **Advantages:** * Real-time revocation status * Smaller response size than CRL * Good for certificates without CRL distribution points * Works when CRL is unavailable **How it works:** 1. Harper sends a request to the OCSP responder specified in the certificate. 2. The responder returns the current status: good, revoked, or unknown. 3. The response is cached (1 hour by default for success, 5 minutes for errors). **Configuration:** ``` http: mtls: certificateVerification: ocsp: timeout: 5000 # 5 seconds for OCSP response cacheTtl: 3600000 # Cache successful responses for 1 hour errorCacheTtl: 300000 # Cache errors for 5 minutes failureMode: fail-closed # Reject on OCSP check failure ``` ## Verification Strategy Harper uses a **CRL-first strategy with OCSP fallback**: 1. **Check CRL** if available (fast; uses cached CRL; no network request if cached). 2. **Fall back to OCSP** if the certificate has no CRL distribution point, the CRL download fails, or the CRL is expired and cannot be refreshed. 3. **Apply failure mode** if both methods fail. This provides the best balance of performance, reliability, and security. ## Configuration ### Enable with Defaults ``` http: mtls: required: true certificateVerification: true ``` This enables CRL checking (10s timeout, 24h cache), OCSP checking (5s timeout, 1h cache), and fail-closed mode. ### Custom Configuration ``` http: mtls: required: true certificateVerification: failureMode: fail-closed # Global setting crl: timeout: 15000 # 15 seconds for CRL download cacheTtl: 43200000 # Cache CRLs for 12 hours gracePeriod: 86400000 # 24 hour grace period failureMode: fail-closed # CRL-specific setting ocsp: timeout: 8000 # 8 seconds for OCSP response cacheTtl: 7200000 # Cache results for 2 hours errorCacheTtl: 600000 # Cache errors for 10 minutes failureMode: fail-closed # OCSP-specific setting ``` ### CRL Only (No OCSP) ``` http: mtls: certificateVerification: ocsp: false # Disable OCSP; CRL remains enabled ``` Only disable OCSP if all client certificates have CRL distribution points. Otherwise, certificates without CRL URLs won't be checked for revocation. ### OCSP Only (No CRL) ``` http: mtls: certificateVerification: crl: false # Disable CRL; OCSP remains enabled ``` ### Environment Variables All settings can be configured via environment variables: ``` # Enable certificate verification HTTP_MTLS_CERTIFICATEVERIFICATION=true # Global failure mode HTTP_MTLS_CERTIFICATEVERIFICATION_FAILUREMODE=fail-closed # CRL settings HTTP_MTLS_CERTIFICATEVERIFICATION_CRL=true HTTP_MTLS_CERTIFICATEVERIFICATION_CRL_TIMEOUT=15000 HTTP_MTLS_CERTIFICATEVERIFICATION_CRL_CACHETTL=43200000 HTTP_MTLS_CERTIFICATEVERIFICATION_CRL_GRACEPERIOD=86400000 HTTP_MTLS_CERTIFICATEVERIFICATION_CRL_FAILUREMODE=fail-closed # OCSP settings HTTP_MTLS_CERTIFICATEVERIFICATION_OCSP=true HTTP_MTLS_CERTIFICATEVERIFICATION_OCSP_TIMEOUT=8000 HTTP_MTLS_CERTIFICATEVERIFICATION_OCSP_CACHETTL=7200000 HTTP_MTLS_CERTIFICATEVERIFICATION_OCSP_ERRORCACHETTL=600000 HTTP_MTLS_CERTIFICATEVERIFICATION_OCSP_FAILUREMODE=fail-closed ``` For replication servers, use the `REPLICATION_` prefix instead of `HTTP_`. ## Failure Modes ### fail-closed (Recommended) **Default behavior.** Rejects connections when verification fails due to network errors, timeouts, or other operational issues. Use when: * Security is paramount * You can tolerate false positives (rejecting valid certificates due to CA unavailability) * Your CA infrastructure is highly available * You're in a zero-trust environment ``` certificateVerification: failureMode: fail-closed ``` ### fail-open Allows connections when verification fails, but logs a warning. The connection is still rejected if the certificate is explicitly found to be revoked. Use when: * Availability is more important than perfect security * Your CA infrastructure may be intermittently unavailable * You have other compensating controls * You're gradually rolling out certificate verification ``` certificateVerification: failureMode: fail-open ``` **Important:** Invalid signatures on CRLs always result in rejection regardless of failure mode, as this indicates potential tampering. ## Performance Considerations ### CRL Performance * **First verification**: Downloads CRL (10s timeout by default) * **Subsequent verifications**: Instant (reads from cache) * **Background refresh**: CRL is refreshed before expiration without blocking requests * **Memory usage**: \~10–100KB per CRL depending on size * **Network usage**: One download per CRL per `cacheTtl` period ### OCSP Performance * **First verification**: OCSP query (5s timeout by default) * **Subsequent verifications**: Reads from cache (1 hour default) * **Memory usage**: Minimal (\~1KB per cached response) * **Network usage**: One query per unique certificate per `cacheTtl` period ### Optimization Tips Increase CRL cache TTL for stable environments: ``` --- crl: cacheTtl: 172800000 # 48 hours ``` Increase OCSP cache TTL for long-lived connections: ``` --- ocsp: cacheTtl: 7200000 # 2 hours ``` Reduce grace period for tighter revocation enforcement: ``` --- crl: gracePeriod: 0 # No grace period ``` ## Production Best Practices ### High-Security Environments ``` http: mtls: required: true certificateVerification: failureMode: fail-closed crl: timeout: 15000 cacheTtl: 43200000 # 12 hours gracePeriod: 0 # No grace period for strict enforcement ocsp: timeout: 8000 cacheTtl: 3600000 # 1 hour ``` ### High-Availability Environments ``` http: mtls: required: true certificateVerification: failureMode: fail-open # Prioritize availability crl: timeout: 5000 cacheTtl: 86400000 # 24 hours gracePeriod: 86400000 # 24 hour grace period ocsp: timeout: 3000 cacheTtl: 7200000 # 2 hours ``` ### Performance-Critical Environments ``` http: mtls: required: true certificateVerification: crl: cacheTtl: 172800000 # 48 hours gracePeriod: 86400000 ocsp: cacheTtl: 7200000 # 2 hours errorCacheTtl: 600000 ``` ## Troubleshooting ### Connection Rejected: Certificate Verification Failed **Cause:** Certificate was found to be revoked, or verification failed in fail-closed mode. **Solutions:** 1. Check if the certificate is actually revoked in the CRL or OCSP responder. 2. Verify CA infrastructure is accessible. 3. Check timeout settings — increase if needed. 4. Temporarily switch to fail-open mode while investigating. ### High Latency on First Connection **Cause:** CRL is being downloaded for the first time. **Solutions:** 1. This is normal; only happens once per CRL per `cacheTtl` period. 2. Subsequent connections will be fast (cached CRL). 3. Increase CRL timeout if downloads are slow: ``` crl: timeout: 20000 # 20 seconds ``` ### Frequent CRL Downloads **Cause:** `cacheTtl` is too short, or the CRL's `nextUpdate` period is very short. **Solutions:** 1. Increase `cacheTtl`: ``` crl: cacheTtl: 172800000 # 48 hours ``` 2. Increase `gracePeriod` to allow using slightly expired CRLs. ### OCSP Responder Unavailable **Cause:** OCSP responder is down or unreachable. **Solutions:** 1. CRL will be used as fallback automatically. 2. Use fail-open mode to allow connections: ``` ocsp: failureMode: fail-open ``` 3. Disable OCSP and rely on CRL only (ensure all certs have CRL URLs): ``` ocsp: false ``` ### Network or Firewall Blocking Outbound Requests **Cause:** Secure hosting environments often restrict outbound HTTP/HTTPS traffic. This prevents Harper from reaching CRL distribution points and OCSP responders. **Symptoms:** * Certificate verification timeouts in fail-closed mode * Logs show connection failures to CRL/OCSP URLs * First connection may succeed (no cached data), but subsequent connections fail after cache expires **Solutions:** 1. **Allow outbound traffic to CA infrastructure** (recommended): * Whitelist CRL distribution point URLs from your certificates * Whitelist OCSP responder URLs from your certificates * Example for Let's Encrypt: allow `http://x1.c.lencr.org/` and `http://ocsp.int-x3.letsencrypt.org/` 2. **Use fail-open mode:** ``` certificateVerification: failureMode: fail-open ``` 3. **Set up an internal CRL mirror/proxy:** ``` certificateVerification: crl: cacheTtl: 172800000 # 48 hours ocsp: false ``` 4. **Disable verification** (if you have alternative security controls): ``` certificateVerification: false ``` ## Security Considerations Enable certificate verification when: * Certificates have long validity periods (> 1 day) * You need immediate revocation capability * Compliance requires revocation checking (PCI DSS, HIPAA, etc.) * You're in a zero-trust security model * Client certificates are used for API authentication Consider skipping it when: * Certificates have very short validity periods (< 24 hours) * You rotate certificates automatically (e.g., with cert-manager) * You have alternative revocation mechanisms * Your CA doesn't publish CRLs or support OCSP Certificate verification is one layer of security. Also consider: short certificate validity periods, certificate pinning, network segmentation, access logging, and regular certificate rotation. ## Replication Certificate verification works identically for replication servers. Use the `replication.mtls` configuration: ``` replication: hostname: server-one routes: - server-two mtls: certificateVerification: true ``` mTLS is always required for replication and cannot be disabled. This configuration only controls whether certificate revocation checking is performed. For complete replication configuration, see [Replication Configuration](/reference/v4/replication/clustering.md). --- # Authentication Configuration Harper's authentication system is configured via the top-level `authentication` section of `harperdb-config.yaml`. ``` authentication: authorizeLocal: true cacheTTL: 30000 enableSessions: true operationTokenTimeout: 1d refreshTokenTimeout: 30d hashFunction: sha256 ``` ## Options ### `authorizeLocal` *Type: boolean — Default: `true`* Automatically authorizes requests from the loopback IP address (`127.0.0.1`) as the superuser, without requiring credentials. Disable this for any Harper server that may be accessed by untrusted users from the same instance — for example, when using a local proxy or for general server hardening. ### `cacheTTL` *Type: number — Default: `30000`* How long (in milliseconds) an authentication result — a particular `Authorization` header or token — can be cached. Increasing this improves performance at the cost of slower revocation. ### `enableSessions` *Type: boolean — Default: `true`* *Added in: v4.2.0* Enables cookie-based sessions to maintain an authenticated session across requests. This is the preferred authentication mechanism for web browsers: cookies hold the token securely without exposing it to JavaScript, reducing XSS vulnerability risk. ### `operationTokenTimeout` *Type: string — Default: `1d`* How long a JWT operation token remains valid before expiring. Accepts [`jsonwebtoken`-compatible](https://github.com/auth0/node-jsonwebtoken#token-expiration-exp-claim) duration strings (e.g., `1d`, `12h`, `60m`). See [JWT Authentication](/reference/v4/security/jwt-authentication.md). ### `refreshTokenTimeout` *Type: string — Default: `30d`* How long a JWT refresh token remains valid before expiring. Accepts [`jsonwebtoken`-compatible](https://github.com/auth0/node-jsonwebtoken#token-expiration-exp-claim) duration strings. See [JWT Authentication](/reference/v4/security/jwt-authentication.md). ### `hashFunction` *Type: string — Default: `sha256`* *Added in: v4.5.0* Password hashing algorithm used when storing user passwords. Replaced the previous MD5 hashing. Options: * **`sha256`** — Default. Good security and excellent performance. * **`argon2id`** — Highest security. More CPU-intensive; recommended for environments that do not require frequent password verifications. ## Related * [JWT Authentication](/reference/v4/security/jwt-authentication.md) * [Basic Authentication](/reference/v4/security/basic-authentication.md) * [Users & Roles / Configuration](/reference/v4/users-and-roles/configuration.md) --- # JWT Authentication Available since: v4.1.0 Harper supports token-based authentication using JSON Web Tokens (JWTs). Rather than sending credentials on every request, a client authenticates once and receives tokens that are used for subsequent requests. ## Tokens JWT authentication uses two token types: * **`operation_token`** — Used to authenticate all Harper operations via a `Bearer` token `Authorization` header. Default expiry: 1 day. * **`refresh_token`** — Used to obtain a new `operation_token` when the current one expires. Default expiry: 30 days. ## Create Authentication Tokens Call `create_authentication_tokens` with your Harper credentials. No `Authorization` header is required for this operation. ``` { "operation": "create_authentication_tokens", "username": "username", "password": "password" } ``` cURL example: ``` curl --location --request POST 'http://localhost:9925' \ --header 'Content-Type: application/json' \ --data-raw '{ "operation": "create_authentication_tokens", "username": "username", "password": "password" }' ``` Response: ``` { "operation_token": "", "refresh_token": "" } ``` ## Using the Operation Token Pass the `operation_token` as a `Bearer` token in the `Authorization` header on subsequent requests: ``` curl --location --request POST 'http://localhost:9925' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data-raw '{ "operation": "search_by_hash", "schema": "dev", "table": "dog", "hash_values": [1], "get_attributes": ["*"] }' ``` ## Refreshing the Operation Token When the `operation_token` expires, use the `refresh_token` to obtain a new one. Pass the `refresh_token` as the `Bearer` token: ``` curl --location --request POST 'http://localhost:9925' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data-raw '{ "operation": "refresh_operation_token" }' ``` Response: ``` { "operation_token": "" } ``` When both tokens have expired, call `create_authentication_tokens` again with your username and password. ## Token Expiry Configuration Token timeouts are configurable in `harperdb-config.yaml` under the top-level `authentication` section: ``` authentication: operationTokenTimeout: 1d # Default: 1 day refreshTokenTimeout: 30d # Default: 30 days ``` Valid duration string values follow the [`jsonwebtoken` package format](https://github.com/auth0/node-jsonwebtoken#token-expiration-exp-claim) (e.g., `1d`, `12h`, `60m`). See [Security / Configuration](/reference/v4/security/configuration.md) for the full authentication config reference. ## When to Use JWT Auth JWT authentication is preferred over Basic Auth when: * You want to avoid sending credentials on every request * Your client can store and manage tokens * You have multiple sequential requests and want to avoid repeated credential encoding For simple or server-to-server scenarios, see [Basic Authentication](/reference/v4/security/basic-authentication.md). ## Security Notes * Always use HTTPS in production to protect tokens in transit. See [HTTP / TLS](/reference/v4/http/tls.md). * Store tokens securely; treat them like passwords. * If a token is compromised, it will remain valid until it expires. Consider setting shorter `operationTokenTimeout` values in high-security environments. --- # mTLS Authentication *Added in: v4.3.0* Harper supports Mutual TLS (mTLS) authentication for incoming HTTP connections. When enabled, the client must present a certificate signed by a trusted Certificate Authority (CA). If the certificate is valid and trusted, the connection is authenticated using the user whose username matches the `CN` (Common Name) from the client certificate's `subject`. ## How It Works 1. The client presents a TLS certificate during the handshake. 2. Harper validates the certificate against the configured CA (`tls.certificateAuthority`). 3. If valid, Harper extracts the `CN` from the certificate subject and uses it as the username for the request. 1. Or it is configurable via the `http.mtls.user` option in the relevant configuration object. 4. Optionally, Harper checks whether the certificate has been revoked (see [Certificate Verification](/reference/v4/security/certificate-verification.md)). ## Configuration mTLS is configured via the `http.mtls` section in `harperdb-config.yaml`. **Require mTLS for all connections:** ``` http: mtls: required: true tls: certificateAuthority: ~/hdb/keys/ca.pem ``` **Make mTLS optional (accept both mTLS and non-mTLS connections):** ``` http: mtls: required: false tls: certificateAuthority: ~/hdb/keys/ca.pem ``` When `required` is `false`, clients that do not present a certificate will fall back to other authentication methods (Basic Auth or JWT). For more configuration information see the [HTTP / Configuration](/reference/v4/http/configuration.md) and [HTTP / TLS](/reference/v4/http/tls.md) sections. ## Certificate Revocation Checking When using mTLS, you can optionally enable certificate revocation checking to ensure that revoked certificates cannot authenticate, even if they are otherwise valid and trusted. To enable: ``` http: mtls: required: true certificateVerification: true ``` Certificate revocation checking is **disabled by default** and must be explicitly enabled. For full details on CRL and OCSP configuration, see [Certificate Verification](/reference/v4/security/certificate-verification.md). ## User Identity The username for the mTLS-authenticated request is derived from the `CN` field of the client certificate's subject. Ensure the CN value matches an existing Harper user account. See [Users and Roles](/reference/v4/users-and-roles/overview.md) for managing user accounts. ## Setup Requirements To use mTLS you need: 1. A Certificate Authority (CA) certificate configured in `tls.certificateAuthority`. 2. Client certificates signed by that CA, with a `CN` matching a Harper username. 3. The `http.mtls` configuration enabled. For help generating and managing certificates, see [Certificate Management](/reference/v4/security/certificate-management.md). ## Replication mTLS is always required for Harper replication and cannot be disabled. For replication-specific mTLS configuration, see [Replication Configuration](/reference/v4/replication/clustering.md). --- # Security Harper uses role-based, attribute-level security to ensure that users can only gain access to the data they are supposed to be able to access. Granular permissions allow for unparalleled flexibility and control, and can lower the total cost of ownership compared to other database solutions, since you no longer need to replicate subsets of data to isolate use cases. ## Security Philosophy Harper's security model has two distinct layers: **Authentication** determines *who* is making a request. Harper validates each request using one of the methods above, then resolves the caller to a known Harper user account. **Authorization** determines *what* the caller can do. Each Harper user is assigned a role. Roles carry a permissions set that grants or denies CRUD access at the table and attribute level, in addition to controlling access to system operations. For details on how roles and permissions work, see [Users and Roles](/reference/v4/users-and-roles/overview.md). ## Authentication Methods Harper supports three authentication methods: * [Basic Authentication](/reference/v4/security/basic-authentication.md) — Username and password sent as a Base64-encoded `Authorization` header on every request. * [JWT Authentication](/reference/v4/security/jwt-authentication.md) — Token-based authentication using JSON Web Tokens. Clients authenticate once and receive short-lived operation tokens and longer-lived refresh tokens. * [mTLS Authentication](/reference/v4/security/mtls-authentication.md) — Mutual TLS certificate-based authentication. ## Certificate Management * [Certificate Management](/reference/v4/security/certificate-management.md) — Managing TLS certificates and Certificate Authorities for HTTPS and mTLS. * [Certificate Verification](/reference/v4/security/certificate-verification.md) — Certificate revocation checking via CRL and OCSP. ## Access Control * CORS — Cross-Origin Resource Sharing. * For HTTP server configuration see [HTTP / Configuration / CORS](/reference/v4/http/configuration.md#cors) * For Operations API configuration see [Operations API / Configuration](/reference/v4/configuration/operations.md) * SSL & HTTPS — Enabling HTTPS and configuring TLS for the HTTP server. * For HTTP server configuration see [HTTP / Configuration / TLS](/reference/v4/http/tls.md) * For Operations API configuration see [Operations API / Configuration](/reference/v4/configuration/operations.md) * [Users and Roles](/reference/v4/users-and-roles/overview.md) — Role-Based Access Control (RBAC): defining roles, assigning permissions, and managing users. ## API * [Security API](/reference/v4/security/api.md) — JavaScript globals for security operations (e.g. `auth()`). ## Default Behavior Out of the box, Harper: * Generates self-signed TLS certificates at `/keys/` on first run. * Runs with HTTPS disabled (HTTP only on port 9925 for the Operations API). It is recommended that you never directly expose Harper's HTTP interface through a publicly available port. * Enables CORS for all origins (configurable). * Supports Basic Auth and JWT Auth by default; mTLS must be explicitly configured. --- # Static Files * Added in: v4.5.0 * Changed in: v4.7.0 - (Migrated to Plugin API and new options added) The `static` built-in plugin serves static files from your Harper application over HTTP. Use it to host websites, SPAs, downloadable assets, or any static content alongside your Harper data and API endpoints. `static` does **not** need to be installed — it is built into Harper and only needs to be declared in your `config.yaml`. ## Basic Usage Configure `static` with the `files` option pointing to the files you want to serve: ``` static: files: 'site/**' ``` Given a component with this structure: ``` my-app/ ├─ site/ │ ├─ index.html │ ├─ about.html │ ├─ blog/ │ ├─ post-1.html │ ├─ post-2.html ├─ config.yaml ``` Files are accessed relative to the matched directory root, so `GET /index.html` returns `site/index.html` and `GET /blog/post-1.html` returns `site/blog/post-1.html`. ## `files` and `urlPath` Options *Added in: v4.5* `static` is a [Plugin](/reference/v4/components/overview.md) and supports the standard `files` and `urlPath` configuration options for controlling which files to serve and at what URL path. Use `urlPath` to mount the files at a specific URL prefix: ``` static: files: 'site/**' urlPath: 'app' ``` Now `GET /app/index.html` returns `site/index.html` and `GET /app/blog/post-1.html` returns `site/blog/post-1.html`. See [Components Overview](/reference/v4/components/overview.md) for full `files` glob pattern and `urlPath` documentation. ## Additional Options *Added in: v4.7* In addition to the standard `files`, `urlPath`, and `timeout` options, `static` supports these configuration options: * **`index`** - `boolean` - *optional* - If `true`, automatically serves `index.html` when a request targets a directory. Defaults to `false`. * **`extensions`** - `string[]` - *optional* - File extensions to try when an exact path match is not found. For example, `extensions: ['html']` means a request for `/page-1` will also try `/page-1.html`. * **`fallthrough`** - `boolean` - *optional* - If `true`, passes the request to the next handler when the requested file is not found. Set to `false` when using `notFound` to customize 404 responses. Defaults to `true`. * **`notFound`** - `string | { file: string; statusCode: number }` - *optional* - A custom file (or file + status code) to return when a path is not found. Useful for serving a custom 404 page or for SPAs that use client-side routing. ## Auto-Updates *Added in: v4.7.0* Because `static` uses the Plugin API, it automatically responds to changes without requiring a Harper restart. Adding, removing, or modifying files — or updating `config.yaml` — takes effect immediately. ## Examples ### Basic static file serving Serve all files in the `static/` directory. Requests must match file names exactly. ``` static: files: 'static/**' ``` ### Automatic `index.html` serving Serve `index.html` automatically when a request targets a directory: ``` static: files: 'static/**' index: true ``` With this structure: ``` my-app/ ├─ static/ │ ├─ index.html │ ├─ blog/ │ ├─ index.html │ ├─ post-1.html ``` Request mappings: ``` GET / -> static/index.html GET /blog -> static/blog/index.html GET /blog/post-1.html -> static/blog/post-1.html ``` ### Automatic extension matching Combine `index` and `extensions` for clean URLs without file extensions: ``` static: files: 'static/**' index: true extensions: ['html'] ``` Request mappings with the same structure: ``` GET / -> static/index.html GET /blog -> static/blog/index.html GET /blog/post-1 -> static/blog/post-1.html ``` ### Custom 404 page Return a specific file when a requested path is not found: ``` static: files: 'static/**' notFound: 'static/404.html' fallthrough: false ``` A request to `/non-existent` returns the contents of `static/404.html` with a `404` status code. > **Note:** When using `notFound`, set `fallthrough: false` so the request does not pass through to another handler before the custom 404 response is returned. ### SPA client-side routing For SPAs that handle routing in the browser, return the main application file for any unmatched path: ``` static: files: 'static/**' fallthrough: false notFound: file: 'static/index.html' statusCode: 200 ``` A request to any unmatched path returns `static/index.html` with a `200` status code, allowing the client-side router to handle navigation. ## Related * [Components Overview](/reference/v4/components/overview.md) --- # Local Studio * Added in: v4.1.0 * Changed in: v4.3.0 (Upgrade to match Cloud client) * Changed in: v4.7.0 (Upgraded to match Fabric client) Harper Local Studio is a web-based GUI that enables you to administer, navigate, and monitor your Harper instance through a simple, user-friendly interface without requiring knowledge of the underlying Harper APIs. It is automatically bundled with all Harper instances and is enabled by default on the Operations API port. If you're looking for the platform as a service interface, go to [Harper Fabric](https://fabric.harper.fast) instead. ## Configuration To enable the local Studio, set `localStudio.enabled` to `true` in your [configuration file](/reference/v4/configuration/options.md#localstudio): ``` localStudio: enabled: true ``` The local studio is provided by the [Operations API](/reference/v4/operations-api/overview.md) and is available on the configured `operationsApi.port` or `operationsApi.securePort` values. This is `9925` by default. ## Accessing Local Studio The local Studio can be accessed through your browser at: ``` http://localhost:9925 ``` All database interactions from the local Studio are made directly from your browser to your Harper instance. Authentication is maintained via session cookies. --- # Configuration ## Managing Roles with Config Files In addition to managing roles via the Operations API, Harper supports declaring roles in a configuration file. When the application starts, Harper ensures all declared roles exist with the specified permissions. Configure in your application's `config.yaml`: ``` roles: files: roles.yaml ``` Example `roles.yaml`: ``` analyst: super_user: false data: Sales: read: true insert: false update: false delete: false editor: data: Articles: read: true insert: true update: true attributes: title: read: true update: true author: read: true update: false ``` **Startup behavior:** * If a declared role does not exist, Harper creates it. * If a declared role already exists, Harper updates its permissions to match the definition. ## Password Hashing *Added in: v4.5.0* Harper supports two password hashing algorithms, replacing the previous MD5 hashing: * **`sha256`** — Default algorithm. Good security and excellent performance. * **`argon2id`** — Highest security. More CPU-intensive; recommended for high-security environments. Password hashing is configured via the `authentication.hashFunction` key in `harperdb-config.yaml`. See [Security / Configuration](/reference/v4/security/configuration.md#hashfunction) for details. ## Related * [Overview](/reference/v4/users-and-roles/overview.md) * [Operations](/reference/v4/users-and-roles/operations.md) --- # Operations ## Roles ### List Roles *Restricted to `super_user` roles.* ``` { "operation": "list_roles" } ``` ### Add Role *Restricted to `super_user` roles.* * `role` *(required)* — Name for the new role. * `permission` *(required)* — Permissions object. See [Permission Structure](/reference/v4/users-and-roles/overview.md#permission-structure). * `super_user` *(optional)* — If `true`, grants full access. Defaults to `false`. * `structure_user` *(optional)* — Boolean or array of database names. If `true`, can create/drop databases and tables. If array, limited to specified databases. ``` { "operation": "add_role", "role": "developer", "permission": { "super_user": false, "structure_user": false, "dev": { "tables": { "dog": { "read": true, "insert": true, "update": true, "delete": false, "attribute_permissions": [ { "attribute_name": "name", "read": true, "insert": true, "update": true } ] } } } } } ``` ### Alter Role *Restricted to `super_user` roles.* * `id` *(required)* — The `id` of the role to alter (from `list_roles`). * `role` *(optional)* — New name for the role. * `permission` *(required)* — Updated permissions object. ``` { "operation": "alter_role", "id": "f92162e2-cd17-450c-aae0-372a76859038", "role": "another_developer", "permission": { "super_user": false, "structure_user": false, "dev": { "tables": { "dog": { "read": true, "insert": true, "update": true, "delete": false, "attribute_permissions": [] } } } } } ``` ### Drop Role *Restricted to `super_user` roles. Roles with associated users cannot be dropped.* * `id` *(required)* — The `id` of the role to drop. ``` { "operation": "drop_role", "id": "developer" } ``` ## Users ### List Users *Restricted to `super_user` roles.* ``` { "operation": "list_users" } ``` ### User Info Returns user data for the currently authenticated user. Available to all roles. ``` { "operation": "user_info" } ``` ### Add User *Restricted to `super_user` roles.* * `role` *(required)* — Role name to assign. * `username` *(required)* — Username. Cannot be changed after creation. * `password` *(required)* — Plain-text password. Harper encrypts it on receipt. * `active` *(required)* — Boolean. If `false`, user cannot access Harper. ``` { "operation": "add_user", "role": "role_name", "username": "hdb_user", "password": "password", "active": true } ``` ### Alter User *Restricted to `super_user` roles.* * `username` *(required)* — Username to modify. * `password` *(optional)* — New password. * `role` *(optional)* — New role name. * `active` *(optional)* — Enable/disable user access. ``` { "operation": "alter_user", "role": "role_name", "username": "hdb_user", "password": "new_password", "active": true } ``` ### Drop User *Restricted to `super_user` roles.* ``` { "operation": "drop_user", "username": "harper" } ``` ## Related * [Overview](/reference/v4/users-and-roles/overview.md) * [Configuration](/reference/v4/users-and-roles/configuration.md) --- # Users & Roles Harper uses a Role-Based Access Control (RBAC) framework to manage access to Harper instances. Each user is assigned a role that determines their permissions to access database resources and run operations. ## Roles Role permissions in Harper are divided into two categories: **Database Manipulation** — CRUD (create, read, update, delete) permissions against database data (tables and attributes). **Database Definition** — Permissions to manage databases, tables, roles, users, and other system settings. These are restricted to the built-in `super_user` role. ### Built-In Roles | Role | Description | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `super_user` | Full access to all operations and methods. The admin role. | | `structure_user` | Access to create and delete databases and tables. Can be set to `true` (all databases) or an array of database names (specific databases only). | ### User-Defined Roles Admins (`super_user` users) can create custom roles with explicit permissions on specific tables and attributes. * Unless a user-defined role has `super_user: true`, all permissions must be defined explicitly. * Any table or database not included in the role's permission set will be inaccessible. * `describe` operations return metadata only for databases, tables, and attributes that the role has CRUD permissions for. ## Permission Structure When creating or altering a role, you define a `permission` object: ``` { "operation": "add_role", "role": "software_developer", "permission": { "super_user": false, "database_name": { "tables": { "table_name1": { "read": true, "insert": true, "update": true, "delete": false, "attribute_permissions": [ { "attribute_name": "attribute1", "read": true, "insert": true, "update": true } ] }, "table_name2": { "read": true, "insert": true, "update": true, "delete": false, "attribute_permissions": [] } } } } } ``` ### Table Permissions Each table entry defines CRUD access: ``` { "table_name": { "read": boolean, // Access to read from this table "insert": boolean, // Access to insert data "update": boolean, // Access to update data "delete": boolean, // Access to delete rows "attribute_permissions": [ { "attribute_name": "attribute_name", "read": boolean, "insert": boolean, "update": boolean // Note: "delete" is not an attribute-level permission } ] } } ``` ### Important Rules **Table-level:** * If a database or table is not included in the permissions, the role has no access to it. * If a table-level CRUD permission is `false`, setting the same CRUD permission to `true` on an attribute returns an error. **Attribute-level:** * If `attribute_permissions` is a non-empty array, only the listed attributes are accessible (plus the table's hash attribute — see below). * If `attribute_permissions` is empty (`[]`), attribute access follows the table-level CRUD permissions. * If any non-hash attribute is given CRUD access, the table's `hash_attribute` (primary key) automatically receives the same access, even if not explicitly listed. * Any attribute not explicitly listed in a non-empty `attribute_permissions` array has no access. * `DELETE` is not an attribute-level permission. Deleting rows is controlled at the table level. * The `__createdtime__` and `__updatedtime__` attributes managed by Harper can have `read` permissions set; other attribute-level permissions for these fields are ignored. ## Role-Based Operation Restrictions The following table shows which operations are restricted to `super_user` roles. Non-`super_user` roles are also restricted within their accessible operations by their CRUD permission set. ### Databases and Tables | Operation | Restricted to Super User | | ------------------- | ------------------------ | | `describe_all` | | | `describe_database` | | | `describe_table` | | | `create_database` | X | | `drop_database` | X | | `create_table` | X | | `drop_table` | X | | `create_attribute` | | | `drop_attribute` | X | ### NoSQL Operations | Operation | Restricted to Super User | | ---------------------- | ------------------------ | | `insert` | | | `update` | | | `upsert` | | | `delete` | | | `search_by_hash` | | | `search_by_value` | | | `search_by_conditions` | | ### SQL Operations | Operation | Restricted to Super User | | --------- | ------------------------ | | `select` | | | `insert` | | | `update` | | | `delete` | | ### Bulk Operations | Operation | Restricted to Super User | | ---------------- | ------------------------ | | `csv_data_load` | | | `csv_file_load` | | | `csv_url_load` | | | `import_from_s3` | | ### Users and Roles | Operation | Restricted to Super User | | ------------ | ------------------------ | | `list_roles` | X | | `add_role` | X | | `alter_role` | X | | `drop_role` | X | | `list_users` | X | | `user_info` | | | `add_user` | X | | `alter_user` | X | | `drop_user` | X | ### Clustering | Operation | Restricted to Super User | | ----------------------- | ------------------------ | | `cluster_set_routes` | X | | `cluster_get_routes` | X | | `cluster_delete_routes` | X | | `add_node` | X | | `update_node` | X | | `cluster_status` | X | | `remove_node` | X | | `configure_cluster` | X | ### Components | Operation | Restricted to Super User | | -------------------- | ------------------------ | | `get_components` | X | | `get_component_file` | X | | `set_component_file` | X | | `drop_component` | X | | `add_component` | X | | `package_component` | X | | `deploy_component` | X | ### Registration | Operation | Restricted to Super User | | ------------------- | ------------------------ | | `registration_info` | | | `get_fingerprint` | X | | `set_license` | X | ### Jobs | Operation | Restricted to Super User | | --------------------------- | ------------------------ | | `get_job` | | | `search_jobs_by_start_date` | X | ### Logs | Operation | Restricted to Super User | | -------------------------------- | ------------------------ | | `read_log` | X | | `read_transaction_log` | X | | `delete_transaction_logs_before` | X | | `read_audit_log` | X | | `delete_audit_logs_before` | X | ### Utilities | Operation | Restricted to Super User | | ----------------------- | ------------------------ | | `delete_records_before` | X | | `export_local` | X | | `export_to_s3` | X | | `system_information` | X | | `restart` | X | | `restart_service` | X | | `get_configuration` | X | ### Token Authentication | Operation | Restricted to Super User | | ------------------------------ | ------------------------ | | `create_authentication_tokens` | | | `refresh_operation_token` | | ## Troubleshooting: "Must execute as User" If you see the error `Error: Must execute as <>`, it means Harper was installed as a specific OS user and must be run by that same user. Harper stores files natively on the operating system and only allows the Harper executable to be run by a single user — this prevents file permission issues and keeps the installation secure. To resolve: run Harper with the same OS user account used during installation. ## Related * [Configuration](/reference/v4/users-and-roles/configuration.md) * [Operations](/reference/v4/users-and-roles/operations.md) --- # Harper v5 Reference Complete technical reference for Harper v5. Each section covers a core feature or subsystem — configuration options, APIs, and operational details. For concept introductions, tutorials, and guides, see the [Learn](/learn.md) section. ## Sections ### Data & Application | Section | Description | | -------------------------------------------------- | ----------------------------------------------------------------------- | | [Database](/reference/v5/database/overview.md) | Schema system, storage, indexing, transactions, and the database JS API | | [Resources](/reference/v5/resources/overview.md) | Custom resource classes, the Resource API, and query optimization | | [Components](/reference/v5/components/overview.md) | Applications, plugins, and the JS environment | ### Access & Security | Section | Description | | ---------------------------------------------------------- | -------------------------------------------------------------------------- | | [REST](/reference/v5/rest/overview.md) | Auto-REST interface, querying, content types, headers, WebSockets, and SSE | | [HTTP](/reference/v5/http/overview.md) | HTTP server configuration, TLS, and the `server` API | | [MCP](/reference/v5/mcp/overview.md) | Built-in Model Context Protocol server, profiles, CLI, and migration | | [Security](/reference/v5/security/overview.md) | Authentication mechanisms, certificates, and CORS/SSL configuration | | [Users & Roles](/reference/v5/users-and-roles/overview.md) | RBAC, roles configuration, and user management operations | ### Setup & Operation | Section | Description | | ---------------------------------------------------------- | ------------------------------------------------------------- | | [CLI](/reference/v5/cli/overview.md) | All CLI commands, Operations API commands, and authentication | | [Configuration](/reference/v5/configuration/overview.md) | `harper-config.yaml` options and configuration operations | | [Operations API](/reference/v5/operations-api/overview.md) | Full index of all Operations API operations | ### Features | Section | Description | | ------------------------------------------------------------------------ | ------------------------------------------------------------------ | | [Logging](/reference/v5/logging/overview.md) | Log configuration, the `logger` API, and log management operations | | [Analytics](/reference/v5/analytics/overview.md) | Resource and storage analytics, system tables | | [MQTT](/reference/v5/mqtt/overview.md) | MQTT broker configuration and usage | | [Static Files](/reference/v5/static-files/overview.md) | Static file serving via the `static` plugin | | [Environment Variables](/reference/v5/environment-variables/overview.md) | Environment variable loading via the `loadEnv` plugin | | [Replication](/reference/v5/replication/overview.md) | Native replication, clustering, and sharding | | [GraphQL Querying](/reference/v5/graphql-querying/overview.md) | Experimental GraphQL support | | [Studio](/reference/v5/studio/overview.md) | Local Studio UI configuration and access | | [Fastify Routes](/reference/v5/fastify-routes/overview.md) | Fastify route definitions (discouraged in favor of components) | ### Legacy | Section | Description | | ------------------------------------------------------------ | ------------------------------------------------------------------------ | | [Harper Cloud](/reference/v5/legacy/cloud.md) | Legacy Harper Cloud documentation — see Fabric for current cloud hosting | | [Custom Functions](/reference/v5/legacy/custom-functions.md) | Deprecated predecessor to Components | --- # Analytics Operations Operations for querying Harper analytics data. All operations require `superuser` permission. Analytics data can also be queried directly via `search_by_conditions` on the `hdb_raw_analytics` and `hdb_analytics` tables in the `system` database — see [Analytics Overview](/reference/v5/analytics/overview.md) for details on the table structure. *** ## `list_metrics` Returns the list of available metric names that can be queried with `get_analytics`. ### Parameters | Parameter | Required | Type | Description | | -------------- | -------- | --------- | ------------------------------------------------------------------------ | | `operation` | Yes | string | Must be `"list_metrics"` | | `metric_types` | No | string\[] | Filter by type: `"builtin"`, `"custom"`, or both. Default: `["builtin"]` | ### Request ``` { "operation": "list_metrics", "metric_types": ["custom", "builtin"] } ``` ### Response ``` ["resource-usage", "table-size", "database-size", "main-thread-utilization", "utilization", "storage-volume"] ``` *** ## `describe_metric` Returns the structure and available attributes for a specific metric. ### Parameters | Parameter | Required | Type | Description | | ----------- | -------- | ------ | ------------------------------ | | `operation` | Yes | string | Must be `"describe_metric"` | | `metric` | Yes | string | Name of the metric to describe | ### Request ``` { "operation": "describe_metric", "metric": "resource-usage" } ``` ### Response ``` { "attributes": [ { "name": "id", "type": "number" }, { "name": "metric", "type": "string" }, { "name": "userCPUTime", "type": "number" }, { "name": "systemCPUTime", "type": "number" }, { "name": "node", "type": "string" } ] } ``` *** ## `get_analytics` Queries analytics data for a specific metric over a time range. ### Parameters | Parameter | Required | Type | Description | | ---------------- | -------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `operation` | Yes | string | Must be `"get_analytics"` | | `metric` | Yes | string | Metric name — use `list_metrics` to get valid values | | `start_time` | No | number | Start of time range as Unix timestamp in milliseconds | | `end_time` | No | number | End of time range as Unix timestamp in milliseconds | | `get_attributes` | No | string\[] | Attributes to include in each result. If omitted, all attributes are returned | | `conditions` | No | object\[] | Additional filter conditions. Same format as [`search_by_conditions`](/reference/v5/operations-api/operations.md#nosql-operations) | ### Request ``` { "operation": "get_analytics", "metric": "resource-usage", "start_time": 1769198332754, "end_time": 1769198532754, "get_attributes": ["id", "metric", "userCPUTime", "systemCPUTime"], "conditions": [ { "attribute": "node", "operator": "equals", "value": "node1.example.com" } ] } ``` ### Response ``` [ { "id": "12345", "metric": "resource-usage", "userCPUTime": 100, "systemCPUTime": 50 }, { "id": "67890", "metric": "resource-usage", "userCPUTime": 150, "systemCPUTime": 75 } ] ``` ### Read behavior `get_analytics` queries read against the latest committed data without holding a consistent read snapshot open for the duration of the query. This minimizes the impact of long-running analytics scans on the rest of the system — a query never pins a snapshot that would otherwise block storage compaction for its duration. The trade-off is that results are not guaranteed to reflect a single point in time: rows written while the query is iterating may be included. ## Related * [Analytics Overview](/reference/v5/analytics/overview.md) * [Operations API Overview](/reference/v5/operations-api/overview.md) --- # Analytics *Added in: v4.5.0* (resource and storage analytics) Harper collects real-time telemetry and statistics across all operations, URL endpoints, and messaging topics. This data can be used to monitor server health, understand traffic and usage patterns, identify resource-intensive queries, and inform scaling decisions. ## Storage Tables Analytics data is stored in two system tables in the `system` database: | Table | Description | | ------------------- | ------------------------------------------------------------------------------------------- | | `hdb_raw_analytics` | Per-second raw entries recorded by each thread. One record per second per active thread. | | `hdb_analytics` | Aggregate entries recorded once per minute, summarizing all per-second data across threads. | Both tables require `superuser` permission to query. ## Raw Analytics (`hdb_raw_analytics`) Raw entries are recorded once per second (when there is activity) by each thread. Each record captures all activity in the last second along with system resource information. Records use the timestamp in milliseconds since epoch as the primary key. Query raw analytics using `search_by_conditions` on the `hdb_raw_analytics` table. The example below fetches 10 seconds of raw entries: ``` POST http://localhost:9925 Content-Type: application/json { "operation": "search_by_conditions", "schema": "system", "table": "hdb_raw_analytics", "conditions": [{ "search_attribute": "id", "search_type": "between", "search_value": [1688594000000, 1688594010000] }] } ``` Example raw entry: ``` { "time": 1688594390708, "period": 1000.8336279988289, "metrics": [ { "metric": "bytes-sent", "path": "search_by_conditions", "type": "operation", "median": 202, "mean": 202, "p95": 202, "p90": 202, "count": 1 }, { "metric": "memory", "threadId": 2, "rss": 1492664320, "heapTotal": 124596224, "heapUsed": 119563120, "external": 3469790, "arrayBuffers": 798721 }, { "metric": "utilization", "idle": 138227.52767700003, "active": 70.5066209952347, "utilization": 0.0005098165086230495 } ], "threadId": 2, "totalBytesProcessed": 12182820, "id": 1688594390708.6853 } ``` ## Aggregate Analytics (`hdb_analytics`) Aggregate entries are recorded once per minute, combining per-second raw entries from all threads into a single summary record. Use `search_by_conditions` on the `hdb_analytics` table with a broader time range: ``` POST http://localhost:9925 Content-Type: application/json { "operation": "search_by_conditions", "schema": "system", "table": "hdb_analytics", "conditions": [{ "search_attribute": "id", "search_type": "between", "search_value": [1688194100000, 1688594990000] }] } ``` Example aggregate entry: ``` { "period": 60000, "metric": "bytes-sent", "method": "connack", "type": "mqtt", "median": 4, "mean": 4, "p95": 4, "p90": 4, "count": 1, "id": 1688589569646, "time": 1688589569646 } ``` ## Standard Metrics Harper automatically tracks the following metrics for all services. Applications can also define custom metrics via [`server.recordAnalytics()`](/reference/v5/http/api.md#serverrecordanalyticsvalue-metric-path-method-type). ### HTTP Metrics | `metric` | `path` | `method` | `type` | Unit | Description | | ------------------ | ------------- | -------------- | --------------------------- | ----- | ---------------------------------------- | | `duration` | resource path | request method | `cache-hit` or `cache-miss` | ms | Duration of request handler | | `duration` | route path | request method | `fastify-route` | ms | Duration of Fastify route handler | | `duration` | operation | | `operation` | ms | Duration of Operations API operation | | `success` | resource path | request method | | % | Percentage of successful requests | | `success` | route path | request method | `fastify-route` | % | | | `success` | operation | | `operation` | % | | | `bytes-sent` | resource path | request method | | bytes | Response bytes sent | | `bytes-sent` | route path | request method | `fastify-route` | bytes | | | `bytes-sent` | operation | | `operation` | bytes | | | `transfer` | resource path | request method | `operation` | ms | Duration of response transfer | | `transfer` | route path | request method | `fastify-route` | ms | | | `transfer` | operation | | `operation` | ms | | | `socket-routed` | | | | % | Percentage of sockets immediately routed | | `tls-handshake` | | | | ms | TLS handshake duration | | `tls-reused` | | | | % | Percentage of TLS sessions reused | | `cache-hit` | table name | | | % | Percentage of cache hits | | `cache-resolution` | table name | | | ms | Duration of resolving uncached entries | ### MQTT / WebSocket Metrics | `metric` | `path` | `method` | `type` | Unit | Description | | ------------------ | ------ | ------------ | ------ | ----- | ------------------------------------------------ | | `mqtt-connections` | | | | count | Number of open direct MQTT connections | | `ws-connections` | | | | count | Number of open WebSocket connections | | `connection` | `mqtt` | `connect` | | % | Percentage of successful direct MQTT connections | | `connection` | `mqtt` | `disconnect` | | % | Percentage of explicit direct MQTT disconnects | | `connection` | `ws` | `connect` | | % | Percentage of successful WebSocket connections | | `connection` | `ws` | `disconnect` | | % | Percentage of explicit WebSocket disconnects | | `bytes-sent` | topic | mqtt command | `mqtt` | bytes | Bytes sent for a given MQTT command and topic | ### Replication Metrics | `metric` | `path` | `method` | `type` | Unit | Description | | --------------------- | ------------------- | ------------- | --------- | ----- | ---------------------------------------------------------- | | `bytes-sent` | node.database | `replication` | `egress` | bytes | Bytes sent for replication | | `bytes-sent` | node.database | `replication` | `blob` | bytes | Bytes sent for blob replication | | `bytes-received` | node.database | `replication` | `ingress` | bytes | Bytes received for replication | | `bytes-received` | node.database | `replication` | `blob` | bytes | Bytes received for blob replication | | `replication-latency` | node.database.table | | `ingest` | ms | Time difference from source commit timestamp to local time | ### Resource Usage Metrics | `metric` | Key attributes | Other | Unit | Description | | ------------------------- | ------------------------------------------------------------------------------------------------ | ------------------- | ------- | --------------------------------------------------------------------------------- | | `database-size` | `size`, `used`, `free`, `audit` | `database` | bytes | Database file size breakdown | | `main-thread-utilization` | `idle`, `active`, `taskQueueLatency`, `rss`, `heapTotal`, `heapUsed`, `external`, `arrayBuffers` | `time` | various | Main thread resource usage: idle/active time, queue latency, and memory breakdown | | `resource-usage` | (see below) | | various | Node.js process resource usage (see [resource-usage](#resource-usage-metric)) | | `storage-volume` | `available`, `free`, `size` | `database` | bytes | Storage volume size breakdown | | `table-size` | `size` | `database`, `table` | bytes | Table file size | | `utilization` | | | % | Percentage of time the worker thread was processing requests | #### `resource-usage` Metric Includes everything returned by Node.js [`process.resourceUsage()`](https://nodejs.org/api/process.html#processresourceusage) (with `userCPUTime` and `systemCPUTime` converted to milliseconds), plus: | Field | Unit | Description | | ---------------- | ---- | ------------------------------------------- | | `time` | ms | Unix timestamp when the metric was recorded | | `period` | ms | Duration of the measurement period | | `cpuUtilization` | % | CPU utilization (user + system combined) | ## Custom Metrics Applications can record custom metrics using the `server.recordAnalytics()` API. See [HTTP API](/reference/v5/http/api.md) for details. ## Analytics Configuration The `analytics` configuration section controls aggregation, replication, and storage-volume sampling. All options are optional. ``` analytics: aggregatePeriod: 60 storageInterval: 10 replicate: false logging: level: info ``` ### `analytics.aggregatePeriod` Type: `number` (seconds) Default: `60` How frequently Harper aggregates raw per-second entries into the `hdb_analytics` summary table. Lowering this gives higher-resolution aggregate data at the cost of more frequent aggregation work and more rows in `hdb_analytics`. ### `analytics.storageInterval` Type: `number` Default: `10` Number of aggregation cycles between disk-volume measurements. With the default `aggregatePeriod` of `60` and `storageInterval` of `10`, Harper records `database-size`, `table-size`, and `storage-volume` metrics every 10 minutes. Set to `0` to disable storage-volume sampling entirely — useful when running on systems where `statfs` is expensive or unavailable (e.g., some FUSE mounts). ### `analytics.replicate` Type: `boolean` Default: `false` When enabled, aggregate analytics entries are replicated across the cluster so a single peer can answer aggregate queries for the whole topology. Raw per-thread entries (`hdb_raw_analytics`) are always node-local. Enable when running a centralized analytics consumer; leave disabled in large clusters to avoid replication overhead for high-cardinality metrics. ### `analytics.logging` Type: `object` Per-subsystem logging override for the analytics writer. See [Logging Configuration — `analytics.logging`](/reference/v5/logging/configuration.md#analyticslogging). ## Related * [Analytics Operations](/reference/v5/analytics/operations.md) * [HTTP API](/reference/v5/http/api.md) * [Logging Configuration](/reference/v5/logging/configuration.md) * [Configuration Overview](/reference/v5/configuration/overview.md) --- # CLI Authentication The Harper CLI handles authentication differently for local and remote operations. ## Local Operations Available since: v4.1.0 For local operations (operations executed on the same machine where Harper is installed), the CLI communicates with Harper via Unix domain sockets instead of HTTP. Domain socket requests are automatically authenticated as the superuser, so no additional authentication parameters are required. **Example**: ``` # No authentication needed for local operations harper describe_database database=dev harper get_components harper set_configuration logging_level=info ``` When no `target` parameter is specified, the CLI defaults to using the local domain socket connection, providing secure, authenticated access to the local Harper instance. ## Remote Operations Available since: v4.1.0; expanded in: v4.3.0 For remote operations (operations executed on a remote Harper instance via the `target` parameter), you must provide authentication credentials. ### Authentication Methods #### Method 1: Persistent Login (Recommended for Local Development) Available since: v5.1.0 Use `harper login` to store authentication tokens for a specific target. This is the most convenient method for local development as it removes the need to pass credentials with every command. ``` # Log in once harper login https://server.com:9925 # Provide username and password when prompted # Subsequently execute operations without credentials harper describe_database database=dev target=https://server.com:9925 harper deploy target=https://server.com:9925 ``` When you are finished, you can log out to remove the stored token: ``` harper logout https://server.com:9925 ``` **Benefits**: * Credentials are not stored in command history for every operation * Simplifies frequent remote operations * No need to maintain environment variables in multiple terminal sessions #### Method 2: Environment Variables (Recommended for CI/CD) The CLI supports loading environment variables from your shell environment (or optionally from a `.env` file in the current directory). This is the recommended method for CI/CD pipelines and for pre-populating the `target` parameter for specific projects. **Supported Variables**: * `HARPER_CLI_TARGET` (or `CLI_TARGET`) - Sets the default `target` for CLI commands. * `HARPER_CLI_USERNAME` (or `CLI_TARGET_USERNAME`) - Harper admin username for the target. * `HARPER_CLI_PASSWORD` (or `CLI_TARGET_PASSWORD`) - Harper admin password for the target. **Example `.env` file**: ``` HARPER_CLI_TARGET=https://example.com:9925 HARPER_CLI_USERNAME=HDB_ADMIN HARPER_CLI_PASSWORD=password ``` **Manual Environment Variables**: Set these variables in your shell to avoid exposing credentials in command history: ``` export HARPER_CLI_USERNAME=HDB_ADMIN export HARPER_CLI_PASSWORD=password ``` **Benefits**: * Credentials not visible in command history * More secure for scripting and CI/CD systems * Can be set once per session or project directory * Automatically populated by `harper login` **Automatic `.env` Updates**: When you run `harper login `, the CLI will automatically update your `.env` file in your current directory and set `HARPER_CLI_TARGET` to the specified URL. ``` # Automatically sets HARPER_CLI_TARGET in .env harper login https://my-project.harperdb.cloud ``` Then you can run commands without specifying the `target` or credentials (if they are also in `.env` or exported): ``` # Respects HARPER_CLI_TARGET from .env harper deploy harper describe_database database=dev harper get_components harper logout ``` **Example Script**: ``` #!/bin/bash # Set credentials from secure environment export HARPER_CLI_USERNAME=HDB_ADMIN export HARPER_CLI_PASSWORD=$SECURE_PASSWORD # from secret manager # Execute operations without passing credentials or target (if set) harper deploy target=https://prod-server.com:9925 replicated=true harper restart target=https://prod-server.com:9925 replicated=true ``` #### Method 3: Command Parameters Provide credentials directly as command parameters: ``` harper describe_database \ database=dev \ target=https://server.com:9925 \ username=HDB_ADMIN \ password=password ``` **Parameters**: * `username=` - Harper admin username * `password=` - Harper admin password **Cautions**: * Credentials visible in command history * Less secure for production environments * Exposed in process listings * Not recommended for scripts ### Target Parameter The `target` parameter specifies the full HTTP/HTTPS URL of the remote Harper instance: **Format**: `target=://:` **Examples**: ``` # HTTPS on default operations API port target=https://server.example.com:9925 # HTTP (not recommended for production) target=http://localhost:9925 # Custom port target=https://server.example.com:8080 ``` ## Security Best Practices ### 1. Use Environment Variables Always use environment variables for credentials in scripts and automation: ``` export HARPER_CLI_USERNAME=HDB_ADMIN export HARPER_CLI_PASSWORD=$SECURE_PASSWORD ``` ### 2. Use HTTPS Always use HTTPS for remote operations to encrypt credentials in transit: ``` # Good target=https://server.com:9925 # Bad - credentials sent in plain text target=http://server.com:9925 ``` ### 3. Manage Secrets Securely Store credentials in secure secret management systems: * Environment variables from secret managers (AWS Secrets Manager, HashiCorp Vault, etc.) * CI/CD secret storage (GitHub Secrets, GitLab CI Variables, etc.) * Operating system credential stores **Example with AWS Secrets Manager**: ``` #!/bin/bash # Retrieve credentials from AWS Secrets Manager export HARPER_CLI_USERNAME=$(aws secretsmanager get-secret-value \ --secret-id harper-admin-user \ --query SecretString \ --output text) export HARPER_CLI_PASSWORD=$(aws secretsmanager get-secret-value \ --secret-id harper-admin-password \ --query SecretString \ --output text) # Execute operations harper deploy target=https://prod.example.com:9925 ``` ### 4. Use Least Privilege Create dedicated users with minimal required permissions for CLI operations instead of using the main admin account. See [Users and Roles](/reference/v5/users-and-roles/overview.md) for more information. ### 5. Rotate Credentials Regularly rotate credentials, especially for automated systems and CI/CD pipelines. ### 6. Audit Access Monitor and audit CLI operations, especially for production environments. See [Logging](/reference/v5/logging/overview.md) for more information on logging. ## Troubleshooting ### Authentication Failures If you receive authentication errors: 1. **Verify credentials are correct**: * Check username and password * Ensure no extra whitespace 2. **Verify the target URL**: * Ensure the URL is correct and reachable * Check the port number * Verify HTTPS/HTTP protocol 3. **Check network connectivity**: ``` curl https://server.com:9925 ``` 4. **Verify user permissions**: * Ensure the user has permission to execute the operation * Check user roles and permissions ### Environment Variable Issues If environment variables aren't working: 1. **Verify variables are set**: ``` echo $HARPER_CLI_USERNAME echo $HARPER_CLI_PASSWORD ``` 2. **Export variables**: Ensure you used `export`, not just assignment: ``` # Wrong - variable only available in current shell HARPER_CLI_USERNAME=admin # Correct - variable available to child processes export HARPER_CLI_USERNAME=admin ``` 3. **Check variable scope**: * Variables must be exported before running commands * Variables set in one terminal don't affect other terminals ## See Also * [CLI Overview](/reference/v5/cli/overview.md) - General CLI information * [CLI Commands](/reference/v5/cli/commands.md) - Core CLI commands * [Operations API Commands](/reference/v5/cli/operations-api-commands.md) - Operations available through CLI * [Security Overview](/reference/v5/security/overview.md) - Harper security features * [Users and Roles](/reference/v5/users-and-roles/overview.md) - User management --- # CLI Commands This page documents the core Harper CLI commands for managing Harper instances. For Operations API commands available through the CLI, see [Operations API Commands](/reference/v5/cli/operations-api-commands.md). ## Process Management Commands ### `harper` *Added in: v4.1.0* Run Harper in the foreground as a standard process. This is the recommended way to run Harper. ``` harper ``` When you run `harper`: * If Harper is not installed, it will guide you through the installation process * Once installed, it runs Harper in the foreground as a standard process, compatible with systemd, Docker, and other process management tools **First-Time Installation**: If Harper is not installed, you can provide configuration parameters via environment variables or command line arguments: **Using Environment Variables**: ``` # Minimum required parameters for no additional CLI prompts export TC_AGREEMENT=yes export HDB_ADMIN_USERNAME=HDB_ADMIN export HDB_ADMIN_PASSWORD=password export ROOTPATH=/hdb/ harper ``` note If you specify `DEFAULT_MODE=dev` you will also need to specify the `REPLICATION_HOSTNAME=localhost` **Using Command Line Arguments**: ``` # Minimum required parameters for no additional CLI prompts harper \ --TC_AGREEMENT=yes \ --HDB_ADMIN_USERNAME=HDB_ADMIN \ --HDB_ADMIN_PASSWORD=password \ --ROOTPATH='/hdb' ``` **Note**: When used in conjunction, command line arguments override environment variables. See [Configuration](/reference/v5/configuration/overview.md) for a full list of configuration parameters. info For more information on installation, see [Getting Started / Install and Connect Harper](/learn/getting-started/install-and-connect-harper.md). ### `harper run` *Added in: v4.2.0* Run a Harper application from any location as a foreground, standard process (similar to `harper`). ``` harper run /path/to/app ``` This command runs Harper with the specified application directory without automatic reloading or dev-specific features. ### `harper dev` *Added in: v4.2.0* Run Harper in development mode from a specified directory with automatic reloading. Recommended for local application development. Operates similar to `harper` and `harper run`. ``` harper dev /path/to/app ``` **Features**: * Pushes logs to standard streams automatically * Uses a single thread for simpler debugging * Auto-restart on file changes ### `harper restart` Available since: v4.1.0 Restart a running Harper instance regardless if its a foreground (`harper`, `harper run`, or `harper dev`) or background (`harper start`) process. ``` harper restart ``` ### `harper start` Available since: v4.1.0 Start Harper in background (daemon mode). ``` harper start ``` After installation, this command launches Harper as a background process. Remember that the Harper PID is available in a `hdb.pid` file within the installation directory. ### `harper stop` Available since: v4.1.0 Stop a running Harper instance. ``` harper stop ``` ## Installation Commands ### `harper install` Available since: v4.1.0 Install Harper with interactive prompts or automated configuration. ``` harper install ``` The `harper install` command operates exactly like the [`harper`](#harper) command, but exits as soon as the installation completes. See the [`harper`](#harper) command documentation above for details on providing configuration parameters via environment variables or command line arguments. **Note**: We recommend using `harper` instead of `harper install` as it provides a consistent workflow for both installation and running Harper. ### `harper login` Available since: v5.0.0 Log in to a Harper instance to store authentication tokens locally. Once logged in, subsequent commands targeting this instance (via `target`) will automatically use the stored token. The CLI also supports `.env` files. When you log in, the `HARPER_CLI_TARGET` environment variable will be automatically added to a `.env` file in your current directory if it exists. This allows you to omit the `target` parameter in subsequent commands within that directory. ``` harper login ``` **Optional Parameters**: * `` - The URL of the Harper instance to log in to. **Prompts**: You'll be asked to type in the following information: * `` - If a URL parameter is not provided, you'll be prompted to enter the URL of the Harper instance to log in to. * `` - Harper admin username. * `` - Harper admin password. ### `harper logout` Available since: v5.0.0 Log out of a Harper instance and remove the stored authentication token. ``` harper logout ``` **Optional Parameters**: * `` - The URL of the Harper instance to log out from. If none is provided, you'll be signed out of all instances. ## Information Commands ### `harper version` Available since: v4.1.0 Display the installed Harper version. ``` harper version ``` **Example Output**: ``` 4.7.0 ``` ### `harper status` Available since: v4.1.0 Display the status of Harper and clustering. ``` harper status ``` Shows: * Harper process status * Clustering network status * Replication statuses In Harper versions where NATS is supported, this command also shows the clustering hub and leaf processes too. ### `harper help` Available since: v4.1.0 Display all available Harper CLI commands with brief descriptions. ``` harper help ``` ## Maintenance Commands ### `harper renew-certs` Available since: v4.1.0 Renew Harper-generated self-signed certificates. ``` harper renew-certs ``` This command regenerates the self-signed SSL/TLS certificates used by Harper. ### `harper copy-db` Available since: v4.1.0 Copy a Harper database with compaction to eliminate free-space and fragmentation. ``` harper copy-db ``` **Parameters**: * `` - Name of the source database * `` - Full path to the target database file **Example**: ``` harper copy-db data /home/user/hdb/database/copy.mdb ``` This copies the default `data` database to a new location with compaction applied. **Use Cases**: * Database optimization * Eliminating fragmentation * Creating compacted backups * Reclaiming free space See also: [Database Compaction](/reference/v5/database/compaction.md) for more information. #### How Backups Work Harper uses a transactional commit process that ensures data on disk is always transactionally consistent with storage. This means Harper maintains database integrity in the event of a crash and allows you to use standard volume snapshot tools to make backups. **Backup Process**: Database files are stored in the `hdb/database` directory. As long as the snapshot is an atomic snapshot of these database files, the data can be copied/moved back into the database directory to restore a previous backup (with Harper shut down), and database integrity will be preserved. **Important Notes**: * **Atomic Snapshots**: Use volume snapshot tools that create atomic snapshots * **Not Safe**: Simply copying an in-use database file using `cp` is **not reliable** * Progressive reads occur at different points in time * Results in an unreliable copy that likely won't be usable * **Safe Copying**: Standard file copying is only reliable for database files that are **not in use** **Recommended Backup Tools**: * LVM snapshots * ZFS snapshots * BTRFS snapshots * Cloud provider volume snapshots (AWS EBS, Azure Disk, GCP Persistent Disk) * Enterprise backup solutions with snapshot capabilities ## Remote Operations The CLI supports executing commands on remote Harper instances. For details, see [CLI Overview - Remote Operations](/reference/v5/cli/overview.md#remote-operations). ## See Also * [CLI Overview](/reference/v5/cli/overview.md) - General CLI information * [Operations API Commands](/reference/v5/cli/operations-api-commands.md) - Operations available through CLI * [CLI Authentication](/reference/v5/cli/authentication.md) - Authentication mechanisms * [Configuration](/reference/v5/configuration/overview.md) - Configuration parameters for installation * [Database Compaction](/reference/v5/database/compaction.md) - More on database compaction --- # Operations API Commands *Added in: v4.3.0* The Harper CLI supports executing operations from the [Operations API](/reference/v5/operations-api/overview.md) directly from the command line. This enables powerful automation and scripting capabilities. ## General Syntax ``` harper = ``` **Output Format**: * Default: YAML * JSON: Pass `json=true` as a parameter ## Supported Operations The following operations are available through the CLI. Operations that require complex nested parameters or object structures are not supported via CLI and must be executed through the HTTP API. ### Complete Operations List note This is just a brief overview of all operations available as CLI commands. Review the respective operation documentation for more information on available arguments and expected behavior. Keep in mind that all operations options are converted to CLI arguments in the same way (using `snake_case`). | Operation | Description | Category | Available Since | | -------------------------------- | ------------------------------------- | --------------------------------------------------------------------------------- | --------------- | | `describe_table` | Describe table structure and metadata | [Database](/reference/v5/operations-api/operations.md#databases--tables) | v4.3.0 | | `describe_all` | Describe all databases and tables | [Database](/reference/v5/operations-api/operations.md#databases--tables) | v4.3.0 | | `describe_database` | Describe database structure | [Database](/reference/v5/operations-api/operations.md#databases--tables) | v4.3.0 | | `create_database` | Create a new database | [Database](/reference/v5/operations-api/operations.md#databases--tables) | v4.3.0 | | `drop_database` | Delete a database | [Database](/reference/v5/operations-api/operations.md#databases--tables) | v4.3.0 | | `create_table` | Create a new table | [Database](/reference/v5/operations-api/operations.md#databases--tables) | v4.3.0 | | `drop_table` | Delete a table | [Database](/reference/v5/operations-api/operations.md#databases--tables) | v4.3.0 | | `create_attribute` | Create a table attribute | [Database](/reference/v5/operations-api/operations.md#databases--tables) | v4.3.0 | | `drop_attribute` | Delete a table attribute | [Database](/reference/v5/operations-api/operations.md#databases--tables) | v4.3.0 | | `search_by_id` | Search records by ID | [Data](/reference/v5/operations-api/operations.md#nosql-operations) | v4.3.0 | | `search_by_value` | Search records by attribute value | [Data](/reference/v5/operations-api/operations.md#nosql-operations) | v4.3.0 | | `insert` | Insert new records | [Data](/reference/v5/operations-api/operations.md#nosql-operations) | v4.4.9 | | `update` | Update existing records | [Data](/reference/v5/operations-api/operations.md#nosql-operations) | v4.4.9 | | `upsert` | Insert or update records | [Data](/reference/v5/operations-api/operations.md#nosql-operations) | v4.4.9 | | `delete` | Delete records | [Data](/reference/v5/operations-api/operations.md#nosql-operations) | v4.3.0 | | `sql` | Execute SQL queries | [Data](/reference/v5/operations-api/operations.md#nosql-operations) | v4.3.0 | | `csv_file_load` | Load data from CSV file | [Data](/reference/v5/operations-api/operations.md#nosql-operations) | v4.3.0 | | `csv_url_load` | Load data from CSV URL | [Data](/reference/v5/operations-api/operations.md#nosql-operations) | v4.3.0 | | `list_users` | List all users | [Security](/reference/v5/operations-api/operations.md#certificate-management) | v4.3.0 | | `add_user` | Create a new user | [Security](/reference/v5/operations-api/operations.md#certificate-management) | v4.3.0 | | `alter_user` | Modify user properties | [Security](/reference/v5/operations-api/operations.md#certificate-management) | v4.3.0 | | `drop_user` | Delete a user | [Security](/reference/v5/operations-api/operations.md#certificate-management) | v4.3.0 | | `list_roles` | List all roles | [Security](/reference/v5/operations-api/operations.md#certificate-management) | v4.3.0 | | `drop_role` | Delete a role | [Security](/reference/v5/operations-api/operations.md#certificate-management) | v4.3.0 | | `create_csr` | Create certificate signing request | [Security](/reference/v5/operations-api/operations.md#certificate-management) | v4.4.0 | | `sign_certificate` | Sign a certificate | [Security](/reference/v5/operations-api/operations.md#certificate-management) | v4.4.0 | | `list_certificates` | List SSL/TLS certificates | [Security](/reference/v5/operations-api/operations.md#certificate-management) | v4.4.0 | | `add_certificate` | Add SSL/TLS certificate | [Security](/reference/v5/operations-api/operations.md#certificate-management) | v4.4.0 | | `remove_certificate` | Remove SSL/TLS certificate | [Security](/reference/v5/operations-api/operations.md#certificate-management) | v4.4.0 | | `add_ssh_key` | Add SSH key | [Security](/reference/v5/operations-api/operations.md#certificate-management) | v4.4.0 | | `get_ssh_key` | Get SSH key | [Security](/reference/v5/operations-api/operations.md#certificate-management) | v4.7.2 | | `update_ssh_key` | Update SSH key | [Security](/reference/v5/operations-api/operations.md#certificate-management) | v4.4.0 | | `delete_ssh_key` | Delete SSH key | [Security](/reference/v5/operations-api/operations.md#certificate-management) | v4.4.0 | | `list_ssh_keys` | List all SSH keys | [Security](/reference/v5/operations-api/operations.md#certificate-management) | v4.4.0 | | `set_ssh_known_hosts` | Set SSH known hosts | [Security](/reference/v5/operations-api/operations.md#certificate-management) | v4.4.0 | | `get_ssh_known_hosts` | Get SSH known hosts | [Security](/reference/v5/operations-api/operations.md#certificate-management) | v4.4.0 | | `cluster_get_routes` | Get cluster routing information | [Clustering](/reference/v5/operations-api/operations.md#replication--clustering) | v4.3.0 | | `cluster_network` | Get cluster network status | [Clustering](/reference/v5/operations-api/operations.md#replication--clustering) | v4.3.0 | | `cluster_status` | Get cluster status | [Clustering](/reference/v5/operations-api/operations.md#replication--clustering) | v4.3.0 | | `remove_node` | Remove node from cluster | [Clustering](/reference/v5/operations-api/operations.md#replication--clustering) | v4.3.0 | | `add_component` | Add a component | [Components](/reference/v5/operations-api/operations.md#components) | v4.3.0 | | `deploy_component` | Deploy a component | [Components](/reference/v5/operations-api/operations.md#components) | v4.3.0 | | `deploy` (alias) | Alias for `deploy_component` | [Components](/reference/v5/operations-api/operations.md#components) | v4.3.0 | | `package_component` | Package a component | [Components](/reference/v5/operations-api/operations.md#components) | v4.3.0 | | `package` (alias) | Alias for `package_component` | [Components](/reference/v5/operations-api/operations.md#components) | v4.3.0 | | `drop_component` | Remove a component | [Components](/reference/v5/operations-api/operations.md#components) | v4.3.0 | | `get_components` | List all components | [Components](/reference/v5/operations-api/operations.md#components) | v4.3.0 | | `get_component_file` | Get component file contents | [Components](/reference/v5/operations-api/operations.md#components) | v4.3.0 | | `set_component_file` | Set component file contents | [Components](/reference/v5/operations-api/operations.md#components) | v4.3.0 | | `install_node_modules` | Install Node.js dependencies | [Components](/reference/v5/operations-api/operations.md#components) | v4.3.0 | | `set_configuration` | Update configuration settings | [Configuration](/reference/v5/operations-api/operations.md#configuration) | v4.3.0 | | `get_configuration` | Get current configuration | [Configuration](/reference/v5/operations-api/operations.md#configuration) | v4.3.0 | | `create_authentication_tokens` | Create authentication tokens | [Authentication](/reference/v5/operations-api/operations.md#token-authentication) | v4.3.0 | | `refresh_operation_token` | Refresh operation token | [Authentication](/reference/v5/operations-api/operations.md#token-authentication) | v4.3.0 | | `restart_service` | Restart Harper service | [System](/reference/v5/operations-api/operations.md#registration--licensing) | v4.3.0 | | `restart` | Restart Harper instance | [System](/reference/v5/operations-api/operations.md#registration--licensing) | v4.3.0 | | `system_information` | Get system information | [System](/reference/v5/operations-api/operations.md#registration--licensing) | v4.3.0 | | `registration_info` | Get registration information | [Licensing](/reference/v5/operations-api/operations.md#registration--licensing) | v4.3.0 | | `get_fingerprint` | Get instance fingerprint | [Licensing](/reference/v5/operations-api/operations.md#registration--licensing) | v4.3.0 | | `set_license` | Set license key | [Licensing](/reference/v5/operations-api/operations.md#registration--licensing) | v4.3.0 | | `get_usage_licenses` | Get usage and license info | [Licensing](/reference/v5/operations-api/operations.md#registration--licensing) | v4.7.3 | | `get_job` | Get job status | [Jobs](/reference/v5/operations-api/operations.md#jobs) | v4.3.0 | | `search_jobs_by_start_date` | Search jobs by start date | [Jobs](/reference/v5/operations-api/operations.md#jobs) | v4.3.0 | | `read_log` | Read application logs | [Logging](/reference/v5/operations-api/operations.md#logs) | v4.3.0 | | `read_transaction_log` | Read transaction logs | [Logging](/reference/v5/operations-api/operations.md#logs) | v4.3.0 | | `read_audit_log` | Read audit logs | [Logging](/reference/v5/operations-api/operations.md#logs) | v4.3.0 | | `delete_transaction_logs_before` | Delete old transaction logs | [Logging](/reference/v5/operations-api/operations.md#logs) | v4.3.0 | | `purge_stream` | Purge streaming data | [Maintenance](/reference/v5/operations-api/operations.md#jobs) | v4.3.0 | | `delete_records_before` | Delete old records | [Maintenance](/reference/v5/operations-api/operations.md#jobs) | v4.3.0 | | `get_status` | Get custom status information | [Status](/reference/v5/operations-api/operations.md#registration--licensing) | v4.6.0 | | `set_status` | Set custom status information | [Status](/reference/v5/operations-api/operations.md#registration--licensing) | v4.6.0 | | `clear_status` | Clear custom status information | [Status](/reference/v5/operations-api/operations.md#registration--licensing) | v4.6.0 | ### Command Aliases The following aliases are available for convenience: * `deploy` → `deploy_component` * `package` → `package_component` For detailed parameter information for each operation, see the [Operations API documentation](/reference/v5/operations-api/operations.md). ## Command Examples ### Database Operations **Describe a database**: ``` harper describe_database database=dev ``` **Describe a table** (with YAML output): ``` harper describe_table database=dev table=dog ``` **Example Output**: ``` schema: dev name: dog hash_attribute: id audit: true schema_defined: false attributes: - attribute: id is_primary_key: true - attribute: name indexed: true clustering_stream_name: 3307bb542e0081253klnfd3f1cf551b record_count: 10 last_updated_record: 1724483231970.9949 ``` tip For detailed information on database and table structures, see the [Database Reference](/reference/v5/database/overview.md). ### Data Operations **Search by ID** (with JSON output): ``` harper search_by_id database=dev table=dog ids='["1"]' get_attributes='["*"]' json=true ``` **Search by value**: ``` harper search_by_value table=dog search_attribute=name search_value=harper get_attributes='["id", "name"]' ``` tip For more information on querying data, see the [REST Reference](/reference/v5/rest/overview.md) and [GraphQL Querying](/reference/v5/graphql-querying/overview.md). ### Configuration Operations **Set configuration**: ``` harper set_configuration logging_level=error ``` **Get configuration**: ``` harper get_configuration ``` tip For comprehensive configuration options, see the [Configuration Reference](/reference/v5/configuration/overview.md). ### Component Operations **Deploy a component**: ``` harper deploy_component project=my-cool-app package=https://github.com/HarperDB/application-template ``` **Get all components**: ``` harper get_components ``` **Note**: `deploy` is an alias for `deploy_component`: ``` harper deploy project=my-app package=https://github.com/user/repo ``` tip For more information on components and applications, see the [Components Reference](/reference/v5/components/overview.md). ### User and Role Operations **List users**: ``` harper list_users ``` **List roles**: ``` harper list_roles ``` tip For detailed information on users, roles, and authentication, see the [Security Reference](/reference/v5/security/overview.md). ## Remote Operations All CLI operations can be executed on remote Harper instances. See [CLI Overview - Remote Operations](/reference/v5/cli/overview.md#remote-operations) for details on authentication and remote execution. ### Remote Component Deployment When using remote operations, you can deploy a local component or application to the remote instance. **Deploy current directory**: If you omit the `package` parameter, the current directory will be packaged and deployed: ``` harper deploy target=https://server.com:9925 ``` **Note**: `deploy` is an alias for `deploy_component`. **Deploy to clustered environment**: For clustered environments, use the `replicated=true` parameter to ensure the deployment is replicated to all nodes: ``` harper deploy target=https://server.com:9925 replicated=true ``` **Restart after deployment** (with replication): After deploying to a clustered environment, restart all nodes to apply changes: ``` harper restart target=https://server.com:9925 replicated=true ``` For more information on Harper applications and components, see: * [Components](/reference/v5/components/overview.md) - Application architecture and structure * [Deploying Harper Applications](/learn/getting-started/install-and-connect-harper.md) - Step-by-step deployment guide ## Parameter Formatting ### String Parameters Simple string values can be passed directly: ``` harper describe_table database=dev table=dog ``` ### Array Parameters Array parameters must be quoted and formatted as JSON: ``` harper search_by_id database=dev table=dog ids='["1","2","3"]' ``` ### Object Parameters Object parameters are not supported via CLI. For operations requiring complex nested objects, use: * The [Operations API](/reference/v5/operations-api/overview.md) via HTTP * A custom script or tool ### Boolean Parameters Boolean values can be passed as strings: ``` harper get_configuration json=true harper deploy target=https://server.com:9925 replicated=true ``` ## Output Formatting ### YAML (Default) By default, CLI operation results are formatted as YAML for readability: ``` harper describe_table database=dev table=dog ``` ### JSON Pass `json=true` to get JSON output (useful for scripting): ``` harper describe_table database=dev table=dog json=true ``` ## Scripting and Automation The Operations API commands through the CLI are ideal for: * Build and deployment scripts * Automation workflows * CI/CD pipelines * Administrative tasks * Monitoring and health checks **Check status**: ``` # Log in to the cluster harper login https://cluster-node-1.example.com:9925 # Provide your cluster's username and password when prompted. # Deploy component to remote cluster harper deploy \ target=https://cluster-node-1.example.com:9925 \ replicated=true \ package=https://github.com/myorg/my-component # Restart the cluster harper restart \ target=https://cluster-node-1.example.com:9925 \ replicated=true # Check status harper get_components \ target=https://cluster-node-1.example.com:9925 \ json=true ``` ## Limitations The following operation types are **not supported** via CLI: * Operations requiring complex nested JSON structures * Operations with array-of-objects parameters * File upload operations * Streaming operations For these operations, use the [Operations API](/reference/v5/operations-api/overview.md) directly via HTTP. ## See Also * [CLI Overview](/reference/v5/cli/overview.md) - General CLI information * [CLI Commands](/reference/v5/cli/commands.md) - Core CLI commands * [Operations API Overview](/reference/v5/operations-api/overview.md) - Operations API documentation * [Operations API Reference](/reference/v5/operations-api/operations.md) - Complete operations list * [CLI Authentication](/reference/v5/cli/authentication.md) - Authentication details --- # Harper CLI Overview The Harper command line interface (CLI) is used to administer self-installed Harper instances. ## Installation Available since: v4.1.0 Harper is typically installed globally via npm: ``` npm i -g harperdb ``` The installation includes the Harper CLI, which provides comprehensive management capabilities for local and remote Harper instances. For detailed installation instructions, see the [Getting Started / Install And Connect Harper](https://docs.harperdb.io/docs/getting-started/install-and-connect-harper) guide. ## Command Name *Changed in: v4.7.0* The CLI command is `harper`. From v4.1.0 to v4.6.x, the command was only available as `harperdb`. Starting in v4.7.0, the preferred command is `harper`, though `harperdb` continues to work as an alias for backward compatibility. **Examples**: ``` # Modern usage (v4.7.0+) harper harper describe_table database=dev table=dog # Legacy usage (still supported) harperdb harperdb describe_table database=dev table=dog ``` All examples in this documentation use `harper`. ## General Usage The primary way to use Harper is to run the `harper` command. When you run `harper`: * If Harper is not installed, it will guide you through the installation process * Once installed, it runs Harper in the foreground as a standard process * This makes it compatible with systemd, Docker, and other process management tools * Output logs directly to the console for easy monitoring The CLI supports two main categories of commands: 1. **System Commands** - Core Harper management commands (start, stop, restart, status, etc.) 2. **Operations API Commands** - Execute operations from the Operations API directly via the CLI Both system and operations commands can be executed on local or remote Harper instances. For remote operations, authentication credentials can be provided via command parameters or environment variables. ### CLI Installation Targeting By default, the CLI targets the Harper installation path stored in `~/.harperdb/hdb_boot_properties.file`. You can override this to target a specific Harper installation by specifying the `--ROOTPATH` command line argument or the `ROOTPATH` environment variable. **Example: Target a specific installation**: ``` # Using command line argument harper status --ROOTPATH /custom/path/to/hdb # Using environment variable export ROOTPATH=/custom/path/to/hdb harper status ``` ### Process ID File When Harper is running, the process identifier (PID) is stored in a file named `hdb.pid` located in the Harper installation directory. This file can be used by external process management tools or scripts to monitor or manage the Harper process. **Location**: `/hdb.pid` **Example**: ``` # Read the PID cat /path/to/hdb/hdb.pid # Use with external tools kill -0 $(cat /path/to/hdb/hdb.pid) # Check if process is running ``` ## System Management Commands | Command | Description | Available Since | | ---------------------------------- | ------------------------------------------------------------ | --------------- | | `harper` | Run Harper in foreground mode (default behavior) | v4.1.0 | | `harper run ` | Run Harper application from any directory | v4.2.0 | | `harper dev ` | Run Harper in dev mode with auto-restart and console logging | v4.2.0 | | `harper restart` | Restart Harper | v4.1.0 | | `harper start` | Start Harper in background (daemon mode) | v4.1.0 | | `harper stop` | Stop a running Harper instance | v4.1.0 | | `harper login` | Log in to a Harper instance | v5.0.0 | | `harper logout` | Log out of a Harper instance | v5.0.0 | | `harper status` | Display Harper and clustering status | v4.1.0 | | `harper version` | Show installed Harper version | v4.1.0 | | `harper renew-certs` | Renew Harper-generated self-signed certificates | v4.1.0 | | `harper copy-db ` | Copy a database with compaction | v4.1.0 | | `harper help` | Display all available CLI commands | v4.1.0 | See [CLI Commands](/reference/v5/cli/commands.md) for detailed documentation on each command. ## Operations API Commands *Added in: v4.3.0* The Harper CLI supports executing most operations from the [Operations API](/reference/v5/operations-api/overview.md) directly from the command line. This includes operations that do not require complex nested parameters. **Syntax**: `harper =` **Output Format**: Results are formatted as YAML by default. Pass `json=true` for JSON output. **Examples**: ``` # Describe a table harper describe_table database=dev table=dog # Set configuration harper set_configuration logging_level=error # Deploy a component harper deploy_component project=my-app package=https://github.com/user/repo # Get all components harper get_components # Search by ID (JSON output) harper search_by_id database=dev table=dog ids='["1"]' json=true # SQL query harper sql sql='select * from dev.dog where id="1"' ``` See [Operations API Commands](/reference/v5/cli/operations-api-commands.md) for the complete list of available operations. ## Remote Operations *Changed in: v4.3.0* (expanded remote operations support) The CLI can execute operations on remote Harper instances by passing the `target` parameter with the HTTP address of the remote instance. **Authentication**: Provide credentials via: * **Persistent Login**: `harper login` to store tokens (recommended for local development) * **Environment variables and `.env` files**: Use `HARPER_CLI_TARGET`, `HARPER_CLI_USERNAME`, and `HARPER_CLI_PASSWORD` (recommended for CI/CD and project-specific configuration) * **Parameters**: `username= password=` See [CLI Authentication](/reference/v5/cli/authentication.md) for detailed information on authentication methods and best practices. **Example: Persistent Login and `.env`**: ``` # Log in to a specific target harper login https://server.com:9925 # This automatically sets HARPER_CLI_TARGET in your local .env file # Subsequently execute operations without target or credentials harper describe_database database=dev ``` **Example: CLI Target Environment Variables**: ``` export HARPER_CLI_USERNAME=HDB_ADMIN export HARPER_CLI_PASSWORD=password harper describe_database database=dev target=https://server.com:9925 ``` **Example: CLI Options**: ``` harper describe_database database=dev target=https://server.com:9925 username=HDB_ADMIN password=password ``` ## Development Mode *Added in: v4.2.0* For local application and component development, use `harper dev`: ``` harper dev /path/to/app ``` **Features**: * Console logging for immediate feedback * Debugging enabled * Auto-restart on file changes * Ideal for rapid iteration during development See [CLI Commands](/reference/v5/cli/commands.md) for detailed information on `harper dev` and other development commands. ## See Also * [CLI Commands](/reference/v5/cli/commands.md) - Detailed reference for each CLI command * [Operations API Commands](/reference/v5/cli/operations-api-commands.md) - Operations available through CLI * [CLI Authentication](/reference/v5/cli/authentication.md) - Authentication mechanisms * [Configuration](/reference/v5/configuration/overview.md) - Harper configuration options * [Operations API](/reference/v5/operations-api/overview.md) - Full operations API reference --- # Applications > The contents of this page primarily relate to **application** components. The term "components" in the Operations API and CLI generally refers to applications specifically. See the [Components Overview](/reference/v5/components/overview.md) for a full explanation of terminology. Harper offers several approaches to managing applications that differ between local development and remote Harper instances. ## Local Development ### `dev` and `run` Commands *Added in: v4.2.0* The quickest way to run an application locally is with the `dev` command inside the application directory: ``` harper dev . ``` The `dev` command watches for file changes and restarts Harper worker threads automatically. The `run` command is similar but does not watch for changes. Use `run` when the main thread needs to be restarted (the `dev` command does not restart the main thread). Stop either process with SIGINT (Ctrl+C). ### Deploying to a Local Harper Instance To mimic interaction with a hosted Harper instance locally: 1. Start Harper: `harper` 2. Deploy the application: ``` harper deploy \ project= \ package= \ restart=true ``` * Omit `target` to deploy to the locally running instance. * Setting `package=` creates a symlink so file changes are picked up automatically between restarts. * `restart=true` restarts worker threads after deploy. Use `restart=rolling` for a rolling restart. 3. Use `harper restart` in another terminal to restart threads at any time. 4. Remove an application: `harper drop_component project=` > Not all [component operations](#operations-api) are available via CLI. When in doubt, use the Operations API via direct HTTP requests to the local Harper instance. Example: ``` harper deploy \ project=test-application \ package=/Users/dev/test-application \ restart=true ``` > Use `package=$(pwd)` if your current directory is the application directory. ## Remote Management Managing applications on a remote Harper instance uses the same operations as local management. The recommended approach is to log in first using `harper login` to store an authentication token: ``` # Log in once harper login # Provide your username and password when prompted # Subsequently deploy without credentials harper deploy \ project= \ package= \ target= \ restart=true \ replicated=true ``` Alternatively, credentials can be provided via environment variables (recommended for CI/CD): ``` export HARPER_CLI_USERNAME= export HARPER_CLI_PASSWORD= harper deploy \ project= \ package= \ target= \ restart=true \ replicated=true ``` Or directly via command parameters (not recommended for production): ``` harper deploy \ project= \ package= \ username= \ password= \ target= \ restart=true \ replicated=true ``` ### Package Sources When deploying remotely, the `package` field can be any valid npm dependency value: * **Omit** `package` to package and deploy the current local directory * **npm package**: `package="@harperdb/status-check"` * **GitHub**: `package="HarperDB/status-check"` or `package="https://github.com/HarperDB/status-check"` * **Private repo (SSH)**: `package="git+ssh://git@github.com:HarperDB/secret-app.git"` * **Tarball**: `package="https://example.com/application.tar.gz"` When using git tags, use the `semver` directive for reliable versioning: ``` HarperDB/application-template#semver:v1.0.0 ``` Harper generates a `package.json` from component configurations and uses a form of `npm install` to resolve them. This is why specifying a local file path creates a symlink (changes are picked up between restarts without redeploying). For SSH-based private repos, use the [Add SSH Key](#add_ssh_key) operation to register keys first. ## Dependency Management Harper uses `npm` and `package.json` for dependency management. During application loading, Harper follows this resolution order to determine how to install dependencies: 1. If `node_modules` exists, or if `package.json` is absent — skip installation 2. Check the application's `harper-config.yaml` for `install: { command, timeout }` fields 3. Derive the package manager from [`package.json#devEngines#packageManager`](https://docs.npmjs.com/cli/v10/configuring-npm/package-json#devengines) 4. Default to `npm install` The `add_component` and `deploy_component` operations support `install_command` and `install_timeout` fields for customizing this behavior. ### Example `harper-config.yaml` with Custom Install ``` myApp: package: ./my-app install: command: yarn install timeout: 600000 # 10 minutes allowInstallScripts: true ``` ### Example `package.json` with `devEngines` ``` { "name": "my-app", "version": "1.0.0", "devEngines": { "packageManager": { "name": "pnpm", "onFail": "error" } } } ``` > If you plan to use an alternative package manager, ensure it is installed on the host machine. Harper does not support the `"onFail": "download"` option and falls back to `"onFail": "error"` behavior. ## Advanced: Direct `harper-config.yaml` Configuration Applications can be added to Harper by adding them directly to `harper-config.yaml` (located in the Harper `rootPath`, typically `~/hdb`). ``` status-check: package: '@harperdb/status-check' ``` The entry name does not need to match a `package.json` dependency. Harper transforms these entries into a `package.json` and runs `npm install`. Any valid npm dependency specifier works: ``` myGithubComponent: package: HarperDB-Add-Ons/package#v2.2.0 myNPMComponent: package: harper myTarBall: package: /Users/harper/cool-component.tar myLocal: package: /Users/harper/local myWebsite: package: https://harperdb-component ``` Harper generates a `package.json` and installs all components into `` (default: `~/hdb/components`). A symlink back to `/node_modules` is created for dependency resolution. > Use `harper get_configuration` to find the `rootPath` and `componentsRoot` values on your instance. ## Operations API Component operations are restricted to `super_user` roles. ### `add_component` Creates a new component project in the component root directory using a template. * `project` *(required)* — Name of the project to create * `template` *(optional)* — Git URL of a template repository. Defaults to `https://github.com/HarperFast/application-template` * `install_command` *(optional)* — Install command. Defaults to `npm install` * `install_timeout` *(optional)* — Install timeout in milliseconds. Defaults to `300000` (5 minutes) * `install_allowInstallScripts` *(optional)* — Allow install scripts to run. Defaults to `false`, which causes `--ignore-scripts` to be passed to the install command (this is ignored with `install_command`). * `replicated` *(optional)* — Replicate to all cluster nodes ``` { "operation": "add_component", "project": "my-component" } ``` ### `deploy_component` Deploys a component using a package reference or a base64-encoded `.tar` payload. * `project` *(required)* — Name of the project * `package` *(optional)* — Any valid npm reference (GitHub, npm, tarball, local path, URL) * `payload` *(optional)* — Base64-encoded `.tar` file content * `force` *(optional)* — Allow deploying over protected core components. Defaults to `false` * `restart` *(optional)* — `true` for immediate restart, `'rolling'` for sequential cluster restart * `replicated` *(optional)* — Replicate to all cluster nodes * `install_command` *(optional)* — Install command override * `install_timeout` *(optional)* — Install timeout override in milliseconds * `install_allowInstallScripts` *(optional)* — Allow install scripts to run. Defaults to `false`, which causes `--ignore-scripts` to be passed to the install command (this is ignored with `install_command`). ``` { "operation": "deploy_component", "project": "my-component", "package": "HarperDB/application-template#semver:v1.0.0", "replicated": true, "restart": "rolling" } ``` ### `drop_component` Deletes a component project or a specific file within it. * `project` *(required)* — Project name * `file` *(optional)* — Path relative to project folder. If omitted, deletes the entire project * `replicated` *(optional)* — Replicate deletion to all cluster nodes * `restart` *(optional)* — Restart Harper after dropping ``` { "operation": "drop_component", "project": "my-component" } ``` ### `package_component` Packages a project folder as a base64-encoded `.tar` string. * `project` *(required)* — Project name * `skip_node_modules` *(optional)* — Exclude `node_modules` from the package ``` { "operation": "package_component", "project": "my-component", "skip_node_modules": true } ``` ### `get_components` Returns all local component files, folders, and configuration from `harper-config.yaml`. ``` { "operation": "get_components" } ``` ### `get_component_file` Returns the contents of a file within a component project. * `project` *(required)* — Project name * `file` *(required)* — Path relative to project folder * `encoding` *(optional)* — File encoding. Defaults to `utf8` ``` { "operation": "get_component_file", "project": "my-component", "file": "resources.js" } ``` ### `set_component_file` Creates or updates a file within a component project. * `project` *(required)* — Project name * `file` *(required)* — Path relative to project folder * `payload` *(required)* — File content to write * `encoding` *(optional)* — File encoding. Defaults to `utf8` * `replicated` *(optional)* — Replicate update to all cluster nodes ``` { "operation": "set_component_file", "project": "my-component", "file": "test.js", "payload": "console.log('hello world')" } ``` ### SSH Key Management For deploying from private repositories, SSH keys must be registered on the Harper instance. #### `add_ssh_key` * `name` *(required)* — Key name * `key` *(required)* — Private key contents (must be ed25519; use `\n` for line breaks with trailing `\n`) * `host` *(required)* — Host alias for SSH config (used in `package` URL) * `hostname` *(required)* — Actual domain (e.g., `github.com`) * `known_hosts` *(optional)* — Public SSH keys of the host. Auto-retrieved for `github.com` * `replicated` *(optional)* — Replicate to all cluster nodes ``` { "operation": "add_ssh_key", "name": "my-key", "key": "-----BEGIN OPENSSH PRIVATE KEY-----\n...\n-----END OPENSSH PRIVATE KEY-----\n", "host": "my-key.github.com", "hostname": "github.com" } ``` After adding a key, use the configured host in deploy package URLs: ``` "package": "git+ssh://git@my-key.github.com:my-org/my-repo.git#semver:v1.0.0" ``` Additional SSH key operations: `update_ssh_key`, `delete_ssh_key`, `list_ssh_keys`, `set_ssh_known_hosts`, `get_ssh_known_hosts`. --- # Extension API *Deprecated in: v5.0.0* > As of Harper v4.6 and now stable in v5.0, a new iteration of the extension system called **plugins** was released. Plugins simplify the API and are recommended for new extension development. See the [Plugin API](/reference/v5/components/plugin-api.md) reference. Extensions are still supported as of v5.0 but are considered deprecated in favor of plugins. Extensions are components that provide reusable building blocks for applications. There are two key types: * **Resource Extensions** — Handle specific files or directories * **Protocol Extensions** — More advanced extensions that can return a Resource Extension; primarily used for implementing higher-level protocols and custom networking handlers An extension is distinguished from a plain component by implementing one or more of the Resource Extension or Protocol Extension API methods. ## Declaring an Extension All extensions must define a `config.yaml` with an `extensionModule` option pointing to the extension source code (path resolves from the module root directory): ``` extensionModule: ./extension.js ``` If written in TypeScript or another compiled language, point to the built output: ``` extensionModule: ./dist/index.js ``` ## Resource Extension A Resource Extension processes specific files or directories. It is comprised of four function exports: | Method | Thread | Timing | | ------------------- | ------------------ | ------------------------- | | `handleFile()` | All worker threads | Executed on every restart | | `handleDirectory()` | All worker threads | Executed on every restart | | `setupFile()` | Main thread only | Once, at initial start | | `setupDirectory()` | Main thread only | Once, at initial start | > **Important**: `harper restart` only restarts worker threads. Code in `setupFile()` and `setupDirectory()` runs only when Harper fully shuts down and starts again—not on `deploy` or `restart`. `handleFile()` and `setupFile()` have identical signatures. `handleDirectory()` and `setupDirectory()` have identical signatures. ### Resource Extension Configuration Resource Extensions can be configured with `files` and `urlPath` options in `config.yaml`: * `files` — `string | string[] | FilesOptionObject` *(required)* — Glob pattern(s) determining which files and directories are resolved. Harper uses [fast-glob](https://github.com/mrmlnc/fast-glob) for matching. * `source` — `string | string[]` *(required when object form)* — Glob pattern string(s) * `only` — `'all' | 'files' | 'directories'` *(optional)* — Restrict matching to a single entry type. Defaults to `'all'` * `ignore` — `string[]` *(optional)* — Patterns to exclude from matches * `urlPath` — `string` *(optional)* — Base URL path prepended to resolved entries * Starting with `./` (e.g., `'./static/'`) prepends the component name to the URL path * Value of `.` uses the component name as the base path * `..` is invalid and causes an error * Leading/trailing slashes are handled automatically (`/static/`, `static/`, and `/static` are equivalent) Examples: ``` # Serve HTML files from web/ at the /static/ URL path static: files: 'web/*.html' urlPath: 'static' # Load all GraphQL schemas from src/schema/ graphqlSchema: files: 'src/schema/*.graphql' # Match files in web/, excluding web/images/ static: files: source: 'web/**/*' ignore: ['web/images'] # Match only files (not directories) myExtension: files: source: 'dir/**/*' only: 'files' ``` ### Resource Extension API At minimum, a Resource Extension must implement one of the four methods. As a standalone extension, export them directly: ``` // ESM export function handleFile() {} export function setupDirectory() {} // CJS function handleDirectory() {} function setupFile() {} module.exports = { handleDirectory, setupFile }; ``` When returned by a Protocol Extension, define them on the returned object: ``` export function start() { return { handleFile() {}, }; } ``` #### `handleFile(contents, urlPath, absolutePath, resources): void | Promise` #### `setupFile(contents, urlPath, absolutePath, resources): void | Promise` Process individual files. Can be async. Parameters: * `contents` — `Buffer` — File contents * `urlPath` — `string` — Recommended URL path for the file * `absolutePath` — `string` — Absolute filesystem path * `resources` — `Object` — Currently loaded resources #### `handleDirectory(urlPath, absolutePath, resources): boolean | void | Promise` #### `setupDirectory(urlPath, absolutePath, resources): boolean | void | Promise` Process directories. Can be async. If the function returns a truthy value, the component loading sequence ends and no other entries in the directory are processed. Parameters: * `urlPath` — `string` — Recommended URL path for the directory * `absolutePath` — `string` — Absolute filesystem path * `resources` — `Object` — Currently loaded resources ## Protocol Extension A Protocol Extension is a more advanced form of Resource Extension, primarily used for implementing higher-level protocols (e.g., building and running a Next.js project) or adding custom networking handlers. Protocol Extensions use the [`server`](/reference/v5/http/api.md) global API for custom networking. ### Protocol Extension Configuration In addition to the `files`, `urlPath`, and `package` options, Protocol Extensions accept any additional configuration options defined under the extension name in `config.yaml`. These options are passed through to the `options` object of `start()` and `startOnMainThread()`. Many protocol extensions accept `port` and `securePort` options for configuring networking handlers. Example Protocol Extension configuration: ``` '@my-org/my-protocol-extension': package: '@my-org/my-protocol-extension' files: './' port: 9000 dev: false ``` ### Protocol Extension API A Protocol Extension defines up to two methods: | Method | Thread | Timing | | --------------------- | ------------------ | ------------------------- | | `start()` | All worker threads | Executed on every restart | | `startOnMainThread()` | Main thread only | Once, at initial start | Both methods receive the same `options` object and can return a Resource Extension (an object with any of the Resource Extension methods). #### `start(options): ResourceExtension | Promise` #### `startOnMainThread(options): ResourceExtension | Promise` Parameters: * `options` — `Object` — Extension configuration options from `config.yaml` Returns: An object implementing any of the Resource Extension methods ## Version History * **v4.2.0** — Extension system introduced as part of the component architecture * **v4.6.0** — New extension API with support for dynamic reloading; Plugin API introduced as the recommended alternative --- # JavaScript Environment Harper executes component JavaScript in distinct module caches, using Node.js's VM module loader. This provides contextualized module environments that share the same Node.js runtime but have their own set of modules isolated from other applications. This means each application runs in its own module context while still being able to access Harper's full set of APIs. ## Module Loading Harper supports both ESM and CommonJS module formats. The full set of Harper APIs are accessible by importing from the `harper` package, for example:: ``` import { tables, Resource } from 'harper'; ``` ``` const { tables, Resource } = require('harper'); ``` The Harper APIs are also available as global variables. This can be a quick and easy way to use the APIs and is preserved for backward compatibility, but is not the recommended approach. There are also some APIs that are not fully functional as globals. For components in their own directory, link the package to your local `harper` installation to ensure any typings use the current/correction version of Harper: ``` npm link harper ``` All installed components have `harper` automatically linked. Whether you reach them as globals or as `harper` imports, `tables`, `databases`, and the other APIs are the **same live, process-wide objects** — Harper runs as a single process, so a record written through one component is immediately visible to every other. The automatic link points `harper` at the **running** installation (not a separately-installed copy), so `import { tables } from 'harper'` resolves to that live runtime from any module Harper loads. Application module contexts are seeded from the same process globals, not given an isolated set of these objects. This includes bundler-built code. A Vite **server-side-render** entry, for example, can read data straight from Harper and render it into the HTML (no client-side fetch): ``` import { tables } from 'harper'; export async function render(url: string): Promise { const product = await tables.Product.get(idFromUrl(url)); return renderToString(/* */); } ``` When bundling for SSR, keep `harper` external so it resolves to the runtime instead of being bundled (symlinked dependencies are not reliably auto-externalized) — e.g. `ssr: { external: ['harper'] }` in `vite.config`. ## TypeScript Support Harper runs `.ts` files directly via Node.js's built-in [type stripping](https://nodejs.org/api/typescript.html#type-stripping). No build step or transpiler is required. Requirements and conventions: * **Node.js 22.6 or later.** Type stripping is unavailable on earlier versions. * **Use the `.ts` extension on the source files** referenced from `config.yaml`. The `jsResource` plugin loads `.js` and `.ts` files; point its `files` glob at the `.ts` files you want loaded: ``` jsResource: files: 'resources/*.ts' ``` * **Use explicit `.ts` extensions in imports** between local modules. Node's loader does not resolve `'./helper'` to `'./helper.ts'`: ``` import { helper } from './helper.ts'; ``` * **Only type-stripping is performed.** Enum values, namespaces with runtime semantics, and other features that require code transformation are not supported — declarations and type annotations are simply removed. Type imports from the `harper` package work as usual: ``` import { type RequestTargetOrId, Resource, tables } from 'harper'; export class MyResource extends Resource { async get(target?: RequestTargetOrId): Promise<{ message: string }> { return { message: 'Hello from TS' }; } } ``` ## Harper API The following objects and functions are available as exports from the `harper` package (and also available as global variables). ### `tables` An object whose properties are the tables in the default database (`data`). Each table defined in `schema.graphql` is accessible as a property and implements the Resource API. See [Database API](/reference/v5/database/api.md) for full reference. ### `databases` An object containing all databases defined in Harper. Each database is an object of its tables — `databases.data` is always equivalent to `tables`. See [Database API](/reference/v5/database/api.md) for full reference. ### `transaction(fn)` Executes a function inside a database transaction. Changes made within the function are committed atomically, or rolled back if an error is thrown. See [Transactions](/reference/v5/database/transaction.md) for full reference. ### `createBlob(data, options?)` *Added in: v4.5.0* Creates a [`Blob`](https://developer.mozilla.org/en-US/docs/Web/API/Blob) backed by Harper's storage engine. Use it to store large binary content (images, audio, video, etc.) in `Blob`-typed schema fields. See [Database API](/reference/v5/database/api.md) for full reference. ### `Resource` The base class for all Harper resources, including tables and custom data sources. Extend `Resource` to implement custom data providers. See [Resource API](/reference/v5/resources/resource-api.md) for full reference. ### `server` Provides access to Harper's HTTP server middleware chain, WebSocket server, authentication helpers, resource registry, and cluster information. Also exposes `server.contentTypes` as an alias for the `contentTypes` global. See [HTTP API](/reference/v5/http/api.md) for full reference. ### `contentTypes` A `Map` of MIME type strings to content type handler objects. Harper uses this map for content negotiation — deserializing incoming request bodies and serializing outgoing responses. You can register custom handlers to support additional formats. See [HTTP API](/reference/v5/http/api.md) for full reference. ### `logger` Provides structured logging methods (`trace`, `debug`, `info`, `warn`, `error`, `fatal`, `notify`) that write to Harper's log file. Note that using the global variable may not provide full application tagging and configurability. See [Logging API](/reference/v5/logging/api.md) for full reference. ### `resources` A `Map` of all resources registered on this Harper server, keyed by their URL path. Each entry contains the resource class, path, and routing metadata. Use this to look up or enumerate registered resources programmatically. ### `config` The configuration object for the current application, as provided by the component's configuration (e.g. `config.yaml`). Defaults to an empty object if no configuration is provided. ### `RequestTarget` A class (extending `URLSearchParams`) that represents a parsed resource request — including the target record ID, query conditions, pagination (`limit`/`offset`), sort order, selected attributes, and caching directives. Use `RequestTarget` to construct typed resource requests when calling resource methods directly: ``` import { tables, RequestTarget } from 'harper'; const target = new RequestTarget('/my-record-id?limit=10'); const results = await tables.myTable.search(target); ``` ### `getContext()` Returns the current async context object for the active request. The context holds request-scoped state including the active transaction, user, response, and timestamp. Returns an empty object if called outside of a request context. ``` import { getContext } from 'harper'; const ctx = getContext(); console.log(ctx.user, ctx.transaction); ``` ### `getUser()` Returns the authenticated user from the current async request context, or `undefined` if called outside a request or the request is unauthenticated. Equivalent to `getContext().user`. ``` import { getUser } from 'harper'; export class MyResource extends Resource { async get(id) { const user = getUser(); if (!user) throw new Error('Unauthorized'); // ... } } ``` ### `getResponse()` Returns the outgoing `Response` object for the current request, or `undefined` if called outside a request context. Use this to set response headers or inspect the response mid-handler. Equivalent to `getContext().response`. ### Current Working Directory Harper has a multi-threaded server architecture and uses the harper data root path as the current working directory. Components should not and cannot change the current working directory, and must not use `process.chdir()` or any package that does. --- # Next.js Plugin (`@harperfast/nextjs`) `@harperfast/nextjs` is a Harper [Plugin](/reference/v5/components/plugin-api.md) for running [Next.js](https://nextjs.org/) applications directly on Harper. It supports Next.js v14, v15, and v16. Source: [github.com/HarperFast/nextjs](https://github.com/HarperFast/nextjs) — [Example app](https://github.com/HarperFast/nextjs-example) ## Setup ### 1. Install ``` npm install @harperfast/nextjs ``` ### 2. Wrap your Next.js config Use `withHarper()` to wrap your existing Next.js config. All config formats are supported: ``` // next.config.js (CommonJS) const { withHarper } = require('@harperfast/nextjs'); module.exports = withHarper({ // your existing Next.js config }); ``` ``` // next.config.mjs (ESM) import { withHarper } from '@harperfast/nextjs'; export default withHarper({ // your existing Next.js config }); ``` ``` // next.config.ts (TypeScript) import { withHarper } from '@harperfast/nextjs'; export default withHarper({ // your existing Next.js config }); ``` `withHarper()` automatically adds `harper` and `harper-pro` to `serverExternalPackages` so Harper's native dependencies are handled correctly by the Next.js bundler. ### 3. Configure in `config.yaml` ``` '@harperfast/nextjs': package: '@harperfast/nextjs' ``` ### 4. Run ``` harper run nextjs-app ``` ## Using Harper Globals Within any server-side code path, import `harper` to access Harper's global APIs and interact with Harper tables directly: > Ensure you are using `withHarper()`, or have manually added `harper` (or `harper-pro`) to the `serverExternalPackages` list in your Next.js config. ``` // app/actions.js 'use server'; import 'harper'; export async function listDogs() { const dogs = []; for await (const dog of tables.Dog.search()) { dogs.push({ id: dog.id, name: dog.name }); } return dogs; } ``` ``` // app/dogs/[id]/page.jsx import { getDog, listDogs } from '@/app/actions'; export async function generateStaticParams() { const dogs = await listDogs(); return dogs; } export default async function Dog({ params }) { const dog = await getDog(params.id); return (

{dog.name}

Breed: {dog.get('breed')}

); } ``` ## Options All options are configured in `config.yaml` under the `@harperfast/nextjs` key. All options are optional. ### `bundler` Type: `'webpack' | 'turbopack'` Selects the bundler used for building and serving the Next.js application. The default depends on the Next.js version: * Next.js v16: defaults to `turbopack` (matching the Next.js v16 default) * Next.js v15: defaults to `webpack` (matching the Next.js v15 default) * Next.js v14: always `webpack` (Turbopack is not supported) ``` '@harperfast/nextjs': package: '@harperfast/nextjs' bundler: webpack ``` ### `dev` Type: `boolean` Default: `false` Enables Next.js development mode with hot module replacement (HMR). > Dev mode relies on WebSockets. If you encounter an `Invalid WebSocket frame:` error, disable other WebSocket services running on the same port. ### `prebuilt` Type: `boolean` Default: `false` When `true`, the plugin looks for an existing `.next` directory and skips the build step. Useful when the app is pre-built before deployment. ### `buildOnly` Type: `boolean` Default: `false` Builds the Next.js application and then exits, including shutting down Harper. Useful as a CI build step. ### `port` Type: `number` Default: Harper default port (`9926`) Custom HTTP port for the Next.js server. ### `securePort` Type: `number` Default: Harper default secure port Custom HTTPS port for the Next.js server. ### `runFirst` Type: `boolean` Default: `false` When `true`, the Next.js request handler runs before any other Harper HTTP middleware. Useful for scenarios where Next.js handles authentication directly. Note: enabling `runFirst` conflicts with Harper's REST API on the same port. Use a dedicated `port` to avoid conflicts. ## Caching `@harperfast/nextjs` includes a Harper-backed cache handler for Next.js [Incremental Static Regeneration (ISR)](https://nextjs.org/docs/app/guides/incremental-static-regeneration), the [Data Cache](https://nextjs.org/docs/app/deep-dive/caching#data-cache), and [`unstable_cache`](https://nextjs.org/docs/app/api-reference/functions/unstable_cache). Cached entries live in Harper instead of the worker's local filesystem, so a cache write on one cluster node is immediately visible to every other node. > **Note:** The cache handler is currently a work in progress. ### Enabling Use the `cacheHandlerPath()` helper to set the `cacheHandler` path. This helper resolves the path relative to your config file, which is required for Turbopack compatibility: ``` // next.config.js (CommonJS) const { withHarper, cacheHandlerPath } = require('@harperfast/nextjs'); module.exports = withHarper({ cacheHandler: cacheHandlerPath(__dirname), }); ``` ``` // next.config.mjs (ESM) import { withHarper, cacheHandlerPath } from '@harperfast/nextjs'; export default withHarper({ cacheHandler: cacheHandlerPath(import.meta.dirname), }); ``` ``` // next.config.ts (TypeScript) import { withHarper, cacheHandlerPath } from '@harperfast/nextjs'; export default withHarper({ cacheHandler: cacheHandlerPath(import.meta.dirname), }); ``` ### Tag Invalidation [`revalidateTag()`](https://nextjs.org/docs/app/api-reference/functions/revalidateTag) is supported and propagates across the cluster automatically. A typical flow: ``` // app/products/[id]/page.js import { unstable_cache } from 'next/cache'; const getProduct = unstable_cache( async (id) => { const res = await fetch(`https://api.example.com/products/${id}`); return res.json(); }, ['product'], { tags: ['products'], revalidate: 3600 } ); export default async function ProductPage({ params }) { const product = await getProduct(params.id); return

{product.name}

; } ``` ``` // app/api/revalidate/route.js import { revalidateTag } from 'next/cache'; import { NextResponse } from 'next/server'; export async function POST(request) { const tag = new URL(request.url).searchParams.get('tag'); revalidateTag(tag); return NextResponse.json({ revalidated: true }); } ``` `fetch()` calls with `next: { tags: [...] }` and the `'use cache'` directive (with `cacheTag()`) are also supported - anywhere Next.js attaches tags to a cached value, the handler picks them up. ### How Invalidation Works The cache handler uses a soft-invalidation model: 1. `revalidateTag(tag)` writes a `{ tag, timestamp }` row to the `nextjs_cache_invalidation` table and updates an in-memory map in the calling worker. 2. Every other Harper worker subscribes to that table and updates its own map when the row is replicated - typically within milliseconds. 3. On the next `cache.get()`, if any of the cached entry's tags has an invalidation timestamp newer than the entry's `lastModified`, the handler returns `null` and Next.js regenerates the entry. The `nextjs_cache_invalidation` rows expire after 7 days so abandoned tags do not accumulate indefinitely. ### Cache Tables Enabling the cache handler adds two tables to the `harperfast_nextjs` database: | Table | Purpose | | --------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | `nextjs_isr_cache` | One row per cached entry. Stores `data` (the Next.js `IncrementalCacheValue`), `tags`, and `lastModified`. | | `nextjs_cache_invalidation` | One row per invalidated tag. `id` is the tag; `timestamp` is when `revalidateTag` was called. Auto-expires after 7 days. | ## See Also * [Components Overview](/reference/v5/components/overview.md) * [Plugin API](/reference/v5/components/plugin-api.md) * [Next.js Example App](https://github.com/HarperFast/nextjs-example) --- # Components **Components** are the high-level concept for modules that extend the Harper core platform with additional functionality. Components encapsulate both applications and extensions. > Harper is actively working to disambiguate component terminology. When you see "component" in the Operations API or CLI, it generally refers to an application. Documentation does its best to clarify which classification of component is meant wherever possible. ## Concepts ### Applications *Added in: v4.2.0* **Applications** implement specific user-facing features or functionality. Applications are built on top of extensions and represent the end product that users interact with. For example, a Next.js application serving a web interface or an Apollo GraphQL server providing a GraphQL API are both applications. Also, a collection of Harper Schemas and/or custom Resources is also an application. ### Plugins *Stable in: v5.0.0* *Added in: v4.6.0* **Plugins** are the building blocks of the Harper component system. Applications depend on plugins to provide the functionality they implement. For example, the built-in `graphqlSchema` plugin enables applications to define databases and tables using GraphQL schemas. The `@harperfast/nextjs` plugin provides the functionality to build a Next.js application on Harper. Plugins export a single `handleApplication` method and are always executed on worker threads. Plugins can also depend on other plugins. For example, `@harperfast/nextjs` depends on the built-in `graphqlSchema` to create caching tables. ### Extensions *Deprecated in: v5.0.0* *Added in: v4.2.0* **Extensions** were replace by **plugins** in Harper v5; however, they are still supported for backwards compatibility. Extensions make use of the `start`, `startOnMainThread`, `handleFile`, `setupFile`, `handleDirectory`, and `setupDirectory` methods. ### Built-In vs. Custom Components **Built-in** components are included with Harper by default and referenced directly by name. Examples include `graphqlSchema`, `rest`, `jsResource`, `static`, and `loadEnv`. **Custom** components use external references—npm packages, GitHub repositories, or local directories—and are typically included as `package.json` dependencies. Harper does not currently include built-in applications. All applications are custom. ## Architecture The relationship between applications, extensions, and Harper core: ``` Applications ├── Next.js App → @harperfast/nextjs plugin ├── Apollo App → @harperdb/apollo extension └── Custom Resource → jsResource + graphqlSchema + rest plugins Plugins / Extensions ├── Custom Plugins: @harperfast/nextjs ├── Custom Extensions: @harperdb/apollo, @harperdb/astro └── Built-In: graphqlSchema, jsResource, rest, static, loadEnv, ... Core └── database, file-system, networking ``` ## Configuration Harper components are configured with a `config.yaml` file in the root of the component module directory. This file is how a component configures other components it depends on. Each entry starts with a component name, with configuration values indented below: ``` componentName: option-1: value option-2: value ``` ### Default Configuration Components without a `config.yaml` get this default configuration automatically: ``` rest: true graphqlSchema: files: '*.graphql' roles: files: 'roles.yaml' jsResource: files: 'resources.js' fastifyRoutes: files: 'routes/*.js' urlPath: '.' static: files: 'web/**' ``` If a `config.yaml` is provided, it **replaces** the default config entirely (no merging). ### Custom Component Configuration Any custom component must be configured with a `package` option for Harper to load it. The component name must match a `package.json` dependency: ``` { "dependencies": { "@harperfast/nextjs": "2.0.0" } } ``` ``` '@harperfast/nextjs': package: '@harperfast/nextjs' ``` The `package` value supports any valid npm dependency specifier: npm packages, GitHub repos, tarballs, local paths, and URLs. This is because Harper generates a `package.json` from component configurations and uses `npm install` to resolve them. ### Extension and Plugin Configuration Extensions require an `extensionModule` option pointing to the extension source. Plugins require a `pluginModule` option. See [Extension API](/reference/v5/components/extension-api.md) and [Plugin API](/reference/v5/components/plugin-api.md) for details. ## Built-In Extensions Reference | Name | Description | | ------------------------------------------------------------ | ------------------------------------------------- | | [`dataLoader`](/reference/v5/database/data-loader.md) | Load data from JSON/YAML files into Harper tables | | [`fastifyRoutes`](/reference/v5/fastify-routes/overview.md) | Define custom endpoints with Fastify | | [`graphql`](/reference/v5/graphql-querying/overview.md) | Enable GraphQL querying (experimental) | | [`graphqlSchema`](/reference/v5/database/schema.md) | Define table schemas with GraphQL syntax | | [`jsResource`](/reference/v5/resources/overview.md) | Define custom JavaScript-based resources | | [`loadEnv`](/reference/v5/environment-variables/overview.md) | Load environment variables from `.env` files | | [`rest`](/reference/v5/rest/overview.md) | Enable automatic REST endpoint generation | | [`roles`](/reference/v5/users-and-roles/overview.md) | Define role-based access control from YAML files | | [`static`](/reference/v5/static-files/overview.md) | Serve static files via HTTP | ## Known Custom Components ### Applications * [`@harperdb/status-check`](https://github.com/HarperDB/status-check) * [`@harperdb/prometheus-exporter`](https://github.com/HarperDB/prometheus-exporter) * [`@harperdb/acl-connect`](https://github.com/HarperDB/acl-connect) ### Plugins * [`@harperfast/nextjs`](https://github.com/HarperFast/nextjs) — Run a Next.js application on Harper ([docs](/reference/v5/components/nextjs.md)) * [`@harperfast/vite-plugin`](https://github.com/HarperFast/vite-plugin) — Develop and serve Vite-built front-ends from Harper, with HMR via `harper run` ### Extensions * [`@harperdb/apollo`](https://github.com/HarperFast/apollo) — Serve an Apollo GraphQL server backed by Harper * [`@harperdb/astro`](https://github.com/HarperFast/astro) — Run an Astro application on Harper Each component owns its own documentation in its repository — refer to the linked README for installation, configuration, and version-specific details. ## Component Status Monitoring *Added in: v4.7.0* Harper collects status from each component at load time and tracks any registered status change notifications. This provides visibility into the health and state of running components. ## Evolution History * **v4.1.0** — Custom functions with worker threads (predecessor to components) * **v4.2.0** — Component architecture introduced; Resource API, REST interface, MQTT, WebSockets, SSE, configurable schemas * **v4.3.0** — Component configuration improvements * **v4.6.0** — New extension API with dynamic reloading; Plugin API introduced (experimental) * **v4.7.0** — Component status monitoring; further plugin API improvements ## See Also * [Applications](/reference/v5/components/applications.md) — Managing and deploying applications * [Extension API](/reference/v5/components/extension-api.md) — Building custom extensions * [Plugin API](/reference/v5/components/plugin-api.md) — Building plugins (experimental, recommended for new extensions) * [Resource API](/reference/v5/resources/resource-api.md) — Resource class interface * [Database Schema](/reference/v5/database/schema.md) — Defining schemas with graphqlSchema --- # Plugin API *Stable in: v5.0.0* *Added in: v4.6.0* **Plugins** are the building blocks of the Harper component system. Applications depend on plugins to provide the functionality they implement. For example, the built-in `graphqlSchema` plugin enables applications to define databases and tables using GraphQL schemas. The `@harperfast/nextjs` plugin provides the functionality to build a Next.js application on Harper. Plugins export a single `handleApplication` method and are always executed on worker threads. Plugins can also depend on other plugins. For example, `@harperfast/nextjs` depends on the built-in `graphqlSchema` to create caching tables. ## Declaring a Plugin A plugin must specify a `pluginModule` option in `config.yaml` pointing to the plugin source: ``` pluginModule: plugin.js ``` For TypeScript or other compiled languages, point to the built output: ``` pluginModule: ./dist/index.js ``` It is recommended that plugins have a `package.json` with standard JavaScript package metadata (name, version, type, etc.). Plugins are standard JavaScript packages and can be published to npm, written in TypeScript, or export executables. ## Configuration General plugin configuration options: * `files` — `string | string[] | FilesOptionObject` *(optional)* — Glob pattern(s) for files and directories handled by the plugin's default `EntryHandler`. Pattern rules: * Cannot contain `..` or start with `/` * `.` or `./` is transformed to `**/*` automatically * `urlPath` — `string` *(optional)* — Base URL path prepended to resolved `files` entries. Cannot contain `..`. If starts with `./` or is `.`, the plugin name is automatically prepended * `timeout` — `number` *(optional)* — Timeout in milliseconds for plugin operations. Takes precedence over the plugin's `defaultTimeout` and the system default (30 seconds) ### File Entries ``` # Serve files from web/ at /static/ static: files: 'web/**/*' urlPath: '/static/' # Load only *.graphql files from src/schema/ graphqlSchema: files: 'src/schema/*.graphql' # Exclude a subdirectory static: files: source: 'web/**/*' ignore: 'web/images/**' ``` > Note: Unlike the Extension API, the Plugin API `files` object does **not** support an `only` field. Use `entryEvent.entryType` or `entryEvent.eventType` in your handler instead. ### Timeouts The system default timeout is **30 seconds**. If `handleApplication()` does not complete within this time, the component loader throws an error to prevent indefinite hanging. Plugins can override the system default by exporting a `defaultTimeout`: ``` export const defaultTimeout = 60_000; // 60 seconds ``` Users can override at the application level in `config.yaml`: ``` customPlugin: package: '@org/custom-plugin' files: 'foo.js' timeout: 45_000 # 45 seconds ``` ## TypeScript Support All classes and types are exported from the `harper` package: ``` import type { Scope, Config } from 'harper'; ``` ## API Reference ### Function: `handleApplication(scope: Scope): void | Promise` The only required export from a plugin module. The component loader executes it sequentially across all worker threads. It can be async and is awaited. Avoid event-loop-blocking operations within `handleApplication()`. ``` export function handleApplication(scope: Scope) { // Use scope to access config, resources, server, etc. } ``` Parameters: * `scope` — [`Scope`](#class-scope) — Access to the application's configuration, resources, and APIs The `handleApplication()` method cannot coexist with Extension API methods (`start`, `handleFile`, etc.). Defining both will throw an error. ### Class: `Scope` Extends [`EventEmitter`](https://nodejs.org/docs/latest/api/events.html#class-eventemitter) The central object passed to `handleApplication()`. Provides access to configuration, file entries, server APIs, and logging. #### Events * **`'close'`** — Emitted after `scope.close()` is called * **`'error'`** — `error: unknown` — An error occurred * **`'ready'`** — Emitted when the Scope is ready after loading the config file #### `scope.handleEntry([files][, handler])` Returns an [`EntryHandler`](#class-entryhandler) for watching and processing file system entries. Overloads: * `scope.handleEntry()` — Returns the default `EntryHandler` based on `files`/`urlPath` in `config.yaml` * `scope.handleEntry(handler)` — Returns default `EntryHandler`, registers `handler` for the `'all'` event * `scope.handleEntry(files)` — Returns a new `EntryHandler` for custom `files` config * `scope.handleEntry(files, handler)` — Returns a new `EntryHandler` with a custom `'all'` event handler Example: ``` export function handleApplication(scope) { // Default handler with inline callback scope.handleEntry((entry) => { switch (entry.eventType) { case 'add': case 'change': // handle file add/change break; case 'unlink': // handle file deletion break; } }); // Custom handler for specific files const tsHandler = scope.handleEntry({ files: 'src/**/*.ts' }); } ``` #### `scope.requestRestart()` Request a Harper restart. Does not restart immediately—indicates to the user that a restart is required. Called automatically if no `scope.options.on('change')` handler is defined or if a required handler is missing. #### `scope.resources` Returns: `Map` — Currently loaded [Resource](/reference/v5/resources/resource-api.md) instances. #### `scope.server` Returns: `server` — Reference to the [server](/reference/v5/http/api.md) global API. Use for registering HTTP middleware, custom networking, etc. #### `scope.options` Returns: [`OptionsWatcher`](#class-optionswatcher) — Access to the application's configuration options. Emits `'change'` events when the plugin's section of the config file is modified. #### `scope.logger` Returns: `logger` — Scoped logger instance. Recommended over the global `logger`. #### `scope.name` Returns: `string` — The plugin name as configured in `config.yaml`. #### `scope.directory` Returns: `string` — Root directory of the application component (where `config.yaml` lives). #### `scope.close()` Closes all associated entry handlers and the `scope.options` instance, emits `'close'`, and removes all listeners. ### Class: `OptionsWatcher` Extends [`EventEmitter`](https://nodejs.org/docs/latest/api/events.html#class-eventemitter) Provides reactive access to plugin configuration options, scoped to the specific plugin within the application's `config.yaml`. #### Events * **`'change'`** — `key: string[], value: ConfigValue, config: ConfigValue` — Emitted when a config option changes * `key` — Option key split into parts (e.g., `foo.bar` → `['foo', 'bar']`) * `value` — New value * `config` — Entire plugin configuration object * **`'close'`** — Emitted when the watcher is closed * **`'error'`** — `error: unknown` — An error occurred * **`'ready'`** — `config: ConfigValue | undefined` — Emitted on initial load and after `'remove'` recovery * **`'remove'`** — Config was removed (file deleted, config key deleted, or parse failure) Example: ``` export function handleApplication(scope) { scope.options.on('change', (key, value, config) => { if (key[0] === 'files') { scope.logger.info(`Files option changed to: ${value}`); } }); } ``` #### `options.get(key: string[]): ConfigValue | undefined` Get the value at a specific config key path. #### `options.getAll(): ConfigValue | undefined` Get the entire plugin configuration object. #### `options.getRoot(): Config | undefined` Get the root `config.yaml` object (all plugins and options). #### `options.close()` Close the watcher, preventing further events. ### Class: `EntryHandler` Extends [`EventEmitter`](https://nodejs.org/docs/latest/api/events.html#class-eventemitter) Created by [`scope.handleEntry()`](#scopehandleentry). Watches file system entries matching a `files` glob pattern and emits events as files are added, changed, or removed. #### Events * **`'all'`** — `entry: FileEntryEvent | DirectoryEntryEvent` — Emitted for all entry events (add, change, unlink, addDir, unlinkDir). This is the event registered by the `scope.handleEntry(handler)` shorthand. * **`'add'`** — `entry: AddFileEvent` — File created or first seen * **`'addDir'`** — `entry: AddDirectoryEvent` — Directory created or first seen * **`'change'`** — `entry: ChangeFileEvent` — File modified * **`'close'`** — Entry handler closed * **`'error'`** — `error: unknown` — An error occurred * **`'ready'`** — Handler ready and watching * **`'unlink'`** — `entry: UnlinkFileEvent` — File deleted * **`'unlinkDir'`** — `entry: UnlinkDirectoryEvent` — Directory deleted Recommended pattern for handling all events: ``` scope.handleEntry((entry) => { switch (entry.eventType) { case 'add': break; case 'change': break; case 'unlink': break; case 'addDir': break; case 'unlinkDir': break; } }); ``` #### `entryHandler.name` Returns: `string` — Plugin name. #### `entryHandler.directory` Returns: `string` — Application root directory. #### `entryHandler.close()` Closes the entry handler, removing all listeners. Can be restarted with `update()`. #### `entryHandler.update(config: FilesOption | FileAndURLPathConfig)` Update the handler to watch new entries. Closes and recreates the underlying watcher while preserving existing listeners. Returns a Promise that resolves when the updated handler is ready. ### Interfaces #### `FilesOption` `string | string[] | FilesOptionObject` #### `FilesOptionObject` * `source` — `string | string[]` *(required)* — Glob pattern(s) * `ignore` — `string | string[]` *(optional)* — Patterns to exclude #### `FileAndURLPathConfig` * `files` — `FilesOption` *(required)* * `urlPath` — `string` *(optional)* #### `BaseEntry` * `stats` — `fs.Stats | undefined` — File system stats (may be absent depending on event, entry type, and platform) * `urlPath` — `string` — URL path of the entry, resolved from `files` + `urlPath` options * `absolutePath` — `string` — Absolute filesystem path #### `FileEntry` Extends `BaseEntry` * `contents` — `Buffer` — File contents (automatically read) #### `EntryEvent` Extends `BaseEntry` * `eventType` — `string` — Type of event * `entryType` — `'file' | 'directory'` — Entry type #### `AddFileEvent` * `eventType: 'add'` * `entryType: 'file'` * Extends `EntryEvent`, `FileEntry` #### `ChangeFileEvent` * `eventType: 'change'` * `entryType: 'file'` * Extends `EntryEvent`, `FileEntry` #### `UnlinkFileEvent` * `eventType: 'unlink'` * `entryType: 'file'` * Extends `EntryEvent`, `FileEntry` #### `FileEntryEvent` `AddFileEvent | ChangeFileEvent | UnlinkFileEvent` #### `AddDirectoryEvent` * `eventType: 'addDir'` * `entryType: 'directory'` * Extends `EntryEvent` #### `UnlinkDirectoryEvent` * `eventType: 'unlinkDir'` * `entryType: 'directory'` * Extends `EntryEvent` #### `DirectoryEntryEvent` `AddDirectoryEvent | UnlinkDirectoryEvent` #### `Config` `{ [key: string]: ConfigValue }` Parsed representation of `config.yaml`. #### `ConfigValue` `string | number | boolean | null | undefined | ConfigValue[] | Config` #### `onEntryEventHandler` `(entryEvent: FileEntryEvent | DirectoryEntryEvent): void` Function signature for the `'all'` event handler passed to `scope.handleEntry()`. ## Example: Static File Server Plugin A simplified form of the built-in `static` plugin demonstrating key Plugin API patterns: ``` export function handleApplication(scope) { const staticFiles = new Map(); // React to config changes scope.options.on('change', (key, value, config) => { if (key[0] === 'files' || key[0] === 'urlPath') { staticFiles.clear(); scope.logger.info(`Static files reset due to change in ${key.join('.')}`); } }); // Handle file entry events scope.handleEntry((entry) => { if (entry.entryType === 'directory') return; switch (entry.eventType) { case 'add': case 'change': staticFiles.set(entry.urlPath, entry.contents); break; case 'unlink': staticFiles.delete(entry.urlPath); break; } }); // Register HTTP middleware scope.server.http( (req, next) => { if (req.method !== 'GET') return next(req); const file = staticFiles.get(req.pathname); return file ? { statusCode: 200, body: file } : { statusCode: 404, body: 'File not found' }; }, { runFirst: true } ); } ``` ## Version History * **v4.6.0** — Plugin API introduced (experimental) * **v4.7.0** — Further improvements to the Plugin API --- # Worker Thread Debugging Harper runs as a main thread plus a pool of worker threads (configurable via `threads.count`). The `threads.debug` option exposes the Node.js inspector on each thread so you can attach Chrome DevTools, VS Code, or any [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/) (CDP) client to step through component code, inspect heap state, or capture CPU profiles. For the worker thread architecture, see [Architecture Overview](/reference/v5/database/overview.md#architecture-overview). ## Enabling the Debugger The simplest form starts the inspector on the main thread at the default port (`9229`): ``` threads: debug: true ``` For per-thread debugging, expand to an object: ``` threads: debug: startingPort: 9229 host: 127.0.0.1 waitForDebugger: false ``` | Property | Type | Default | Description | | ----------------- | --------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `port` | `integer` | `9229` | Port for the main thread inspector. Use this when you only need to debug startup or main-thread behavior. | | `startingPort` | `integer` | *(none)* | When set, each worker thread gets a sequential inspector port starting from this value. Thread N uses port `startingPort + N`. The main thread keeps `port`. | | `host` | `string` | `127.0.0.1` | Interface the inspector binds to. Leave on loopback in production; use `0.0.0.0` only when tunneling over SSH or operating in a trusted network. | | `waitForDebugger` | `boolean` | `false` | Pause each thread at startup until a debugger attaches. Useful for catching bugs that occur during component initialization. | ## Attaching Chrome DevTools 1. Set `threads.debug.startingPort: 9230` (so worker threads use 9230, 9231, … and the main thread keeps 9229). 2. Start Harper. 3. Open `chrome://inspect` in Chrome. 4. Click **Configure** and add `localhost:9229`, `localhost:9230`, … for each thread you want to inspect. 5. The threads appear under **Remote Target**. Click **inspect** to open DevTools for that thread. ## Attaching VS Code Add an entry per thread to `.vscode/launch.json`. The example below attaches to the main thread and two workers: ``` { "version": "0.2.0", "configurations": [ { "type": "node", "request": "attach", "name": "Harper main", "port": 9229 }, { "type": "node", "request": "attach", "name": "Harper worker 1", "port": 9230 }, { "type": "node", "request": "attach", "name": "Harper worker 2", "port": 9231 } ], "compounds": [ { "name": "Harper (all threads)", "configurations": ["Harper main", "Harper worker 1", "Harper worker 2"] } ] } ``` Run the compound configuration to attach to every thread at once. ## Debugging Remote Instances Inspector ports must remain on `127.0.0.1` in production. To reach them from a developer workstation, tunnel each port over SSH: ``` ssh -L 9229:127.0.0.1:9229 \ -L 9230:127.0.0.1:9230 \ -L 9231:127.0.0.1:9231 \ harper.example.com ``` Then point Chrome DevTools or VS Code at `localhost:9229–9231` as if they were local. ## Waiting for the Debugger at Startup When a bug only reproduces during component initialization, set `waitForDebugger: true`. Each thread starts paused on its first line until a debugger attaches and resumes execution. This is also the safest way to debug an initialization sequence that completes too quickly to manually attach. ``` threads: debug: startingPort: 9230 waitForDebugger: true ``` **Health checks and load balancers** will fail while the threads are paused — only enable `waitForDebugger` in dedicated debug environments. ## Heap Snapshots Near the Limit ### `threads.heapSnapshotNearLimit` Type: `boolean` Default: `false` When the V8 heap approaches the limit set by `threads.maxHeapMemory`, the thread writes a `.heapsnapshot` file to the Harper root directory before the process exits with an out-of-memory error. The snapshot can be loaded into Chrome DevTools (Memory tab → **Load profile**) to identify retained objects responsible for the leak. ``` threads: maxHeapMemory: 1024 heapSnapshotNearLimit: true ``` Snapshots can be large (often a sizable fraction of the heap limit) and writing them blocks the thread briefly — leave disabled for normal operation and enable only when investigating an out-of-memory pattern. ## Related * [Configuration Options — `threads`](/reference/v5/configuration/options.md#threads) — full thread configuration reference * [Architecture Overview](/reference/v5/database/overview.md#architecture-overview) — how worker threads fit into Harper * [Node.js Inspector documentation](https://nodejs.org/en/learn/getting-started/debugging) — debugger protocol details --- # Configuration Operations Operations API endpoints for reading and modifying Harper configuration. *All operations in this section are restricted to `super_user` roles.* For the full list of configurable options, see [Configuration Options](/reference/v5/configuration/options.md). *** ## Set Configuration Modifies one or more Harper configuration parameters. **Requires a [restart](/reference/v5/operations-api/operations.md#restart) or [restart\_service](/reference/v5/operations-api/operations.md#restart_service) to take effect.** `operation` *(required)* — must be `set_configuration` `replicated` *(optional)* *Added in: v5.2.0* — set to `true` to apply the same configuration change to all cluster nodes in a single call. The origin node applies the change first; if its local write fails, nothing is sent to peers. Requires replication to be configured. Additional properties correspond to configuration keys in underscore-separated format (e.g. `logging_level` for `logging.level`, `clustering_enabled` for `clustering.enabled`). Replicate cluster-appropriate parameters only A replicated call sends the exact same values to every node. Do not include node-local parameters — ports, `node.hostname`, `rootPath`, storage/log file paths, TLS certificate or key paths, or `replication.hostname`/`url`/`routes` — as they would overwrite each peer's local values. ### Body ``` { "operation": "set_configuration", "logging_level": "trace", "clustering_enabled": true } ``` ### Response: 200 ``` { "message": "Configuration successfully set. You must restart HarperDB for new config settings to take effect." } ``` ### Replicated body ``` { "operation": "set_configuration", "http_corsAccessList": ["app.example.com"], "replicated": true } ``` ### Response: 200 (replicated) The `replicated` array reports per-node outcomes. A failed peer appears as `{ "status": "failed", "reason": "...", "node": "..." }` while `message` still reports overall success — always check the array when replicating. ``` { "message": "Configuration successfully set. You must restart Harper for new config settings to take effect.", "replicated": [ { "message": "Configuration successfully set. You must restart Harper for new config settings to take effect.", "node": "node-2" } ] } ``` To restart the whole cluster afterward, follow with [restart\_service](/reference/v5/operations-api/operations.md#restart_service) using `"replicated": true` — it restarts nodes one at a time, so the cluster stays available: ``` { "operation": "restart_service", "service": "http", "replicated": true } ``` *** ## Get Configuration Returns the current Harper configuration. `operation` *(required)* — must be `get_configuration` ### Body ``` { "operation": "get_configuration" } ``` ### Response: 200 ``` { "http": { "compressionThreshold": 1200, "cors": false, "corsAccessList": [null], "keepAliveTimeout": 30000, "port": 9926, "securePort": null, "timeout": 120000 }, "threads": 11, "authentication": { "cacheTTL": 30000, "enableSessions": true, "operationTokenTimeout": "1d", "refreshTokenTimeout": "30d" }, "analytics": { "aggregatePeriod": 60 }, "replication": { "hostname": "node1", "databases": "*", "routes": null, "url": "wss://127.0.0.1:9925" }, "componentsRoot": "/Users/hdb/components", "localStudio": { "enabled": false }, "logging": { "auditAuthEvents": { "logFailed": false, "logSuccessful": false }, "auditLog": true, "auditRetention": "3d", "file": true, "level": "error", "root": "/Users/hdb/log", "rotation": { "enabled": false, "compress": false, "interval": null, "maxSize": null, "path": "/Users/hdb/log" }, "stdStreams": false }, "mqtt": { "network": { "port": 1883, "securePort": 8883 }, "webSocket": true, "requireAuthentication": true }, "operationsApi": { "network": { "cors": true, "corsAccessList": ["*"], "domainSocket": "/Users/hdb/operations-server", "port": 9925, "securePort": null } }, "rootPath": "/Users/hdb", "storage": { "writeAsync": false, "caching": true, "compression": false, "noReadAhead": true, "path": "/Users/hdb/database", "prefetchWrites": true }, "tls": { "privateKey": "/Users/hdb/keys/privateKey.pem" } } ``` --- # Configuration Options Quick reference for all `harper-config.yaml` top-level sections. For how to apply configuration (YAML file, environment variables, CLI, Operations API), see [Configuration Overview](/reference/v5/configuration/overview.md). *** ## `http` Configures the Harper component server (HTTP, REST API, WebSocket). See [HTTP Configuration](/reference/v5/http/configuration.md) for full details. ``` http: port: 9926 securePort: 4443 cors: true timeout: 120000 mtls: false logging: level: info path: ~/hdb/log/http.log ``` * `sessionAffinity` — Route requests from same client to same worker thread (`ip` or header name) * `compressionThreshold` — Response size threshold for Brotli compression; *Default*: `1200` (bytes) * `cors` — Enable CORS; *Default*: `true` * `corsAccessList` — Allowed domains for CORS requests * `corsAccessControlAllowHeaders` — `Access-Control-Allow-Headers` value for OPTIONS preflight * `headersTimeout` — Max wait for complete HTTP headers (ms); *Default*: `60000` * `maxHeaderSize` — Max HTTP header size (bytes); *Default*: `16394` * `requestQueueLimit` — Max estimated request queue time (ms) before 503; *Default*: `20000` * `keepAliveTimeout` — Inactivity before closing keep-alive connection (ms); *Default*: `30000` * `port` — HTTP port; *Default*: `9926` * `securePort` — HTTPS port; requires [TLS configuration](/reference/v5/http/tls.md); *Default*: `null` * `http2` — Enable HTTP/2; *Default*: `false` (Added in: v4.5.0) * `timeout` — Request timeout (ms); *Default*: `120000` * `mtls` — Enable [mTLS authentication](/reference/v5/security/mtls-authentication.md) for incoming connections; sub-options: `user`, `required`, `certificateVerification` (see [Certificate Verification](/reference/v5/security/certificate-verification.md)) * `logging` — HTTP request logging (disabled by default, Added in: v4.6.0); sub-options: `level`, `path`, `timing`, `headers`, `id`. See [Logging Configuration](/reference/v5/logging/configuration.md) *** ## `threads` Worker thread pool configuration. ``` threads: count: 11 maxHeapMemory: 300 ``` * `count` — Number of worker threads; *Default*: CPU count minus one * `maxHeapMemory` — Heap limit per thread (MB) * `heapSnapshotNearLimit` — Write a `.heapsnapshot` file when a thread nears its heap limit (loadable in Chrome DevTools Memory tab); *Default*: `false`. See [Worker Thread Debugging](/reference/v5/configuration/debugging.md#heap-snapshots-near-the-limit) * `debug` — Enable Node.js inspector; sub-options: `port`, `startingPort`, `host`, `waitForDebugger`. See [Worker Thread Debugging](/reference/v5/configuration/debugging.md) * `preload` *Added in: v5.2.0* — Module, or list of modules, to load (via Node's `--import`) before any Harper or application module on each worker thread. Intended for instrumentation/APM agents that must load first to instrument subsequent module loads. Use the agent's ESM/register entry — e.g. `dd-trace/register.js`, which registers the loader hooks that instrument worker threads (where Harper runs its work); the plain `dd-trace/init` (`--require`) entry only covers the main thread. Bare specifiers resolve against the `node_modules` of your installed [components](/reference/v5/components/overview.md) — so the agent can be shipped as a dependency of a deployed component — and absolute paths are also accepted. Applies to worker threads only (not under Bun). ``` threads: preload: dd-trace/register.js ``` Or several modules: ``` threads: preload: - dd-trace/register.js - /opt/instrumentation/agent.mjs ``` * `preloadRequire` *Added in: v5.2.0* — Same as `preload`, but loads modules via Node's `--require` (CommonJS) instead of `--import`. Use this for agents that document the `--require` path and do not need ESM loader hooks (e.g. `dd-trace/init`, Dynatrace OneAgent). Same resolution rules as `preload`. ``` threads: preloadRequire: dd-trace/init ``` *** ## `authentication` Authentication and session configuration. Added in: v4.1.0; `enableSessions` added in v4.2.0. See [Authentication Configuration](/reference/v5/security/configuration.md). ``` authentication: authorizeLocal: true cacheTTL: 30000 enableSessions: true operationTokenTimeout: 1d refreshTokenTimeout: 30d ``` * `authorizeLocal` — Auto-authorize loopback requests as superuser; *Default*: `true` * `cacheTTL` — Session cache duration (ms); *Default*: `30000` * `enableSessions` — Cookie-based sessions; *Default*: `true` * `operationTokenTimeout` — Access token lifetime; *Default*: `1d` * `refreshTokenTimeout` — Refresh token lifetime; *Default*: `1d` * `logging` — Authentication event logging (Added in: v4.6.0); sub-options: `path`, `level`, `tag`, `stdStreams`. See [Logging Configuration](/reference/v5/logging/configuration.md) *** ## `operationsApi` Harper Operations API endpoint configuration. See [Operations API Overview](/reference/v5/operations-api/overview.md). ``` operationsApi: network: port: 9925 cors: true tls: certificate: ~/hdb/keys/certificate.pem privateKey: ~/hdb/keys/privateKey.pem ``` * `componentFile.maxSize` — Maximum file size in bytes returned by `get_component_file`. Requests for files larger than this limit are rejected with HTTP 413. *Default*: `5242880` (5 MB) * `network.cors` / `network.corsAccessList` — CORS settings * `network.domainSocket` — Unix socket path for CLI communication; *Default*: `/hdb/operations-server` * `network.headersTimeout` / `network.keepAliveTimeout` / `network.timeout` — Timeout settings (ms) * `network.port` — Operations API port; *Default*: `9925` * `network.securePort` — HTTPS port; *Default*: `null` * `tls` — TLS override for the Operations API; sub-options: `certificate`, `certificateAuthority`, `privateKey`. See [`tls`](#tls) *** ## `tls` Global TLS configuration for HTTPS and TLS sockets (used by HTTP and MQTT). Can be a single object or an array for SNI. See [TLS](/reference/v5/http/tls.md) and [Certificate Management](/reference/v5/security/certificate-management.md). ``` tls: certificate: ~/hdb/keys/certificate.pem certificateAuthority: ~/hdb/keys/ca.pem privateKey: ~/hdb/keys/privateKey.pem ``` * `certificate` — Path to TLS certificate; *Default*: `/keys/certificate.pem` * `certificateAuthority` — Path to CA file; *Default*: `/keys/ca.pem` * `privateKey` — Path to private key; *Default*: `/keys/privateKey.pem` * `ciphers` — Allowed TLS cipher suites *** ## `mqtt` MQTT protocol configuration. Added in: v4.2.0. See [MQTT Configuration](/reference/v5/mqtt/configuration.md). ``` mqtt: network: port: 1883 securePort: 8883 webSocket: true requireAuthentication: true ``` * `network.port` — Insecure MQTT port; *Default*: `1883` * `network.securePort` — Secure MQTT port; *Default*: `8883` * `network.mtls` — Enable [mTLS](/reference/v5/security/mtls-authentication.md) for MQTT connections; sub-options: `user`, `required`, `certificateAuthority`, `certificateVerification` * `webSocket` — Enable MQTT over WebSocket on HTTP port; *Default*: `true` * `requireAuthentication` — Require credentials or mTLS; *Default*: `true` * `logging` — MQTT event logging (Added in: v4.6.0); sub-options: `path`, `level`, `tag`, `stdStreams`. See [Logging Configuration](/reference/v5/logging/configuration.md) *** ## `logging` Application logging. Added in: v4.1.0; per-component logging added in v4.6.0. See [Logging Configuration](/reference/v5/logging/configuration.md). ``` logging: level: warn root: ~/hdb/log stdStreams: false auditLog: false rotation: interval: 1D maxSize: 100M ``` * `level` — Log verbosity (`trace` → `debug` → `info` → `warn` → `error` → `fatal` → `notify`); *Default*: `warn` * `file` — Write to file; *Default*: `true` * `root` — Log directory; *Default*: `/log` * `path` — Explicit log file path (overrides `root`) * `stdStreams` — Write to stdout/stderr; *Default*: `false` * `console` — Include `console.*` output; *Default*: `true` * `auditLog` — Enable table transaction audit logging; *Default*: `false` * `auditRetention` — Audit log retention duration; *Default*: `3d` * `external` — Logging for components using the logger API; sub-options: `level`, `path` * `rotation.enabled` / `rotation.compress` / `rotation.interval` / `rotation.maxSize` / `rotation.path` — Log file rotation (activates when `interval` or `maxSize` is set) * `auditAuthEvents.logFailed` / `auditAuthEvents.logSuccessful` — Log failed/successful authentication events; *Default*: `false` *** ## `replication` Native WebSocket-based replication (Plexus). Added in: v4.4.0. See [Replication](/reference/v5/replication/overview.md) and [Clustering](/reference/v5/replication/clustering.md). ``` replication: hostname: server-one url: wss://server-one:9933 databases: '*' routes: - wss://server-two:9933 ``` * `hostname` — This instance's hostname within the cluster * `url` — WebSocket URL peers use to connect to this instance * `databases` — Databases to replicate; *Default*: `"*"` (all). Each entry supports `name` and `sharded` * `routes` — Peer nodes; URL strings or `{hostname, port, startTime, revokedCertificates}` objects * `port` — Replication port * `securePort` — Secure replication port; *Default*: `9933` (changed from `9925` in v4.5.0) * `enableRootCAs` — Verify against Node.js Mozilla CA store; *Default*: `true` * `blobTimeout` — Blob transfer timeout (ms); *Default*: `120000` * `blobSendDrainTimeout` — Maximum time (ms) a worker waits for in-flight replication blob **sends** to finish before shutting down during a restart, so a rolling restart (e.g., a component deploy reload) doesn't interrupt a transfer in progress. Only sends that are still making progress are waited on; `0` disables draining; *Default*: `600000` * `failOver` — Failover to alternate node if peer unreachable; *Default*: `true` * `shard` — Shard ID for traffic routing; see [Sharding](/reference/v5/replication/sharding.md) * `mtls.certificateVerification` — Certificate revocation checking (CRL/OCSP) for replication connections; see [Certificate Verification](/reference/v5/security/certificate-verification.md) * `logging` — Replication event logging; sub-options: `path`, `level`, `tag`, `stdStreams`. See [Logging Configuration](/reference/v5/logging/configuration.md) *** ## `storage` Database storage configuration. See [Storage Tuning](/reference/v5/database/storage-tuning.md) for guidance on tuning these options for production workloads, [Database Overview](/reference/v5/database/overview.md) for general database concepts, and [Compaction](/reference/v5/database/compaction.md) for reclaiming space inside existing files. ``` storage: path: ~/hdb/database caching: true compression: true compactOnStart: false engine: rocksdb ``` * `engine` — The database storage engine to use for new databases. Currently supported engines are `rocksdb` and `lmdb`. The default is `rocksdb`. Existing databases will use the engine that was used when they were created. * `writeAsync` — Disable disk sync for higher throughput (**disables durability guarantees**); *Default*: `false` * `caching` — In-memory record caching; *Default*: `true` * `compression` — LZ4 record compression; *Default*: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold` * `compactOnStart` — Compact all non-system databases on startup; *Default*: `false` (Added in: v4.3.0) * `compactOnStartKeepBackup` — Retain compaction backups; *Default*: `false` * `maxTransactionQueueTime` — Max write queue time before 503; *Default*: `45s` * `noReadAhead` — Advise OS against read-ahead; *Default*: `false` * `prefetchWrites` — Prefetch before write transactions; *Default*: `true` * `path` — Database files directory; *Default*: `/database` * `blobPaths` — Blob storage directory or directories; *Default*: `/blobs` (Added in: v4.5.0) * `pageSize` — Database page size (bytes); *Default*: OS default * `reclamation.threshold` — Free-space ratio below which reclamation begins evicting from caching tables; *Default*: `0.4` (Added in: v4.5.0) * `reclamation.interval` — Free-space check interval; *Default*: `1h` * `reclamation.evictionFactor` — Heuristic factor for early eviction under disk pressure; *Default*: `100000`. See [Storage Tuning — Reclamation](/reference/v5/database/storage-tuning.md#storage-reclamation) * `rocks.blockCacheSize` — RocksDB shared block cache size in bytes; *Default*: 25% of constrained memory. See [Storage Tuning — RocksDB Memory](/reference/v5/database/storage-tuning.md#rocksdb-memory) (Added in: v5.1.0) * `rocks.writeBufferManagerSize` — Process-wide cap (bytes) on RocksDB memtable memory across all databases. `0` disables; *Default*: one third of `blockCacheSize` (enabled). See [Storage Tuning — RocksDB Memory](/reference/v5/database/storage-tuning.md#rocksdb-memory) (Added in: v5.1.0) * `rocks.writeBufferManagerCostToCache` — When enabled, memtable memory and the block cache share a single, unified memory pool. During heavy write bursts, the block cache dynamically shrinks to give memtables more room. Once those memtables flush to disk, the block cache automatically reclaims that memory space. *Default*: `true`. Has no effect when `writeBufferManagerSize` is `0`. (Added in: v5.1.0) * `rocks.writeBufferManagerAllowStall` — Stall writes when memtable memory exceeds `writeBufferManagerSize` (hard cap) instead of allowing brief overshoot with more aggressive flushing (soft cap); *Default*: `true`. (Added in: v5.1.0) *** ## `databases` Per-database and per-table file path overrides. Must be set before the database/table is created. See [Database Overview](/reference/v5/database/overview.md). ``` databases: myDatabase: path: /data/myDatabase auditPath: /data/myDatabase-audit tables: myTable: path: /data/myTable ``` * `.path` — Database files directory * `.auditPath` — Audit log directory for this database * `.tables..path` — Table files directory *** ## `analytics` Analytics aggregation configuration. See [Analytics Overview](/reference/v5/analytics/overview.md). ``` analytics: aggregatePeriod: 60 replicate: false ``` * `aggregatePeriod` — Aggregation interval (seconds); *Default*: `60` (Added in: v4.5.0) * `storageInterval` — Aggregation cycles between storage-volume measurements (`0` disables); *Default*: `10` * `replicate` — Replicate analytics data across cluster; *Default*: `false` * `logging` — Per-subsystem logger override for analytics writes. See [Logging Configuration](/reference/v5/logging/configuration.md#analyticslogging) *** ## `localStudio` Local Harper Studio GUI. See [Studio](/reference/v5/studio/overview.md). ``` localStudio: enabled: true ``` * `enabled` — Enable local Studio at `http://localhost:`; *Default*: `false` *** ## `componentsRoot` Path to local component files. Added in: v4.2.0 (previously `customFunctionsRoot`). See [Components](/reference/v5/components/overview.md). ``` componentsRoot: ~/hdb/components ``` *** ## `rootPath` Root directory for all Harper persistent data, config, logs, and components. ``` rootPath: /var/lib/harper ``` *** ## `applications` Added in: v5.0.0 ``` applications: lockdown: freeze moduleLoader: vm dependencyLoader: auto allowedSpawnCommands: - npm - node ``` * `lockdown` — Indicates if intrinsic/built-in objects should be locked down/frozen. This provides additional security and protection against prototype pollution attacks. The options can be `freeze` (default, which freezes the important built-in objects, without interfering with most packages), 'none', or 'ses' (lockdown provided by `ses` package, which is more strict). * `moduleLoader` — The method used to load modules (and isolate the application). The default is `vm`, which uses Node's VM to load modules. This can also be set to `native` (use standard Node module loader), or `compartment`, which uses the `ses` implementation of the proposed `Compartment` functionality. * `dependencyLoader` — The application module loader can be used to load packages/dependencies (installed as `dependencies` from the package.json). The default is 'auto', which only use the VM module loader if the package specifies `harper` as a dependency. This can also be set to `app` to always use the application module loader or `native` to always native module loader for packages. * `allowedSpawnCommands` - This lists the specific commands that can be spawned by the application (using `child_process`'s `spawn()`, `exec()`, and `execFile()` functions). You can add commands that you are application will need to launch (this is to protect against malicious code spawning processes). ## Component Configuration Installed components are configured directly at the root of `harper-config.yaml` using the component name as the key — not nested under a `components:` section. See [Components](/reference/v5/components/overview.md). ``` my-component: package: 'HarperDB-Add-Ons/my-component' port: 4321 ``` * `.package` — NPM package name, GitHub repo (`user/repo`), or local path * `.port` — Port for the component; *Default*: value of `http.port` --- # Configuration Harper is configured through a [YAML](https://yaml.org/) file called `harper-config.yaml` located in the Harper root directory. By default the root directory is a folder named `hdb` in the home directory of the current user. Some configuration values are pre-populated in the config file on install, regardless of whether they are used. For a complete reference of all available configuration options, see [Configuration Options](/reference/v5/configuration/options.md). *** ## The Configuration File To change a configuration value, edit `harper-config.yaml` and save. **Harper must be restarted for changes to take effect.** Configuration keys use camelCase (e.g. `operationsApi`). Nested keys use dot notation conceptually (e.g. `operationsApi.network.port`). *** ## Setting Configuration Values All configuration values can be set through four mechanisms: ### 1. YAML File (direct edit) Edit `harper-config.yaml` directly: ``` http: port: 9926 logging: level: warn ``` ### 2. Environment Variables Map YAML keys to `SCREAMING_SNAKE_CASE`. Use underscores for nesting. Keys are case-insensitive. Examples: * `http.port` → `HTTP_PORT=9926` * `logging.rotation.enabled` → `LOGGING_ROTATION_ENABLED=false` * `operationsApi.network.port` → `OPERATIONSAPI_NETWORK_PORT=9925` ``` HTTP_PORT=9926 harper ``` > **Note:** Component configuration cannot be set via environment variables or CLI arguments. To set multiple values (or a whole config object) from a single environment variable, use `HARPER_CONFIG` — see [Environment Variable-Based Configuration](#environment-variable-based-configuration). ### 3. CLI Arguments Same naming convention as environment variables, prefixed with `--`: ``` harper --HTTP_PORT 9926 --LOGGING_LEVEL warn ``` ### 4. Operations API Use `set_configuration` with underscore-separated key paths: ``` { "operation": "set_configuration", "http_port": 9926, "logging_level": "warn" } ``` See [Configuration Operations](/reference/v5/configuration/operations.md) for the full `set_configuration` and `get_configuration` API reference. *** ## Configuration File Backups When `harper-config.yaml` is updated through the [Operations API](#4-operations-api) (`set_configuration`) or by passing CLI arguments / environment variables to Harper at startup, Harper writes a timestamped copy of the previous file to `{rootPath}/backup/` before applying the change. Backup files are named `-harper-config.yaml.bak`, for example: ``` hdb/backup/2026-05-06T14-22-08.123Z-harper-config.yaml.bak ``` Notes: * Backups are not produced when you edit `harper-config.yaml` directly — Harper only sees that change on its next start. * The `HARPER_CONFIG`, `HARPER_DEFAULT_CONFIG`, and `HARPER_SET_CONFIG` mechanisms do not produce timestamped backups; their provenance is tracked separately in `{rootPath}/backup/.harper-config-state.json` (see [State Tracking](#state-tracking)). * Backups are not rotated. Old `.bak` files accumulate in `{rootPath}/backup/` until you remove them. * A failure to write the backup is logged at `error` level and does not block the configuration update. *** ## Custom Config File Path To specify a custom config file location at install time, use the `HDB_CONFIG` variable: ``` # Use a custom config file path HDB_CONFIG=/path/to/custom-config.yaml harper # Install over an existing config HDB_CONFIG=/existing/rootpath/harper-config.yaml harper ``` *** ## Environment Variable-Based Configuration *Added in: v4.7.2* Harper provides three special environment variables for managing configuration across deployments: `HARPER_CONFIG`, `HARPER_DEFAULT_CONFIG`, and `HARPER_SET_CONFIG`. All accept JSON-formatted configuration that mirrors the structure of `harper-config.yaml`, and all **merge** into the configuration — they set only the keys you name, leaving every other key untouched. ``` # Recommended: set configuration from the environment export HARPER_CONFIG='{"http":{"port":8080},"logging":{"level":"info"}}' # Specialized: defaults that yield to other sources, and forced/locked values export HARPER_DEFAULT_CONFIG='{"logging":{"level":"info"}}' export HARPER_SET_CONFIG='{"authentication":{"enabled":true}}' ``` `HARPER_CONFIG` *Added in: v5.1.0* is the recommended choice for "how do I set configuration from the environment?" — it applies with least-surprise merge semantics. Reach for `HARPER_DEFAULT_CONFIG` only when you specifically need overridable defaults, and `HARPER_SET_CONFIG` only when you need to force a value and lock it against drift. ### HARPER\_CONFIG *Added in: v5.1.0* The recommended way to set configuration from the environment. It applies as a **merge on top of** the existing configuration: it sets exactly the keys you name (at any depth) and leaves everything else — including sibling keys — untouched. * Sets only the keys it names; unspecified keys at any depth are preserved * **Reasserts its keys on every restart** — while the variable names a key, that value wins over the config file and over manual edits (a hand-edit to a named key is reapplied on the next restart) * Yields only to `HARPER_SET_CONFIG` * Individual environment variables and CLI arguments (e.g. `HTTP_PORT`) still win over `HARPER_CONFIG` for the specific keys they set * When a key is removed from the variable, its original value is restored (or the key is deleted if `HARPER_CONFIG` introduced it) **Example:** ``` export HARPER_CONFIG='{"http":{"port":8080},"logging":{"level":"info"}}' harper # http.port and logging.level are set to these values on every restart. # http.securePort, other logging keys, and the rest of the config are untouched. # If an administrator edits http.port to 9000 in harper-config.yaml, # it is reset to 8080 on the next restart (the variable still names http.port). # If http.port is later removed from HARPER_CONFIG, the original value is restored. ``` ### HARPER\_DEFAULT\_CONFIG Provides default configuration values while respecting user modifications. Ideal for supplying sensible defaults without preventing administrators from customizing their instances. **At installation time:** * Overrides template default values * Respects values set by `HARPER_SET_CONFIG` * Respects values from existing config files (when using `HDB_CONFIG`) **At runtime:** * Only updates values it originally set * Detects and respects manual user edits to the config file * When a key is removed from the variable, the original value is restored **Example:** ``` export HARPER_DEFAULT_CONFIG='{"http":{"port":8080},"logging":{"level":"info"}}' harper # If an administrator manually changes the port to 9000, Harper will # detect this edit and respect it on subsequent restarts. # If http.port is removed from HARPER_DEFAULT_CONFIG later, # the port reverts to the original template default (9926). ``` ### HARPER\_SET\_CONFIG Forces configuration values that cannot be overridden by user edits. Designed for security policies, compliance requirements, or critical operational settings. **At runtime:** * Always overrides all other configuration sources * Takes precedence over user edits, file values, `HARPER_CONFIG`, and `HARPER_DEFAULT_CONFIG` * When a key is removed from the variable, it is deleted from the config (not restored) **Example:** ``` export HARPER_SET_CONFIG='{"authentication":{"enabled":true},"logging":{"level":"error","stdStreams":true}}' harper # Any change to these values in harper-config.yaml will be # overridden on the next restart. ``` ### Array values and the `$union` directive *Added in: v5.1.0* By default an array in any of these variables **replaces** the existing array wholesale — the same as a plain value. To instead **compose** an array, wrap it in a `$union` directive: ``` export HARPER_SET_CONFIG='{"tls":{"uses":{"$union":["server","operations-api"]}}}' ``` `$union` guarantees the listed items are present in the target array while preserving any entries already there — the order-preserving union of (existing ∪ listed). It is: * **Idempotent** — reapplying the same `$union` on every restart is a no-op; it never grows duplicates * **Non-destructive** — it only adds the items it names and never removes other entries, even under `HARPER_SET_CONFIG`'s drift handling This lets a platform layer guarantee required array entries (for example `tls.uses`, the list of services a TLS certificate serves) on every restart without clobbering entries an application adds. `$union` works in all three variables. A bare array (no `$union`) still replaces, which remains the default. > **Note:** Only `$`-prefixed keys that Harper recognizes as directives (currently `$union`) are treated specially. Other `$`-prefixed keys — such as a JSON Schema's `$schema` or `$ref` inside component configuration — pass through as ordinary configuration values. ### Combining the variables ``` # Recommended: set configuration from the environment export HARPER_CONFIG='{"http":{"port":8080,"cors":true},"logging":{"level":"info"}}' # Provide sensible defaults (can be overridden by admins) export HARPER_DEFAULT_CONFIG='{"operationsApi":{"network":{"port":9925}}}' # Enforce critical settings (cannot be changed) export HARPER_SET_CONFIG='{"authentication":{"enabled":true}}' ``` Because each variable holds a single string, two parties cannot both write the *same* variable without one overwriting the other. The split lets them coexist: a platform layer can pin values via `HARPER_SET_CONFIG` (with `$union` for required array entries) while an application sets its own keys via `HARPER_CONFIG`, and the two merge rather than clobbering each other. ### Configuration Precedence From highest to lowest: 1. **`HARPER_SET_CONFIG`** — Always wins; locked against drift 2. **Individual env vars / CLI arguments** (e.g. `HTTP_PORT`) — for the specific keys they set 3. **`HARPER_CONFIG`** — Reasserted on every restart; wins over the config file and user edits 4. **User manual edits** — Detected via drift detection (relative to `HARPER_DEFAULT_CONFIG`) 5. **`HARPER_DEFAULT_CONFIG`** — Applied if no user edits detected 6. **File defaults** — Original template values ### State Tracking Harper maintains a state file at `{rootPath}/backup/.harper-config-state.json` to track the source of each configuration value. This enables: * **Drift detection**: Identifying when users manually edit values set by `HARPER_DEFAULT_CONFIG` * **Restoration**: Restoring original values when keys are removed from `HARPER_CONFIG` or `HARPER_DEFAULT_CONFIG` * **Conflict resolution**: Determining which source should take precedence ### Format Reference The JSON structure mirrors the YAML config file: **YAML:** ``` http: port: 8080 cors: true logging: level: info rotation: enabled: true ``` **Environment variable (JSON):** ``` { "http": { "port": 8080, "cors": true }, "logging": { "level": "info", "rotation": { "enabled": true } } } ``` ### Important Notes * All three variables must contain valid JSON matching the structure of `harper-config.yaml` * Invalid values are caught by Harper's configuration validator at startup * Changes to these variables require a Harper restart to take effect * The state file is per-instance (stored in the root path) *** ## HARPER\_SAFE\_MODE Setting `HARPER_SAFE_MODE` to any value starts Harper without loading user applications or components. Harper's core services (database, operations API, HTTP server) start normally, but no applications from the components directory are loaded and no package-based extensions are initialized. ``` HARPER_SAFE_MODE=1 harperdb ``` This is useful when a broken or misbehaving component prevents Harper from starting. Safe mode lets you access the operations API to inspect, repair, or remove the problematic component without needing to manually edit files on disk. Safe mode skips: * Installing registered applications * Loading components from the components directory * Loading package-based extensions defined in component configs Built-in plugins (REST, HTTP, operations API, etc.) and the core database are unaffected. ## See Also * [Configuration Options](/reference/v5/configuration/options.md) — complete reference for every `harper-config.yaml` key * [Worker Thread Debugging](/reference/v5/configuration/debugging.md) — attaching Node.js inspector to Harper's worker threads * [Storage Tuning](/reference/v5/database/storage-tuning.md) — production tuning of `storage.*` options * [Logging Configuration](/reference/v5/logging/configuration.md) — main and per-subsystem log settings --- # API Harper exposes a set of global variables and functions that JavaScript code (in components, applications, and plugins) can use to interact with the database system. ## `tables` `tables` is an object whose properties are the tables in the default database (`data`). Each table defined in your `schema.graphql` file is available as a property, and the value is the table class that implements the [Resource API](/reference/v5/resources/resource-api.md). ``` # schema.graphql type Product @table { id: Long @primaryKey name: String price: Float } ``` ``` const { Product } = tables; // same as: databases.data.Product ``` ### Example ``` // Create a new record (id auto-generated) const created = await Product.create({ name: 'Shirt', price: 9.5 }); // Modify the record await Product.patch(created.id, { price: Math.round(created.price * 0.8 * 100) / 100 }); // Retrieve by primary key const record = await Product.get(created.id); // Query with conditions const query = { conditions: [{ attribute: 'price', comparator: 'less_than', value: 8.0 }], }; for await (const record of Product.search(query)) { // ... } ``` For the full set of methods available on table classes, see the [Resource API](/reference/v5/resources/resource-api.md). Data safety Programmatic `update`, `patch`, and `delete` calls operate directly on stored data, which may be live production data. A query that matches more records than intended — or a delete issued without confirmation — can be destructive and is not easily reversible. Scope destructive operations with specific conditions, validate the affected set before writing, and gate them behind your application's approval and authorization controls. ## `databases` `databases` is an object whose properties are Harper databases. Each database contains its tables as properties, the same way `tables` does for the default database. In fact, `databases.data === tables` is always true. Use `databases` when you need to access tables from a non-default database. ### Example ``` const { Product } = databases.data; // default database const Events = databases.analytics.Events; // another database // Create an event record const event = await Events.create({ eventType: 'login', timestamp: Date.now() }); // Query events for await (const e of Events.search({ conditions: [{ attribute: 'eventType', value: 'login' }] })) { // handle each event } ``` To define tables in a non-default database, use the `database` argument on the `@table` directive in your schema: ``` type Events @table(database: "analytics") { id: Long @primaryKey eventType: String @indexed } ``` See [Schema](/reference/v5/database/schema.md) for full schema definition syntax. ## `transaction(context?, callback)` `transaction` executes a callback within a database transaction and returns a promise that resolves when the transaction commits. The callback may be async. ``` transaction(context?: object, callback: (txn: Transaction) => any | Promise): Promise ``` For most operations — HTTP request handlers, for example — Harper automatically starts a transaction. Use `transaction()` explicitly when your code runs outside of a natural transaction context, such as in timers or background jobs. ### Basic Usage ``` import { tables } from 'harper'; const { MyTable } = tables; if (isMainThread) { setInterval(async () => { let data = await (await fetch('https://example.com/data')).json(); transaction(async (txn) => { for (let item of data) { await MyTable.put(item, txn); } }); }, 3600000); // every hour } ``` ### Nesting If `transaction()` is called with a context that already has an active transaction, it reuses that transaction, executes the callback immediately, and returns. This makes `transaction()` safe to call defensively to ensure a transaction is always active. ### Transaction Object The callback receives a `txn` object with the following members: | Member | Type | Description | | --------------------- | --------------- | ------------------------------------------------------ | | `commit()` | `() => Promise` | Commits the current transaction | | `abort()` | `() => void` | Aborts the transaction and resets it | | `resetReadSnapshot()` | `() => void` | Resets the read snapshot to the latest committed state | | `timestamp` | `number` | Timestamp associated with the current transaction | On normal callback completion the transaction is committed automatically. If the callback throws, the transaction is aborted. ### Transaction Scope and Atomicity Transactions span a single database. All tables within the same database share a single transactional context: reads return a consistent snapshot, and writes across multiple tables are committed atomically. If code accesses tables in different databases, each database gets its own transaction with no cross-database atomicity guarantee. For deeper background on Harper's transaction model, see [Storage Algorithm](/reference/v5/database/storage-algorithm.md). ## Read Consistency Each request reads from a consistent point-in-time snapshot. Within a single request: * a query response never mixes partially-written records; * a long or streamed scan holds the snapshot it started with, so it does not observe writes that land mid-scan; * multiple reads in the same request transaction see the same snapshot. Across separate requests there is no shared snapshot. A common pattern — querying for matching rows in one request, then fetching each by id in a later request — runs the two reads at two different points in time. If the data changes in between (a row is deleted or expires, say), the second read can disagree with the first; a query may return an id whose subsequent fetch returns a 404. This is expected: it is two independent snapshots, not a consistency bug. For repeatable-read consistency across a query and a fetch, perform both reads within a single resource method or `transaction()` callback. ## `createBlob(data, options?)` *Added in: v4.5.0* `createBlob` creates a [`Blob`](https://developer.mozilla.org/en-US/docs/Web/API/Blob) backed by Harper's storage engine. Use it to store large binary content (images, audio, video, large HTML, etc.) in a `Blob`-typed schema field. ``` createBlob(data: Buffer | Uint8Array | ReadableStream | string, options?: BlobOptions): Blob ``` Harper's `Blob` extends the [Web API `Blob`](https://developer.mozilla.org/en-US/docs/Web/API/Blob), so standard methods (`.text()`, `.arrayBuffer()`, `.stream()`, `.slice()`, `.bytes()`) are all available. Unlike `Bytes`, blobs are stored separately from the record, support streaming, and do not need to fit in memory. ### Basic Usage Declare a blob field in your schema (see [Schema — Blob Type](/reference/v5/database/schema.md#blob-type)): ``` type MyTable @table { id: Any! @primaryKey data: Blob } ``` Create and store a blob: ``` let blob = createBlob(largeBuffer); await MyTable.put({ id: 'my-record', data: blob }); ``` Retrieve blob data using standard `Blob` methods: ``` let record = await MyTable.get('my-record'); let buffer = await record.data.bytes(); // ArrayBuffer let text = await record.data.text(); // string let stream = record.data.stream(); // ReadableStream ``` ### Streaming `createBlob` supports streaming data in as data is streamed out — useful for large media where low-latency transmission from origin is critical: ``` let blob = createBlob(incomingStream); // blob exists, but data is still streaming to storage await MyTable.put({ id: 'my-record', data: blob }); let record = await MyTable.get('my-record'); // blob data is accessible as it arrives let outgoingStream = record.data.stream(); ``` Blobs are inherently a streaming API designed to work asynchronously, so they can be referenced before they are fully written and are **not** ACID-compliant by default. Use `saveBeforeCommit: true` to ensure the blob is fully saved before the transaction commits. Even then, a `Blob` is stored alongside the record rather than within it — if you need the data to be a true, ACID-committed part of the record, use a `Bytes` field instead of a `Blob`: ``` let blob = createBlob(stream, { saveBeforeCommit: true }); await MyTable.put({ id: 'my-record', data: blob }); // put() resolves only after blob is fully written and record is committed ``` ### `BlobOptions` | Option | Type | Default | Description | | ------------------ | --------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------- | | `type` | `string` | `undefined` | MIME type to associate with the blob (e.g., `image/jpeg`, `audio/mpeg`). Readable via `blob.type` and used when serving HTTP | | `size` | `number` | `undefined` | Size of the data in bytes, if known ahead of time. Otherwise inferred from a buffer or determined as a stream completes | | `saveBeforeCommit` | `boolean` | `false` | Wait until the blob is fully written before the transaction commits | | `compress` | `boolean` | `false` | Compress the stored data with deflate | | `flush` | `boolean` | `false` | Flush the file to disk after writing, before the `createBlob` promise chain resolves | Example with MIME type: ``` let blob = createBlob(imageBuffer, { type: 'image/jpeg' }); await Photo.put({ id, data: blob }); ``` ### Accepting Binary in JSON Requests REST clients that can't post raw binary typically send base64 inside JSON. Decode in the resource override and wrap with `createBlob`, recording the MIME type so it round-trips on read: ``` import { type RequestTargetOrId, tables, createBlob } from 'harper'; export class Photo extends tables.Photo { static async post(target: RequestTargetOrId, record: any) { if (record.data) { record.data = createBlob(Buffer.from(record.data, record.encoding || 'base64'), { type: record.contentType || 'application/octet-stream', }); } return super.post(target, record); } } ``` ### Serving Binary from a Resource Return a response object with the blob's MIME type in headers and the blob itself as the body. Harper will stream it to the client: ``` export class Photo extends tables.Photo { static async get(target: RequestTargetOrId) { const record = await super.get(target); if (record?.data) { return { status: 200, headers: { 'Content-Type': record.data.type || 'application/octet-stream' }, body: record.data, }; } return record; } } ``` ### Error Handling Blobs written from a stream can fail mid-stream after the record is committed. Register an error handler to respond to interrupted writes: ``` export class MyEndpoint extends MyTable { static async get(target) { const record = await super.get(target); let blob = record.data; blob.on('error', () => { MyTable.invalidate(target); }); return { status: 200, headers: {}, body: blob }; } } ``` ### `size` Property Blobs created from a stream may not have `size` available immediately. Listen for the `size` event if you need it: ``` let blob = record.data; if (blob.size === undefined) { blob.on('size', (size) => { // called once size is determined }); } ``` ### Blob Coercion When a field is typed as `Blob` in the schema, any string or buffer assigned via `put`, `patch`, or `publish` is automatically coerced to a `Blob`. This means plain JSON HTTP bodies and MQTT messages work without manual `createBlob()` calls in most cases. ## Related Documentation * [Schema](/reference/v5/database/schema.md) — Defining tables and blob fields * [Resource API](/reference/v5/resources/resource-api.md) — Full table class method reference * [Transaction Logging](/reference/v5/database/transaction.md) — Audit log and transaction log for data change history * [Configuration](/reference/v5/configuration/options.md) — Blob storage path configuration --- # Compaction *Added in: v4.3.0* Database files grow over time as records are inserted, updated, and deleted. Deleted records and updated values leave behind free space (fragmentation) in the database file, which can increase file size and potentially affect performance. Compaction eliminates this free space, creating a smaller, contiguous database file. > **Note:** Compaction does not compress your data. It removes internal fragmentation to make the file smaller. To enable compression on a database, use compaction to copy the database with updated storage configuration applied. Compaction is also the mechanism to apply storage configuration changes (such as enabling compression) to existing databases, since some storage settings cannot be changed in-place. ## Copy Compaction Creates a compacted copy of a database file. The original database is left unchanged. > **Recommendation:** Stop Harper before performing copy compaction to prevent any record loss during the copy operation. Run using the [CLI](/reference/v5/cli/commands.md): ``` harper copy-db ``` The `source-database` is the database name (not a file path). The target is the full file path where the compacted copy will be written. To replace the original database with the compacted copy, move or rename the output file to the original database path after Harper is stopped. **Example — compact the default `data` database:** ``` harper copy-db data /home/user/hdb/database/copy.mdb ``` ## Compact on Start Automatically compacts all non-system databases when Harper starts. Harper will not start until compaction is complete. Under the hood, it loops through all user databases, creates a backup of each, compacts it, replaces the original with the compacted copy, and removes the backup. Configure in `harper-config.yaml`: ``` storage: compactOnStart: true compactOnStartKeepBackup: false ``` Using CLI environment variables: ``` STORAGE_COMPACTONSTART=true STORAGE_COMPACTONSTARTKEEPBACKUP=true harper ``` ### Options | Option | Type | Default | Description | | -------------------------- | ------- | ------- | ------------------------------------------------------------------------------- | | `compactOnStart` | Boolean | `false` | Compact all databases at startup. Automatically reset to `false` after running. | | `compactOnStartKeepBackup` | Boolean | `false` | Retain the backup copy created during compact on start | > **Note:** `compactOnStart` is automatically set back to `false` after it runs, so compaction only happens on the next start if you explicitly re-enable it. ## Related Documentation * [Storage Algorithm](/reference/v5/database/storage-algorithm.md) — How Harper stores data using LMDB * [CLI Commands](/reference/v5/cli/commands.md) — `copy-db` CLI command reference * [Configuration](/reference/v5/configuration/options.md) — Full storage configuration options including compression settings --- # Data Loader *Added in: v4.6.0* The Data Loader is a built-in component that loads data from JSON or YAML files into Harper tables as part of component deployment. It is designed for seeding tables with initial records — configuration data, reference data, default users, or other records that should exist when a component is first deployed or updated. ## Configuration In your component's `config.yaml`, use the `dataLoader` key to specify the data files to load: ``` dataLoader: files: 'data/*.json' ``` `dataLoader` is a [plugin](/reference/v5/components/plugin-api.md) and supports the standard `files` configuration option, including glob patterns. ## Data File Format Each data file loads records into a single table. The file specifies the target database, table, and an array of records. ### JSON Example ``` { "database": "myapp", "table": "users", "records": [ { "id": 1, "username": "admin", "email": "admin@example.com", "role": "administrator" }, { "id": 2, "username": "user1", "email": "user1@example.com", "role": "standard" } ] } ``` ### YAML Example ``` database: myapp table: settings records: - id: 1 setting_name: app_name setting_value: My Application - id: 2 setting_name: version setting_value: '1.0.0' ``` One table per file. To load data into multiple tables, create a separate file for each table. ## File Patterns The `files` option accepts a single path, a list of paths, or a glob pattern: ``` # Single file dataLoader: files: 'data/seed-data.json' # Multiple specific files dataLoader: files: - 'data/users.json' - 'data/settings.yaml' - 'data/initial-products.json' # Glob pattern dataLoader: files: 'data/**/*.{json,yaml,yml}' ``` ## Loading Behavior The Data Loader runs on every full system start and every component deployment — this includes fresh installs, restarts of the Harper process or threads, and redeployments of the component. Because the Data Loader runs on every startup and deployment, change detection is central to how it works safely. On each run: 1. All specified data files are read (JSON or YAML) 2. Each file is validated to reference a single table 3. Records are inserted or updated based on content hash comparison: * New records are inserted if they don't exist * Existing records are updated only if the data file content has changed * Records created outside the Data Loader (via Operations API, REST, etc.) are never overwritten * Records modified by users after being loaded are preserved and not overwritten * Extra fields added by users to data-loaded records are preserved during updates 4. SHA-256 content hashes are stored in the [`hdb_dataloader_hash`](/reference/v5/database/system-tables.md#hdb_dataloader_hash) system table to track which records have been loaded and detect changes ### Change Detection | Scenario | Behavior | | -------------------------------------------------- | ------------------------------------------------ | | New record | Inserted; content hash stored | | Unchanged record | Skipped (no writes) | | Changed data file | Updated via `patch`, preserving any extra fields | | Record created by user (not data loader) | Never overwritten | | Record modified by user after load | Preserved, not overwritten | | Extra fields added by user to a data-loaded record | Preserved during updates | This design makes data files safe to redeploy repeatedly — across deployments, node scaling, and system restarts — without losing manual modifications or causing unnecessary writes. ## Best Practices **Define schemas first.** While the Data Loader can infer schemas from the records it loads, it is strongly recommended to define table schemas explicitly using the [graphqlSchema component](/reference/v5/database/schema.md) before loading data. This ensures proper types, constraints, and relationships. **One table per file.** Each data file must target a single table. Organize files accordingly. **Idempotent data.** Design files to be safe to load multiple times without creating duplicate or conflicting records. **Version control.** Include data files in version control for consistency across deployments and environments. **Environment-specific data.** Consider using different data files for different environments (development, staging, production) to avoid loading inappropriate records. **Validate before deploying.** Ensure data files are valid JSON or YAML and match your table schemas before deployment to catch type mismatches early. **No sensitive data.** Do not include passwords, API keys, or secrets directly in data files. Use environment variables or secure configuration management instead. ## Example Component Structure A common production use case is shipping reference data — lookup tables like countries and regions — as part of a component. The records are version-controlled alongside the code, consistent across every environment, and the data loader keeps them in sync on every deployment without touching any user-modified fields. ``` my-component/ ├── config.yaml ├── schemas.graphql ├── roles.yaml └── data/ ├── countries.json # ISO country codes — reference data, ships with component └── regions.json # region/subdivision codes ``` **`config.yaml`**: ``` graphqlSchema: files: 'schemas.graphql' roles: files: 'roles.yaml' dataLoader: files: 'data/*.json' rest: true ``` **`schemas.graphql`**: ``` type Country @table(database: "myapp") @export { id: String @primaryKey # ISO 3166-1 alpha-2, e.g. "US" name: String @indexed region: String @indexed } type Region @table(database: "myapp") @export { id: String @primaryKey # ISO 3166-2, e.g. "US-CA" name: String @indexed countryId: String @indexed country: Country @relationship(from: countryId) } ``` **`data/countries.json`**: ``` { "database": "myapp", "table": "Country", "records": [ { "id": "US", "name": "United States", "region": "Americas" }, { "id": "GB", "name": "United Kingdom", "region": "Europe" }, { "id": "DE", "name": "Germany", "region": "Europe" } // ... all ~250 ISO countries ] } ``` **`data/regions.json`**: ``` { "database": "myapp", "table": "Region", "records": [ { "id": "US-CA", "name": "California", "countryId": "US" }, { "id": "US-NY", "name": "New York", "countryId": "US" }, { "id": "GB-ENG", "name": "England", "countryId": "GB" } // ... ] } ``` Because the data loader uses content hashing, adding new countries or correcting a name in the file will update only the changed records on the next deployment — existing records that haven't changed are skipped entirely. ## Related Documentation * [Schema](/reference/v5/database/schema.md) — Defining table structure before loading data * [Jobs](/reference/v5/database/jobs.md) — Bulk data operations via the Operations API (CSV/JSON import from file, URL, or S3) * [Components](/reference/v5/components/overview.md) — Plugin system that the data loader is built on --- # Jobs Harper uses an asynchronous job system for long-running data operations. When a bulk operation is initiated — such as loading a large CSV file or exporting millions of records — Harper starts a background job and immediately returns a job ID. Use the job ID to check progress and status. Job status values: * `IN_PROGRESS` — the job is currently running * `COMPLETE` — the job finished successfully ## Bulk Operations The following operations create jobs. All bulk operations are sent to the [Operations API](/reference/v5/operations-api/overview.md). ### CSV Data Load Ingests CSV data provided directly in the request body. * `operation` *(required)* — `csv_data_load` * `database` *(optional)* — target database; defaults to `data` * `table` *(required)* — target table * `action` *(optional)* — `insert`, `update`, or `upsert`; defaults to `insert` * `data` *(required)* — CSV content as a string ``` { "operation": "csv_data_load", "database": "dev", "action": "insert", "table": "breed", "data": "id,name,country\n1,Labrador,Canada\n2,Poodle,France\n" } ``` Response: ``` { "message": "Starting job with id 2fe25039-566e-4670-8bb3-2db3d4e07e69", "job_id": "2fe25039-566e-4670-8bb3-2db3d4e07e69" } ``` *** ### CSV File Load Ingests CSV data from a file on the server's local filesystem. > The CSV file must reside on the same machine running Harper. * `operation` *(required)* — `csv_file_load` * `database` *(optional)* — target database; defaults to `data` * `table` *(required)* — target table * `action` *(optional)* — `insert`, `update`, or `upsert`; defaults to `insert` * `file_path` *(required)* — absolute path to the CSV file on the host ``` { "operation": "csv_file_load", "action": "insert", "database": "dev", "table": "breed", "file_path": "/home/user/imports/breeds.csv" } ``` *** ### CSV URL Load Ingests CSV data from a URL. * `operation` *(required)* — `csv_url_load` * `database` *(optional)* — target database; defaults to `data` * `table` *(required)* — target table * `action` *(optional)* — `insert`, `update`, or `upsert`; defaults to `insert` * `csv_url` *(required)* — URL pointing to the CSV file ``` { "operation": "csv_url_load", "action": "insert", "database": "dev", "table": "breed", "csv_url": "https://s3.amazonaws.com/mydata/breeds.csv" } ``` *** ### Import from S3 Imports CSV or JSON files from an AWS S3 bucket. * `operation` *(required)* — `import_from_s3` * `database` *(optional)* — target database; defaults to `data` * `table` *(required)* — target table * `action` *(optional)* — `insert`, `update`, or `upsert`; defaults to `insert` * `s3` *(required)* — S3 connection details: * `aws_access_key_id` * `aws_secret_access_key` * `bucket` * `key` — filename including extension (`.csv` or `.json`) * `region` ``` { "operation": "import_from_s3", "action": "insert", "database": "dev", "table": "dog", "s3": { "aws_access_key_id": "YOUR_KEY", "aws_secret_access_key": "YOUR_SECRET_KEY", "bucket": "BUCKET_NAME", "key": "dogs.json", "region": "us-east-1" } } ``` *** ### Export Local Exports table data to a local file in JSON or CSV format. * `operation` *(required)* — `export_local` * `format` *(required)* — `json` or `csv` * `path` *(required)* — local directory path where the export file will be written * `search_operation` *(required)* — query to select records: `search_by_hash`, `search_by_value`, `search_by_conditions`, or `sql` *Changed in: v4.3.0* — `search_by_conditions` added as a supported search operation for exports * `filename` *(optional)* — filename without extension; auto-generated from epoch timestamp if omitted ``` { "operation": "export_local", "format": "json", "path": "/data/exports/", "search_operation": { "operation": "sql", "sql": "SELECT * FROM dev.breed" } } ``` *** ### Export to S3 Exports table data to an AWS S3 bucket in JSON or CSV format. *Changed in: v4.3.0* — `search_by_conditions` added as a supported search operation * `operation` *(required)* — `export_to_s3` * `format` *(required)* — `json` or `csv` * `s3` *(required)* — S3 connection details (same fields as Import from S3, plus `key` for the output object name) * `search_operation` *(required)* — `search_by_hash`, `search_by_value`, `search_by_conditions`, or `sql` ``` { "operation": "export_to_s3", "format": "json", "s3": { "aws_access_key_id": "YOUR_KEY", "aws_secret_access_key": "YOUR_SECRET_KEY", "bucket": "BUCKET_NAME", "key": "exports/dogs.json", "region": "us-east-1" }, "search_operation": { "operation": "sql", "sql": "SELECT * FROM dev.dog" } } ``` *** ### Delete Records Before Deletes records older than a given timestamp from a table. Operates only on the local node — clustered replicas retain their data. *Restricted to `super_user` roles.* * `operation` *(required)* — `delete_records_before` * `schema` *(required)* — database name * `table` *(required)* — table name * `date` *(required)* — records with `__createdtime__` before this timestamp are deleted. Format: `YYYY-MM-DDThh:mm:ss.sZ` ``` { "operation": "delete_records_before", "date": "2024-01-01T00:00:00.000Z", "schema": "dev", "table": "breed" } ``` ## Managing Jobs ### Get Job Returns status, metrics, and messages for a specific job by ID. * `operation` *(required)* — `get_job` * `id` *(required)* — job ID ``` { "operation": "get_job", "id": "4a982782-929a-4507-8794-26dae1132def" } ``` Response: ``` [ { "__createdtime__": 1611615798782, "__updatedtime__": 1611615801207, "created_datetime": 1611615798774, "end_datetime": 1611615801206, "id": "4a982782-929a-4507-8794-26dae1132def", "job_body": null, "message": "successfully loaded 350 of 350 records", "start_datetime": 1611615798805, "status": "COMPLETE", "type": "csv_url_load", "user": "HDB_ADMIN", "start_datetime_converted": "2021-01-25T23:03:18.805Z", "end_datetime_converted": "2021-01-25T23:03:21.206Z" } ] ``` *** ### Search Jobs by Start Date Returns all jobs started within a time window. *Restricted to `super_user` roles.* * `operation` *(required)* — `search_jobs_by_start_date` * `from_date` *(required)* — start of the search window (ISO 8601 format) * `to_date` *(required)* — end of the search window (ISO 8601 format) ``` { "operation": "search_jobs_by_start_date", "from_date": "2024-01-01T00:00:00.000+0000", "to_date": "2024-01-02T00:00:00.000+0000" } ``` ## Related Documentation * [Data Loader](/reference/v5/database/data-loader.md) — Component-based data loading as part of deployment * [Operations API](/reference/v5/operations-api/overview.md) — Sending operations to Harper * [Transaction Logging](/reference/v5/database/transaction.md) — Recording a history of changes made to tables --- # Database Harper's database system is the foundation of its data storage and retrieval capabilities. Harper supports two storage enginers, [RocksDB](https://github.com/facebook/rocksdb/wiki/RocksDB-Overview) and [LMDB](https://www.symas.com/lmdb) (Lightning Memory-Mapped Database) and is designed to provide high performance, ACID-compliant storage with indexing and flexible schema support. ## How Harper Stores Data Harper organizes data in a three-tier hierarchy: * **Databases** — containers that group related tables together in a single transactional file * **Tables** — collections of records with a common data pattern * **Records** — individual data objects with a primary key and any number of attributes All tables within a database share the same transaction context, meaning reads and writes across tables in the same database can be performed atomically. ### The Schema System and Auto-REST The most common way to use Harper's database is through the **schema system**. By defining a [GraphQL schema](/reference/v5/database/schema.md), you can: * Declare tables and their attribute types * Control which attributes are indexed * Define relationships between tables * Automatically expose data via REST, MQTT, and other interfaces You do not need to build custom application code to use the database. A schema definition alone is enough to create fully functional, queryable REST endpoints for your data. For more advanced use cases, you can extend table behavior using the [Resource API](/reference/v5/resources/resource-api.md). ### Architecture Overview ``` ┌──────────┐ ┌──────────┐ │ Clients │ │ Clients │ └────┬─────┘ └────┬─────┘ │ │ ▼ ▼ ┌────────────────────────────────────────┐ │ │ │ Socket routing/management │ ├───────────────────────┬────────────────┤ │ │ │ │ Server Interfaces ─►│ Authentication │ │ RESTful HTTP, MQTT │ Authorization │ │ ◄─┤ │ │ ▲ └────────────────┤ │ │ │ │ ├───┼──────────┼─────────────────────────┤ │ │ │ ▲ │ │ ▼ Resources ▲ │ ┌───────────┐ │ │ │ └─┤ │ │ ├─────────────────┴────┐ │ App │ │ │ ├─►│ resources │ │ │ Database tables │ └───────────┘ │ │ │ ▲ │ ├──────────────────────┘ │ │ │ ▲ ▼ │ │ │ ┌────────────────┐ │ │ │ │ External │ │ │ │ │ data sources ├────┘ │ │ │ │ │ │ └────────────────┘ │ │ │ └────────────────────────────────────────┘ ``` ## Databases Harper databases hold a collection of tables in a single transactionally-consistent file. This means reads and writes can be performed atomically across all tables in the same database, and multi-table transactions are replicated as a single atomic unit. The default database is named `data`. Most applications will use this default. Additional databases can be created for namespace separation — this is particularly useful for components designed for reuse across multiple applications, where a unique database name avoids naming collisions. > **Note:** Transactions do not preserve atomicity across different databases, only across tables within the same database. ## Tables Tables group records with a common data pattern. A table must have: * **Table name** — used to identify the table * **Primary key** — the unique identifier for each record (also referred to as `hash_attribute` in the Operations API) Primary keys must be unique. If a primary key is not provided on insert, Harper auto-generates one: * A **UUID string** for primary keys typed as `String` or `ID` * An **auto-incrementing integer** for primary keys typed as `Int`, `Long`, or `Any` Numeric primary keys are more efficient than UUIDs for large tables. ## Dynamic vs. Defined Schemas Harper tables can operate in two modes: **Defined schemas** (recommended): Tables with schemas explicitly declared using [GraphQL schema syntax](/reference/v5/database/schema.md). This provides predictable structure, precise control over indexing, and data integrity. Schemas are declared in a component's `schema.graphql` file. **Dynamic schemas**: Tables created through the Operations API or Studio without a schema definition. Attributes are reflexively added as data is ingested. All top-level attributes are automatically indexed. Dynamic schema tables automatically maintain `__createdtime__` and `__updatedtime__` audit attributes on every record. It is best practice to define schemas for production tables. Dynamic schemas are convenient for experimentation and prototyping. ## Key Concepts For deeper coverage of each database feature, see the dedicated pages in this section: * **[Schema](/reference/v5/database/schema.md)** — Defining table structure, types, indexes, relationships, and computed properties using GraphQL schema syntax * **[API](/reference/v5/database/api.md)** — The `tables`, `databases`, `transaction()`, and `createBlob()` globals for interacting with the database from code * **[Data Loader](/reference/v5/database/data-loader.md)** — Loading seed or initial data into tables as part of component deployment * **[Storage Algorithm](/reference/v5/database/storage-algorithm.md)** — How Harper stores data using LMDB with universal indexing and ACID compliance * **[Storage Tuning](/reference/v5/database/storage-tuning.md)** — Tuning `storage.*` options for production: durability vs. throughput, compression, blob paths, reclamation * **[Jobs](/reference/v5/database/jobs.md)** — Asynchronous bulk data operations (CSV import/export, S3 import/export) * **[System Tables](/reference/v5/database/system-tables.md)** — Harper internal tables for analytics, data loader state, and other system features * **[Compaction](/reference/v5/database/compaction.md)** — Reducing database file size by eliminating fragmentation and free space * **[Transaction Logging](/reference/v5/database/transaction.md)** — Recording and querying a history of data changes via audit log and transaction log ## Related Documentation * [REST](/reference/v5/rest/overview.md) — HTTP interface built on top of the database resource system * [Resources](/reference/v5/resources/overview.md) — Custom application logic extending database tables * [Operations API](/reference/v5/operations-api/overview.md) — Direct database management operations (create/drop databases and tables, insert/update/delete records) * [Configuration](/reference/v5/configuration/overview.md) — Storage configuration options (compression, blob paths, compaction) --- # Schema Harper uses GraphQL Schema Definition Language (SDL) to declaratively define table structure. Schema definitions are loaded from `.graphql` files in a component directory and control table creation, attribute types, indexing, and relationships. ## Overview *Added in: v4.2.0* Schemas are defined using standard [GraphQL type definitions](https://graphql.org/learn/schema/) with Harper-specific directives. A schema definition: * Ensures required tables exist when a component is deployed * Enforces attribute types and required constraints * Controls which attributes are indexed * Defines relationships between tables * Configures computed properties, expiration, and audit behavior Schemas are flexible by default — records may include additional properties beyond those declared in the schema. Use the `@sealed` directive to prevent this. A minimal example: ``` type Dog @table { id: Long @primaryKey name: String breed: String age: Int } type Breed @table { id: Long @primaryKey name: String @indexed } ``` ### Loading Schemas In a component's `config.yaml`, specify the schema file with the `graphqlSchema` plugin: ``` graphqlSchema: files: 'schema.graphql' ``` Keep in mind that both plugins and applications can specify schemas. ## Type Directives Type directives apply to the entire table type definition. ### `@table` Marks a GraphQL type as a Harper database table. The type name becomes the table name by default. ``` type MyTable @table { id: Long @primaryKey } ``` Optional arguments: | Argument | Type | Default | Description | | -------------- | --------- | ----------------------------- | --------------------------------------------------------------------------- | | `table` | `String` | type name | Override the table name | | `database` | `String` | `"data"` | Database to place the table in | | `expiration` | `Int` | — | Seconds until a record goes stale (useful for caching tables) | | `eviction` | `Int` | `0` | Additional seconds after `expiration` before a record is physically removed | | `scanInterval` | `Int` | `(expiration + eviction) / 4` | Seconds between eviction scans | | `replicate` | `Boolean` | true | Enable replication of this table | **`expiration`, `eviction`, and `scanInterval`** These three arguments work together to control the full lifecycle of a cached record: * **`expiration`** — When elapsed, a record is considered *stale*. The next request for a stale record triggers a fetch from the source. The record may still be served while revalidation is in progress. * **`eviction`** — Additional time after `expiration` before the record is physically removed from the table. Setting `eviction > 0` lets you serve the stale record while revalidation happens and controls how long after expiration the data is kept on disk. * **`scanInterval`** — How often Harper scans the table for records to evict. Defaults to one quarter of `expiration + eviction`. You can provide a single `expiration` value and all three behaviors share the same TTL. To tune them independently: ``` # Expire after 5 minutes, evict after 1 hour, scan every 10 minutes type WeatherCache @table(expiration: 300, eviction: 3300, scanInterval: 600) { id: ID @primaryKey temperature: Float } ``` #### How `scanInterval` Determines the Eviction Cycle `scanInterval` determines fixed clock-aligned times when eviction runs. Harper divides the clock into evenly spaced anchors based on the interval, calculated in the server's local timezone. As a result: * The server's startup time does not affect when eviction runs. * Eviction timings are deterministic and timezone-aware. * For any given configuration, the eviction schedule is the same across restarts and across servers in the same local timezone. **Example: 1-hour expiration** — default `scanInterval` = 15 minutes (one quarter of `expiration`). Eviction schedule: ``` 00:00, 00:15, 00:30, 00:45, 01:00, ... ``` If the server starts at 12:05, the first eviction runs at 12:15 — not 12:20. The schedule is clock-aligned, not startup-aligned. **Example: 1-day expiration** — default `scanInterval` = 6 hours. Eviction schedule: ``` 00:00, 06:00, 12:00, 18:00, ... ``` #### Eviction with Indexing Eviction removes non-indexed record data, but it does *not* remove a record from its secondary indexes. If an evicted record matches a search query, Harper fetches the full record from the source on demand to satisfy the query. This means indexes remain fully functional even when most of the data has been evicted. **Examples:** ``` # Override table name type Product @table(table: "products") { id: Long @primaryKey } # Place in a specific database type Order @table(database: "commerce") { id: Long @primaryKey } # Auto-expire records after 1 hour (e.g., a session cache) type Session @table(expiration: 3600) { id: Long @primaryKey userId: String } # Disable replication for this table explicitly type LocalRecord @table(replicate: false) { id: Long @primaryKey value: String } # Combine multiple arguments type Event @table(database: "analytics", expiration: 86400) { id: Long @primaryKey name: String @indexed } ``` **Database naming:** Since all tables default to the `data` database, when designing plugins or applications, consider using unique database names to avoid table naming collisions. **Replication:** Replication is enabled by default for all tables. Note that if you disable replication on a table and re-enable it later, it will not catch-up on previous writes during when the replication was disabled. ### `@export` Exposes the table as an externally accessible resource endpoint, available via REST, MQTT, and other interfaces. ``` type MyTable @table @export(name: "my-table") { id: Long @primaryKey } ``` The optional `name` parameter specifies the URL path segment (e.g., `/my-table/`). Without `name`, the type name is used. ### `@sealed` Prevents records from including any properties beyond those explicitly declared in the type. By default, Harper allows records to have additional properties. ``` type StrictRecord @table @sealed { id: Long @primaryKey name: String } ``` ### `@hidden` (Type Directive) Suppresses the type from introspectable surfaces — MCP tool descriptors and the OpenAPI document. The table still exists; data is still queryable through Harper's other interfaces subject to RBAC. `@hidden` is a **metadata-visibility** directive, not an access-control mechanism: use `attribute_permissions` on roles to control data access. ``` type InternalConfig @table @hidden { id: Long @primaryKey value: String } ``` `@hidden` is also available as a [field directive](#hidden-field-directive) to suppress individual attributes. ## Documenting Types and Fields Harper picks up GraphQL's standard triple-quoted docstrings on type and field definitions. Docstrings flow through to: * **MCP** — `Table.description` (consumed as a prefix on every verb-tool description) and `inputSchema.properties[*].description` on derived tool schemas * **OpenAPI** — `components.schemas[*].description`, per-property `description`, and the path-level `description` for every verb on the resource ``` """ Product catalog row — what shows up in the storefront listing, search, and inventory feeds. One row per SKU. """ type Product @table @export { """ Stock keeping unit — globally unique across catalogs. """ sku: String! @primaryKey """ Display name shown in the storefront. """ name: String! """ Retail price in cents (USD). """ priceCents: Int! } ``` Docstrings on `@hidden` fields are dropped from the descriptive surfaces alongside the field itself. > **Trust model.** Docstrings reach LLMs and public OpenAPI consumers verbatim. Treat them as code: don't put secrets, internal-only commentary, or speculative prose in them. Use `@hidden` to suppress fields that shouldn't surface publicly. ## Field Directives Field directives apply to individual attributes in a type definition. ### `@primaryKey` Designates the attribute as the table's primary key. Primary keys must be unique; inserts with a duplicate primary key are rejected. ``` type Product @table { id: Long @primaryKey name: String } ``` If no primary key is provided on insert, Harper auto-generates one: * **UUID string** — when type is `String` or `ID` * **Auto-incrementing integer** — when type is `Int`, `Long`, or `Any` *Changed in: v4.4.0* Auto-incrementing integer primary keys were added. Previously only UUID generation was supported for `ID` and `String` types. Using `Long` or `Any` is recommended for auto-generated numeric keys. `Int` is limited to 32-bit and may be insufficient for large tables. ### `@indexed` Creates a secondary index on the attribute for fast querying. Required for filtering by this attribute in REST queries, SQL, or NoSQL operations. ``` type Product @table { id: Long @primaryKey category: String @indexed price: Float @indexed } ``` If the field value is an array, each element in the array is individually indexed, enabling queries by any individual value. Null values are indexed by default (added in v4.3.0), enabling queries like `GET /Product/?category=null`. ### `@embed` *Added in: v5.1.0* Automatically computes an embedding vector for the attribute whenever the source field is written, using a configured [embedding model](/reference/v5/models/overview.md): ``` type Document @table { id: Long @primaryKey text: String embedding: [Float] @embed(source: "text", model: "default") } ``` * `source` — the name of the field to embed. Must be a declared field on the same type, passed as a string literal. * `model` — the logical name of a configured embedding model, passed as a string literal. The attribute type must be `[Float]`. The attribute is automatically indexed with an [HNSW vector index](#vector-indexing), so it is immediately searchable by similarity; an explicit `@indexed` on the same attribute is allowed only if it is also HNSW. Write semantics: * Creating a record with the source field, or updating the source field, computes the vector before the write commits (with `inputType: 'document'`). A failure to compute the embedding fails the write. * An update that does not touch the source field leaves the vector unchanged. * Setting the source field to `null` sets the vector to `null`. * Replicated writes and audit-log replays do not re-embed — the vector travels with the record, and only the node that accepted the original write calls the model. Multiple `@embed` attributes on one type are computed concurrently. ### `@createdTime` Automatically assigns a creation timestamp (Unix epoch milliseconds) to the attribute when a record is created. ``` type Event @table { id: Long @primaryKey createdAt: Long @createdTime } ``` ### `@updatedTime` Automatically assigns a timestamp (Unix epoch milliseconds) each time the record is updated. ``` type Event @table { id: Long @primaryKey updatedAt: Long @updatedTime } ``` ### `@hidden` (Field Directive) Suppresses the field from MCP tool descriptors and the OpenAPI document. The attribute still exists in the table; data is still queryable through other interfaces subject to RBAC. Use this for fields that should not appear in introspectable surfaces. ``` type Customer @table { id: Long @primaryKey name: String """ Internal — do not surface to external consumers. """ creditScore: Int @hidden } ``` `@hidden` is a metadata-visibility directive, not access control: `attribute_permissions` on roles remains the data-access enforcement mechanism. ## Relationships *Added in: v4.3.0* The `@relationship` directive defines how one table relates to another through a foreign key. Relationships enable join queries and allow related records to be selected as nested properties in query results. ### `@relationship(from: attribute)` — many-to-one or many-to-many The foreign key is in this table, referencing the primary key of the target table. ``` type RealityShow @table @export { id: Long @primaryKey networkId: Long @indexed # foreign key network: Network @relationship(from: networkId) # many-to-one title: String @indexed } type Network @table @export { id: Long @primaryKey name: String @indexed # e.g. "Bravo", "Peacock", "Netflix" } ``` Query shows by network name: ``` GET /RealityShow?network.name=Bravo ``` If the foreign key is an array, this establishes a many-to-many relationship (e.g., a show with multiple streaming homes): ``` type RealityShow @table @export { id: Long @primaryKey networkIds: [Long] @indexed networks: [Network] @relationship(from: networkIds) } ``` ### `@relationship(to: attribute)` — one-to-many or many-to-many The foreign key is in the target table, referencing the primary key of this table. The result type must be an array. ``` type Network @table @export { id: Long @primaryKey name: String @indexed # e.g. "Bravo", "Peacock", "Netflix" shows: [RealityShow] @relationship(to: networkId) # one-to-many # shows like "Real Housewives of Atlanta", "The Traitors", "Vanderpump Rules" } ``` ### `@relationship(from: attribute, to: attribute)` — foreign key to foreign key Both `from` and `to` can be specified together to define a relationship where neither side uses the primary key — a foreign key to foreign key join. This is useful for many-to-many relationships that join on non-primary-key attributes. ``` type OrderItem @table @export { id: Long @primaryKey orderId: Long @indexed productSku: Long @indexed product: Product @relationship(from: productSku, to: sku) # join on sku, not primary key } type Product @table @export { id: Long @primaryKey sku: Long @indexed name: String } ``` Schemas can also define self-referential relationships, enabling parent-child hierarchies within a single table. ## Computed Properties *Added in: v4.4.0* The `@computed` directive marks a field as derived from other fields at query time. Computed properties are not stored in the database but are evaluated when the field is accessed. ``` type Product @table { id: Long @primaryKey price: Float taxRate: Float totalPrice: Float @computed(from: "price + (price * taxRate)") } ``` The `from` argument is a JavaScript expression that can reference other record fields. Computed properties can also be defined in JavaScript for complex logic: ``` type Product @table { id: Long @primaryKey totalPrice: Float @computed } ``` ``` tables.Product.setComputedAttribute('totalPrice', (record) => { return record.price + record.price * record.taxRate; }); ``` Computed properties are not included in query results by default — use `select` to include them explicitly. ### Computed Indexes Computed properties can be indexed with `@indexed`, enabling custom indexing strategies such as composite indexes, full-text search, or vector indexing: ``` type Product @table { id: Long @primaryKey tags: String tagsSeparated: String[] @computed(from: "tags.split(/\\s*,\\s*/)") @indexed } ``` When using a JavaScript function for an indexed computed property, use the `version` argument to ensure re-indexing when the function changes: ``` type Product @table { id: Long @primaryKey totalPrice: Float @computed(version: 1) @indexed } ``` Increment `version` whenever the computation function changes. Failing to do so can result in an inconsistent index. ## Vector Indexing *Added in: v4.6.0* Use `@indexed(type: "HNSW")` to create a vector index using the Hierarchical Navigable Small World algorithm, designed for fast approximate nearest-neighbor search on high-dimensional vectors. ``` type Document @table { id: Long @primaryKey textEmbeddings: [Float] @indexed(type: "HNSW") } ``` Embedding vectors can also be computed automatically at write time from a text field with the [`@embed` directive](#embed), which creates the HNSW index implicitly. Query by nearest neighbors using the `sort` parameter: ``` let results = Document.search({ sort: { attribute: 'textEmbeddings', target: searchVector }, limit: 5, }); ``` HNSW can be combined with filter conditions: ``` let results = Document.search({ conditions: [{ attribute: 'price', comparator: 'lt', value: 50 }], sort: { attribute: 'textEmbeddings', target: searchVector }, limit: 5, }); ``` *Changed in: v5.2.0* — Conditions combined with a vector sort are evaluated *during* graph traversal (predicate-aware search): the search keeps exploring until it has enough *matching* nearest neighbors, instead of finding the nearest candidates first and then dropping the ones that fail the filter. With a selective filter this is the difference between a full result set and an under-filled one. When a companion condition is very selective, Harper instead computes exact distances over just the records matching that condition, which is both exact and faster than traversing the graph. ### Filtered Vector Search with a Function Predicate *Added in: v5.2.0* A `vectorFilter` function on the query participates in the traversal the same way, for predicates that are not expressible as conditions: ``` let results = Document.search( { sort: { attribute: 'textEmbeddings', target: searchVector }, vectorFilter: (record) => record.tenantId === context.user.tenantId && record.status === 'published', limit: 10, }, context ); ``` `vectorFilter` is available from the JavaScript API only (it cannot be expressed in a REST query string). The function receives the candidate record and must return a boolean — `true` to include the record in results, `false` to exclude it (it still routes traversal either way). It must be synchronous, side-effect free, and fast — it can run once per candidate record visited during traversal (verdicts are memoized per query). Records passed to it are frozen. ### Record-Level Access Control (Record-Scoped `allowRead`) *Added in: v5.2.0* Overriding `allowRead(user, target, context)` on a table resource makes it a **record-scoped** check: during query execution it is evaluated once per record with `this` bound to the record, so row-level logic reads naturally from `this`. For vector queries the check participates in the graph traversal, so a restricted user receives the k nearest records *they are allowed to see* rather than "nearest k, minus redacted" (which under-fills results and reveals that nearby restricted records exist): ``` export class Reports extends tables.Reports { allowRead(user, target, context) { // Compose the table/RBAC grant first, so losing the role's read denies (at request entry and // when a live subscription is re-authorized). super.allowRead is safe to call at any scope. if (!super.allowRead(user, target, context)) return false; if (user.role.permission.super_user) return true; // Collection scope — a whole-table subscribe or the subscription re-auth check — has no record // loaded (`this.ownerId` is undefined). Return true to open the connection; rows are filtered // per record during delivery / query execution. if (this.ownerId == null) return true; return this.ownerId === user.id; // per record } } ``` How the one definition applies at each scope (when permission checking is active, e.g. any external request): * **Collection queries** (REST collection `GET`, `search()`, including vector sorts) — evaluated per record; rows failing the check are filtered out of results. The default (non-overridden) `allowRead` is a table-level RBAC check and continues to run once at request entry with no per-record cost. * **Single-record `get(id)`** — evaluated at request entry with the record loaded (attribute reads like `this.ownerId` resolve against the record); a denied record returns a 403. * **Subscriptions** (SSE, WebSocket, MQTT) — the entry check grants the connection at subscribe time (evaluated at collection scope — return the base grant to open), then delivery is filtered per event so a subscriber receives only the row-change events for records the check permits. A live subscription is periodically re-authorized by re-running this same `allowRead` against the current user, so revoking the role's read (or, for a connection-level override, its grant) tears the subscription down. Delete tombstones and published message payloads do not carry the full record, so an override keyed on row fields will deny those event types. Constraints: the check must be synchronous, side-effect free, and fast — it can run once per candidate record visited during traversal (verdicts are memoized per query). `this` is the frozen record during per-record evaluation. A thrown exception denies that record (fail closed). On caching tables the check is enforced against the record actually returned, after any source revalidation. ### Tuning Filtered Traversal Filtered traversal is bounded by a visit budget of `ef * filterExpansion` nodes (`filterExpansion` defaults to 24). If the budget is exhausted before the result list fills — which happens when the filter matches only a tiny fraction of records — the search returns the matches found so far rather than erroring. Both knobs can be set per query: ``` let results = Document.search( { sort: { attribute: 'textEmbeddings', target: searchVector, ef: 200, filterExpansion: 40 }, vectorFilter: (record) => record.category === 'rare', limit: 10, }, context ); ``` Raise `filterExpansion` (or `ef`) to trade latency for recall under selective function predicates. Condition-based filters rarely need tuning: very selective conditions are automatically diverted to the exact-scan strategy instead of graph traversal. ### Filtering by Distance Threshold To return only records whose distance to a target vector is below a threshold, place `target` directly on the condition (alongside `comparator` and `value`). This returns matches within the threshold without using `sort`: ``` let results = Document.search({ conditions: { attribute: 'textEmbeddings', comparator: 'lt', value: 0.1, target: searchVector, }, }); ``` This form is useful when you want to bound result quality by a similarity cutoff rather than ranking by similarity. ### Selecting the Distance Use the special `$distance` field in `select` to include the computed distance from the target vector in returned records: ``` let results = Document.search({ select: ['name', '$distance'], sort: { attribute: 'textEmbeddings', target: searchVector }, limit: 5, }); ``` `$distance` is available in both `sort`-based ranking and `conditions`-based threshold queries. ### Per-Query Search Options The `sort` descriptor (and threshold condition) accepts options that tune an individual query: ``` let results = Document.search({ sort: { attribute: 'textEmbeddings', target: searchVector, distance: 'dotProduct', ef: 200 }, limit: 5, }); ``` * `distance` — overrides the index's distance function for this query: `"cosine"`, `"euclidean"`, or `"dotProduct"` (`dotProduct` *Added in: v5.1.0*). * `ef` *Added in: v5.1.0* — overrides the search exploration budget for this query. Higher values improve recall at the cost of latency. *Changed in: v5.1.0* — When a query passes no `ef` and the index does not explicitly configure `efConstructionSearch` (or `efConstruction`), the search budget auto-scales with the size of the index, so recall holds as the table grows instead of decaying with a fixed budget. ### HNSW Parameters | Parameter | Default | Description | | ---------------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `distance` | `"cosine"` | Distance function: `"cosine"` (negative cosine similarity), `"euclidean"`, or `"dotProduct"` (added in v5.1.0) | | `efConstruction` | `100` | Max nodes explored during index construction. Higher = better recall, lower = better performance | | `M` | `16` | Preferred connections per graph layer. Higher = more space, better recall for high-dimensional data | | `optimizeRouting` | `0.5` | Heuristic aggressiveness for omitting redundant connections (0 = off, 1 = most aggressive) | | `mL` | computed from `M` | Normalization factor for level generation | | `efConstructionSearch` | auto-scaled | Max nodes explored during search. When unset, auto-scales with index size (see above); setting it (or `efConstruction`, which seeds it) fixes the budget | | `quantization` | — | `"int8"` stores vectors quantized to int8 (added in v5.1.0, see below) | | `filterExpansion` | `24` | Visit-budget multiplier for filtered (predicate-aware) search: a filtered query visits at most `ef * filterExpansion` nodes (added in v5.2.0, see above) | Example with custom parameters: ``` type Document @table { id: Long @primaryKey textEmbeddings: [Float] @indexed(type: "HNSW", distance: "euclidean", optimizeRouting: 0, efConstructionSearch: 100) } ``` Note: this parameter was previously documented as `efSearchConstruction`; the option name Harper reads is `efConstructionSearch`. *Changed in: v5.1.0* — Changing `efConstructionSearch` on an existing index no longer triggers a rebuild; it only affects searches. Structural parameters (`distance`, `M`, `efConstruction`, `quantization`) still rebuild the index when changed. ### Vector Quantization *Added in: v5.1.0* `quantization: "int8"` stores the index's vectors quantized to 8-bit integers, substantially reducing index size and memory traffic: ``` type Document @table { id: Long @primaryKey textEmbeddings: [Float] @indexed(type: "HNSW", quantization: "int8") } ``` Graph navigation runs on the quantized (approximate) distances. For nearest-neighbor `sort` queries, Harper re-ranks the results against the full-precision vectors stored on the records, restoring exact ordering and exact `$distance` values. Distance-threshold (`lt`/`le`) queries currently filter on the approximate distance. ## Field Types Harper supports the following field types: | Type | Description | | --------- | ----------------------------------------------------------------------------------------------- | | `String` | Unicode text, UTF-8 encoded | | `Int` | 32-bit signed integer (−2,147,483,648 to 2,147,483,647) | | `Long` | 54-bit signed integer (−9,007,199,254,740,992 to 9,007,199,254,740,992) | | `Float` | 64-bit double precision floating point | | `BigInt` | Integer up to \~300 digits. Note: distinct JavaScript type; handle appropriately in custom code | | `Boolean` | `true` or `false` | | `ID` | String; indicates a non-human-readable identifier | | `Any` | Any primitive, object, or array | | `Date` | JavaScript `Date` object | | `Bytes` | Binary data as `Buffer` or `Uint8Array` | | `Blob` | Binary large object; designed for streaming content >20KB | Added `BigInt` in v4.3.0 Added `Blob` in v4.5.0 ### Large integers: `Long` vs `BigInt` Despite the name, `Long` is bounded by the JavaScript safe-integer range — values must satisfy `|value| < 2^53` (9,007,199,254,740,991, i.e. `Number.MAX_SAFE_INTEGER`). A `Long` attribute rejects integers beyond that range, so it is not a full 64-bit type. For true 64-bit integers — IDs, counters, or timestamps that can exceed 2^53 — use `BigInt`, and send the value as an actual bigint (for example, via CBOR or MessagePack) rather than a JSON number. A JSON number above 2^53 has already lost precision before Harper receives it, so the larger value cannot be recovered. `BigInt` attributes, including `@indexed` range queries, store and order distinct values above 2^53 correctly. Arrays of a type are expressed with `[Type]` syntax (e.g., `[Float]` for a vector). ### Blob Type *Added in: v4.5.0* `Blob` fields are designed for large binary content. Harper's `Blob` type implements the [Web API `Blob` interface](https://developer.mozilla.org/en-US/docs/Web/API/Blob), so all standard `Blob` methods (`.text()`, `.arrayBuffer()`, `.stream()`, `.slice()`) are available. Unlike `Bytes`, blobs are stored separately from the record, support streaming, and do not need to be held entirely in memory. Use `Blob` for content typically larger than 20KB (images, video, audio, large HTML, etc.). See [Blob usage details](#blob-usage) below. #### Blob Usage Declare a blob field: ``` type MyTable @table { id: Any! @primaryKey data: Blob } ``` Create and store a blob using [`createBlob()`](/reference/v5/database/api.md#createblobdata-options): ``` let blob = createBlob(largeBuffer); await MyTable.put({ id: 'my-record', data: blob }); ``` Retrieve blob data using standard Web API `Blob` methods: ``` let record = await MyTable.get('my-record'); let buffer = await record.data.bytes(); // ArrayBuffer let text = await record.data.text(); // string let stream = record.data.stream(); // ReadableStream ``` Blobs support asynchronous streaming, meaning a record can reference a blob before it is fully written to storage. Use `saveBeforeCommit: true` to wait for full write before committing: ``` let blob = createBlob(stream, { saveBeforeCommit: true }); await MyTable.put({ id: 'my-record', data: blob }); ``` Any string or buffer assigned to a `Blob` field in a `put`, `patch`, or `publish` is automatically coerced to a `Blob`. When returning a blob via REST, register an error handler to handle interrupted streams: ``` export class MyEndpoint extends MyTable { static async get(target) { const record = super.get(target); let blob = record.data; blob.on('error', () => { MyTable.invalidate(target); }); return { status: 200, headers: {}, body: blob }; } } ``` ## Dynamic Schema Behavior When a table is created through the Operations API or Studio without a schema definition, it follows dynamic schema behavior: * Attributes are reflexively created as data is ingested * All top-level attributes are automatically indexed * Records automatically get `__createdtime__` and `__updatedtime__` audit attributes Dynamic schema tables are additive — new attributes are added as new data arrives. Existing records will have `null` for any newly added attributes. Use `create_attribute` and `drop_attribute` operations to manually manage attributes on dynamic schema tables. See the [Operations API](/reference/v5/operations-api/operations.md#databases--tables) for details. ## OpenAPI Specification Tables exported with `@export` are described via an `/openapi` endpoint on the main HTTP server associated with the REST service (default port 9926). ``` GET http://localhost:9926/openapi ``` This provides an OpenAPI 3.x description of all exported resource endpoints. The endpoint is a starting guide and may not cover every edge case. ## Renaming Tables Harper does **not** support renaming tables. Changing a type name in a schema definition creates a new, empty table — the original table and its data are unaffected. ## Related Documentation * [JavaScript API](/reference/v5/database/api.md) — `tables`, `databases`, `transaction()`, and `createBlob()` globals for working with schema-defined tables in code * [Data Loader](/reference/v5/database/data-loader.md) — Seed tables with initial data alongside schema deployment * [REST Querying](/reference/v5/rest/querying.md) — Querying tables via HTTP using schema-defined attributes and relationships * [Resources](/reference/v5/resources/resource-api.md) — Extending table behavior with custom application logic * [Storage Algorithm](/reference/v5/database/storage-algorithm.md) — How Harper indexes and stores schema-defined data * [Configuration](/reference/v5/configuration/options.md) — Component configuration for schemas --- # Storage Algorithm Harper's storage algorithm is the foundation of all database functionality. It is built on top of [RocksDB](https://rocksdb.org/) (the default) or [LMDB](https://www.symas.com/lmdb) (legacy), both high-performance key-value stores, and extends them with automatic indexing, query-language-agnostic data access, and ACID compliance. RocksDB is the default storage engine for new installations. LMDB databases from prior versions are still supported and loaded automatically when detected. ## Query Language Agnostic Harper's storage layer is decoupled from any specific query language. Data inserted via NoSQL operations can be read via SQL, REST, or the Resource API — all accessing the same underlying storage. This architecture allows Harper to add new query interfaces without changing how data is stored. ## ACID Compliance Harper provides full ACID compliance on each node: * **Atomicity**: All writes in a transaction either fully commit or fully roll back * **Consistency**: Each transaction moves data from one valid state to another * **Isolation**: Reads use snapshots and do not block writes; writes do not block reads * **Durability**: RocksDB commits are persisted via its Write-Ahead Log (WAL); LMDB uses memory-mapped file writes Harper uses application-level locking to serialize schema changes and table creation, ensuring write ordering without deadlocks. ## Universally Indexed *Changed in: v4.3.0* — Storage performance improvements including better free-space management For [dynamic schema tables](/reference/v5/database/overview.md#dynamic-vs-defined-schemas), all top-level attributes are automatically indexed immediately upon ingestion — Harper reflexively creates the attribute and its index as new data arrives. For [schema-defined tables](/reference/v5/database/schema.md), indexes are created for all attributes marked with `@indexed`. Indexes are type-agnostic, ordering values as follows: 1. Booleans 2. Numbers (ordered numerically) 3. Strings (ordered lexically) ### Storage Layout Each Harper database corresponds to a separate storage environment: * **RocksDB** (default): a directory on disk containing all stores for that database * **LMDB** (legacy): a single `.mdb` file containing all sub-databases for that database Within each database, a table is represented by multiple key-value stores: * **Primary store** (`tableName/`): stores the full record for each primary key * **Secondary index stores** (`tableName/attributeName`): one store per indexed attribute, mapping attribute values to primary keys * **Metadata store** (`__internal_dbis__`): tracks table and attribute definitions for the database All stores for a given database reside within the same RocksDB directory (or LMDB environment file), so cross-table operations within a database share the same underlying I/O path. ## Compression *Changed in: v4.3.0* — Compression is now enabled by default for all records over 4KB Harper compresses record data automatically for records over 4KB. Compression settings can be configured in the [storage configuration](/reference/v5/configuration/options.md). Note that compression settings cannot be changed on existing databases without creating a new compacted copy — see [Compaction](/reference/v5/database/compaction.md). ## Performance Characteristics Harper inherits strong performance properties from its storage engines: **RocksDB (default)**: * **LSM-tree writes**: Optimized for write-heavy workloads via log-structured merge trees * **Block cache**: Configurable in-memory block cache (defaults to 25% of available system memory) * **WAL durability**: Write-Ahead Log provides crash recovery without sacrificing throughput * **Compression**: Native support for multiple compression algorithms per level **LMDB (legacy)**: * **Memory-mapped I/O**: Data is accessed via memory mapping, enabling fast reads without data duplication between disk and memory * **Buffer cache integration**: Fully exploits the OS buffer cache for reduced I/O * **Zero-copy reads**: Readers access data directly from the memory map without copying * **Deadlock-free writes**: Full serialization of writers guarantees write ordering without deadlocks ## Indexing Example Given a table with records like this: ``` ┌────┬────────┬────────┐ │ id │ field1 │ field2 │ ├────┼────────┼────────┤ │ 1 │ A │ X │ │ 2 │ 25 │ X │ │ 3 │ -1 │ Y │ │ 4 │ A │ │ │ 5 │ true │ 2 │ └────┴────────┴────────┘ ``` Harper maintains three separate key-value stores for that table, all within the same database: ``` Database (RocksDB directory or LMDB environment) │ ├── primary store: "MyTable/" │ ┌─────┬──────────────────────────────────────┐ │ │ Key │ Value (full record) │ │ ├─────┼──────────────────────────────────────┤ │ │ 1 │ { id:1, field1:"A", field2:"X" } │ │ │ 2 │ { id:2, field1:25, field2:"X" } │ │ │ 3 │ { id:3, field1:-1, field2:"Y" } │ │ │ 4 │ { id:4, field1:"A" } │ │ │ 5 │ { id:5, field1:true, field2:2 } │ │ └─────┴──────────────────────────────────────┘ │ ├── secondary index: "MyTable/field1" secondary index: "MyTable/field2" │ ┌────────┬───────┐ ┌────────┬───────┐ │ │ Key │ Value │ │ Key │ Value │ │ ├────────┼───────┤ ├────────┼───────┤ │ │ -1 │ 3 │ │ 2 │ 5 │ │ │ 25 │ 2 │ │ X │ 1 │ │ │ A │ 1 │ │ X │ 2 │ │ │ A │ 4 │ │ Y │ 3 │ │ │ true │ 5 │ └────────┴───────┘ │ └────────┴───────┘ ``` Secondary indexes store the attribute value as the key and the record's primary key (`id`) as the value. To resolve a query result, Harper looks up the matching ids in the secondary index, then fetches the full records from the primary store. Indexes are ordered — booleans first, then numbers (numerically), then strings (lexically) — enabling efficient range queries across all types. ## Related Documentation * [Schema](/reference/v5/database/schema.md) — Defining indexed attributes and vector indexes * [Compaction](/reference/v5/database/compaction.md) — Reclaiming free space and applying new storage configuration to existing databases * [Configuration](/reference/v5/configuration/options.md) — Storage configuration options (compression, memory maps, blob paths) --- # Storage Tuning Harper's `storage` configuration section controls how database files are written, cached, and reclaimed on disk. Defaults are tuned for safety and balanced throughput; this page covers the knobs that matter for production deployments with specific workload profiles. For a quick reference of every option, see [Configuration Options — `storage`](/reference/v5/configuration/options.md#storage). For the underlying mechanics, see [Storage Algorithm](/reference/v5/database/storage-algorithm.md). ## Durability vs. Throughput ### `storage.writeAsync` Type: `boolean` Default: `false` Disables `fsync` on commit. Writes return as soon as data is queued to the page cache, dramatically increasing throughput on write-heavy workloads. **This disables durability guarantees.** A power loss or OS crash between the application commit and the OS flushing pages to disk can lose recently committed transactions. The database itself remains structurally consistent — only the most recent writes are at risk. Enable only when: * The data is reproducible from an upstream source (e.g., caches with `sourcedFrom`). * The workload is bulk ingest where some loss on crash is acceptable and the operation can be re-run. * Replication provides durability — peers acknowledge writes before they could be lost. ``` storage: writeAsync: true ``` ### `storage.maxTransactionQueueTime` Type: `string` (duration) Default: `45s` The maximum estimated time a write may wait in the commit queue before Harper rejects new writes with HTTP 503. Acts as backpressure when downstream disk I/O cannot keep up with incoming writes. Lower this in latency-sensitive systems where it is better to shed load early than to let request queues grow. Raise it when occasional disk-write bursts are expected and the application can tolerate longer commit latency. ``` storage: maxTransactionQueueTime: 30s ``` ## Compression ### `storage.compression` Type: `boolean | object` Default: `true` LZ4 record compression is enabled by default. It typically reduces on-disk size by 2–4× for JSON-like records with modest CPU cost. For object form: | Property | Type | Description | | ------------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `dictionary` | `string` | Path to a [Zstd-style](https://github.com/facebook/zstd) compression dictionary. Training a dictionary on representative records improves the compression ratio for small records. | | `threshold` | `number` | Records smaller than this many bytes are stored uncompressed. Useful when small records dominate and the overhead of compression headers outweighs gains. | ``` storage: compression: threshold: 256 dictionary: ~/hdb/keys/records.dict ``` Disable entirely for workloads dominated by small numeric records or pre-compressed payloads (e.g., images, video): ``` storage: compression: false ``` ## Blob Storage Paths ### `storage.blobPaths` Type: `string | string[]` Default: `/blobs` Blob attributes (declared with `Blob` in `schema.graphql` or written via [`createBlob`](/reference/v5/database/api.md)) are stored outside the main database files. `blobPaths` accepts a single path or an array — Harper distributes blob writes across the listed paths. Common configurations: * **Separate fast disk for blobs:** Place `blobPaths` on a higher-bandwidth volume (e.g., NVMe) while keeping the index on a smaller, lower-latency drive. * **Multiple volumes:** Provide an array to spread storage across drives. Harper picks the path with the most free space for each new blob, providing crude load-balancing without RAID. * **Storage tiers:** Mount large but slower object storage at one of the paths to absorb older blobs while keeping hot blobs on fast storage. ``` storage: blobPaths: - /mnt/nvme0/harper-blobs - /mnt/nvme1/harper-blobs ``` Blobs are not relocated when `blobPaths` changes — only new blobs honor the updated configuration. Existing blob references continue to resolve at their original path. ## Read & Write Behavior ### `storage.prefetchWrites` Type: `boolean` Default: `true` Before a write transaction commits, Harper loads the affected pages into memory if not already present. This avoids stalling the commit on a page fault. Disable only when memory pressure makes the prefetch counterproductive — for example, when transactions touch records on cold pages that are not expected to be re-read soon. ### `storage.noReadAhead` Type: `boolean` Default: `false` Advises the OS via `posix_fadvise` not to read ahead beyond the requested pages. Useful for random-access workloads on rotational disks where speculative reads pollute the page cache. Leave at the default for sequential or scan-heavy workloads. ### `storage.pageSize` Type: `number` Default: OS page size (typically 4096 bytes) Changes the database page size. Larger pages can reduce write amplification for large records but increase the minimum I/O unit. **Only set this on a fresh database** — existing files cannot be migrated to a different page size. ### `storage.caching` Type: `boolean` Default: `true` In-memory record caching of decoded records. Disable to reduce heap usage when records are large and unlikely to be re-read in the same process. ## RocksDB Memory RocksDB exposes two large native memory pools that Harper makes tunable: a shared **block cache** for hot SST blocks and a **WriteBufferManager** (enabled by default) that caps total memtable memory across every database in the process. These options apply only when `storage.engine` is `rocksdb`. ### How RocksDB reads are cached A read of a record that isn't in the memtable goes through three tiers before reaching disk: 1. **Block cache** (in-process, decompressed) — sized by `storage.rocks.blockCacheSize`. A hit returns in roughly a microsecond with no syscall and no decompression cost. 2. **OS page cache** (kernel, compressed SST file pages) — sized dynamically by the kernel from whatever memory isn't claimed by the process. A block-cache miss that hits the page cache costs a `read` syscall plus decompression — still on the order of microseconds, just an order of magnitude slower than the block cache. 3. **Disk** — if neither cache holds the page, RocksDB reads from the SST file directly. Harper uses buffered I/O, so the OS page cache is always in play. The implication for sizing: shrinking the block cache doesn't directly translate to more disk reads — it shifts hits from the block cache (decompressed) to the OS page cache (compressed). The OS page cache also adjusts dynamically to host-wide memory pressure, which the block cache does not. Reserving less memory for the block cache leaves more for the page cache and for unrelated allocations on the host. The trade-off favors a larger block cache when read latency matters and the working set fits; it favors a smaller block cache when memory pressure or noisy neighbors are the dominant concern. ### `storage.rocks.blockCacheSize` *Added in: v5.1.0* Type: `number` (bytes) Default: 25% of constrained (cgroup) or total memory The shared LRU cache for decompressed SST blocks. Every RocksDB database in the process draws from this single pool. The cache fills as blocks are read; it does **not** shrink on idle. Once the cache reaches its high-water mark for a workload, entries persist until LRU eviction or a manual capacity change. A long-running instance with a brief burst of activity will hold the cached blocks for the lifetime of the process. ``` storage: rocks: blockCacheSize: 268435456 # 256 MB ``` Lower the cache size when: * The host has limited memory headroom and the OS page cache is a meaningful second tier. * Read access patterns favor a warm working set far smaller than 25% of memory. * The instance runs under a strict cgroup limit and the headroom is needed for memtables or application heap. Raise it (or leave at the default) when reads dominate and the working set comfortably fits at 25%. ### `storage.rocks.writeBufferManagerSize` *Added in: v5.1.0* Type: `number` (bytes) Default: one third of `blockCacheSize` (enabled). Set to `0` to disable. Harper attaches a single RocksDB `WriteBufferManager` to every opened database in the process. Total memtable memory — including active memtables, immutable memtables awaiting flush, and the maintain-window history that RocksDB's `OptimisticTransactionDB` retains for conflict checking — is capped at this size across the entire process. Without a `WriteBufferManager`, each column family (table) manages its own memtable budget. The total grows with the number of column families: each one retains roughly `max_write_buffer_size_to_maintain` worth of recently-flushed memtables for snapshot reads and conflict detection. A database with many tables can accumulate hundreds of megabytes to a few gigabytes of resident anonymous memory before any cap is reached. The manager is enabled by default to bound that growth at a single limit. Override the default to set an explicit budget, or disable it entirely: ``` storage: rocks: writeBufferManagerSize: 268435456 # 256 MB total memtable budget (0 disables) ``` The configured size affects new databases opened after it is changed; existing open databases retain whatever budget they were attached with. ### `storage.rocks.writeBufferManagerCostToCache` *Added in: v5.1.0* Type: `boolean` Default: `true` When `true`, memtable memory tracked by the `WriteBufferManager` is **charged against the block cache** as pinned cache entries. The block cache and write buffers then share a single accounting pool, visible through one operational metric (`rocksdb.block-cache-usage`). This does not let the cache "shrink" to make room for writes — pinned entries cannot be evicted by LRU — but it unifies observability and bounds the combined memory footprint when `writeBufferManagerSize` is at or below `blockCacheSize`. Has no effect when `storage.rocks.writeBufferManagerSize` is `0` or when the block cache is disabled. ``` storage: rocks: blockCacheSize: 536870912 # 512 MB writeBufferManagerSize: 268435456 # 256 MB writeBufferManagerCostToCache: true ``` ### `storage.rocks.writeBufferManagerAllowStall` *Added in: v5.1.0* Type: `boolean` Default: `true` Controls behavior when memtable memory reaches `writeBufferManagerSize`: * `false` (soft cap) — Memtables may briefly exceed the limit. RocksDB compensates by flushing more aggressively. Writes proceed without latency spikes; total memory may temporarily overshoot during bursts. * `true` (hard cap) — Writes are stalled until flushes free up memory. Total memtable memory is strictly bounded; write latency can spike during bursts. The default (`true`) strictly bounds total memtable memory, applying write backpressure rather than letting memtables overshoot — which also keeps bulk ingest from outrunning the memtable flush/conflict-check window. Set to `false` for a soft cap when write-latency smoothness matters more than a strict memory bound and brief overshoot during bursts is acceptable. This option is the only `WriteBufferManager` setting that can be changed at runtime — `costToCache` is fixed at first creation. ## Storage Reclamation `storage.reclamation` controls how Harper evicts data from caching tables (tables with [`sourcedFrom`](/reference/v5/resources/resource-api.md#sourcedfromresource-options)) when disk usage runs high. Reclamation does **not** affect non-caching tables — those rely on explicit deletion, TTL expiration, or [compaction](/reference/v5/database/compaction.md). ### `storage.reclamation.threshold` Type: `number` (ratio) Default: `0.4` Minimum fraction of the volume that should remain free. When free space falls below this ratio, reclamation begins evicting expired and lightly-used entries from caching tables. A larger value reclaims earlier and more aggressively; a smaller value defers reclamation closer to the volume filling. ### `storage.reclamation.interval` Type: `string` (duration) Default: `1h` How often Harper checks free space against the threshold. Lower intervals catch fast-filling volumes sooner at the cost of more periodic I/O. ### `storage.reclamation.evictionFactor` Type: `number` Default: `100000` Tunes the heuristic used to evict entries early when reclamation priority is high. The heuristic considers each entry's remaining time-to-expiration, record size, and how long ago it was last refreshed. Lowering this evicts more aggressively when free space is critical; raising it preserves entries longer. ``` storage: reclamation: threshold: 0.3 # start reclaiming at 30% free interval: 30m evictionFactor: 50000 ``` For a deeper discussion of how `sourcedFrom` interacts with reclamation, see [Resource API — `sourcedFrom`](/reference/v5/resources/resource-api.md#sourcedfromresource-options). ## Compaction on Start ### `storage.compactOnStart` Type: `boolean` Default: `false` Runs [compaction](/reference/v5/database/compaction.md) on all non-system databases at startup. Useful when deployments include scheduled restarts and you want to reclaim fragmented space as part of the maintenance window. ### `storage.compactOnStartKeepBackup` Type: `boolean` Default: `false` Retains the pre-compaction backup files after `compactOnStart` runs. Recommended for the first few cycles in production while validating compaction behavior; the backups can be removed manually once confidence is established. ## Workload Recipes **Write-heavy ingest, durability via replication:** ``` storage: writeAsync: true prefetchWrites: true maxTransactionQueueTime: 60s ``` **Read-heavy cache layer with large blobs:** ``` storage: caching: true blobPaths: - /mnt/fast-ssd/harper-blobs reclamation: threshold: 0.2 interval: 15m ``` **Memory-constrained edge deployment:** ``` storage: caching: false noReadAhead: true compression: true ``` ## Related * [Configuration Options](/reference/v5/configuration/options.md) — full list of `storage` options * [Storage Algorithm](/reference/v5/database/storage-algorithm.md) — how Harper stores records and indexes on disk * [Compaction](/reference/v5/database/compaction.md) — reclaiming space inside existing database files * [Resource API — `sourcedFrom`](/reference/v5/resources/resource-api.md#sourcedfromresource-options) — caching tables that interact with reclamation * [Database API — `createBlob`](/reference/v5/database/api.md) — creating blobs that live under `blobPaths` --- # System Tables Harper maintains a set of internal system tables in the `system` database. These tables store analytics, job tracking, replication configuration, and other internal state. Most are read-only from the application perspective; some can be queried for observability or management purposes. System tables are prefixed with `hdb_` and reside in the `system` database. ## Analytics Tables *Added in: v4.5.0* (resource and storage analytics expansion) ### `hdb_raw_analytics` Stores per-second, per-thread performance metrics. Records are written once per second (when there is activity) and include metrics for all operations, URL endpoints, and messaging topics, plus system resource information such as memory and CPU utilization. Records have a primary key equal to the timestamp in milliseconds since Unix epoch. Query with `search_by_conditions` (requires `superuser` permission): ``` { "operation": "search_by_conditions", "schema": "system", "table": "hdb_raw_analytics", "conditions": [ { "search_attribute": "id", "search_type": "between", "search_value": [1688594000000, 1688594010000] } ] } ``` A typical record: ``` { "time": 1688594390708, "period": 1000.8336279988289, "metrics": [ { "metric": "bytes-sent", "path": "search_by_conditions", "type": "operation", "median": 202, "mean": 202, "p95": 202, "p90": 202, "count": 1 }, { "metric": "memory", "threadId": 2, "rss": 1492664320, "heapTotal": 124596224, "heapUsed": 119563120, "external": 3469790, "arrayBuffers": 798721 }, { "metric": "utilization", "idle": 138227.52767700003, "active": 70.5066209952347, "utilization": 0.0005098165086230495 } ], "threadId": 2, "totalBytesProcessed": 12182820, "id": 1688594390708.6853 } ``` ### `hdb_analytics` Stores per-minute aggregate analytics. Once per minute, Harper aggregates all per-second raw entries from all threads into summary records in this table. Query it for longer-term performance trends. ``` { "operation": "search_by_conditions", "schema": "system", "table": "hdb_analytics", "conditions": [ { "search_attribute": "id", "search_type": "between", "search_value": [1688194100000, 1688594990000] } ] } ``` A typical aggregate record: ``` { "period": 60000, "metric": "bytes-sent", "method": "connack", "type": "mqtt", "median": 4, "mean": 4, "p95": 4, "p90": 4, "count": 1, "id": 1688589569646, "time": 1688589569646 } ``` For a full reference of available metrics and their fields, see [Analytics](/reference/v5/analytics/overview.md "Complete analytics metrics reference"). ## Data Loader Table ### `hdb_dataloader_hash` *Added in: v4.6.0* Used internally by the [Data Loader](/reference/v5/database/data-loader.md) to track which records have been loaded and detect changes. Stores SHA-256 content hashes of data file records so that unchanged records are not re-written on subsequent deployments. This table is managed automatically by the Data Loader. No direct interaction is required. ## Replication Tables ### `hdb_nodes` Stores the configuration and state of known nodes in a cluster, including connection details, replication settings, and revoked certificate serial numbers. Can be queried to inspect the current replication topology: ``` { "operation": "search_by_hash", "schema": "system", "table": "hdb_nodes", "hash_values": ["node-id"] } ``` Used by the `add_node`, `update_node`, and related clustering operations. See [Replication](/reference/v5/replication/clustering.md) for details. ### `hdb_certificate` Stores TLS certificates used in replication. Can be queried to inspect the certificates currently known to the cluster. ## Related Documentation * [Analytics](/reference/v5/analytics/overview.md) — Full reference for analytics metrics tracked in `hdb_analytics` and `hdb_raw_analytics` * [Data Loader](/reference/v5/database/data-loader.md) — Component that writes to `hdb_dataloader_hash` * [Replication](/reference/v5/replication/overview.md) — Clustering and replication system that uses `hdb_nodes` and `hdb_certificate` * [Operations API](/reference/v5/operations-api/overview.md) — Querying system tables using `search_by_conditions` --- # Transaction Logging Harper provides two complementary mechanisms for recording a history of data changes on a table: the **audit log** and the **transaction log**. Both are available at the table level and serve different use cases. | Feature | Audit Log | Transaction Log | | ----------------------------- | --------------------------------- | ------------------------------ | | Storage | Standard Harper table (per-table) | Clustering streams (per-table) | | Requires clustering | No | Yes | | Available since | v4.1.0 | v4.1.0 | | Stores original record values | Yes | No | | Query by username | Yes | No | | Query by primary key | Yes | No | | Used for real-time messaging | Yes (required) | No | ## Audit Log Available since: v4.1.0 The audit log is a data store that tracks every transaction across all tables in a database. Harper automatically creates and maintains a single audit log per database. The audit log captures the operation type, the user who made the change, the timestamp, and both the new and original record values. The audit log is **enabled by default**. To disable it, set [`logging.auditLog`](/reference/v5/logging/configuration.md) to `false` in `harper-config.yaml` and restart Harper. > The audit log is required for real-time messaging (WebSocket and MQTT subscriptions) and replication. Do not disable it if real-time features or replication are in use. ### Audit Log Operations #### `read_audit_log` Queries the audit log for a specific table. Supports filtering by timestamp, username, or primary key value. **By timestamp:** ``` { "operation": "read_audit_log", "schema": "dev", "table": "dog", "search_type": "timestamp", "search_values": [1660585740558] } ``` Timestamp behavior: | `search_values` | Result | | --------------- | ---------------------------------------- | | `[]` | All records for the table | | `[timestamp]` | All records after the provided timestamp | | `[from, to]` | Records between the two timestamps | **By username:** ``` { "operation": "read_audit_log", "schema": "dev", "table": "dog", "search_type": "username", "search_values": ["admin"] } ``` **By primary key:** ``` { "operation": "read_audit_log", "schema": "dev", "table": "dog", "search_type": "hash_value", "search_values": [318] } ``` **Response example:** ``` { "operation": "update", "user_name": "HDB_ADMIN", "timestamp": 1607035559122.277, "hash_values": [1, 2], "records": [ { "id": 1, "breed": "Muttzilla", "age": 6, "__updatedtime__": 1607035559122 } ], "original_records": [ { "__createdtime__": 1607035556801, "__updatedtime__": 1607035556801, "age": 5, "breed": "Mutt", "id": 1, "name": "Harper" } ] } ``` The `original_records` field contains the record state before the operation was applied. #### `delete_audit_logs_before` Deletes audit log entries older than the specified timestamp. *Changed in: v4.3.0* — Audit log cleanup improved to reduce resource consumption during scheduled cleanups *Changed in: v4.5.0* — Storage reclamation: Harper automatically evicts older audit log entries when free storage drops below a configurable threshold ``` { "operation": "delete_audit_logs_before", "schema": "dev", "table": "dog", "timestamp": 1598290282817 } ``` *** ## Enabling Audit Log Per Table You can enable or disable the audit log for individual tables using the `@table` directive's `audit` argument in your schema: ``` type Dog @table(audit: true) { id: Long @primaryKey name: String } ``` This overrides the [`logging.auditLog`](/reference/v5/logging/configuration.md) global configuration for that specific table. ## Related Documentation * [Logging](/reference/v5/logging/overview.md) — Application and system logging (separate from transaction/audit logging) * [Replication](/reference/v5/replication/overview.md) — Clustering setup required for transaction logs * [Logging Configuration](/reference/v5/logging/configuration.md) — Global audit log configuration (`logging.auditLog`) * [Operations API](/reference/v5/operations-api/overview.md) — Sending operations to Harper --- # Environment Variables Harper supports loading environment variables in Harper applications `process.env` using the built-in `loadEnv` plugin. This is the standard way to supply secrets and configuration to your Harper components without hardcoding values. `loadEnv` does **not** need to be installed as it is built into Harper and only needs to be declared in your `config.yaml`. note If you are looking for information on how to configure your Harper installation using environment variables, see [Configuration](/reference/v5/configuration/overview.md) section for more information. ## Configuration | Option | Type | Required | Description | | ---------- | -------------------- | -------- | -------------------------------------------------------------------------------------- | | `files` | `string \| string[]` | **Yes** | Path(s) or glob pattern(s) to the env file(s) to load. | | `override` | `boolean` | No | If `true`, loaded values override existing environment variables. Defaults to `false`. | ## Basic Usage ``` loadEnv: files: '.env' ``` This loads the `.env` file from the root of your component directory into `process.env`. ## Load Order > **Important:** Specify `loadEnv` first in your `config.yaml` so that environment variables are loaded before any other components start. ``` # config.yaml — loadEnv must come first loadEnv: files: '.env' rest: true myApp: files: './src/*.js' ``` Because Harper is a single-process application, environment variables are loaded onto `process.env` and are shared across all components. As long as `loadEnv` is listed before dependent components, those components will have access to the loaded variables. ## Override Behavior By default, `loadEnv` follows the standard dotenv convention: **existing environment variables take precedence** over values in `.env` files. This means variables already set in the shell or container environment will not be overwritten. To override existing environment variables, use the `override` option: ``` loadEnv: files: '.env' override: true ``` ## Multiple Files As a Harper plugin, `loadEnv` supports multiple files using either glob patterns or a list of files in the configuration: ``` loadEnv: files: - '.env' - '.env.local' ``` or ``` loadEnv: files: 'env-vars/*' ``` Files are loaded in the order specified. ## Related * [Components Overview](/reference/v5/components/overview.md) * [Configuration](/reference/v5/configuration/overview.md) --- # Define Fastify Routes note Fastify routes are discouraged in favor of modern routing with [Custom Resources](/reference/v5/resources/overview.md), but remain a supported feature for backwards compatibility and specific use cases. Harper provides a build-in plugin for loading [Fastify](https://www.fastify.io/) routes as a way to define custom endpoints for your Harper application. While we generally recommend building your endpoints/APIs with Harper's [REST interface](/reference/v5/rest/overview.md) for better performance and standards compliance, Fastify routes can provide an extensive API for highly customized path handling. Below is a very simple example of a route declaration. The fastify route handler can be configured in your application's config.yaml (this is the default config if you used the [application template](https://github.com/HarperDB/application-template)): ``` fastifyRoutes: files: routes/*.js # specify the location of route definition modules ``` By default, route URLs are configured to be: ``` :// ``` However, you can specify the path to be `/` if you wish to have your routes handling the root path of incoming URLs. * The route below, using the default config, within the **dogs** project, with a route of **breeds** would be available at ****. In effect, this route is just a pass-through to Harper. The same result could have been achieved by hitting the core Harper API, since it uses **hdbCore.preValidation** and **hdbCore.request**, which are defined in the "helper methods" section, below. ``` export default async (server, { hdbCore, logger }) => { server.route({ url: '/', method: 'POST', preValidation: hdbCore.preValidation, handler: hdbCore.request, }); }; ``` ## Custom Handlers For endpoints where you want to execute multiple operations against Harper, or perform additional processing (like an ML classification, or an aggregation, or a call to a 3rd party API), you can define your own logic in the handler. The function below will execute a query against the dogs table, and filter the results to only return those dogs over 4 years in age. **IMPORTANT: This route has NO preValidation and uses hdbCore.requestWithoutAuthentication, which- as the name implies- bypasses all user authentication. See the security concerns and mitigations in the "helper methods" section, below.** ``` export default async (server, { hdbCore, logger }) => { server.route({ url: '/:id', method: 'GET', handler: (request) => { request.body= { operation: 'sql', sql: `SELECT * FROM dev.dog WHERE id = ${request.params.id}` }; const result = await hdbCore.requestWithoutAuthentication(request); return result.filter((dog) => dog.age > 4); } }); } ``` ## Custom preValidation Hooks The simple example above was just a pass-through to Harper- the exact same result could have been achieved by hitting the core Harper API. But for many applications, you may want to authenticate the user using custom logic you write, or by conferring with a 3rd party service. Custom preValidation hooks let you do just that. Below is an example of a route that uses a custom validation hook: ``` import customValidation from '../helpers/customValidation'; export default async (server, { hdbCore, logger }) => { server.route({ url: '/:id', method: 'GET', preValidation: (request) => customValidation(request, logger), handler: (request) => { request.body = { operation: 'sql', sql: `SELECT * FROM dev.dog WHERE id = ${request.params.id}`, }; return hdbCore.requestWithoutAuthentication(request); }, }); }; ``` Notice we imported customValidation from the **helpers** directory. To include a helper, and to see the actual code within customValidation, see [Helper Methods](#helper-methods). ## Helper Methods When declaring routes, you are given access to 2 helper methods: hdbCore and logger. ### hdbCore hdbCore contains three functions that allow you to authenticate an inbound request, and execute operations against Harper directly, by passing the standard Operations API. #### preValidation This is an array of functions used for fastify authentication. The second function takes the authorization header from the inbound request and executes the same authentication as the standard Harper Operations API (for example, `hdbCore.preValidation[1](req, resp, callback)`). It will determine if the user exists, and if they are allowed to perform this operation. **If you use the request method, you have to use preValidation to get the authenticated user**. #### request This will execute a request with Harper using the operations API. The `request.body` should contain a standard Harper operation and must also include the `hdb_user` property that was in `request.body` provided in the callback. #### requestWithoutAuthentication Executes a request against Harper without any security checks around whether the inbound user is allowed to make this request. For security purposes, you should always take the following precautions when using this method: * Properly handle user-submitted values, including url params. User-submitted values should only be used for `search_value` and for defining values in records. Special care should be taken to properly escape any values if user-submitted values are used for SQL. ### logger This helper allows you to write directly to the log file, hdb.log. It's useful for debugging during development, although you may also use the console logger. There are 5 functions contained within logger, each of which pertains to a different **logging.level** configuration in your harper-config.yaml file. * logger.trace('Starting the handler for /dogs') * logger.debug('This should only fire once') * logger.warn('This should never ever fire') * logger.error('This did not go well') * logger.fatal('This did not go very well at all') --- # GraphQL Querying *Added in: v4.4.0* (provisional) *Changed in: v4.5.0* (disabled by default, configuration options) Harper supports GraphQL in a variety of ways. It can be used for [defining schemas](/reference/v5/components/applications.md), and for querying [Resources](/reference/v5/resources/overview.md). Get started by setting `graphql: true` in `config.yaml`. This configuration option was added in v4.5.0 to allow more granular control over the GraphQL endpoint. This automatically enables a `/graphql` endpoint that can be used for GraphQL queries. > Harper's GraphQL component is inspired by the [GraphQL Over HTTP](https://graphql.github.io/graphql-over-http/draft/#) specification; however, it does not fully implement neither that specification nor the [GraphQL](https://spec.graphql.org/) specification. Queries can either be `GET` or `POST` requests, and both follow essentially the same request format. `GET` requests must use search parameters, and `POST` requests use the request body. For example, to request the GraphQL Query: ``` query GetDogs { Dog { id name } } ``` The `GET` request would look like: ``` GET /graphql?query=query+GetDogs+%7B+Dog+%7B+id+name+%7D+%7D+%7D Accept: application/graphql-response+json ``` And the `POST` request would look like: ``` POST /graphql/ Content-Type: application/json Accept: application/graphql-response+json { "query": "query GetDogs { Dog { id name } } }" } ``` > Tip: For the best user experience, include the `Accept: application/graphql-response+json` header in your request. This provides better status codes for errors. The Harper GraphQL querying system is strictly limited to exported Harper Resources. This will typically be a table that uses the `@exported` directive in its schema or `export`'ed custom resources. Queries can only specify Harper Resources and their attributes in the selection set. Queries can filter using [arguments](https://graphql.org/learn/queries/#arguments) on the top-level Resource field. Harper provides a short form pattern for simple queries, and a long form pattern based off of the [Resource Query API](/reference/v5/rest/querying.md) for more complex queries. Unlike REST queries, GraphQL queries can specify multiple resources simultaneously: ``` query GetDogsAndOwners { Dog { id name breed } Owner { id name occupation } } ``` This will return all dogs and owners in the database. And is equivalent to executing two REST queries: ``` GET /Dog/?select(id,name,breed) # and GET /Owner/?select(id,name,occupation) ``` ## Request Parameters There are three request parameters for GraphQL queries: `query`, `operationName`, and `variables` 1. `query` - *Required* - The string representation of the GraphQL document. 1. Limited to [Executable Definitions](https://spec.graphql.org/October2021/#executabledefinition) only. 2. i.e. GraphQL [`query`](https://graphql.org/learn/queries/#fields) or `mutation` (coming soon) operations, and [fragments](https://graphql.org/learn/queries/#fragments). 3. If an shorthand, unnamed, or singular named query is provided, they will be executed by default. Otherwise, if there are multiple queries, the `operationName` parameter must be used. 2. `operationName` - *Optional* - The name of the query operation to execute if multiple queries are provided in the `query` parameter 3. `variables` - *Optional* - A map of variable values to be used for the specified query ## Type Checking The Harper GraphQL Querying system is designed to handle GraphQL queries and map them directly to Harper's tables, schemas, fields, and relationships to easily query with GraphQL syntax with minimal configuration, code, and overhead. However, the "GraphQL", as a technology has come to encompass an entire model of resolvers and a type checking system, which is outside of the scope of using GraphQL as a *query* language for data retrieval from Harper. Therefore, the querying system generally does **not** type check, and type checking behavior is outside the scope of resolving queries and is only loosely defined in Harper. In variable definitions, the querying system will ensure non-null values exist (and error appropriately), but it will not do any type checking of the value itself. For example, the variable `$name: String!` states that `name` should be a non-null, string value. * If the request does not contain the `name` variable, an error will be returned * If the request provides `null` for the `name` variable, an error will be returned * If the request provides any non-string value for the `name` variable, i.e. `1`, `true`, `{ foo: "bar" }`, the behavior is undefined and an error may or may not be returned. * If the variable definition is changed to include a default value, `$name: String! = "John"`, then when omitted, `"John"` will be used. * If `null` is provided as the variable value, an error will still be returned. * If the default value does not match the type specified (i.e. `$name: String! = 0`), this is also considered undefined behavior. It may or may not fail in a variety of ways. * Fragments will generally extend non-specified types, and the querying system will do no validity checking on them. For example, `fragment Fields on Any { ... }` is just as valid as `fragment Fields on MadeUpTypeName { ... }`. See the Fragments sections for more details. The only notable place the querying system will do some level of type analysis is the transformation of arguments into a query. * Objects will be transformed into properly nested attributes * Strings and Boolean values are passed through as their AST values * Float and Int values will be parsed using the JavaScript `parseFloat` and `parseInt` methods respectively. * List and Enums are not supported. ## Fragments The querying system loosely supports fragments. Both fragment definitions and inline fragments are supported, and are entirely a composition utility. Since this system does very little type checking, the `on Type` part of fragments is entirely pointless. Any value can be used for `Type` and it will have the same effect. For example, in the query ``` query Get { Dog { ...DogFields } } fragment DogFields on Dog { name breed } ``` The `Dog` type in the fragment has no correlation to the `Dog` resource in the query (that correlates to the Harper `Dog` resource). You can literally specify anything in the fragment and it will behave the same way: ``` fragment DogFields on Any { ... } # this is recommended fragment DogFields on Cat { ... } fragment DogFields on Animal { ... } fragment DogFields on LiterallyAnything { ... } ``` As an actual example, fragments should be used for composition: ``` query Get { Dog { ...sharedFields breed } Owner { ...sharedFields occupation } } fragment sharedFields on Any { id name } ``` ## Short Form Querying Any attribute can be used as an argument for a query. In this short form, multiple arguments is treated as multiple equivalency conditions with the default `and` operation. For example, the following query requires an `id` variable to be provided, and the system will search for a `Dog` record matching that id. ``` query GetDog($id: ID!) { Dog(id: $id) { name breed owner { name } } } ``` And as a properly formed request: ``` POST /graphql/ Content-Type: application/json Accept: application/graphql-response+json { "query": "query GetDog($id: ID!) { Dog(id: $id) { name breed owner {name}}", "variables": { "id": "0" } } ``` The REST equivalent would be: ``` GET /Dog/?id==0&select(name,breed,owner{name}) # or GET /Dog/0?select(name,breed,owner{name}) ``` Short form queries can handle nested attributes as well. For example, return all dogs who have an owner with the name `"John"` ``` query GetDog { Dog(owner: { name: "John" }) { name breed owner { name } } } ``` Would be equivalent to ``` GET /Dog/?owner.name==John&select(name,breed,owner{name}) ``` And finally, we can put all of these together to create semi-complex, equality based queries! The following query has two variables and will return all dogs who have the specified name as well as the specified owner name. ``` query GetDog($dogName: String!, $ownerName: String!) { Dog(name: $dogName, owner: { name: $ownerName }) { name breed owner { name } } } ``` --- # HTTP API The `server` global object is available in all Harper component code. It provides access to the HTTP server middleware chain, WebSocket server, authentication, resource registry, and cluster information. ## `server.http(listener, options)` Add a handler to the HTTP request middleware chain. ``` server.http(listener: RequestListener, options?: HttpOptions): HttpServer[] ``` Returns an array of `HttpServer` instances based on the `options.port` and `options.securePort` values. **Example:** ``` server.http( (request, next) => { if (request.url === '/graphql') return handleGraphQLRequest(request); return next(request); }, { runFirst: true } ); ``` ### `RequestListener` ``` type RequestListener = (request: Request, next: RequestListener) => Promise; ``` To continue the middleware chain, call `next(request)`. To short-circuit, return a `Response` (or `Response`-like object) directly. ### `HttpOptions` | Property | Type | Default | Description | | ------------ | ------- | ----------------- | --------------------------------------------- | | `runFirst` | boolean | `false` | Insert this handler at the front of the chain | | `port` | number | `http.port` | Target the HTTP server on this port | | `securePort` | number | `http.securePort` | Target the HTTPS server on this port | ### `HttpServer` A Node.js [`http.Server`](https://nodejs.org/api/http.html#class-httpserver) or [`https.SecureServer`](https://nodejs.org/api/https.html#class-httpsserver) instance. *** ## `Request` A `Request` object is passed to HTTP middleware handlers and direct static REST handlers. It follows the [WHATWG `Request`](https://developer.mozilla.org/en-US/docs/Web/API/Request) API with additional Harper-specific properties. ### Properties | Property | Type | Description | | ---------- | --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `url` | string | The request target (path + query string), e.g. `/path?query=string` | | `method` | string | HTTP method: `GET`, `POST`, `PUT`, `DELETE`, etc. | | `headers` | [`Headers`](https://developer.mozilla.org/en-US/docs/Web/API/Headers) | Request headers | | `pathname` | string | Path portion of the URL, without query string | | `protocol` | string | `http` or `https` | | `data` | any | Deserialized body, based on `Content-Type` header | | `ip` | string | Remote IP address of the client (or last proxy) | | `host` | string | Host from the request headers | | `session` | object | Current cookie-based session (a `Table` record instance). Update with `request.session.update({ key: value })`. A cookie is set automatically the first time a session is updated or a login occurs. | ### Methods #### `request.login(username, password)` ``` login(username: string, password: string): Promise ``` Authenticates the user by username and password. On success, creates a session and sets a cookie on the response. Rejects if authentication fails. #### `request.sendEarlyHints(link, headers?)` ``` sendEarlyHints(link: string, headers?: object): void ``` Sends an [Early Hints](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/103) (HTTP 103) response before the final response. Useful in cache resolution functions to hint at preloadable resources: ``` class Origin { async get(request) { this.getContext().requestContext.sendEarlyHints(''); return fetch(request); } } Cache.sourcedFrom(Origin); ``` ### Low-Level Node.js Access caution These properties expose the raw Node.js request/response objects and should be used with caution. Using them can break other middleware handlers that depend on the layered `Request`/`Response` pattern. | Property | Description | | --------------- | ----------------------------------------------------------------------------------------------------- | | `_nodeRequest` | Underlying [`http.IncomingMessage`](https://nodejs.org/api/http.html#http_class_http_incomingmessage) | | `_nodeResponse` | Underlying [`http.ServerResponse`](https://nodejs.org/api/http.html#http_class_http_serverresponse) | *** ## `Response` REST method handlers can return: * **Data directly** — Serialized using Harper's content negotiation * **A `Response` object** — The WHATWG [`Response`](https://developer.mozilla.org/en-US/docs/Web/API/Response) * **A `Response`-like object** — A plain object with the following properties: | Property | Type | Description | | --------- | --------------------------------------------------------------------- | ------------------------------------------------- | | `status` | number | HTTP status code (e.g. `200`, `404`) | | `headers` | [`Headers`](https://developer.mozilla.org/en-US/docs/Web/API/Headers) | Response headers | | `data` | any | Response data, serialized via content negotiation | | `body` | Buffer \| string \| ReadableStream \| Blob | Raw response body (alternative to `data`) | *** ## `server.ws(listener, options)` Add a handler to the WebSocket connection middleware chain. ``` server.ws(listener: WsListener, options?: WsOptions): HttpServer[] ``` **Example:** ``` server.ws((ws, request, chainCompletion) => { chainCompletion.then(() => { ws.on('message', (data) => console.log('received:', data)); ws.send('hello'); }); }); ``` ### `WsListener` ``` type WsListener = (ws: WebSocket, request: Request, chainCompletion: Promise, next: WsListener) => Promise; ``` | Parameter | Description | | ----------------- | ------------------------------------------------------------------------------------------------------------------------- | | `ws` | [`WebSocket`](https://github.com/websockets/ws/blob/main/doc/ws.md#class-websocket) instance | | `request` | Harper `Request` object from the upgrade event | | `chainCompletion` | `Promise` that resolves when the HTTP request chain finishes. Await before sending to ensure the HTTP request is handled. | | `next` | Continue chain: `next(ws, request, chainCompletion)` | ### `WsOptions` | Property | Type | Default | Description | | ------------ | ------- | ----------------- | ----------------------------------------------- | | `maxPayload` | number | 100 MB | Maximum WebSocket payload size | | `runFirst` | boolean | `false` | Insert this handler at the front of the chain | | `port` | number | `http.port` | Target the WebSocket server on this port | | `securePort` | number | `http.securePort` | Target the secure WebSocket server on this port | *** ## `server.upgrade(listener, options)` Add a handler to the HTTP server `upgrade` event. Use this to delegate upgrade events to an external WebSocket server. ``` server.upgrade(listener: UpgradeListener, options?: UpgradeOptions): void ``` **Example** (from the Harper Next.js component): ``` server.upgrade( (request, socket, head, next) => { if (request.url === '/_next/webpack-hmr') { return upgradeHandler(request, socket, head).then(() => { request.__harperRequestUpgraded = true; next(request, socket, head); }); } return next(request, socket, head); }, { runFirst: true } ); ``` When `server.ws()` is registered, Harper adds a default upgrade handler. The default handler sets `request.__harperRequestUpgraded = true` after upgrading, and checks for this flag before upgrading again (so external upgrade handlers can detect whether Harper has already handled the upgrade). ### `UpgradeListener` ``` type UpgradeListener = (request: IncomingMessage, socket: Socket, head: Buffer, next: UpgradeListener) => void; ``` ### `UpgradeOptions` | Property | Type | Default | Description | | ------------ | ------- | ----------------- | ------------------------------------ | | `runFirst` | boolean | `false` | Insert at the front of the chain | | `port` | number | `http.port` | Target the HTTP server on this port | | `securePort` | number | `http.securePort` | Target the HTTPS server on this port | *** ## `server.socket(listener, options)` Create a raw TCP or TLS socket server. ``` server.socket(listener: ConnectionListener, options: SocketOptions): SocketServer ``` Only one socket server is created per call. A `securePort` takes precedence over `port`. ### `ConnectionListener` Node.js connection listener as in [`net.createServer`](https://nodejs.org/api/net.html#netcreateserveroptions-connectionlistener) or [`tls.createServer`](https://nodejs.org/api/tls.html#tlscreateserveroptions-secureconnectionlistener). ### `SocketOptions` | Property | Type | Description | | ------------ | ------ | -------------------------------------------------------------------------- | | `port` | number | Port for a [`net.Server`](https://nodejs.org/api/net.html#class-netserver) | | `securePort` | number | Port for a [`tls.Server`](https://nodejs.org/api/tls.html#class-tlsserver) | ### `SocketServer` A Node.js [`net.Server`](https://nodejs.org/api/net.html#class-netserver) or [`tls.Server`](https://nodejs.org/api/tls.html#class-tlsserver) instance. *** ## `server.authenticateUser(username, password)` *Added in: v4.5.0* ``` server.authenticateUser(username: string, password: string): Promise ``` Returns the user object for the given username after verifying the password. Throws if the password is incorrect. Use this when you need to explicitly verify a user's credentials (e.g., in a custom login endpoint). For lookup without password verification, use [`server.getUser()`](#servergetuserusername). *** ## `server.getUser(username)` ``` server.getUser(username: string): Promise ``` Returns the user object for the given username without verifying credentials. Use for authorization checks when the user is already authenticated. *** ## `server.resources` The central registry of all resources exported for REST, MQTT, and other protocols. ### `server.resources.set(name, resource, exportTypes?)` Register a resource: ``` class NewResource extends Resource {} server.resources.set('NewResource', NewResource); // Limit to specific protocols: server.resources.set('NewResource', NewResource, { rest: true, mqtt: false }); ``` ### `server.resources.getMatch(path, exportType?)` Find a resource matching a path: ``` server.resources.getMatch('/NewResource/some-id'); server.resources.getMatch('/NewResource/some-id', 'rest'); ``` *** ## `server.operation(operation, context?, authorize?)` Execute an [Operations API](/reference/v5/operations-api/overview.md) operation programmatically. ``` server.operation(operation: object, context?: { username: string }, authorize?: boolean): Promise ``` | Parameter | Type | Description | | ----------- | ---------------------- | ---------------------------------------------------- | | `operation` | object | Operations API request body | | `context` | `{ username: string }` | Optional: execute as this user | | `authorize` | boolean | Whether to apply authorization. Defaults to `false`. | *** ## `server.recordAnalytics(value, metric, path?, method?, type?)` Record a metric into Harper's analytics system. ``` server.recordAnalytics(value: number, metric: string, path?: string, method?: string, type?: string): void ``` | Parameter | Description | | --------- | ---------------------------------------------------------------------------- | | `value` | Numeric value (e.g. duration in ms, bytes) | | `metric` | Metric name | | `path` | Optional URL path for grouping (omit per-record IDs — use the resource name) | | `method` | Optional HTTP method for grouping | | `type` | Optional type for grouping | Metrics are aggregated and available via the [analytics API](/reference/v5/analytics/overview.md). *** ## `server.config` The parsed `harper-config.yaml` configuration object. Read-only access to Harper's current runtime configuration. *** ## `server.nodes` Returns an array of node objects registered in the cluster. ## `server.shards` Returns a map of shard number to an array of associated nodes. ## `server.hostname` Returns the hostname of the current node. ## `server.contentTypes` Returns the `Map` of registered content type handlers. Same as the global [`contentTypes`](#contenttypes) object. *** ## `contentTypes` A [`Map`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map) of content type handlers for HTTP request/response serialization. Harper uses content negotiation: the `Content-Type` header selects the deserializer for incoming requests, and the `Accept` header selects the serializer for responses. ### Built-in Content Types | MIME type | Description | | --------------------- | ------------------ | | `application/json` | JSON | | `application/cbor` | CBOR | | `application/msgpack` | MessagePack | | `text/csv` | CSV | | `text/event-stream` | Server-Sent Events | ### Custom Content Type Handlers Register or replace a handler by setting it on the `contentTypes` map: ``` import { contentTypes } from 'harper'; contentTypes.set('text/xml', { serialize(data) { return '' + serialize(data) + ''; }, q: 0.8, // quality: lower = less preferred during content negotiation }); ``` ### Handler Interface | Property | Type | Description | | --------------------------- | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `serialize(data)` | `(any) => Buffer \| Uint8Array \| string` | Serialize data for a response | | `serializeStream(data)` | `(any) => ReadableStream` | Serialize as a stream (for async iterables or large data) | | `deserialize(buffer)` | `(Buffer \| string) => any` | Deserialize an incoming request body. Used when `deserializeStream` is absent. String for `text/*` types, Buffer for binary types. | | `deserializeStream(stream)` | `(ReadableStream) => any` | Deserialize an incoming request stream | | `q` | number (0–1) | Quality indicator for content negotiation. Defaults to `1`. | *** ## Related * [HTTP Overview](/reference/v5/http/overview.md) * [HTTP Configuration](/reference/v5/http/configuration.md) * [REST Overview](/reference/v5/rest/overview.md) * [Security API](/reference/v5/security/api.md) --- # HTTP Configuration The `http` section in `harper-config.yaml` controls the built-in HTTP server that serves REST, WebSocket, component, and Operations API traffic. Harper must be restarted for configuration changes to take effect. ## Ports ### `http.port` Type: `integer` Default: `9926` The port the HTTP server listens on. This is the primary port for REST, WebSocket, MQTT-over-WebSocket, and component traffic. ### `http.securePort` Type: `integer` Default: `null` The port for HTTPS connections. Requires a valid `tls` section configured with certificate and key. When set, Harper accepts both plaintext (`http.port`) and TLS connections (`http.securePort`) simultaneously. ## TLS TLS is configured in its own top-level `tls` section in `harper-config.yaml`, separate from the `http` section. It is shared by the HTTP server (HTTPS), the MQTT broker (secure MQTT), and any TLS socket servers. See [TLS Configuration](/reference/v5/http/tls.md) for all options including multi-domain (SNI) certificates and the Operations API override. To enable HTTPS, set `http.securePort` and add a `tls` block: ``` http: securePort: 9927 tls: certificate: ~/hdb/keys/certificate.pem certificateAuthority: ~/hdb/keys/ca.pem privateKey: ~/hdb/keys/privateKey.pem ``` ## HTTP/2 ### `http.http2` *Added in: v4.5.0* Type: `boolean` Default: `false` Enables HTTP/2 for all API endpoints. HTTP/2 requires TLS, so `http.securePort` must also be set. ``` http: http2: true securePort: 9927 ``` ## Timeouts and Limits ### `http.headersTimeout` Type: `integer` Default: `60000` (ms) Maximum time in milliseconds the server waits to receive the complete HTTP headers for a request. ### `http.keepAliveTimeout` Type: `integer` Default: `30000` (ms) Milliseconds of inactivity after which the server closes an idle keep-alive connection. ### `http.timeout` Type: `integer` Default: `120000` (ms) Maximum time in milliseconds before a request times out. ### `http.maxHeaderSize` Type: `integer` Default: `16394` (bytes) Maximum allowed size of HTTP request headers. ### `http.requestQueueLimit` Type: `integer` Default: `20000` (ms) The maximum estimated request queue time in milliseconds. When the queue exceeds this limit, requests are rejected with HTTP 503. ## Compression ### `http.compressionThreshold` *Added in: v4.2.0* Type: `number` Default: `1200` (bytes) For clients that support Brotli encoding (`Accept-Encoding: br`), responses larger than this threshold are compressed. Streaming query responses are always compressed for supporting clients, regardless of this setting (since their size is unknown upfront). ``` http: compressionThreshold: 1200 ``` ## CORS ### `http.cors` Type: `boolean` Default: `true` Enables Cross-Origin Resource Sharing, allowing requests from different origins. ### `http.corsAccessList` Type: `string[]` Default: `null` An array of allowed origin domains when CORS is enabled. When `null`, all origins are allowed. ``` http: cors: true corsAccessList: - https://example.com - https://app.example.com ``` ### `http.corsAccessControlAllowHeaders` *Added in: v4.5.0* Type: `string` Default: `"Accept, Content-Type, Authorization"` Comma-separated list of headers allowed in the [`Access-Control-Allow-Headers`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Headers) response header for OPTIONS (preflight) requests. ## Session Affinity ### `http.sessionAffinity` *Added in: v4.1.0* Type: `string` Default: `null` Routes repeated requests from the same client to the same worker thread. This can improve caching locality and provide fairness in request handling. Accepted values: * `ip` — Route by the remote IP address. Use this when Harper is the public-facing server and each client has a distinct IP. * `` — Route by the value of any HTTP header (e.g., `Authorization`). Use this when Harper is behind a proxy where all requests share the same source IP. ``` http: sessionAffinity: ip ``` caution If Harper is behind a reverse proxy and you use `ip`, all requests will share the proxy's IP and will be routed to a single thread. Use a header-based value instead. ## mTLS ### `http.mtls` *Added in: v4.3.0* Type: `boolean | object` Default: `false` Enables mutual TLS (mTLS) authentication for HTTP connections. When set to `true`, client certificates are verified against the CA specified in `tls.certificateAuthority`. Authenticated connections use the `CN` (common name) from the certificate subject as the Harper username. ``` http: mtls: true ``` For granular control, specify an object: | Property | Type | Default | Description | | ------------------------- | ----------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `user` | string | (CN from cert) | Authenticate all mTLS connections as this specific user. Set to `null` to skip credential-based authentication (requires combining with `required: true`). | | `required` | boolean | `false` | Reject any connection that does not provide a valid client certificate. | | `certificateVerification` | boolean \| object | `false` | Enable CRL/OCSP certificate revocation checking. See below. | ### `http.mtls.certificateVerification` *Added in: v4.7.0* (OCSP support) Type: `boolean | object` Default: `false` When mTLS is enabled, Harper can verify the revocation status of client certificates using CRL (Certificate Revocation List) and/or OCSP (Online Certificate Status Protocol). Disabled by default; must be explicitly enabled for environments that require certificate revocation checking. Set to `true` to enable with all defaults, or configure as an object: **Global:** * `failureMode` — `'fail-closed'` (default) | `'fail-open'`. Whether to reject or allow connections when revocation checking fails. **CRL** (enabled by default when `certificateVerification` is enabled): * `crl.enabled` — boolean, default `true` * `crl.timeout` — ms to wait for CRL download, default `10000` * `crl.cacheTtl` — ms to cache CRL, default `86400000` (24h) * `crl.gracePeriod` — ms grace period after CRL `nextUpdate`, default `86400000` (24h) * `crl.failureMode` — CRL-specific failure mode **OCSP** (enabled by default as CRL fallback): * `ocsp.enabled` — boolean, default `true` * `ocsp.timeout` — ms to wait for OCSP response, default `5000` * `ocsp.cacheTtl` — ms to cache successful responses, default `3600000` (1h) * `ocsp.errorCacheTtl` — ms to cache errors, default `300000` (5m) * `ocsp.failureMode` — OCSP-specific failure mode Harper uses a CRL-first strategy with OCSP fallback. If both fail, the configured `failureMode` is applied. **Examples:** ``` # Basic mTLS, no revocation checking http: mtls: true # mTLS with revocation checking (recommended for production) http: mtls: certificateVerification: true # Require mTLS for all connections + revocation checking http: mtls: required: true certificateVerification: true # Custom verification settings http: mtls: certificateVerification: failureMode: fail-closed crl: timeout: 15000 cacheTtl: 43200000 ocsp: timeout: 8000 cacheTtl: 7200000 ``` ## Logging HTTP request logging is disabled by default. Enabling the `http.logging` block turns on request logging. ### `http.logging` *Added in: v4.6.0* Type: `object` Default: disabled ``` http: logging: level: info # info = all requests, warn = 4xx+, error = 5xx path: ~/hdb/log/http.log timing: true # log request timing headers: false # log request headers (verbose) id: true # assign and log a unique request ID ``` The `level` controls which requests are logged: * `info` (or more verbose) — All HTTP requests * `warn` — Requests with status 400 or above * `error` — Requests with status 500 or above ## Complete Example ``` http: port: 9926 securePort: 9927 http2: true cors: true corsAccessList: - null compressionThreshold: 1200 headersTimeout: 60000 keepAliveTimeout: 30000 timeout: 120000 maxHeaderSize: 16384 requestQueueLimit: 20000 sessionAffinity: null mtls: false logging: level: warn path: ~/hdb/log/http.log timing: true # tls is a top-level section — see TLS Configuration tls: certificate: ~/hdb/keys/certificate.pem certificateAuthority: ~/hdb/keys/ca.pem privateKey: ~/hdb/keys/privateKey.pem ``` ## Related * [HTTP Overview](/reference/v5/http/overview.md) * [HTTP API](/reference/v5/http/api.md) * [TLS Configuration](/reference/v5/http/tls.md) * [Security Overview](/reference/v5/security/overview.md) * [Configuration Overview](/reference/v5/configuration/overview.md) --- # HTTP Server Harper includes a built-in HTTP server that serves as the primary interface for REST, WebSocket, MQTT-over-WebSocket, and component-defined endpoints. The same server handles all application traffic on a configurable port (default `9926`). ## Architecture Harper's HTTP server is multi-threaded. Each thread runs an independent copy of the HTTP stack, and incoming connections are distributed across threads using `SO_REUSEPORT` socket sharing — the most performant mechanism available for multi-threaded socket handling. *Added in: v4.1.0* (worker threads for HTTP requests) *Changed in: v4.2.0* (switched from process-per-thread model with session-affinity delegation to `SO_REUSEPORT` socket sharing) In previous versions: Session-affinity based socket delegation was used to route requests. This has been deprecated in favor of `SO_REUSEPORT`. ## Request Handling Harper uses a layered middleware chain for HTTP request processing. Components and applications can add handlers to this chain using the [`server.http()`](/reference/v5/http/api.md#serverhttplistener-options) API. Handlers are called in order; each handler can either process the request and return a `Response`, or pass it along to the next handler with `next(request)`. Request and response objects follow the [WHATWG Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) conventions (`Request` and `Response` classes), providing good composability for layered middleware and clean mapping to REST resource handlers. ## Protocols Served The HTTP server handles multiple protocols on the same port: * **REST** — CRUD operations on Harper resources via standard HTTP methods * **WebSockets** — Real-time bidirectional connections (via `server.ws()`) * **MQTT over WebSocket** — MQTT clients connecting over WebSocket (sub-protocol `mqtt`) * **Server-Sent Events** — Streaming updates to browser clients * **Operations API** — Management API (configurable to share or use separate port) ## TLS / HTTPS HTTPS support is enabled by setting `http.securePort` in `harper-config.yaml` and configuring the `tls` section with a certificate and private key. The same `tls` configuration is shared by HTTPS and MQTT secure connections. See [Configuration](/reference/v5/http/configuration.md) for TLS options and [Security](/reference/v5/security/overview.md) for certificate management details. ## HTTP/2 *Added in: v4.5.0* HTTP/2 can be enabled with the `http2: true` option in `harper-config.yaml`. When enabled, HTTP/2 applies to all API endpoints served on `http.securePort` (HTTP/2 requires TLS). ## Compression Harper automatically compresses HTTP responses using Brotli for clients that advertise `Accept-Encoding: br`. Compression applies when the response body exceeds the configured `compressionThreshold` (default 1200 bytes). Streaming query responses are always compressed for clients that support it (since their size is not known upfront). ## Logging HTTP request logging is not enabled by default. To enable it, add an `http.logging` block to your configuration. See [Configuration](/reference/v5/http/configuration.md#logging) for details. ## Related * [HTTP Configuration](/reference/v5/http/configuration.md) * [HTTP API](/reference/v5/http/api.md) * [REST Overview](/reference/v5/rest/overview.md) * [Security Overview](/reference/v5/security/overview.md) --- # TLS Configuration Harper uses a top-level `tls` section in `harper-config.yaml` to configure Transport Layer Security. This configuration is shared by the HTTP server (HTTPS), the MQTT broker (secure MQTT), and any TLS socket servers created via the [HTTP API](/reference/v5/http/api.md#serversocketlistener-options). The `operationsApi` section can optionally define its own `tls` block, which overrides the root `tls` for Operations API traffic only. See the [Operations API Configuration](/reference/v5/configuration/operations.md) for more details. Harper must be restarted for TLS configuration changes to take effect. ## TLS Configuration ``` tls: certificate: ~/hdb/keys/certificate.pem certificateAuthority: ~/hdb/keys/ca.pem privateKey: ~/hdb/keys/privateKey.pem ``` ### `tls.certificate` Type: `string` Default: `"/keys/certificate.pem"` Path to the PEM-encoded certificate file. ### `tls.certificateAuthority` Type: `string` Default: `"/keys/ca.pem"` Path to the PEM-encoded certificate authority (CA) file. Used to verify client certificates when mTLS is enabled. ### `tls.privateKey` Type: `string` Default: `"/keys/privateKey.pem"` Path to the PEM-encoded private key file. ### `tls.host` Type: `string | undefined` The domain name this certificate entry applies to, used for SNI (Server Name Indication) matching. Only relevant when `tls` is defined as an array. When omitted, the certificate's common name (CN) is used as the host name. ### `tls.ciphers` Type: `string | undefined` Default: `crypto.defaultCipherList` Colon-separated list of allowed TLS cipher suites. When omitted, Node.js [default ciphers](https://nodejs.org/api/crypto.html#nodejs-crypto-constants) are used. See Node.js [Modifying the default TLS cipher suite](https://nodejs.org/api/tls.html#modifying-the-default-tls-cipher-suite) for more information. ## Enabling HTTPS To enable HTTPS, set `http.securePort` in addition to the `tls` section: ``` http: securePort: 9927 tls: certificate: ~/hdb/keys/certificate.pem certificateAuthority: ~/hdb/keys/ca.pem privateKey: ~/hdb/keys/privateKey.pem ``` When `http.securePort` is set, Harper accepts plaintext connections on `http.port` and TLS connections on `http.securePort` simultaneously. ## Multi-Domain Certificates (SNI) To serve different certificates for different domains using Server Name Indication (SNI), define `tls` as an array of configuration objects. Each entry can optionally include a `host` property specifying which domain it applies to. If `host` is omitted, the certificate's common name and subject alternate names (SANs) are used. ``` tls: - certificate: ~/hdb/keys/certificate1.pem certificateAuthority: ~/hdb/keys/ca1.pem privateKey: ~/hdb/keys/privateKey1.pem host: example.com - certificate: ~/hdb/keys/certificate2.pem certificateAuthority: ~/hdb/keys/ca2.pem privateKey: ~/hdb/keys/privateKey2.pem # host omitted: certificate's CN is used ``` ## Operations API Override The `operationsApi` section can define its own `tls` block to use a separate certificate for the Operations API: ``` tls: certificate: ~/hdb/keys/certificate.pem certificateAuthority: ~/hdb/keys/ca.pem privateKey: ~/hdb/keys/privateKey.pem operationsApi: network: securePort: 9924 tls: certificate: ~/hdb/keys/ops-certificate.pem certificateAuthority: ~/hdb/keys/ops-ca.pem privateKey: ~/hdb/keys/ops-privateKey.pem ``` See the [Operations API Configuration](/reference/v5/configuration/operations.md) for more details. ## Related * [HTTP Configuration](/reference/v5/http/configuration.md) — `http.securePort`, `http.http2`, `http.mtls` * [HTTP Overview](/reference/v5/http/overview.md) * [Security mTLS Authentication](/reference/v5/security/mtls-authentication.md) --- # Harper Cloud Harper Cloud (also sometimes referred to as Harper Studio) was Harper's original PaaS offering. It has been fully replaced by [Harper Fabric](https://fabric.harper.fast). All users are encouraged to migrate or get started using Harper Fabric immediately. [Local Studio](/reference/v5/studio/overview.md) is still an available feature, and now uses the same client as Harper Fabric. Reach out to or join our community [Discord](https://harper.fast/discord) if you have questions. --- # Custom Functions Custom Functions were Harper's original mechanism for adding custom API endpoints and application logic to a Harper instance. They allowed developers to define Fastify-based HTTP routes that ran inside Harper with direct access to the database, and could be deployed across instances via Studio. Custom Functions were superseded by the [Components](/reference/v5/components/overview.md) system introduced in v4.2.0. Components provide the same capabilities with a more robust architecture, better tooling, and support for extensions and plugins. All users are encouraged to migrate Custom Functions to Components. See the [Components](/reference/v5/components/overview.md) documentation for the modern approach. Reach out to or join our community [Discord](https://harper.fast/discord) if you have questions. --- # Logging API ## `logger` The `logger` global is available in all JavaScript components without any imports. It writes structured log entries to the standard Harper log file (`hdb.log`) at the configured `logging.external` level and path. See [Logging Configuration](/reference/v5/logging/configuration.md#loggingexternal) for per-component log configuration. The `logger` global is a `MainLogger`. Calling `logger.withTag(tag)` returns a `TaggedLogger` scoped to that tag. ### `MainLogger` `MainLogger` always has all log-level methods defined. It also exposes `withTag()` to create a `TaggedLogger`. ``` interface MainLogger { trace(...messages: any[]): void; debug(...messages: any[]): void; info(...messages: any[]): void; warn(...messages: any[]): void; error(...messages: any[]): void; fatal(...messages: any[]): void; notify(...messages: any[]): void; withTag(tag: string): TaggedLogger; } ``` Each method corresponds to a log level. Only entries at or above the configured `logging.level` (or `logging.external.level`) are written. See [Log Levels](/reference/v5/logging/overview.md#log-levels) for the full hierarchy. ### `TaggedLogger` `TaggedLogger` is returned by `logger.withTag(tag)`. It prefixes every log entry with the given tag, making it easy to filter log output by component or context. Because `TaggedLogger` is bound to the configured log level at creation time, methods for levels that are currently disabled are `null`. Always use optional chaining (`?.`) when calling methods on a `TaggedLogger`. ``` interface TaggedLogger { trace: ((...messages: any[]) => void) | null; debug: ((...messages: any[]) => void) | null; info: ((...messages: any[]) => void) | null; warn: ((...messages: any[]) => void) | null; error: ((...messages: any[]) => void) | null; fatal: ((...messages: any[]) => void) | null; notify: ((...messages: any[]) => void) | null; } ``` `TaggedLogger` does not have a `withTag()` method. ### Console Capture When `logging.console: true` is set, writes to `process.stdout` and `process.stderr` from component code are also written verbatim to the Harper log file. This includes anything written via `console.log`, `console.warn`, `console.error`, etc. Captured lines do not pass through `logger`'s level filter — they are appended as-is. Prefer `logger` directly in production code so that level filtering and tagging apply. Console capture is intended as a convenience for porting existing code and for debugging. ### Usage #### Basic logging with `logger` ``` export class MyResource extends Resource { async get(id) { logger.debug('Fetching record', { id }); const record = await super.get(id); if (!record) { logger.warn('Record not found', { id }); } return record; } async put(record) { logger.info('Updating record', { id: record.id }); try { return await super.put(record); } catch (err) { logger.error('Failed to update record', err); throw err; } } } ``` #### Tagged logging with `withTag()` Create a tagged logger once per module or class and reuse it. Always use `?.` when calling methods since a given level may be `null` if it is below the configured log level. ``` const log = logger.withTag('my-resource'); export class MyResource extends Resource { async get(id) { log.debug?.('Fetching record', { id }); const record = await super.get(id); if (!record) { log.warn?.('Record not found', { id }); } return record; } async put(record) { log.info?.('Updating record', { id: record.id }); try { return await super.put(record); } catch (err) { log.error?.('Failed to update record', err); throw err; } } } ``` Tagged entries appear in the log with the tag included in the entry header: ``` 2023-03-09T14:25:05.269Z [info] [my-resource]: Updating record ``` ### Log Entry Format Entries written via `logger` appear in `hdb.log` with the standard format: ``` [] [/]: ``` Entries written via a `TaggedLogger` include the tag: ``` [] []: ``` For external components, the thread context is set automatically based on which worker thread executes the code. ## Related * [Logging Overview](/reference/v5/logging/overview.md) * [Logging Configuration](/reference/v5/logging/configuration.md) * [Logging Operations](/reference/v5/logging/operations.md) --- # Logging Configuration The `logging` section in `harper-config.yaml` controls standard log output. Many logging settings are applied dynamically without a restart (added in v4.6.0). ## Main Logger ### `logging.level` Type: `string` Default: `warn` Controls the verbosity of logs. Levels from least to most severe: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `notify`. Setting a level includes that level and all more-severe levels. ``` logging: level: warn ``` For example, `level: warn` results in `warn`, `error`, `fatal`, and `notify` logs. ### `logging.path` Type: `string` Default: `/log/hdb.log` Full file path for the log file. ``` logging: path: ~/hdb/log/hdb.log ``` ### `logging.root` Type: `string` Default: `/log` Directory path where log files are written. Use `path` to specify the full filename; use `root` to specify only the directory (Harper determines the filename). ``` logging: root: ~/hdb/log ``` ### `logging.file` Type: `boolean` Default: `true` Whether to write logs to a file. Disable if you want to use only standard streams. ``` logging: file: true ``` ### `logging.stdStreams` Type: `boolean` Default: `false` Log to `stdout`/`stderr` in addition to (or instead of) the log file. When enabled, run Harper in the foreground (`harper`, not `harper start`). ``` logging: stdStreams: true ``` ### `logging.console` Type: `boolean` Default: `false` Controls whether `console.log` and other `console.*` calls (and anything writing to `process.stdout`/`process.stderr` from JS components) are captured to the log file. ``` logging: console: true ``` ### `logging.auditLog` Type: `boolean` Default: `false` Enables audit (table transaction) logging. When enabled, Harper records every insert, update, and delete to a corresponding audit table. Audit log data is accessed via the `read_audit_log` operation. See [Database / Transaction Logging](/reference/v5/database/transaction.md) for details on using audit logs. ``` logging: auditLog: false ``` ### `logging.auditRetention` Type: `string | number` Default: `3d` How long audit log entries are retained before automatic eviction. Accepts duration strings (e.g., `3d`, `12h`) or milliseconds. ``` logging: auditRetention: 3d ``` ## Log Rotation Rotation provides systematic management of the `hdb.log` file — compressing, archiving, and replacing it on a schedule or size threshold. Rotation is triggered when either `interval` or `maxSize` is set. > `interval` and `maxSize` are approximates only. The log file may exceed these values slightly before rotation occurs. ### `logging.rotation.enabled` Type: `boolean` Default: `true` Enables log rotation. Rotation only activates when `interval` or `maxSize` is also set. ### `logging.rotation.compress` Type: `boolean` Default: `false` Compress rotated log files with gzip. ### `logging.rotation.interval` Type: `string` Default: `null` Time between rotations. Accepted units: `D` (days), `H` (hours), `M` (minutes). Example: `1D`, `12H`. ### `logging.rotation.maxSize` Type: `string` Default: `null` Maximum log file size before rotation. Accepted units: `K` (kilobytes), `M` (megabytes), `G` (gigabytes). Example: `100M`, `1G`. ### `logging.rotation.path` Type: `string` Default: `/log` Directory for storing rotated log files. Rotated files are named: `HDB-YYYY-MM-DDT-HH-MM-SSSZ.log`. ``` logging: rotation: enabled: true compress: false interval: 1D maxSize: 100M path: ~/hdb/log ``` ## Authentication Logging ### `logging.auditAuthEvents.logFailed` *Added in: v4.2.0* Type: `boolean` Default: `false` Log all failed authentication attempts. Example log entry: ``` [error] [auth-event]: {"username":"admin","status":"failure","type":"authentication","originating_ip":"127.0.0.1","request_method":"POST","path":"/","auth_strategy":"Basic"} ``` ### `logging.auditAuthEvents.logSuccessful` *Added in: v4.2.0* Type: `boolean` Default: `false` Log all successful authentication events. Example log entry: ``` [notify] [auth-event]: {"username":"admin","status":"success","type":"authentication","originating_ip":"127.0.0.1","request_method":"POST","path":"/","auth_strategy":"Basic"} ``` ``` logging: auditAuthEvents: logFailed: false logSuccessful: false ``` ## Per-Component Logging *Added in: v4.6.0* Harper supports independent logging configurations for different components. Each component logger can have its own `path`, `root`, `level`, `tag`, and `stdStreams` settings. All components default to the main `logging` configuration unless overridden. ### `logging.external` Logging configuration for all external components that use the [`logger` API](/reference/v5/logging/api.md). ``` logging: external: level: warn path: ~/hdb/log/apps.log ``` ### `http.logging` HTTP request logging. Disabled by default — defining this section enables it. ``` http: logging: level: info # info = all requests, warn = 4xx+, error = 5xx path: ~/hdb/log/http.log timing: true # log request duration headers: false # log request headers (verbose) id: true # assign and log a unique request ID per request ``` See [HTTP Configuration](/reference/v5/http/configuration.md) for full details. ### `mqtt.logging` MQTT logging configuration. Accepts standard logging options. ``` mqtt: logging: level: warn path: ~/hdb/log/mqtt.log stdStreams: false ``` ### `authentication.logging` Authentication subsystem logging. Accepts standard logging options. ``` authentication: logging: level: warn path: ~/hdb/log/auth.log ``` ### `replication.logging` Replication subsystem logging. Accepts standard logging options. ``` replication: logging: level: warn path: ~/hdb/log/replication.log ``` ### `tls.logging` TLS subsystem logging. Accepts standard logging options. ``` tls: logging: level: warn path: ~/hdb/log/tls.log ``` ### `storage.logging` Database storage subsystem logging. Accepts standard logging options. ``` storage: logging: level: warn path: ~/hdb/log/storage.log ``` ### `analytics.logging` Analytics subsystem logging. Accepts standard logging options. ``` analytics: logging: level: warn path: ~/hdb/log/analytics.log ``` ## Clustering Log Level Clustering has a separate log level due to its verbosity. Configure with `clustering.logLevel`. Valid levels from least verbose: `error`, `warn`, `info`, `debug`, `trace`. ``` clustering: logLevel: warn ``` ## Complete Example ``` logging: level: warn path: ~/hdb/log/hdb.log file: true stdStreams: false console: false auditLog: false auditRetention: 3d rotation: enabled: true compress: false interval: 1D maxSize: 100M path: ~/hdb/log auditAuthEvents: logFailed: false logSuccessful: false external: level: warn path: ~/hdb/log/apps.log http: logging: level: warn path: ~/hdb/log/http.log timing: true ``` ## Related * [Logging Overview](/reference/v5/logging/overview.md) * [Logging API](/reference/v5/logging/api.md) * [Logging Operations](/reference/v5/logging/operations.md) * [Database / Transaction Logging](/reference/v5/database/transaction.md) * [Configuration Overview](/reference/v5/configuration/overview.md) --- # Logging Operations Operations for reading the standard Harper log (`hdb.log`). All operations are restricted to `super_user` roles only. > Audit log and transaction log operations (`read_audit_log`, `read_transaction_log`, `delete_audit_logs_before`, `delete_transaction_logs_before`) are documented in [Database / Transaction Logging](/reference/v5/database/transaction.md). *** ## `read_log` Returns log entries from the primary Harper log (`hdb.log`) matching the provided criteria. *Restricted to super\_user roles only.* ### Parameters | Parameter | Required | Type | Description | | ----------- | -------- | ------ | ------------------------------------------------------------------------------------------------------ | | `operation` | Yes | string | Must be `"read_log"` | | `start` | No | number | Result offset to start from. Default: `0` (first entry in `hdb.log`). | | `limit` | No | number | Maximum number of entries to return. Default: `1000`. | | `level` | No | string | Filter by log level. One of: `notify`, `error`, `warn`, `info`, `debug`, `trace`. Default: all levels. | | `from` | No | string | Start of time window. Format: `YYYY-MM-DD` or `YYYY-MM-DD hh:mm:ss`. Default: first entry in log. | | `until` | No | string | End of time window. Format: `YYYY-MM-DD` or `YYYY-MM-DD hh:mm:ss`. Default: last entry in log. | | `order` | No | string | Sort order: `asc` or `desc` by timestamp. Default: maintains `hdb.log` order. | | `filter` | No | string | A substring that must appear in each returned log line. | ### Request ``` { "operation": "read_log", "start": 0, "limit": 1000, "level": "error", "from": "2021-01-25T22:05:27.464+0000", "until": "2021-01-25T23:05:27.464+0000", "order": "desc" } ``` ### Response ``` [ { "level": "notify", "message": "Connected to cluster server.", "timestamp": "2021-01-25T23:03:20.710Z", "thread": "main/0", "tags": [] }, { "level": "warn", "message": "Login failed", "timestamp": "2021-01-25T22:24:45.113Z", "thread": "http/9", "tags": [] }, { "level": "error", "message": "unknown attribute 'name and breed'", "timestamp": "2021-01-25T22:23:24.167Z", "thread": "http/9", "tags": [] } ] ``` ### Response Fields | Field | Type | Description | | ----------- | ------ | ------------------------------------------------------------------------------------------------------- | | `level` | string | Log level of the entry. | | `message` | string | Log message. | | `timestamp` | string | ISO 8601 timestamp when the event occurred. | | `thread` | string | Thread name and ID (e.g., `main/0`, `http/3`). | | `tags` | array | Additional context tags. Entries from components may include `custom-function` or other component tags. | ## Related * [Logging Overview](/reference/v5/logging/overview.md) * [Logging Configuration](/reference/v5/logging/configuration.md) * [Database / Transaction Logging](/reference/v5/database/transaction.md) * [Operations API Overview](/reference/v5/operations-api/overview.md) --- # Logging Harper's core logging system is used for diagnostics, monitoring, and observability. It has an extensive configuration system, and even supports feature-specific (per-component) configurations in latest versions. Furthermore, the `logger` global API is available for creating custom logs from any JavaScript application or plugin code. > If you are looking for information on Harper's Audit and Transaction logging system, refer to the [Database](/reference/v5/database/transaction.md) section. ## Log File *Changed in: v4.1.0* — All logs consolidated into a single `hdb.log` file All standard log output is written to `/log/hdb.log` (default: `~/hdb/log/hdb.log`). ## Log Entry Format Each log entry follows this structure: ``` [/] [] ...[]: ``` Example: ``` 2023-03-09T14:25:05.269Z [main/0] [notify]: HarperDB successfully started. ``` Fields: | Field | Description | | ----------- | -------------------------------------------------------------------------------------------- | | `timestamp` | ISO 8601 date/time when the event occurred. | | `level` | Severity level. See [Log Levels](#log-levels) below. | | `thread/id` | Name and ID of the thread that produced the log entry (generally, `main`, `http`, or `job`). | | `tags` | Additional context tags (e.g., `custom-function`, `auth-event`). Most entries have no tags. | | `message` | The log message. | ### Log Levels From least to most severe (most verbose to least verbose): | Level | Description | | -------- | --------------------------------------------------------------------------------------------- | | `trace` | Highly detailed internal execution tracing. | | `debug` | Diagnostic information useful during development. | | `info` | General operational events. | | `warn` | Potential issues that don't prevent normal operation. | | `error` | Errors that affect specific operations. | | `fatal` | Critical errors causing process termination. | | `notify` | Important operational milestones (e.g., "server started"). Always logged regardless of level. | The default log level is `warn`. Setting a level includes that level and all more-severe levels. For example, `warn` logs `warn`, `error`, `fatal`, and `notify`. ## Standard Streams *Changed in: v4.6.0* By default, logs are written only to the log file. To also log to `stdout`/`stderr`, set [`logging.stdStreams: true`](/reference/v5/logging/configuration.md#loggingstdstreams) (this is automatically enabled by the `DEFAULT_MODE=dev` configuration during installation). When logging to standard streams, run Harper in the foreground (i.e. `harper`, not `harper start`). As of v4.6.0, logging to standard streams does **not** include timestamps, and console logging (`console.log`, etc.) does not get forwarded to log files unless the [`logging.console: true`](/reference/v5/logging/configuration.md#loggingconsole) option is enabled. ## Logger API JavaScript components can use the `logger` global to write structured log entries: ``` logger.trace('detailed trace message'); logger.debug('debug info', { someContext: 'value' }); logger.info('informational message'); logger.warn('potential issue'); logger.error('error occurred', error); logger.fatal('fatal error'); logger.notify('server is ready'); ``` The `logger` global provides `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `notify` methods. The logger is based on the Node.js Console API. See [Logging API](/reference/v5/logging/api.md) for full details. ## Related * [Logging Configuration](/reference/v5/logging/configuration.md) * [Logging API](/reference/v5/logging/api.md) * [Logging Operations](/reference/v5/logging/operations.md) * [Database / Transaction Logging](/reference/v5/database/transaction.md) --- # Harper MCP CLI *Added in: v5.1.0* `harper mcp` is a stdio bridge that lets MCP hosts (Claude Desktop, Cursor, Zed, or any client that speaks the [stdio MCP transport](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#stdio)) talk to a running Harper instance over Harper's Streamable HTTP MCP endpoint. The CLI is bundled with Harper itself; if `harper` is on your `PATH`, so is `harper mcp`. ## Subcommands ``` harper mcp [subcommand] [flags] ``` | Subcommand | Purpose | | -------------- | ------------------------------------------------------------------------------------------------ | | *(default)* | Run the stdio bridge — JSON-RPC frames on stdin, responses on stdout, until stdin closes. | | `print-config` | Emit a paste-ready config block for an MCP host (`--client claude-desktop`, `cursor`, or `zed`). | | `doctor` | Connect, complete an `initialize` handshake, list tools, clean up; report each step. | | `help` | Print the help text. | ## Connection modes `harper mcp` connects in one of two modes, chosen automatically based on whether `--target` is set: ### Local UDS (default) With no `--target` flag, the CLI connects to the Harper running on the same host via the operations API Unix Domain Socket — the same socket `bin/cliOperations` uses. Filesystem permissions on the socket are the access gate; no credentials are required or sent. The UDS path is derived from `operationsApi.network.domainSocket` in `harperdb-config.yaml` and is typically `/sockets/operations-server`. ### Network HTTPS / HTTP `--target https://node.example.com:9926` connects over the network to a remote Harper. Credentials are resolved with this precedence (highest first): 1. `--bearer ` — explicit bearer token. 2. `--username` + `--password` — explicit Basic auth. 3. URL-embedded user/pass, e.g. `--target https://alice:pw@node:9926`. 4. Saved JWT from `~/.harperdb/credentials.json` for the resolved target (populated by `harper login`). If none of these are present, the request goes out unauthenticated and Harper gates the response accordingly. Use `--insecure` to skip TLS certificate validation (network mode only) — useful for local self-signed certs during development. ## Flags | Flag | Default | Purpose | | --------------------- | ----------------------------- | --------------------------------------------------------------------------------- | | `--profile ` | `application` | `operations` or `application`. Determines which MCP endpoint the CLI connects to. | | `--target ` | *(local UDS)* | Network endpoint. Switches the CLI into network mode. | | `--mount-path ` | `/mcp` | Overrides the mount path. Match whatever `mcp..mountPath` is set to. | | `--username ` | *(none)* | Basic auth username (network mode). | | `--password

` | *(none)* | Basic auth password (network mode). | | `--bearer ` | *(none)* | Bearer token (network mode). Wins over `--username` / `--password`. | | `--insecure` | *(off)* | Skip TLS certificate validation (network mode only). | | `--client ` | *(required for print-config)* | `claude-desktop`, `cursor`, or `zed`. | | `--help`, `-h` | *(off)* | Print the help text and exit. | ## `harper mcp` (the bridge) The default subcommand runs until stdin closes. The expected use is to invoke it from an MCP host's configuration block (see [print-config](#harper-mcp-print-config) below). Each line of stdin is parsed as a JSON-RPC frame and POSTed to the Harper MCP endpoint; each response (whether JSON or SSE-streamed) is emitted to stdout as line-delimited JSON-RPC. After the `initialize` handshake completes, the bridge opens a long-lived `GET /mcp` request — Harper's server-push channel — and forwards every `notifications/tools/list_changed` and `notifications/resources/list_changed` frame to stdout. The host sees one unified MCP stream. Logs (status, errors, dropped invalid stdin lines) go to stderr so they never collide with the JSON-RPC channel. ## `harper mcp print-config` Emits a paste-ready JSON block for the requested MCP host along with a comment indicating where to put it. ``` harper mcp print-config --client claude-desktop ``` Produces: ``` # Target file: ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows) { "mcpServers": { "harper": { "command": "harper", "args": ["mcp"] } } } # Note: Restart Claude Desktop after editing the file. # Note: Merge into an existing `mcpServers` block if you already have one. ``` The generated `args` array reflects whatever flags you pass to `print-config` (other than `--client`). For instance: ``` harper mcp print-config --client cursor --target https://node.example.com:9926 --profile operations ``` emits a block whose `args` is `["mcp", "--profile", "operations", "--target", "https://node.example.com:9926"]`. This lets you generate fully-resolved config blocks for non-default deployments without hand-editing. Supported clients: * **`claude-desktop`** — Anthropic Claude Desktop (`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS, `%APPDATA%\Claude\claude_desktop_config.json` on Windows). * **`cursor`** — Cursor IDE (`~/.cursor/mcp.json`). * **`zed`** — Zed editor (Zed `settings.json`, under `context_servers`). ## `harper mcp doctor` Runs a three-step smoke check against the configured connection: 1. POST `initialize` — verifies the transport works and Harper accepts the handshake. Captures the negotiated protocol version and session id. 2. POST `tools/list` — verifies the session is usable and reports how many tools are visible to the authenticated user. 3. DELETE `/mcp` — cleans up the session. This step is allowed to fail (when `mcp.session.allowClientDelete` is `false` the server returns 405); overall doctor exit is still success. Each step prints `[OK]` or `[FAIL]` with a short detail line. Exit code is `0` on success, `1` on any non-tolerable failure. ``` $ harper mcp doctor --target https://node.example.com:9926 --bearer $TOKEN [OK ] initialize - session=ab12... protocol=2025-06-18 [OK ] tools/list - 14 tool(s) visible [OK ] session cleanup All checks passed. ``` Use `doctor` as a quick "is the wire path healthy?" check before pointing a real MCP host at the server. --- # MCP Configuration *Added in: v5.1.0* All MCP configuration lives under the top-level `mcp:` block in `harperdb-config.yaml`. Each profile (`operations`, `application`) is enabled by the **presence** of its sub-block — there is no separate `enabled` flag. A minimal "turn it on" config is therefore just: ``` mcp: operations: {} application: {} ``` That boots both profiles with default settings: the operations profile mounts at `/mcp` on the operations server, the application profile mounts at `/mcp` on the application HTTP server, and the default allow lists / rate limits apply. ## `mcp.operations.*` Configures the operations-profile MCP endpoint that wraps Harper's operation catalog. ### `mcp.operations.mountPath` Type: `string` Default: `/mcp` URL path the MCP endpoint mounts on. Change it if `/mcp` collides with another route in your application. ### `mcp.operations.allow` Type: `array` (glob patterns or literal operation names) Default: `['describe_*', 'list_*', 'search_*', 'get_job', 'get_status', 'get_analytics', 'get_metrics', 'system_information', 'read_log', 'read_audit_log']` Operations exposed as MCP tools. Glob `*` matches any sequence of characters; literal names match exactly. Setting `allow` **replaces** the default list; it does not merge. To add destructive or sensitive operations to the surface (e.g. `set_configuration`, `drop_table`), include them here explicitly. The default list intentionally avoids `get_*` as a glob because that pulls in `get_configuration` (which can return TLS / S3 / authentication secrets), `get_components` / `get_component_file` / `get_custom_function*` (which return component source that can embed secrets), `get_backup`, and `get_deployment*`. These are all gated by `verifyPerms` at dispatch, but defaulting to "expose them to an LLM if a super\_user calls them" is the wrong default — opt them in deliberately. ### `mcp.operations.deny` Type: `array` (glob patterns or literal operation names) Default: `[]` Operations to filter out **after** the allow list has been applied. Useful for taking back a single operation that a broad allow glob would otherwise expose. ### `mcp.operations.maxTools` Type: `integer` (minimum 1) Default: `200` Maximum number of tools returned in a single `tools/list` response page. The MCP cursor is used to page through any overflow. ### `mcp.operations.rateLimit.*` See [`mcp..rateLimit.*`](#mcpprofileratelimit) below — the schema is identical for both profiles. Default per-profile values for `operations`: `perToolPerSecond: 10`, `perToolBurst: 20`, `sessionConcurrency: 25`, `sessionPerSecond: 100`. ## `mcp.application.*` Configures the application-profile MCP endpoint that walks your exported `Resource` classes. All `mcp.operations.*` keys above also apply here; the additional knob is: ### `mcp.application.searchMaxResults` Type: `integer` (minimum 1) Default: `1000` Hard cap on the number of records a generated `search_` tool can return per call. Clients still pass `limit`, but the server clamps to this ceiling regardless of what the client requests — bounded to keep a runaway agent from exhausting memory. Default per-profile rate-limit values for `application`: `perToolPerSecond: 25`, `perToolBurst: 50`, `sessionConcurrency: 50`, `sessionPerSecond: 200`. ## `mcp..rateLimit.*` Per-session, per-tool token-bucket rate limits. A bucket exists per `(session, tool)` pair plus one per-session bucket across all tools. When either bucket is exhausted, `tools/call` returns `result.isError = true` with `kind: "rate_limited"` — **not** a JSON-RPC error — so the LLM can read the message and back off without the transport tearing down. ### `mcp..rateLimit.perToolPerSecond` Type: `number` (minimum 0) Default: `10` (operations) / `25` (application) Sustained rate at which the per-tool token bucket refills. Set to `0` to disable per-tool throttling on this profile. ### `mcp..rateLimit.perToolBurst` Type: `number` (minimum 0) Default: `20` (operations) / `50` (application) Burst capacity of the per-tool token bucket — how many back-to-back calls a single tool can absorb before sustained-rate refill kicks in. ### `mcp..rateLimit.sessionConcurrency` Type: `integer` (minimum 0) Default: `25` (operations) / `50` (application) Maximum number of `tools/call` invocations a single session may have in flight at once. Subsequent attempts return `kind: "rate_limited"` with `scope: "concurrency"`. ### `mcp..rateLimit.sessionPerSecond` Type: `number` (minimum 0) Default: `100` (operations) / `200` (application) Sustained per-session rate across **all** tools combined. Protects a worker from a single session that spreads its calls across many distinct tools (and so would otherwise dodge `perTool*`). ### `mcp..rateLimit.perClientPerSecond` Type: `number` (minimum 0) Default: `0` (disabled) Sustained `tools/call` rate keyed on **client identity** rather than session (5.2.0+). Session-scoped buckets can be evaded by an anonymous client that cycles sessions (`initialize` → call → drop → repeat); the client bucket survives that loop. Identity is the client socket IP by default, or the value derived from [`identityHeader`](#mcpprofileratelimitidentityheader). Denials surface like other rate-limit hits: an `isError` tool result with `kind: 'rate_limited'`, `scope: 'per_client'`. Like the other buckets, state is in-memory per worker — it does not survive a restart and is not shared across workers. For durable quotas, see [`mcp..quota.*`](#mcpprofilequota). ### `mcp..rateLimit.perClientBurst` Type: `number` (minimum 0) Default: the `perClientPerSecond` value, floored at `1` Burst capacity of the per-client bucket. Defaults to the sustained rate so enabling the limit is a one-key change — floored at 1 whole token: a fractional `perClientPerSecond` (e.g. `0.1` for "6 per minute") still yields `perClientBurst: 1` by default, since a bucket capped below one token could never admit a call. ### `mcp..rateLimit.identityHeader` Type: `string` Default: unset (client identity = socket IP) Name of a trusted header whose first (client-most) value supplies client identity — for deployments behind a reverse proxy, where every socket IP is the proxy's. Typically `x-forwarded-for`. **Only set this when the fronting proxy strips or replaces the header on untrusted traffic.** A client-controlled identity header lets callers mint fresh identities per request and bypass per-client limits entirely; Harper logs a startup warning when this key is configured. ## `mcp..quota.*` An operator-pluggable **durable** quota hook for `tools/call` (5.2.0+). The in-memory buckets above bound instantaneous rates but reset on restart and are per-worker — insufficient as a cost control for a public, unauthenticated, cost-bearing tool (an LLM-backed `answer`, say). This hook delegates the policy to your code, where it can be backed by a table: ``` mcp: application: quota: resource: McpQuota ``` ``` // resources.js const DAILY_LIMIT = 100; export class McpQuota extends tables.QuotaCounter { static async allowMcpCall({ identity, tool, user, profile, sessionId }) { const id = identity ?? 'unknown'; const existing = await McpQuota.get(id); const used = (existing?.used ?? 0) + 1; await McpQuota.put({ id, used }); if (used > DAILY_LIMIT) { return { allowed: false, message: 'daily quota reached', retryAfterSeconds: 3600 }; } return true; } } ``` ### `mcp..quota.resource` Type: `string` Default: unset (no durable quota) Path of an exported Resource whose static quota method Harper calls before each admitted `tools/call`. Dispatch uses the live registry class, so an exported subclass (and its policy) wins on reload. ### `mcp..quota.method` Type: `string` Default: `allowMcpCall` Name of the static method to call. It receives `{ identity, tool, user, profile, sessionId }` (`identity` may be `undefined` when no socket IP or header value is available) and returns `true` to allow or `{ allowed: false, message?, retryAfterSeconds? }` to deny. Denials surface as an `isError` tool result with `kind: 'quota_exceeded'` plus the author-supplied `message`/`retryAfterSeconds`. Semantics to know: * The hook runs **after** the in-memory buckets admit the call, so rate-limited clients cannot spam a table-backed hook. * **Fail-closed**: a hook that throws — or a configured `resource`/`method` that doesn't resolve — **denies** the call. Cost protection that silently disables itself on a bug is worse than a hard failure. The raw error is written to the server log only; the client sees a sanitized message. * Harper calls the hook once per attempted tool call; counting strategy (increment on check vs on success) is the hook's business. * **Race-safety is the hook's business too.** The hook can run concurrently for the same identity — within a worker (interleaving across `await` boundaries) and across workers. A naive read-then-write counter like the example above can undercount under concurrency and admit calls past the limit; make the read-modify-write atomic (a transaction that serializes conflicting writers, a compare-and-set retry loop, or a store with native atomic increments) for production use. ## `mcp.session.*` Settings that apply to MCP session lifecycle on both profiles. ### `mcp.session.idleTimeoutSeconds` Type: `integer` (minimum 1) Default: `1800` (30 minutes) Idle window after which a session record in `system.mcp_session` is TTL-evicted. The next request bearing the evicted session id receives HTTP 404 and the client is expected to re-`initialize`. ### `mcp.session.allowClientDelete` Type: `boolean` Default: `false` When `true`, Harper accepts client-issued `DELETE /mcp` requests that explicitly terminate a session. When `false` (the default), `DELETE` returns 405 with an `Allow` header — sessions only end via idle eviction or explicit server-side cleanup. ## Security: Origin validation The MCP endpoint validates the request `Origin` header to defend against DNS-rebinding attacks (a requirement of the MCP Streamable HTTP transport). Validation reuses each profile's existing CORS configuration rather than introducing a separate MCP setting: * When CORS is **disabled** — or enabled with a `*` (wildcard) allow-list — any `Origin` is accepted. This is appropriate for localhost-only or non-browser clients, where no DNS-rebinding vector exists. * When CORS is **enabled** with an explicit allow-list, a request whose `Origin` is not in the list is rejected with `403 Forbidden`. * A request with no `Origin` header at all (for example `curl` or server-to-server traffic) is always accepted — DNS rebinding only applies to browser-initiated requests. **Secure default:** any deployment that exposes the MCP endpoint to browsers beyond loopback should enable CORS with an explicit (non-`*`) allow-list — that is what activates Origin-based DNS-rebinding protection. The two profiles ship with **different CORS defaults**, but both accept any `Origin` out of the box: * Application profile (HTTP port): `http.cors` + `http.corsAccessList`. Default: CORS **disabled**, so any `Origin` is accepted. * Operations profile (operations port): `operationsApi.network.cors` + `operationsApi.network.corsAccessList`. Default: CORS **enabled with a `*` allow-list**, so any `Origin` is still accepted until you replace `*` with explicit origins. ## Example A common deployment pattern that locks down the operations profile to a small explicit set, enables MCP DELETE for graceful client logout, and raises per-tool throughput for the application profile: ``` mcp: operations: allow: - describe_all - describe_database - system_information - get_job rateLimit: perToolPerSecond: 5 perToolBurst: 10 application: searchMaxResults: 500 rateLimit: perToolPerSecond: 50 perToolBurst: 100 session: idleTimeoutSeconds: 3600 allowClientDelete: true ``` --- # Migrating from the External MCP Server *Added in: v5.1.0* Earlier deployments of Harper used the standalone [`HarperFast/mcp-server`](https://github.com/HarperFast/mcp-server) addon — a separate Node.js process that wrapped Harper's Operations API and exposed MCP over stdio. With v5.1, MCP is now a first-class **built-in** server-side surface; the external addon is deprecated and will be archived alongside this release. This page covers what changes for you and how to migrate. ## What changed | Concern | External `mcp-server` addon | Built-in MCP (v5.1+) | | ---------------------------- | ------------------------------------------------ | ----------------------------------------------------------------------------------------------- | | Deployment | Separate Node.js process spawned by the LLM host | In-process inside Harper; nothing else to install or run | | Transport | stdio only | Streamable HTTP (POST + GET-SSE) **plus** stdio via `harper mcp` | | Authentication | Operations API JWT, passed via env var | Harper's native Basic / JWT / mTLS / UDS-via-filesystem-perms | | Tool surface | Hand-curated wrapper around the Operations API | Operations profile + Application profile (Resources walker) | | Resource exposure | None | `harper://about`, `harper://operations`, `harper://schema/*`, `harper://openapi`, `https://...` | | `notifications/list_changed` | Not supported | Supported per-session on both profiles | | Rate limiting | Not present | Per-session, per-tool token-bucket on both profiles | | Audit logging | Operations API audit log only | Dedicated `mcp.audit` category with credential redaction | | Per-attribute permissions | Not honored in the tool surface | Narrowed at schema-derivation time | | Config | Env vars + addon's own JSON | Top-level `mcp:` block in `harperdb-config.yaml` | ## Migration checklist ### 1. Enable the built-in MCP surface Add an `mcp:` block to `harperdb-config.yaml`. The minimal "turn it on" form is: ``` mcp: operations: {} application: {} ``` If you only need the operation-wrapper functionality the external addon provided (which is roughly what `mcp.operations` does), the operations block alone is enough: ``` mcp: operations: {} ``` See [MCP Configuration](/reference/v5/mcp/configuration.md) for the full set of knobs. ### 2. Switch your MCP host to `harper mcp` The `harper mcp` CLI ships with Harper itself — see [Harper MCP CLI](/reference/v5/mcp/cli.md). For Claude Desktop, the new config block is: ``` { "mcpServers": { "harper": { "command": "harper", "args": ["mcp"] } } } ``` `harper mcp print-config --client claude-desktop|cursor|zed` emits paste-ready blocks for the three supported hosts. By default the CLI connects to the local Harper via the operations API UDS — no credentials, gated by filesystem permissions on the socket. For a remote Harper, add `--target https://node.example.com:9926` and either supply credentials with `--bearer` / `--username` + `--password`, or run `harper login https://node.example.com:9926` once and let `harper mcp` pick up the saved JWT automatically. ### 3. Audit the operations exposed to your LLM The built-in default-allow list is intentionally narrower than the external addon's wrapper. If your agents rely on operations outside `describe_*` / `list_*` / `search_*` / `system_information` / `read_log` / `read_audit_log` / `get_job` / `get_status` / `get_analytics` / `get_metrics`, add them explicitly to `mcp.operations.allow`. See the [Default-allow list](/reference/v5/mcp/tools-and-resources.md#default-allow-list) and the rationale for excluding `get_*` as a glob (it would pull in `get_configuration`, `get_components`, etc., which can return secrets). ### 4. Decommission the addon Once your MCP hosts are pointing at `harper mcp` and tools are dispatching correctly: * Stop the separate `mcp-server` process. * Remove the addon's config block from your MCP host's `mcpServers` (replaced in step 2). * Remove the `mcp-server` package from any deployment scripts. ## Differences to watch for * **Operation names are unchanged.** A tool that the external addon called `search_by_value` is still `search_by_value` in the built-in MCP server. * **Result envelopes differ slightly.** The built-in MCP server wraps operation results in MCP's `result.content[]` (with a `type: 'text'` JSON-encoded entry) per spec. Hosts that consume the raw operation JSON directly will need to parse it out of `content[0].text`. * **Permission filtering is now enforced at `tools/list` time** — users see only operations their role can invoke. The external addon listed everything and let the dispatch fail. * **Sessions are real now.** Each MCP host opens an `Mcp-Session-Id` session, can hold an SSE channel open for notifications, and is cleaned up on idle eviction. The external addon was stateless per-request. ## After migration * Validate with `harper mcp doctor --target ` from any machine. The output gives an OK/FAIL line per handshake step. * If your agents need richer per-resource invocation (`get_`, `search_`, etc., generated from your application's exported `Resource` classes), enable `mcp.application` as well — that surface has no equivalent in the external addon. --- # MCP Overview *Added in: v5.1.0* Harper implements the [Model Context Protocol](https://modelcontextprotocol.io) (MCP) as a first-class server-side surface, letting large-language-model hosts (Claude Desktop, Cursor, Zed, custom agents) discover and invoke Harper operations, resources, and tables through a standard wire protocol. The MCP server runs in-process inside Harper — there is no separate addon, no sidecar, and no out-of-process broker. ## What MCP gives you * **Tool discovery and invocation.** LLM hosts get a typed list of operations they can call (`tools/list`) and a uniform JSON-RPC invocation envelope (`tools/call`). Per-tool input schemas come from Harper's operation catalog (operations profile) or from your `Table.attributes` and exported `Resource` classes (application profile). * **Resource exposure.** Synthetic `harper://` URIs surface metadata (server info, OpenAPI document, table schemas, operations catalog), and `https://` URIs mirror your application's REST surface so hosts can resolve real REST endpoints in-process. * **Server-push notifications.** `notifications/tools/list_changed` and `notifications/resources/list_changed` fire over an open Server-Sent Events channel when role mutations or schema changes alter what a session can see. * **Resource subscriptions.** *Added in: v5.1.10* Subscribe to a row-backed application resource with `resources/subscribe` and receive a `notifications/resources/updated` push whenever that record (or table) changes, driven by Harper's audit-log change feed. See [Resource Subscriptions](/reference/v5/mcp/subscriptions.md). * **Resumable notification streams.** *Added in: v5.1.10* A dropped GET-SSE connection can reconnect with a `Last-Event-ID` header and replay the notifications it missed from a bounded per-session buffer, so a brief network blip doesn't lose `list_changed` / `resources/updated` events. See [Resource Subscriptions](/reference/v5/mcp/subscriptions.md#resuming-a-dropped-stream). * **Per-session bookkeeping.** Sessions persist for the configured idle window; `Mcp-Session-Id` ties JSON-RPC requests, GET-SSE notifications, and the optional DELETE-session cleanup together. * **Built-in auth and RBAC.** Harper's existing Basic, JWT, and mTLS authentication paths run unchanged on the MCP endpoint. Tool and resource visibility is filtered through your role's `permission` block (`super_user`, `structure_user`, per-operation, per-table, per-attribute). * **Audit + rate limits.** Every `tools/call` writes to Harper's audit log (with credential redaction); per-session and per-tool token-bucket rate limits prevent a runaway agent from overwhelming a Harper worker. ## Protocol versions supported | Version | Status | Notes | | ------------ | ---------- | -------------------------------------------------------------------------------------- | | `2025-06-18` | Preferred | The version Harper reports in the `initialize` response. | | `2025-03-26` | Backcompat | Accepted for clients that pin to the earlier rev. | | Other | Negotiated | Per spec, Harper responds with the preferred version; clients downgrade or disconnect. | The negotiation behavior follows the MCP spec's "server MUST respond with a value it does support" rule, so newer SDKs (which may default to a later protocol version Harper does not yet implement) connect cleanly by accepting the downgrade. ## The two profiles Harper exposes MCP through one or two profiles, each mounted on its own endpoint and gated by its own config block. A profile is enabled if its sub-block exists in `mcp:` config — there is no separate `enabled` flag. ### Operations profile (`mcp.operations`) Wraps Harper's operation catalog (the same set of operations the REST `/operation` endpoint and the legacy `OPERATIONS_API` accept). Mounts on the **operations server** (default port `9925`). * Default-allowed surface (read-only): `describe_*`, `list_*`, `search_*`, plus an explicit safe-getter list (`get_job`, `get_status`, `get_analytics`, `get_metrics`), `system_information`, `read_log`, `read_audit_log`. * Operators opt destructive or sensitive operations in via `mcp.operations.allow`. Destructive operations carry `destructiveHint: true` so well-behaved MCP clients can prompt before invoking. * Tool dispatch goes through the same `chooseOperation` + `processLocalTransaction` path as REST `/operation` — `verifyPerms` runs unchanged. ### Application profile (`mcp.application`) Walks your application's exported `Resource` classes and generates one MCP tool per implemented REST verb. Mounts on the **application HTTP server** (the same listener that serves your REST endpoints). * For each exported Resource, Harper emits `get_`, `search_`, `create_`, `update_`, and `delete_` tools when the corresponding verb is implemented on the prototype. * Input schemas are derived from `Table.attributes` and narrowed by your role's `attribute_permissions`. * Components can opt non-verb instance methods into the MCP surface by declaring a static `mcpTools` array on the Resource class. * A Resource is excluded from the MCP surface when its registration sets `exportTypes.mcp = false`. See [MCP Tools and Resources](/reference/v5/mcp/tools-and-resources.md) for the full generation rules and visibility model. ## What's next * **Configuration** — see [MCP Configuration](/reference/v5/mcp/configuration.md) for the full set of config knobs. * **CLI** — see [Harper MCP CLI](/reference/v5/mcp/cli.md) for the `harper mcp` subcommand that bridges stdio MCP hosts (Claude Desktop, Cursor, Zed) to a running Harper instance. * **Migration** — if you are coming from the `HarperFast/mcp-server` external addon, see [MCP Migration](/reference/v5/mcp/migration.md). * **Tool metadata** — see [Tool Metadata](/reference/v5/mcp/tool-metadata.md) for what fields appear on each generated tool descriptor and where the data comes from. ## Out of scope for v1 The following items are explicitly deferred to a follow-on release: * OAuth 2.1 PRM (Protected Resource Metadata) authorization. * Cross-worker session sharing (each MCP session is bound to the worker that accepted the GET stream). * TypeScript type reflection into JSON Schema for custom `mcpTools` entries (schemas are hand-authored). * Global REST/operations rate limiting — only per-session/per-tool limits apply on the MCP surface. --- # MCP Resource Subscriptions *Added in: v5.1.10* Beyond the `notifications/*/list_changed` events that tell a client *what tools and resources exist*, Harper can push a notification whenever the *contents* of a specific application resource change. A client subscribes to a resource URI with `resources/subscribe`; Harper watches that record (or table) through its audit-log change feed and emits `notifications/resources/updated` on every commit. This page covers `resources/subscribe` / `resources/unsubscribe`, the `notifications/resources/updated` push, which URIs are subscribable, and `Last-Event-ID` stream resumability. The `list_changed` events themselves are documented in [MCP Tools and Resources](/reference/v5/mcp/tools-and-resources.md#notificationslist_changed). Harper advertises the capability on `initialize`: ``` { "capabilities": { "tools": { "listChanged": true }, "resources": { "listChanged": true, "subscribe": true }, "prompts": { "listChanged": true }, "completions": {}, "logging": {} } } ``` ## Prerequisites A subscription delivers its updates over the session's GET-SSE channel, so the client **must open `GET /mcp` before calling `resources/subscribe`**. Subscribing without a live stream returns `-32602`: ``` { "error": { "code": -32602, "message": "open the GET SSE stream before subscribing to resources" } } ``` The open stream is also the subscription's lifecycle boundary: when it closes (by any of the paths in [Unsubscribing](#unsubscribing)), the subscription's underlying change iterator is released. This is why Harper rejects a stream-less subscribe rather than letting the iterator leak with nowhere to deliver. ## Subscribing Send `resources/subscribe` with the resource URI: ``` { "jsonrpc": "2.0", "id": 7, "method": "resources/subscribe", "params": { "uri": "https://node.example.com:9926/Product/42" } } ``` A successful subscribe returns an empty result: ``` { "jsonrpc": "2.0", "id": 7, "result": {} } ``` * `params.uri` is required and must be a string — otherwise `-32602` (`resources/subscribe requires params.uri`). * Re-subscribing to the same URI is idempotent: the prior subscription is stopped and replaced. * The URI is persisted on the durable `mcp_session` record so it can be restored after an SSE reconnect (see [Resuming a dropped stream](#resuming-a-dropped-stream)). ### Which URIs are subscribable Only **row-backed application resources** can be subscribed to — the `https://` / `http://` URIs that mirror an exported `Resource` (the same ones `resources/list` advertises on the application profile). The URI's path is matched against the `Resources` registry, and the resource is subscribable only if its class implements `subscribe`. Anything else returns `-32602`: ``` { "error": { "code": -32602, "message": "resource is not subscribable: harper://schema/data/product" } } ``` Synthetic `harper://*` metadata URIs (`harper://about`, `harper://openapi`, `harper://schema/...`, `harper://operations`) have no change source and are **not** subscribable. They participate in `notifications/resources/list_changed` only. ### Record vs. collection URIs The granularity of a subscription is determined by the path: | URI form | Watches | | ------------------------------ | ------------------------------------------------------------ | | `https://host:port/Product/42` | The single record with primary key `42`. | | `https://host:port/Product` | The whole `Product` table (any record insert/update/delete). | A collection URI (no record key — the form `resources/list` advertises) subscribes to the entire table; a record URI narrows the watch to one primary key. Subscriptions use `omitCurrent` semantics: Harper does **not** replay the current value at subscribe time. You only receive notifications for changes that commit *after* the subscription is established. The notification carries no snapshot — re-read the resource (or call the corresponding `get_*` tool) to fetch the new state. ## The `notifications/resources/updated` push On each committed change to a subscribed resource, Harper pushes a notification onto the session's GET-SSE channel: ``` { "jsonrpc": "2.0", "method": "notifications/resources/updated", "params": { "uri": "https://node.example.com:9926/Product/42" } } ``` There is no diff payload — `params.uri` is the only field. The client decides whether and how to re-read. ## Unsubscribing Send `resources/unsubscribe` with the same URI: ``` { "jsonrpc": "2.0", "id": 8, "method": "resources/unsubscribe", "params": { "uri": "https://node.example.com:9926/Product/42" } } ``` * `params.uri` is required and must be a string — otherwise `-32602` (`resources/unsubscribe requires params.uri`). * Unsubscribing a URI that was never subscribed is a no-op and still returns `{}`. Subscriptions are also torn down automatically when the session's GET-SSE stream closes — via explicit `DELETE /mcp`, idle-timeout eviction from `system.mcp_session`, a dropped TCP connection, or an idle-prune sweep. ## Scope and durability Subscriptions are **per-worker**: a subscription is bound to the worker that holds the session's SSE stream, and the audit-log change feed delivers commits locally on that worker. The live subscription objects can't be persisted, but the durable list of subscribed URIs lives on the `mcp_session` record, which is how they survive a reconnect. ## Resuming a dropped stream Every frame Harper sends on the GET-SSE channel carries a monotonic SSE event id (the `id:` field). Harper keeps a bounded per-session **replay buffer** of the most recent 100 frames. When a dropped connection reconnects, the client sends the id of the last frame it saw in the standard `Last-Event-ID` header: ``` GET /mcp HTTP/1.1 Mcp-Session-Id: 0c8f... Last-Event-ID: 137 ``` Harper replays every buffered frame with a higher id (exclusive) before live frames resume, so a brief network blip doesn't drop `notifications/resources/updated` or `list_changed` events. On reconnect, Harper also restores the session's persisted subscriptions from the `mcp_session` record (best-effort — a URI that is no longer subscribable is dropped). Caveats, all best-effort by design: * **Buffer bound.** Only the last 100 frames are retained; a client that was disconnected long enough to miss more than 100 frames will not get the older ones. A non-numeric `Last-Event-ID` replays the entire buffer. * **Per-worker.** The replay buffer and event-id sequence live on the worker that served the original stream. If the reconnecting `GET` lands on a different worker (one that never held the prior stream), its buffer is empty and there is nothing to replay. This is the same per-worker binding that scopes sessions and subscriptions. ## Not yet supported See [the v1 out-of-scope list](/reference/v5/mcp/overview.md#out-of-scope-for-v1). Notably, cross-worker session sharing is deferred, so subscription delivery and `Last-Event-ID` replay are both best-effort across a reconnect that changes workers. --- # MCP tool payload sourcing Harper's MCP server publishes tools via the Model Context Protocol (rev 2025-06-18, Streamable HTTP transport). This page is a reference for what fields appear on each generated tool descriptor and where the data comes from. For authoring guidance, see the [Writing quality MCP and OpenAPI descriptions](/learn/developers/mcp-and-openapi-metadata.md) how-to. ## Tool descriptor fields Per the MCP spec, every tool descriptor returned from `tools/list` carries: | Field | Required | Purpose | | -------------- | -------- | ------------------------------------------------------------------------------------------------ | | `name` | yes | Machine identifier used by `tools/call` | | `description` | yes | LLM-facing prose explaining what the tool does and when to pick it over siblings | | `inputSchema` | yes | JSON Schema for the arguments the tool accepts | | `outputSchema` | no | JSON Schema for the data the tool returns (added in spec rev 2025-06-18) | | `annotations` | no | Hints for clients: `readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`, `title` | Harper populates each from a different source depending on the tool's profile and origin. ## Profiles Harper exposes two MCP profiles, each with its own port and registration loop: * **Operations profile** — registers one MCP tool per Harper operation that survives the `mcp.operations.allow` / `deny` filter. Walks `OPERATION_FUNCTION_MAP`. The default allow list is read-only and intentionally narrow: `describe_*`, `list_*`, `search_*`, plus an explicit list of safe getters. * **Application profile** — registers verb tools (`get_*`, `search_*`, `create_*`, `update_*`, `patch_*`, `delete_*`) for every exported Resource, plus any custom tools declared via `static mcpTools`. ## Sourcing per profile ### Operations profile For operations tools, the descriptor fields are sourced from: | Field | Source | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Operation name (from `OPERATION_FUNCTION_MAP` key) | | `description` | Hand-authored entry in the operations descriptions catalog; falls back to a generic template when an operator opts in a non-cataloged operation | | `inputSchema` | Hand-curated JSON Schema in the operations input-schemas catalog; falls back to `{ type: 'object', additionalProperties: true }` | | `annotations.readOnlyHint` | `true` if the operation matches a read-only prefix (`describe_`, `list_`, `search_`, `get_`, `read_`) or is the literal `system_information` | | `annotations.destructiveHint` | `true` for operations in the curated destructive set (`drop_*`, `delete_*`, `restart`, `set_configuration`, etc.) | | `annotations.idempotentHint` | Default empty; opt-in per operation after end-to-end verification that the second call produces the same observable outcome | Operations registered outside core (for example, `cluster_status` from harper-pro) don't have catalog entries; they fall back to the generic description template until the per-operation metadata registry lands. ### Application profile — verb tools For verb tools generated from exported Resources: | Field | Source | | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | `${verb}_${sanitized-path}` (e.g. `get_Product`, `search_Customer`) | | `description` | Composed: `[ResourceClass.description \n\n] ${verb sentence} ${runtime RBAC note}` | | `inputSchema` | Derived per verb from `ResourceClass.attributes` and the caller's `attribute_permissions`. Per-attribute `description` propagates to `inputSchema.properties[*].description` | | `outputSchema` | Derived per verb from `ResourceClass.attributes` for `get_*` / `create_*` / `update_*` / `patch_*`. `delete_*` returns `{ deleted: true, }`. `search_*` deliberately omits `outputSchema` | | `annotations.readOnlyHint` | `true` on `get_*` and `search_*` | | `annotations.destructiveHint` | `true` on `delete_*` | | `annotations.idempotentHint` | `true` on `update_*` (PUT semantics); other verbs default off | `static description` and `static properties` on the Resource class override the auto-derived values. `static outputSchemas[verb]` overrides per-verb output schemas. `static mcp.annotations[verb]` overrides annotations per verb. `static hidden === true` suppresses the entire Resource from MCP listing. ### Application profile — custom `mcpTools` For tools declared via `static mcpTools`: | Field | Source | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | `def.name` from the `mcpTools` entry | | `description` | `def.description` from the entry; falls back to a generic template if omitted (with a warn-once log at registration) | | `inputSchema` | `def.inputSchema` from the entry; falls back to `{ type: 'object', additionalProperties: true }` if omitted (with a warn-once log at registration) | | `outputSchema` | `def.outputSchema` from the entry (optional; no fallback) | | `annotations` | `def.annotations` from the entry (optional pass-through) | Custom tools have no per-user listing filter beyond authentication — the Resource's instance method is responsible for whatever RBAC it needs to enforce. ## Sample descriptors ### `get_Product` (with docstrings) Given a `Product` table with type-level and field-level docstrings: ``` { "name": "get_Product", "description": "Product catalog row — what shows up in the storefront listing, search, and inventory feeds. One row per SKU.\n\nFetches a single Product record by sku. Runtime RBAC (allowGet) enforces per-record access at call time.", "inputSchema": { "type": "object", "properties": { "id": { "type": "string", "description": "Primary key (sku)." }, "get_attributes": { "type": "array", "items": { "type": "string" } } }, "required": ["id"] }, "outputSchema": { "type": "object", "properties": { "sku": { "type": "string", "description": "Stock keeping unit — globally unique across catalogs." }, "name": { "type": "string", "description": "Display name shown in the storefront." }, "priceCents": { "type": "integer", "description": "Retail price in cents (USD)." } }, "required": ["sku", "name", "priceCents"], "additionalProperties": false }, "annotations": { "readOnlyHint": true } } ``` ### `add_user` (operations profile) ``` { "name": "add_user", "description": "Creates a new Harper user with username, password, and role. Requires super_user. Username is immutable after creation — use drop_user + add_user to rename.", "inputSchema": { "type": "object", "properties": { "username": { "type": "string" }, "password": { "type": "string" }, "role": { "type": "string", "description": "Role name. Use list_roles to discover available roles." }, "active": { "type": "boolean", "default": true } }, "required": ["username", "password", "role"] } } ``` Note that `add_user` does NOT carry `idempotentHint: true`. Under MCP semantics this would claim the second call produces the same observable outcome — but `add_user("bob")` on first call returns the created user and on second call returns an "already exists" error. Different observable outcomes → not idempotent. ## `harper://*` resources Harper also publishes a small set of synthetic resources via the MCP `resources/list` endpoint: | URI | Profile | Description | | ------------------------------ | ----------- | ----------------------------------------------------- | | `harper://about` | both | Server version, profile, MCP protocol versions | | `harper://operations` | operations | User-filtered operations catalog | | `harper://openapi` | application | Full OpenAPI 3.0.3 document | | `harper://schema/{db}/{table}` | application | Per-table schema, filtered by `attribute_permissions` | | `https://{host}/{path}` | application | Application HTTP Resources, in-process | For `harper://schema/{db}/{table}` and `https://{host}/{path}` entries, the descriptor description prepends `Table.description` / `ResourceClass.description` when present. ## See also * [Writing quality MCP and OpenAPI descriptions](/learn/developers/mcp-and-openapi-metadata.md) — authoring how-to * [Schema reference: docstrings and `@hidden`](/reference/v5/database/schema.md) — GraphQL surface * [Resource API reference: class-level metadata](/reference/v5/resources/resource-api.md#class-level-metadata-for-mcp-and-openapi) — programmatic surface --- # MCP Tools and Resources *Added in: v5.1.0* This page documents what the MCP server actually exposes — which tools land on `tools/list` for which user, how their input schemas are built, and what shows up in `resources/list` for each profile. Configuration knobs that gate this surface are documented in [MCP Configuration](/reference/v5/mcp/configuration.md). ## Operations profile — tool generation Tools are generated by walking Harper's `OPERATION_FUNCTION_MAP` and filtering through the configured allow/deny lists. Each tool is named for its operation (`describe_all`, `search_by_value`, `system_information`, …) and dispatches through the same `chooseOperation` + `processLocalTransaction` path the REST `/operation` endpoint uses, so existing `verifyPerms` enforcement runs unchanged. ### Default-allow list The default `mcp.operations.allow` list is intentionally narrow and read-only: * `describe_*` — schema / database / table descriptions. * `list_*` — enumerations (users, roles, databases). * `search_*` — search operations. * `get_job`, `get_status`, `get_analytics`, `get_metrics` — explicit safe getters. * `system_information` — server-level information. * `read_log`, `read_audit_log` — log readers. `get_*` is deliberately **not** a wildcard. That glob would otherwise pull in: * `get_configuration` — returns TLS, S3, and authentication secrets. * `get_components`, `get_component_file`, `get_custom_function`, `get_custom_functions` — return component source code, which can embed secrets. * `get_backup` — backup metadata / payload. * `get_deployment`, `get_deployment_payload` — deployment artifacts. These are all gated by `verifyPerms`, but defaulting to "expose them to the LLM if a super\_user invokes them" is the wrong posture for MCP — the LLM provider sees and may log every input/output. Operators who want any of them on the surface opt them in via `mcp.operations.allow`. ### Tool annotations Each generated tool carries MCP annotations the client can use to decide how to surface it: * `readOnlyHint: true` — operations matching the read-only set (`describe_*`, `list_*`, `search_*`, `get_*`, `read_*`, `system_information`, `status`). MCP hosts can render these as "safe to call without confirmation". * `destructiveHint: true` — operations Harper knows are destructive (`drop_*`, `delete*`, `restart*`, `set_configuration`, `remove_node`). MCP hosts SHOULD prompt before invoking. Neither hint is an authorization check — `verifyPerms` runs at dispatch. ### Per-user filtering `tools/list` is filtered through `canRoleInvokeOperation` so each session sees only the operations its user can actually call: * `super_user` sees everything in the allow list. * A user with `structure_user: true` sees schema-structure operations (`create_schema`, `drop_table`, `create_attribute`, etc.) in addition to anything in `permission.operations`. * Other users see only operations listed in `permission.operations`. The list is cached per session and recomputed when a `notifications/tools/list_changed` event would fire. ## Application profile — tool generation The application profile walks Harper's `Resources` registry. For each exported `Resource` whose registration does not set `exportTypes.mcp = false`, Harper emits one MCP tool per implemented REST verb: | Verb on Resource prototype | Tool name | Schema source | | -------------------------- | ------------------------ | ------------------------------------------------------------- | | `get(target, request)` | `get_` | Primary key + optional `get_attributes` | | `search(target, request)` | `search_` | `conditions`, `operator`, `get_attributes`, `limit`, `cursor` | | `post(target, data)` | `create_` | All writable attributes; non-nullable non-PK fields required | | `put(target, data)` | `update_` (`put`) | PK + writable attributes | | `patch(target, data)` | `patch_` (`patch`) | PK + writable attributes | | `delete(target, request)` | `delete_` | Primary key | A Resource that implements both `put` and `patch` emits `update_` (favoring `put`). ### Tool-name sanitization The Resource's path is sanitized into a valid tool name: `/` and `.` become `_`. If two Resources sanitize to the same name, Harper disambiguates by prefixing the database name; if a collision still occurs, a 6-character hash suffix is appended. ### Input schema derivation Input schemas come from `Table.attributes`: * Harper types map to JSON Schema primitive types (`Int`/`Long`/`BigInt` → `integer`, `Float` → `number`, `String`/`ID` → `string`, `Boolean` → `boolean`, `Date` → `[string, number]`, `Bytes`/`Blob` → `string` with `contentEncoding: base64`). * Nested `Object` and `Array` attributes recurse into their `properties` / `elements`. * `nullable: true` adds `"null"` to the type union. * Auto-managed columns (`assignCreatedTime`, `assignUpdatedTime`, `expiresAt`) and computed columns are stripped from write schemas (`create_*`, `update_*`) — the server fills them in. * Per-attribute `attribute_permissions` narrow the schema **per requesting user**: attributes the user cannot read are stripped from `get_*` / `search_*` schemas; attributes the user cannot insert/update are stripped from `create_*` / `update_*` schemas. The schema narrowing is a UX optimization, not a security boundary — runtime `Table.allowUpdate` / `Table.allowCreate` still enforces. The narrowing just avoids burning LLM tokens on fields the user couldn't write anyway. ### Custom `mcpTools` opt-in A component author can expose non-verb instance methods as MCP tools by declaring a static `mcpTools` array on the Resource class: ``` class Orders extends Tables.orders { static mcpTools = [ { name: 'reconcile_unsettled', method: 'reconcileUnsettled', description: 'Reconcile all orders flagged as unsettled and emit a summary', inputSchema: { type: 'object', properties: { since: { type: 'string', description: 'ISO 8601 timestamp' } }, }, }, ]; async reconcileUnsettled({ since }) { /* ... */ } } ``` The corresponding instance method runs through Harper's normal `transactional()` envelope, so per-record `allow*` predicates and audit logging behave the same way as regular verb dispatch. **Custom tools are exposed to any MCP session — including anonymous, unauthenticated ones.** Unlike the auto-generated verb tools (which are RBAC-filtered per user at `tools/list` time and enforce table permissions on call), the MCP layer performs no authentication or ACL check for a custom tool: it is listed to every session and its method executes even when no user is logged in (`context.user` may be empty). Access control is entirely the method's responsibility — to restrict a tool to authenticated users or specific roles, check `context.user` (or rely on the per-record `allow*` predicates its data access triggers) inside the method and throw when the caller doesn't qualify. ### `exportTypes` gating The MCP surface mirrors the public REST surface. A Resource is filtered out of MCP enumeration entirely when its registration sets `exportTypes.mcp = false`: ``` server.http(Resource, { name: 'internal-thing', exportTypes: { mcp: false } }); ``` This is independent of the `http` exportType — the only switch that operators set to scope MCP visibility is `mcp`. ## Resources surface Both profiles serve `resources/list`, `resources/read`, and `resources/templates/list`. Row-backed application resources can additionally be watched with `resources/subscribe` — see [MCP Resource Subscriptions](/reference/v5/mcp/subscriptions.md). ### `harper://` URIs | URI | Profile | Content | | ------------------------------------ | ----------- | -------------------------------------------------------------- | | `harper://about` | both | Server version, profile name, protocol versions, capabilities. | | `harper://operations` | operations | User-filtered list of allowed operation names. | | `harper://openapi` | application | The OpenAPI 3.0.3 document for the application's REST surface. | | `harper://schema/{database}/{table}` | application | Per-table attribute definitions, RBAC-filtered at read time. | The schema URIs honor each user's `permission[db].tables[table]` walk — a user with no `read` or `describe` perm on a table gets a "permission denied" response from `resources/read`. ### `https://` URIs The application profile additionally exposes every exported `Resource` (that passes the `exportTypes.mcp` gate **and** the `hasRestVerbs` check) as an `https://:/` URI. These resolve in-process via `Resources.getMatch(path, 'mcp')` — there is no outbound HTTP request. The body returned by `resources/read` is a small descriptor: ``` { "uri": "https://node.example.com:9926/Product", "path": "Product", "database": "data", "table": "product", "hint": "Use the corresponding `get_*` or `search_*` tool from `tools/list` to fetch records." } ``` Per-record reads go through the tools surface, where each Resource's `allow{Read,…}` predicates run. The `resources/read` descriptor itself is a fast, side-effect-free hint — not a capability. ## `notifications/*/list_changed` After the `initialize` handshake, an MCP client opens `GET /mcp` to keep an SSE channel open for server-push frames. Harper subscribes to its existing role-cache and schema-reload event channels and, whenever one fires: 1. Walks the per-worker session registry. 2. For each session on that profile, re-resolves the bound user (so any role/permission mutations occurring between the handshake and the event are evaluated against current permissions, rather than a frozen snapshot). 3. Recomputes the session's `tools/list` and `resources/list` against the fresh user. 4. Compares to the snapshot taken at session start (or after the last fire). 5. Emits `notifications/tools/list_changed` and / or `notifications/resources/list_changed` if and only if the visible set actually changed. Sessions whose visible surface is unchanged see nothing — there is no broadcast. The notification carries no diff payload; clients call `tools/list` and `resources/list` again to fetch the new state. The GET-SSE channel itself closes on: * Explicit `DELETE /mcp` (when `mcp.session.allowClientDelete` is `true`). * The session being TTL-evicted from `system.mcp_session` after `mcp.session.idleTimeoutSeconds`. * The client dropping the underlying TCP connection (Harper's HTTP server propagates that to the iterator's `return()`, which the registry's on-close listener catches). * An idle-prune sweep (belt-and-braces against the cases above missing — see [Configuration / `mcp.session.idleTimeoutSeconds`](/reference/v5/mcp/configuration.md#mcpsessionidletimeoutseconds)). --- # Analytics *Added in: v5.1.0* Every model call is recorded for observability and usage accounting, at two levels of granularity: a per-call log table for forensics, and aggregate counters in Harper's [general analytics](/reference/v5/analytics/overview.md) for dashboards and trends. ## Per-call log: `hdb_model_calls` Each `embed()`, `generate()`, and `generateStream()` call writes one row to the `hdb_model_calls` system table — on success and on failure. With `toolMode: 'auto'`, each backend round inside the loop records its own row (the outer loop itself does not add one). | Field | Description | | ------------------- | --------------------------------------------------------------------------------------------------------------- | | `tenant` | Tenant identifier, when the call carried one | | `app` | Resource path of the calling resource, when called from one | | `model` | Logical model name the caller used | | `backend` | Backend that served the call (`ollama`, `openai`, …); `unknown` for pre-dispatch failures | | `method` | `embed`, `generate`, or `generateStream` | | `prompt_tokens` | Prompt token count, when the backend reported usage | | `completion_tokens` | Completion token count, when the backend reported usage | | `embedding_tokens` | Embedding token count, when the backend reported usage | | `latency_ms` | Wall-clock call duration | | `success` | Whether the call completed | | `error_code` | On failure: `backend_error`, `aborted`, `capability_unsupported`, `backend_not_found`, or `pending_unsupported` | Rows are buffered in memory and flushed every 10 seconds, or immediately once 1,000 rows accumulate; rows older than 90 days are purged. Buffered rows may be lost on abrupt shutdown — treat the table as operational telemetry, not an audit log. Query it like any table, for example through the operations API: ``` { "operation": "search_by_conditions", "database": "system", "table": "hdb_model_calls", "conditions": [{ "search_attribute": "success", "search_type": "equals", "search_value": false }] } ``` ## Aggregate metrics Each call also increments Harper's aggregate analytics (visible in `hdb_raw_analytics` alongside the other [analytics metrics](/reference/v5/analytics/overview.md)): * `model-embed`, `model-generate`, `model-generateStream` — call counts * `model-embed-tokens`, `model-generate-tokens`, `model-generateStream-tokens` — token totals Metrics are broken down by backend name, so usage can be charted per provider. --- # API *Added in: v5.1.0* The `models` object exposes three methods. All of them accept an optional `model` option naming the configured logical model to use; when omitted, the logical name `default` is used. Calling a logical name with no configured backend, or asking a backend for a capability it does not support (for example, embeddings from a generation-only backend), throws an error — capability checks run up front, before any request is made. ## embed() ``` models.embed(input: string | string[], options?: EmbedOpts): Promise ``` Converts one or more strings into embedding vectors. The result is always an array of `Float32Array`, one per input string, in input order — including when a single string is passed. ``` import { models } from 'harper'; const [single] = await models.embed('What is Harper?', { inputType: 'query' }); const batch = await models.embed(['first document', 'second document']); ``` | Option | Type | Default | Description | | ----------- | ------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `model` | `string` | `'default'` | Logical name of a configured embedding model | | `requires` | `Capability[]` | — | Capabilities the chosen backend must satisfy; used by [routing](/reference/v5/models/routing.md#capability-routing) to select a candidate in the group | | `inputType` | `'document'` \| `'query'` | — | Hint for models that distinguish document embeddings from query embeddings (e.g. `nomic-embed-text`); ignored by models that do not | | `signal` | `AbortSignal` | — | Cancels the call; composed with the backend's configured `requestTimeoutMs` | ## generate() ``` models.generate(input: GenerateInput, options?: GenerateOpts): Promise ``` Generates a completion. The input may be: * a `string` — shorthand for a single user message, * an array of messages: `{ role: 'system' | 'user' | 'assistant' | 'tool', content: string }`, * an object `{ messages, tools?, system? }` — the form required to declare [tools](/reference/v5/models/tool-calling.md) or pass a system prompt alongside the messages. ``` const result = await models.generate( [ { role: 'system', content: 'You are a terse assistant.' }, { role: 'user', content: 'What is an HNSW index?' }, ], { temperature: 0.2, maxTokens: 300 } ); console.log(result.content); ``` | Option | Type | Default | Description | | ---------------- | -------------------------------------------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | `string` | `'default'` | Logical name of a configured generative model | | `requires` | `Capability[]` | — | Capabilities the backend must satisfy (e.g. `tools`); used by [routing](/reference/v5/models/routing.md#capability-routing). Tools in the input auto-require `tools` | | `temperature` | `number` | backend | Sampling temperature, passed through to the backend | | `maxTokens` | `number` | backend | Completion token limit, passed through to the backend | | `responseFormat` | `'text'` \| `'json'` \| `{ schema: object }` | `'text'` | Structured output. `{ schema }` requests output conforming to a JSON Schema; support varies by backend | | `toolMode` | `'return'` \| `'auto'` | `'return'` | How tool calls are handled — see [Tool Calling](/reference/v5/models/tool-calling.md) | | `signal` | `AbortSignal` | — | Cancels the call; composed with the backend's configured `requestTimeoutMs` | Additional options apply only when `toolMode: 'auto'`; they are documented in [Tool Calling](/reference/v5/models/tool-calling.md). ### GenerateResult | Field | Type | Description | | -------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `content` | `string` | The generated text | | `finishReason` | `'stop'` \| `'length'` \| `'tool_calls'` \| `'content_filter'` | Why generation stopped, normalized across backends | | `toolCalls` | `ToolCall[]` | Tool calls the model requested, when `finishReason` is `'tool_calls'` (each `{ id, name, arguments }`, with `arguments` parsed to an object) | | `usage` | `TokenUsage` | Token usage reported by the backend (`promptTokens`, `completionTokens`, …), when available | | `trace` | `ToolTraceEntry[]` | Per-tool-invocation trace; only populated by the `toolMode: 'auto'` loop — see [Tool Calling](/reference/v5/models/tool-calling.md) | ## generateStream() ``` models.generateStream(input: GenerateInput, options?: GenerateOpts): AsyncIterable ``` Identical to `generate()` but yields the completion incrementally: ``` let text = ''; for await (const chunk of models.generateStream('Write a haiku about databases.')) { if (chunk.deltaContent) text += chunk.deltaContent; } ``` Each chunk may carry: | Field | Type | Description | | ---------------- | ------------------------------- | ---------------------------------------------------------------------------------------------------- | | `deltaContent` | `string` | Text appended since the previous chunk | | `deltaToolCalls` | `Partial[]` | Tool-call deltas; a backend may deliver the same tool call across several chunks with partial fields | | `finishReason` | same values as `GenerateResult` | Set on the final chunk only | Errors detected before the call starts (unknown model name, missing capability) throw synchronously; errors during generation propagate through the iterable. ## registerBackend() *Added in: v5.1.15* ``` models.registerBackend(kind: 'embedding' | 'generative', id: string, backend: ModelBackend): void ``` Registers a custom backend under a logical name, selectable by the `model` option on later calls. This is the programmatic path for in-process or third-party backends; pair it with `models.defineBackend()` to build the backend from a few methods. Both are methods on `models` — reachable as `models.registerBackend(...)` / `scope.models.registerBackend(...)` (and likewise for `defineBackend`), not standalone `harper` exports. See [Custom backends](/reference/v5/models/backends.md#custom-backends) for the full guide. ## Errors and timeouts * An unconfigured logical model name throws a not-found error. The error names the missing logical name only — it does not enumerate configured names. * A capability mismatch (embedding call to a generation-only backend, tool declarations against a backend without tool support) throws before any request is made. * Each backend supports a `requestTimeoutMs` configuration field; when set, it is composed with any caller-provided `signal` so whichever fires first cancels the request. * Backend/network failures throw backend-specific errors with sanitized messages. Every call — successful or failed — is recorded in the [model-call analytics](/reference/v5/models/analytics.md). --- # Backends *Added in: v5.1.0* Four model backends ship with Harper. Each model entry in the [`models` configuration](/reference/v5/models/overview.md#configuration) selects one with its `backend` field. | Backend | Embeddings | Generation | Streaming | Tools | | ----------- | ---------- | ---------- | --------- | --------------- | | `ollama` | ✓ | ✓ | ✓ | — | | `openai` | ✓ | ✓ | ✓ | ✓ | | `anthropic` | — | ✓ | ✓ | ✓ | | `bedrock` | ✓ | ✓ | ✓ | varies by model | All backends support these common fields: | Field | Description | | ------------------ | ---------------------------------------------------------------------------------------------------- | | `backend` | Which backend to use (required) | | `model` | Provider-side model identifier (e.g. `gpt-4o`) used when a call does not pass its own `model` option | | `requestTimeoutMs` | Per-request timeout in milliseconds; composed with any caller-provided `AbortSignal` | ## Ollama Calls a local or remote [Ollama](https://ollama.com) server. No credentials. ``` models: embedding: default: backend: ollama host: localhost:11434 model: nomic-embed-text:latest generative: local: backend: ollama host: ollama.internal:11434 model: mistral:7b ``` | Field | Default | Description | | ------- | ----------------- | --------------------------------------------------------------------------------------------------------------- | | `host` | `localhost:11434` | Ollama server origin. A scheme-less value is treated as `http://`; a full origin (`https://…`) is used as given | | `model` | — | Ollama model name, e.g. `nomic-embed-text:latest`, `mistral:7b` | When embedding with `nomic-embed-text`, the `inputType` option (`'document'` / `'query'`) is applied using the model's task prefixes; other models ignore it. The Ollama backend does not advertise tool support — declaring tools against it fails up front. ## OpenAI Calls the OpenAI API — or any service exposing an OpenAI-compatible API with bearer-token authentication, by pointing `baseUrl` at it. This includes vLLM's OpenAI-compatible server, Google's Gemini OpenAI-compatible endpoint, Azure OpenAI's `/openai/v1` endpoint, and hosted gateways such as OpenRouter or Together AI. ``` models: embedding: default: backend: openai apiKey: ${OPENAI_API_KEY} model: text-embedding-3-large generative: default: backend: openai apiKey: ${OPENAI_API_KEY} model: gpt-4o vllm: backend: openai apiKey: ${VLLM_API_KEY} baseUrl: http://vllm.internal:8000/v1 model: meta-llama/Llama-3.1-8B-Instruct ``` | Field | Default | Description | | -------------- | --------------------------- | ---------------------------------------------------------------------------------- | | `apiKey` | — (required) | API key, sent as a bearer token. Use `${VAR}` indirection | | `baseUrl` | `https://api.openai.com/v1` | API root; point at any OpenAI-compatible endpoint | | `model` | — | Model name, e.g. `gpt-4o`, `text-embedding-3-large` | | `organization` | — | Sent as the `OpenAI-Organization` header, for keys spanning multiple organizations | `responseFormat: 'json'` maps to OpenAI's JSON mode and `responseFormat: { schema }` to strict structured outputs (`json_schema`); OpenAI-compatible servers vary in their support for these. ## Anthropic Calls the Anthropic Messages API. Generation only — Anthropic does not offer an embeddings API. ``` models: generative: claude: backend: anthropic apiKey: ${ANTHROPIC_API_KEY} model: claude-sonnet-4-6 ``` | Field | Default | Description | | --------- | --------------------------- | --------------------------------------- | | `apiKey` | — (required) | API key, sent as the `x-api-key` header | | `baseUrl` | `https://api.anthropic.com` | API root | | `model` | — | Model name, e.g. `claude-sonnet-4-6` | The Anthropic API requires a completion token limit on every request; when a call does not pass `maxTokens`, Harper sends `4096`. ## Amazon Bedrock Calls AWS Bedrock. Credentials come from the standard AWS SDK chain (environment variables, shared credentials file, IAM instance/task roles) — there is no `apiKey` field. The AWS SDK is not bundled with Harper. Install it in your project to use this backend: ``` npm install @aws-sdk/client-bedrock-runtime ``` ``` models: embedding: titan: backend: bedrock region: us-east-1 model: amazon.titan-embed-text-v2:0 generative: claude: backend: bedrock region: us-east-1 model: anthropic.claude-sonnet-4-5-20250929-v1:0 ``` | Field | Default | Description | | -------- | ------------ | ---------------------------------------------------------------------- | | `region` | — (required) | AWS region hosting the Bedrock models | | `model` | — | Bedrock model identifier; the vendor prefix selects the request format | The model identifier's vendor prefix (`anthropic.`, `meta.`, `amazon.titan-`, `cohere.`, `mistral.`) determines the request/response format Harper uses; an unrecognized prefix is rejected with an error. Tool support depends on the underlying model family. Bedrock embedding APIs accept one text per request, so batch `embed()` calls are issued sequentially. ## Custom backends *Added in: v5.1.15* Beyond the four built-ins, a component or application can register its own backend — including an in-process one that runs inference locally instead of calling an HTTP service. A registered backend is selected by its logical name through the same `model` option as a configured backend. Custom backends can be added two ways: **registered programmatically** (below), or **selected in config** by pointing the `backend` field at a module — see [Config-selectable backends](#config-selectable-backends). ### defineBackend() ``` models.defineBackend(spec: DefineBackendSpec): ModelBackend ``` A method on `models` (reachable as `models.defineBackend(...)` / `scope.models.defineBackend(...)`). Builds a `ModelBackend` from the methods it implements. `capabilities()` is derived from which of `embed` / `generate` / `generateStream` are supplied; `tools` and `adapters` cannot be inferred from method presence, so declare them explicitly. | Field | Type | Default | Description | | ---------------- | ---------- | ------- | -------------------------------------------------------------------- | | `name` | `string` | — | Backend name, used in analytics and error messages (required) | | `embed` | `function` | — | `embed(input, opts)` implementation, if the backend embeds | | `generate` | `function` | — | `generate(input, opts)` implementation, if the backend generates | | `generateStream` | `function` | — | `generateStream(input, opts)` implementation, if the backend streams | | `tools` | `boolean` | `false` | Whether `generate` supports tool calls | | `adapters` | `boolean` | `false` | Whether the backend supports per-call adapter selection | `embed` and `generate` return the shape the built-in backends return: `{ status: 'completed', output, usage? }`, where `output` is `Float32Array[]` for `embed` and `{ content, finishReason }` for `generate`. `generateStream` is an async generator yielding incremental `{ deltaContent?, deltaToolCalls?, finishReason? }` chunks — the same [`generateStream()`](/reference/v5/models/api.md#generatestream) shape, not a wrapped result. At least one method must be supplied. A backend that supplies only `generateStream` still satisfies `generate()`: Harper drains the stream into a single result. ### registerBackend() ``` models.registerBackend(kind: 'embedding' | 'generative', id: string, backend: ModelBackend): void ``` Registers `backend` under the logical name `id` for the given `kind`. A method on `models` (reachable as `models.registerBackend(...)` / `scope.models.registerBackend(...)`). Register during component initialization (for example, in `handleApplication`) so the backend is in place before requests arrive; the registry is process-wide, so each worker thread that loads the component registers its own instance. Use a provider-namespaced `id` (e.g. `local:bge-small`) to avoid collisions when more than one component registers backends. ``` import { models } from 'harper'; import { init, embed } from 'some-local-embedding-library'; await init(); models.registerBackend( 'embedding', 'local:bge-small', models.defineBackend({ name: 'local:bge-small', async embed(input) { const texts = Array.isArray(input) ? input : [input]; const vectors = await embed(texts); return { status: 'completed', output: vectors.map((v) => Float32Array.from(v)) }; }, }) ); // Selected like any other model: const [vector] = await models.embed('What is Harper?', { model: 'local:bge-small' }); ``` A registered backend takes precedence over a configuration entry with the same logical name, because registration runs after the configuration is loaded. A backend whose `capabilities()` disagrees with the methods it actually implements is registered as-is and fails at call time — `defineBackend()` keeps the two consistent. ### Config-selectable backends A `backend` value in the [`models` configuration](/reference/v5/models/overview.md#configuration) that isn't a built-in name is resolved as a **module specifier** and imported at startup; the module's default export — or a `register` export — is a factory that registers the backend. This lets an operator select a custom backend entirely from config, the same way the built-ins are selected. ``` models: embedding: default: backend: '@acme/embedder' # an installed package model: bge-small ``` The `backend` specifier is resolved as: * a **bare package** (`@acme/embedder`) — resolved from the Harper instance's `node_modules`; install the backend as a dependency. Preferred, since it carries no filesystem path and travels with the deployment. * an **instance-root-relative path** (`./backends/local.js`) — resolved against the Harper instance root. * an **absolute path**. The factory has the signature `({ logicalName, kind, config }) => void | Promise` and registers via [`models.registerBackend`](#registerbackend); it receives the config entry with `${VAR}` placeholders already resolved. A `backend` that is neither a built-in nor an importable module is logged and skipped at startup, leaving other entries unaffected. --- # Models *Added in: v5.1.0* Harper provides a unified API for calling AI models — text embeddings and text generation — from application code. Models are configured by an operator under logical names; application code requests a model by its logical name and Harper routes the call to the configured backend (Ollama, OpenAI, Anthropic, or Amazon Bedrock) — or to a [custom backend](/reference/v5/models/backends.md#custom-backends) a component registers. Swapping providers is a configuration change, not a code change. A logical name can also name an ordered group of backends to try, and calls can require specific capabilities — see [Routing & Fallback](/reference/v5/models/routing.md). The API is exposed as a single process-wide `models` object: ``` import { models } from 'harper'; const [vector] = await models.embed('What is Harper?'); const reply = await models.generate('Describe the Harper resource API in one sentence.'); ``` The same object is available as `scope.models` in component scopes and as the `models` global. All three refer to the same instance. The API surface is three methods: | Method | Purpose | | -------------------------------------------------------------------------------------- | ------------------------------------------ | | [`models.embed(input, options?)`](/reference/v5/models/api.md#embed) | Convert text to embedding vectors | | [`models.generate(input, options?)`](/reference/v5/models/api.md#generate) | Generate a completion for a prompt or chat | | [`models.generateStream(input, options?)`](/reference/v5/models/api.md#generatestream) | Stream a completion as it is produced | Generation supports [tool calling](/reference/v5/models/tool-calling.md), including a built-in agent loop (`toolMode: 'auto'`) that resolves tool calls in-process. Tables can compute embedding vectors automatically at write time with the [`@embed` schema directive](/reference/v5/database/schema.md#embed), and vectors can be searched with [HNSW vector indexes](/reference/v5/database/schema.md#vector-indexing). Every model call is recorded for [observability and usage accounting](/reference/v5/models/analytics.md). ## Configuration Models are configured in the `models` section of `harper-config.yaml`, split by capability into `embedding` and `generative` maps. Each key is a logical model name; each entry names a `backend` plus backend-specific settings: ``` models: embedding: default: backend: ollama host: localhost:11434 model: nomic-embed-text:latest generative: default: backend: openai apiKey: ${OPENAI_API_KEY} model: gpt-4o fast: backend: ollama model: mistral:7b ``` The logical name `default` is used when application code does not pass an explicit `model` option. Calling a logical name that is not configured throws an error. See [Backends](/reference/v5/models/backends.md) for the full set of configuration fields supported by each backend. ### Credentials String values in model entries support environment-variable indirection with `${VAR_NAME}` syntax, resolved at startup. Use this for API keys rather than placing the literal key in the configuration file — Harper logs a warning at startup when a credential field contains a literal value. If the referenced environment variable is unset, the placeholder is left as-is; for credential fields the backend rejects the unresolved placeholder at startup, while other fields (such as `host` or `model`) carry the literal placeholder into requests — surfacing as per-request failures rather than a startup error. Indirection applies to string-typed fields only; numeric fields such as `requestTimeoutMs` must be literal values. ### Startup behavior Model entries are registered when Harper boots, before components load, so `models` is usable from component initialization onward. Model entries are validated with the rest of the configuration file at startup: a structurally invalid entry — a missing required field such as `apiKey`, an unrecognized field name, or a wrong value type — fails configuration validation and prevents Harper from starting, like any other configuration error. Errors at registration time (for example, an unrecognized `backend` name) are logged and skipped without blocking startup or other model entries. --- # Routing & Fallback *Added in: v5.1.15* Every `models` call resolves through a **router** that returns an ordered list of candidate backends for the requested logical name. The facade uses the first candidate whose capabilities satisfy the call and falls through to the next on failure — capability-aware selection and fallback are the same mechanism. By default the router is a name-lookup + capability filter over a logical name's [fallback group](#fallback-groups). A component can replace it with [`models.registerRouter()`](#custom-routers) for cost-, latency-, or tenant-aware policies. ## Fallback groups A model entry can name other logical models to try, in order, after itself. Configure the group with `fallback` in the [models configuration](/reference/v5/models/overview.md#configuration): ``` models: generative: default: backend: openai model: gpt-4o-mini fallback: [local-llama] # try `default`, then `local-llama` local-llama: backend: ollama model: llama3.2 ``` A call to the `default` model tries `openai` first; if it fails, the call falls through to `local-llama`. The group is `[default, ...fallback]`, de-duplicated and in order. Groups are rebuilt on every configuration reload, so removing a `fallback` takes effect on the next reload. ## Capability routing A call can require capabilities of the backend it lands on. The router keeps only the candidates whose `capabilities()` satisfy the requirement, in group order. * **`opts.requires`** — an explicit list of capabilities (`embed`, `generate`, `stream`, `tools`, `adapters`). * **Tools auto-require `tools`** — a `generate()` call whose input carries a `tools` array routes to a tools-capable candidate in the group instead of erroring on a backend that can't do tools. ``` // Tools in the input auto-route to a tools-capable candidate in the group: const reply = await models.generate( { messages: [{ role: 'user', content: 'What is the weather?' }], tools: [weatherTool] }, { model: 'default' } ); // Or require a capability explicitly: const reply = await models.generate('…', { model: 'default', requires: ['tools'] }); ``` If no candidate in the group satisfies the required capabilities, the call throws a capability error naming the primary backend — no request is made. ## Fallback on error When a candidate fails, `embed` / `generate` record the attempt and try the next candidate. Every attempt — success or failure — is written to [model-call analytics](/reference/v5/models/analytics.md), so a fallthrough is observable. * **Any backend error falls through** to the next candidate — the facade's default is to fall back on any error. Candidates are heterogeneous (a limit or input error on one backend may succeed on another with different constraints), so *filtering* which errors should skip the fallback is a router or caller policy, not something the facade decides. * **A caller abort short-circuits.** If the call's `signal` is already aborted, the loop stops and surfaces the abort rather than spending another backend call. * **The primary error is surfaced.** If every candidate fails, the error thrown is the **first (primary)** candidate's — the model you asked for, and usually the most diagnostic — not the last fallback's. `generateStream` resolves to the **first** candidate only. There is no mid-stream fallback: once chunks have been yielded, switching backends would mean replaying already-delivered output. ## Custom routers Replace the default policy with `models.registerRouter()`. A router is a single **synchronous** `route()` method that returns the ordered candidate backends for a request; an empty array means "no candidate." ``` models.registerRouter(router: ModelRouter): void interface ModelRouter { route(req: RouteRequest): ModelBackend[]; } interface RouteRequest { kind: 'embedding' | 'generative'; logicalName: string; // from opts.model; defaults to 'default' requires: Capability[]; // capabilities the chosen backend must satisfy hints?: Record; // free-form (tenant, prompt size, …) for custom policies } ``` `route()` is synchronous so the facade keeps its up-front resolution errors (notably `generateStream`'s synchronous throw for an unknown model). A registered router **fully replaces** the default — it is responsible for every `kind` and logical name it should serve. Register during component initialization; the override is process-wide and one router serves the whole process (last registration wins), so it is a deployment/application control rather than something independent components should each set. ``` import { models } from 'harper'; // Backends this policy chooses among (or capture already-registered ones): const fast = models.defineBackend({ name: 'fast', generate: generateFast }); const cheap = models.defineBackend({ name: 'cheap', generate: generateCheap }); models.registerRouter({ route({ kind, requires, hints }) { if (kind !== 'generative') return []; // this policy only routes generation // e.g. prefer the cheap backend for small prompts, the fast one otherwise: const order = hints?.small ? [cheap, fast] : [fast, cheap]; return order.filter((b) => requires.every((cap) => b.capabilities()[cap])); }, }); ``` A custom router that returns no candidates when a backend *does* satisfy the requirement surfaces a plain "no routing candidates available" error — not a misleading capability error against a backend that actually supports the call. --- # Tool Calling *Added in: v5.1.0* Generative models can be given tools — functions the model may request calls to while producing a response. Tools are declared on the input object (they are model-facing content, like messages), and the `toolMode` option selects who resolves them: * **`toolMode: 'return'`** (default) — `generate()` returns as soon as the model requests tool calls; your code dispatches them and continues the conversation. * **`toolMode: 'auto'`** — Harper runs an in-process loop: it dispatches each requested tool to a handler you supply, feeds results back to the model, and repeats until the model produces a final answer or a budget is exhausted. Tool calling requires a backend that supports tools (see the [backend capability table](/reference/v5/models/backends.md)). Declaring tools against a backend without tool support fails up front rather than silently dropping the tools. ## Declaring tools Use the object form of the generation input. Each tool has a name, a description, and a JSON Schema for its arguments: ``` const input = { system: 'You are a helpful assistant.', messages: [{ role: 'user', content: 'What is the weather in Denver?' }], tools: [ { name: 'get_weather', description: 'Get the current weather for a city.', parameters: { type: 'object', properties: { city: { type: 'string' } }, required: ['city'], }, }, ], }; ``` ## toolMode: 'return' The model's requested calls come back on the result, and `finishReason` is `'tool_calls'`. Your code runs the tools, appends the results as `tool`-role messages, and calls `generate()` again: ``` const result = await models.generate(input); if (result.finishReason === 'tool_calls') { const followUp = [...input.messages, { role: 'assistant', content: result.content, toolCalls: result.toolCalls }]; for (const call of result.toolCalls) { const output = await getWeather(call.arguments); // your dispatch followUp.push({ role: 'tool', toolCallId: call.id, content: JSON.stringify(output) }); } const final = await models.generate({ ...input, messages: followUp }); } ``` `ToolCall.arguments` is always a parsed object — backends that deliver stringified JSON normalize it before returning. ## toolMode: 'auto' Supply handlers keyed by tool name and Harper resolves the calls in-process: ``` const result = await models.generate(input, { toolMode: 'auto', toolHandlers: { get_weather: async ({ city }, ctx) => fetchWeather(city, { signal: ctx.signal }), }, maxToolIterations: 5, includeToolTrace: true, }); console.log(result.content); // final answer, tool round-trips already resolved ``` A handler receives the parsed arguments and a context object `{ signal, accounting }`. The `signal` is the composed cancellation signal for the iteration — it fires if the caller aborts or a budget trips, so long-running handlers should honor it. The handler's return value is JSON-serialized and fed back to the model; a thrown error is routed by `toolErrorMode` (below). `generateStream()` supports `toolMode: 'auto'` as well: content deltas stream out as each round produces them, and `finishReason` is emitted exactly once, on the final chunk of the final round. ### Options All options below apply only with `toolMode: 'auto'`. | Option | Type | Default | Description | | -------------------- | ----------------------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `toolHandlers` | `Record` | — | Dispatch table keyed by tool name. A tool declared in `tools` with no handler here is a configuration error (400) | | `maxToolIterations` | `number` | `10` | Hard cap on model → tools → model rounds | | `maxToolTokens` | `number` | — | Cumulative prompt+completion token cap across rounds. Best-effort: requires the backend to report usage; if it does not, Harper warns once and `maxToolIterations` remains the bound. Not supported on `generateStream()` (throws) | | `toolParallelism` | `'parallel'` \| `'serial'` | `'parallel'` | When one round requests multiple tool calls, run handlers concurrently or in order | | `toolResultMaxBytes` | `number` | `65536` | Per-result byte cap (JSON-stringified). Larger results are truncated with a marker; the model sees the truncated form, the trace records the original size | | `toolErrorMode` | `'recover'` \| `'abort'` | `'recover'` | `'recover'` feeds a handler error back to the model as the tool result so it can react; `'abort'` stops the loop and throws with the trace attached | | `includeToolTrace` | `boolean` | `false` | Populate `result.trace` with one entry per tool invocation (iteration, name, arguments, result size, duration, error) | | `conversation` | `ConversationAppender` | — | Optional persistence hook — see below | ### Budgets and errors When `maxToolIterations` or `maxToolTokens` is exhausted, the loop throws a budget-exceeded error (HTTP status 429) carrying a `partialTrace` of everything that ran — the trace is attached on error paths regardless of `includeToolTrace`. With `toolErrorMode: 'abort'`, a handler failure throws an error carrying the same trace. If the model requests a tool name that was never declared in `tools` (a hallucinated tool), the call is treated as a tool error and routed by `toolErrorMode` — with `'recover'`, the model is told the tool is unknown and can correct itself. ### Conversation persistence The `conversation` option accepts any object with an `append(turn)` method returning a promise. The loop awaits `append` for each new turn it produces — assistant turns (with their tool calls) and tool-result turns — in order, giving the appender back-pressure over the loop. The caller's own input messages are not echoed back through the hook. Appenders should catch their own recoverable failures; a throw from `append` becomes the loop's terminal error. ### Reserved options `toolArgValidation` (`'strict'` / `'lenient'` JSON Schema validation of tool arguments), `maxCostUsd`, and `conversationId` exist on the type surface but are not functional in 5.1 — the validation modes and streaming token budgets throw a `501` error, and the cost cap has no rate card behind it yet. Don't rely on them. --- # MQTT Configuration The `mqtt` section in `harper-config.yaml` controls Harper's built-in MQTT broker. MQTT is enabled by default. Harper must be restarted for configuration changes to take effect. ## Minimal Example ``` mqtt: network: port: 1883 securePort: 8883 webSocket: true requireAuthentication: true ``` ## Ports ### `mqtt.network.port` Type: `integer` Default: `1883` The port for plaintext (non-TLS) MQTT connections. ### `mqtt.network.securePort` Type: `integer` Default: `8883` The port for secure MQTT connections (MQTTS). Uses the `tls` configuration for certificates. See [TLS Configuration](/reference/v5/http/tls.md) for certificate setup. ## WebSocket ### `mqtt.webSocket` Type: `boolean` Default: `true` Enables MQTT over WebSockets. When enabled, Harper handles WebSocket connections on the HTTP port (default `9926`) that specify the `mqtt` sub-protocol (`Sec-WebSocket-Protocol: mqtt`). This is required by the MQTT specification and should be set by any conformant MQTT-over-WebSocket client. ``` mqtt: webSocket: true ``` ## Authentication ### `mqtt.requireAuthentication` Type: `boolean` Default: `true` Controls whether credentials are required to establish an MQTT connection. When `true`, clients must authenticate with either a username/password or a valid mTLS client certificate. When set to `false`, unauthenticated connections are allowed. Unauthenticated clients are still subject to authorization on each publish and subscribe operation — by default, tables and resources do not grant access to unauthenticated users, but this can be configured at the resource level. ``` mqtt: requireAuthentication: true ``` ## mTLS ### `mqtt.network.mtls` *Added in: v4.3.0* Type: `boolean | object` Default: `false` Enables mutual TLS (mTLS) authentication for MQTT connections. When set to `true`, client certificates are verified against the CA specified in the root `tls.certificateAuthority` section. Authenticated connections use the `CN` (common name) from the client certificate's subject as the Harper username by default. ``` mqtt: network: mtls: true ``` For granular control, specify an object with the following optional properties: ### `mqtt.network.mtls.user` Type: `string | null` Default: Common Name from client certificate Specifies a fixed username to authenticate all mTLS connections as. When set, any connection that passes certificate verification authenticates as this user regardless of the certificate's CN. Setting to `null` disables credential-based authentication for mTLS connections. When combined with `required: true`, this enforces that clients must have a valid certificate AND provide separate credential-based authentication. ### `mqtt.network.mtls.required` Type: `boolean` Default: `false` When `true`, all incoming MQTT connections must provide a valid client certificate. Connections without a valid certificate are rejected. By default, clients can authenticate with either mTLS or standard username/password credentials. ### `mqtt.network.mtls.certificateAuthority` Type: `string` Default: Path from `tls.certificateAuthority` Path to the certificate authority (CA) file used to verify MQTT client certificates. By default, uses the CA configured in the root `tls` section. Set this if MQTT clients should be verified against a different CA than the one used for HTTP/TLS. ### `mqtt.network.mtls.certificateVerification` Type: `boolean | object` Default: `true` When mTLS is enabled, Harper verifies the revocation status of client certificates using OCSP (Online Certificate Status Protocol). This ensures revoked certificates cannot be used for authentication. Set to `false` to disable revocation checking, or configure as an object: | Property | Type | Default | Description | | ------------- | ------- | ------------- | ------------------------------------------------------------------------------------------------------ | | `timeout` | integer | `5000` | Maximum milliseconds to wait for an OCSP response. | | `cacheTtl` | integer | `3600000` | Milliseconds to cache successful verification results (default 1h). | | `failureMode` | string | `'fail-open'` | Behavior when OCSP verification fails: `'fail-open'` (allow, log warning) or `'fail-closed'` (reject). | ``` mqtt: network: mtls: required: true certificateVerification: failureMode: fail-closed timeout: 5000 cacheTtl: 3600000 ``` ## mTLS Examples ``` # Require client certificate + standard credentials (combined auth) mqtt: network: mtls: user: null required: true # Authenticate all mTLS connections as a fixed user mqtt: network: mtls: user: mqtt-service-account required: true # mTLS optional — clients can use mTLS or credentials mqtt: network: mtls: true ``` ## Logging ### `mqtt.logging` Type: `object` Default: disabled Configures logging for MQTT activity. Accepts the standard logging configuration options. ``` mqtt: logging: path: ~/hdb/log/mqtt.log level: warn stdStreams: false ``` | Option | Description | | ------------ | ------------------------------------------------- | | `path` | File path for the MQTT log output. | | `root` | Alternative to `path` — sets the log directory. | | `level` | Log level: `error`, `warn`, `info`, `debug`, etc. | | `tag` | Custom tag to prefix log entries. | | `stdStreams` | When `true`, also logs to stdout/stderr. | ## Complete Example ``` mqtt: network: port: 1883 securePort: 8883 mtls: required: false certificateAuthority: ~/hdb/keys/ca.pem certificateVerification: failureMode: fail-open timeout: 5000 cacheTtl: 3600000 webSocket: true requireAuthentication: true logging: level: warn path: ~/hdb/log/mqtt.log # TLS is a top-level section, shared with HTTP tls: certificate: ~/hdb/keys/certificate.pem certificateAuthority: ~/hdb/keys/ca.pem privateKey: ~/hdb/keys/privateKey.pem ``` ## Related * [MQTT Overview](/reference/v5/mqtt/overview.md) * [TLS Configuration](/reference/v5/http/tls.md) * [Security Overview](/reference/v5/security/overview.md) * [Configuration Overview](/reference/v5/configuration/overview.md) --- # MQTT *Added in: v4.2.0* Harper includes a built-in MQTT broker that provides real-time pub/sub messaging deeply integrated with the database. Unlike a generic MQTT broker, Harper's MQTT implementation connects topics directly to database records — publishing to a topic writes to the database, and subscribing to a topic delivers live updates for the corresponding record. ## How Topics Map to Database Records MQTT topics in Harper follow the same path convention as REST endpoints. If you define a table or resource with an endpoint path of `my-resource`, the corresponding MQTT topic namespace is `my-resource`. A topic of `my-resource/some-id` corresponds to the record with id `some-id` in the `my-resource` table (or custom resource). This means: * **Subscribing** to `my-resource/some-id` delivers notification messages whenever that record is updated or deleted. * The **current value** of the record is treated as the retained message for that topic. On subscription, the subscriber immediately receives the current record as the initial retained message — no separate GET request needed. * **Publishing** with the `retain` flag set replaces the record in the database (equivalent to a PUT operation). * **Publishing without** the `retain` flag delivers the message to current subscribers without writing to the database. Defining a table that creates a topic can be as simple as adding a table with no attributes to your [schema.graphql](/reference/v5/database/schema.md) in a Harper application: ``` type MyTopic @table @export ``` ## Protocol Support Harper supports MQTT versions **v3.1.1** and **v5**, with standard publish/subscribe capabilities. ### Topics and Wildcards Harper supports multi-level topics for both publishing and subscribing: * **Multi-level wildcard (`#`)** — Subscribe to `my-resource/#` to receive notifications for all records in that resource, including nested paths (`my-resource/some-id`, `my-resource/nested/id`). * **Single-level wildcard (`+`)** — Added in v4.3.0. Subscribe to `my-resource/+/status` to match any single path segment. ### QoS Levels * **QoS 0** — At most once delivery (fire and forget). * **QoS 1** — At least once delivery (acknowledged delivery). * **QoS 2** — Harper can perform the QoS 2 conversation but does not guarantee exactly-once delivery. ### Sessions * **Clean sessions** — Subscriptions and queued messages are discarded on disconnect. * **Durable sessions** — Subscriptions and queued messages are persisted across reconnects. See [Durable Sessions](#durable-sessions) below. ### Durable Sessions A durable session retains a client's subscription list and any unacknowledged messages across disconnects. When the client reconnects with the same client ID, it picks up from where it left off — including any messages published while it was offline. Durable sessions in Harper are persisted as records in the `hdb_durable_session` system table, indexed by client ID. The session record holds the list of subscriptions (topic + QoS) and the timestamp of the last delivered message per topic. Because durable sessions are records rather than in-memory state, an abandoned session sits idle with no runtime cost until the client reconnects or the record is deleted. **Establishing a durable session** — Connect with a stable client ID and `cleanSession: false` (MQTT v3.1.1) or `cleanStart: false` (MQTT v5): ``` // MQTT v5 with the `mqtt` npm package mqtt.connect('mqtts://harper.example.com:8883', { clientId: 'sensor-42', clean: false, // request a durable session protocolVersion: 5, properties: { sessionExpiryInterval: 86400, // keep session for 24h after disconnect }, }); ``` **Catch-up on reconnect** — When the client reconnects, Harper replays missed messages on subscribed topics by reading the audit log. For this to work, audit logging must be enabled on the tables backing the subscribed topics. See [Transaction Logging](/reference/v5/database/transaction.md) and [`logging.auditLog`](/reference/v5/logging/configuration.md#loggingauditlog). **Session expiry** — In MQTT v5, the `sessionExpiryInterval` property on `CONNECT` controls how long the session is retained after the client disconnects. With `sessionExpiryInterval: 0` (or a clean session connect), Harper deletes the session record at disconnect. Connecting with the same client ID and `clean: true` also explicitly deletes any existing durable session. **Distributed durable sessions** — Harper does not currently maintain a single logical durable session across multiple cluster nodes; a durable session record lives on the node where it was created. Clients that may reconnect to different nodes (e.g., behind a load balancer with round-robin DNS) should use sticky routing or terminate on a fixed node. ### Last Will *Added in: v4.3.0* Harper supports the MQTT Last Will and Testament feature. If a client disconnects unexpectedly, the broker publishes the configured will message on its behalf. Will messages are persisted in the `hdb_session_will` system table at CONNECT time, so they survive a broker restart and fire reliably on unexpected disconnect. ## Content Negotiation Harper handles structured data natively. Messages can be published and received in any supported structured format — JSON, CBOR, or MessagePack — and Harper stores and delivers them as structured objects. Different clients can independently choose their preferred format: one client may publish in JSON while another subscribes and receives in CBOR. ## Ordering and Distributed Delivery Harper is designed for distributed, low-latency message delivery. Messages are delivered to subscribers immediately on arrival — Harper does not delay delivery to coordinate consensus across nodes. In a distributed cluster, messages may arrive out of order due to network topology. The behavior depends on whether the message is retained or non-retained: * **Retained messages** (published with `retain: true`, or written via PUT/upsert) maintain eventual consistency across the cluster. Harper keeps the message with the latest timestamp as the winning record state. An out-of-order earlier message will not be re-delivered to clients; the cluster converges to the most recent state. * **Non-retained messages** are always delivered to local subscribers when received, even if they arrive out of order. Every message is delivered, prioritizing completeness over strict ordering. **Non-retained messages** are suited for applications like chat where every message must be delivered. **Retained messages** are suited for sensor readings or state updates where only the latest value matters. ## Authentication MQTT connections support two authentication methods: * **Credential-based** — Standard MQTT username/password in the CONNECT packet. * **mTLS** — Added in v4.3.0. Mutual TLS authentication using client certificates. The `CN` (common name) from the client certificate subject is used as the Harper username by default. Authentication is required by default (`requireAuthentication: true`). See [MQTT Configuration](/reference/v5/mqtt/configuration.md) for details on disabling authentication or configuring mTLS options. ## Server Events API JavaScript components can listen for MQTT connection events via `server.mqtt.events`: ``` server.mqtt.events.on('connected', (session, socket) => { console.log('client connected with id', session.clientId); }); ``` Available events: | Event | Description | | -------------- | ---------------------------------------------------- | | `connection` | Client establishes a TCP or WebSocket connection | | `connected` | Client completes MQTT handshake and is authenticated | | `auth-failed` | Client fails to authenticate | | `disconnected` | Client disconnects | ## Feature Support Matrix | Feature | Support | | --------------------------------------------- | ------------------------------------------------------------ | | MQTT v3.1.1 connections | ✅ | | MQTT v5 connections | ✅ | | Secure MQTTS (TLS) | ✅ | | MQTT over WebSockets | ✅ | | Authentication via username/password | ✅ | | Authentication via mTLS | ✅ (added v4.3.0) | | Publish | ✅ | | Subscribe | ✅ | | Multi-level wildcard (`#`) | ✅ | | Single-level wildcard (`+`) | ✅ (added v4.3.0) | | QoS 0 | ✅ | | QoS 1 | ✅ | | QoS 2 | Not fully supported — conversation supported, not guaranteed | | Keep-Alive monitoring | ✅ | | Clean session | ✅ | | Durable session | ✅ | | Distributed durable session | Not supported | | Last Will | ✅ | | MQTT V5 Subscribe retain handling | ✅ (added v4.3.0) | | MQTT V5 User properties | Not supported | | MQTT V5 Will properties | Not supported | | MQTT V5 Connection properties | Not supported | | MQTT V5 Connection acknowledgement properties | Not supported | | MQTT V5 Publish properties | Not supported | | MQTT V5 Subscribe properties (general) | Not supported | | MQTT V5 Ack properties | Not supported | | MQTT V5 AUTH command | Not supported | | MQTT V5 Shared subscriptions | Not supported | ## Related * [MQTT Configuration](/reference/v5/mqtt/configuration.md) * [HTTP Overview](/reference/v5/http/overview.md) * [Security Overview](/reference/v5/security/overview.md) * [Database Schema](/reference/v5/database/schema.md) * [REST Overview](/reference/v5/rest/overview.md) --- # Operations Reference This page lists all available Operations API operations, grouped by category. Each entry links to the feature section where the full documentation lives. For endpoint and authentication setup, see the [Operations API Overview](/reference/v5/operations-api/overview.md). *** ## Databases & Tables Operations for managing databases, tables, and attributes. Detailed documentation: [Database Overview](/reference/v5/database/overview.md) | Operation | Description | Role Required | | ------------------- | ------------------------------------------------------------------- | ------------- | | `describe_all` | Returns definitions of all databases and tables, with record counts | any | | `describe_database` | Returns all table definitions for a specified database | any | | `describe_table` | Returns the definition of a specified table | any | | `create_database` | Creates a new database | super\_user | | `drop_database` | Drops a database and all its tables/records | super\_user | | `create_table` | Creates a new table with optional schema and expiration | super\_user | | `drop_table` | Drops a table and all its records | super\_user | | `create_attribute` | Adds a new attribute to a table | super\_user | | `drop_attribute` | Removes an attribute and all its values from a table | super\_user | | `get_backup` | Returns a binary snapshot of a database for backup purposes | super\_user | ### `describe_all` Returns the definitions of all databases and tables within the database. Record counts above 5000 records are estimated; the response includes `estimated_record_range` when estimated. To force an exact count (requires full table scan), include `"exact_count": true`. ``` { "operation": "describe_all" } ``` ### `describe_database` Returns all table definitions within the specified database. ``` { "operation": "describe_database", "database": "dev" } ``` ### `describe_table` Returns the definition of a specific table. ``` { "operation": "describe_table", "table": "dog", "database": "dev" } ``` ### `create_database` Creates a new database. ``` { "operation": "create_database", "database": "dev" } ``` ### `drop_database` Drops a database and all its tables/records. Supports `"replicated": true` to propagate to all cluster nodes. ``` { "operation": "drop_database", "database": "dev" } ``` ### `create_table` Creates a new table. Optional fields: `database` (defaults to `data`), `attributes` (array defining schema), `expiration` (TTL in seconds). ``` { "operation": "create_table", "database": "dev", "table": "dog", "primary_key": "id" } ``` ### `drop_table` Drops a table and all associated records. Supports `"replicated": true`. ``` { "operation": "drop_table", "database": "dev", "table": "dog" } ``` ### `create_attribute` Creates a new attribute within a table. Harper auto-creates attributes on insert/update, but this can be used to pre-define them (e.g., for role-based permission setup). ``` { "operation": "create_attribute", "database": "dev", "table": "dog", "attribute": "is_adorable" } ``` ### `drop_attribute` Drops an attribute and all its values from the specified table. ``` { "operation": "drop_attribute", "database": "dev", "table": "dog", "attribute": "is_adorable" } ``` ### `get_backup` Returns a binary snapshot of the specified database (or individual table). Safe for backup while Harper is running. Specify `"table"` for a single table or `"tables"` for a set. ``` { "operation": "get_backup", "database": "dev" } ``` *** ## NoSQL Operations Operations for inserting, updating, deleting, and querying records using NoSQL. Detailed documentation: [REST Querying Reference](/reference/v5/rest/querying.md) | Operation | Description | Role Required | | ---------------------- | ------------------------------------------------------------------------- | ------------- | | `insert` | Inserts one or more records | any | | `update` | Updates one or more records by primary key | any | | `upsert` | Inserts or updates records | any | | `delete` | Deletes records by primary key | any | | `search_by_id` | Retrieves records by primary key | any | | `search_by_value` | Retrieves records matching a value on any attribute | any | | `search_by_conditions` | Retrieves records matching complex conditions with sorting and pagination | any | ### `insert` Inserts one or more records. If a primary key is not provided, a GUID or auto-increment value is generated. ``` { "operation": "insert", "database": "dev", "table": "dog", "records": [{ "id": 1, "dog_name": "Penny" }] } ``` ### `update` Updates one or more records. Primary key must be supplied for each record. ``` { "operation": "update", "database": "dev", "table": "dog", "records": [{ "id": 1, "weight_lbs": 38 }] } ``` ### `upsert` Updates existing records and inserts new ones. Matches on primary key if provided. ``` { "operation": "upsert", "database": "dev", "table": "dog", "records": [{ "id": 1, "weight_lbs": 40 }] } ``` ### `delete` Deletes records by primary key values. ``` { "operation": "delete", "database": "dev", "table": "dog", "ids": [1, 2] } ``` ### `search_by_id` Returns records matching the given primary key values. Use `"get_attributes": ["*"]` to return all attributes. ``` { "operation": "search_by_id", "database": "dev", "table": "dog", "ids": [1, 2], "get_attributes": ["dog_name", "breed_id"] } ``` ### `search_by_value` Returns records with a matching value on any attribute. Supports wildcards (e.g., `"Ky*"`). ``` { "operation": "search_by_value", "database": "dev", "table": "dog", "attribute": "owner_name", "value": "Ky*", "get_attributes": ["id", "dog_name"] } ``` ### `search_by_conditions` Returns records matching one or more conditions. Supports `operator` (`and`/`or`), `offset`, `limit`, nested `conditions` groups, and `sort` with multi-level tie-breaking. ``` { "operation": "search_by_conditions", "database": "dev", "table": "dog", "operator": "and", "limit": 10, "get_attributes": ["*"], "conditions": [{ "attribute": "age", "comparator": "between", "value": [5, 8] }] } ``` *** ## Bulk Operations Operations for bulk import/export of data. Detailed documentation: [Database Jobs](/reference/v5/database/jobs.md) | Operation | Description | Role Required | | ----------------------- | -------------------------------------------------------------- | ------------- | | `export_local` | Exports query results to a local file in JSON or CSV | super\_user | | `csv_data_load` | Ingests CSV data provided inline | any | | `csv_file_load` | Ingests CSV data from a server-local file path | any | | `csv_url_load` | Ingests CSV data from a URL | any | | `export_to_s3` | Exports query results to AWS S3 | super\_user | | `import_from_s3` | Imports CSV or JSON data from AWS S3 | any | | `delete_records_before` | Deletes records older than a given timestamp (local node only) | super\_user | All bulk import/export operations are asynchronous and return a job ID. Use [`get_job`](#get_job) to check status. ### `export_local` Exports query results to a local path on the server. Formats: `json` or `csv`. ``` { "operation": "export_local", "format": "json", "path": "/data/", "search_operation": { "operation": "sql", "sql": "SELECT * FROM dev.dog" } } ``` ### `csv_data_load` Ingests inline CSV data. Actions: `insert` (default), `update`, `upsert`. ``` { "operation": "csv_data_load", "database": "dev", "table": "dog", "action": "insert", "data": "id,name\n1,Penny\n" } ``` ### `csv_file_load` Ingests CSV from a file path on the server running Harper. ``` { "operation": "csv_file_load", "database": "dev", "table": "dog", "file_path": "/home/user/imports/dogs.csv" } ``` ### `csv_url_load` Ingests CSV from a URL. ``` { "operation": "csv_url_load", "database": "dev", "table": "dog", "csv_url": "https://example.com/dogs.csv" } ``` ### `export_to_s3` Exports query results to an AWS S3 bucket as JSON or CSV. ``` { "operation": "export_to_s3", "format": "json", "s3": { "aws_access_key_id": "YOUR_KEY", "aws_secret_access_key": "YOUR_SECRET", "bucket": "my-bucket", "key": "dogs.json", "region": "us-east-1" }, "search_operation": { "operation": "sql", "sql": "SELECT * FROM dev.dog" } } ``` ### `import_from_s3` Imports CSV or JSON from an AWS S3 bucket. File must include a valid `.csv` or `.json` extension. ``` { "operation": "import_from_s3", "database": "dev", "table": "dog", "s3": { "aws_access_key_id": "YOUR_KEY", "aws_secret_access_key": "YOUR_SECRET", "bucket": "my-bucket", "key": "dogs.csv", "region": "us-east-1" } } ``` ### `delete_records_before` Deletes records older than the specified timestamp from the local node only. Clustered nodes retain their data. ``` { "operation": "delete_records_before", "date": "2021-01-25T23:05:27.464", "schema": "dev", "table": "dog" } ``` *** ## SQL Operations Operations for executing SQL statements. warning Harper SQL is intended for data investigation and use cases where performance is not a priority. For production workloads, use NoSQL or REST operations. SQL performance optimizations are on the roadmap. Detailed documentation: [SQL Reference](/reference/v5/operations-api/sql.md) | Operation | Description | Role Required | | --------- | ------------------------------------------------------------------ | ------------- | | `sql` | Executes a SQL `SELECT`, `INSERT`, `UPDATE`, or `DELETE` statement | any | ### `sql` Executes a standard SQL statement. ``` { "operation": "sql", "sql": "SELECT * FROM dev.dog WHERE id = 1" } ``` *** ## Users & Roles Operations for managing users and role-based access control (RBAC). Detailed documentation: [Users & Roles Operations](/reference/v5/users-and-roles/operations.md) | Operation | Description | Role Required | | ------------ | --------------------------------------------------- | ------------- | | `list_roles` | Returns all roles | super\_user | | `add_role` | Creates a new role with permissions | super\_user | | `alter_role` | Modifies an existing role's permissions | super\_user | | `drop_role` | Deletes a role (role must have no associated users) | super\_user | | `list_users` | Returns all users | super\_user | | `user_info` | Returns data for the authenticated user | any | | `add_user` | Creates a new user | super\_user | | `alter_user` | Modifies an existing user's credentials or role | super\_user | | `drop_user` | Deletes a user | super\_user | ### `list_roles` Returns all roles defined in the instance. ``` { "operation": "list_roles" } ``` ### `add_role` Creates a new role with the specified permissions. The `permission` object maps database names to table-level access rules (`read`, `insert`, `update`, `delete`). Set `super_user: true` to grant full access. ``` { "operation": "add_role", "role": "developer", "permission": { "super_user": false, "dev": { "tables": { "dog": { "read": true, "insert": true, "update": true, "delete": false } } } } } ``` ### `alter_role` Modifies an existing role's name or permissions. Requires the role's `id` (returned by `list_roles`). ``` { "operation": "alter_role", "id": "f92162e2-cd17-450c-aae0-372a76859038", "role": "senior_developer", "permission": { "super_user": false, "dev": { "tables": { "dog": { "read": true, "insert": true, "update": true, "delete": true } } } } } ``` ### `drop_role` Deletes a role. The role must have no associated users before it can be dropped. ``` { "operation": "drop_role", "id": "f92162e2-cd17-450c-aae0-372a76859038" } ``` ### `list_users` Returns all users. ``` { "operation": "list_users" } ``` ### `user_info` Returns data for the currently authenticated user. ``` { "operation": "user_info" } ``` ### `add_user` Creates a new user. `username` cannot be changed after creation. `password` is stored encrypted. ``` { "operation": "add_user", "role": "developer", "username": "hdb_user", "password": "password", "active": true } ``` ### `alter_user` Modifies an existing user's password, role, or active status. All fields except `username` are optional. ``` { "operation": "alter_user", "username": "hdb_user", "password": "new_password", "role": "senior_developer", "active": true } ``` ### `drop_user` Deletes a user by username. ``` { "operation": "drop_user", "username": "hdb_user" } ``` See [Users & Roles Operations](/reference/v5/users-and-roles/operations.md) for full documentation including permission object structure. *** ## Token Authentication Operations for JWT token creation and refresh. Detailed documentation: [JWT Authentication](/reference/v5/security/jwt-authentication.md) | Operation | Description | Role Required | | ------------------------------ | ------------------------------------------------------- | ---------------------- | | `create_authentication_tokens` | Creates an operation token and refresh token for a user | none (unauthenticated) | | `refresh_operation_token` | Creates a new operation token from a refresh token | any | ### `create_authentication_tokens` Does not require prior authentication. Returns `operation_token` (short-lived JWT) and `refresh_token` (long-lived JWT). ``` { "operation": "create_authentication_tokens", "username": "my-user", "password": "my-password" } ``` ### `refresh_operation_token` Creates a new operation token from an existing refresh token. ``` { "operation": "refresh_operation_token", "refresh_token": "EXISTING_REFRESH_TOKEN" } ``` *** ## Components Operations for deploying and managing Harper components (applications, plugins). Detailed documentation: [Components Overview](/reference/v5/components/overview.md) | Operation | Description | Role Required | | --------------------------- | ----------------------------------------------------------------------- | ------------- | | `add_component` | Creates a new component project from a template | super\_user | | `deploy_component` | Deploys a component via payload (tar) or package reference (NPM/GitHub) | super\_user | | `package_component` | Packages a component project into a base64-encoded tar | super\_user | | `drop_component` | Deletes a component or a file within a component | super\_user | | `get_components` | Lists all component files and config | super\_user | | `get_component_file` | Returns the contents of a file within a component | super\_user | | `set_component_file` | Creates or updates a file within a component | super\_user | | `list_deployments` | Lists deployment records with optional filters | super\_user | | `get_deployment` | Fetches a single deployment record by ID; supports SSE streaming | super\_user | | `get_deployment_payload` | Returns the tarball stored for a deployment | super\_user | | `delete_deployment_payload` | Removes the stored tarball to free space | super\_user | | `add_ssh_key` | Adds an SSH key for deploying from private repositories | super\_user | | `update_ssh_key` | Updates an existing SSH key | super\_user | | `delete_ssh_key` | Deletes an SSH key | super\_user | | `list_ssh_keys` | Lists all configured SSH key names | super\_user | | `set_ssh_known_hosts` | Overwrites the SSH known\_hosts file | super\_user | | `get_ssh_known_hosts` | Returns the contents of the SSH known\_hosts file | super\_user | | `install_node_modules` | *(Deprecated)* Run npm install on component projects | super\_user | ### `deploy_component` Deploys a component. The `package` option accepts any valid NPM reference including GitHub repos (`HarperDB/app#semver:v1.0.0`), tarballs, or NPM packages. The `payload` option accepts a base64-encoded tar string from `package_component`. Supports `"replicated": true` and `"restart": true` or `"restart": "rolling"`. Additional parameters: * `urlPath` — override the HTTP URL path the component is mounted at (e.g. `"/api/v2"`) * `install_allow_scripts` — set to `true` to allow npm pre/post install scripts (disabled by default) The response includes a `deployment_id` that can be used to query the deployment record: ``` { "operation": "deploy_component", "project": "my-app", "package": "my-org/my-app#semver:v1.2.3", "replicated": true, "restart": "rolling" } ``` Response: ``` { "deployment_id": "a3f8c2d1...", "message": "Component deployed successfully" } ``` ### Deployment Operations Harper records every `deploy_component` call in the `system.hdb_deployment` table, capturing the full lifecycle of a deployment including phase transitions (prepare → load → replicate → restart → success/failed), per-node outcomes, and a bounded event log of install output. ### `list_deployments` Returns a list of deployment records, newest first. All filter parameters are optional. | Parameter | Type | Description | | --------- | ------ | ------------------------------------------------ | | `project` | string | Filter to a specific component project | | `status` | string | Filter by status: `pending`, `success`, `failed` | | `since` | number | Start of time range (Unix timestamp ms) | | `until` | number | End of time range (Unix timestamp ms) | | `limit` | number | Maximum number of results (default: 100) | | `offset` | number | Pagination offset | ``` { "operation": "list_deployments", "project": "my-app", "status": "success", "limit": 20 } ``` Response includes a `deployments` array and a `total` count. The `payload_blob` field is stripped from list responses for size; use `get_deployment_payload` to retrieve the tarball. ### `get_deployment` Returns a single deployment record by `deployment_id`. When called on an in-progress deployment via a request that accepts `text/event-stream`, the response streams live phase events and install output as Server-Sent Events, replaying the buffered event log then tailing until the deployment reaches a terminal status. ``` { "operation": "get_deployment", "deployment_id": "a3f8c2d1..." } ``` The deployment record includes: | Field | Description | | -------------------- | ----------------------------------------------------------------------- | | `deployment_id` | Unique identifier (content hash) | | `project` | Component project name | | `package_identifier` | Package reference or `payload` for tar uploads | | `status` | `pending`, `success`, `failed`, or `rolled_back` | | `phase` | Current lifecycle phase: `prepare`, `load`, `replicate`, `restart` | | `event_log` | Bounded log of install output and phase transitions (up to 200 entries) | | `peer_results` | Per-node outcome map for replicated deployments | | `payload_hash` | SHA-256 hash of the deployment tarball | | `payload_size` | Byte size of the deployment tarball | | `started_at` | Timestamp when deployment began | | `completed_at` | Timestamp when deployment finished | | `user` | User who initiated the deployment | | `rollback_of` | `deployment_id` of the deployment this rolls back, if applicable | | `error` | Error message for failed deployments | ### `get_deployment_payload` Returns the raw tarball for a deployment. Useful for inspecting or re-deploying a specific version. ``` { "operation": "get_deployment_payload", "deployment_id": "a3f8c2d1..." } ``` ### `delete_deployment_payload` Removes the tarball blob from a deployment record. The deployment record itself is retained; only the binary payload is deleted. Use this to reclaim storage after confirming a deployment is stable. ``` { "operation": "delete_deployment_payload", "deployment_id": "a3f8c2d1..." } ``` ### `add_ssh_key` Adds an SSH key (must be ed25519) for authenticating deployments from private repositories. ``` { "operation": "add_ssh_key", "name": "my-key", "key": "-----BEGIN OPENSSH PRIVATE KEY-----\n...\n-----END OPENSSH PRIVATE KEY-----\n", "host": "my-key.github.com", "hostname": "github.com" } ``` *** ## Replication & Clustering Operations for configuring and managing Harper cluster replication. Detailed documentation: [Replication & Clustering](/reference/v5/replication/clustering.md) | Operation | Description | Role Required | | ----------------------- | --------------------------------------------------------------- | ------------- | | `add_node` | Adds a Harper instance to the cluster | super\_user | | `update_node` | Modifies an existing node's subscriptions | super\_user | | `remove_node` | Removes a node from the cluster | super\_user | | `cluster_status` | Returns current cluster connection status | super\_user | | `configure_cluster` | Bulk-creates/resets cluster subscriptions across multiple nodes | super\_user | | `cluster_set_routes` | Adds routes to the replication routes config (PATCH/upsert) | super\_user | | `cluster_get_routes` | Returns the current replication routes config | super\_user | | `cluster_delete_routes` | Removes routes from the replication routes config | super\_user | ### `add_node` Adds a remote Harper node to the cluster. If `subscriptions` are not provided, a fully replicating cluster is created. Optional fields: `verify_tls`, `authorization`, `retain_authorization`, `revoked_certificates`, `shard`. ``` { "operation": "add_node", "hostname": "server-two", "verify_tls": false, "authorization": { "username": "admin", "password": "password" } } ``` ### `cluster_status` Returns connection state for all cluster nodes, including per-database socket status and replication timing statistics (`lastCommitConfirmed`, `lastReceivedRemoteTime`, `lastReceivedLocalTime`). ``` { "operation": "cluster_status" } ``` ### `configure_cluster` Resets and replaces the entire clustering configuration. Each entry follows the `add_node` schema. ``` { "operation": "configure_cluster", "connections": [ { "hostname": "server-two", "subscriptions": [{ "database": "dev", "table": "dog", "subscribe": true, "publish": true }] } ] } ``` *** ## Configuration Operations for reading and updating Harper configuration. Detailed documentation: [Configuration Overview](/reference/v5/configuration/overview.md) | Operation | Description | Role Required | | ------------------- | ---------------------------------------------------------------- | ------------- | | `set_configuration` | Modifies Harper configuration file parameters (requires restart) | super\_user | | `get_configuration` | Returns the current Harper configuration | super\_user | ### `set_configuration` Updates configuration parameters in `harper-config.yaml`. A restart (`restart` or `restart_service`) is required for changes to take effect. Supports `"replicated": true` *Added in: v5.2.0* to apply the same change to all cluster nodes in one call; per-node outcomes are returned in the response's `replicated` array. Only send cluster-appropriate parameters when replicating — node-local parameters (ports, `node.hostname`, file paths, TLS material, `replication.hostname`/`url`/`routes`) would overwrite every peer's local values. To apply the change cluster-wide, follow with `restart_service` using `"replicated": true` (which restarts nodes one at a time). See [Configuration Operations](/reference/v5/configuration/operations.md#set-configuration) for details. ``` { "operation": "set_configuration", "logging_level": "trace", "replicated": true } ``` ### `get_configuration` Returns the full current configuration object. ``` { "operation": "get_configuration" } ``` *** ## System Operations for restarting Harper and managing system state. | Operation | Description | Role Required | | -------------------- | ----------------------------------------------------- | ------------- | | `restart` | Restarts the Harper instance | super\_user | | `restart_service` | Restarts a specific Harper service | super\_user | | `system_information` | Returns detailed host system metrics | super\_user | | `set_status` | Sets an application-specific status value (in-memory) | super\_user | | `get_status` | Returns a previously set status value | super\_user | | `clear_status` | Removes a status entry | super\_user | ### `restart` Restarts all Harper processes. May take up to 60 seconds. ``` { "operation": "restart" } ``` ### `restart_service` Restarts a specific service. `service` must be one of: `http`, `http_workers`, `custom_functions`, `harperdb` (all currently restart the HTTP workers). Supports `"replicated": true` for a rolling cluster restart. ``` { "operation": "restart_service", "service": "http_workers" } ``` ### `system_information` Returns system metrics including CPU, memory, disk, network, and Harper process info. Optionally filter by `attributes` array (e.g., `["cpu", "memory", "replication"]`). ``` { "operation": "system_information" } ``` ### `set_status` / `get_status` / `clear_status` Manage in-memory application status values. Status types: `primary`, `maintenance`, `availability` (availability only accepts `'Available'` or `'Unavailable'`). Status is not persisted across restarts. ``` { "operation": "set_status", "id": "primary", "status": "active" } ``` *** ## Jobs Operations for querying background job status. Detailed documentation: [Database Jobs](/reference/v5/database/jobs.md) | Operation | Description | Role Required | | --------------------------- | ------------------------------------------------ | ------------- | | `get_job` | Returns status and results for a specific job ID | any | | `search_jobs_by_start_date` | Returns jobs within a specified time window | super\_user | ### `get_job` Returns job status (`COMPLETE`, `IN_PROGRESS`, `ERROR`), timing, and result message for the specified job ID. Bulk import/export operations return a job ID on initiation. ``` { "operation": "get_job", "id": "4a982782-929a-4507-8794-26dae1132def" } ``` ### `search_jobs_by_start_date` Returns all jobs started within the specified datetime range. ``` { "operation": "search_jobs_by_start_date", "from_date": "2021-01-25T22:05:27.464+0000", "to_date": "2021-01-25T23:05:27.464+0000" } ``` *** ## Logs Operations for reading Harper logs. Detailed documentation: [Logging Operations](/reference/v5/logging/operations.md) | Operation | Description | Role Required | | -------------------------------- | ---------------------------------------------------------------------- | ------------- | | `read_log` | Returns entries from the primary `hdb.log` | super\_user | | `read_transaction_log` | Returns transaction history for a table | super\_user | | `delete_transaction_logs_before` | Deletes transaction log entries older than a timestamp | super\_user | | `read_audit_log` | Returns verbose audit history for a table (requires audit log enabled) | super\_user | | `delete_audit_logs_before` | Deletes audit log entries older than a timestamp | super\_user | ### `read_log` Returns entries from `hdb.log`. Filter by `level` (`notify`, `error`, `warn`, `info`, `debug`, `trace`), date range (`from`, `until`), and text `filter`. ``` { "operation": "read_log", "start": 0, "limit": 100, "level": "error" } ``` ### `read_transaction_log` Returns transaction history for a specific table. Optionally filter by `from`/`to` (millisecond epoch) and `limit`. ``` { "operation": "read_transaction_log", "schema": "dev", "table": "dog", "limit": 10 } ``` ### `read_audit_log` Returns verbose audit history including original record state. Requires `logging.auditLog: true` in configuration. Filter by `search_type`: `hash_value`, `timestamp`, or `username`. ``` { "operation": "read_audit_log", "schema": "dev", "table": "dog", "search_type": "username", "search_values": ["admin"] } ``` *** ## Certificate Management Operations for managing TLS certificates in the `hdb_certificate` system table. Detailed documentation: [Certificate Management](/reference/v5/security/certificate-management.md) | Operation | Description | Role Required | | -------------------- | ---------------------------------------------- | ------------- | | `add_certificate` | Adds or updates a certificate | super\_user | | `remove_certificate` | Removes a certificate and its private key file | super\_user | | `list_certificates` | Lists all certificates | super\_user | ### `add_certificate` Adds a certificate to `hdb_certificate`. If a `private_key` is provided, it is written to `/keys/` (not stored in the table). If no private key is provided, the operation searches for a matching one on disk. ``` { "operation": "add_certificate", "name": "my-cert", "certificate": "-----BEGIN CERTIFICATE-----...", "is_authority": false, "private_key": "-----BEGIN RSA PRIVATE KEY-----..." } ``` *** ## Analytics Operations for querying analytics metrics. Detailed documentation: [Analytics Operations](/reference/v5/analytics/operations.md) | Operation | Description | Role Required | | ----------------- | ----------------------------------------------- | ------------- | | `get_analytics` | Retrieves analytics data for a specified metric | any | | `list_metrics` | Lists available analytics metrics | any | | `describe_metric` | Returns the schema of a specific metric | any | ### `get_analytics` Retrieves analytics data. Supports `start_time`/`end_time` (Unix ms), `get_attributes`, and `conditions` (same format as `search_by_conditions`). ``` { "operation": "get_analytics", "metric": "resource-usage", "start_time": 1769198332754, "end_time": 1769198532754 } ``` ### `list_metrics` Returns available metric names. Filter by `metric_types`: `custom`, `builtin` (default: `builtin`). ``` { "operation": "list_metrics" } ``` *** ## Registration & Licensing Operations for license management. | Operation | Description | Role Required | | ----------------------- | -------------------------------------------------- | ------------- | | `registration_info` | Returns registration and version information | any | | `install_usage_license` | Installs a Harper usage license block | super\_user | | `get_usage_licenses` | Returns all usage licenses with consumption counts | super\_user | | `get_fingerprint` | *(Deprecated)* Returns the machine fingerprint | super\_user | | `set_license` | *(Deprecated)* Sets a license key | super\_user | ### `registration_info` Returns the instance registration status, version, RAM allocation, and license expiration. ``` { "operation": "registration_info" } ``` ### `install_usage_license` Installs a usage license block. A license is a JWT-like structure (`header.payload.signature`) signed by Harper. Multiple blocks may be installed; earliest blocks are consumed first. ``` { "operation": "install_usage_license", "license": "abc...0123.abc...0123.abc...0123" } ``` ### `get_usage_licenses` Returns all usage licenses (including expired/exhausted) with current consumption counts. Optionally filter by `region`. ``` { "operation": "get_usage_licenses" } ``` *** ## Deprecated Operations The following operations are deprecated and should not be used in new code. ### Custom Functions (Deprecated) Custom Functions were the precursor to the Component architecture introduced in v4.2.0. These operations are preserved for backward compatibility. *Deprecated in: v4.2.0* (moved to legacy in v4.7+) For modern equivalents, see [Components Overview](/reference/v5/components/overview.md). | Operation | Description | | --------------------------------- | ------------------------------------------------ | | `custom_functions_status` | Returns Custom Functions server status | | `get_custom_functions` | Lists all Custom Function projects | | `get_custom_function` | Returns a Custom Function file's content | | `set_custom_function` | Creates or updates a Custom Function file | | `drop_custom_function` | Deletes a Custom Function file | | `add_custom_function_project` | Creates a new Custom Function project | | `drop_custom_function_project` | Deletes a Custom Function project | | `package_custom_function_project` | Packages a Custom Function project as base64 tar | | `deploy_custom_function_project` | Deploys a packaged Custom Function project | ### Other Deprecated Operations | Operation | Replaced By | | ---------------------- | ------------------------------------------------------------------- | | `install_node_modules` | Handled automatically by `deploy_component` and `restart` | | `get_fingerprint` | Use `registration_info` | | `set_license` | Use `install_usage_license` | | `search_by_hash` | Use `search_by_id` | | `search_attribute` | Use `attribute` field in `search_by_value` / `search_by_conditions` | | `search_value` | Use `value` field in `search_by_value` / `search_by_conditions` | | `search_type` | Use `comparator` field in `search_by_conditions` | --- # Operations API The Operations API provides a comprehensive set of capabilities for configuring, deploying, administering, and controlling Harper. It is the primary programmatic interface for all administrative and operational tasks that are not handled through the REST interface. ## Endpoint All Operations API requests are sent as HTTP POST requests to the Operations API endpoint. By default, this listens on port `9925` on the root path: ``` POST http://:9925/ ``` See [Configuration Overview](/reference/v5/configuration/overview.md) for how to change the port and other network settings (`operationsApi.network.port`, `operationsApi.network.securePort`). ## Request Format Each request body must be a JSON object with an `operation` field that identifies the operation to perform: ``` POST https://my-harper-server:9925/ Authorization: Basic YourBase64EncodedUser:Pass Content-Type: application/json { "operation": "create_table", "table": "my-table" } ``` ## Authentication Operations API requests must be authenticated. Harper supports two authentication methods: * **Basic Auth**: Base64-encoded `username:password` in the `Authorization` header. See [Basic Authentication](/reference/v5/security/basic-authentication.md). * **JWT**: A Bearer token in the `Authorization` header, obtained via `create_authentication_tokens`. See [JWT Authentication](/reference/v5/security/jwt-authentication.md). The `create_authentication_tokens` operation itself does not require prior authentication — it accepts a username and password and returns an operation token and refresh token. ## Example with curl ``` curl --location --request POST 'https://my-harper-server:9925/' \ --header 'Authorization: Basic YourBase64EncodedUser:Pass' \ --header 'Content-Type: application/json' \ --data-raw '{ "operation": "create_table", "table": "my-table" }' ``` ## Authorization Most operations are restricted to `super_user` roles. This is noted in the documentation for each operation. Some operations (such as `user_info`, `get_job`, and `create_authentication_tokens`) are available to all authenticated users. ## Operations Reference Operations are grouped by topic. See [Operations](/reference/v5/operations-api/operations.md) for the complete reference list. **Topic categories:** | Category | Description | Detailed Docs | | ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------- | | [Databases & Tables](/reference/v5/operations-api/operations.md#databases--tables) | Create and manage databases, tables, and attributes | [Database Overview](/reference/v5/database/overview.md) | | [NoSQL Operations](/reference/v5/operations-api/operations.md#nosql-operations) | Insert, update, upsert, delete, and query records | [REST Querying Reference](/reference/v5/rest/querying.md) | | [Bulk Operations](/reference/v5/operations-api/operations.md#bulk-operations) | CSV/S3 import and export, batch delete | [Database Jobs](/reference/v5/database/jobs.md) | | [SQL Operations](/reference/v5/operations-api/operations.md#sql-operations) | Execute SQL statements (use for investigation, not production) | — | | [Users & Roles](/reference/v5/operations-api/operations.md#users--roles) | Manage users and role-based access control | [Users & Roles Operations](/reference/v5/users-and-roles/operations.md) | | [Token Authentication](/reference/v5/operations-api/operations.md#token-authentication) | Create and refresh JWT tokens | [JWT Authentication](/reference/v5/security/jwt-authentication.md) | | [Components](/reference/v5/operations-api/operations.md#components) | Deploy and manage Harper components | [Components Overview](/reference/v5/components/overview.md) | | [Replication & Clustering](/reference/v5/operations-api/operations.md#replication--clustering) | Configure cluster topology and replication | [Replication & Clustering](/reference/v5/replication/clustering.md) | | [Configuration](/reference/v5/operations-api/operations.md#configuration) | Read and update Harper configuration | — | | [System](/reference/v5/operations-api/operations.md#system) | Restart, system information, status management | — | | [Jobs](/reference/v5/operations-api/operations.md#jobs) | Query background job status | [Database Jobs](/reference/v5/database/jobs.md) | | [Logs](/reference/v5/operations-api/operations.md#logs) | Read standard, transaction, and audit logs | [Logging Operations](/reference/v5/logging/operations.md) | | [Certificate Management](/reference/v5/operations-api/operations.md#certificate-management) | Manage TLS certificates | [Certificate Management](/reference/v5/security/certificate-management.md) | | [Analytics](/reference/v5/operations-api/operations.md#analytics) | Query analytics metrics | [Analytics Operations](/reference/v5/analytics/operations.md) | | [Registration & Licensing](/reference/v5/operations-api/operations.md#registration--licensing) | License management | — | ## Past Release API Documentation For API documentation prior to v4.0, see [olddocs.harperdb.io](https://olddocs.harperdb.io). --- # SQL warning SQL querying is not recommended for production use or on large tables. SQL queries often do not utilize indexes and are not optimized for performance. Use the [REST interface](/reference/v5/rest/overview.md) for production data access — it provides a more stable, secure, and performant interface. SQL is intended for ad-hoc data investigation and administrative queries. Harper includes a SQL interface supporting SELECT, INSERT, UPDATE, and DELETE operations. Tables are referenced using `database.table` notation (e.g., `dev.dog`). ## Operations API SQL queries are executed via the Operations API using the `sql` operation: * `operation` *(required)* — must be `sql` * `sql` *(required)* — the SQL statement to execute ### Select ``` { "operation": "sql", "sql": "SELECT * FROM dev.dog WHERE id = 1" } ``` ### Insert ``` { "operation": "sql", "sql": "INSERT INTO dev.dog (id, dog_name) VALUE (22, 'Simon')" } ``` Response: ``` { "message": "inserted 1 of 1 records", "inserted_hashes": [22], "skipped_hashes": [] } ``` ### Update ``` { "operation": "sql", "sql": "UPDATE dev.dog SET dog_name = 'penelope' WHERE id = 1" } ``` ### Delete ``` { "operation": "sql", "sql": "DELETE FROM dev.dog WHERE id = 1" } ``` *** ## SELECT Syntax ``` SELECT * FROM dev.dog SELECT id, dog_name, age FROM dev.dog SELECT * FROM dev.dog ORDER BY age SELECT * FROM dev.dog ORDER BY age DESC SELECT DISTINCT breed_id FROM dev.dog SELECT COUNT(*) FROM dev.dog WHERE age > 3 ``` ### Joins Supported join types: `INNER JOIN`, `LEFT [OUTER] JOIN`, `RIGHT [OUTER] JOIN`, `FULL OUTER JOIN`, `CROSS JOIN`. ``` SELECT d.id, d.dog_name, b.name FROM dev.dog AS d INNER JOIN dev.breed AS b ON d.breed_id = b.id WHERE d.owner_name IN ('Kyle', 'Zach') ORDER BY d.dog_name ``` *** ## Features Matrix | INSERT | | | ---------------------------------- | - | | Values — multiple values supported | ✔ | | Sub-SELECT | ✗ | | UPDATE | | | -------------- | - | | SET | ✔ | | Sub-SELECT | ✗ | | Conditions | ✔ | | Date Functions | ✔ | | Math Functions | ✔ | | DELETE | | | ---------- | - | | FROM | ✔ | | Sub-SELECT | ✗ | | Conditions | ✔ | | SELECT | | | ------------------- | - | | Column SELECT | ✔ | | Aliases | ✔ | | Aggregate Functions | ✔ | | Date Functions | ✔ | | Math Functions | ✔ | | Constant Values | ✔ | | DISTINCT | ✔ | | Sub-SELECT | ✗ | | FROM | | | ---------------- | - | | Multi-table JOIN | ✔ | | INNER JOIN | ✔ | | LEFT OUTER JOIN | ✔ | | LEFT INNER JOIN | ✔ | | RIGHT OUTER JOIN | ✔ | | RIGHT INNER JOIN | ✔ | | FULL JOIN | ✔ | | UNION | ✗ | | Sub-SELECT | ✗ | | TOP | ✔ | | WHERE | | | ---------------- | - | | Multi-Conditions | ✔ | | Wildcards | ✔ | | IN | ✔ | | LIKE | ✔ | | AND, OR, NOT | ✔ | | NULL | ✔ | | BETWEEN | ✔ | | EXISTS, ANY, ALL | ✔ | | Compare columns | ✔ | | Date Functions | ✔ | | Sub-SELECT | ✗ | | GROUP BY | | | --------------------- | - | | Multi-Column GROUP BY | ✔ | | HAVING | | | ----------------------------- | - | | Aggregate function conditions | ✔ | | ORDER BY | | | --------------------- | - | | Multi-Column ORDER BY | ✔ | | Aliases | ✔ | *** ## Functions ### Aggregate | Function | Description | | ---------------------- | ----------------------------------------------------- | | `AVG(expr)` | Average of a numeric expression. | | `COUNT(col)` | Count of rows matching the criteria (nulls excluded). | | `MAX(col)` | Largest value in a column. | | `MIN(col)` | Smallest value in a column. | | `SUM(col)` | Sum of numeric values. | | `GROUP_CONCAT(expr)` | Comma-separated string of non-null values. | | `ARRAY(expr)` | Returns a list of data as a field. | | `DISTINCT_ARRAY(expr)` | Returns a deduplicated list. | ### Conversion | Function | Description | | ---------------------------------- | ------------------------------------------ | | `CAST(expr AS datatype)` | Converts a value to the specified type. | | `CONVERT(datatype, expr[, style])` | Converts a value from one type to another. | ### String | Function | Description | | ----------------------------- | ------------------------------------------------------- | | `CONCAT(s1, s2, ...)` | Joins strings together. | | `CONCAT_WS(sep, s1, s2, ...)` | Joins strings with a separator. | | `INSTR(s1, s2)` | Position of s2 within s1. | | `LEN(s)` | Length of a string. | | `LOWER(s)` | Converts to lower-case. | | `UPPER(s)` | Converts to upper-case. | | `REPLACE(s, old, new)` | Replaces all instances of old with new. | | `SUBSTRING(s, pos, len)` | Extracts a substring. | | `TRIM([chars FROM] s)` | Removes leading and trailing spaces or specified chars. | | `REGEXP pattern` | Matches a regular expression pattern. | | `REGEXP_LIKE(col, pattern)` | Matches a regular expression pattern (function form). | ### Mathematical | Function | Description | | ------------------ | --------------------------------------- | | `ABS(expr)` | Absolute value. | | `CEIL(n)` | Smallest integer ≥ n. | | `FLOOR(n)` | Largest integer ≤ n. | | `EXP(n)` | e to the power of n. | | `ROUND(n, places)` | Rounds to the specified decimal places. | | `SQRT(expr)` | Square root. | | `RANDOM(seed)` | Pseudo-random number. | ### Logical | Function | Description | | -------------------------------- | ------------------------------------------------------- | | `IF(cond, true_val, false_val)` | Returns one of two values based on a condition. | | `IIF(cond, true_val, false_val)` | Alias for IF. | | `IFNULL(expr, alt)` | Returns alt if expr is null. | | `NULLIF(expr1, expr2)` | Returns null if expr1 = expr2, otherwise returns expr1. | *** ## Date & Time Functions All SQL date operations use UTC internally. Dates are parsed as [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601), then [RFC 2822](https://tools.ietf.org/html/rfc2822#section-3.3), then `new Date(string)`. | Function | Returns | | ------------------------------------- | ------------------------------------------------------------------------------------------------ | | `CURRENT_DATE()` | Current date as `YYYY-MM-DD`. | | `CURRENT_TIME()` | Current time as `HH:mm:ss.SSS`. | | `CURRENT_TIMESTAMP` | Current Unix timestamp in milliseconds. | | `NOW()` | Current Unix timestamp in milliseconds. | | `GETDATE()` | Current Unix timestamp in milliseconds. | | `GET_SERVER_TIME()` | Current date/time in server's timezone as `YYYY-MM-DDTHH:mm:ss.SSSZZ`. | | `DATE([date_string])` | Date formatted as `YYYY-MM-DDTHH:mm:ss.SSSZZ`. | | `DATE_ADD(date, value, interval)` | Adds time to a date; returns Unix ms. | | `DATE_SUB(date, value, interval)` | Subtracts time from a date; returns Unix ms. | | `DATE_DIFF(date1, date2[, interval])` | Difference between two dates. | | `DATE_FORMAT(date, format)` | Formats a date using [moment.js format strings](https://momentjs.com/docs/#/displaying/format/). | | `EXTRACT(date, date_part)` | Extracts a part (year, month, day, hour, minute, second, millisecond). | | `OFFSET_UTC(date, offset)` | Returns the date adjusted by offset minutes (or hours if < 16). | | `DAY(date)` | Day of the month. | | `DAYOFWEEK(date)` | Day of the week (0=Sunday … 6=Saturday). | | `HOUR(datetime)` | Hour part (0–838). | | `MINUTE(datetime)` | Minute part (0–59). | | `MONTH(date)` | Month (1–12). | | `SECOND(datetime)` | Seconds part (0–59). | | `YEAR(date)` | Year. | `DATE_ADD` and `DATE_SUB` accept these interval values: | Key | Shorthand | | ------------ | --------- | | years | y | | quarters | Q | | months | M | | weeks | w | | days | d | | hours | h | | minutes | m | | seconds | s | | milliseconds | ms | *** ## JSON Search `SEARCH_JSON(expression, attribute)` queries nested JSON data that is not indexed by Harper. It uses the [JSONata](https://docs.jsonata.org/overview.html) library and works in both SELECT and WHERE clauses. ``` -- Find records where the name array contains "Harper" SELECT * FROM dev.dog WHERE SEARCH_JSON('"Harper" in *', name) ``` ``` -- Select and filter nested JSON in one query SELECT m.title, SEARCH_JSON($[name in ["Actor A", "Actor B"]].{"actor": name}, c.`cast`) AS cast FROM movies.credits c INNER JOIN movies.movie m ON c.movie_id = m.id WHERE SEARCH_JSON($count($[name in ["Actor A", "Actor B"]]), c.`cast`) >= 2 ``` *** ## Geospatial Functions Geospatial data must be stored using the [GeoJSON standard](https://geojson.org/) in a single column. All coordinates are in `[longitude, latitude]` format. | Function | Description | | -------------------------------------------- | ------------------------------------------------------------------ | | `geoArea(geoJSON)` | Area of features in square meters. | | `geoLength(geoJSON[, units])` | Length in km (default), or degrees/radians/miles. | | `geoDistance(point1, point2[, units])` | Distance between two points. | | `geoNear(point1, point2, distance[, units])` | Returns boolean: true if points are within the specified distance. | | `geoContains(geo1, geo2)` | Returns boolean: true if geo2 is completely contained by geo1. | | `geoDifference(polygon1, polygon2)` | Returns a new polygon with polygon2 clipped from polygon1. | | `geoEqual(geo1, geo2)` | Returns boolean: true if two GeoJSON features are identical. | | `geoCrosses(geo1, geo2)` | Returns boolean: true if the geometries cross each other. | | `geoConvert(coordinates, geo_type[, props])` | Converts coordinates into a GeoJSON of the specified type. | `units` options: `'degrees'`, `'radians'`, `'miles'`, `'kilometers'` (default). `geo_type` options for `geoConvert`: `'point'`, `'lineString'`, `'multiLineString'`, `'multiPoint'`, `'multiPolygon'`, `'polygon'`. *** ## Logical Operators | Keyword | Description | | --------- | ------------------------------------------------ | | `BETWEEN` | Returns values within a given range (inclusive). | | `IN` | Specifies multiple values in a WHERE clause. | | `LIKE` | Searches for a pattern. | *** ## Reserved Words If a database, table, or attribute name conflicts with a reserved word, wrap it in backticks or brackets: ``` SELECT * FROM data.`ASSERT` SELECT * FROM data.[ASSERT] ``` Full reserved word list ABSOLUTE, ACTION, ADD, AGGR, ALL, ALTER, AND, ANTI, ANY, APPLY, ARRAY, AS, ASSERT, ASC, ATTACH, AUTOINCREMENT, AUTO\_INCREMENT, AVG, BEGIN, BETWEEN, BREAK, BY, CALL, CASE, CAST, CHECK, CLASS, CLOSE, COLLATE, COLUMN, COLUMNS, COMMIT, CONSTRAINT, CONTENT, CONTINUE, CONVERT, CORRESPONDING, COUNT, CREATE, CROSS, CUBE, CURRENT\_TIMESTAMP, CURSOR, DATABASE, DECLARE, DEFAULT, DELETE, DELETED, DESC, DETACH, DISTINCT, DOUBLEPRECISION, DROP, ECHO, EDGE, END, ENUM, ELSE, EXCEPT, EXISTS, EXPLAIN, FALSE, FETCH, FIRST, FOREIGN, FROM, GO, GRAPH, GROUP, GROUPING, HAVING, HDB\_HASH, HELP, IF, IDENTITY, IS, IN, INDEX, INNER, INSERT, INSERTED, INTERSECT, INTO, JOIN, KEY, LAST, LET, LEFT, LIKE, LIMIT, LOOP, MATCHED, MATRIX, MAX, MERGE, MIN, MINUS, MODIFY, NATURAL, NEXT, NEW, NOCASE, NO, NOT, NULL, OFF, ON, ONLY, OFFSET, OPEN, OPTION, OR, ORDER, OUTER, OVER, PATH, PARTITION, PERCENT, PLAN, PRIMARY, PRINT, PRIOR, QUERY, READ, RECORDSET, REDUCE, REFERENCES, RELATIVE, REPLACE, REMOVE, RENAME, REQUIRE, RESTORE, RETURN, RETURNS, RIGHT, ROLLBACK, ROLLUP, ROW, SCHEMA, SCHEMAS, SEARCH, SELECT, SEMI, SET, SETS, SHOW, SOME, SOURCE, STRATEGY, STORE, SYSTEM, SUM, TABLE, TABLES, TARGET, TEMP, TEMPORARY, TEXTSTRING, THEN, TIMEOUT, TO, TOP, TRAN, TRANSACTION, TRIGGER, TRUE, TRUNCATE, UNION, UNIQUE, UPDATE, USE, USING, VALUE, VERTEX, VIEW, WHEN, WHERE, WHILE, WITH, WORK --- # Clustering Operations API for managing Harper's replication system. For an overview of how replication works, see [Replication Overview](/reference/v5/replication/overview.md). For sharding configuration, see [Sharding](/reference/v5/replication/sharding.md). All clustering operations require `super_user` role. *** ### Add Node Adds a new Harper instance to the cluster. If `subscriptions` are provided, it creates the specified replication relationships between the nodes. Without `subscriptions`, a fully replicating system is created (all data in all databases). **Parameters**: * `operation` *(required)* — must be `add_node` * `hostname` or `url` *(required)* — the hostname or URL of the node to add * `verify_tls` *(optional)* — whether to verify the TLS certificate. Set to `false` temporarily on fresh installs with self-signed certificates. Defaults to `true` * `authorization` *(optional)* — credentials for the node being added. Either an object with `username` and `password`, or an HTTP `Authorization` style string * `retain_authorization` *(optional)* — if `true`, stores credentials and uses them on every reconnect. Generally not recommended; prefer certificate-based authentication. Defaults to `false` * `revoked_certificates` *(optional)* — array of revoked certificate serial numbers that will not be accepted for any connections * `shard` *(optional)* — shard number for this node. Only needed when using sharding * `start_time` *(optional)* — ISO 8601 UTC datetime. If set, only data after this time is downloaded during initial synchronization instead of the entire database * `subscriptions` *(optional)* — explicit table-level replication relationships. This is optional (and discouraged). Each subscription is an object with: * `database` — database name * `table` — table name * `subscribe` — if `true`, transactions on the remote table are replicated locally * `publish` — if `true`, transactions on the local table are replicated to the remote node **Request**: ``` { "operation": "add_node", "hostname": "server-two", "verify_tls": false, "authorization": { "username": "admin", "password": "password" } } ``` **Response**: ``` { "message": "Successfully added 'server-two' to cluster" } ``` > **Note**: `set_node` is an alias for `add_node`. *** ### Update Node Modifies an existing Harper instance in the cluster. Will attempt to add the node if it does not exist. **Parameters**: * `operation` *(required)* — must be `update_node` * `hostname` *(required)* — hostname of the remote node to update * `revoked_certificates` *(optional)* — array of revoked certificate serial numbers * `shard` *(optional)* — shard number to assign to this node * `subscriptions` *(required)* — array of subscription objects (same structure as `add_node`) **Request**: ``` { "operation": "update_node", "hostname": "server-two" } ``` **Response**: ``` { "message": "Successfully updated 'server-two'" } ``` *** ### Remove Node Removes a Harper node from the cluster and stops all replication to and from that node. **Parameters**: * `operation` *(required)* — must be `remove_node` * `hostname` *(required)* — hostname of the node to remove **Request**: ``` { "operation": "remove_node", "hostname": "server-two" } ``` **Response**: ``` { "message": "Successfully removed 'server-two' from cluster" } ``` *** ### Cluster Status Returns an array of status objects from the cluster, including active WebSocket connections and replication timing statistics. *Added in: v4.4.0* ; timing statistics added in v4.5.0 **Parameters**: * `operation` *(required)* — must be `cluster_status` **Request**: ``` { "operation": "cluster_status" } ``` **Response**: ``` { "type": "cluster-status", "connections": [ { "replicateByDefault": true, "replicates": true, "url": "wss://server-2.domain.com:9933", "name": "server-2.domain.com", "subscriptions": null, "database_sockets": [ { "database": "data", "connected": true, "latency": 0.7, "thread_id": 1, "nodes": ["server-2.domain.com"], "lastCommitConfirmed": "Wed, 12 Feb 2025 19:09:34 GMT", "lastReceivedRemoteTime": "Wed, 12 Feb 2025 16:49:29 GMT", "lastReceivedLocalTime": "Wed, 12 Feb 2025 16:50:59 GMT", "lastSendTime": "Wed, 12 Feb 2025 16:50:59 GMT" } ] } ], "node_name": "server-1.domain.com", "is_enabled": true } ``` `database_sockets` shows the actual WebSocket connections between nodes — one socket per database per node. Timing fields: | Field | Description | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------- | | `lastCommitConfirmed` | Last time a receipt of confirmation was received for an outgoing commit | | `lastReceivedRemoteTime` | Timestamp (from the originating node) of the last received transaction | | `lastReceivedLocalTime` | Local time when the last transaction was received. A gap between this and `lastReceivedRemoteTime` suggests the node is catching up | | `sendingMessage` | Timestamp of the transaction actively being sent. Absent when waiting for the next transaction | *** ### Configure Cluster Bulk creates or resets subscriptions for any number of remote nodes. **Resets and replaces any existing clustering setup.** **Parameters**: * `operation` *(required)* — must be `configure_cluster` * `connections` *(required)* — array of node objects following the `add_node` schema **Request**: ``` { "operation": "configure_cluster", "connections": [ { "hostname": "server-two", "verify_tls": false, "authorization": { "username": "admin", "password": "password2" } }, { "hostname": "server-three", "verify_tls": false, "authorization": { "username": "admin", "password": "password3" } } ] } ``` **Response**: ``` { "message": "Cluster successfully configured." } ``` *** ### Cluster Set Routes Adds routes to the `replication.routes` configuration. Behaves as a PATCH/upsert — adds new routes while leaving existing routes untouched. **Parameters**: * `operation` *(required)* — must be `cluster_set_routes` * `routes` *(required)* — array of route strings (`wss://host:port`) or objects with `hostname` and `port` properties **Request**: ``` { "operation": "cluster_set_routes", "routes": [ "wss://server-two:9925", { "hostname": "server-three", "port": 9930 } ] } ``` **Response**: ``` { "message": "cluster routes successfully set", "set": ["wss://server-two:9925", { "hostname": "server-three", "port": 9930 }], "skipped": [] } ``` *** ### Cluster Get Routes Returns the replication routes from the Harper config file. **Parameters**: * `operation` *(required)* — must be `cluster_get_routes` **Request**: ``` { "operation": "cluster_get_routes" } ``` **Response**: ``` ["wss://server-two:9925", { "hostname": "server-three", "port": 9930 }] ``` *** ### Cluster Delete Routes Removes routes from the Harper config file. **Parameters**: * `operation` *(required)* — must be `cluster_delete_routes` * `routes` *(required)* — array of route objects to remove **Request**: ``` { "operation": "cluster_delete_routes", "routes": [ { "hostname": "server-three", "port": 9930 } ] } ``` **Response**: ``` { "message": "cluster routes successfully deleted", "deleted": [{ "hostname": "server-three", "port": 9930 }], "skipped": [] } ``` --- # Replication Overview Harper's replication system is designed to make distributed data replication fast and reliable across multiple nodes. You can build a distributed database that ensures high availability, disaster recovery, and data localization — all without complex setup. Nodes can be added or removed dynamically, you can choose which data to replicate, and you can monitor cluster health without jumping through hoops. ## Peer-to-Peer Model Harper replication uses a peer-to-peer model where every node in your cluster can send data to and receive data from other nodes. Nodes communicate over WebSockets, allowing data to flow in both directions. Harper automatically manages these connections and subscriptions, so you don't need to manually track data consistency. Connections between nodes are secured and reliable by default. ## Configuration ### Connecting Nodes To connect nodes to each other, provide hostnames or URLs in the `replication` section of `harper-config.yaml`. Each node specifies its own hostname and the routes (other nodes) it should connect to: ``` replication: hostname: server-one routes: - server-two - server-three ``` Routes can also be specified as URLs or with explicit port numbers: ``` replication: hostname: server-one routes: - wss://server-two:9933 - hostname: server-three port: 9933 ``` By default, replication connects on the secure port `9933`. ``` replication: securePort: 9933 ``` You can also manage nodes dynamically through the [Operations API](/reference/v5/replication/clustering.md) without editing the config file. ### Gossip Discovery Harper automatically replicates node information to other nodes in the cluster using [gossip-style discovery](https://highscalability.com/gossip-protocol-explained/). This means you only need to connect to one existing node in a cluster, and Harper will automatically detect and connect to all other nodes bidirectionally. ### Data Selection By default, Harper replicates all data in all databases. You can narrow replication to specific databases: ``` replication: databases: - data - system ``` All tables within a replicated database are replicated by default. To exclude a specific table from replication, set `replicate: false` in the table definition: ``` type LocalTableForNode @table(replicate: false) { id: ID! name: String! } ``` Transactions are replicated atomically, which may span multiple tables. You can also control how many nodes data is replicated to using [sharding configuration](/reference/v5/replication/sharding.md). ## Securing Connections Harper supports PKI-based security and authorization for replication connections. Two authentication methods are supported: * **Certificate-based authentication** (recommended for production): Nodes are identified by the certificate's common name (CN) or Subject Alternative Names (SANs). * **IP-based authentication** (for development/testing): Nodes are identified by IP address when using insecure connections. Harper can automatically perform CRL (Certificate Revocation List) and OCSP (Online Certificate Status Protocol) verification to ensure revoked certificates cannot be used. OCSP and CRL work automatically with certificates from public CAs when `enableRootCAs` is enabled. For self-signed certificates or private CAs without OCSP/CRL support, use Harper's manual certificate revocation feature. Certificate verification settings follow the same configuration as HTTP mTLS connections (see [Certificate Verification](/reference/v5/security/certificate-verification.md)). ### Providing Your Own Certificates If you have certificates from a public or corporate CA, enable `enableRootCAs` so nodes validate against the standard root CA list: ``` replication: enableRootCAs: true ``` Ensure the certificate's CN matches the node's hostname. ### Setting Up Custom Certificates There are two ways to configure Harper with your own certificates: 1. Use the `add_certificate` operation to upload them. 2. Specify certificate paths directly in `harper-config.yaml`: ``` tls: certificate: /path/to/certificate.pem certificateAuthority: /path/to/ca.pem privateKey: /path/to/privateKey.pem ``` Harper will load the provided certificates into the certificate table and use them to secure and authenticate connections. If you have a publicly-signed certificate, you can omit the `certificateAuthority` and enable `enableRootCAs` to use the bundled Mozilla CA store instead. ### Cross-Generated Certificates Harper can generate its own certificates for secure connections — useful when no existing certificates are available. When you run `add_node` over SSL with temporary credentials, Harper automatically handles certificate generation and signing: ``` { "operation": "add_node", "hostname": "server-two", "verify_tls": false, "authorization": { "username": "admin", "password": "password" } } ``` On a fresh install, set `verify_tls: false` temporarily to accept the self-signed certificate. Harper then: 1. Creates a certificate signing request (CSR) and sends it to `server-two`. 2. `server-two` signs the CSR and returns the signed certificate and CA. 3. The signed certificate is stored for all future connections. Credentials are not stored — they are discarded immediately after use. You can also provide credentials in HTTP Authorization format (Basic, Token, or JWT). ### Revoking Certificates *Added in: v4.5.0* Certificates used in replication can be revoked using the certificate serial number. Use either the `revoked_certificates` attribute in the `hdb_nodes` system table or route config: Via the operations API: ``` { "operation": "update_node", "hostname": "server-two", "revoked_certificates": ["1769F7D6A"] } ``` Via `harper-config.yaml`: ``` replication: routes: - hostname: server-three port: 9930 revokedCertificates: - 1769F7D6A - QA69C7E2S ``` ### Insecure IP-Based Authentication For development, testing, or secure private networks, you can disable TLS and use IP addresses to authenticate nodes. Configure replication on an insecure port and set up IP-based routes: ``` replication: port: 9933 routes: - 127.0.0.2 - 127.0.0.3 ``` > **Warning**: Never use insecure connections for production systems accessible from the public internet. Loopback addresses (`127.0.0.X`) are a convenient way to run multiple nodes on a single machine for local development. ## Controlling Replication Flow By default, Harper replicates all data in all databases with symmetric bidirectional flow. To restrict replication to one direction between certain nodes, set `sends` and `receives` on the route configuration: ``` replication: databases: - data routes: - hostname: node-two replicates: sends: false receives: true - hostname: node-three replicates: sends: true receives: false ``` In this example, the local node only receives from `node-two` (one-way inbound) and only sends to `node-three` (one-way outbound). > **Note**: When using controlled flow replication, avoid replicating the `system` database. The `system` database contains node configurations, so replicating it would cause all nodes to have identical (and incorrect) route configurations. ### Explicit Subscriptions By default, Harper automatically manages connections and subscriptions between nodes. Explicit subscriptions exist only for testing, debugging, and legacy migration — they should not be used for production replication and will likely be removed in v5. With explicit subscriptions, Harper no longer guarantees data consistency. If you want unidirectional replication, use [controlled replication flow](#controlling-replication-flow) instead. To explicitly subscribe, use `add_node` with subscription definitions: ``` { "operation": "add_node", "hostname": "server-two", "subscriptions": [ { "database": "dev", "table": "my-table", "publish": true, "subscribe": false } ] } ``` Update a subscription with `update_node`: ``` { "operation": "update_node", "hostname": "server-two", "subscriptions": [ { "database": "dev", "table": "my-table", "publish": true, "subscribe": true } ] } ``` ## Monitoring Replication *Added in: v4.5.0* (cluster status timing statistics) Use `cluster_status` to monitor the state of replication: ``` { "operation": "cluster_status" } ``` See [Clustering Operations](/reference/v5/replication/clustering.md#cluster-status) for the full response schema and field descriptions. ## Initial Synchronization and Resynchronization When a new node is added and its database has not been previously synced, Harper downloads the full database from the first node it connects to. After the initial sync completes, the node enters replication mode and receives incremental updates. If a node goes offline and comes back, it resynchronizes automatically to catch up on missed transactions. You can also specify a `start_time` in the `add_node` operation to limit the initial download to data since a given point in time: ``` { "operation": "add_node", "hostname": "server-two", "start_time": "2024-01-01T00:00:00.000Z" } ``` ## Replicated Transactions The following data operations are replicated across the cluster: * Insert * Update * Upsert * Delete * Bulk loads (CSV data load, CSV file load, CSV URL load, import from S3) **Destructive schema operations are not replicated**: `drop_database`, `drop_table`, and `drop_attribute` must be run on each node independently. Users and roles are not replicated across the cluster. Certain management operations — including component deployment and rolling restarts — can also be replicated across the cluster. ## Inspecting Cluster Configuration Query the `hdb_nodes` system table to inspect the current known nodes and their configuration: ``` { "operation": "search_by_value", "database": "system", "table": "hdb_nodes", "attribute": "name", "value": "*" } ``` The `hdb_certificate` table contains the certificates used for replication connections. ## See Also * [Clustering Operations](/reference/v5/replication/clustering.md) — Operations API for managing cluster nodes and subscriptions * [Sharding](/reference/v5/replication/sharding.md) — Distributing data across a subset of nodes * [Certificate Management](/reference/v5/security/certificate-management.md) --- # Sharding *Added in: v4.4.0* (provisional) *Changed in: v4.5.0* — expanded sharding functionality: Harper now honors write requests with residency information that will not be stored on the local node, and nodes can be declaratively configured as part of a shard. Harper's replication system supports sharding — storing different data across different subsets of nodes — while still allowing data to be accessed from any node in the cluster. This enables horizontal scalability for storage and write performance, while maintaining optimal data locality and consistency. When sharding is configured, requests for records that don't reside on the handling node are automatically forwarded to the appropriate node transparently. Clients do not need to know where data is stored. By default (without sharding), Harper replicates all data to all nodes. ## Approaches to Sharding There are two main approaches: **Dynamic sharding** — the location (residency) of records is determined dynamically based on where the record was written, the record's data, or a custom function. Records can be relocated dynamically based on where they are accessed. Residency information is specific to each record. **Static sharding** — each node is assigned to a specific numbered shard, and each record is replicated to the nodes in that shard based on the primary key, regardless of where the data was written or accessed. More predictable than dynamic sharding: data location is always determinable from the primary key. ## Dynamic Sharding ### Replication Count The simplest way to limit replication is to configure a replication count. Set `replicateTo` in the `replication` section of `harper-config.yaml` to specify how many additional nodes data should be replicated to: ``` replication: replicateTo: 2 ``` This ensures each record is stored on three nodes total (the node that first stored it, plus two others). ### Replication Control via REST Header With the REST interface, you can specify replication targets and confirmation requirements per request using the `X-Replicate-To` header: ``` PUT /MyTable/3 X-Replicate-To: 2;confirm=1 ``` * `2` — replicate to two additional nodes * `confirm=1` — wait for confirmation from one additional node before responding Specify exact destination nodes by hostname: ``` PUT /MyTable/3 X-Replicate-To: node1,node2 ``` The `confirm` parameter can be combined with explicit node lists. ### Replication Control via Operations API Specify `replicateTo` and `replicatedConfirmation` in the operation body: ``` { "operation": "update", "schema": "dev", "table": "MyTable", "hashValues": [3], "record": { "name": "John Doe" }, "replicateTo": 2, "replicatedConfirmation": 1 } ``` Or specify explicit nodes: ``` { // ... "replicateTo": ["node-1", "node-2"], // ... } ``` ### Programmatic Replication Control Set `replicateTo` and `replicatedConfirmation` programmatically in a resource method: ``` class MyTable extends tables.MyTable { put(record) { const context = this.getContext(); context.replicateTo = 2; // or an array of node names context.replicatedConfirmation = 1; return super.put(record); } } ``` ## Static Sharding ### Basic Static Shard Configuration Assign a node to a numbered shard in `harper-config.yaml`: ``` replication: shard: 1 ``` Or assign shards per route: ``` replication: routes: - hostname: node1 shard: 1 - hostname: node2 shard: 2 ``` Or dynamically via the operations API by including `shard` in an `add_node` or `set_node` operation: ``` { "operation": "add_node", "hostname": "node1", "shard": 1 } ``` Once shards are configured, use `setResidency` or `setResidencyById` (described below) to assign records to specific shards. ## Custom Sharding ### By Record Content (`setResidency`) Define a custom residency function that is called with the full record. Return an array of node hostnames or a shard number. With this approach, record metadata (including residency information) and indexed properties are replicated to all nodes, but the full record is only stored on the specified nodes. Return node hostnames: ``` MyTable.setResidency((record) => { return record.id % 2 === 0 ? ['node1'] : ['node2']; }); ``` Return a shard number (replicates to all nodes in that shard): ``` MyTable.setResidency((record) => { return record.id % 2 === 0 ? 1 : 2; }); ``` ### By Primary Key Only (`setResidencyById`) Define a residency function based solely on the primary key. Records (including metadata) are only replicated to the specified nodes — metadata does not need to be replicated everywhere, which allows data to be retrieved without needing access to record data or metadata on the requesting node. Return a shard number: ``` MyTable.setResidencyById((id) => { return id % 2 === 0 ? 1 : 2; }); ``` Return node hostnames: ``` MyTable.setResidencyById((id) => { return id % 2 === 0 ? ['node1'] : ['node2']; }); ``` ## Disabling Cross-Node Access By default, sharding allows data stored on specific nodes to be accessed from any node — requests are forwarded transparently. To disable this and only return data if it is stored on the local node, set `replicateFrom` to `false`. Via the operations API: ``` { "operation": "search_by_id", "table": "MyTable", "ids": [3], "replicateFrom": false } ``` Via the REST API: ``` GET /MyTable/3 X-Replicate-From: none ``` ## See Also * [Replication Overview](/reference/v5/replication/overview.md) — How Harper's replication system works * [Clustering Operations](/reference/v5/replication/clustering.md) — Operations API for managing cluster nodes --- # Resources Harper's Resource API is the foundation for building custom data access logic and connecting data sources. Resources are JavaScript classes that define how data is accessed, modified, subscribed to, and served over HTTP, MQTT, and WebSocket protocols. ## What Is a Resource? A **Resource** is a class that provides a unified interface for a set of records or entities. Harper's built-in tables extend the base `Resource` class, and you can extend either `Resource` or a table class to implement custom behavior for any data source — internal or external. *Added in: v4.2.0* The Resource API is designed to mirror REST/HTTP semantics: methods map directly to HTTP verbs (`get`, `put`, `patch`, `post`, `delete`), making it straightforward to build API endpoints alongside custom data logic. ## Relationship to Other Features * **Database tables** extend `Resource` automatically. You can use tables through the Resource API without writing any custom code. * The **REST plugin** maps incoming HTTP requests to Resource methods. See [REST Overview](/reference/v5/rest/overview.md). * The **MQTT plugin** routes publish/subscribe messages to `publish` and `subscribe` Resource methods. See [MQTT Overview](/reference/v5/mqtt/overview.md). * **Global APIs** (`tables`, `databases`, `transaction`) provide access to resources from JavaScript code. * The **`jsResource` plugin** (configured in `config.yaml`) registers a JavaScript file's exported Resource classes as endpoints. ## Extending a Table The most common use case is extending an existing table to add custom logic. Starting with a table definition in a `schema.graphql`: ``` # Omit the `@export` directive type MyTable @table { id: Long @primaryKey # ... } ``` `@export` on the schema type registers Harper's default table resource at `/MyTable`. When you extend the table in JavaScript and want your subclass to serve those endpoints instead, **omit `@export`** from the schema and let the exported JavaScript class own the URL. Leaving `@export` on the schema while also exporting a subclass with the same name produces conflicting endpoints. When overriding handlers, call `super.get/post/...` to preserve Harper's default behavior unless you intend to replace it entirely. > For more info on the schema API see [`Database / Schema`](/reference/v5/database/schema.md) Then, in a `resources.js` extend from the `tables.MyTable` global: ``` export class MyTable extends tables.MyTable { static async get(target) { // get the record from the database const record = await super.get(target); // add a computed property before returning return { ...record, computedField: 'value' }; } static async post(target, data) { // custom action on POST this.create({ ...(await data), status: 'pending' }); } } ``` When delegating to `super`, match the argument form to the operation: a collection create passes the collection target and the record (`super.post(target, record)` — the target identifies the collection and carries no id, since the primary key is auto-generated), while updates pass the target and data (`super.put(target, data)` / `super.patch(target, data)`) and reads/deletes pass the target (`super.get(target)` / `super.delete(target)`). To return a specific HTTP status from a thrown error, set `.statusCode` (e.g. `400`) on the error object — a plain `.status` property is ignored: ``` const error = new Error('Name is required'); error.statusCode = 400; // use statusCode, NOT status throw error; ``` Finally, ensure everything is configured appropriately: ``` rest: true graphqlSchema: files: schema.graphql jsResource: files: resources.js ``` ## Custom External Data Source You can also extend the base `Resource` class directly to implement custom endpoints, or even wrap an external API or service as a custom caching layer: ``` export class CustomEndpoint extends Resource { static get(target) { return { data: doSomething(), }; } } export class MyExternalData extends Resource { static async get(target) { const response = await fetch(`https://api.example.com/${target.id}`); return response.json(); } static async put(target, data) { return fetch(`https://api.example.com/${target.id}`, { method: 'PUT', body: JSON.stringify(await data), }); } } // Use as a cache source for a local table tables.MyCache.sourcedFrom(MyExternalData); ``` Resources are the true customization point for Harper. This is where the business logic of a Harper application really lives. There is a lot more to this API than these examples show. Ensure you fully review the [Resource API](/reference/v5/resources/resource-api.md) documentation, and consider exploring the Learn guides for more information. ## Exporting Resources as Endpoints Resources become HTTP/MQTT endpoints when they are exported. As the examples demonstrated if a Resource extends an existing table, make sure to not have conflicting exports between the schema and the JavaScript implementation. Alternatively, you can register resources programmatically using `server.resources.set()`. See [HTTP API](/reference/v5/http/api.md) for `server` API documentation. The shape of the export controls the resulting URL: | Export form | URL | Notes | | ------------------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------ | | `export class Foo extends Resource {}` | `/Foo/` | The class name becomes the path segment. Path segments are case-sensitive. | | `export const Bar = { Foo };` | `/Bar/Foo/` | Nest a class under an object to add a path prefix. | | `export const bar = { 'foo-baz': Foo };` | `/bar/foo-baz/` | Use object keys when you need lowercase, hyphens, or any non-identifier URL. | | `export { Foo as '/widget/:id' }` | `/widget/:id` | Rename the export to set the path directly, including a leading-slash top-level path. See [Path Parameters](#path-parameters). | | `static path = '/widget/:id'` (class field) | `/widget/:id` | Declare the path on the class itself; overrides the export name. See [Path Parameters](#path-parameters). | | `server.resources.set('my-path', Foo);` | `/my-path/` | Programmatic registration; useful when the path is dynamic. | URL path matching is case-sensitive — `/Foo/` and `/foo/` are different endpoints. ### Path Parameters *Added in: v5.1.13* A resource's path can declare dynamic segments. A `:name` segment matches a single path segment, and a `*name` segment is a catch-all that matches the rest of the path. Matched values are bound onto the request target so a handler reads them as `target.`: ``` export class Widget extends Resource { // GET /widget/10/action/jump -> target.id === '10', target.action === 'jump' static path = '/widget/:id/action/:action'; static get(target) { return { id: target.id, action: target.action }; } } export class Files extends Resource { // GET /files/a/b/c.txt -> target.rest === 'a/b/c.txt' static path = '/files/*rest'; static get(target) { return { path: target.rest }; } } ``` A bare `*` (no name) binds under `target.wildcard`. A wildcard must be the final segment of the path. **Declaring the path.** The path can come from a `static path` field on the class (the recommended lever, since it leaves the exports untouched) or from the export name (`export { Widget as '/widget/:id' }`). A `static path` takes precedence over the export name. In both forms: * A leading `/` makes the path **root-relative** (top-level), independent of the file's location — useful for fixed top-level routes such as `static path = '/.well-known/acme-challenge/:token'`. * A leading `./` or a bare name resolves **relative to the component directory**, the same as the default export-name behavior. **Resolution order.** Exact and static paths always win over parameterized ones — `/resource/admin` is matched by a `static`-pathed `/resource/admin` resource even when a `/resource/:id` resource is also registered. Among parameterized routes, more specific paths win: a literal segment beats a `:param`, which beats a `*` wildcard, compared left to right. Parameterized routes also surface in the generated OpenAPI document (as templated paths like `/widget/{id}/action/{action}`) and in MCP `resources/templates/list` (as `{param}` URI templates). ## Pages in This Section | Page | Description | | ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | [Resource API](/reference/v5/resources/resource-api.md) | Complete reference for instance methods, static methods, the Query object, RequestTarget, and response handling | | [Query Optimization](/reference/v5/resources/query-optimization.md) | How Harper executes queries and how to write performant conditions | --- # Query Optimization *Added in: v4.3.0* (query planning and execution improvements) Harper has powerful query functionality with excellent performance characteristics. Like any database, different queries can vary significantly in performance. Understanding how querying works helps you write queries that perform well as your dataset grows. ## Query Execution At a fundamental level, querying involves defining conditions to find matching data and then executing those conditions against the database. Harper supports indexed fields, and these indexes are used to speed up query execution. When conditions are specified in a query, Harper attempts to utilize indexes to optimize the speed of query execution. When a field is not indexed, Harper checks each potential record to determine if it matches the condition — this is a full table scan and degrades as data grows (`O(n)`). When a query has multiple conditions, Harper attempts to optimize their execution order. For intersecting conditions (the default `and` operator), Harper applies the most selective and performant condition first. If one condition can use an index and is more selective than another, it is used first to narrow the candidate set before filtering on the remaining conditions. The `search` method supports an `explain` flag that returns the query execution order Harper determined, useful for debugging and optimization: ``` const result = await MyTable.search({ conditions: [...], explain: true, }); ``` For union queries (`or` operator), each condition is executed separately and the results are merged. ## Conditions, Operators, and Indexing When a query is executed, conditions are evaluated against the database. Indexed fields significantly improve query performance. ### Index Performance Characteristics | Operator | Uses index | Notes | | -------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------ | | `equals` | Yes | Fast lookup in sorted index | | `greater_than`, `greater_than_equal`, `less_than`, `less_than_equal` | Yes | Range scan in sorted index; narrower range = faster | | `starts_with` | Yes | Prefix search in sorted index | | `not_equal` | No | Full scan required (unless combined with selective indexed condition) | | `contains` | No | Full scan required | | `ends_with` | No | Full scan required | | `!= null` | Yes (special case) | Can use indexes to find non-null records; only helpful for sparse fields | **Rule of thumb**: Use `equals`, range operators, and `starts_with` on indexed fields. Avoid `contains`, `ends_with`, and `not_equal` as the sole or first condition in large datasets. ### Indexed vs. Non-Indexed Fields Indexed fields provide `O(log n)` lookup — fast even as the dataset grows. Non-indexed fields require `O(n)` full table scans. Trade-off: indexes speed up reads but add overhead to writes (insert/update/delete must update the index). This is usually worth it for frequently queried fields. ### Primary Key vs. Secondary Index Querying on a **primary key** is faster than querying on a secondary (non-primary) index, because the primary key directly addresses the record without cross-referencing. Secondary indexes are still valuable for query conditions on other fields, but expect slightly more overhead than primary key lookups. ### Cardinality More unique values (higher cardinality) = more efficient indexed lookups. For example, an index on a boolean field has very low cardinality (only two possible values) and is less efficient than an index on a `UUID` field. High-cardinality fields benefit most from indexing. ### Querying an Attribute Whose Index Is Still Building *Added in: v5.1.5* When a secondary index is still being built — for example, just after a new `@indexed` attribute is added, or while an index rebuilds following a schema change or upgrade — a query that relies on that index cannot return correct results yet. Rather than silently returning a partial result set, Harper rejects the query with a retryable `503`: | Property | Value | | --------- | --------------------------------------------------------------------- | | Status | `503` | | Code | `INDEX_REBUILDING` | | Retryable | `true` | | Message | `"" is not indexed yet, can not search for this attribute` | This is a transient condition — the index finishes building in the background, after which the same query succeeds. Clients should retry (with backoff) rather than treating it as an empty result or a permanent failure. On the operations API the error body includes `code` and `retryable`, so callers can branch on `error.code === 'INDEX_REBUILDING'` or `error.retryable`. ## Relationships and Joins Harper supports relationship-based queries that join data across tables. See [Schema documentation](/reference/v5/database/schema.md) for how to define relationships. Join queries involve more lookups and naturally carry more overhead. The same indexing principles apply: * Conditions on joined table fields should use indexed columns for best performance. * If a relationship uses a foreign key, that foreign key should be indexed in both tables. * Higher cardinality foreign keys make joins more efficient. Example of an indexed foreign key that enables efficient join queries: ``` type Product @table { id: Long @primaryKey brandId: Long @indexed # foreign key — index this brand: Related @relationship(from: "brandId") } type Brand @table { id: Long @primaryKey name: String @indexed # indexed — enables efficient brand.name queries products: Product @relationship(to: "brandId") } ``` *Added in: v4.3.0* ## Sorting Sorting can significantly impact query performance. * **Aligned sort and index**: If the sort attribute is the same indexed field used in the primary condition, Harper can use the index to retrieve results already in order — very fast. * **Unaligned sort**: If the sort is on a different field than the condition, or the sort field is not indexed, Harper must retrieve and sort all matching records. For large result sets this can be slow, and it also **defeats streaming** (see below). Best practice: sort on the same indexed field you are filtering on, or sort on a secondary indexed field with a narrow enough condition to produce a manageable result set. ## Streaming Harper can stream query results — returning records as they are found rather than waiting for the entire query to complete. This improves time-to-first-byte for large queries and reduces peak memory usage. **Streaming is defeated** when: * A sort order is specified that is not aligned with the condition's index * The full result set must be materialized to perform sorting When streaming is possible, results are returned as an `AsyncIterable`: ``` for await (const record of MyTable.search({ conditions: [...] })) { // process each record as it arrives } ``` Failing to iterate the `AsyncIterable` to completion keeps a read transaction open, degrading performance. Always ensure you either fully iterate or explicitly release the query. ### Draining or Releasing a Query An open query holds an active read transaction. While that transaction is open, the underlying data pages and internal state for the query cannot be freed — they remain pinned in memory until the transaction closes. In long-running processes or under high concurrency, accumulating unreleased transactions degrades throughput and increases memory pressure. The transaction closes automatically once the `AsyncIterable` is fully iterated. If you need to stop early, you must explicitly signal that iteration is complete so Harper can release the transaction. **Breaking out of a `for await...of` loop** is the most natural way. The JavaScript runtime automatically calls `.return()` on the iterator when a `break`, `return`, or `throw` exits the loop: ``` for await (const record of MyTable.search({ conditions: [...] })) { if (meetsStopCriteria(record)) { break; // iterator.return() is called automatically — transaction is released } process(record); } ``` **Calling `.return()` manually** is useful when you hold an iterator reference directly: ``` const iterator = MyTable.search({ conditions: [...] })[Symbol.asyncIterator](); try { const { value } = await iterator.next(); process(value); } finally { await iterator.return(); // explicitly closes the iterator and releases the transaction } ``` Avoid storing an iterator and abandoning it (e.g. never using it in a for-loop or calling `.return()`), as the transaction will remain open until the iterator is garbage collected — which is non-deterministic. ## Practical Guidance ### Index fields you query on frequently ``` type Product @table { id: Long @primaryKey name: String @indexed # queried frequently category: String @indexed # queried frequently description: String # not indexed (rarely in conditions) } ``` ### Use `explain` to diagnose slow queries ``` const result = await Product.search({ conditions: [ { attribute: 'category', value: 'electronics' }, { attribute: 'price', comparator: 'less_than', value: 100 }, ], explain: true, }); // result shows the actual execution order Harper selected ``` ### Prefer selective conditions first When Harper cannot auto-reorder (e.g. with `enforceExecutionOrder`), put the most selective condition first: ``` // Better: indexed, selective condition first Product.search({ conditions: [ { attribute: 'sku', value: 'ABC-001' }, // exact match on indexed unique field { attribute: 'active', value: true }, // low cardinality filter ], }); ``` ### Use `limit` and `offset` for pagination ``` Product.search({ conditions: [...], sort: { attribute: 'createdAt', descending: true }, limit: 20, offset: page * 20, }); ``` ### Avoid wide range queries on non-indexed fields ``` // Slow: non-indexed field with range condition Product.search({ conditions: [{ attribute: 'description', comparator: 'contains', value: 'sale' }], }); // Better: use an indexed field condition to narrow first Product.search({ conditions: [ { attribute: 'category', value: 'clothing' }, // indexed — narrows to subset { attribute: 'description', comparator: 'contains', value: 'sale' }, // non-indexed, applied to smaller set ], }); ``` --- # Resource API *Added in: v4.2.0* The Resource API provides a unified JavaScript interface for accessing, querying, modifying, and subscribing to data resources in Harper. Tables extend the base `Resource` class, and all resource interactions — whether from HTTP requests, MQTT messages, or application code — flow through this interface. A Resource class represents a collection of entities/records with methods for querying and accessing records and inserting/updating records. Instances of a Resource class represent a single record that can be modified through various methods or queries. A Resource instance holds the primary key/identifier and any pending updates to the record, so any instance methods can act on the record and have full access to this information during execution. Resource classes have static methods that directly map to RESTful methods or HTTP verbs (`get`, `put`, `patch`, `post`, `delete`), which can be called to interact with records, with general create, read, update, and delete capabilities. And these static methods can be overridden for defining custom API endpoint handling. ## Resource Static Methods Static methods are defined on a Resource class and are the preferred way to interact with tables and resources from application code. They handle transaction setup, access checks, and request parsing automatically. These methods also map to RESTful HTTP verbs and can be overridden to define custom behavior for requests. ### `get(target: RequestTarget | Id | Query, context?: Resource | Context): Promise | ExtendedIterable` Retrieves a record by primary key, or queries for records when given a `Query` object or a collection `RequestTarget`. ``` // By primary key const product = await Product.get(34); // By query object with select const product = await Product.get({ id: 34, select: ['name', 'price'] }); // Iterate a collection query for await (const record of Product.get({ conditions: [{ attribute: 'inStock', value: true }] })) { // ... } ``` The default `get` method returns a `RecordObject` — a frozen plain object with the record's properties plus `getUpdatedTime()` and `getExpiresAt()`. The record object is immutable because it represents the current state of the record in the database. `get` is also called for HTTP GET requests and is always called with a `RequestTarget` as the `target` parameter. When the request targets a single record (e.g. `/Table/some-id`), the default `get` returns a single record object. When the request targets a collection (e.g. `/Table/?name=value`), the `target.isCollection` property is `true` and the default behavior calls `search()`, returning an `ExtendedIterable`. ``` class MyResource extends Resource { static get(target) { const id = target.id; // primary key from URL path const param = target.get('param1'); // query string param const path = target.pathname; // path relative to resource return super.get(target); // default: return the record } } ``` If you want a modified object from this record, you can copy the record and change properties: ``` static async get(target) { const record = await super.get(target); return { ...record, changedProperty: 'value' }; } ``` #### RecordObject The `get()` method returns a `RecordObject` — a frozen plain object with all record properties, plus: * `getUpdatedTime(): number` — Last updated time (milliseconds since epoch) * `getExpiresAt(): number` — Expiration time, if set *** ### `search(query: RequestTarget | Query, context?): ExtendedIterable` Performs a query on the resource or table. This is called by `get()` on collection requests and can be overridden to define custom query behavior. The default implementation on tables queries by the `conditions`, `limit`, `offset`, `select`, and `sort` properties parsed from the URL or query object. See [Query Object](#query-object) below for available query options. See the [ExtendedIterable](#extendediterable) below for how to interact with the query results. *** ### `put(target: RequestTarget | Id, data: Promise, context?: Resource | Context): Promise | Response` ### `put(record: object, context?): Promise` Writes the full record to the table, creating or replacing the existing record. This does *not* merge the `data` into the existing record — it replaces it. The second form reads the primary key from the record object. ``` await Product.put(34, { name: 'New Product Name' }); // or with the primary key in the object: await Product.put({ id: 34, name: 'New Product Name' }); ``` This is called for HTTP PUT requests, and can be overridden to implement a custom `PUT` handler: ``` class MyResource extends Resource { static async put(target, data) { const record = await data; return super.put(target, { ...record, status: record.status ?? 'active' }); } } ``` *** ### `patch(target: RequestTarget | Id, data: Promise, context?: Resource | Context): Promise | Response` Writes a partial record to the table, merging `data` into the existing record. ``` await Product.patch(34, { description: 'Updated description' }); ``` This is called for HTTP PATCH requests, and can be overridden to implement a custom `PATCH` handler: ``` class MyResource extends Resource { static async patch(target, data) { const record = await data; return super.patch(target, { ...record, status: record.status ?? 'active' }); } } ``` *Added in: v4.3.0* (CRDT support for individual property updates via PATCH) *** ### `post(target: RequestTarget, data: Promise, context?: Resource | Context): Promise | Response` Called for HTTP POST requests. The default behavior creates a new record, but it can be overridden to implement custom actions. Prefer more explicit methods like `create()` or `update()` over calling `post` directly. `data` is a promise that resolves to the deserialized request body — the body is read and decoded lazily, so you must `await` `data` before reading its fields. Once awaited, an `application/json` body resolves to the parsed JSON value (typically an object): ``` class MyResource extends Resource { static async post(target, data) { data = await data; if (data.action === 'create') { return this.create(target, data.content); } else if (data.action === 'update') { let resource = await this.update(target); resource.set('status', data.status); resource.save(); } } } ``` The shape of the resolved value depends on the request's `Content-Type`. The built-in deserializers produce: | `Content-Type` | resolved `data` | | -------------------------------------------- | --------------------- | | `application/json` | parsed JSON value | | `application/cbor`, `application/x-msgpack` | decoded value | | `application/x-ndjson`, `application/ndjson` | array of parsed lines | | `text/plain` | string | Custom content types registered through `contentTypes.set(...)` receive whatever value the deserializer returns. *** ### `delete(target: RequestTarget | Id, context?): Promise` Deletes a record from the table. ``` await Product.delete(34); ``` This is called for HTTP DELETE requests, and can be overridden to implement a custom `DELETE` handler: ``` class MyResource extends Resource { static async delete(target) { return super.delete(target); } } ``` *** ### `create(record: object, context?): Promise` Creates a new record with an auto-generated primary key. Returns the created Resource instance. Do not include a primary key in the `record` argument. *Added in: v4.2.0* *** ### `update(target: RequestTarget | Id, updates?: object): Promise` Returns a mutable Resource instance for a given record. Property changes on the instance are written to the table when the transaction commits. This is the primary method for getting a Resource instance and accessing all instance methods. ``` const product = await Product.update(32); product.status = 'active'; product.subtractFrom('quantity', 1); product.save(); ``` *** ### `publish(target: RequestTarget | Id, message: object, context?: Resource | Context): Promise` Publishes a message to a record/topic using the same primary key structure as records. ``` await Product.publish(34, { event: 'product-purchased', purchasePrice: 100 }); ``` This is called for MQTT publish commands. The default behavior records the message and notifies subscribers without changing the record's stored data. This can be overridden to implement custom message handling. *** ### `subscribe(subscriptionRequest?: SubscriptionRequest, context?): Promise` Called for MQTT subscribe commands. Returns a `Subscription` — an `AsyncIterable` of messages/changes. #### `SubscriptionRequest` options All properties are optional: | Property | Description | | -------------------- | ---------------------------------------------------------------------------------------------- | | `includeDescendants` | Include all updates with an id prefixed by the subscribed id (e.g. `sub/*`) | | `startTime` | Start from a past time (catch-up of historical messages). Cannot be used with `previousCount`. | | `previousCount` | Return the last N updates/messages. Cannot be used with `startTime`. | | `omitCurrent` | Do not send the current/retained record as the first update. | *** ### `connect(target: RequestTarget, incomingMessages?: AsyncIterable): AsyncIterable` Called for WebSocket and Server-Sent Events connections. `incomingMessages` is provided for WebSocket connections (not SSE). Returns an `AsyncIterable` of messages to send to the client. *** ### `invalidate(target: RequestTarget | Id, context?)` Marks the specified record as invalid in a caching table, so it will be reloaded from the source on next access. *** ### `sourcedFrom(Resource, options?)` Configure a table to use another resource as its data source. When a record is not found locally or has expired, it is fetched from the source and cached. Writes to the table are optionally delegated to the source if the source implements `put`, `patch`, or `delete`. ``` tables.MyCache.sourcedFrom(MyDataSource); ``` Options (all optional; prefer setting these via `@table` schema directives): | Option | Description | | -------------- | ------------------------------------------------ | | `expiration` | Seconds until a record goes stale | | `eviction` | Seconds after expiration before physical removal | | `scanInterval` | Seconds between eviction scans | Harper automatically serializes concurrent requests for the same missing or stale record — all waiting requests share a single upstream fetch, preventing cache stampedes. #### Observing cache disposition Each `get` on a caching table records whether the record came from the cache or from the source in the `loadedFromSource` property of the **`RequestTarget`** for that get. Cache disposition is a per-get result, so it lives on the per-get target — pass an explicit `RequestTarget` to observe it: ``` import { RequestTarget } from 'harper'; const target = new RequestTarget(); target.id = recordId; const record = await MyCache.get(target); console.log(target.loadedFromSource); // true = went to the source, false = served from cache ``` The flag settles as follows: * `true` — the get went to the source: either it fetched the record, or the source errored and a stale cached record was served as a fallback (`staleIfError`). `true` means a source request was made, not necessarily that the returned data is fresh. * `false` — the record was served from the cache: fresh hits, `onlyIfCached` requests, stale-while-revalidate responses (the source fetch continues in the background), and requests that waited on another request's in-flight fetch of the same record. This last case means a cache hit can still take as long as an upstream fetch. Note that `get()` returns a plain `RecordObject`, not a resource instance — the record itself does not carry cache disposition; the explicitly passed `RequestTarget` is the supported way to observe it. There is deliberately no request-`Context` mirror of this flag: a context is shared across every `get` in the request (including a caching table's internal gets), so a context-level value would be silently overwritten before the caller could read it. #### Source `get` — controlling timestamp and expiration Inside a source `get()` method, the context (`this.getContext()`) exposes caching-specific properties: ``` class MySource extends Resource { async get() { const context = this.getContext(); // Pass If-Modified-Since to origin using the existing cached version const headers = new Headers(); if (context.replacingVersion) { headers.set('If-Modified-Since', new Date(context.replacingVersion).toUTCString()); } const response = await fetch(`https://api.example.com/${this.getId()}`, { headers }); // Propagate the origin's Last-Modified timestamp to Harper's ETag context.lastModified = response.headers.get('Last-Modified'); // Honor origin's Cache-Control max-age for per-record TTL const maxAge = response.headers.get('Cache-Control')?.match(/max-age=(\d+)/)?.[1]; if (maxAge) { context.expiresAt = Date.now() + Number(maxAge) * 1000; } // Return origin's 304 as a cache revalidation (no re-download) if (response.status === 304) return context.replacingRecord; return response.json(); } } ``` Context properties available inside a source `get()`: | Property | Description | | ------------------ | --------------------------------------------------------------------------------------------- | | `replacingVersion` | Timestamp of the currently cached record being replaced (useful for `If-Modified-Since`) | | `replacingRecord` | The currently cached record value (return this on a 304 to skip a re-download) | | `lastModified` | Set this to propagate the origin's timestamp as Harper's `ETag` / `Last-Modified` | | `expiresAt` | Set this (milliseconds epoch) to give the record a per-entry TTL overriding the table default | #### Source `subscribe` — active caching For data sources that can push change notifications, implement a `subscribe` method returning an async iterable of events. Harper calls `subscribe` once per process and propagates updates to the cache automatically — no polling needed. ``` class MySource extends Resource { async *subscribe() { // Option A: async generator const stream = connectToExternalEventStream(); for await (const event of stream) { yield { type: 'put', // 'put' | 'invalidate' | 'delete' | 'message' | 'transaction' id: event.id, value: event.data, timestamp: event.ts, }; } } } ``` Alternatively, use the default subscription stream to push events from a callback-based source: ``` class MySource extends Resource { subscribe() { const subscription = super.subscribe(); remoteClient.on('update', (event) => { subscription.send({ type: 'put', id: event.id, value: event.data }); }); return subscription; } } ``` **Supported event types:** | Type | Description | | ------------- | ---------------------------------------------------------------------------------- | | `put` | Record updated — `value` contains the new record | | `invalidate` | Record changed but value not provided — cache evicts and re-fetches on next access | | `delete` | Record deleted | | `message` | Pub/sub message passing through the record; record data is not changed | | `transaction` | Atomic group of writes; include an array of events in the `writes` property | **Event properties:** | Property | Description | | ----------- | ----------------------------------------------------------------- | | `type` | Event type (see above) | | `id` | Primary key of the affected record | | `value` | New record value (for `put` and `message`) | | `writes` | Array of events for `transaction` events | | `table` | Target table name (for cross-table writes inside a `transaction`) | | `timestamp` | Timestamp of the change | By default, `subscribe` runs on a single thread to avoid duplicate notifications and race conditions. To run on multiple threads: ``` class MySource extends Resource { static subscribeOnThisThread(threadIndex) { return threadIndex < 2; // run on first two threads only } async *subscribe() { ... } } ``` #### Write-through caching If the source implements `put`, `patch`, or `delete`, writes to the caching table are forwarded to the source before being committed locally: ``` class MySource extends Resource { async get() { ... } async put(data) { await fetch(`https://api.example.com/${this.getId()}`, { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(await data), }); } async delete() { await fetch(`https://api.example.com/${this.getId()}`, { method: 'DELETE' }); } } ``` Harper waits for the source to confirm the write before committing to the local cache (two-phase write). **Loading from source in write methods:** Methods other than `get()` do not automatically load data from the source. Call `ensureLoaded()` first if you need the existing record: ``` class MyCache extends tables.MyCache { async post(data) { await this.ensureLoaded(); // loads from source if not cached this.quantity = this.quantity - (await data).purchases; } } ``` #### Passive-active updates A source `get()` can proactively populate *other* tables as a side effect, atomically with the main record. Pass `this` (the current context) to any write call to include it in the same transaction: ``` const { Post, Comment } = tables; class BlogSource extends Resource { async get() { const post = await (await fetch(`https://my-blog/${this.getId()}`)).json(); for (const comment of post.comments) { await Comment.put(comment, this); // atomic with the Post write } return post; } } Post.sourcedFrom(BlogSource); ``` *** ### `getUpdatedTime(): number` Returns the last updated time of the resource (milliseconds since epoch). *** ### `getRecordCount({ exactCount?: boolean }): Promise<{ recordCount: number, estimatedRange?: [number, number] }>` Returns the number of records in the table. By default returns an approximate (fast) count. Pass `{ exactCount: true }` for a precise count. *Added in: v4.5.0* *** ### `primaryKey` The name of the primary key attribute for the table. ### `static path?: string` *Added in: v5.1.13* Declares the URL path the resource is registered at, overriding the default export-name convention. A leading `/` makes the path root-relative (top-level); a leading `./` or a bare name resolves relative to the component directory. The path may contain `:name` (single-segment) and `*name` (trailing catch-all) parameters, whose matched values are bound onto the request target (e.g. `static path = '/widget/:id'` populates `target.id`). ``` export class Widget extends Resource { static path = '/widget/:id/action/:action'; static get(target) { // GET /widget/10/action/jump -> target.id === '10', target.action === 'jump' } } ``` See [Exporting Resources as Endpoints → Path Parameters](/reference/v5/resources/overview.md#path-parameters) for matching and precedence rules. *** ## Class-level metadata for MCP and OpenAPI Resource classes — both `@table @export` Resources and programmatic Resource subclasses — can declare class-level static fields that drive the MCP tool descriptors and OpenAPI document. These statics are JSON-Schema-aligned and tool-agnostic; the same data feeds every introspectable surface. For `@table @export` Resources, `static description` and `static properties` are auto-derived from the GraphQL schema (docstrings and field types). For programmatic Resources, you declare them directly. ### `static description?: string` Class-level docstring. Consumed by: * **MCP** — prefixed onto every verb-tool description (`get_X`, `search_X`, etc.) and onto the `harper://schema/{db}/{table}` resource description * **OpenAPI** — set as the schema-level `description` on the resource's component schema; also prepended to the `description` of every path operation (POST/GET/PUT/PATCH/DELETE) ``` import { Resource } from 'harperdb'; export class ProductInventory extends Resource { static description = 'Aggregate inventory analytics computed over the Product catalog. ' + 'Read-only; the underlying Product table is the system of record.'; async get(id) { /* ... */ } } ``` ### `static properties?: Record` JSON-Schema-shaped attribute map keyed by name. This is the canonical public API for class-level metadata. For `@table @export` Resources it's auto-derived from the GraphQL schema. For programmatic Resources, declare it directly: ``` export class ProductInventory extends Resource { static description = '...'; static properties = { sku: { type: 'string', primaryKey: true, description: 'Stock keeping unit; matches Product.sku.' }, onHand: { type: 'integer', description: 'Current warehouse count.' }, reserved: { type: 'integer', description: 'Units allocated to open orders but not yet shipped.' }, stockStatus: { type: 'string', enum: ['in_stock', 'out_of_stock', 'backorder'], description: 'Derived from onHand vs reserved.', }, }; async get(id) { /* ... */ } } ``` For complex types and nested structures, JSON Schema vocabulary applies (`type`, `enum`, `required`, `additionalProperties`, etc.). Per-property `description` flows into both MCP `inputSchema.properties[*].description` and OpenAPI `components.schemas[*].properties[*].description`. **Inheritance composes naturally.** Extend a `@table @export` Resource and override individual entries with spread: ``` const { Product } = tables; class CustomProduct extends Product { static properties = { ...Product.properties, priceCents: { ...Product.properties.priceCents, description: 'Retail price in cents, including any per-customer adjustments.', }, }; } ``` The author writes against `properties` (the public API). Internal code that needs ordered iteration / index metadata continues to read `Class.attributes` (the internal Array form, also inherited). ### `static outputSchemas?: { [verb: string]: JsonSchemaFragment }` Per-verb output schema overrides for programmatic Resources whose verb methods return a projection rather than the full record. When omitted, the MCP deriver falls back to `static properties` for the cheap verbs (`get`/`create`/`update`/`patch`) and a synthesized `{deleted: true, }` envelope for `delete`. `search_*` deliberately has no output schema. ``` export class ProductInventory extends Resource { static description = '...'; static properties = { /* full record shape */ }; static outputSchemas = { get: { type: 'object', properties: { sku: { type: 'string' }, onHand: { type: 'integer' }, stockStatus: { type: 'string', enum: ['in_stock', 'out_of_stock', 'backorder'] }, }, required: ['sku', 'onHand', 'stockStatus'], }, }; async get(id) { /* returns the projection above */ } } ``` ### `static hidden?: boolean` When `true`, the Resource is dropped from MCP tool registration and OpenAPI path generation entirely. Data remains accessible via direct Harper interfaces subject to RBAC. Equivalent to applying `@hidden` to the `@table @export` declaration. ``` export class InternalDiagnostics extends Resource { static hidden = true; async get() { /* ... */ } } ``` ### `static mcp?: { annotations?: { [verb: string]: Annotations } }` Narrow, MCP-only override for annotation hints that don't fit JSON Schema (such as `idempotentHint` per verb). Documented as discouraged — most authors should only need `static description` + `static properties`. Use this when you need to claim, for example, that your custom `update` semantics are observably idempotent on repeat calls. ``` export class ProductInventory extends Resource { static description = '...'; static properties = { /* ... */ }; static mcp = { annotations: { update: { idempotentHint: true }, }, }; } ``` > **Under-annotate before mis-annotate.** Under MCP semantics, `idempotentHint: true` is a strong claim: the second call must produce the same observable outcome as the first. `add_*`-style operations that return "already exists" on the second call are NOT idempotent in this sense, even though they don't crash. Verify repeat-call behavior end-to-end before annotating. ### `static mcpTools?: McpToolDefinition[]` Component-author opt-in for exposing non-verb instance methods as MCP tools. Each entry maps an instance-method name to an MCP tool descriptor. RBAC is enforced by the Resource method itself; the MCP layer does not invent new ACLs. ``` export class ProductInventory extends Resource { static mcpTools = [ { name: 'reconcile_inventory', method: 'reconcileInventory', description: 'Triggers an immediate reconciliation against the warehouse system. ' + 'Returns the diff applied. Heavy — do not call in a loop.', inputSchema: { type: 'object', properties: { sku: { type: 'string', description: 'SKU to reconcile, or omit for full sweep.' }, }, }, annotations: { destructiveHint: false, idempotentHint: false }, }, ]; async reconcileInventory(args) { /* ... */ } } ``` Custom `mcpTools` declarations without a `description` or `inputSchema` are still registered, but Harper emits a warn-once log at registration time — LLM tool selection degrades without these fields. *** ### `setComputedAttribute(name: string, computeFunction: (record) => any)` Define the compute function for a `@computed` schema attribute. *Added in: v4.4.0* ``` MyTable.setComputedAttribute('fullName', (record) => `${record.firstName} ${record.lastName}`); ``` *** ### `getContext(): Context` Returns the current context, which includes: * `user` — User object with username, role, and authorization information * `transaction` — The current transaction When triggered by HTTP, the context is the `Request` object with these additional properties: * `url` — Full local path including query string * `method` — HTTP method * `headers` — Request headers (access with `context.headers.get(name)`) * `responseHeaders` — Response headers (set with `context.responseHeaders.set(name, value)`) * `pathname` — Path without query string * `host` — Host from the `Host` header * `ip` — Client IP address * `body` — Raw Node.js `Readable` stream (if a request body exists) * `data` — Promise resolving to the deserialized request body * `lastModified` — Controls the `ETag`/`Last-Modified` response header * `requestContext` — (For source resources only) Context of the upstream resource making the data request *** ### `operation(operationObject: object, authorize?: boolean): Promise` Executes a Harper operations API call using this table as the target. Set `authorize` to `true` to enforce current-user authorization. *** ### `getCurrentUser(): User | undefined` Returns the user associated with the current request, or `undefined` if no user is authenticated. The returned object exposes the username, role, and `role.permission` flags. ``` async get(target) { const user = this.getCurrentUser(); if (!user) return new Response(null, { status: 401 }); return { username: user.username, role: user.role }; } ``` *** ### Session and Login from a Resource The context returned by `getContext()` exposes `login` and `session` for handling sign-in/out flows in a custom Resource. Sessions require `authentication.enableSessions: true` in `harperdb-config.yaml`. ``` export class SignIn extends Resource { async post(_target, data) { const context = this.getContext(); try { await context.login(data.username, data.password); } catch { return new Response('Invalid credentials', { status: 403 }); } return new Response('Logged in', { status: 200 }); } } export class SignOut extends Resource { async post() { const context = this.getContext(); if (!context.session) return new Response(null, { status: 401 }); await context.session.delete(context.session.id); return new Response('Logged out', { status: 200 }); } } ``` `context.login(username, password)` verifies credentials and establishes the session cookie on success. To end a session, delete it via `context.session.delete(context.session.id)`. Cookie-based sessions are intended for browser clients. For non-browser clients (CLI tools, mobile apps, service-to-service), use JWT issuance — see [JWT Authentication](/reference/v5/security/jwt-authentication.md). *** ## Resource Instance Methods A Resource instance is used to update and interact with a single record/resource. It provides functionality for updating properties, accessing property values, and managing record lifecycle. The Resource instance is normally retrieved from the static `update()` method. An instance from a table has updatable properties that can used to access and update individual properties (for properties declared in the table's schema), as well methods for more advanced updates and saving data. For example: ``` const product = await Product.update(32); product.status = 'active'; // we can directly change properties on the updatable record, if they are declared in the schema product.subtractFrom('quantity', 1); // We can use CRDT incrementation/decrementation to safely update the quantity product.save(); ``` ### `save()` This saves the current state of the resource to the database in the current transaction. This method can be called after making changes to the resource to ensure that those changes have been saved to the current transaction and can be queried within the same transaction. Any pending changes are automatically saved when the transaction commits (if `save()` has not already saved them). This method only saves data when using RocksDB storage engine, and is a no-op when using LMDB. ### `addTo(property: string, value: number)` Adds `value` to `property` using CRDT incrementation — safe for concurrent updates across threads and nodes. *Added in: v4.3.0* ``` static async post(target, data) { const record = await this.update(target.id); record.addTo('quantity', -1); // decrement safely across nodes } // GET /MyTable/test?foo=bar → primary key is 'test?foo=bar' ``` ### `subtractFrom(property: string, value: number)` Subtracts `value` from `property` using CRDT incrementation. ### `set(property: string, value: any): void` Sets a property to `value`. Equivalent to direct property assignment (`record.property = value`), but can be used when the property name is dynamic and not declared in the schema. ``` const record = await Table.update(target.id); record.set('status', 'active'); record.save(); ``` ### `put(record: object): void` This replaces the current record data in the instance with the provided `record` object. ### `patch(record: object): void` This merges the provided `record` object into the current record data for the instance. ### `validate(record: object, partial?: boolean): void` This validates the provided `record` object against the schema, throwing an error if validation fails. If `partial` is true, only validates the provided properties, otherwise validates all required properties. This can be overridden to implement custom validation logic. This is called at the beginning of a transaction commit, prior to writing data to the transaction and fully committing it. ### `publish(message: object): void` This publishes a message to the current instance's primary key. ### `invalidate(): void` This invalidates the current instance's record in a caching table, forcing it to be reloaded from the source on next access. ### `getId(): Id` Returns the primary key of the current instance. ### `getProperty(property: string): any` Returns the current value of `property` from the record. Useful when the property name is dynamic or when you want an explicit read rather than direct property access. ``` const record = await Table.update(target.id); const current = record.getProperty('status'); ``` ### `getUpdatedTime(): number` Returns the last updated time as milliseconds since epoch. ### `getExpiresAt(): number` Returns the expiration time, if one is set. ### `allowStaleWhileRevalidate(entry, id): boolean` For caching tables: return `true` to serve the stale entry while revalidation happens concurrently; `false` to wait for the fresh value. Entry properties: * `version` — Timestamp/version from the source * `localTime` — When the resource was last refreshed locally * `expiresAt` — When the entry became stale * `value` — The stale record value The following instances are also implemented on Resource instances for [backwards compatibility with 4.x](/reference/v4/resources/resource-api.md), but generally not necessary to directly use: * `get` * `search` * `post` * `create` * `subscribe` ## Concurrency and Safe Concurrent Writes When multiple writers may touch the same record concurrently, only atomic deltas are safe. Harper resolves deltas at commit time, so the committed result is exact regardless of how the writers interleave: ``` record.addTo('counter', 1); // increment record.subtractFrom('stock', 1); // decrement ``` Read-then-write patterns — "compare-and-set" — are not safe under concurrency. This includes version-guarded updates (`ifVersion`), create-only "first write wins", reading a value to decide and then writing it back, and HTTP `If-Match` / conditional writes. All of these read a value *before* commit and are not re-validated *at* commit, so two concurrent writers can both pass the check. Modeling state as commutative deltas avoids the problem entirely. A few limitations to design around: * `addTo` and `subtractFrom` return no value, and a read-back immediately after a write is not guaranteed to reflect your own committed result. You cannot use them to read "your position" for an exactly-once claim or a hard limit. * `subtractFrom` has no floor; concurrent decrements can drive a value negative. For non-commutative operations — claim-once, hard caps, inventory floors, state-machine transitions — serialize the operation through a single owning process, or use an external lock or coordinator. This matters more in a cluster: a counter replicates and converges eventually, but a decision made on one node's not-yet-converged view can be wrong during the replication window. A rate limiter built this way, for example, can transiently over-admit when requests are spread across nodes. ## Returning Errors and Status Codes A custom resource method controls the HTTP status in a few ways. The key distinction is **throw vs. return**: throwing rolls the transaction back, while returning (resolving) commits it. To return an **error** status, throw — either an `Error` carrying a status, or a plain object. Both `statusCode` and `status` are honored: ``` async get() { const err = new Error('Not found'); err.statusCode = 404; throw err; } ``` ``` async get() { throw { status: 400, message: 'Invalid input' }; } ``` To set the status on a **successful** response, return a `Response`, or set it on the context and return your data — either commits the transaction: ``` async post(target, data) { return new Response(body, { status: 201 }); } ``` ``` async get() { this.getContext().response.status = 202; return data; } ``` A thrown `Response` is honored too — it short-circuits with its own status, headers, and body — but, being a throw, it **rolls back the transaction**. Use it for error or redirect short-circuits; `return` a `Response` when a preceding write in the same method should commit: ``` async post(target, data) { // short-circuits to 422; any write earlier in this method is rolled back throw new Response(body, { status: 422 }); } ``` One pattern looks like it should work but does not: * `return { status: 400, data: {...} }` — a bare `status` on a returned plain object is ignored (the response is 200) unless the object also carries a `headers` field. `status` collides with ordinary record attributes, so `headers` is the disambiguator. Return a `Response`, or set `context.response.status`, instead. Note that an unhandled throw currently surfaces its message in the response body, so avoid putting sensitive detail in thrown error messages. ## Query Object The `Query` object is accepted by `search()` and the static `get()` method. ### `conditions` Array of condition objects to filter records. Each condition: | Property | Description | | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `attribute` | Property name, or an array for chained/joined properties (e.g. `['brand', 'name']`) | | `value` | The value to match | | `comparator` | `equals` (default), `greater_than`, `greater_than_equal`, `less_than`, `less_than_equal`, `starts_with`, `contains`, `ends_with`, `between`, `not_equal` | | `conditions` | Nested conditions array | | `operator` | `and` (default) or `or` for the nested `conditions` | Example with nested conditions: ``` Product.search({ conditions: [ { attribute: 'price', comparator: 'less_than', value: 100 }, { operator: 'or', conditions: [ { attribute: 'rating', comparator: 'greater_than', value: 4 }, { attribute: 'featured', value: true }, ], }, ], }); ``` **Chained attribute references** (for relationships/joins): Use an array to traverse relationship properties: ``` Product.search({ conditions: [{ attribute: ['brand', 'name'], value: 'Harper' }] }); ``` *Added in: v4.3.0* ### `operator` Top-level `and` (default) or `or` for the `conditions` array. ### `limit` Maximum number of records to return. ### `offset` Number of records to skip (for pagination). ### `select` Properties to include in each returned record. Can be: * Array of property names: `['name', 'price']` * Nested select for related records: `[{ name: 'brand', select: ['id', 'name'] }]` * String to return a single property per record: `'id'` Special properties: * `$id` — Returns the primary key regardless of its name * `$updatedtime` — Returns the last-updated timestamp * `$distance` — When the query ranks or filters by a vector index, returns the computed distance from the target vector. See [Vector Indexing](/reference/v5/database/schema.md#vector-indexing). #### Selecting related records When a field is defined as a [relationship](/reference/v5/database/schema.md#relationships), `select` can pull the related record(s) into each result as a nested property — the programmatic equivalent of the REST `select(name,author{name})` syntax. * **Whole related record** — list the relationship field by name. The related record (or an array of records for a to-many relationship) is attached as a nested property: ``` // `author` is a relationship field const book = await Book.get({ id: 42, select: ['id', 'title', 'author'] }); book.author.name; // the full related Author record ``` * **Partial related record** — use an object `{ name, select }` to choose which fields of the related record to return. Unselected fields are omitted: ``` const book = await Book.get({ id: 42, select: ['id', 'title', { name: 'author', select: ['name'] }] }); ``` * **Nesting** — a `select` inside an object entry may itself contain more `{ name, select }` objects, traversing multiple relationships in one query: ``` select: ['id', 'name', { name: 'segments', select: ['id', 'name', { name: 'client', select: ['id', 'name'] }] }]; ``` A to-many relationship resolves to an array of records; depending on access pattern you may need to `await` the property before iterating it. **Join behavior:** selecting a relationship *without* filtering on it behaves as a **LEFT JOIN** — records with no related row are still returned (the relationship property is omitted or empty). Adding a condition on a related attribute (e.g. `attribute: ['author', 'name']`) behaves as an **INNER JOIN** — only records with a matching related row are returned. ### `sort` Sort order object: | Property | Description | | ------------ | ---------------------------------------------------------- | | `attribute` | Property name (or array for chained relationship property) | | `descending` | Sort descending if `true` (default: `false`) | | `next` | Secondary sort to resolve ties (same structure) | Harper uses an index to provide sort order, so a `sort` needs one of: * An `@indexed` sort `attribute`. Harper aligns the scan with the index automatically — **no condition is required**, and the condition (if any) does not have to be on the sort attribute. * At least one `conditions` entry (on **any** attribute) when the sort `attribute` is not indexed. Harper filters by the condition and then orders the result set in memory. Only the combination of a non-indexed sort `attribute` **and** zero conditions is rejected: > `HdbError: is not indexed and not combined with any other conditions` Note the bare `@primaryKey` is treated as not indexed for this purpose (it has its own primary store rather than a secondary index), so sorting by the primary key alone — with no conditions — hits this error. To iterate a whole table in primary-key order, add an open-ended range condition (which can be on the primary key or any other attribute): ``` Product.search({ conditions: [{ attribute: 'id', comparator: 'greater_than', value: '' }], sort: { attribute: 'id' }, }); ``` Alternatively, pass `allowFullScan: true` to permit an unconditional ordered scan, or — if order doesn't matter — omit `sort` entirely and `search({})` will iterate without an index requirement. ### `explain` If `true`, returns conditions reordered as Harper will execute them (for debugging and optimization). ### `enforceExecutionOrder` If `true`, forces conditions to execute in the order supplied, disabling Harper's automatic re-ordering optimization. *** ## RequestTarget `RequestTarget` represents a URL path mapped to a resource. It is a subclass of `URLSearchParams`. Properties: * `pathname` — Path relative to the resource, without query string * `search` — The query/search string portion of the URL * `id` — Primary key derived from the path * `isCollection` — `true` when the request targets a collection * `checkPermission` — Set to indicate authorization should be performed; has `action`, `resource`, and `user` sub-properties Standard `URLSearchParams` methods are available: * `get(name)`, `getAll(name)`, `set(name, value)`, `append(name, value)`, `delete(name)`, `has(name)` * Iterable: `for (const [name, value] of target) { ... }` When a URL uses Harper's extended query syntax, these are parsed onto the target: * `conditions`, `limit`, `offset`, `sort`, `select` *** ## RecordObject The `get()` method returns a `RecordObject` — a frozen plain object with all record properties, plus: * `getUpdatedTime(): number` — Last updated time (milliseconds since epoch) * `getExpiresAt(): number` — Expiration time, if set *** ## ExtendedIterable The `ExtendedIterable` extends and behaves like an `AsyncIterable`, but also includes a set of array-like methods: * `map`: Returns a new `ExtendedIterable` with the results of calling a provided function on every element in the calling `ExtendedIterable` (lazily evaluated as the iterable is consumed). * `filter`: Returns a new `ExtendedIterable` with the elements that pass the test implemented by the provided function (lazily evaluated as the iterable is consumed). * `flatMap`: Returns a new `ExtendedIterable` with the results of calling a provided function on every element in the calling `ExtendedIterable` and then flattening the result by one-level (lazily evaluated as the iterable is consumed). * `concat`: Returns a new `ExtendedIterable` that contains the elements of the calling `ExtendedIterable` followed by the elements of the iterable passed as an argument (lazily evaluated as the iterable is consumed). * `forEach`: Iterates the `ExtendedIterable`, calling the provided function once per element. This is executed eagerly/immediately. * `slice`: Returns a new `ExtendedIterable` containing a subset of the elements of the calling `ExtendedIterable` (lazily evaluated as the iterable is consumed). * `mapError`: Returns a new `ExtendedIterable` with that matches the calling `ExtendedIterable`, but maps any element evaluation that throws an error (lazily evaluated as the iterable is consumed) to a new value. These methods are intended to allow you to easily interact with the results of search queries, without having to convert the `ExtendedIterable` to an array. Generally, converting results to an array is discouraged because it can consume a excessive memory for large results, and undermines Harper's efficient iteration/streaming system. For example, you might write a `get` method like: ``` static async function get(target) { const records = this.search(target); // we can filter records here const filteredRecords = records.map((record) => record.quantity > 100); // we can map to new values const mappedRecords = filteredRecords.map((record) => ({ ...record, extraProperty: 'value' })); // we never converted this to an array, large results can efficiently to be streamed to the client return mappedRecords; } ``` If we do want to iterate the results within a function using a for-loop, you can use the `for await` syntax: ``` for await (const record of records) { if (record.name === 'I found what I was looking for') { return record; } } ``` (but again, using a for-loop to convert to an array is discouraged) See the [ExtendedIterable documentation](https://github.com/harperfast/extended-iterable) for more details. ## Response Object Resource methods can return: 1. **Plain data** — serialized using content negotiation 2. **`Response`-like object** with `status`, `headers`, and `data` or `body`: ``` // Redirect return { status: 302, headers: { Location: '/new-location' } }; // Custom header with data return { status: 200, headers: { 'X-Custom-Header': 'value' }, data: { message: 'ok' } }; ``` `body` must be a string, `Buffer`, Node.js stream, or `ReadableStream`. `data` is an object that will be serialized. *Added in: v4.4.0* ### Throwing Errors Uncaught errors are caught by the protocol handler. For REST, they produce error responses. Set `error.statusCode` to control the HTTP status: ``` if (!authorized) { const error = new Error('Forbidden'); error.statusCode = 403; throw error; } ``` *** ## Context and Transactions Harper's HTTP/REST request handler automatically starts a transaction for each request, and assigns the `Request` object as the current context. The current context is available via `getContext()` as export from the `harper` module, or as a global variable. All database interactions that are called from the request will automatically use that transaction, for reading and writing data. Transactions and context are tracking using [asynchronous context tracking](https://nodejs.org/dist/latest/docs/api/async_context.html). However, you can explicitly create transactions to control the scope of atomicity and isolation. Transactions are created with the `transaction()` method, which establishes a transaction and context that are used for all subsequent database operations within the asynchronous context of the transaction. For example: ``` import { transaction } from 'harper'; function receivedShipment(products) { let myContext = {}; trasaction(myContext, async () => { for (let received of products) { let product = await Product.update(received.productId); product.addTo('quantity', received.quantity); } }); // all the product updates will be atomically commmited in this transaction } ``` See [Global APIs — transaction](/reference/v5/components/javascript-environment.md#transaction) for more information on starting transactions outside of request handlers. *** ### `getContext(): Context` getContext is availabe as export from the `harper` module, or as a global variable, and returns the current context, which includes: * `user` — User object with username, role, and authorization information * `transaction` — The current transaction When triggered by HTTP, the context is the `Request` object with these additional properties: * `url` — Full local path including query string * `method` — HTTP method * `headers` — Request headers (access with `context.headers.get(name)`) * `responseHeaders` — Response headers (set with `context.responseHeaders.set(name, value)`) * `pathname` — Path without query string * `host` — Host from the `Host` header * `ip` — Client IP address * `body` — Raw Node.js `Readable` stream (if a request body exists) * `data` — Promise resolving to the deserialized request body * `lastModified` — Controls the `ETag`/`Last-Modified` response header * `requestContext` — (For source resources only) Context of the upstream resource making the data request --- # Content Types Harper supports multiple content types (MIME types) for both HTTP request bodies and response bodies. Harper follows HTTP standards: use the `Content-Type` request header to specify the encoding of the request body, and use the `Accept` header to request a specific response format. ``` Content-Type: application/cbor Accept: application/cbor ``` All content types work with any standard Harper operation. ## Supported Formats ### JSON — `application/json` JSON is the most widely used format, readable and easy to work with. It is well-supported across all HTTP tooling. **Limitations**: JSON does not natively support all Harper data types — binary data, `Date`, `Map`, and `Set` values require special handling. JSON also produces larger payloads than binary formats. **When to use**: Web development, debugging, interoperability with third-party clients, or when the standard JSON type set is sufficient. Pairing JSON with compression (`Accept-Encoding: gzip, br`) often yields compact network transfers due to favorable Huffman coding characteristics. ### CBOR — `application/cbor` CBOR is the recommended format for most production use cases. It is a highly efficient binary format with native support for the full range of Harper data types, including binary data, typed dates, and explicit Maps/Sets. **Advantages**: Very compact encoding, fast serialization, native streaming support (indefinite-length arrays for optimal time-to-first-byte on query results). Well-standardized with growing ecosystem support. **When to use**: Production APIs, performance-sensitive applications, or any scenario requiring rich data types. ### MessagePack — `application/x-msgpack` MessagePack is another efficient binary format similar to CBOR, with broader adoption in some ecosystems. It supports all Harper data types. **Limitations**: MessagePack does not natively support streaming arrays, so query results are returned as a concatenated sequence of MessagePack objects. Decoders must be prepared to handle a sequence of values rather than a single document. **When to use**: Systems with existing MessagePack support that don't have CBOR available, or when interoperability with MessagePack clients is required. CBOR is generally preferred when both are available. ### CSV — `text/csv` Comma-separated values format, suitable for data export and spreadsheet import/export. CSV lacks hierarchical structure and explicit typing. **When to use**: Ad-hoc data export, spreadsheet workflows, batch data processing. Not recommended for frequent or production API use. ## Content Type via URL Extension As an alternative to the `Accept` header, responses can be requested in a specific format using file-style URL extensions: ``` GET /product/some-id.csv GET /product/.msgpack?category=software ``` Using the `Accept` header is the recommended approach for clean, standard HTTP interactions. ## Custom Content Types Harper's content type system is extensible. Custom handlers for any serialization format (XML, YAML, proprietary formats, etc.) can be registered in the [`contentTypes`](/reference/v5/components/javascript-environment.md#contenttypes) global Map. ## Storing Arbitrary Content Types When a `PUT` or `POST` is made with a non-standard content type (e.g., `text/calendar`, `image/gif`), Harper stores the content as a record with `contentType` and `data` properties: ``` PUT /my-resource/33 Content-Type: text/calendar BEGIN:VCALENDAR VERSION:2.0 ... ``` This stores a record equivalent to: ``` { "contentType": "text/calendar", "data": "BEGIN:VCALENDAR\nVERSION:2.0\n..." } ``` Retrieving a record that has `contentType` and `data` properties returns the response with the specified `Content-Type` and body. If the content type is not from the `text` family, the data is treated as binary (a Node.js `Buffer`). Use `application/octet-stream` for binary data or for uploading to a specific property: ``` PUT /my-resource/33/image Content-Type: image/gif ...image data... ``` ## See Also * [REST Overview](/reference/v5/rest/overview.md) — HTTP methods and URL structure * [Headers](/reference/v5/rest/headers.md) — Content negotiation headers * [Querying](/reference/v5/rest/querying.md) — URL query syntax --- # REST Headers Harper's REST interface uses standard HTTP headers for content negotiation, caching, and performance instrumentation. ## Response Headers These headers are included in all Harper REST API responses: | Header | Example Value | Description | | --------------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `server-timing` | `db;dur=7.165` | Duration of the operation in milliseconds. Follows the [Server-Timing](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Server-Timing) standard and can be consumed by network monitoring tools. | | `content-type` | `application/json` | MIME type of the returned content, negotiated based on the `Accept` request header. | | `etag` | `"abc123"` | Encoded version/last-modification time of the returned record. Used for conditional requests. | | `location` | `/MyTable/new-id` | Returned on `POST` responses. Contains the path to the newly created record. | ## Request Headers ### Content-Type Specifies the format of the request body (for `PUT`, `PATCH`, `POST`): ``` Content-Type: application/json Content-Type: application/cbor Content-Type: application/x-msgpack Content-Type: text/csv ``` See [Content Types](/reference/v5/rest/content-types.md) for the full list of supported formats. ### Accept Specifies the preferred response format: ``` Accept: application/json Accept: application/cbor Accept: application/x-msgpack Accept: text/csv ``` ### If-None-Match Used for conditional GET requests. Provide the `ETag` value from a previous response to avoid re-fetching unchanged data: ``` GET /MyTable/123 If-None-Match: "abc123" ``` If the record has not changed, Harper returns `304 Not Modified` with no body. This avoids serialization and network transfer overhead and works seamlessly with browser caches and external HTTP caches. ### Accept-Encoding Harper supports standard HTTP compression. Including this header enables compressed responses: ``` Accept-Encoding: gzip, br ``` Compression is particularly effective for JSON responses. For binary formats like CBOR, compression provides diminishing returns compared to the already-compact encoding. ### Authorization Credentials for authenticating requests. See [Security Overview](/reference/v5/security/overview.md) for details on supported authentication mechanisms (Basic, JWT, mTLS). ### Sec-WebSocket-Protocol When connecting via WebSocket for MQTT, the sub-protocol must be set to `mqtt` as required by the MQTT specification: ``` Sec-WebSocket-Protocol: mqtt ``` ## Content Type via URL Extension As an alternative to the `Accept` header, content types can be specified using file-style extensions in the URL path: ``` GET /product/some-id.csv GET /product/.msgpack?category=software ``` This is not recommended for production use — prefer the `Accept` header for clean, standard HTTP interactions. ## See Also * [REST Overview](/reference/v5/rest/overview.md) — HTTP methods and URL structure * [Content Types](/reference/v5/rest/content-types.md) — Supported encoding formats * [Security Overview](/reference/v5/security/overview.md) — Authentication headers and mechanisms --- # REST Overview *Added in: v4.2.0* Harper provides a powerful, efficient, and standard-compliant HTTP REST interface for interacting with tables and other resources. The REST interface is the recommended interface for data access, querying, and manipulation over HTTP, providing the best performance and HTTP interoperability with different clients. ## How the REST Interface Works Harper's REST interface exposes database tables and custom resources as RESTful endpoints. Tables are **not** exported by default; they must be explicitly exported in a schema definition. The name of the exported resource defines the base of the endpoint path, served on the application HTTP server port (default `9926`). For more on defining schemas and exporting resources, see [Database / Schema](/reference/v5/database/schema.md). ## Configuration Enable the REST interface by adding the `rest` plugin to your application's `config.yaml`: ``` rest: true ``` **Options**: ``` rest: lastModified: true # enables Last-Modified response header support webSocket: false # disables automatic WebSocket support (enabled by default) ``` ## URL Structure The REST interface follows a consistent URL structure: | Path | Description | | -------------------------------------------- | ------------------------------------------------------------------------------------------------- | | `/my-resource` | Root path — returns a description of the resource (e.g., table metadata) | | `/my-resource/` | Trailing slash indicates a collection — represents all records; append query parameters to search | | `/my-resource/record-id` | A specific record identified by its primary key | | `/my-resource/record-id/` | Trailing slash — the collection of records with the given id prefix | | `/my-resource/record-id/with/multiple/parts` | Record id with multiple path segments | *Changed in: v4.5.0* — Resources can be defined with nested paths and accessed by exact path without a trailing slash. The `id.property` dot syntax for accessing properties via URL is only applied to properties declared in a schema. ## HTTP Methods REST operations map to HTTP methods following uniform interface principles: ### GET Retrieve a record or perform a search. Handled by the resource's `get()` method. ``` GET /MyTable/123 ``` Returns the record with primary key `123`. ``` GET /MyTable/?name=Harper ``` Returns records matching `name=Harper`. See [Querying](/reference/v5/rest/querying.md) for the full query syntax. ``` GET /MyTable/123.propertyName ``` Returns a single property of a record. Only works for properties declared in the schema. #### Conditional Requests and Caching GET responses include an `ETag` header encoding the record's version/last-modification time. Clients with a cached copy can include `If-None-Match` on subsequent requests. If the record hasn't changed, Harper returns `304 Not Modified` with no body — avoiding serialization and network transfer overhead. ### PUT Create or replace a record with a specified primary key (upsert semantics). Handled by the resource's `put(record)` method. The stored record will exactly match the submitted body — any properties not included in the body are removed from the previous record. ``` PUT /MyTable/123 Content-Type: application/json { "name": "some data" } ``` Creates or replaces the record with primary key `123`. ### POST Create a new record without specifying a primary key, or trigger a custom action. Handled by the resource's `post(data)` method. The auto-assigned primary key is returned in the `Location` response header. ``` POST /MyTable/ Content-Type: application/json { "name": "some data" } ``` ### PATCH Partially update a record, merging only the provided properties (CRDT-style update). Unspecified properties are preserved. *Added in: v4.3.0* ``` PATCH /MyTable/123 Content-Type: application/json { "status": "active" } ``` ### DELETE Delete a specific record or all records matching a query. ``` DELETE /MyTable/123 ``` Deletes the record with primary key `123`. ``` DELETE /MyTable/?status=archived ``` Deletes all records matching `status=archived`. ## Content Types Harper supports multiple content types for both request bodies and responses. Use the `Content-Type` header for request bodies and the `Accept` header to request a specific response format. See [Content Types](/reference/v5/rest/content-types.md) for the full list of supported formats and encoding recommendations. ## OpenAPI *Added in: v4.3.0* Harper automatically generates an OpenAPI specification for all resources exported via a schema. This endpoint is available at: ``` GET /openapi ``` ## See Also * [Querying](/reference/v5/rest/querying.md) — Full URL query syntax, operators, and examples * [Headers](/reference/v5/rest/headers.md) — HTTP headers used by the REST interface * [Content Types](/reference/v5/rest/content-types.md) — Supported formats (JSON, CBOR, MessagePack, CSV) * [WebSockets](/reference/v5/rest/websockets.md) — Real-time connections via WebSocket * [Server-Sent Events](/reference/v5/rest/server-sent-events.md) — One-way streaming via SSE * [HTTP Server](/reference/v5/http/overview.md) — Underlying HTTP server configuration * [Database / Schema](/reference/v5/database/schema.md) — How to define and export resources --- # REST Querying Harper's REST interface supports a rich URL-based query language for filtering, sorting, selecting, and limiting records. Queries are expressed as URL query parameters on collection paths. ## Basic Attribute Filtering Search by attribute name and value using query parameters. The queried attribute must be indexed. ``` GET /Product/?category=software ``` Multiple attributes can be combined — only one needs to be indexed for the query to execute: ``` GET /Product/?category=software&inStock=true ``` ### Null Queries *Added in: v4.3.0* Query for null values or non-null values: ``` GET /Product/?discount=null ``` Note: Only indexes created in v4.3.0 or later support null indexing. Existing indexes must be rebuilt (removed and re-added) to support null queries. ## Comparison Operators (FIQL) Harper uses [FIQL](https://datatracker.ietf.org/doc/html/draft-nottingham-atompub-fiql-00) syntax for comparison operators: | Operator | Meaning | | -------------------- | -------------------------------------- | | `==` | Equal | | `=lt=` | Less than | | `=le=` | Less than or equal | | `=gt=` | Greater than | | `=ge=` | Greater than or equal | | `=ne=`, `!=` | Not equal | | `=ct=` | Contains (strings) | | `=sw=`, `==*` | Starts with (strings) | | `=ew=` | Ends with (strings) | | `=`, `===` | Strict equality (no type conversion) | | `!==` | Strict inequality (no type conversion) | **Examples**: ``` GET /Product/?price=gt=100 GET /Product/?price=le=20 GET /Product/?name==Keyboard* GET /Product/?category=software&price=gt=100&price=lt=200 ``` For date fields, colons must be URL-encoded as `%3A`: ``` GET /Product/?listDate=gt=2017-03-08T09%3A30%3A00.000Z ``` ### Chained Conditions (Range) Omit the attribute name on the second condition to chain it against the same attribute: ``` GET /Product/?price=gt=100<=200 ``` Chaining supports `gt`/`ge` combined with `lt`/`le` for range queries. No other chaining combinations are currently supported. ### Type Conversion For FIQL comparators (`==`, `!=`, `=gt=`, etc.), Harper applies automatic type conversion: | Syntax | Behavior | | ----------------------------------------- | ------------------------------------------- | | `name==null` | Converts to `null` | | `name==123` | Converts to number if attribute is untyped | | `name==true` | Converts to boolean if attribute is untyped | | `name==number:123` | Explicit number conversion | | `name==boolean:true` | Explicit boolean conversion | | `name==string:some%20text` | Keep as string with URL decode | | `name==date:2024-01-05T20%3A07%3A27.955Z` | Explicit Date conversion | If the attribute specifies a type in the schema (e.g., `Float`), values are always converted to that type before searching. For strict operators (`=`, `===`, `!==`), no automatic type conversion is applied — the value is decoded as a URL-encoded string, and the attribute type (if declared in the schema) dictates type conversion. ## Unions (OR Logic) Use `|` instead of `&` to combine conditions with OR logic: ``` GET /Product/?rating=5|featured=true ``` ## Grouping Use parentheses or square brackets to control order of operations: ``` GET /Product/?rating=5|(price=gt=100&price=lt=200) ``` Square brackets are recommended when constructing queries from user input because standard URI encoding safely encodes `[` and `]` (but not `(`): ``` GET /Product/?rating=5&[tag=fast|tag=scalable|tag=efficient] ``` Constructing from JavaScript: ``` let url = `/Product/?rating=5&[${tags.map(encodeURIComponent).join('|')}]`; ``` Groups can be nested for complex conditions: ``` GET /Product/?price=lt=100|[rating=5&[tag=fast|tag=scalable|tag=efficient]&inStock=true] ``` ## Query Functions Harper supports special query functions using call syntax, included in the query string separated by `&`. ### `select(properties)` Specify which properties to include in the response. | Syntax | Returns | | -------------------------------------- | ------------------------------------------- | | `?select(property)` | Values of a single property directly | | `?select(property1,property2)` | Objects with only the specified properties | | `?select([property1,property2])` | Arrays of property values | | `?select(property1,)` | Objects with a single specified property | | `?select(property{subProp1,subProp2})` | Nested objects with specific sub-properties | **Examples**: ``` GET /Product/?category=software&select(name) GET /Product/?brand.name=Microsoft&select(name,brand{name}) ``` ### `limit(end)` or `limit(start,end)` Limit the number of results returned, with an optional starting offset. ``` GET /Product/?rating=gt=3&inStock=true&select(rating,name)&limit(20) GET /Product/?rating=gt=3&limit(10,30) ``` ### `sort(property)` or `sort(+property,-property,...)` Sort results by one or more properties. Prefix `+` or no prefix = ascending; `-` = descending. Multiple properties break ties in order. ``` GET /Product/?rating=gt=3&sort(+name) GET /Product/?sort(+rating,-price) ``` *Added in: v4.3.0* ## Relationships and Joins *Added in: v4.3.0* Harper supports querying across related tables through dot-syntax chained attributes. Relationships must be defined in the schema using `@relationship`. **Schema example**: ``` type Product @table @export { id: Long @primaryKey name: String brandId: Long @indexed brand: Brand @relationship(from: "brandId") } type Brand @table @export { id: Long @primaryKey name: String products: [Product] @relationship(to: "brandId") } ``` **Query by related attribute** (INNER JOIN behavior): ``` GET /Product/?brand.name=Microsoft GET /Brand/?products.name=Keyboard ``` ### Nested Select with Joins Relationship attributes are not included by default. Use `select()` to include them: ``` GET /Product/?brand.name=Microsoft&select(name,brand) GET /Product/?brand.name=Microsoft&select(name,brand{name}) GET /Product/?name=Keyboard&select(name,brand{name,id}) ``` When selecting without a filter on the related table, this acts as a LEFT JOIN — the relationship property is omitted if the foreign key is null or references a non-existent record. ### Many-to-Many Relationships Many-to-many relationships can be modeled with an array of foreign key values, without a junction table: ``` type Product @table @export { id: Long @primaryKey name: String resellerIds: [Long] @indexed resellers: [Reseller] @relationship(from: "resellerId") } ``` ``` GET /Product/?resellers.name=Cool Shop&select(id,name,resellers{name,id}) ``` The array order of `resellerIds` is preserved when resolving the relationship. ## Property Access via URL *Changed in: v4.5.0* Access a specific property of a record by appending it with dot syntax to the record id: ``` GET /MyTable/123.propertyName ``` This only works for properties declared in the schema. As of v4.5.0, dots in URL paths are no longer interpreted as property access for undeclared properties, allowing URLs to generally include dots without being misinterpreted. ## `directURLMapping` Option *Added in: v4.5.0* Resources can be configured with `directURLMapping: true` for more direct URL path handling. When enabled, the URL path is mapped more directly to the resource without the default query parameter parsing semantics. See [Database / Schema](/reference/v5/database/schema.md) for configuration details. ## See Also * [REST Overview](/reference/v5/rest/overview.md) — HTTP methods, URL structure, and caching * [Headers](/reference/v5/rest/headers.md) — Request and response headers * [Content Types](/reference/v5/rest/content-types.md) — Encoding formats * [Database / Schema](/reference/v5/database/schema.md) — Defining schemas, relationships, and indexes --- # Server-Sent Events *Added in: v4.2.0* Harper supports Server-Sent Events (SSE), a simple and efficient mechanism for browser-based applications to receive real-time updates from the server over a standard HTTP connection. SSE is a one-directional transport — the server pushes events to the client, and the client has no way to send messages back on the same connection. ## Connecting SSE connections are made by targeting a resource URL. By default, connecting to a resource path subscribes to changes for that resource and streams events as they occur. ``` let eventSource = new EventSource('https://server/my-resource/341', { withCredentials: true, }); eventSource.onmessage = (event) => { let data = JSON.parse(event.data); }; ``` The URL path maps to the resource in the same way as REST and WebSocket connections. Connecting to `/my-resource/341` subscribes to updates for the record with id `341` in the `my-resource` table (or custom resource). ## `connect()` Handler SSE connections use the same `connect()` method as WebSockets on resource classes, with one key difference: since SSE is one-directional, `connect()` is called without an `incomingMessages` argument. ``` export class MyResource extends Resource { async *connect() { // yield messages to send to the client while (true) { await someCondition(); yield { event: 'update', data: { value: 42 } }; } } } ``` The default `connect()` behavior subscribes to the resource and streams changes automatically. ## Reading the request in `connect()` To read the incoming request (query parameters, headers, the path), define `connect()` as a **`static`** method. Its first argument is the `RequestTarget`, which exposes the query string via `target.get()`. Because `target` is an ordinary argument, it is captured in the generator's scope — read it directly inside the `async *connect()` body: ``` export class Search extends Resource { static async *connect(target) { const query = target.get('q'); // GET /Search?q=harper const apiKey = target.get('api_key'); for (const hit of await runSearch(query)) yield { data: hit }; } } ``` This works even though the generator body doesn't begin executing until the first event is requested — the `target` argument is already bound when `connect()` is called. > **Note:** the `RequestTarget` argument is only passed to a **`static`** `connect()`. An instance method's first argument is not the target, so define `connect()` as `static`. ### Doing synchronous work before streaming If you must read the request and act on it *synchronously* — before the stream starts — split `connect()` into a reader that returns a separate generator. This is also required when reading the request `context` via `getContext()`, which must be captured in the synchronous call frame: it is no longer available once the generator body is deferred to the first event. ``` export class Search extends Resource { static connect(target) { const query = target.get('q'); if (!query) throw new Error('q is required'); // reject before streaming starts return this.stream(query); } static async *stream(query) { for (const hit of await runSearch(query)) yield { data: hit }; } } ``` ## When to Use SSE vs WebSockets | | SSE | WebSockets | | --------------- | ------------------------------------- | -------------------------------- | | Direction | Server → Client only | Bidirectional | | Transport | Standard HTTP | HTTP upgrade | | Browser support | Native `EventSource` API | Native `WebSocket` API | | Use case | Live feeds, dashboards, notifications | Interactive real-time apps, MQTT | SSE is simpler to implement and has built-in reconnection in browsers. For scenarios requiring bidirectional communication, use [WebSockets](/reference/v5/rest/websockets.md). ## See Also * [WebSockets](/reference/v5/rest/websockets.md) — Bidirectional real-time connections * [MQTT Overview](/reference/v5/mqtt/overview.md) — Full MQTT pub/sub documentation * [REST Overview](/reference/v5/rest/overview.md) — HTTP methods and URL structure * [Resources](/reference/v5/resources/overview.md) — Custom resource API including `connect()` --- # WebSockets *Added in: v4.2.0* Harper supports WebSocket connections through the REST interface, enabling real-time bidirectional communication with resources. WebSocket connections target a resource URL path — by default, connecting to a resource subscribes to changes for that resource. ## Configuration WebSocket support is enabled automatically when the `rest` plugin is enabled. To disable it: ``` rest: webSocket: false ``` ## Connecting A WebSocket connection to a resource URL subscribes to that resource and streams change events: ``` let ws = new WebSocket('wss://server/my-resource/341'); ws.onmessage = (event) => { let data = JSON.parse(event.data); }; ``` By default, `new WebSocket('wss://server/my-resource/341')` accesses the resource defined for `my-resource` with record id `341` and subscribes to it. When the record changes or a message is published to it, the WebSocket connection receives the update. ## Custom `connect()` Handler WebSocket behavior is driven by the `connect(incomingMessages)` method on a resource class. The method must return an async iterable (or generator) that produces messages to send to the client. For more on implementing custom resources, see [Resource API](/reference/v5/resources/resource-api.md). **Simple echo server**: ``` export class Echo extends Resource { async *connect(incomingMessages) { for await (let message of incomingMessages) { yield message; // echo each message back } } } ``` **Using the default connect with event-style access**: The default `connect()` returns a convenient streaming iterable with: * A `send(message)` method for pushing outgoing messages * A `close` event for cleanup on disconnect ``` export class Example extends Resource { connect(incomingMessages) { let outgoingMessages = super.connect(); let timer = setInterval(() => { outgoingMessages.send({ greeting: 'hi again!' }); }, 1000); incomingMessages.on('data', (message) => { outgoingMessages.send(message); // echo incoming messages }); outgoingMessages.on('close', () => { clearInterval(timer); }); return outgoingMessages; } } ``` ## MQTT over WebSockets Harper also supports MQTT over WebSockets. The sub-protocol must be set to `mqtt` as required by the MQTT specification: ``` Sec-WebSocket-Protocol: mqtt ``` See [MQTT Overview](/reference/v5/mqtt/overview.md) for full MQTT documentation. ## Message Ordering in Distributed Environments Harper prioritizes low-latency delivery in distributed (multi-node) environments. Messages are delivered to local subscribers immediately upon arrival — Harper does not delay messages for inter-node coordination. In a scenario where messages arrive out-of-order across nodes: * **Non-retained messages** (published without a `retain` flag): Every message is delivered to subscribers in the order received, even if out-of-order relative to other nodes. Good for use cases like chat where every message must be delivered. * **Retained messages** (published with `retain`, or PUT/updated in the database): Only the message with the latest timestamp is kept as the "winning" record. Out-of-order older messages are not re-delivered. This ensures eventual consistency of the most recent record state across the cluster. Good for use cases like sensor readings where only the latest value matters. ## See Also * [Server-Sent Events](/reference/v5/rest/server-sent-events.md) — One-way real-time streaming * [MQTT Overview](/reference/v5/mqtt/overview.md) — Full MQTT pub/sub documentation * [REST Overview](/reference/v5/rest/overview.md) — HTTP methods and URL structure * [Resources](/reference/v5/resources/overview.md) — Custom resource API including `connect()` --- # Security API Harper exposes security-related globals accessible in all component JavaScript modules without needing to import them. *** ## `auth(username, password?): Promise` Returns the user object for the given username. If `password` is provided, it is verified before returning the user (throws on incorrect password). ``` const user = await auth('admin', 'secret'); // user.role, user.username, etc. ``` This is useful for implementing custom authentication flows or verifying credentials in component code. For HTTP-level authentication configuration, see [Security Overview](/reference/v5/security/overview.md). --- # Basic Authentication Available since: v4.1.0 Harper supports HTTP Basic Authentication. In the context of an HTTP transaction, [Basic Authentication](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Authentication#basic_authentication_scheme) is the simplest authorization scheme which transmits credentials as username/password pairs encoded using base64. Importantly, this scheme does not encrypt credentials. If used over an insecure connection, such as HTTP, they are susceptible to being compromised. Only ever use Basic Authentication over secured connections, such as HTTPS. Even then, its better to upgrade to an encryption based authentication scheme or certificates. See [HTTP / TLS](/reference/v5/http/tls.md) for more information. ## How It Works Each request must contain the `Authorization` header with a value if `Basic `, where `` is the Base64 encoding of the string `username:password`. ``` Authorization: Basic ``` ## Example The following example shows how to construct the Authorization header using `btoa()`: ``` const username = 'HDB_ADMIN'; const password = 'abc123!'; const authorizationValue = `Basic ${btoa(`${username}:${password}`)}`; ``` Then use the `authorizationValue` as the value for the `Authorization` header such as: ``` fetch('/', { // ... headers: { Authorization: authorizationValue, }, // ... }); ``` ## cURL Example With cURL you can use the `--user` (`-u`) command-line option to automatically handle the Base64 encoding: ``` curl -u "username:password" [URL] ``` ## When to Use Basic Auth Basic authentication is the simplest option and is appropriate for: * Server-to-server requests in trusted environments * Development and testing * Scenarios where token management overhead is undesirable For user-facing applications or when tokens are preferred for performance reasons, see [JWT Authentication](/reference/v5/security/jwt-authentication.md). --- # Certificate Management This page covers certificate management for Harper's external-facing HTTP and Operations APIs. For replication certificate management, see [Replication Certificate Management](/reference/v5/replication/clustering.md). ## Default Behavior On first run, Harper automatically generates self-signed TLS certificates at `/keys/`: * `certificate.pem` — The server certificate * `privateKey.pem` — The server private key * `ca.pem` — A self-signed Certificate Authority These certificates have a valid Common Name (CN), but they are not signed by a root authority. HTTPS can be used with them, but clients must be configured to accept the invalid certificate. ## Development Setup By default, HTTPS is disabled. HTTP is suitable for local development and trusted private networks. If you are developing on a remote server with requests traversing the Internet, enable HTTPS. To enable HTTPS, set `http.securePort` in `harper-config.yaml` and restart Harper: ``` http: securePort: 9926 ``` Harper will use the auto-generated certificates from `/keys/`. ## Production Setup For production, use certificates from your own CA or a public CA, with CNs that match the Fully Qualified Domain Name (FQDN) of your Harper node. ### Option 1: Replace Harper Certificates Enable HTTPS and replace the certificate files: ``` http: securePort: 9926 tls: certificate: ~/hdb/keys/certificate.pem privateKey: ~/hdb/keys/privateKey.pem ``` Either replace the files at `/keys/` in place, or update `tls.certificate` and `tls.privateKey` to point to your new files and restart Harper. The `operationsApi.tls` section is optional. If not set, Harper uses the values from the top-level `tls` section. You can specify different certificates for the Operations API: ``` operationsApi: tls: certificate: ~/hdb/keys/certificate.pem privateKey: ~/hdb/keys/privateKey.pem ``` ### Option 2: Nginx Reverse Proxy Instead of enabling HTTPS directly on Harper, use Nginx as a reverse proxy. Configure Nginx to handle HTTPS with certificates from your own CA or a public CA, then forward HTTP requests to Harper. This approach keeps Harper's HTTP interface internal while Nginx handles TLS termination. ### Option 3: External Reverse Proxy / Load Balancer External services such as an AWS Elastic Load Balancer or Google Cloud Load Balancing can act as TLS-terminating reverse proxies. Configure the service to accept HTTPS connections and forward over a private network to Harper as HTTP. These services typically include integrated certificate management. ## mTLS Setup Mutual TLS (mTLS) requires both client and server to present certificates. To enable mTLS, provide a CA certificate that Harper will use to verify client certificates: ``` http: mtls: required: true tls: certificateAuthority: ~/hdb/keys/ca.pem ``` For full mTLS authentication details, see [mTLS Authentication](/reference/v5/security/mtls-authentication.md). ## Certificate Verification *Added in: v4.5.0* (certificate revocation); v4.7.0 (OCSP support) When using mTLS, enable certificate verification to ensure revoked certificates cannot authenticate even if still within their validity period: ``` http: mtls: required: true certificateVerification: true ``` Harper supports two industry-standard methods: **CRL (Certificate Revocation List)** * Downloaded and cached locally (24 hours by default) * Fast verification after first download (no network requests) * Best for high-volume verification and offline scenarios **OCSP (Online Certificate Status Protocol)** * Real-time query to the CA's OCSP responder * Best for certificates without CRL distribution points * Responses cached (1 hour by default) **Harper's approach: CRL-first with OCSP fallback** 1. Checks CRL if available (fast, cached locally) 2. Falls back to OCSP if CRL is unavailable or fails 3. Applies the configured failure mode if both methods fail For full configuration options and troubleshooting, see [Certificate Verification](/reference/v5/security/certificate-verification.md). ## Dynamic Certificate Management *Added in: v4.4.0* Certificates — including CAs and private keys — can be dynamically managed without restarting Harper. ## Multiple Certificate Authorities It is possible to use different certificates for the Operations API and the HTTP (custom application) API. For example, in scenarios where only your application endpoints need to be exposed to the Internet and the Operations API is reserved for administration, you may use a private CA for the Operations API and a public CA for your application certificates. Configure each separately: ``` # Top-level tls: used by HTTP/application endpoints tls: certificate: ~/hdb/keys/app-certificate.pem privateKey: ~/hdb/keys/app-privateKey.pem # Operations API can use a separate cert operationsApi: tls: certificate: ~/hdb/keys/ops-certificate.pem privateKey: ~/hdb/keys/ops-privateKey.pem ``` ## Renewing Certificates The `harper renew-certs` CLI command renews the auto-generated Harper certificates. See [CLI Commands](/reference/v5/cli/commands.md) for details. **Changes to TLS settings require a restart**, except where dynamic certificate management is used. --- # Certificate Verification *Added in: v4.5.0* *Changed in: v4.7.0* (OCSP support added) Certificate verification (also called certificate revocation checking) ensures that revoked certificates cannot be used for mTLS authentication, even if they are otherwise valid and trusted. This is a critical security control for environments where certificates may need to be revoked before their expiration date — due to compromise, employee departure, or other security concerns. ## Overview When a client presents a certificate for mTLS authentication, Harper performs two levels of checks: 1. **Certificate Validation** (always performed by Node.js TLS): * Certificate signature is valid * Certificate is issued by a trusted CA * Certificate is within its validity period * Certificate chain is properly formed 2. **Certificate Revocation Checking** (optional, must be explicitly enabled): * Certificate has not been revoked by the issuing CA * Uses CRL and/or OCSP Revocation checking is **disabled by default**. ## Revocation Checking Methods ### CRL (Certificate Revocation List) A CRL is a digitally signed list of revoked certificates published by a Certificate Authority. **Advantages:** * Fast verification (cached locally) * Works offline once downloaded * Predictable bandwidth usage * Good for high-volume verification * No privacy concerns (no per-certificate queries) **How it works:** 1. Harper downloads the CRL from the distribution point specified in the certificate. 2. The CRL is cached locally (24 hours by default). 3. Subsequent verifications check the cached CRL — very fast, no network requests. 4. The CRL is refreshed in the background before expiration. **Configuration:** ``` http: mtls: certificateVerification: crl: timeout: 10000 # 10 seconds to download CRL cacheTtl: 86400000 # Cache for 24 hours gracePeriod: 86400000 # 24 hour grace period after nextUpdate failureMode: fail-closed # Reject on CRL check failure ``` ### OCSP (Online Certificate Status Protocol) OCSP provides real-time certificate status checking by querying the CA's OCSP responder. **Advantages:** * Real-time revocation status * Smaller response size than CRL * Good for certificates without CRL distribution points * Works when CRL is unavailable **How it works:** 1. Harper sends a request to the OCSP responder specified in the certificate. 2. The responder returns the current status: good, revoked, or unknown. 3. The response is cached (1 hour by default for success, 5 minutes for errors). **Configuration:** ``` http: mtls: certificateVerification: ocsp: timeout: 5000 # 5 seconds for OCSP response cacheTtl: 3600000 # Cache successful responses for 1 hour errorCacheTtl: 300000 # Cache errors for 5 minutes failureMode: fail-closed # Reject on OCSP check failure ``` ## Verification Strategy Harper uses a **CRL-first strategy with OCSP fallback**: 1. **Check CRL** if available (fast; uses cached CRL; no network request if cached). 2. **Fall back to OCSP** if the certificate has no CRL distribution point, the CRL download fails, or the CRL is expired and cannot be refreshed. 3. **Apply failure mode** if both methods fail. This provides the best balance of performance, reliability, and security. ## Configuration ### Enable with Defaults ``` http: mtls: required: true certificateVerification: true ``` This enables CRL checking (10s timeout, 24h cache), OCSP checking (5s timeout, 1h cache), and fail-closed mode. ### Custom Configuration ``` http: mtls: required: true certificateVerification: failureMode: fail-closed # Global setting crl: timeout: 15000 # 15 seconds for CRL download cacheTtl: 43200000 # Cache CRLs for 12 hours gracePeriod: 86400000 # 24 hour grace period failureMode: fail-closed # CRL-specific setting ocsp: timeout: 8000 # 8 seconds for OCSP response cacheTtl: 7200000 # Cache results for 2 hours errorCacheTtl: 600000 # Cache errors for 10 minutes failureMode: fail-closed # OCSP-specific setting ``` ### CRL Only (No OCSP) ``` http: mtls: certificateVerification: ocsp: false # Disable OCSP; CRL remains enabled ``` Only disable OCSP if all client certificates have CRL distribution points. Otherwise, certificates without CRL URLs won't be checked for revocation. ### OCSP Only (No CRL) ``` http: mtls: certificateVerification: crl: false # Disable CRL; OCSP remains enabled ``` ### Environment Variables All settings can be configured via environment variables: ``` # Enable certificate verification HTTP_MTLS_CERTIFICATEVERIFICATION=true # Global failure mode HTTP_MTLS_CERTIFICATEVERIFICATION_FAILUREMODE=fail-closed # CRL settings HTTP_MTLS_CERTIFICATEVERIFICATION_CRL=true HTTP_MTLS_CERTIFICATEVERIFICATION_CRL_TIMEOUT=15000 HTTP_MTLS_CERTIFICATEVERIFICATION_CRL_CACHETTL=43200000 HTTP_MTLS_CERTIFICATEVERIFICATION_CRL_GRACEPERIOD=86400000 HTTP_MTLS_CERTIFICATEVERIFICATION_CRL_FAILUREMODE=fail-closed # OCSP settings HTTP_MTLS_CERTIFICATEVERIFICATION_OCSP=true HTTP_MTLS_CERTIFICATEVERIFICATION_OCSP_TIMEOUT=8000 HTTP_MTLS_CERTIFICATEVERIFICATION_OCSP_CACHETTL=7200000 HTTP_MTLS_CERTIFICATEVERIFICATION_OCSP_ERRORCACHETTL=600000 HTTP_MTLS_CERTIFICATEVERIFICATION_OCSP_FAILUREMODE=fail-closed ``` For replication servers, use the `REPLICATION_` prefix instead of `HTTP_`. ## Failure Modes ### fail-closed (Recommended) **Default behavior.** Rejects connections when verification fails due to network errors, timeouts, or other operational issues. Use when: * Security is paramount * You can tolerate false positives (rejecting valid certificates due to CA unavailability) * Your CA infrastructure is highly available * You're in a zero-trust environment ``` certificateVerification: failureMode: fail-closed ``` ### fail-open Allows connections when verification fails, but logs a warning. The connection is still rejected if the certificate is explicitly found to be revoked. Use when: * Availability is more important than perfect security * Your CA infrastructure may be intermittently unavailable * You have other compensating controls * You're gradually rolling out certificate verification ``` certificateVerification: failureMode: fail-open ``` **Important:** Invalid signatures on CRLs always result in rejection regardless of failure mode, as this indicates potential tampering. ## Performance Considerations ### CRL Performance * **First verification**: Downloads CRL (10s timeout by default) * **Subsequent verifications**: Instant (reads from cache) * **Background refresh**: CRL is refreshed before expiration without blocking requests * **Memory usage**: \~10–100KB per CRL depending on size * **Network usage**: One download per CRL per `cacheTtl` period ### OCSP Performance * **First verification**: OCSP query (5s timeout by default) * **Subsequent verifications**: Reads from cache (1 hour default) * **Memory usage**: Minimal (\~1KB per cached response) * **Network usage**: One query per unique certificate per `cacheTtl` period ### Optimization Tips Increase CRL cache TTL for stable environments: ``` --- crl: cacheTtl: 172800000 # 48 hours ``` Increase OCSP cache TTL for long-lived connections: ``` --- ocsp: cacheTtl: 7200000 # 2 hours ``` Reduce grace period for tighter revocation enforcement: ``` --- crl: gracePeriod: 0 # No grace period ``` ## Production Best Practices ### High-Security Environments ``` http: mtls: required: true certificateVerification: failureMode: fail-closed crl: timeout: 15000 cacheTtl: 43200000 # 12 hours gracePeriod: 0 # No grace period for strict enforcement ocsp: timeout: 8000 cacheTtl: 3600000 # 1 hour ``` ### High-Availability Environments ``` http: mtls: required: true certificateVerification: failureMode: fail-open # Prioritize availability crl: timeout: 5000 cacheTtl: 86400000 # 24 hours gracePeriod: 86400000 # 24 hour grace period ocsp: timeout: 3000 cacheTtl: 7200000 # 2 hours ``` ### Performance-Critical Environments ``` http: mtls: required: true certificateVerification: crl: cacheTtl: 172800000 # 48 hours gracePeriod: 86400000 ocsp: cacheTtl: 7200000 # 2 hours errorCacheTtl: 600000 ``` ## Troubleshooting ### Connection Rejected: Certificate Verification Failed **Cause:** Certificate was found to be revoked, or verification failed in fail-closed mode. **Solutions:** 1. Check if the certificate is actually revoked in the CRL or OCSP responder. 2. Verify CA infrastructure is accessible. 3. Check timeout settings — increase if needed. 4. Temporarily switch to fail-open mode while investigating. ### High Latency on First Connection **Cause:** CRL is being downloaded for the first time. **Solutions:** 1. This is normal; only happens once per CRL per `cacheTtl` period. 2. Subsequent connections will be fast (cached CRL). 3. Increase CRL timeout if downloads are slow: ``` crl: timeout: 20000 # 20 seconds ``` ### Frequent CRL Downloads **Cause:** `cacheTtl` is too short, or the CRL's `nextUpdate` period is very short. **Solutions:** 1. Increase `cacheTtl`: ``` crl: cacheTtl: 172800000 # 48 hours ``` 2. Increase `gracePeriod` to allow using slightly expired CRLs. ### OCSP Responder Unavailable **Cause:** OCSP responder is down or unreachable. **Solutions:** 1. CRL will be used as fallback automatically. 2. Use fail-open mode to allow connections: ``` ocsp: failureMode: fail-open ``` 3. Disable OCSP and rely on CRL only (ensure all certs have CRL URLs): ``` ocsp: false ``` ### Network or Firewall Blocking Outbound Requests **Cause:** Secure hosting environments often restrict outbound HTTP/HTTPS traffic. This prevents Harper from reaching CRL distribution points and OCSP responders. **Symptoms:** * Certificate verification timeouts in fail-closed mode * Logs show connection failures to CRL/OCSP URLs * First connection may succeed (no cached data), but subsequent connections fail after cache expires **Solutions:** 1. **Allow outbound traffic to CA infrastructure** (recommended): * Whitelist CRL distribution point URLs from your certificates * Whitelist OCSP responder URLs from your certificates * Example for Let's Encrypt: allow `http://x1.c.lencr.org/` and `http://ocsp.int-x3.letsencrypt.org/` 2. **Use fail-open mode:** ``` certificateVerification: failureMode: fail-open ``` 3. **Set up an internal CRL mirror/proxy:** ``` certificateVerification: crl: cacheTtl: 172800000 # 48 hours ocsp: false ``` 4. **Disable verification** (if you have alternative security controls): ``` certificateVerification: false ``` ## Security Considerations Enable certificate verification when: * Certificates have long validity periods (> 1 day) * You need immediate revocation capability * Compliance requires revocation checking (PCI DSS, HIPAA, etc.) * You're in a zero-trust security model * Client certificates are used for API authentication Consider skipping it when: * Certificates have very short validity periods (< 24 hours) * You rotate certificates automatically (e.g., with cert-manager) * You have alternative revocation mechanisms * Your CA doesn't publish CRLs or support OCSP Certificate verification is one layer of security. Also consider: short certificate validity periods, certificate pinning, network segmentation, access logging, and regular certificate rotation. ## Replication Certificate verification works identically for replication servers. Use the `replication.mtls` configuration: ``` replication: hostname: server-one routes: - server-two mtls: certificateVerification: true ``` mTLS is always required for replication and cannot be disabled. This configuration only controls whether certificate revocation checking is performed. For complete replication configuration, see [Replication Configuration](/reference/v5/replication/clustering.md). --- # Authentication Configuration Harper's authentication system is configured via the top-level `authentication` section of `harper-config.yaml`. ``` authentication: authorizeLocal: true cacheTTL: 30000 enableSessions: true operationTokenTimeout: 1d refreshTokenTimeout: 30d hashFunction: sha256 ``` ## Options ### `authorizeLocal` *Type: boolean — Default: `true`* Automatically authorizes requests from the loopback IP address (`127.0.0.1`) as the superuser, without requiring credentials. Disable this for any Harper server that may be accessed by untrusted users from the same instance — for example, when using a local proxy or for general server hardening. ### `cacheTTL` *Type: number — Default: `30000`* How long (in milliseconds) an authentication result — a particular `Authorization` header or token — can be cached. Increasing this improves performance at the cost of slower revocation. ### `enableSessions` *Type: boolean — Default: `true`* *Added in: v4.2.0* Enables cookie-based sessions to maintain an authenticated session across requests. This is the preferred authentication mechanism for web browsers: cookies hold the token securely without exposing it to JavaScript, reducing XSS vulnerability risk. ### `operationTokenTimeout` *Type: string — Default: `1d`* How long a JWT operation token remains valid before expiring. Accepts [`jsonwebtoken`-compatible](https://github.com/auth0/node-jsonwebtoken#token-expiration-exp-claim) duration strings (e.g., `1d`, `12h`, `60m`). See [JWT Authentication](/reference/v5/security/jwt-authentication.md). ### `refreshTokenTimeout` *Type: string — Default: `30d`* How long a JWT refresh token remains valid before expiring. Accepts [`jsonwebtoken`-compatible](https://github.com/auth0/node-jsonwebtoken#token-expiration-exp-claim) duration strings. See [JWT Authentication](/reference/v5/security/jwt-authentication.md). ### `hashFunction` *Type: string — Default: `sha256`* *Added in: v4.5.0* Password hashing algorithm used when storing user passwords. Replaced the previous MD5 hashing. Options: * **`sha256`** — Default. Good security and excellent performance. * **`argon2id`** — Highest security. More CPU-intensive; recommended for environments that do not require frequent password verifications. ## Related * [JWT Authentication](/reference/v5/security/jwt-authentication.md) * [Basic Authentication](/reference/v5/security/basic-authentication.md) * [Users & Roles / Configuration](/reference/v5/users-and-roles/configuration.md) --- # Impersonation Impersonation allows a `super_user` to execute operations API requests as if they were a different, less-privileged user. This is useful for testing permissions, debugging access issues, and building admin tools that preview what a given user or role can see and do — all without needing that user's credentials. ## How It Works Add an `impersonate` property to any operations API request body. Harper will authenticate the request normally (the caller must be a `super_user`), then **downgrade** the effective permissions for that request to match the impersonated identity. ``` POST https://my-harperdb-server:9925/ Authorization: Basic Content-Type: application/json { "operation": "search_by_hash", "database": "dev", "table": "dog", "hash_values": ["1"], "impersonate": { "username": "test_user" } } ``` The request above runs the `search_by_hash` as if `test_user` had made it — subject to that user's role permissions. ## Security Constraints * **Super user only** — only users with `super_user` permissions can use `impersonate`. All other users receive a `403` error. * **Downgrade only** — impersonation can never escalate privileges. The `super_user` and `cluster_user` flags are always forced to `false` on the impersonated identity. * **Audit trail** — every impersonated request is logged, recording who initiated the impersonation and which identity was assumed. ## Impersonation Modes There are three ways to specify the impersonated identity, depending on what you want to test. ### Impersonate an Existing User Provide a `username` to run the request with that user's current role and permissions. ``` { "operation": "search_by_hash", "database": "dev", "table": "dog", "hash_values": ["1"], "impersonate": { "username": "test_user" } } ``` The target user must exist and be active. If the user is not found, a `404` error is returned. If the user is inactive, a `403` error is returned. ### Impersonate an Existing Role Provide a `role_name` to run the request with that role's permissions. You can optionally include a `username` to set the effective username (defaults to the caller's username). ``` { "operation": "search_by_value", "database": "dev", "table": "dog", "search_attribute": "name", "search_value": "Penny", "impersonate": { "role_name": "developer" } } ``` The role must exist. If the role is not found, a `404` error is returned. ### Impersonate with Inline Permissions Provide a `role` object with a `permission` property to test with an ad-hoc set of permissions. This is useful for previewing the effect of a role you haven't created yet. ``` { "operation": "sql", "sql": "SELECT * FROM dev.dog", "impersonate": { "username": "preview_user", "role": { "permission": { "dev": { "tables": { "dog": { "read": true, "insert": false, "update": false, "delete": false, "attribute_permissions": [] } } } } } } } ``` The `username` field is optional and defaults to the caller's username. The `permission` object follows the same structure as [role permissions](/reference/v5/users-and-roles/overview.md#operation-permissions). You can also restrict the impersonated identity to a specific set of operations API calls using the `operations` field inside `permission`: ``` { "operation": "search_by_hash", "database": "dev", "table": "dog", "hash_values": ["1"], "impersonate": { "role": { "permission": { "operations": ["read_only"], "dev": { "tables": { "dog": { "read": true, "insert": false, "update": false, "delete": false, "attribute_permissions": [] } } } } } } } ``` ## Impersonate Payload Reference | Field | Type | Description | | ----------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------- | | `username` | string | Target username. Required for existing-user mode. Optional for role-based modes (defaults to the caller's username). | | `role_name` | string | Name of an existing role to assume. | | `role` | object | Inline role definition. Must include a `permission` object. | | `role.permission` | object | Permission object following the standard [role permissions](/reference/v5/users-and-roles/overview.md#operation-permissions) structure. | Exactly one of `username` (alone), `role_name`, or `role` must be provided. If `role` is present, it takes precedence. ## Use Cases * **Admin dashboards** — preview what a user sees without switching accounts. * **Permission testing** — verify that a role grants (or denies) the expected access before assigning it to users. * **Debugging** — reproduce access issues reported by a user by impersonating them directly. * **CI/CD** — automated tests can verify permission configurations by impersonating different roles against a single `super_user` credential. --- # JWT Authentication Available since: v4.1.0 Harper supports token-based authentication using JSON Web Tokens (JWTs). Rather than sending credentials on every request, a client authenticates once and receives tokens that are used for subsequent requests. ## Tokens JWT authentication uses two token types: * **`operation_token`** — Used to authenticate all Harper operations via a `Bearer` token `Authorization` header. Default expiry: 1 day. * **`refresh_token`** — Used to obtain a new `operation_token` when the current one expires. Default expiry: 30 days. ## Create Authentication Tokens Call `create_authentication_tokens` with your Harper credentials. No `Authorization` header is required for this operation. ``` { "operation": "create_authentication_tokens", "username": "username", "password": "password" } ``` cURL example: ``` curl --location --request POST 'http://localhost:9925' \ --header 'Content-Type: application/json' \ --data-raw '{ "operation": "create_authentication_tokens", "username": "username", "password": "password" }' ``` Response: ``` { "operation_token": "", "refresh_token": "" } ``` ## Using the Operation Token Pass the `operation_token` as a `Bearer` token in the `Authorization` header on subsequent requests: ``` curl --location --request POST 'http://localhost:9925' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data-raw '{ "operation": "search_by_hash", "schema": "dev", "table": "dog", "hash_values": [1], "get_attributes": ["*"] }' ``` ## Refreshing the Operation Token When the `operation_token` expires, use the `refresh_token` to obtain a new one. Pass the `refresh_token` as the `Bearer` token: ``` curl --location --request POST 'http://localhost:9925' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data-raw '{ "operation": "refresh_operation_token" }' ``` Response: ``` { "operation_token": "" } ``` When both tokens have expired, call `create_authentication_tokens` again with your username and password. ## Issuing Tokens from a Custom Resource Custom Resources can mint tokens programmatically by invoking the same operations via [`server.operation()`](/reference/v5/http/api.md#serveroperationoperation-context-authorize). This is useful when you want a Resource-style endpoint (e.g., `POST /IssueTokens`) instead of (or in addition to) the raw Operations API. ``` import { Resource, server } from 'harper'; export class IssueTokens extends Resource { static async get(_target, context) { // Caller is already authenticated (Basic Auth or an existing JWT) — issue // tokens for the current user. const { operation_token, refresh_token } = await server.operation( { operation: 'create_authentication_tokens' }, context, true ); return { operation_token, refresh_token }; } static async post(_target, data) { // Caller provides credentials in the body — issue tokens directly. const { username, password } = await data; if (!username || !password) { return new Response('username and password required', { status: 400 }); } const { operation_token, refresh_token } = await server.operation({ operation: 'create_authentication_tokens', username, password, }); return { operation_token, refresh_token }; } } export class RefreshJWT extends Resource { static async post(_target, data) { const { refresh_token } = await data; if (!refresh_token) { return new Response('refresh_token required', { status: 400 }); } const { operation_token } = await server.operation({ operation: 'refresh_operation_token', refresh_token, }); return { operation_token }; } } ``` Pass `authorize: true` (third argument) when the operation should run as the current authenticated user; omit it (or pass `false`) when the operation supplies its own credentials. ## Token Expiry Configuration Token timeouts are configurable in `harper-config.yaml` under the top-level `authentication` section: ``` authentication: operationTokenTimeout: 1d # Default: 1 day refreshTokenTimeout: 30d # Default: 30 days ``` Valid duration string values follow the [`jsonwebtoken` package format](https://github.com/auth0/node-jsonwebtoken#token-expiration-exp-claim) (e.g., `1d`, `12h`, `60m`). See [Security / Configuration](/reference/v5/security/configuration.md) for the full authentication config reference. ## When to Use JWT Auth JWT authentication is preferred over Basic Auth when: * You want to avoid sending credentials on every request * Your client can store and manage tokens * You have multiple sequential requests and want to avoid repeated credential encoding For simple or server-to-server scenarios, see [Basic Authentication](/reference/v5/security/basic-authentication.md). ## Security Notes * Always use HTTPS in production to protect tokens in transit. See [HTTP / TLS](/reference/v5/http/tls.md). * Store tokens securely; treat them like passwords. * If a token is compromised, it will remain valid until it expires. Consider setting shorter `operationTokenTimeout` values in high-security environments. --- # mTLS Authentication *Added in: v4.3.0* Harper supports Mutual TLS (mTLS) authentication for incoming HTTP connections. When enabled, the client must present a certificate signed by a trusted Certificate Authority (CA). If the certificate is valid and trusted, the connection is authenticated using the user whose username matches the `CN` (Common Name) from the client certificate's `subject`. ## How It Works 1. The client presents a TLS certificate during the handshake. 2. Harper validates the certificate against the configured CA (`tls.certificateAuthority`). 3. If valid, Harper extracts the `CN` from the certificate subject and uses it as the username for the request. 1. Or it is configurable via the `http.mtls.user` option in the relevant configuration object. 4. Optionally, Harper checks whether the certificate has been revoked (see [Certificate Verification](/reference/v5/security/certificate-verification.md)). ## Configuration mTLS is configured via the `http.mtls` section in `harper-config.yaml`. **Require mTLS for all connections:** ``` http: mtls: required: true tls: certificateAuthority: ~/hdb/keys/ca.pem ``` **Make mTLS optional (accept both mTLS and non-mTLS connections):** ``` http: mtls: required: false tls: certificateAuthority: ~/hdb/keys/ca.pem ``` When `required` is `false`, clients that do not present a certificate will fall back to other authentication methods (Basic Auth or JWT). For more configuration information see the [HTTP / Configuration](/reference/v5/http/configuration.md) and [HTTP / TLS](/reference/v5/http/tls.md) sections. ## Certificate Revocation Checking When using mTLS, you can optionally enable certificate revocation checking to ensure that revoked certificates cannot authenticate, even if they are otherwise valid and trusted. To enable: ``` http: mtls: required: true certificateVerification: true ``` Certificate revocation checking is **disabled by default** and must be explicitly enabled. For full details on CRL and OCSP configuration, see [Certificate Verification](/reference/v5/security/certificate-verification.md). ## User Identity The username for the mTLS-authenticated request is derived from the `CN` field of the client certificate's subject. Ensure the CN value matches an existing Harper user account. See [Users and Roles](/reference/v5/users-and-roles/overview.md) for managing user accounts. ## Setup Requirements To use mTLS you need: 1. A Certificate Authority (CA) certificate configured in `tls.certificateAuthority`. 2. Client certificates signed by that CA, with a `CN` matching a Harper username. 3. The `http.mtls` configuration enabled. For help generating and managing certificates, see [Certificate Management](/reference/v5/security/certificate-management.md). ## Replication mTLS is always required for Harper replication and cannot be disabled. For replication-specific mTLS configuration, see [Replication Configuration](/reference/v5/replication/clustering.md). --- # Security Harper uses role-based, attribute-level security to ensure that users can only gain access to the data they are supposed to be able to access. Granular permissions allow for unparalleled flexibility and control, and can lower the total cost of ownership compared to other database solutions, since you no longer need to replicate subsets of data to isolate use cases. ## Security Philosophy Harper's security model has two distinct layers: **Authentication** determines *who* is making a request. Harper validates each request using one of the methods above, then resolves the caller to a known Harper user account. **Authorization** determines *what* the caller can do. Each Harper user is assigned a role. Roles carry a permissions set that grants or denies CRUD access at the table and attribute level, in addition to controlling access to system operations. For details on how roles and permissions work, see [Users and Roles](/reference/v5/users-and-roles/overview.md). ## Authentication Methods Harper supports three authentication methods: * [Basic Authentication](/reference/v5/security/basic-authentication.md) — Username and password sent as a Base64-encoded `Authorization` header on every request. * [JWT Authentication](/reference/v5/security/jwt-authentication.md) — Token-based authentication using JSON Web Tokens. Clients authenticate once and receive short-lived operation tokens and longer-lived refresh tokens. * [mTLS Authentication](/reference/v5/security/mtls-authentication.md) — Mutual TLS certificate-based authentication. ## Certificate Management * [Certificate Management](/reference/v5/security/certificate-management.md) — Managing TLS certificates and Certificate Authorities for HTTPS and mTLS. * [Certificate Verification](/reference/v5/security/certificate-verification.md) — Certificate revocation checking via CRL and OCSP. ## Access Control * CORS — Cross-Origin Resource Sharing. * For HTTP server configuration see [HTTP / Configuration / CORS](/reference/v5/http/configuration.md#cors) * For Operations API configuration see [Operations API / Configuration](/reference/v5/configuration/operations.md) * SSL & HTTPS — Enabling HTTPS and configuring TLS for the HTTP server. * For HTTP server configuration see [HTTP / Configuration / TLS](/reference/v5/http/tls.md) * For Operations API configuration see [Operations API / Configuration](/reference/v5/configuration/operations.md) * [Users and Roles](/reference/v5/users-and-roles/overview.md) — Role-Based Access Control (RBAC): defining roles, assigning permissions, and managing users. * [Impersonation](/reference/v5/security/impersonation.md) — Execute operations as a different user or role. ## API * [Security API](/reference/v5/security/api.md) — JavaScript globals for security operations (e.g. `auth()`). ## Default Behavior Out of the box, Harper: * Generates self-signed TLS certificates at `/keys/` on first run. * Runs with HTTPS disabled (HTTP only on port 9925 for the Operations API). It is recommended that you never directly expose Harper's HTTP interface through a publicly available port. * Enables CORS for all origins (configurable). * Supports Basic Auth and JWT Auth by default; mTLS must be explicitly configured. --- # Static Files * Added in: v4.5.0 * Changed in: v4.7.0 - (Migrated to Plugin API and new options added) The `static` built-in plugin serves static files from your Harper application over HTTP. Use it to host websites, SPAs, downloadable assets, or any static content alongside your Harper data and API endpoints. `static` does **not** need to be installed — it is built into Harper and only needs to be declared in your `config.yaml`. ## Basic Usage Configure `static` with the `files` option pointing to the files you want to serve: ``` static: files: 'site/**' ``` Given a component with this structure: ``` my-app/ ├─ site/ │ ├─ index.html │ ├─ about.html │ ├─ blog/ │ ├─ post-1.html │ ├─ post-2.html ├─ config.yaml ``` Files are accessed relative to the matched directory root, so `GET /index.html` returns `site/index.html` and `GET /blog/post-1.html` returns `site/blog/post-1.html`. ## `files` and `urlPath` Options *Added in: v4.5* `static` is a [Plugin](/reference/v5/components/overview.md) and supports the standard `files` and `urlPath` configuration options for controlling which files to serve and at what URL path. Use `urlPath` to mount the files at a specific URL prefix: ``` static: files: 'site/**' urlPath: 'app' ``` Now `GET /app/index.html` returns `site/index.html` and `GET /app/blog/post-1.html` returns `site/blog/post-1.html`. See [Components Overview](/reference/v5/components/overview.md) for full `files` glob pattern and `urlPath` documentation. ## Additional Options *Added in: v4.7* In addition to the standard `files`, `urlPath`, and `timeout` options, `static` supports these configuration options: * **`index`** - `boolean` - *optional* - If `true`, automatically serves `index.html` when a request targets a directory. Defaults to `false`. * **`extensions`** - `string[]` - *optional* - File extensions to try when an exact path match is not found. For example, `extensions: ['html']` means a request for `/page-1` will also try `/page-1.html`. * **`fallthrough`** - `boolean` - *optional* - If `true`, passes the request to the next handler when the requested file is not found. Set to `false` when using `notFound` to customize 404 responses. Defaults to `true`. * **`notFound`** - `string | { file: string; statusCode: number }` - *optional* - A custom file (or file + status code) to return when a path is not found. Useful for serving a custom 404 page or for SPAs that use client-side routing. ## Auto-Updates *Added in: v4.7.0* Because `static` uses the Plugin API, it automatically responds to changes without requiring a Harper restart. Adding, removing, or modifying files — or updating `config.yaml` — takes effect immediately. ## Examples ### Basic static file serving Serve all files in the `static/` directory. Requests must match file names exactly. ``` static: files: 'static/**' ``` ### Automatic `index.html` serving Serve `index.html` automatically when a request targets a directory: ``` static: files: 'static/**' index: true ``` With this structure: ``` my-app/ ├─ static/ │ ├─ index.html │ ├─ blog/ │ ├─ index.html │ ├─ post-1.html ``` Request mappings: ``` GET / -> static/index.html GET /blog -> static/blog/index.html GET /blog/post-1.html -> static/blog/post-1.html ``` ### Automatic extension matching Combine `index` and `extensions` for clean URLs without file extensions: ``` static: files: 'static/**' index: true extensions: ['html'] ``` Request mappings with the same structure: ``` GET / -> static/index.html GET /blog -> static/blog/index.html GET /blog/post-1 -> static/blog/post-1.html ``` ### Custom 404 page Return a specific file when a requested path is not found: ``` static: files: 'static/**' notFound: 'static/404.html' fallthrough: false ``` A request to `/non-existent` returns the contents of `static/404.html` with a `404` status code. > **Note:** When using `notFound`, set `fallthrough: false` so the request does not pass through to another handler before the custom 404 response is returned. ### SPA client-side routing For SPAs that handle routing in the browser, return the main application file for any unmatched path: ``` static: files: 'static/**' fallthrough: false notFound: file: 'static/index.html' statusCode: 200 ``` A request to any unmatched path returns `static/index.html` with a `200` status code, allowing the client-side router to handle navigation. ## Dynamic Applications The `static` plugin handles pure static file hosting. If you need server-side rendering, API routes, or other dynamic behavior, consider using a plugin designed for that purpose: * [`@harperfast/nextjs`](/reference/v5/components/nextjs.md) - Run a full Next.js application (SSR, ISR, API routes) directly on Harper ## Related * [Components Overview](/reference/v5/components/overview.md) --- # Local Studio * Added in: v4.1.0 * Changed in: v4.3.0 (Upgrade to match Cloud client) * Changed in: v4.7.0 (Upgraded to match Fabric client) Harper Local Studio is a web-based GUI that enables you to administer, navigate, and monitor your Harper instance through a simple, user-friendly interface without requiring knowledge of the underlying Harper APIs. It is automatically bundled with all Harper instances and is enabled by default on the Operations API port. If you're looking for the platform as a service interface, go to [Harper Fabric](https://fabric.harper.fast) instead. ## Configuration To enable the local Studio, set `localStudio.enabled` to `true` in your [configuration file](/reference/v5/configuration/options.md#localstudio): ``` localStudio: enabled: true ``` The local studio is provided by the [Operations API](/reference/v5/operations-api/overview.md) and is available on the configured `operationsApi.port` or `operationsApi.securePort` values. This is `9925` by default. ## Accessing Local Studio The local Studio can be accessed through your browser at: ``` http://localhost:9925 ``` All database interactions from the local Studio are made directly from your browser to your Harper instance. Authentication is maintained via session cookies. --- # Configuration ## Managing Roles with Config Files In addition to managing roles via the Operations API, Harper supports declaring roles in a configuration file. When the application starts, Harper ensures all declared roles exist with the specified permissions. Configure in your application's `config.yaml`: ``` roles: files: roles.yaml ``` Example `roles.yaml`: ``` analyst: super_user: false data: Sales: read: true insert: false update: false delete: false editor: data: Articles: read: true insert: true update: true attributes: title: read: true update: true author: read: true update: false ``` **Startup behavior:** * If a declared role does not exist, Harper creates it. * If a declared role already exists, Harper updates its permissions to match the definition. ## Password Hashing *Added in: v4.5.0* Harper supports two password hashing algorithms, replacing the previous MD5 hashing: * **`sha256`** — Default algorithm. Good security and excellent performance. * **`argon2id`** — Highest security. More CPU-intensive; recommended for high-security environments. Password hashing is configured via the `authentication.hashFunction` key in `harper-config.yaml`. See [Security / Configuration](/reference/v5/security/configuration.md#hashfunction) for details. ## Related * [Overview](/reference/v5/users-and-roles/overview.md) * [Operations](/reference/v5/users-and-roles/operations.md) --- # Operations ## Roles ### List Roles *Restricted to `super_user` roles.* ``` { "operation": "list_roles" } ``` ### Add Role *Restricted to `super_user` roles.* * `role` *(required)* — Name for the new role. * `permission` *(required)* — Permissions object. See [Permission Structure](/reference/v5/users-and-roles/overview.md#permission-structure). * `super_user` *(optional)* — If `true`, grants full access. Defaults to `false`. * `structure_user` *(optional)* — Boolean or array of database names. If `true`, can create/drop databases and tables. If array, limited to specified databases. * `operations` *(optional)* — Array of operation names and/or permission group names (`read_only`, `standard_user`) this role may call. When set, any unlisted operation is denied. See [Operation Permissions](/reference/v5/users-and-roles/overview.md#operation-permissions). ``` { "operation": "add_role", "role": "developer", "permission": { "super_user": false, "structure_user": false, "dev": { "tables": { "dog": { "read": true, "insert": true, "update": true, "delete": false, "attribute_permissions": [ { "attribute_name": "name", "read": true, "insert": true, "update": true } ] } } } } } ``` ### Alter Role *Restricted to `super_user` roles.* * `id` *(required)* — The `id` of the role to alter (from `list_roles`). * `role` *(optional)* — New name for the role. * `permission` *(required)* — Updated permissions object. ``` { "operation": "alter_role", "id": "f92162e2-cd17-450c-aae0-372a76859038", "role": "another_developer", "permission": { "super_user": false, "structure_user": false, "dev": { "tables": { "dog": { "read": true, "insert": true, "update": true, "delete": false, "attribute_permissions": [] } } } } } ``` ### Drop Role *Restricted to `super_user` roles. Roles with associated users cannot be dropped.* * `id` *(required)* — The `id` of the role to drop. ``` { "operation": "drop_role", "id": "developer" } ``` ## Users ### List Users *Restricted to `super_user` roles.* ``` { "operation": "list_users" } ``` ### User Info Returns user data for the currently authenticated user. Available to all roles. ``` { "operation": "user_info" } ``` ### Add User *Restricted to `super_user` roles.* * `role` *(required)* — Role name to assign. * `username` *(required)* — Username. Cannot be changed after creation. * `password` *(required)* — Plain-text password. Harper encrypts it on receipt. * `active` *(required)* — Boolean. If `false`, user cannot access Harper. ``` { "operation": "add_user", "role": "role_name", "username": "hdb_user", "password": "password", "active": true } ``` ### Alter User *Restricted to `super_user` roles.* * `username` *(required)* — Username to modify. * `password` *(optional)* — New password. * `role` *(optional)* — New role name. * `active` *(optional)* — Enable/disable user access. ``` { "operation": "alter_user", "role": "role_name", "username": "hdb_user", "password": "new_password", "active": true } ``` ### Drop User *Restricted to `super_user` roles.* ``` { "operation": "drop_user", "username": "harper" } ``` ## Related * [Overview](/reference/v5/users-and-roles/overview.md) * [Configuration](/reference/v5/users-and-roles/configuration.md) --- # Users & Roles Harper uses a Role-Based Access Control (RBAC) framework to manage access to Harper instances. Each user is assigned a role that determines their permissions to access database resources and run operations. ## Roles Role permissions in Harper are divided into two categories: **Database Manipulation** — CRUD (create, read, update, delete) permissions against database data (tables and attributes). **Database Definition** — Permissions to manage databases, tables, roles, users, and other system settings. These are restricted to the built-in `super_user` role. ### Built-In Roles | Role | Description | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `super_user` | Full access to all operations and methods. The admin role. | | `structure_user` | Access to create and delete databases and tables. Can be set to `true` (all databases) or an array of database names (specific databases only). | ### User-Defined Roles Admins (`super_user` users) can create custom roles with explicit permissions on specific tables and attributes. * Unless a user-defined role has `super_user: true`, all permissions must be defined explicitly. * Any table or database not included in the role's permission set will be inaccessible. * `describe` operations return metadata only for databases, tables, and attributes that the role has CRUD permissions for. ## Permission Structure When creating or altering a role, you define a `permission` object: ``` { "operation": "add_role", "role": "software_developer", "permission": { "super_user": false, "database_name": { "tables": { "table_name1": { "read": true, "insert": true, "update": true, "delete": false, "attribute_permissions": [ { "attribute_name": "attribute1", "read": true, "insert": true, "update": true } ] }, "table_name2": { "read": true, "insert": true, "update": true, "delete": false, "attribute_permissions": [] } } } } } ``` ### Operation Permissions The `operations` field in a permission object restricts which Operations API calls this role can make. When set, it acts as a two-gate check: 1. Only listed operations are reachable — any unlisted operation is denied regardless of table CRUD permissions. 2. For data operations that pass gate one, table-level CRUD permissions still apply as normal. Operations normally restricted to `super_user` can be selectively granted by including them in the list. If `operations` is not set, the role can call any non-`super_user` operation, subject to table CRUD permissions. **Permission Groups** Groups expand to a predefined set of operations and can be mixed with individual operation names: * `read_only` — Search, SQL SELECT, describe, and monitoring operations. No data modification. Operations: `search`, `search_by_conditions`, `search_by_hash`, `search_by_id`, `search_by_value`, `sql`, `describe_all`, `describe_schema`, `describe_database`, `describe_table`, `user_info`, `get_job`, `get_analytics`, `list_metrics`, `describe_metric` * `standard_user` — Everything in `read_only` plus full data manipulation and bulk load. Does not include any `super_user`-restricted operations, schema DDL (`create_attribute`), or token management. Additional operations beyond `read_only`: `insert`, `update`, `upsert`, `delete`, `csv_data_load`, `csv_file_load`, `csv_url_load`, `import_from_s3` **Example: read-only role** ``` { "operation": "add_role", "role": "read_only_analyst", "permission": { "operations": ["read_only"], "orders_db": { "tables": { "orders": { "read": true, "insert": false, "update": false, "delete": false, "attribute_permissions": [] } } } } } ``` **Example: full data access + targeted admin operations** ``` { "operation": "add_role", "role": "ops_engineer", "permission": { "operations": ["standard_user", "get_configuration", "system_information"], "orders_db": { "tables": { "orders": { "read": true, "insert": true, "update": true, "delete": true, "attribute_permissions": [] } } } } } ``` ### Table Permissions Each table entry defines CRUD access: ``` { "table_name": { "read": boolean, // Access to read from this table "insert": boolean, // Access to insert data "update": boolean, // Access to update data "delete": boolean, // Access to delete rows "attribute_permissions": [ { "attribute_name": "attribute_name", "read": boolean, "insert": boolean, "update": boolean // Note: "delete" is not an attribute-level permission } ] } } ``` ### Important Rules **Table-level:** * If a database or table is not included in the permissions, the role has no access to it. * If a table-level CRUD permission is `false`, setting the same CRUD permission to `true` on an attribute returns an error. **Attribute-level:** * If `attribute_permissions` is a non-empty array, only the listed attributes are accessible (plus the table's hash attribute — see below). * If `attribute_permissions` is empty (`[]`), attribute access follows the table-level CRUD permissions. * If any non-hash attribute is given CRUD access, the table's `hash_attribute` (primary key) automatically receives the same access, even if not explicitly listed. * Any attribute not explicitly listed in a non-empty `attribute_permissions` array has no access. * `DELETE` is not an attribute-level permission. Deleting rows is controlled at the table level. * The `__createdtime__` and `__updatedtime__` attributes managed by Harper can have `read` permissions set; other attribute-level permissions for these fields are ignored. ## Role-Based Operation Restrictions ### Databases and Tables | Operation | Restricted to Super User | | ------------------- | ------------------------ | | `describe_all` | | | `describe_database` | | | `describe_table` | | | `create_database` | X | | `drop_database` | X | | `create_table` | X | | `drop_table` | X | | `create_attribute` | | | `drop_attribute` | X | ### NoSQL Operations | Operation | Restricted to Super User | | ---------------------- | ------------------------ | | `insert` | | | `update` | | | `upsert` | | | `delete` | | | `search_by_hash` | | | `search_by_value` | | | `search_by_conditions` | | ### SQL Operations | Operation | Restricted to Super User | | --------- | ------------------------ | | `select` | | | `insert` | | | `update` | | | `delete` | | ### Bulk Operations | Operation | Restricted to Super User | | ---------------- | ------------------------ | | `csv_data_load` | | | `csv_file_load` | | | `csv_url_load` | | | `import_from_s3` | | ### Users and Roles | Operation | Restricted to Super User | | ------------ | ------------------------ | | `list_roles` | X | | `add_role` | X | | `alter_role` | X | | `drop_role` | X | | `list_users` | X | | `user_info` | | | `add_user` | X | | `alter_user` | X | | `drop_user` | X | ### Clustering | Operation | Restricted to Super User | | ----------------------- | ------------------------ | | `cluster_set_routes` | X | | `cluster_get_routes` | X | | `cluster_delete_routes` | X | | `add_node` | X | | `update_node` | X | | `cluster_status` | X | | `remove_node` | X | | `configure_cluster` | X | ### Components | Operation | Restricted to Super User | | -------------------- | ------------------------ | | `get_components` | X | | `get_component_file` | X | | `set_component_file` | X | | `drop_component` | X | | `add_component` | X | | `package_component` | X | | `deploy_component` | X | ### Registration | Operation | Restricted to Super User | | ------------------- | ------------------------ | | `registration_info` | | | `get_fingerprint` | X | | `set_license` | X | ### Jobs | Operation | Restricted to Super User | | --------------------------- | ------------------------ | | `get_job` | | | `search_jobs_by_start_date` | X | ### Logs | Operation | Restricted to Super User | | -------------------------------- | ------------------------ | | `read_log` | X | | `read_transaction_log` | X | | `delete_transaction_logs_before` | X | | `read_audit_log` | X | | `delete_audit_logs_before` | X | ### Utilities | Operation | Restricted to Super User | | ----------------------- | ------------------------ | | `delete_records_before` | X | | `export_local` | X | | `export_to_s3` | X | | `system_information` | X | | `restart` | X | | `restart_service` | X | | `get_configuration` | X | ### Token Authentication | Operation | Restricted to Super User | | ------------------------------ | ------------------------ | | `create_authentication_tokens` | | | `refresh_operation_token` | | ## Troubleshooting: "Must execute as User" If you see the error `Error: Must execute as <>`, it means Harper was installed as a specific OS user and must be run by that same user. Harper stores files natively on the operating system and only allows the Harper executable to be run by a single user — this prevents file permission issues and keeps the installation secure. To resolve: run Harper with the same OS user account used during installation. ## Related * [Configuration](/reference/v5/users-and-roles/configuration.md) * [Operations](/reference/v5/users-and-roles/operations.md) --- # Release Notes ## Current Release - Version 4 (Tucker) [Meet Tucker](/release-notes/v4-tucker/tucker.md) Our 4th Release Pup 1. [4.7.33 Tucker](/release-notes/v4-tucker/4.7.33.md) 2. [4.7.32 Tucker](/release-notes/v4-tucker/4.7.32.md) 3. [4.7.31 Tucker](/release-notes/v4-tucker/4.7.31.md) 4. [4.7.30 Tucker](/release-notes/v4-tucker/4.7.30.md) 5. [4.7.29 Tucker](/release-notes/v4-tucker/4.7.29.md) 6. [4.7.28 Tucker](/release-notes/v4-tucker/4.7.28.md) 7. [4.7.27 Tucker](/release-notes/v4-tucker/4.7.27.md) 8. [4.7.26 Tucker](/release-notes/v4-tucker/4.7.26.md) 9. [4.7.25 Tucker](/release-notes/v4-tucker/4.7.25.md) 10. [4.7.24 Tucker](/release-notes/v4-tucker/4.7.24.md) 11. [4.7.23 Tucker](/release-notes/v4-tucker/4.7.23.md) 12. [4.7.22 Tucker](/release-notes/v4-tucker/4.7.22.md) 13. [4.7.21 Tucker](/release-notes/v4-tucker/4.7.21.md) 14. [4.7.20 Tucker](/release-notes/v4-tucker/4.7.20.md) 15. [4.7.19 Tucker](/release-notes/v4-tucker/4.7.19.md) 16. [4.7.18 Tucker](/release-notes/v4-tucker/4.7.18.md) 17. [4.7.17 Tucker](/release-notes/v4-tucker/4.7.17.md) 18. [4.7.16 Tucker](/release-notes/v4-tucker/4.7.16.md) 19. [4.7.15 Tucker](/release-notes/v4-tucker/4.7.15.md) 20. [4.7.14 Tucker](/release-notes/v4-tucker/4.7.14.md) 21. [4.7.13 Tucker](/release-notes/v4-tucker/4.7.13.md) 22. [4.7.12 Tucker](/release-notes/v4-tucker/4.7.12.md) 23. [4.7.11 Tucker](/release-notes/v4-tucker/4.7.11.md) 24. [4.7.10 Tucker](/release-notes/v4-tucker/4.7.10.md) 25. [4.7.9 Tucker](/release-notes/v4-tucker/4.7.9.md) 26. [4.7.8 Tucker](/release-notes/v4-tucker/4.7.8.md) 27. [4.7.7 Tucker](/release-notes/v4-tucker/4.7.7.md) 28. [4.7.6 Tucker](/release-notes/v4-tucker/4.7.6.md) 29. [4.7.5 Tucker](/release-notes/v4-tucker/4.7.5.md) 30. [4.7.4 Tucker](/release-notes/v4-tucker/4.7.4.md) 31. [4.7.3 Tucker](/release-notes/v4-tucker/4.7.3.md) 32. [4.7.2 Tucker](/release-notes/v4-tucker/4.7.2.md) 33. [4.7.1 Tucker](/release-notes/v4-tucker/4.7.1.md) 34. [4.7.0 Tucker](/release-notes/v4-tucker/4.7.0.md) 35. [4.6.26 Tucker](/release-notes/v4-tucker/4.6.26.md) 36. [4.6.25 Tucker](/release-notes/v4-tucker/4.6.25.md) 37. [4.6.24 Tucker](/release-notes/v4-tucker/4.6.24.md) 38. [4.6.23 Tucker](/release-notes/v4-tucker/4.6.23.md) 39. [4.6.22 Tucker](/release-notes/v4-tucker/4.6.22.md) 40. [4.6.21 Tucker](/release-notes/v4-tucker/4.6.21.md) 41. [4.6.20 Tucker](/release-notes/v4-tucker/4.6.20.md) 42. [4.6.19 Tucker](/release-notes/v4-tucker/4.6.19.md) 43. [4.6.18 Tucker](/release-notes/v4-tucker/4.6.18.md) 44. [4.6.17 Tucker](/release-notes/v4-tucker/4.6.17.md) 45. [4.6.16 Tucker](/release-notes/v4-tucker/4.6.16.md) 46. [4.6.15 Tucker](/release-notes/v4-tucker/4.6.15.md) 47. [4.6.14 Tucker](/release-notes/v4-tucker/4.6.14.md) 48. [4.6.13 Tucker](/release-notes/v4-tucker/4.6.13.md) 49. [4.6.12 Tucker](/release-notes/v4-tucker/4.6.12.md) 50. [4.6.11 Tucker](/release-notes/v4-tucker/4.6.11.md) 51. [4.6.10 Tucker](/release-notes/v4-tucker/4.6.10.md) 52. [4.6.9 Tucker](/release-notes/v4-tucker/4.6.9.md) 53. [4.6.8 Tucker](/release-notes/v4-tucker/4.6.8.md) 54. [4.6.7 Tucker](/release-notes/v4-tucker/4.6.7.md) 55. [4.6.6 Tucker](/release-notes/v4-tucker/4.6.6.md) 56. [4.6.5 Tucker](/release-notes/v4-tucker/4.6.5.md) 57. [4.6.4 Tucker](/release-notes/v4-tucker/4.6.4.md) 58. [4.6.3 Tucker](/release-notes/v4-tucker/4.6.3.md) 59. [4.6.2 Tucker](/release-notes/v4-tucker/4.6.2.md) 60. [4.6.1 Tucker](/release-notes/v4-tucker/4.6.1.md) 61. [4.6.0 Tucker](/release-notes/v4-tucker/4.6.0.md) 62. [4.5.42 Tucker](/release-notes/v4-tucker/4.5.42.md) 63. [4.5.41 Tucker](/release-notes/v4-tucker/4.5.41.md) 64. [4.5.40 Tucker](/release-notes/v4-tucker/4.5.40.md) 65. [4.5.39 Tucker](/release-notes/v4-tucker/4.5.39.md) 66. [4.5.38 Tucker](/release-notes/v4-tucker/4.5.38.md) 67. [4.5.37 Tucker](/release-notes/v4-tucker/4.5.37.md) 68. [4.5.36 Tucker](/release-notes/v4-tucker/4.5.36.md) 69. [4.5.35 Tucker](/release-notes/v4-tucker/4.5.35.md) 70. [4.5.34 Tucker](/release-notes/v4-tucker/4.5.34.md) 71. [4.5.33 Tucker](/release-notes/v4-tucker/4.5.33.md) 72. [4.5.32 Tucker](/release-notes/v4-tucker/4.5.32.md) 73. [4.5.31 Tucker](/release-notes/v4-tucker/4.5.31.md) 74. [4.5.30 Tucker](/release-notes/v4-tucker/4.5.30.md) 75. [4.5.29 Tucker](/release-notes/v4-tucker/4.5.29.md) 76. [4.5.28 Tucker](/release-notes/v4-tucker/4.5.28.md) 77. [4.5.27 Tucker](/release-notes/v4-tucker/4.5.27.md) 78. [4.5.26 Tucker](/release-notes/v4-tucker/4.5.26.md) 79. [4.5.25 Tucker](/release-notes/v4-tucker/4.5.25.md) 80. [4.5.24 Tucker](/release-notes/v4-tucker/4.5.24.md) 81. [4.5.23 Tucker](/release-notes/v4-tucker/4.5.23.md) 82. [4.5.22 Tucker](/release-notes/v4-tucker/4.5.22.md) 83. [4.5.21 Tucker](/release-notes/v4-tucker/4.5.21.md) 84. [4.5.20 Tucker](/release-notes/v4-tucker/4.5.20.md) 85. [4.5.19 Tucker](/release-notes/v4-tucker/4.5.19.md) 86. [4.5.18 Tucker](/release-notes/v4-tucker/4.5.18.md) 87. [4.5.17 Tucker](/release-notes/v4-tucker/4.5.17.md) 88. [4.5.16 Tucker](/release-notes/v4-tucker/4.5.16.md) 89. [4.5.15 Tucker](/release-notes/v4-tucker/4.5.15.md) 90. [4.5.14 Tucker](/release-notes/v4-tucker/4.5.14.md) 91. [4.5.13 Tucker](/release-notes/v4-tucker/4.5.13.md) 92. [4.5.12 Tucker](/release-notes/v4-tucker/4.5.12.md) 93. [4.5.11 Tucker](/release-notes/v4-tucker/4.5.11.md) 94. [4.5.10 Tucker](/release-notes/v4-tucker/4.5.10.md) 95. [4.5.9 Tucker](/release-notes/v4-tucker/4.5.9.md) 96. [4.5.8 Tucker](/release-notes/v4-tucker/4.5.8.md) 97. [4.5.7 Tucker](/release-notes/v4-tucker/4.5.7.md) 98. [4.5.6 Tucker](/release-notes/v4-tucker/4.5.6.md) 99. [4.5.5 Tucker](/release-notes/v4-tucker/4.5.5.md) 100. [4.5.4 Tucker](/release-notes/v4-tucker/4.5.4.md) 101. [4.5.3 Tucker](/release-notes/v4-tucker/4.5.3.md) 102. [4.5.2 Tucker](/release-notes/v4-tucker/4.5.2.md) 103. [4.5.1 Tucker](/release-notes/v4-tucker/4.5.1.md) 104. [4.5.0 Tucker](/release-notes/v4-tucker/4.5.0.md) 105. [4.4.24 Tucker](/release-notes/v4-tucker/4.4.24.md) 106. [4.4.23 Tucker](/release-notes/v4-tucker/4.4.23.md) 107. [4.4.22 Tucker](/release-notes/v4-tucker/4.4.22.md) 108. [4.4.21 Tucker](/release-notes/v4-tucker/4.4.21.md) 109. [4.4.20 Tucker](/release-notes/v4-tucker/4.4.20.md) 110. [4.4.19 Tucker](/release-notes/v4-tucker/4.4.19.md) 111. [4.4.18 Tucker](/release-notes/v4-tucker/4.4.18.md) 112. [4.4.17 Tucker](/release-notes/v4-tucker/4.4.17.md) 113. [4.4.16 Tucker](/release-notes/v4-tucker/4.4.16.md) 114. [4.4.15 Tucker](/release-notes/v4-tucker/4.4.15.md) 115. [4.4.14 Tucker](/release-notes/v4-tucker/4.4.14.md) 116. [4.4.13 Tucker](/release-notes/v4-tucker/4.4.13.md) 117. [4.4.12 Tucker](/release-notes/v4-tucker/4.4.12.md) 118. [4.4.11 Tucker](/release-notes/v4-tucker/4.4.11.md) 119. [4.4.10 Tucker](/release-notes/v4-tucker/4.4.10.md) 120. [4.4.9 Tucker](/release-notes/v4-tucker/4.4.9.md) 121. [4.4.8 Tucker](/release-notes/v4-tucker/4.4.8.md) 122. [4.4.7 Tucker](/release-notes/v4-tucker/4.4.7.md) 123. [4.4.6 Tucker](/release-notes/v4-tucker/4.4.6.md) 124. [4.4.5 Tucker](/release-notes/v4-tucker/4.4.5.md) 125. [4.4.4 Tucker](/release-notes/v4-tucker/4.4.4.md) 126. [4.4.3 Tucker](/release-notes/v4-tucker/4.4.3.md) 127. [4.4.2 Tucker](/release-notes/v4-tucker/4.4.2.md) 128. [4.4.1 Tucker](/release-notes/v4-tucker/4.4.1.md) 129. [4.4.0 Tucker](/release-notes/v4-tucker/4.4.0.md) 130. [4.3.38 Tucker](/release-notes/v4-tucker/4.3.38.md) 131. [4.3.37 Tucker](/release-notes/v4-tucker/4.3.37.md) 132. [4.3.36 Tucker](/release-notes/v4-tucker/4.3.36.md) 133. [4.3.35 Tucker](/release-notes/v4-tucker/4.3.35.md) 134. [4.3.34 Tucker](/release-notes/v4-tucker/4.3.34.md) 135. [4.3.33 Tucker](/release-notes/v4-tucker/4.3.33.md) 136. [4.3.32 Tucker](/release-notes/v4-tucker/4.3.32.md) 137. [4.3.31 Tucker](/release-notes/v4-tucker/4.3.31.md) 138. [4.3.30 Tucker](/release-notes/v4-tucker/4.3.30.md) 139. [4.3.29 Tucker](/release-notes/v4-tucker/4.3.29.md) 140. [4.3.28 Tucker](/release-notes/v4-tucker/4.3.28.md) 141. [4.3.27 Tucker](/release-notes/v4-tucker/4.3.27.md) 142. [4.3.26 Tucker](/release-notes/v4-tucker/4.3.26.md) 143. [4.3.25 Tucker](/release-notes/v4-tucker/4.3.25.md) 144. [4.3.24 Tucker](/release-notes/v4-tucker/4.3.24.md) 145. [4.3.23 Tucker](/release-notes/v4-tucker/4.3.23.md) 146. [4.3.22 Tucker](/release-notes/v4-tucker/4.3.22.md) 147. [4.3.21 Tucker](/release-notes/v4-tucker/4.3.21.md) 148. [4.3.20 Tucker](/release-notes/v4-tucker/4.3.20.md) 149. [4.3.19 Tucker](/release-notes/v4-tucker/4.3.19.md) 150. [4.3.18 Tucker](/release-notes/v4-tucker/4.3.18.md) 151. [4.3.17 Tucker](/release-notes/v4-tucker/4.3.17.md) 152. [4.3.16 Tucker](/release-notes/v4-tucker/4.3.16.md) 153. [4.3.15 Tucker](/release-notes/v4-tucker/4.3.15.md) 154. [4.3.14 Tucker](/release-notes/v4-tucker/4.3.14.md) 155. [4.3.13 Tucker](/release-notes/v4-tucker/4.3.13.md) 156. [4.3.12 Tucker](/release-notes/v4-tucker/4.3.12.md) 157. [4.3.11 Tucker](/release-notes/v4-tucker/4.3.11.md) 158. [4.3.10 Tucker](/release-notes/v4-tucker/4.3.10.md) 159. [4.3.9 Tucker](/release-notes/v4-tucker/4.3.9.md) 160. [4.3.8 Tucker](/release-notes/v4-tucker/4.3.8.md) 161. [4.3.7 Tucker](/release-notes/v4-tucker/4.3.7.md) 162. [4.3.6 Tucker](/release-notes/v4-tucker/4.3.6.md) 163. [4.3.5 Tucker](/release-notes/v4-tucker/4.3.5.md) 164. [4.3.4 Tucker](/release-notes/v4-tucker/4.3.4.md) 165. [4.3.3 Tucker](/release-notes/v4-tucker/4.3.3.md) 166. [4.3.2 Tucker](/release-notes/v4-tucker/4.3.2.md) 167. [4.3.1 Tucker](/release-notes/v4-tucker/4.3.1.md) 168. [4.3.0 Tucker](/release-notes/v4-tucker/4.3.0.md) 169. [4.2.8 Tucker](/release-notes/v4-tucker/4.2.8.md) 170. [4.2.7 Tucker](/release-notes/v4-tucker/4.2.7.md) 171. [4.2.6 Tucker](/release-notes/v4-tucker/4.2.6.md) 172. [4.2.5 Tucker](/release-notes/v4-tucker/4.2.5.md) 173. [4.2.4 Tucker](/release-notes/v4-tucker/4.2.4.md) 174. [4.2.3 Tucker](/release-notes/v4-tucker/4.2.3.md) 175. [4.2.2 Tucker](/release-notes/v4-tucker/4.2.2.md) 176. [4.2.1 Tucker](/release-notes/v4-tucker/4.2.1.md) 177. [4.2.0 Tucker](/release-notes/v4-tucker/4.2.0.md) 178. [4.1.2 Tucker](/release-notes/v4-tucker/4.1.2.md) 179. [4.1.1 Tucker](/release-notes/v4-tucker/4.1.1.md) 180. [4.1.0 Tucker](/release-notes/v4-tucker/4.1.0.md) 181. [4.0.7 Tucker](/release-notes/v4-tucker/4.0.7.md) 182. [4.0.6 Tucker](/release-notes/v4-tucker/4.0.6.md) 183. [4.0.5 Tucker](/release-notes/v4-tucker/4.0.5.md) 184. [4.0.4 Tucker](/release-notes/v4-tucker/4.0.4.md) 185. [4.0.3 Tucker](/release-notes/v4-tucker/4.0.3.md) 186. [4.0.2 Tucker](/release-notes/v4-tucker/4.0.2.md) 187. [4.0.1 Tucker](/release-notes/v4-tucker/4.0.1.md) 188. [4.0.0 Tucker](/release-notes/v4-tucker/4.0.0.md) ## Previous Major Releases ### Version 3 - Monkey [Meet Monkey](/release-notes/v3-monkey.md) Our 3rd Release Pup 1. [3.3.0 Monkey](/release-notes/v3-monkey/3.3.0.md) 2. [3.2.1 Monkey](/release-notes/v3-monkey/3.2.1.md) 3. [3.2.0 Monkey](/release-notes/v3-monkey/3.2.0.md) 4. [3.1.5 Monkey](/release-notes/v3-monkey/3.1.5.md) 5. [3.1.4 Monkey](/release-notes/v3-monkey/3.1.4.md) 6. [3.1.3 Monkey](/release-notes/v3-monkey/3.1.3.md) 7. [3.1.2 Monkey](/release-notes/v3-monkey/3.1.2.md) 8. [3.1.1 Monkey](/release-notes/v3-monkey/3.1.1.md) 9. [3.1.0 Monkey](/release-notes/v3-monkey/3.1.0.md) 10. [3.0.0 Monkey](/release-notes/v3-monkey/3.0.0.md) ### Version 2 - Penny [Meet Penny](/release-notes/v2-penny.md) Our 2nd Release Pup 1. [2.3.1 Penny](/release-notes/v2-penny/2.3.1.md) 2. [2.3.0 Penny](/release-notes/v2-penny/2.3.0.md) 3. [2.2.3 Penny](/release-notes/v2-penny/2.2.3.md) 4. [2.2.2 Penny](/release-notes/v2-penny/2.2.2.md) 5. [2.2.0 Penny](/release-notes/v2-penny/2.2.0.md) 6. [2.1.1 Penny](/release-notes/v2-penny/2.1.1.md) ### Version 1 - Alby [Meet Alby](/release-notes/v1-alby.md) Our 1st Release Pup 1. [1.3.1 Alby](/release-notes/v1-alby/1.3.1.md) 2. [1.3.0 Alby](/release-notes/v1-alby/1.3.0.md) 3. [1.2.0 Alby](/release-notes/v1-alby/1.2.0.md) 4. [1.1.0 Alby](/release-notes/v1-alby/1.1.0.md) --- # HarperDB Alby (Version 1) Did you know our release names are dedicated to employee pups? For our first release, Alby was our pup. Here is a bit about Alby: ![picture of black dog](/assets/images/alby-b4d0a7b708faff57aaf8973e41c180d2.webp) *Hi, I am Alby. My mom is Kaylan Stock, Director of Marketing at HarperDB. I am a 9-year-old Great Dane mix who loves sun bathing, going for swims, and wreaking havoc on the local squirrels. My favorite snack is whatever you are eating, and I love a good butt scratch!* --- # 1.1.0 ### HarperDB 1.1.0, Alby Release 4/18/2018 **Features** * Users & Roles: * Limit/Assign access to all HarperDB operations * Limit/Assign access to schemas, tables & attributes * Limit/Assign access to specific SQL operations (`INSERT`, `UPDATE`, `DELETE`, `SELECT`) * Enhanced SQL parser * Added extensive ANSI SQL Support. * Added Array function, which allows for converting relational data into Object/Hierarchical data * `Distinct_Array` Function: allows for removing duplicates in the Array function. * Enhanced SQL Validation: Improved validation around structure of SQL, validating the schema, etc.. * 10x performance improvement on SQL statements. * Export Function: can now call a NoSQL/SQL search and have it export to CSV or JSON. * Added upgrade function to CLI * Added ability to perform bulk update from CSV * Created landing page for HarperDB. * Added CORS support to HarperDB **Fixes** * Fixed memory leak in CSV bulk loads * Corrected error when attempting to perform a `SQL DELETE` * Added further validation to NoSQL `UPDATE` to validate schema & table exist * Fixed install issue occurring when part of the install path does not exist, the install would silently fail. * Fixed issues with replicated data when one of the replicas is down * Removed logging of initial user’s credentials during install * Can now use reserved words as aliases in SQL * Removed user(s) password in results when calling `list_users` * Corrected forwarding of operations to other nodes in a cluster * Corrected lag in schema meta-data passing to other nodes in a cluster * Drop table & schema now move the table & schema or table to the trash folder under the Database folder for later permanent deletion. * Bulk inserts no longer halt the entire operation if n records already exist, instead the return includes the hashes of records that have been skipped. * Added ability to accept EULA from command line * Corrected `search_by_value` not searching on the correct attribute * Added ability to increase the timeout of a request by adding `SERVER_TIMEOUT_MS` to config/settings.js * Add error handling resulting from SQL calculations. * Standardized error responses as JSON. * Corrected internal process generation to not allow more processes than machine has cores. --- # 1.2.0 ### HarperDB 1.2.0, Alby Release 7/10/2018 **Features** * Time to Live: Conserve the resources of your edge device by setting data on devices to live for a specific period of time. * Geo: HarperDB has implemented turf.js into its SQL parser to enable geo based analytics. * Jobs: CSV Data loads, Exports & Time to Live now all run as back ground jobs. * Exports: Perform queries that export into JSON or CSV and save to disk or S3. **Fixes** * Fixed issue where CSV data loads incorrectly report number of records loaded. * Added validation to stop `BETWEEN` operations in SQL. * Updated logging to not include internal variables in the logs. * Cleaned up `add_role` response to not include internal variables. * Removed old and unused dependencies. * Build out further unit tests and integration tests. * Fixed https to handle certificates properly. * Improved stability of clustering & replication. * Corrected issue where Objects and Arrays were not casting properly in `SQL SELECT` response. * Fixed issue where Blob text was not being returned from `SQL SELECT`s. * Fixed error being returned when querying on table with no data, now correctly returns empty array. * Improved performance in SQL when searching on exact values. * Fixed error when ./harperdb stop is called. * Fixed logging issue causing instability in installer. * Fixed `read_log` operation to accept date time. * Added permissions checking to `export_to_s3`. * Added ability to run SQL on `SELECT` without a `FROM`. * Fixed issue where updating a user’s password was not encrypting properly. * Fixed `user_guide.html` to point to readme on git repo. * Created option to have HarperDB run as a foreground process. * Updated `user_info` to return the correct role for a user. * Fixed issue where HarperDB would not stop if the database root was deleted. * Corrected error message on insert if an invalid schema is provided. * Added permissions checks for user & role operations. --- # 1.3.0 ### HarperDB 1.3.0, Alby Release 11/2/2018 **Features** * Upgrade: Upgrade to newest version via command line. * SQL Support: Added `IS NULL` for SQL parser. * Added attribute validation to search operations. **Fixes** * Fixed `SELECT` calculations, i.e. `SELECT` 2+2. * Fixed select OR not returning expected results. * No longer allowing reserved words for schema and table names. * Corrected process interruptions from improper SQL statements. * Improved message handling between spawned processes that replace killed processes. * Enhanced error handling for updates to tables that do not exist. * Fixed error handling for NoSQL responses when `get_attributes` is provided with invalid attributes. * Fixed issue with new columns not being updated properly in update statements. * Now validating roles, tables and attributes when creating or updating roles. * Fixed an issue where in some cases `undefined` was being returned after dropping a role --- # 1.3.1 ### HarperDB 1.3.1, Alby Release 2/26/2019 **Features** * Clustering connection direction appointment * Foundations for threading/multi processing * UUID autogen for hash attributes that were not provided * Added cluster status operation **Bug Fixes and Enhancements** * More logging * Clustering communication enhancements * Clustering queue ordering by timestamps * Cluster re connection enhancements * Number of system core(s) detection * Node LTS (10.15) compatibility * Update/Alter users enhancements * General performance enhancements * Warning is logged if different versions of harperdb are connected via clustering * Fixed need to restart after user creation/alteration * Fixed SQL error that occurred on selecting from an empty table --- # HarperDB Penny (Version 2) Did you know our release names are dedicated to employee pups? For our second release, Penny was the star. Here is a bit about Penny: ![picture of brindle dog](/assets/images/penny-88847803c8c6215b8392fd131d4c1838.webp) *Hi I am Penny! My dad is Kyle Bernhardy, the CTO of HarperDB. I am a nine-year-old Whippet who lives for running hard and fast while exploring the beautiful terrain of Colorado. My favorite activity is chasing birds along with afternoon snoozes in a sunny spot in my backyard.* --- # 2.1.1 ### HarperDB 2.1.1, Penny Release 05/22/2020 **Highlights** * CORE-1007 Added the ability to perform `SQL INSERT` & `UPDATE` with function calls & expressions on values. * CORE-1023 Fixed minor bug in final SQL step incorrectly trying to translate ordinals to alias in `ORDER BY` statement. * CORE-1020 Fixed bug allowing 'null' and 'undefined' string values to be passed in as valid hash values. * CORE-1006 Added SQL functionality that enables `JOIN` statements across different schemas. * CORE-1005 Implemented JSONata library to handle our JSON document search functionality in SQL, creating the `SEARCH_JSON` function. * CORE-1009 Updated schema validation to allow all printable ASCII characters to be used in schema/table/attribute names, except, forward slashes and backticks. Same rules apply now for hash attribute values. * CORE-1003 Fixed handling of ORDER BY statements with function aliases. * CORE-1004 Fixed bug related to `SELECT*` on `JOIN` queries with table columns with the same name. * CORE-996 Fixed an issue where the `transact_to_cluster` flag is lost for CSV URL loads, fixed an issue where new attributes created in CSV bulk load do not sync to the cluster. * CORE-994 Added new operation `system_information`. This operation returns info & metrics for the OS, time, memory, cpu, disk, network. * CORE-993 Added new custom date functions for AlaSQL & UTC updates. * CORE-991 Changed jobs to spawn a new process which will run the intended job without impacting a main HarperDB process. * CORE-992 HTTPS enabled by default. * CORE-990 Updated `describe_table` to add the record count for the table for LMDB data storage. * CORE-989 Killed the socket cluster processes prior to HarperDB processes to eliminate a false uptime. * CORE-975 Updated time values set by SQL Date Functions to be in epoch format. * CORE-974 Added date functions to `SQL SELECT` column alias functionality. --- # 2.2.0 ### HarperDB 2.2.0, Penny Release 08/24/2020 **Features/Updates** * CORE-997 Updated the data format for CSV data loads being sync'd across a cluster to take up less resources * CORE-1018 Adds SQL functionality for `BETWEEN` statements * CORE-1032 Updates permissions to allow regular users (i.e. non-super users) to call the `get_job` operation * CORE-1036 On create/drop table we auto create/drop the related transactions environments for the schema.table * CORE-1042 Built raw functions to write to a tables transaction log for insert/update/delete operations * CORE-1057 Implemented write transaction into lmdb create/update/delete functions * CORE-1048 Adds `SEARCH` wildcard handling for role permissions standards * CORE-1059 Added config setting to disable transaction logging for an instance * CORE-1076 Adds permissions filter to describe operations * CORE-1043 Change clustering catchup to use the new transaction log * CORE-1052 Removed word "master" from source * CORE-1061 Added new operation called `delete_transactions_before` this will tail a transaction log for a specific schema / table * CORE-1040 On HarperDB startup make sure all tables have a transaction environment * CORE-1055 Added 2 new setting to change the server headersTimeout & keepAliveTimeout from the config file * CORE-1044 Created new operation `read_transaction_log` which will allow a user to get transactions for a table by `timestamp`, `username`, or `hash_value` * CORE-1043 Change clustering catchup to use the new transaction log * CORE-1089 Added new attribute to `system_information` for table/transaction log data size in bytes & transaction log record count * CORE-1101 Fix to store empty strings rather than considering them null & fix to be able to search on empty strings in SQL/NoSQL. * CORE-1054 Updates permissions object to remove delete attribute permission and update table attribute permission key to `attribute_permissions` * CORE-1092 Do not allow the `__createdtime__` to be updated * CORE-1085 Updates create schema/table & drop schema/table/attribute operations permissions to require super user role and adds integration tests to validate * CORE-1071 Updates response messages and status codes from `describe_schema` and `describe_table` operations to provide standard language/status code when a schema item is not found * CORE-1049 Updates response message for SQL update op with no matching rows * CORE-1096 Added tracking of the origin in the transaction log. This origin object stores the node name, timestamp of the transaction from the originating node & the user. **Bug Fixes** * CORE-1028 Fixes bug for simple `SQL SELECT` queries not returning aliases and incorrectly returning hash values when not requested in query * CORE-1037 Fixed an issue where numbers with leading zero i.e. 00123 are converted to numbers rather than being honored as strings. * CORE-1063 Updates permission error response shape to consolidate issues into individual objects per schema/table combo * CORE-1098 Fixed an issue where transaction environments were remaining in the global cache after being dropped. * CORE-1086 Fixed issue where responses from insert/update were incorrect with skipped records. * CORE-1079 Fixes SQL bugs around invalid schema/table and special characters in `WHERE` clause --- # 2.2.2 ### HarperDB 2.2.2, Penny Release 10/27/2020 * CORE-1154 Allowed transaction logging to be disabled even if clustering is enabled. * CORE-1153 Fixed issue where `delete_files_before` was writing to transaction log. * CORE-1152 Fixed issue where no more than 4 HarperDB forks would be created. * CORE-1112 Adds handling for system timestamp attributes in permissions. * CORE-1131 Adds better handling for checking perms on operations with action value in JSON. * CORE-1113 Fixes validation bug checking for super user/cluster user permissions and other permissions. * CORE-1135 Adds validation for valid keys in role API operations. * CORE-1073 Adds new `import_from_s3` operation to API. --- # 2.2.3 ### HarperDB 2.2.3, Penny Release 11/16/2020 * CORE-1158 Performance improvements to core delete function and configuration of `delete_files_before` to run in batches with a pause into between. --- # 2.3.0 ### HarperDB 2.3.0, Penny Release 12/03/2020 **Features/Updates** * CORE-1191, CORE-1190, CORE-1125, CORE-1157, CORE-1126, CORE-1140, CORE-1134, CORE-1123, CORE-1124, CORE-1122 Added JWT Authentication option (See documentation for more information) * CORE-1128, CORE-1143, CORE-1140, CORE-1129 Added `upsert` operation * CORE-1187 Added `get_configuration` operation which allows admins to view their configuration settings. * CORE-1175 Added new internal LMDB function to copy an environment for use in future features. * CORE-1166 Updated packages to address security vulnerabilities. **Bug Fixes** * CORE-1195 Modified `drop_attribute` to drop after data cleanse completes. * CORE-1149 Fix SQL bug regarding self joins and updates alasql to 0.6.5 release. * CORE-1168 Fix inconsistent invalid schema/table errors. * CORE-1162 Fix bug which caused `delete_files_before` to cause tables to grow in size due to an open cursor issue. --- # 2.3.1 ### HarperDB 2.3.1, Penny Release 1/29/2021 **Bug Fixes** * CORE-1218 A bug in HarperDB 2.3.0 was identified related to manually calling the `create_attribute` operation. This bug caused secondary indexes to be overwritten by the most recently inserted or updated value for the index, thereby causing a search operation filtered with that index to only return the most recently inserted/updated row. Note, this issue does not affect attributes that are reflexively/automatically created. It only affects attributes created using `create_attribute`. To resolve this issue in 2.3.0 or earlier, drop and recreate your table using reflexive attribute creation. In 2.3.1, drop and recreate your table and use either reflexive attribute creation or `create_attribute`. * CORE-1219 Increased maximum table attributes from 1000 to 10000 --- # HarperDB Monkey (Version 3) Did you know our release names are dedicated to employee pups? For our third release, we have Monkey. ![picture of tan dog](/assets/images/monkey-127579f8b29e5300b5cf6b111ca4c76f.webp) *Hi, I am Monkey, a.k.a. Monk, a.k.a. Monchichi. My dad is Aron Johnson, the Director of DevOps at HarperDB. I am an eight-year-old Australian Cattle dog mutt whose favorite pastime is hunting and collecting tennis balls from the park next to her home. I love burrowing in the Colorado snow, rolling in the cool grass on warm days, and cheese!* --- # 3.0.0 ### HarperDB 3.0, Monkey Release 5/18/2021 **Features/Updates** * CORE-1217, CORE-1226, CORE-1232 Create new `search_by_conditions` operation. * CORE-1304 Upgrade to Node 12.22.1. * CORE-1235 Adds new upgrade/install functionality. * CORE-1206, CORE-1248, CORE-1252 Implement `lmdb-store` library for optimized performance. * CORE-1062 Added alias operation for `delete_files_before`, named `delete_records_before`. * CORE-1243 Change `HTTPS_ON` settings value to false by default. * CORE-1189 Implement fastify web server, resulting in improved performance. * CORE-1221 Update user API to use role name instead of role id. * CORE-1225 Updated dependencies to eliminate npm security warnings. * CORE-1241 Adds 3.0 update directive and refactors/fixes update functionality. **Bug Fixes** * CORE-1299 Remove all references to the `PROJECT_DIR` setting. This setting is problematic when using node version managers and upgrading the version of node and then installing a new instance of HarperDB. * CORE-1288 Fix bug with drop table/schema that was causing 'env required' error log. * CORE-1285 Update warning log when trying to create an attribute that already exists. * CORE-1254 Added logic to manage data collisions in clustering. * CORE-1212 Add pre-check to `drop_user` that returns error if user doesn't exist. * CORE-1114 Update response code and message from `add_user` when user already exists. * CORE-1111 Update response from `create_attribute` to match the create schema/table response. * CORE-1205 Fixed bug that prevented schema/table from being dropped if name was a number or had a wildcard value in it. Updated validation for insert, upsert and update. --- # 3.1.0 ### HarperDB 3.1.0, Monkey Release 8/24/2021 **Features/Updates** * CORE-1320, CORE-1321, CORE-1323, CORE-1324 Version 1.0 of HarperDB Custom Functions * CORE-1275, CORE-1276, CORE-1278, CORE-1279, CORE-1280, CORE-1282, CORE-1283, CORE-1305, CORE-1314 IPC server for communication between HarperDB processes, including HarperDB, HarperDB Clustering, and HarperDB Functions * CORE-1352, CORE-1355, CORE-1356, CORE-1358 Implement pm2 for HarperDB process management * CORE-1292, CORE-1308, CORE-1312, CORE-1334, CORE-1338 Updated installation process to start HarperDB immediately on install and to accept all config settings via environment variable or command line arguments * CORE-1310 Updated licensing functionality * CORE-1301 Updated validation for performance improvement * CORE-1359 Add `hdb-response-time` header which returns the HarperDB response time in milliseconds * CORE-1330, CORE-1309 New config settings: `LOG_TO_FILE`, `LOG_TO_STDSTREAMS`, `IPC_SERVER_PORT`, `RUN_IN_FOREGROUND`, `CUSTOM_FUNCTIONS`, `CUSTOM_FUNCTIONS_PORT`, `CUSTOM_FUNCTIONS_DIRECTORY`, `MAX_CUSTOM_FUNCTION_PROCESSES` **Bug Fixes** * CORE-1315 Corrected issue in HarperDB restart scenario * CORE-1370 Update some of the validation error handlers so that they don't log full stack --- # 3.1.1 ### HarperDB 3.1.1, Monkey Release 9/23/2021 **Features/Updates** * CORE-1393 Added utility function to add settings from env/cmd vars to the settings file on every run/restart * CORE-1395 Create a setting which will allow to enable the local Studio to be served from an instance of HarperDB * CORE-1397 Update the stock 404 response to not return the request URL * General updates to optimize Docker container **Bug Fixes** * CORE-1399 Added fixes for complex SQL alias issues --- # 3.1.2 ### HarperDB 3.1.2, Monkey Release 10/21/2021 **Features/Updates** * Updated the installation ASCII art to reflect the new HarperDB logo **Bug Fixes** * CORE-1408 Corrects issue where `drop_attribute` was not properly setting the LMDB version number causing tables to behave unexpectedly --- # 3.1.3 ### HarperDB 3.1.3, Monkey Release 1/14/2022 **Bug Fixes** * CORE-1446 Fix for scans on indexes larger than 1 million entries causing queries to never return --- # 3.1.4 ### HarperDB 3.1.4, Monkey Release 2/24/2022 **Features/Updates** * CORE-1460 Added new setting `STORAGE_WRITE_ASYNC`. If this setting is true, LMDB will have faster write performance at the expense of not being crash safe. The default for this setting is false, which results in HarperDB being crash safe. --- # 3.1.5 ### HarperDB 3.1.5, Monkey Release 3/4/2022 **Features/Updates** * CORE-1498 Fixed incorrect autocasting of string that start with "0." that tries to convert to number but instead returns NaN. --- # 3.2.0 ### HarperDB 3.2.0, Monkey Release 3/25/2022 **Features/Updates** * CORE-1391 Bug fix related to orphaned HarperDB background processes. * CORE-1509 Updated node version check, updated Node.js version, updated project dependencies. * CORE-1518 Remove final call from logger. --- # 3.2.1 ### HarperDB 3.2.1, Monkey Release 6/1/2022 **Features/Updates** * CORE-1573 Added logic to track the pid of the foreground process if running in foreground. Then on stop, use that pid to kill the process. Logic was also added to kill the pm2 daemon when stop is called. --- # 3.3.0 ### HarperDB 3.3.0 - Monkey * CORE-1595 Added new role type `structure_user`, this enables non-superusers to be able to create/drop schema/table/attribute. * CORE-1501 Improved performance for drop\_table. * CORE-1599 Added two new operations for custom functions `install_node_modules` & `audit_node_modules`. * CORE-1598 Added `skip_node_modules` flag to `package_custom_function_project` operation. This flag allows for not bundling project dependencies and deploying a smaller project to other nodes. Use this flag in tandem with `install_node_modules`. * CORE-1707 Binaries are now included for Linux on AMD64, Linux on ARM64, and macOS. GCC, Make, Python are no longer required when installing on these platforms. --- # Harper Tucker (Version 4) HarperDB version 4 ([Tucker release](/release-notes/v4-tucker/tucker.md)) represents major step forward in database technology. This release line has ground-breaking architectural advancements including: ## [4.7](/release-notes/v4-tucker/4.7.33.md) * Component status monitoring for tracking loading and status changes across all components * OCSP support for TLS certificate validation in replication and HTTP connections * New analytics and licensing functionality for Fabric services integration * Further improvements to the plugin API ## [4.6](/release-notes/v4-tucker/4.6.26.md) * Vector Indexing - 4.6 introduces a new Vector Indexing system based on Hierarchical Navigable Small World Graphs. * New extension API - 4.6 introduces a new extension API for creating extensions components. * Improved logging configurability - Logging can be dynamically updated and specifically configured for each component. * Resource API - 4.6 has updated Resource APIs for ease of use. * Data loader - 4.6 introduces a new data loader that allows for ensuring records exist as part of a component. ## [4.5](/release-notes/v4-tucker/4.5.42.md) * Blob Storage - 4.5 introduces a new [Blob storage system](/reference/v4/database/schema.md#blob-type). * Password Hashing Upgrade - two new password hashing algorithms for better security (to replace md5). * New resource and storage Analytics ## [4.4](/release-notes/v4-tucker/4.4.24.md) * Native replication (codename "Plexus") which is faster, more efficient, secure, and reliable than the previous replication system and provides provisional sharding capabilities with a foundation for the future * Computed properties that allow applications to define properties that are computed from other properties, allowing for composite properties that are calculated from other data stored in records without requiring actual storage of the computed value * Custom indexing including composite, full-text indexing, and vector indexing ## [4.3](/release-notes/v4-tucker/4.3.38.md) * Relationships, joins, and broad new querying capabilities for complex and nested conditions, sorting, joining, and selecting with significant query optimizations * More advanced transaction support for CRDTs and storage of large integers (with BigInt) * Better management with new upgraded local studio and new CLI features ## [4.2](/release-notes/v4-tucker/4.2.8.md) * New component architecture and Resource API for advanced, robust custom database application development * Real-time capabilites through MQTT, WebSockets, and Server-Sent Events * REST interface for intuitive, fast, and standards-compliant HTTP interaction * Native caching capabilities for high-performance cache scenarios * Clone node functionality ## [4.1](/release-notes/v4-tucker/4.1.2.md) * New streaming iterators mechanism that allows query results to be delivered to clients *while* querying results are being processed, for incredibly fast time-to-first-byte and concurrent processing/delivery * New thread-based concurrency model for more efficient resource usage ## [4.0](/release-notes/v4-tucker/4.0.7.md) * New clustering technology that delivers robust, resilient and high-performance replication * Major storage improvements with highly-efficient adaptive-structure modified MessagePack format, with on-demand deserialization capabilities Did you know our release names are dedicated to employee pups? For our fourth release, [meet Tucker!](/release-notes/v4-tucker/tucker.md) --- # 4.0.0 ### HarperDB 4.0.0, Tucker Release 11/2/2022 **Networking & Data Replication (Clustering)** The HarperDB clustering internals have been rewritten and the underlying technology for Clustering has been completely replaced with [NATS](https://nats.io/), an enterprise grade connective technology responsible for addressing, discovery and exchanging of messages that drive the common patterns in distributed systems. * CORE-1464, CORE-1470, : Remove SocketCluster dependencies and all code related to them. * CORE-1465, CORE-1485, CORE-1537, CORE-1538, CORE-1558, CORE-1583, CORE\_1665, CORE-1710, CORE-1801, CORE-1865 :Add nats-`server` code as dependency, on install of HarperDB download nats-`server` is possible else fallback to building from source code. * CORE-1593, CORE-1761: Add `nats.js` as project dependency. * CORE-1466: Build NATS configs on `harperdb run` based on HarperDB YAML configuration. * CORE-1467, CORE-1508: Launch and manage NATS servers with PM2. * CORE-1468, CORE-1507: Create a process which reads the work queue stream and processes transactions. * CORE-1481, CORE-1529, CORE-1698, CORE-1502, CORE-1696: On upgrade to 4.0, update pre-existing clustering configurations, create table transaction streams, create work queue stream, update `hdb_nodes` table, create clustering folder structure, and rebuild self-signed certs. * CORE-1494, CORE-1521, CORE-1755: Build out internals to interface with NATS. * CORE-1504: Update existing hooks to save transactions to work with NATS. * CORE-1514, CORE-1515, CORE-1516, CORE-1527, CORE-1532: Update `add_node`, `update_node`, and `remove_node` operations to no longer need host and port in payload. These operations now manage dynamically sourcing of table level transaction streams between nodes and work queues. * CORE-1522: Create `NATSReplyService` process which handles the receiving NATS based requests from remote instances and sending back appropriate responses. * CORE-1471, CORE-1568, CORE-1563, CORE-1534, CORE-1569: Update `cluster_status` operation. * CORE-1611: Update pre-existing transaction log operations to be audit log operations. * CORE-1541, CORE-1612, CORE-1613: Create translation log operations which interface with streams. * CORE-1668: Update NATS serialization / deserialization to use MessagePack. * CORE-1673: Add `system_info` param to `hdb_nodes` table and update on `add_node` and `cluster_status`. * CORE-1477, CORE-1493, CORE-1557, CORE-1596, CORE-1577: Both a full HarperDB restart & just clustering restart call the NATS server with a reload directive to maintain full uptime while servers refresh. * CORE-1474 :HarperDB install adds clustering folder structure. * CORE-1530: Post `drop_table` HarperDB purges the related transaction stream. * CORE-1567: Set NATS config to always use TLS. * CORE-1543: Removed the `transact_to_cluster` attribute from the bulk load operations. Now bulk loads always replicate. * CORE-1533, CORE-1556, CORE-1561, CORE-1562, CORE-1564: New operation `configure_cluster`, this operation enables bulk publishing and subscription of multiple tables to multiple instances of HarperDB. * CORE-1535: Create work queue stream on install of HarperDB. This stream receives transactions from remote instances of HarperDB which are then ingested in order. * CORE-1551: Create transaction streams on the remote node if they do not exist when performing `add_node` or `update_node`. * CORE-1594, CORE-1605, CORE-1749, CORE-1767, CORE-1770: Optimize the work queue stream and its consumer to be more performant and validate exact once delivery. * CORE-1621, CORE-1692, CORE-1570, CORE-1693: NATS stream names are MD5 hashed to avoid characters that HarperDB allows, but NATS may not. * CORE-1762: Add a new optional attribute to `add_node` and `update_node` named `opt_start_time`. This attribute sets a starting time to start synchronizing transactions. * CORE-1785: Optimizations and bug fixes in regards to sourcing data from remote instances on HarperDB. * CORE-1588: Created new operation `set_cluster_routes` to enable setting routes for instances of HarperDB to mesh together. * CORE-1589: Created new operation `get_cluster_routes` to allow for retrieval of routes used to connect the instance of HarperDB to the mesh. * CORE-1590: Created new operation `delete_cluster_routes` to allow for removal of routes used to connect the instance of HarperDB to the mesh. * CORE-1667: Fix old environment variable `CLUSTERING_PORT` not mapping to new hub server port. * CORE-1609: Allow `remove_node` to be called when the other node cannot be reached. * CORE-1815: Add transaction lock to `add_node` and `update_node` to avoid concurrent nats source update bug. * CORE-1848: Update stream configs if the node name has been changed in the YAML configuration. * CORE-1873: Update `add_node` and `update_node` so that it auto-creates schema/table on both local and remote node respectively **Data Storage** We have made improvements to how we store, index, and retrieve data. * CORE-1619: Enabled new concurrent flushing technology for improved write performance. * CORE-1701: Optimize search performance for `search_by_conditions` when executing multiple AND conditions. * CORE-1652: Encode the values of secondary indices more efficiently for faster access. * CORE-1670: Store updated timestamp in `lmdb.js`' version property. * CORE-1651: Enabled multiple value indexing of array values which allows for the ability to search on specific elements in an array more efficiently. * CORE-1649, CORE-1659: Large text values (larger than 255 bytes) are no longer stored in separate blob index. Now they are segmented and delimited in the same index to increase search performance. * Complex objects and object arrays are no longer stored in a separate index to preserve storage and increase write throughput. * CORE-1650, CORE-1724, CORE-1738: Improved internals around interpreting attribute values. * CORE-1657: Deferred property decoding allows large objects to be stored, but individual attributes can be accessed (like with get\_attributes) without incurring the cost of decoding the entire object. * CORE-1658: Enable in-memory caching of records for even faster access to frequently accessed data. * CORE-1693: Wrap updates in async transactions to ensure ACID-compliant updates. * CORE-1653: Upgrade to 4.0 rebuilds tables to reflect changes made to index improvements. * CORE-1753: Removed old `node-lmdb` dependency. * CORE-1787: Freeze objects returned from queries. * CORE-1821: Read the `WRITE_ASYNC` setting which enables LMDB nosync. **Logging** HarperDB has increased logging specificity by breaking out logs based on components logging. There are specific log files each for HarperDB Core, Custom Functions, Hub Server, Leaf Server, and more. * CORE-1497: Remove `pino` and `winston` dependencies. * CORE-1426: All logging is output via `stdout` and `stderr`, our default logging is then picked up by PM2 which handles writing out to file. * CORE-1431: Improved `read_log` operation validation. * CORE-1433, CORE-1463: Added log rotation. * CORE-1553, CORE-1555, CORE-1552, CORE-1554, CORE-1704: Performance gain by only serializing objects and arrays if the log is for the level defined in configuration. * CORE-1436: Upgrade to 4.0 updates internals for logging changes. * CORE-1428, CORE-1440, CORE-1442, CORE-1434, CORE-1435, CORE-1439, CORE-1482, CORE-1751, CORE-1752: Bug fixes, performance improvements and improved unit tests. * CORE-1691: Convert non-PM2 managed log file writes to use Node.js `fs.appendFileSync` function. **Configuration** HarperDB has updated its configuration from a properties file to YAML. * CORE-1448, CORE-1449, CORE-1519, CORE-1587: Upgrade automatically converts the pre-existing settings file to YAML. * CORE-1445, CORE-1534, CORE-1444, CORE-1858: Build out new logic to create, update, and interpret the YAML configuration file. * Installer has updated prompts to reflect YAML settings. * CORE-1447: Create an alias for the `configure_cluster` operation as `set_configuration`. * CORE-1461, CORE-1462, CORE-1483: Unit test improvements. * CORE-1492: Improvements to get\_configuration and set\_configuration operations. * CORE-1503: Modify HarperDB configuration for more granular certificate definition. * CORE-1591: Update `routes` IP param to `host` and to `leaf` config in `harperdb.conf` * CORE-1519: Fix issue when switching between old and new versions of HarperDB we are getting the config parameter is undefined error on npm install. **Broad NodeJS and Platform Support** * CORE-1624: HarperDB can now run on multiple versions of NodeJS, from v14 to v19. We primarily test on v18, so that is the preferred version. **Windows 10 and 11** * CORE-1088: HarperDB now runs natively on Windows 10 and 11 without the need to run in a container or installed in WSL. Windows is only intended for evaluation and development purposes, not for production work loads. **Extra Changes and Bug Fixes** * CORE-1520: Refactor installer to remove all waterfall code and update to use Promises. * CORE-1573: Stop the PM2 daemon and any logging processes when stopping hdb. * CORE-1586: When HarperDB is running in foreground stop any additional logging processes from being spawned. * CORE-1626: Update docker file to accommodate new `harperdb.conf` file. * CORE-1592, CORE-1526, CORE-1660, CORE-1646, CORE-1640, CORE-1689, CORE-1711, CORE-1601, CORE-1726, CORE-1728, CORE-1736, CORE-1735, CORE-1745, CORE-1729, CORE-1748, CORE-1644, CORE-1750, CORE-1757, CORE-1727, CORE-1740, CORE-1730, CORE-1777, CORE-1778, CORE-1782, CORE-1775, CORE-1771, CORE-1774, CORE-1759, CORE-1772, CORE-1861, CORE-1862, CORE-1863, CORE-1870, CORE-1869 :Changes for CI/CD pipeline and integration tests. * CORE-1661: Fixed issue where old boot properties file caused an error when attempting to install 4.0.0. * CORE-1697, CORE-1814, CORE-1855: Upgrade fastify dependency to new major version 4. * CORE-1629: Jobs are now running as processes managed by the PM2 daemon. * CORE-1733: Update LICENSE to reflect our EULA on our site. * CORE-1606: Enable Custom Functions by default. * CORE-1714: Include pre-built binaries for most common platforms (darwin-arm64, darwin-x64, linux-arm64, linux-x64, win32-x64). * CORE-1628: Fix issue where setting license through environment variable not working. * CORE-1602, CORE-1760, CORE-1838, CORE-1839, CORE-1847, CORE-1773: HarperDB Docker container improvements. * CORE-1706: Add support for encoding HTTP responses with MessagePack. * CORE-1709: Improve the way lmdb.js dependencies are installed. * CORE-1758: Remove/update unnecessary HTTP headers. * CORE-1756: On `npm install` and `harperdb install` change the node version check from an error to a warning if the installed Node.js version does not match our preferred version. * CORE-1791: Optimizations to authenticated user caching. * CORE-1794: Update README to discuss Windows support & Node.js versions * CORE-1837: Fix issue where Custom Function directory was not being created on install. * CORE-1742: Add more validation to audit log - check schema/table exists and log is enabled. * CORE-1768: Fix issue where when running in foreground HarperDB process is not stopping on `harperdb stop`. * CORE-1864: Fix to semver checks on upgrade. * CORE-1850: Fix issue where a `cluster_user` type role could not be altered. --- # 4.0.1 ### HarperDB 4.0.1, Tucker Release 01/20/2023 **Bug Fixes** * CORE-1992 Local studio was not loading because the path got mangled in the build. * CORE-2001 Fixed deploy\_custom\_function\_project after node update broke it. --- # 4.0.2 ### HarperDB 4.0.2, Tucker Release 01/24/2023 **Bug Fixes** * CORE-2003 Fix bug where if machine had one core thread config would default to zero. * Update to lmdb 2.7.3 and msgpackr 1.7.0 --- # 4.0.3 ### HarperDB 4.0.3, Tucker Release 01/26/2023 **Bug Fixes** * CORE-2007 Add update nodes 4.0.0 launch script to build script to fix clustering upgrade. --- # 4.0.4 ### HarperDB 4.0.4, Tucker Release 01/27/2023 **Bug Fixes** * CORE-2009 Fixed bug where add node was not being called when upgrading clustering. --- # 4.0.5 ### HarperDB 4.0.5, Tucker Release 02/15/2023 **Bug Fixes** * CORE-2029 Improved the upgrade process for handling existing user TLS certificates and correctly configuring TLS settings. Added a prompt to upgrade to determine if new certificates should be created or existing certificates should be kept/used. * Fix the way NATS connections are honored in a local environment. * Do not define the certificate authority path to NATS if it is not defined in the HarperDB config. --- # 4.0.6 ### HarperDB 4.0.6, Tucker Release 03/09/2023 **Bug Fixes** * Fixed a data serialization error that occurs when a large number of different record structures are persisted in a single table. --- # 4.0.7 ### HarperDB 4.0.7, Tucker Release 03/10/2023 **Bug Fixes** * Update lmdb.js dependency --- # 4.1.0 HarperDB 4.1 introduces the ability to use worker threads for concurrently handling HTTP requests. Previously this was handled by processes. This shift provides important benefits in terms of better control of traffic delegation with support for optimized load tracking and session affinity, better debuggability, and reduced memory footprint. This means debugging will be much easier for custom functions. If you install/run HarperDB locally, most modern IDEs like WebStorm and VSCode support worker thread debugging, so you can start HarperDB in your IDE, and set breakpoints in your custom functions and debug them. The associated routing functionality now includes session affinity support. This can be used to consistently route users to the same thread which can improve caching locality, performance, and fairness. This can be enabled in with the [`http.sessionAffinity` option in your configuration](/reference/v4/configuration/options.md#http). HarperDB 4.1's NoSQL query handling has been revamped to consistently use iterators, which provide an extremely memory efficient mechanism for directly streaming query results to the network *as* the query results are computed. This results in faster Time to First Byte (TTFB) (only the first record/value in a query needs to be computed before data can start to be sent), and less memory usage during querying (the entire query result does not need to be stored in memory). These iterators are also available in query results for custom functions and can provide means for custom function code to iteratively access data from the database without loading entire results. This should be a completely transparent upgrade, all HTTP APIs function the same, with the one exception that custom functions need to be aware that they can't access query results by `[index]` (they should use array methods or for-in loops to handle query results). 4.1 includes configuration options for specifying the location of database storage files. This allows you to specifically locate database directories and files on different volumes for better flexibility and utilization of disks and storage volumes. See the [storage configuration](/reference/v4/configuration/options.md#storage) and [schemas configuration](/reference/v4/database/schema.md) for information on how to configure these locations. Logging has been revamped and condensed into one `hdb.log` file. See [logging](/reference/v4/logging/overview.md) for more information. A new operation called `cluster_network` was added, this operation will ping the cluster and return a list of enmeshed nodes. Custom Functions will no longer automatically load static file routes, instead the `@fastify/static` plugin will need to be registered with the Custom Function server. See [Host A Static Web UI-static](/reference/v4/legacy/custom-functions.md). Updates to S3 import and export mean that these operations now require the bucket `region` in the request. Also, if referencing a nested object it should be done in the `key` parameter. See examples [here](https://api.harperdb.io/#aa74bbdf-668c-4536-80f1-b91bb13e5024). Due to the AWS SDK v2 reaching end of life support we have updated to v3. This has caused some breaking changes in our operations `import_from_s3` and `export_to_s3`: * A new attribute `region` will need to be supplied * The `bucket` attribute can no longer have trailing slashes. Slashes will now need to be in the `key`. Starting HarperDB without any command (just `harperdb`) now runs HarperDB like a standard process, in the foreground. This means you can use standard unix tooling for interacting with the process and is conducive for running HarperDB with systemd or any other process management tool. If you wish to have HarperDB launch itself in separate background process (and immediately terminate the shell process), you can do so by running `harperdb start`. Internal Tickets completed: * CORE-609 - Ensure that attribute names are always added to global schema as Strings * CORE-1549 - Remove fastify-static code from Custom Functions server which auto serves content from "static" folder * CORE-1655 - Iterator based queries * CORE-1764 - Fix issue where describe\_all operation returns an empty object for non super-users if schema(s) do not yet have table(s) * CORE-1854 - Switch to using worker threads instead of processes for handling concurrency * CORE-1877 - Extend the csv\_url\_load operation to allow for additional headers to be passed to the remote server when the csv is being downloaded * CORE-1893 - Add last updated timestamp to describe operations * CORE-1896 - Fix issue where Select \* from system.hdb\_info returns wrong HDB version number after Instance Upgrade * CORE-1904 - Fix issue when executing GEOJSON query in SQL * CORE-1905 - Add HarperDB YAML configuration setting which defines the storage location of NATS streams * CORE-1906 - Add HarperDB YAML configuration setting defining the storage location of tables. * CORE-1655 - Streaming binary format serialization * CORE-1943 - Add configuration option to set mount point for audit tables * CORE-1921 - Update NATS transaction lifecycle to handle message deduplication in work queue streams. * CORE-1963 - Update logging for better readability, reduced duplication, and request context information. * CORE-1968 - In server\nats\natsIngestService.js remove the js\_msg.working(); line to improve performance. * CORE-1976 - Fix error when calling describe\_table operation with no schema or table defined in payload. * CORE-1983 - Fix issue where create\_attribute operation does not validate request for required attributes * CORE-2015 - Remove PM2 logs that get logged in console when starting HDB * CORE-2048 - systemd script for 4.1 * CORE-2052 - Include thread information in system\_information for visibility of threads * CORE-2061 - Add a better error msg when clustering is enabled without a cluster user set * CORE-2068 - Create new log rotate logic since pm2 log-rotate no longer used * CORE-2072 - Update to Node 18.15.0 * CORE-2090 - Upgrade Testing from v4.0.x and v3.x to v4.1. * CORE-2091 - Run the performance tests * CORE-2092 - Allow for automatic patch version updates of certain packages * CORE-2109 - Add verify option to clustering TLS configuration * CORE-2111 - Update AWS SDK to v3 --- # 4.1.1 06/16/2023 * HarperDB uses improved logic for determining default heap limits and thread counts. When running in a restricted container and on NodeJS 18.15+, HarperDB will use the constrained memory limit to determine heap limits for each thread. In more memory constrained servers with many CPU cores, a reduced default thread count will be used to ensure that excessive memory is not used by many workers. You may still define your own thread count (with `http`/`threads`) in the [configuration](/reference/v4/configuration/overview.md). * An option has been added for [disabling the republishing NATS messages](/reference/v4/configuration/overview.md), which can provide improved replication performance in a fully connected network. * Improvements to our OpenShift container. * Dependency security updates. * **Bug Fixes** * Fixed a bug in reporting database metrics in the `system_information` operation. --- # 4.1.2 ### HarperDB 4.1.2, Tucker Release 06/16/2023 * HarperDB has updated binary dependencies to support older glibc versions back 2.17. * A new CLI command was added to get the current status of whether HarperDB is running and the cluster status. This is available with `harperdb status`. * Improvements to our OpenShift container. * Dependency security updates. --- # 4.2.0 #### HarperDB 4.2.0 HarperDB 4.2 introduces a new interface to accessing our core database engine with faster access, well-typed idiomatic JavaScript interfaces, ergonomic object mapping, and real-time data subscriptions. 4.2 also had adopted a new component architecture for building extensions to deliver customized external data sources, authentication, file handlers, content types, and more. These architectural upgrades lead to several key new HarperDB capabilities including a new REST interface, advanced caching, real-time messaging and publish/subscribe functionality through MQTT, WebSockets, and Server-Sent Events. 4.2 also introduces configurable database schemas using GraphQL Schema syntax. The new component structure is also configuration-driven, providing easy, low-code paths to building applications. [Check out our new getting starting guide](/learn.md) to see how easy it is to get started with HarperDB apps. ### Resource API The [Resource API](/reference/v4/resources/overview.md) is the new interface for accessing data in HarperDB. It utilizes a uniform interface for accessing data in HarperDB database/tables and is designed to easily be implemented or extended for defining customized application logic for table access or defining custom external data sources. This API has support for connecting resources together for caching and delivering data change and message notifications in real-time. The [Resource API documentation details this interface](/reference/v4/resources/overview.md). ### Component Architecture HarperDB's custom functions have evolved towards a full component architecture; our internal functionality is defined as components, and this can be used in a modular way in conjunction with user components. These can all easily be configured and loaded through configuration files, and there is now a [well-defined interface for creating your own components](/reference/v4/components/extension-api.md). Components can easily be deployed/installed into HarperDB using [NPM and Github references as well](/reference/v4/components/applications.md). ### Configurable Database Schemas HarperDB applications or components support [schema definitions using GraphQL schema syntax](/reference/v4/database/schema.md). This makes it easy to define your table and attribute structure and gives you control over which attributes should be indexed and what types they should be. With schemas in configuration, these schemas can be bundled with an application and deployed together with application code. ### REST Interface HarperDB 4.2 introduces a new REST interface for accessing data through best-practice HTTP APIs using intuitive paths and standards-based methods and headers that directly map to our Resource API. This new interface provides fast and easy access to data via queries through GET requests, modifications of data through PUTs, customized actions through POSTs and more. With standards-based header support built-in, this works seamlessly with external caches (including browser caches) for accelerated performance and reduced network transfers. ### Real-Time HarperDB 4.2 now provides standard interfaces for subscribing to data changes and receiving notifications of changes and messages in real-time. Using these new real-time messaging capabilities with structured data provides a powerful integrated platform for both database style data updates and querying along with message delivery. [Real-time messaging](/reference/v4/rest/websockets.md) of data is available through several protocols: #### MQTT 4.2 now includes MQTT support which is a publish and subscribe messaging protocol, designed for efficiency (designed to be efficient enough for even small Internet of Things devices). This allows clients to connect to HarperDB and publish messages through our data center and subscribe to messages and data for real-time delivery. 4.2 implements support for QoS 0 and 1, along with durable sessions. #### WebSockets HarperDB now also supports WebSockets. This can be used as a transport for MQTT or as a connection for custom connection handling. #### Server-Sent Events HarperDB also includes support for Server-Sent Events. This is a very easy-to-use browser API that allows web sites/applications to connect to HarperDB and subscribe to data changes with minimal effort over standard HTTP. ### Database Structure HarperDB databases contain a collection of tables, and these tables are now contained in a single transactionally-consistent database file. This means reads and writes can be performed transactionally and atomically across tables (as long as they are in the same database). Multi-table transactions are replicated as single atomic transactions as well. Audit logs are also maintained in the same database with atomic consistency as well. Databases are now entirely encapsulated in a file, which means they can be moved/copied to another database without requiring any separate metadata updates in the system tables. ### Clone Node HarperDB includes new functionality for adding new HarperDB nodes in a cluster. New instances can be configured to clone from a leader node, performing and copying a database snapshot from a leader node, and self-configuring from the leader node as well, to facilitate accelerated deployment of new nodes for fast horizontal scaling to meet demand needs. See the replication documentation for more information on node management. ### Operations API terminology updates Any operation that used the `schema` property was updated to make this property optional and alternately support `database` as the property for specifying the database (formerly 'schema'). If both `schema` and `database` are absent, operation defaults to using the `data` database. Term 'primary key' now used in place of 'hash'. noSQL operation `search_by_hash` updated to `search_by_id`. Support was added for defining a table with `primary_key` instead of `hash_attribute`. ## Configuration There have been significant changes to `harperdb-config.yaml`, however none of these changes should affect pre-4.2 versions. If you upgrade to 4.2 any existing configuration should be backwards compatible and will not need to be updated. `harperdb-config.yaml` has had some configuration values added, removed, renamed and defaults changed. Please refer to [harperdb-config.yaml](/reference/v4/configuration/overview.md) for the most current configuration parameters. * The `http` element has been expanded. * `compressionThreshold` was added. * All `customFunction` configuration now lives here, except for the `tls` section. * `threads` has moved out of the `http` element and now is its own top level element. * `authentication` section was moved out of the `operationsApi` section and is now its own top level element/section. * `analytics.aggregatePeriod` was added. * Default logging level was changed to `warn`. * Default clustering log level was changed to `info`. * `clustering.republishMessages` now defaults to `false`. * `operationsApi.foreground` was removed. To start HarperDB in the foreground, from the CLI run `harperdb`. * Made `operationsApi` configuration optional. Any config not defined here will default to the `http` section. * Added a `securePort` parameter to `operationsApi` and `http` used for setting the https port. * Added a new top level `tls` section. * Removed `customFunctions.enabled`, `customFunctions.network.https`, `operationsApi.network.https` and `operationsApi.nodeEnv`. * Added an element called `componentRoot` which replaces `customFunctions.root`. * Updated custom pathing to use `databases` instead of `schemas`. * Added `logging.auditAuthEvents.logFailed` and `logging.auditAuthEvents.logSuccessful` for enabling logging of auth events. * A new `mqtt` section was added. ### Socket Management HarperDB now uses socket sharing to distribute incoming connections to different threads (`SO_REUSEPORT`). This is considered to be the most performant mechanism available for multi-threaded socket handling. This does mean that we have deprecated session-affinity based socket delegation. HarperDB now also supports more flexible port configurations: application endpoints and WebSockets run on 9926 by default, but these can be separated, or application endpoints can be configured to run on the same port as the operations API for a single port configuration. ### Sessions HarperDB now supports cookie-based sessions for authentication for web clients. This can be used with the standard authentication mechanisms to login, and then cookies can be used to preserve the authenticated session. This is generally a more secure way of maintaining authentication in browsers, without having to rely on storing credentials. ### Dev Mode HarperDB can now directly run a HarperDB application from any location using `harperdb run /path/to/app` or `harperdb dev /path/to/app`. The latter starts in dev mode, with logging directly to the console, debugging enabled, and auto-restarting with any changes in your application files. Dev mode is recommended for local application and component development. --- # 4.2.1 ### HarperDB 4.2.1, Tucker Release 11/3/2023 * Downgrade NATS 2.10.3 back to 2.10.1 due to regression in connection handling. * Handle package names with underscores. * Improved validation of queries and comparators * Avoid double replication on transactions with multiple commits * Added file metadata on get\_component\_file --- # 4.2.2 ### HarperDB 4.2.2, Tucker Release 11/8/2023 * Increase timeouts for NATS connections. * Fix for database snapshots for backups (and for clone node). * Fix application of permissions for default tables exposed through REST. * Log replication failures with record information. * Fix application of authorization/permissions for MQTT commands. * Fix copying of local components in clone node. * Fix calculation of overlapping start time in clone node. --- # 4.2.3 ### HarperDB 4.2.3, Tucker Release 11/15/2023 * When setting setting securePort, disable unsecure port setting on same port * Fix `harperdb status` when pid file is missing * Fix/include missing icons/fonts from local studio * Fix crash that can occur when concurrently accessing records > 16KB * Apply a lower heap limit to better ensure that memory leaks are quickly caught/mitigated --- # 4.2.4 ### HarperDB 4.2.4, Tucker Release 11/16/2023 * Prevent coercion of strings to numbers in SQL queries (in WHERE clause) * Address fastify deprecation warning about accessing config --- # 4.2.5 ### HarperDB 4.2.5, Tucker Release 11/22/2023 * Disable compression on server-sent events to ensure messages are immediately sent (not queued for later deliver) * Update geoNear function to tolerate null values * lmdb-js fix to ensure prefetched keys are pinned in memory until retrieved * Add header to indicate start of a new authenticated session (for studio to identify authenticated sessions) --- # 4.2.6 ### HarperDB 4.2.6, Tucker Release 11/29/2023 * Update various geo SQL functions to tolerate invalid values * Properly report component installation/load errors in `get_components` (for studio to load components after an installation failure) --- # 4.2.7 ### HarperDB 4.2.7 12/6/2023 * Add support for cloning over the top of an existing HarperDB instance * Add health checks for NATS consumer with ability to restart consumer loops for better resiliency * Revert Fastify autoload module due to a regression that had caused EcmaScript modules for Fastify route modules to fail to load on Windows --- # 4.2.8 ### HarperDB 4.2.8 12/19/2023 * Added support CLI command line arguments for clone node * Added support for cloning a node without enabling clustering * Clear NATS client cache on closed event * Fix check for attribute permissions so that an empty attribute permissions array is treated as a table level permission definition * Improve speed of cross-node health checks * Fix for using `database` in describe operations --- # 4.3.0 #### HarperDB 4.3.0, Tucker Release 3/19/2024 #### Relationships and Joins HarperDB now supports defining relationships between tables. These relationships can be defined as one-to-many, many-to-one, or many-to-many, and use a foreign key to record the relationship between records from different tables. An example of how to use this to define a many-to-one and one-to-many relationships between a product and brand table: ``` type Product @table { id: ID @primaryKey name: String @indexed # foreign key used to reference a brand brandId: ID @indexed # many-to-one relationship to brand brand: Related @relationship(from: "brandId") } type Brand @table { id: ID @primaryKey name: String @indexed # one-to-many relationship of brand to products of that brand products: Product @relationship(to: "brandId") } ``` This relationships model can be used in queries and selects, which will automatically "join" the data from the tables. For example, you could search for products by brand name: ``` /Product?brand.name=Microsoft ``` HarperDB also now supports querying with a sort order. Multiple sort orders can be provided breaking ties. Nested select have also been added, which also utilizes joins when related records are referenced. For example: ``` /Product?brand.name=Microsoft&sort(price)&select(name,brand{name,size}) ``` See the [schema definition documentation](/reference/v4/database/schema.md) for more information on defining relationships, and the [REST documentation for more information on queries](/reference/v4/rest/overview.md). #### OpenAPI Specification A new default endpoint `GET /openapi` was added for describing endpoints configured through a GraphQL schema. #### Query Optimizations HarperDB has also made numerous improvements to query planning and execution for high performance query results with a broader range of queries. #### Indexing Nulls New tables and indexes now support indexing null values, enabling queries by null (as well as queries for non-null values). For example, you can query by nulls with the REST interface: ``` GET /Table/?attribute=null ``` Note, that existing indexes will remain without null value indexing, and can only support indexing/querying by nulls if they are rebuilt (removed and re-added). #### CLI Expansion The HarperDB now supports an expansive set of commands that execute operations from the operations API. For example, you can list users from the command line: ``` harperdb list_users ``` #### BigInt Support HarperDB now supports `BigInt` attributes/values with integers (with full precision) up to 1000 bits (or 10^301). These can be used as primary keys or standard attributes, and can be used in queries or other operations. Within JSON documents, you can simply use standard JSON integer numbers with up to 300 digits, and large BigInt integers will be returned as standard JSON numbers. #### Local Studio Upgrade HarperDB has upgraded the local studio to match the same version that is offered on . The local studio now has the full robust feature set of the online version. ### MQTT #### mTLS Support HarperDB now supports mTLS based authentication for HTTP, WebSockets, and MQTT. See the [configuration documentation for more information](/reference/v4/configuration/overview.md). #### Single-Level Wildcards HarperDB's MQTT service now supports single-level wildcards (`+`), which facilitates a great range of subscriptions. #### Retain handling HarperDB's MQTT now supports the retain handling flags for subscriptions that are made using MQTT v5. #### CRDT HarperDB now supports basic conflict-free data type (CRDT) updates that allow properties to be individually updated and merged when separate properties are updated on different threads or nodes. Individual property CRDT updates are automatically performed when you update individual properties through the resource API. Individual property CRDT updates are used when making `PATCH` requests through the REST API. The CRDT functionality also supports explicit incrementation to merge multiple parallel incrementation requests with proper summing. See the [Resource API for more information](/reference/v4/resources/overview.md). #### Configuration Improvements The configuration has improved support for detecting port conflicts, handling paths for fastify routes, and now includes support for specifying a heap limit and TLS ciphers. See the [configuration documentation for more information](/reference/v4/configuration/overview.md). #### Balanced Audit Log Cleanup Audit log cleanup has been improved to reduce resource consumption during scheduled cleanups. #### `export_*` support for `search_by_conditions` The `export_local` and `export_to_s3` operations now support `search_by_conditions` as one of the allowed search operators. ### Storage Performance Improvements Significant improvements were made to handling of free-space to decrease free-space fragmentation and improve performance of reusing free-space for new data. This includes prioritizing reuse of recently released free-space for more better memory/caching utilization. #### Compact Database In addition to storage improvements, HarperDB now includes functionality for [compacting a database](/reference/v4/cli/overview.md) (while offline), which can be used to eliminate all free-space to reset any fragmentation. #### Compression Compression is now enabled by default for all records over 4KB. To learn more on how to configure compression visit [configuration](/reference/v4/configuration/overview.md). --- # 4.3.1 ### HarperDB 4.3.1 3/25/2024 * Fix Fastify warning about responseTime usage * Add access to the MQTT topic in the context * Fix for ensuring local NATS streams are created --- # 4.3.10 ### HarperDB 4.3.10 5/5/2024 * Provide a `data` property on the request/context with deserialized data from the request body for any request including methods that don't typically have a request body * Ensure that CRDTs are not double applied after committing a transaction * Delete MQTT will after publishing even if it fails to publish * Improve transaction retry logic to use async non-optimistic transactions after multiple retries --- # 4.3.11 ### HarperDB 4.3.11 5/15/2024 * Add support for multiple certificates with SNI-based selection of certificates for HTTPS/TLS * Fix warning in Node v22 --- # 4.3.12 ### HarperDB 4.3.12 5/16/2024 * Fix for handling ciphers in multiple certificates * Allow each certificate config to have multiple hostnames --- # 4.3.13 ### HarperDB 4.3.13 5/22/2024 * Fix for handling HTTPS/TLS with IP address targets (no hostname) where SNI is not available * Fix for memory leak when a node is down and consumers are trying to reconnect * Faster cross-thread notification mechanism for transaction events --- # 4.3.14 ### HarperDB 4.3.14 5/24/2024 * Fix application of ciphers to multi-certificate TLS configuration --- # 4.3.15 ### HarperDB 4.3.15 5/29/2024 * Add support for wildcards in hostnames for SNI * Properly apply ciphers settings on multiple TLS configurations --- # 4.3.16 ### HarperDB 4.3.16 6/3/2024 * Properly shim legacy TLS configuration with new multi-certificate support * Show the changed filenames when an application is reloaded --- # 4.3.17 ### HarperDB 4.3.17 6/13/2024 * Add MQTT analytics of incoming messages and separate by QoS level * Ensure that any installed `harperdb` package in components is relinked to running harperdb. * Upgrade storage to more efficiently avoid storage increases * Fix to improve database metrics in system\_information * Fix for pathing on Windows with extension modules * Add ability to define a range of listening threads --- # 4.3.18 ### HarperDB 4.3.18 6/18/2024 * Immediately terminate an MQTT connection when there is a keep-alive timeout. --- # 4.3.19 ### HarperDB 4.3.19 7/2/2024 * Properly return records for the existing value for subscriptions used for retained messages, so they are correctly serialized. * Ensure that deploy components empty the target directory for a clean installation and expansion of a `package` sub-directory. * Ensure that we do not double load components that are referenced by symlink from node\_modules and in components directory. --- # 4.3.2 ### HarperDB 4.3.2 3/29/2024 * Clone node updates to individually clone missing parts * Fixes for publishing OpenShift container * Increase purge stream timeout * Fixed declaration of analytics schema so queries work before a restart * Fix for iterating queries when deleted records exist * LMDB stability upgrade * Fix for cleanup of last will in MQTT --- # 4.3.20 ### HarperDB 4.3.20 7/11/2024 * The restart\_service operation is now executed as a job, making it possible to track the progress of a restart (which is performed as a rolling restart of threads) * Disable Nagle's algorithm for TCP connections to improve performance * Append Server-Timing header if a fastify route has already added one * Avoid symlinking the harperdb directory to itself * Fix for deleting an empty database * Upgrade ws and pm2 packages for security vulnerabilities * Improved TypeScript definitions for Resource and Context. * The context of a source can set `noCacheStore` to avoid caching the results of a retrieval from source * Better error reporting of MQTT parsing errors and termination of connections for compliance --- # 4.3.21 ### HarperDB 4.3.21 8/21/2024 * Fixed an issue with iterating/serializing query results with a `limit`. * Fixed an issue that was preventing the caching of structured records in memory. * Fixed and added several TypeScript exported types including `tables`, `databases`, `Query`, and `Context`. * Fixed logging warnings about license limits after a license is updated. * Don't register a certificate as the default certificate for non-SNI connections unless it lists an IP address in the SAN field. --- # 4.3.22 ### HarperDB 4.3.22 9/6/2024 * Adding improved back-pressure handling for large subscriptions and backlogs with durable MQTT sessions * Allow .extension in URL paths to indicate both preferred encoding and decoding * Added support for multi-part ids in query parameters * Limit describe calls by time before using statistical sampling * Proper cleanup of a transaction when it is aborted due to running out of available read transactions * Updates to release/builds --- # 4.3.23 ### HarperDB 4.3.23 9/12/2024 * Avoid long-running read transactions on subscription catch-ups * Reverted change to setting default certificate for IP address only * Better handling of last-will messages on startup --- # 4.3.24 ### HarperDB 4.3.24 9/12/2024 * Fix for querying for large strings (over 255 characters) --- # 4.3.25 ### HarperDB 4.3.25 9/24/2024 * Add analytics for replication latency * Fix iteration issue over asynchronous joined queries * Local studio fix for loading applications in insecure context (HTTP) * Local studio fix for loading configuration tab --- # 4.3.26 ### HarperDB 4.3.26 9/27/2024 * Fixed a security issue that allowed users to bypass access controls with the operations API * Previously expiration handling was limited to tables with a source, but now it can be applied to any table --- # 4.3.27 ### HarperDB 4.3.27 10/2/2024 * Fixed handling HTTP upgrade with Connection header that does not use Upgrade as the sole value (for Firefox) * Added metrics for requests by status code * Properly remove attributes from the stored metadata when removed from GraphQL schema * Fixed a regression in clustering retrieval of schema description * Fix attribute validation/handling to ensure that sequential ids can be assigned with insert/upsert operations --- # 4.3.28 ### HarperDB 4.3.28 10/3/2024 * Tolerate user with no role when building NATS config * Change metrics for requests by status code to be prefixed with "response\_" * Log error `cause`, and other properties, when available. --- # 4.3.29 ### HarperDB 4.3.29 10/7/2024 * Avoid unnecessary cookie session creation without explicit login * Added support for caching directives in operations API * Fixed issue with creating metadata for table with no primary key * Local studio upgrade: * Added support for "cache only" mode to view table data without origin resolution * Added partial support for cookie-based authentication * Added support for browsing tables with no primary key * Improved performance for sorting tables --- # 4.3.3 ### HarperDB 4.3.3 4/01/2024 * Improve MQTT logging by properly logging auth failures, logging disconnections --- # 4.3.30 ### HarperDB 4.3.30 10/9/2024 * Properly assign transaction timestamp to writes from cache resolutions (ensuring that latencies can be calculated on replicating nodes) --- # 4.3.31 ### HarperDB 4.3.31 10/10/2024 * Reset the restart limit for manual restarts to ensure that NATS process will continue to restart after more than 10 manual restarts * Only apply caching directives (from headers) to tables/resources that are configured to be caching, sourced from another resource * Catch/tolerate errors on serializing objects for logging --- # 4.3.32 ### HarperDB 4.3.32 10/16/2024 * Fix a memory leak when cluster\_network closes a hub connection * Improved MQTT error handling, with less verbose logging of more common errors, and treat a missing subscription as an invalid/missing topic * Record analytics and server-timing header even when cache resolution fails --- # 4.3.33 ### HarperDB 4.3.33 10/24/2024 * Change the default maximum length for a fastify route parameter from 100 to 1000 characters. --- # 4.3.34 ### HarperDB 4.3.34 10/24/2024 * lmdb-js upgrade --- # 4.3.35 ### HarperDB 4.3.35 11/12/2024 * Upgrades for supporting Node.js V23 * Fix for handling a change in the schema for nested data structures --- # 4.3.36 ### HarperDB 4.3.36 11/14/2024 * lmdb-js upgrade for better free-space management --- # 4.3.37 ### HarperDB 4.3.37 12/6/2024 * lmdb-js upgrade for preventing crashes with shared user buffers --- # 4.3.38 ### HarperDB 4.3.38 1/10/2025 * Fixes for audit log cleanup --- # 4.3.4 ### HarperDB 4.3.4 4/9/2024 * Fixed a buffer overrun issue with decompressing compressed data * Better keep-alive of transactions with long running queries --- # 4.3.5 ### HarperDB 4.3.5 4/10/2024 * Fixed a buffer overrun issue with decompressing compressed data --- # 4.3.6 ### HarperDB 4.3.6 4/12/2024 * Fixed parsing of dates from epoch millisecond times in queries * Fixed CRDT incrementation of different data types * Adjustments to text/plain content type q-value handling * Fixed parsing of passwords with a colon * Added MQTT events for connections, authorization, and disconnections --- # 4.3.7 ### HarperDB 4.3.7 4/16/2024 * Fixed transaction handling to stay on open on long compaction operations * Fixed handling of sorting on non-indexed attributes * Storage stability improvements * Fixed authentication/authorization of WebSockets connection and use of cookies * Fixes for clone node operations --- # 4.3.8 ### HarperDB 4.3.8 4/26/2024 * Added support for the MQTT keep-alive feature (disconnecting if no control messages are received within keep-alive window) * Improved handling of write queue timeouts, with configurability * Fixed a memory leak that can occur with NATS reconnections after heartbeat misses * Fixed a bug in clone node with a null port * Add error events to MQTT events system --- # 4.3.9 ### HarperDB 4.3.9 4/30/2024 * lmdb-js upgrade --- # 4.4.0 #### HarperDB 4.4.0 10/14/2024 ### Native Replication HarperDB has a completely [new native replication system](/reference/v4/replication/overview.md) which is faster, more efficient, secure, and reliable than the previous replication system. The new system (codenamed "Plexus") uses direct WebSocket connections between servers with highly optimized encoding and is driven by direct tracking audit/transaction log for efficient and flexible data transfer. This replication has improved resilience with the ability to reach consensus consistency when one node goes down through cross-node catch-up. Network connections can be performed over the existing operations API port or a separate port, for improved configurability. The native replication system is much easier to configure, with multiple options for authentication and security, including PKI/mTLS security that is highly robust and easy to use in conjunction with existing PKI certificates. Replication can be configured through explicit subscriptions or for automated replication of all data in a database. With automated replication, gossiping is used to automatically discover and connect to other nodes in the cluster. #### Sharding The new replication system also includes provisional support for [sharding](/reference/v4/replication/sharding.md). This sharding mechanism paves the way for greater scalability and performance, by allow data to be distributed across multiple nodes. #### Replicated Operations Certain operations can now be replicated across the cluster, including the deployment and management of components. This allows for a more seamless experience when managing a cluster of HarperDB instances. Restarts can also be "replicated", and if used, will perform a rolling restart of all the nodes in a cluster. ### Computed Properties Computed properties allow applications to define properties that are computed from other properties, allowing for composite properties that are calculated from other data stored in records without requiring actual storage of the computed value. For example, you could have a computed property for a full name based on first and last, or age/duration based on a date. Computed properties are also foundational for custom indexes. See the [schema documentation](/reference/v4/database/schema.md), [Resource API](/reference/v4/resources/overview.md), and our blog post on [computed properties](https://www.harpersystems.dev/development/tutorials/how-to-create-custom-indexes-with-computed-properties) for more information. ### Custom Indexing Custom indexes can now be defined using computed properties to allow for unlimited possibilities of indexing, including composite, full-text indexing, vector indexing. Again, see the [schema documentation](/reference/v4/database/schema.md) for more information. ### Native Graph Support HarperDB now includes provisional support for native [GraphQL querying functionality](/reference/v4/graphql-querying/overview.md). This allows for querying of graph data using GraphQL syntax. This is provisional and some APIs may be updated in the future. ### Dynamic Certificate Management Certificates are now stored in system tables and can be dynamically managed. Certificates can be added, replaced, and deleted without restarting HarperDB. This includes both standard certificates and certificate authorities, as well as private keys (private keys are not stored in table, they securely stored in a file). #### Status Report on Startup On startup, HarperDB will now print out an informative status of all running services and ports they are listening on. #### Support for Response object Resource methods can now return a `Response` object (or an object with `headers` and `status`) to allow for more control over the response. ### Auto-incrementing Primary Keys Primary keys can now be auto-incrementing, allowing for automatic generation of numeric primary keys on insert/creation. Primary keys defined with `ID` or `String` will continue to use GUIDs for auto-assigned primary keys, which occurs on insert or creation if the primary key is not provided. However, for keys that are defined as `Any`, `Int`, or `Long`, the primary key will be assigned using auto-incrementation. This is significantly more efficient than GUIDs since the key only requires 8 bytes of storage instead of 31 bytes, and doesn't require random number generation. #### Developer/Production Mode for Configuration When using interactive installation (when configuration is not provided through arguments or env vars), HarperDB now provides an option for developer or production mode with a set of default configuration for each mode better suited for developer or production environments. **Export by Protocol** Exported resources can be configured to be specifically exported by protocol (REST, MQTT, etc.) for more granular control over what is exported where. --- # 4.4.1 ### HarperDB 4.4.1 10/17/2024 * Fix issue where non-RSA keys were not being parsed correctly on startup. * Fix a memory leak when cluster\_network closes a hub connection * Improved MQTT error handling, with less verbose logging of more common errors, and treat a missing subscription as an invalid/missing topic * Record analytics and server-timing header even when cache resolution fails --- # 4.4.10 ### HarperDB 4.4.10 12/17/2024 * Fix for deploying packages and detecting node\_modules directory --- # 4.4.11 ### HarperDB 4.4.11 12/18/2024 * Fix for initial certification creation on upgrade * Docker build fix --- # 4.4.12 ### HarperDB 4.4.12 12/19/2024 * Move components installed by reference into hdb/components for consistency and compatibility with next.js * Use npm install --force to ensure modules are installed --- # 4.4.13 ### HarperDB 4.4.13 1/2/2025 * Fix for not using requestCert if the port doesn't need replication * Fix for applying timeouts HTTP server for ancient node versions * Updates for different replication configuration settings, including sharding and replication using stored credentials * Mitigation crashing due GC'ed shared array buffers * Fix for error handling with CLI failures * Updated dependencies * Fix for allow securePort to be set on authentication --- # 4.4.14 ### HarperDB 4.4.14 1/3/2025 * Fix for starting HTTP server if headersTimeout is omitted in the configuration * Fix for avoiding ping timeouts for large/long-duration WS messages between nodes * Don't report errors for component that only uses a directory * Add flag for disabling WebSocket on REST component --- # 4.4.15 ### HarperDB 4.4.15 1/8/2025 * Fix for manage the state of replication sequences for node * Fix for better concurrency with ongoing replication * Fix for accessing audit log entries --- # 4.4.16 ### HarperDB 4.4.16 1/22/2025 * Fix for cleaning up old audit entries and associated deletion entries * Allow CLI operations to be run when cloning is enabled * Report table size in describe operations * Fix for cleaning up symlinks when dropping components * Fix for enumerating components when symlinks are used * Add an option for using a specific installation command with deploys * Add an API for registering an HTTP upgrade listener with `server.upgrade` --- # 4.4.17 ### HarperDB 4.4.17 1/29/2025 * Provide statistics on the size of the audit log store * Fix handling of symlinks to HarperDB package that to avoid NPM's errors in restricted containers * Add option for rolling/consecutive restarts for deployments * Fix for enabling root CAs for replication authorization --- # 4.4.18 ### HarperDB 4.4.18 1/29/2025 * Add option for disabling full table copy in replication * Add option for startTime in route configuration * Add/fix option to deploy with package from CLI --- # 4.4.19 ### HarperDB 4.4.19 2/4/2025 * LMDB upgrade for free-list verification on commit * Add check to avoid compacting database multiple times with compactOnStart * Fix handling of denied/absent subscription * Add support for including symlinked directories in packaging a deployed component --- # 4.4.2 ### HarperDB 4.4.2 10/18/2024 * Republish of 4.4.1 with Git merge correction. --- # 4.4.20 ### HarperDB 4.4.20 2/11/2025 * LMDB upgrade for improved handling of page boundaries with free-space lists --- # 4.4.21 ### HarperDB 4.4.21 2/25/2025 * Fix for saving audit log entries for large keys (> 1KB) * Security fix for handling missing passwords * Skip bin links for NPM installation to avoid access issues --- # 4.4.22 ### HarperDB 4.4.22 3/5/2025 * Add new http configuration option `corsAccessControlAllowHeaders` --- # 4.4.23 ### HarperDB 4.4.23 3/7/2025 * Fix for subscriptions to children of segmented id * Fix for better error reporting on NPM failures --- # 4.4.24 ### HarperDB 4.4.24 3/10/2025 * Use process.exit(0) to restart when enabled by env var * Reset the cwd on thread restart --- # 4.4.3 ### HarperDB 4.4.3 10/25/2024 * Fix for notification of records through classes that override get for multi-tier caching * Fix for CLI operations * Support for longer route parameters in Fastify routes * Fix for accessing `harperdb` package/module from user threads * Improvements to clone node for cloning without credentials --- # 4.4.4 ### HarperDB 4.4.4 11/4/2024 * Re-introduce declarative roles and permissions * Fix for OpenAPI endpoint * Fix for exports of `harperdb` package/module --- # 4.4.5 ### HarperDB 4.4.5 11/15/2024 * Fix for DOS vulnerability in large headers with cache-control and replication headers * Fix for handling a change in the schema type for sub-fields in a nested object * Add support for content type handlers to return iterators * Fix for session management with custom authentication handler * Updates for Node.js V23 compatibility * Fix for sorting on nested properties * Fix for querying on not\_equal to a null with object values --- # 4.4.6 ### HarperDB 4.4.6 11/25/2024 * Fix queries with only sorting applied * Fix for handling invalidation events propagating through sources * Expanded CLI support for deploying packages * Support for deploying large packages --- # 4.4.7 ### HarperDB 4.4.7 11/27/2024 * Allow for package to deploy own modules * Fix for preventing double sourcing of resources --- # 4.4.8 ### HarperDB 4.4.8 12/2/2024 * Add multiple node versions of published docker containers --- # 4.4.9 ### HarperDB 4.4.9 12/12/2024 * Change enableRootCAs to default to true * Fixes for install and clone commands * Add rejectUnauthorized to the CLI options * Fixes for cloning * Install modules in own component when deploying package by payload --- # 4.5.0 #### HarperDB 4.5.0 3/13/2025 ### Blob Storage 4.5 introduces a new [Blob storage system](/reference/v4/database/schema.md#blob-type), that is designed to efficiently handle large binary objects, with built-in support for streaming large content/media in and out of storage. This provides significantly better performance and functionality for large unstructured data, such as HTML, images, video, and other large files. Components can leverage this functionality through the JavaScript `Blob` interface, and the new `createBlob` function. Blobs are fully replicated and integrated. Harper can also coerce strings to `Blob`s (when dictated by the field type), making it feasible to use blobs for large string data, including with MQTT messaging. ### Password Hashing Upgrade 4.5 adds two new password hashing algorithms for better security (to replace md5): `sha256`: This is a solid general purpose of password hashing, with good security properties and excellent performance. This is the default algorithm in 4.5. `argon2id`: This provides the highest level of security, and is the recommended algorithm that do not require frequent password verifications. However, it is more CPU intensive, and may not be suitable for environments with a high frequency of password verifications. ### Resource and Storage Analytics 4.5 includes numerous new analytics for resources and storage, including page faults, context switches, free space, disk usage, and other metrics. #### Default Replication Port The default port for replication has been changed from 9925 to 9933. ### Property Forwarding Accessing record properties from resource instances should be accessible through standard property access syntax, regardless of whether the property was declared in a schema. Previously only properties declared in a schema were accessible through standard property access syntax. This change allows for more consistent and intuitive access to record properties, regardless of how they were defined. It is still recommended to declare properties in a schema for better performance and documentation. ### Storage Reclamation Harper now includes functionality for automatically trying to clean up and evict non-essential data when storage is running low. When free space drops below 40% (configurable), Harper will start to: * Evict older entries from caching tables * Evict older audit log entries * Remove older rotated logs files These efforts will become progressively more aggressive as free space decreases. ### Expanded Sharding Functionality When sharding is being used, Harper can now honor write requests with residency information that will not be written to the local node's table. Harper also now allows nodes to be declaratively configured as part of a shard. ### Certificate Revocation Certificates can now be revoked by configuring nodes with a list of revoked certificate serial numbers. ### Built-in `loadEnv` Component There is a new `loadEnv` component loader that can be used to load environmental variables from a .env in a component. ### Cluster Status Information The [`cluster_status` operation](/reference/v4/replication/clustering.md) now includes new statistics for replication, including the timestamps of last received transactions, sent transactions, and committed transactions. ### Improved URL path parsing Resources can be defined with nested paths and directly accessed by the exact path without requiring a trailing slash. The `id.property` syntax for accessing properties in URLs will only be applied to properties that are declared in a schema. This allows for URLs to generally include dots in paths without being interpreted as property access. A new [`directURLMapping` option/flag](/reference/v4/configuration/overview.md) on resources that allows for more direct URL path handling as well. ### `server.authenticateUser` API In addition to the `server.getUser` API that allows for retrieval of users by username, the `server.authenticateUser` API is now available which will *always* verify the user by the provided password. #### Improved Message Delivery Performance of delivery of messages has been improved. ### HTTP/2 HarperDB now supports HTTP/2 for all API endpoints. This can be enabled with the `http2` option in the configuration file. ### `harperdb` symlink Using `import from 'harperdb'` will more consistently work when directly running a component locally. ### Transaction Reuse By default, transactions can now be reused after calling `transaction.commit()`. ### GraphQL configuration The GraphQL query endpoint can be configured to listen on different ports. GraphQL query endpoing is now also disabled by default, to avoid any conflicts. ### Glob support for components Glob file handling for specifying files used by components has been improved for better consistency. ### Table.getRecordCount `Table.getRecordCount()` is now available to get the number of records in a table. ### Removal of record counts from REST API Previously the root path for a resource in the REST API would return a record count. However, this is a significant performance hazard and was never documented to exist, so this has been removed to ensure better performance and reliability. Note that downgrading from 4.5 to 4.4 is *not* supported. --- # 4.5.1 ### HarperDB 4.5.1 3/18/2025 * Fix/implementation for sharding data that is written for cache resolution * Add support for replication.shard in configuration for defining local node's shard id * Fix for source map handling in stack traces * Improved error reporting for syntax errors in component code * Improved logging on deployment and NPM installation * Added shard information to cluster\_status * Fix for audit entry eviction when a table is deleted --- # 4.5.10 ### HarperDB 4.5.10 5/20/2025 * Expose the `resources` map for being able to set and access custom resources * Fix for cleaning up blob files that are used when a database is deleted --- # 4.5.11 ### HarperDB 4.5.11 6/27/2025 * Fix bug (workaround Node.js bug) with assigning the ciphers to a server and applying to TLS connections * Fix for handling TLS array when checking certificates configuration --- # 4.5.12 ### HarperDB 4.5.12 7/9/2025 * Fix for dynamically setting `harperdb` package symlink on deploy * Assign shard numbers from each node's config rather than from routes * Handle certificates without a common name, falling back to the SANs * Properly clean up blobs that are only transiently used for replication * Ensure that we always set up server.shards even when there are no TLS connections --- # 4.5.13 ### HarperDB 4.5.13 7/12/2025 * Fix cleaning out audit entries when a blob has been removed --- # 4.5.14 ### HarperDB 4.5.14 7/15/2025 * Use proper back-pressure when copying a table for initial database sync --- # 4.5.15 ### HarperDB 4.5.15 7/21/2025 * Removed the `copyTablesToCatchUp` option and instead utilized the clone node designation of the leader node to copy tables * Ensure that skipping large number of audit entries does not lock up the thread and cause a connection reset --- # 4.5.16 ### HarperDB 4.5.16 7/30/2025 * Do not free/remove the shared user buffer that is used by all threads as an atomic counter for ids (for blobs and incremented ids), but retain it as a stable allocated buffer --- # 4.5.17 ### HarperDB 4.5.17 8/4/2025 * Ensure that blob paths exist when determining blob volume sizes * Increase and make timeouts for blob transfers configurable * Add debugging information on the record ID if a blob transfer times out * Fix for copying tables when doing an initial clone node (will not affect an existing cluster) * Fix for ensuring node IDs are not duplicated across subscriptions to different nodes. This eliminates the duplication of replication data * Fix for building the local studio --- # 4.5.18 ### HarperDB 4.5.18 Release was incorrect version --- # 4.5.19 ### HarperDB 4.5.19 8/5/2025 * Update subscription manager to only use failover subscriptions to nodes with the same shard * Do not attempt to resolve a proxied request to the oneself * Delay deletion of blobs when doing proxied requests of blobs --- # 4.5.2 ### HarperDB 4.5.2 3/25/2025 * For defined schemas, don't allow updates from remote nodes that could cause conflicts and repeated schema change requests * New harper-chrome docker container for accessing Chrome binaries for use with tools like Puppeteer * Improved rolling restart handling of errors with reaching individual nodes * Defined cleaner operation object to avoid accident leaking of credentials with logging --- # 4.5.20 ### HarperDB 4.5.20 8/7/2025 * Properly handle and aggregate multiple requests for the same blob using the same stream * Added analytics for blob replication and replication egress transfer * Added receiving status to indicate if there is a replication backlog or waiting for future transactions * Fix for properly saving blob in a recently updated/relocated record --- # 4.5.21 ### HarperDB 4.5.21 8/8/2025 * Fix for small blobs (less than 8KB) having their contents swapped on repeated access * Completely separate the on-demand request connections and the subscription replication connections for more robust connection management --- # 4.5.22 ### HarperDB 4.5.22 8/14/2025 * Cache the TLS context for replication to avoid high CPU usage of building new TLS contexts * Cleanup subscription listeners that were accumulating * Better handling of establishing connections without effecting failover connections --- # 4.5.23 ### HarperDB 4.5.23 8/18/2025 * Fix issue with connection registration that was causing repeated connections to be established * Increase timeout for retrieval connections * Add support for disabling failover connections * Add additional logging for blob errors in replication --- # 4.5.24 ### HarperDB 4.5.24 8/21/2025 * Ensure that record retrieval during replication can be performed asynchronously to minimize page faults on JS threads * Fix for updating connectivity status when failover is disabled * Fix for handling certificates that do not have a subject CN --- # 4.5.25 ### HarperDB 4.5.25 8/23/2025 * Applies throttling to POST requests to ensure that the event queue gets a chance to process events to avoid starvation of other ongoing functions like replication. --- # 4.5.26 ### HarperDB 4.5.26 8/25/2025 * Applies throttling to all non-safe requests (POST, PUT, DELETE, etc.) * Make the request queue limit configurable --- # 4.5.27 ### HarperDB 4.5.27 9/15/2025 * For blobs that are to be saved before the transaction is committed, do not save them until the commit is initiated, to avoid orphaned blobs when transactions abort * Fix for handling paths when deploying components on Windows * Block all write operations when we are performing a database copy/compact --- # 4.5.28 ### HarperDB 4.5.28 9/19/2025 * Add operation for cleaning up orphaned blobs --- # 4.5.29 ### HarperDB 4.5.29 9/23/2025 * Execute delete sequentially to avoid overloading memory --- # 4.5.3 ### HarperDB 4.5.3 4/3/2025 * Fix for immediately reloading updated certificates and private key files to ensure that certificates properly match the private key * Fix for analytics of storage size when tables are deleted --- # 4.5.30 ### HarperDB 4.5.30 10/1/2025 * Delete blobs when they are saved with saveBeforeCommit: true and the stream is interrupted * Fix serialization of error message from error blobs * Fix CLI handling in clone mode * Completely remove evicted entries rather than leaving "orphaned" deletion entries * Tolerate a deleted table when clearing audit entries --- # 4.5.31 ### HarperDB 4.5.31 10/17/2025 * copy-db and compactOnStart should omit deleted records * Upgrade lmdb for extra integrity check for zero-valued free-space entry size --- # 4.5.32 ### HarperDB 4.5.32 10/21/2025 * Remove append flag for audit log additions --- # 4.5.33 ### HarperDB 4.5.33 11/5/2025 * Add option to mark a database as "sharded" so replication only connects to databases in the same shard, improving efficiency for sharded clusters. * Reliability: ensure we never attempt to reference an audit entry when one is not recorded and always create an audit entry during conflict/CRDT resolutions; this prevents orphaned blobs and guarantees they can be cleaned up. * Data integrity: assert that each transaction is recorded in the audit log. * Maintenance: throttle blob cleanup to reduce system load and add a final log message indicating when orphan cleanup finishes. * Fix: do not relocate records when their location is determined by id. * Add a final cleanup log message when orphan blob cleanup finishes. --- # 4.5.34 ### HarperDB 4.5.34 11/8/2025 * Upgrade LMDB to address issue with returning loose pages to the free-list --- # 4.5.35 ### HarperDB 4.5.35 11/11/2025 * compactOnStart updates: * Don't rollback to pre-compaction if the compaction has a record discrepancy * Move compacted databases into main database directory after each database compaction, in case of failure and some databases don't complete --- # 4.5.36 ### HarperDB 4.5.36 11/20/2025 * Reduced frequency of unnecessary `end_txn` messages that were causing excess traffic from node without originating writes. * Fix to allow `databases` config in replication support a database name as a string or an object * Backport fix for ensuring that we don't failover connections (when failover is disabled) when re-connections take place. * Update last received status when receiving sequence id updates for better reporting of cluster status --- # 4.5.37 ### HarperDB 4.5.37 1/23/2026 * Upgrade lmdb-js and Node.js --- # 4.5.38 ### HarperDB 4.5.38 1/27/2026 * Don't save blobs that are to be written before commit until the transaction is validated --- # 4.5.39 ### HarperDB 4.5.39 4/4/2026 * Upgrade LMDB to fix segfault in printing error message * Add support for re-signing replication certificates to enable cross-signed certificates for replication * Add support for dynamic reloading of certificate configuration * Ensure that we check record expiration after re-retrieving cached record after waiting for another thread to do cache resolution --- # 4.5.4 ### HarperDB 4.5.4 4/11/2025 * Fix for replication of (non-retained) published messages * Make cookie domain be configurable to allow for cookies shared across sub-hostnames * Fix for on-demand loading of shared blobs --- # 4.5.40 ### HarperDB 4.5.40 4/18/2026 * Fix accessing entries with a null/deleted record value * Upgrade LMDB to fix edge case of binary search with trailing zeros --- # 4.5.41 ### HarperDB 4.5.41 4/19/2026 * Fix x509\_cert variable name in certificate handling --- # 4.5.42 ### HarperDB 4.5.42 6/1/2026 * Upgrade LMDB to 3.5.5 to fix orphaned read snapshot not being released when only renewing cursors remain --- # 4.5.5 ### HarperDB 4.5.5 4/15/2025 * Updates for better messaging with symlinks in Windows * Fix for saving replicated blobs --- # 4.5.6 ### HarperDB 4.5.6 4/17/2025 * Fix for changing the type of the primary key attribute * Added a new `includeExpensiveRecordCountEstimates` property to the REST component for returning record count estimates * Fix for dropping attributes --- # 4.5.7 ### HarperDB 4.5.7 4/23/2025 * Fix for handling buffers from replicated sharded blob records to prevent overwriting while using * Updated included studio version for fix for logging in --- # 4.5.8 ### HarperDB 4.5.8 4/30/2025 * Fix MQTT subscription topics with trailing slashes to ensure they are not treated as a wildcard * Fix the arguments that are used for the default connect/subscribe calls so they pass the second argument from connect like `connect(incomingMessages, query) -> subscribe(query)` * Add support for replication connections using any configured certificate authorities to verify the server certificates * Added more descriptive error messages on errors in user residency functions --- # 4.5.9 ### HarperDB 4.5.9 5/14/2025 * Remove --no-bin-links directive for NPM that was causing installs of dependencies to fail --- # 4.6.0 #### HarperDB 4.6.0 6/13/2025 ### Vector Indexing: Hierarchical Navigable Small World Harper 4.6 now includes support for vector indexing, which allows for efficient and fast queries on large semantic data sets. Vector indexing is powered by the [Hierarchical Navigable Small World (HNSW) algorithm](https://arxiv.org/abs/1603.09320) and can be used to index any vector-valued property, and is particularly useful for vector text-embedding data. This provides powerful efficient vector-based searching for semantic and AI-based querying functionality. HNSW is a preferred algorithm for vector indexing and searching because it provides an excellent balance of recall and performance. ### New Extension API with support for dynamic reloading 4.6 introduces a new extension API with significant ergonomic improvements for creating new extension components that are more robust and dynamic. The new API also provides a mechanism for dynamic reloading of some files and configuration without restarts. ### Logging Improvements 4.6 includes significant expansions to logging configurability, allowing for specific logging configurations of individual components. This also leverages the new extension API to allow for dynamic reloading of logging configuration. With the more granular logging, logs can be directed to different files and/or different log levels. The logger includes support for HTTP logging, which configurability for logging standard HTTP methods and paths as well headers, ids, and timing information. It also supports distinct logging configuration for different components. The new logger is now based on the Node.js Console API, with improved the formatting of log messages for various types of objects. An important change is that logging to standard out/error will *not* include the timestamp. And console logging does not get logged to the log files by default. ### Data Loader 4.6 includes a new [data loader](/reference/v4/database/data-loader.md) that can be used to load data into HarperDB as part of a component. The data loader can be used to load data from JSON file and can be deployed and distributed with a component to provide a reliable mechanism for ensuring specific records are loaded into Harper. ### Resource API Upgrades 4.6 includes an upgraded form of the Resource API that can be selected with significant improvements in ease of use. ### only-if-cached behavior Previously when the `only-in-cached` caching directive was used and the entry was not cached, Harper would return a 504, but still make a request to origin in the background. Now, Harper will no longer a request to origin for `only-if-cached`. --- # 4.6.1 7/10/2025 * Plugin API updates to use plugin nomenclature * Fix for dynamically setting `harperdb` package symlink on deploy * Assign shard numbers from each node's config rather than from routes * Handle certificates without a common name, falling back to the SANs * Properly clean up blobs that are only transiently used for replication * Ensure that we always set up server.shards even when there are no TLS connections * Fix for clone node getting the cluster status * Properly initialize config on CLI operations to avoid path error * Fix for lmdb for compiling for MacOS and using little-endian * Allow secure cookies with localhost --- # 4.6.10 9/22/2025 * Add operation for cleaning up orphaned blobs * Fix for directory permission on deploying from Windows * Remove any blobs when messages are removed from the audit log * Block all write operations when we are performing a database copy/compact --- # 4.6.11 9/26/2025 * Add support for querying of dynamic properties for get\_analytics --- # 4.6.12 10/1/2025 * Delete blobs when they are saved with saveBeforeCommit: true and the stream is interrupted * Fix serialization of error message from error blobs * Fix CLI handling in clone mode --- # 4.6.13 10/2/2025 * Completely remove evicted entries rather than leaving "orphaned" deletion entries * Ensure that files are really deleted when commit is aborted with saveBeforeCommit * Add loggerWithTag method to logger --- # 4.6.14 10/21/2025 * Skip deleted records when doing compactions * Tighten constraints on blob cleanup memory usage * Remove append flag for audit log additions --- # 4.6.15 11/5/2025 * Audit logging robustness: * Always record an audit entry when conflict/CRDT resolutions are performed so all blobs retain a reference for eventual cleanup * Avoid referencing an audit entry when audit logging is disabled; ensure a new version is created when resolving a cache entry that replaces an existing one * Blob cleanup improvements: * Reduce cleanup rate to limit resource usage under load * Add a final "orphan cleanup finished" log message * Sharded deployments: do not connect to nodes with a database connection if the database is marked as sharded and is not in the same shard * Stability: clear the application timeout timer to allow threads to exit gracefully when still running * HTTP/router: fix missing `urlPath` option on initial entry handler creation * Memory monitoring: make the near‑limit heap snapshot threshold configurable and enable taking a snapshot before exhausting heap memory * Data placement: avoid relocating records when location is determined by id * Developer ergonomics: fix camelCase conversion typos in resource code --- # 4.6.16 11/8/2025 * Upgrade LMDB to address issue with returning loose pages to the free-list --- # 4.6.17 11/11/2025 * compactOnStart updates: * Don't rollback to pre-compaction if the compaction has a record discrepancy * Move compacted databases into main database directory after each database compaction, in case of failure and some databases don't complete --- # 4.6.18 ### HarperDB 4.6.18 11/21/2025 * Reduced frequency of unnecessary `end_txn` messages that were causing excess traffic from node without originating writes. * Fix to allow `databases` config in replication support a database name as a string or an object * Backport fix for ensuring that we don't failover connections (when failover is disabled) when re-connections take place. * Update last received status when receiving sequence id updates for better reporting of cluster status --- # 4.6.19 ### HarperDB 4.6.19 11/24/2025 * Put a time window on custom metric lookup to improve performance --- # 4.6.2 7/15/2025 * Use proper back-pressure when copying a table for initial database sync * Fix cleaning out audit entries when a blob has been removed * Fix for running CLI operations when a Harper DB is not installed --- # 4.6.20 ### HarperDB 4.6.20 11/25/2025 * Fix bug where replication shared status variable should be camelCase instead of snake\_case --- # 4.6.21 ### HarperDB 4.6.21 11/26/2025 * Fix undefined snake\_case references that should be camelCase variables --- # 4.6.22 ### HarperDB 4.6.22 12/02/2025 * No relevant changes --- # 4.6.23 ### HarperDB 4.6.23 1/23/2026 * Upgrade lmdb-js and Node.js --- # 4.6.24 ### HarperDB 4.6.24 1/27/2026 * Don't save blobs that are to be written before commit until the transaction is validated * Properly check time stamps of analytics to ensure reliable aggregation --- # 4.6.25 ### HarperDB 4.6.25 2/23/2026 * Tolerate headers that have already been sent in 404 handler --- # 4.6.26 ### HarperDB 4.6.26 4/4/2026 * Upgrade LMDB to fix segfault in printing error message * Add support for re-signing replication certificates to enable cross-signed certificates for replication * Add support for dynamic reloading of certificate configuration * Ensure that we check record expiration after re-retrieving cached record after waiting for another thread to do cache resolution --- # 4.6.3 7/30/2025 * Fix for calling table.operation() * Do not free/remove the shared user buffer that is used by all threads as an atomic counter for ids (for blobs and incremented ids), but retain it as a stable allocated buffer * Removed the `copyTablesToCatchUp` option and instead utilized the clone node designation of the leader node to copy tables * Ensure that skipping large number of audit entries does not lock up the thread and cause a connection reset --- # 4.6.4 8/1/2025 * Fix for sending blobs on-demand without deleting before sent * Fix for copying blobs in full table copy in a clone * Freeze records in new resource mode * Improvement in connection handling with disconnections --- # 4.6.5 8/7/2025 * Add more analytics for replication, including transfer for blobs * Avoid attempts to connect to oneself for missing data * Adjustments to logging --- # 4.6.6 8/15/2025 * Fix the auditing of hdb\_nodes on install to ensure full connection of nodes when joining * Copy blob buffers to ensure that small buffers are not replaced with frequent usage * Separate replication subscription connections from on-demand connection retrieval * Fix for fail over connection handling --- # 4.6.7 8/18/2025 * Fix for registering replication connections to avoid repeated connections * Add support for disabling fail-over connections * Tolerate certificates without a subject CN * Additional blob error logging --- # 4.6.8 9/3/2025 * Fix for applying offset/limit on searches with remaining deleted records * Throttling of non-safe HTTP requests and cache resolutions * Fix for skipping application of field select() after a custom get method * Fix for applying the correct permissions when deploying from Windows * Disable fail-over handling by default for replication --- # 4.6.9 9/9/2025 * For blobs that are to be saved before the transaction is committed, do not save them until the commit is initiated, to avoid orphaned blobs when transactions abort --- # 4.7.0 10/16/2025 * A new component status monitoring collects status from each component from loading and any registered notification of status changes. * OCSP is now supported, and can be used to invalidate TLS certificates used for replication and HTTP through an OCSP server. * New analytics and licensing functionality has been added for integration with Fabric services. * Further improvements to the plugin API --- # 4.7.1 10/16/2025 * Fix an issue with an incorrect logger call --- # 4.7.10 11/26/2025 * Fix undefined snake\_case references that should be camelCase variables * Ensure `HARPER_SET_CONFIG` can override any other config source, especially at install time --- # 4.7.11 12/02/2025 * Limit message size to prevent oversized messages * Restart 1/8th of the workers at a time for smoother restarts * Add host and hostname to `listSSHKeys` response * Lazy load the trusted components to avoid requiring the component loader too early * Only guard certain core component names from being deployed * Have `deploy-component` fail by default if a conflicting config entry exists (can be forced) * Ensure that the resource cache is reset when a transaction is committed to avoid reading stale data * Reset the `restartRequired` flag on restart * Fix link to the configuration file documentation * Fix `mergeHeaders` where set-cookie headers could get combined, and handle when array is passed with non-string values --- # 4.7.12 12/08/2025 * Warn users if they are running HarperDB in an application folder or root folder to prevent potential issues --- # 4.7.13 1/2/2026 ### Improvements * **Resource Prototype Handling**: Enhanced the Resource API to ensure correct prototypes are maintained for loaded objects, improving consistency when working with custom classes. * **Replication Configuration**: Added support for passing `replicates` definitions through the configuration, allowing for more flexible replication setups. * **Dependency Updates**: Updated `msgpackr` to properly serialize non-Object plain objects as random access structures and upgraded several other internal dependencies. ### Bug Fixes * Improved the dev mode watcher for better reliability during development. --- # 4.7.14 1/6/2026 ### Improvements * Added support for `replicated` flag for `add_certificate` and `remove_certificate` operations. * Added a safe-mode that allows Harper to be started without running any application code. ### Bug Fixes * Better support for older versions of Node.js. --- # 4.7.15 1/9/2026 ### Improvements * Revamped exported Resource types for significantly improved typing experience. --- # 4.7.16 1/23/2026 * Upgrade lmdb-js and Node.js * Many improvements to types * Add filtering support to read\_log operation --- # 4.7.17 1/27/2026 * Don't save blobs that are to be written before commit until the transaction is validated * Properly check time stamps of analytics to ensure reliable aggregation * Updates for Resource API types --- # 4.7.18 2/6/2026 * Fix indexing issue when a deletion query is executed. * Update clone node to clone JWT passphrase. * Improved typing of sessions on context. * Correctly print stdout/stderr correctly when running package install. * Properly record blob errors when blob timeouts occur in replication. * Make concurrency configuration (with lower default) in replication. * Add `coalesce_time` option to get\_analytics for better tracking of cross-node analytics. * Fix for tracking/exclusion of analytics of system database reads. --- # 4.7.19 2/7/2026 * Fix a race condition in reading blob as a stream from files that are being written to. --- # 4.7.2 10/21/2025 * Data loader updated to detect changes via hash comparison * Fix for Windows inserting backslashes into exported paths * Fix for Windows closing connections on thread restart * Fix for setLogLevel error on upgrade * Change `static` handler to default to using index.html * Switch clone node back to using cluster status information to determine cloning is finished and only query the leader * Added a get\_ssh\_key operation so that ssh keys could be cloned in new nodes * Skip deleted records when doing compactions * Tighten constraints on blob cleanup memory usage * Added new `HARPER_SET_CONFIG` and `HARPER_DEFAULT_CONFIG` env variables for setting and resetting config values * Rename read\_log's `until` to `to` --- # 4.7.20 2/27/2026 * Don't register a WS message listener multiple times when replication is waiting for authorization * Update the get\_analytics coalesce window to match the aggregation window * Broad improvements to TypeScript definitions * Separate app and plugin name in plugin scope --- # 4.7.21 3/6/2026 * Don't send a `Content-Length: 0` header for responses to HEAD requests. * Fix the coalescing time function to properly stream instead of buffering. * Improve support for prioritizing certifications by usage and allow for alternate name/primary key assignment. * Add support for configuring the maximum payload of replication connections. --- # 4.7.22 3/10/2026 * Fix listening on unix domain socket for operations API --- # 4.7.23 3/16/2026 * Fix for preventing segmentation fault when reporting incorrect freelist entry * Fix for not returning an expired entry when cache resolution is concurrent with another attempt at cache resolution * Improvements to get\_analytics performance * Improvements to query concurrency with other actions * Fix for timing with user changes * Fix for application name assignment on initial deployment --- # 4.7.24 3/19/2026 * Add support for reloading certificate configuration on config file changes * Fix for backwards compatibility in honoring synchronous iteration of query results * Fix for handling certificate configuration with HARPER\_SET\_CONFIG --- # 4.7.25 3/30/2026 * Fix for applying correct TLS configuration to each certificate (not incorrectly applying mlts to replication) * Add close event for scopes --- # 4.7.26 ### HarperDB 4.7.26 4/4/2026 * Upgrade LMDB to fix segfault in printing error message * Add support for re-signing replication certificates to enable cross-signed certificates for replication * Add support for dynamic reloading of certificate configuration * Ensure that we check record expiration after re-retrieving cached record after waiting for another thread to do cache resolution * Fix for license warning after renewal --- # 4.7.27 ### HarperDB 4.7.27 4/4/2026 * Error handling of fired events in MQTT * Bypass write queue checks for MQTT updates * Add support for JWT auth with clone node * Improved certification/key selection for CSRs --- # 4.7.28 ### HarperDB 4.7.28 4/16/2026 * Fix for LMDB handling of a binary search for a key with a large number of trailing zeros * Always reassign the analytics hostname to ensure replication --- # 4.7.29 ### HarperDB 4.7.29 4/28/2026 * Allow downgrading to a previous version (with confirmation prompt) * Removed NATS install script * Backported env-var config fixes from Harper: when `securePort` is set to the same value as the HTTP port, the HTTP port is now automatically nulled instead of failing validation; added `composeConfigFromEnv()` helper for inspecting merged config without modifying the config state file * Improved certificate handling for replication: default certificate now includes replication key usages, and certificate scoring updated to heavily favor certs with specific usage types --- # 4.7.3 10/28/2025 * Fix urlPath for initialization of components * Enable option to make heap snapshot near memory limit * Fix trailing slash issue with static component * Avoid deleting UDS file during thread shutdown * Switch to using @datadog/pprof for profiling * Clear application timer on startup to avoid preventing graceful thread shutdown * Add get\_usage\_licenses as an operations API --- # 4.7.30 ### HarperDB 4.7.30 5/20/2026 * Force structures reload on first audit record per replication subscription * Backported Harper CA cert for CSR and nuanced TLS scoring --- # 4.7.31 ### HarperDB 4.7.31 5/20/2026 * Added libatomic1 dependency for pnpm-based installs --- # 4.7.32 ### HarperDB 4.7.32 5/20/2026 * Fix replication table copy audit entries to include expiresAt --- # 4.7.33 ### HarperDB 4.7.33 6/1/2026 * Upgrade LMDB to 3.5.5 to fix orphaned read snapshot not being released when only renewing cursors remain --- # 4.7.4 11/07/2025 * Allow licenses to be partially unlimited * Improve transaction timeout tracking: * Transactions have a timeout property that can be individually adjusted for known long transactions * Transaction timeout (default) is configurable * Transactions record the `startedFrom` resource name and method for better error reporting * Transactions will commit instead of aborting when they timeout, so processes can continue with minimal disruption * Improve TypeScript type accuracy for resource interfaces and pub/sub bits * Fix JSResource to wait for resources to be imported before considering the component loaded * Improve JSResource error handling * Ensure authorize is consistently disabled on appropriate REST methods * Audit log improvements: * When conflict/CRDT resolutions are performed, always save an audit log entry to ensure all blobs have a reference for eventual cleanup * Ensure that audit entries are not referenced if not being recorded * Ensure a new version when resolving a cache entry that replaces an existing one * Improve logging: * Operations API reply errors at info level * Reduce operation request error logging level * Add final cleanup log message for orphan cleanup completion * Blob cleanup optimizations to reduce system load * Fix camelCase conversion issues in variable references * Do not connect to nodes with a database connection if the database is marked as sharded and it is not in the same shard * Don't relocate records if the location is determined by id * Fix Docker container restart behavior --- # 4.7.5 11/11/2025 * Transactions: * Reset transaction timestamp on commit so reused transactions receive a new timestamp and do not conflict with prior operations * Fix transaction timeout message to be accurate * Disallow `get()` with null/undefined more precisely; prevent misuse and improve safety * Fix handling of `create` with one/two arguments when `loadAsInstance=false` * Compaction and database maintenance: * Move each compacted database into place immediately after its compaction completes * Do not disable database deletion when a record count discrepancy is expected due to skipped deletion entries * Component loading/router: * Await `loadDataFile` so subsequent components can access the loaded data * Keep `scope.handleEntry()` promise-free while ensuring the initial load completes before loading the next component * API/OpenAPI: * Fix hostname access for OpenAPI generation * Dependencies: * Upgrade LMDB to address issue with returning loose pages to the free-list * Bug fixes: * Correct nullish comparison logic to properly handle `null` values * Applications: * Add Harper application lock file to avoid reinstalling application on every restart --- # 4.7.6 11/12/2025 * Reducing sampling frequency for standard profiling for usage * Don't warn for no argument for subscribe/search --- # 4.7.7 11/17/2025 * Fix lockfile property spelling and prevent race condition for application installation * Run UI tests for self-hosted cluster with localhost HarperDB --- # 4.7.8 11/21/2025 * Handle thrown plain objects correctly and properly report errors for a bad body * Fix connect to properly pass on query to subscribe * Put a time window on custom metric lookup to improve performance * Use faster compression setting * Further lock down node rearranging when failover is disabled * Update status of last received commit when receiving a sequence id update * Allow databases config to support a direct database name or a data config object * Avoid unnecessary `end_txn` messages when no transactions have been delivered * Rename HDBMS to Studio * Keep multiple set-cookie headers separate --- # 4.7.9 11/25/2025 * Fix bug where replication shared status variable should be camelCase instead of snake\_case --- # Harper Tucker (Version 4) Did you know our release names are dedicated to employee pups? For our fourth release, we have Tucker. ![picture of grey and white dog](/assets/images/tucker-604f9676e934a03c305f9af808c49bf0.png) *G’day, I’m Tucker. My dad is David Cockerill, a software engineer here at Harper. I am a 3-year-old Labrador Husky mix. I love to protect my dad from all the squirrels and rabbits we have in our yard. I have very ticklish feet and love belly rubs!* --- # Harper Lincoln (Version 5) All specific full patch version release notes for 5.x.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5\&expanded=true). ## [5.0](/release-notes/v5-lincoln/5.0.md) * **Open Source & Pro Editions** — Harper v5.0 ships as two editions: a free Apache 2.0 open source edition (`npm i -g harper`) and a Pro edition with replication and certificate management (`npm i -g @harperfast/harper-pro`). * **RocksDB Storage Engine** — RocksDB replaces LMDB as the default storage engine, bringing more robust transactions, a native write-ahead log for ACID compliance, and powerful background compaction. LMDB remains supported for existing databases. * **Application Context Isolation** — Each application now runs in its own JavaScript context with isolated globals and module imports, protecting against prototype pollution and enabling per-application logging and configuration. * **Transitive Replication** — An exclusion-based subscription model lets nodes replicate through intermediaries, enabling complex cluster topologies without requiring direct connections between all nodes. * **Resource API Upgrades** — Static REST methods are now first-class, `getContext()` enables async request access anywhere, and a new `save()` method makes in-transaction writes visible to subsequent reads. * **Granular Role Permissions & Impersonation** — Roles can now be scoped to specific operations, and super users can impersonate other users with configurable identity and role bindings. --- # 5.0 Release Notes ### Patch Releases All patch release notes for 5.0.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.0\&expanded=true). ## Open Source and Pro Editions Harper v5.0 is available in two editions: Open Source and Pro. The Open Source edition is free and open source under the Apache 2.0 license, while the Pro edition includes replication, certificate management, and licensing functionality (the source code is available under the Elastic 2.0 License). The open source edition can be installed with: `npm i -g harper` And the pro edition can be installed with: `npm i -g @harperfast/harper-pro` ### Naming Updates Along with new names for Harper packages, Harper now uses the name "harper" more consistently: * For a fresh installation, the data, configuration, logs, and applications will be installed in the directory `~/harper` directory by default (instead of `~/hdb`). * The configuration file will be named `harper-config.yaml` by default (`harperdb-config.yaml` will still be supported for backwards compatibility). * Applications should import from the `harper` module instead of `harperdb`, to access the Harper APIs. ( `harperdb` will still be supported for backwards compatibility). ## RocksDB Harper 5.0 now uses RocksDB as its default underlying storage engine. RocksDB provides a significantly more robust and reliable storage engine with consistent performance characteristics. RocksDB is well-maintained, has powerful background compaction capabilities, and a wide array of tuning options and features. Harper also introduces its own native transaction log as a write-ahead log (WAL) for RocksDB, which drives ACID compliance in RocksDB, as well as powers real-time delivery of data. This is a highly optimized transaction log designed for high throughput of messaging and data. This is all powered by our new [open source rocksdb-js library](https://github.com/harperfast/rocksdb-js). The transaction log also utilizes separate log files for each node origin, for improved performance and reliability with replication. RocksDB enables robust transactions in Harper, with the complete ability to read and query after writes (and get data from those writes) within a transaction. Harper will continue to support the existing LMDB storage engine for v5.0, and will continue to load databases created with LMDB. Switching a database from LMDB to RocksDB requires a database migration. This can be done using replication by creating new nodes and replicating the data from the old nodes to the new nodes. ### Current Limitations of RocksDB Currently, there are a number of optimizations with querying, caching, contention monitoring, and write batching that have not yet been implemented in v5.0, but are planned for a future release. LMDB exhibits better performance for data that is cached in-memory. Retrieval of past events is not guaranteed to return every event when concurrent events take place on different nodes. This is often used by non-clean MQTT sessions. However, the latest message is always guaranteed to be delivered, sequences of messages from the same node are guaranteed to be delivered in order, ensuring the correctness of most applications and message retain consistency. Retrieval of past events for subscriptions will not support a `count` option. Published messages do not support streamed blobs. ## Resource API Updates Harper v5.0 has upgraded the resource API with several important changes: * Harper v5.0 is specifically encouraging the use of `static` REST methods, and providing functionality to easily use these methods. * The `target` (`RequestTarget` type) will now be parsed prior to calling static REST methods, for access to any query information in the URL. * The current request (`Request` type) will be available in any function through asynchronous context tracking, using the `getContext()` function available from the `harper` module. * The `get` method will always return a frozen enumerable record. The return value does not include all the methods from the Resource API (like `wasLoadedFromSource`, `getContext`, etc.). It only includes methods `getUpdatedTime` and `getExpiresAt`. * A source resource can return a standard `Response` object (the resolved return value from a `fetch` call) in a cache resolving `get` method, and Harper will automatically handle the response, streaming the body and saving the headers into a cached record. * A `getResponse` function is available as part of the standard `harper` module exports, allowing for easy access to the response object from within a resource method. * When using the LMDB storage engine, Harper will no longer attempt to cache resource instances that can make were used to make a record stored in a write visible in a subsequent read. * All the default singular REST methods on tables will consistently return a `Promise`. This includes `Table.get(id)`, `Table.put(...)`, `Table.delete(id)`, `Table.patch(...)`, and `Table.invalidate(id)`. * The Table resource API now includes a `save()` method to explicitly save a record to the database within the current transaction, making it visible to subsequent reads/queries. * RESTful methods can return a "Response-like" object, which is now identified as any object with a `headers` property. * Source resources can return a standard `Response` object (the resolved return value from a `fetch` call) in a cache resolving `get` method, and Harper will automatically handle the response, streaming the body and saving the headers into a cached record. ## Application Context Separation Harper now runs each application its own separate JavaScript "context", which has its own global object, top level variables, and module imports. This provides isolation of applications and access to application-specific configuration data and functionality. These contexts will limit access to certain functionality including spawning new processes. This functionality can be controlled with configuration options. Specifically, any new processes that will be spawned need to be listed in `applications.allowedShellCommands`. Harper will also "freeze" many of the intrinsic objects in the global object, to protect against prototype pollution type attacks and vulnerabilities. This application context separation will also allow the logger to apply application-specific tagging to log messages, and leverage the application-specific configuration for logging. ## Transitive Replication Harper now uses an exclusion-based subscription model for replication. This means that replication will request data from nodes, excluding any other known nodes that will be sending data to the current node. With this approach, complex topologies can be created where additional nodes can be added and transitively replicate through other nodes, without explicit knowledge of all the nodes. Previously, replication required direct connections between all nodes, but transitive replication enables topologies with limited connections to proxy data throughout the cluster. By default, with the replication of the system database, all nodes are reachable and will fully connect. To leverage transitive replication, you can disable the replication of the system database and individually configure the routing of each node (with `replication.routes` in the configuration). ## Granular Operations Access for Roles Roles can now be configured with granular access to operations. A role can be designed to have access to a subset of operations, or reference a named group of operations. ### Impersonation Super users can now impersonate other users, with the ability to specify the user's identity and roles for the impersonated session. ## REST API Updates For errors that occur during the execution of a REST method, Harper now follows [RFC 9457](https://www.rfc-editor.org/rfc/rfc9457.html) for error responses. This means that the error response will include`type`, `title`, and `code` properties to describe the error. Harper will no longer add a `Server` header in the response. `blob.save()` was removed, as it was deprecated and superceded by the `saveBeforeCommit` flag. ## Operation API Updates The `update` operation, and the `upsert` operation when applied to an existing record, will now follow the semantics of a `patch` method which means they will fully utilize CRDT semantics for resolution of conflicting updates across a cluster (separate properties can be independently updated and merged). The operations API has fully switched from using `hash_attribute` to `primary_key` for all operations. The response from `describe_all`, `describe_database`, and `describe_table` operations will now include the `primary_key` attribute (instead of the legacy `hash_attribute` name). The `get_components` operation will now return all files including files and directories that begin with a period (although `node_modules` will still be excluded). ### Configuration The `harper-config.yaml` file will now use relative paths to the root directory of the Harper data installation. Please see the [migration guide](/release-notes/v5-lincoln/v5-migration.md) for suggestions on how to migrate from v4 to v5. --- # 5.1 Release Notes ### Patch Releases All patch release notes for 5.1.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.1\&expanded=true). ## AI Models Integration Harper 5.1 introduces a built-in models layer that provides a unified interface for AI model backends. This enables embedding generation and text generation from directly within Harper applications, without managing external API connections per-application. The models layer is exposed via `scope.models` in application code: ``` // Generate embeddings const vector = await scope.models.embed('text to embed', { model: 'my-embedding-model' }); // Generate text const response = await scope.models.generate([{ role: 'user', content: 'Hello' }], { model: 'my-chat-model' }); // Streaming generation for await (const chunk of scope.models.generateStream(messages, { model: 'my-chat-model' })) { // ... } ``` Supported backends are Anthropic, AWS Bedrock, OpenAI, and Ollama, configured under the `models` key in `harper-config.yaml`: ``` models: anthropic: apiKey: your-api-key openai: apiKey: your-api-key baseUrl: https://api.openai.com/v1 # optional override ollama: baseUrl: http://localhost:11434 bedrock: region: us-east-1 ``` ### `@embed` Schema Directive The `@embed` directive automates vector embedding at the schema level, eliminating the need for application code to compute and store embeddings on every write. Add it to any `Float` array field with a `source` pointing to the text field to embed: ``` type Document @table { id: ID @primaryKey content: String embedding: [Float] @embed(source: "content", model: "my-embedding-model") @indexed(type: "HNSW") } ``` On every write, Harper automatically calls the specified model to compute the embedding from `source` and stores it on the record. The `@indexed(type: "HNSW")` index is attached automatically when not explicitly specified. ### Agent Loop `scope.models.generate` now supports `toolMode: 'auto'`, which runs an agentic loop — the model can invoke tools and Harper will automatically dispatch them until the model produces a final non-tool response. This makes it straightforward to build tool-using agents directly in Harper application code. ## MCP Server Harper 5.1 includes a built-in [Model Context Protocol](https://modelcontextprotocol.io/) server, allowing LLM clients such as Claude Desktop, Cursor, and Zed to connect directly to a Harper instance and interact with its data and operations. The MCP server exposes two profiles: * **Operations profile** — wraps Harper's operations catalog as tools, with a curated default allow-list of read-only operations * **Application profile** — auto-generates tools from a Harper application's Resource verb methods The `harper mcp` CLI command provides a stdio bridge for use with MCP clients: ``` # Generate a config block for Claude Desktop harper mcp print-config --client claude-desktop # Run diagnostics against a running instance harper mcp doctor ``` MCP is enabled by adding an `mcp` block to `harper-config.yaml`. See the MCP reference documentation for full configuration options, authentication, and tool customization. ## Application Routing & Middleware Harper 5.1 adds a named middleware ordering and URL routing system for application components, giving fine-grained control over how request handlers are composed. ### `urlPath` routing A component's HTTP handler can now declare a `urlPath` to scope its routes to a URL prefix. Harper strips the prefix before dispatching to the handler, so the handler's logic sees clean relative paths regardless of where it's mounted. ``` // This handler only receives requests under /api/v2/ export const handleHttpRequest = { urlPath: '/api/v2', async handleRequest(request) { // request.url here is the path *after* /api/v2/ is stripped }, }; ``` ### `before`/`after` ordering Components declare their execution order relative to other named middleware using `before` or `after`. The primary use case is ordering relative to Harper's built-in `authentication` middleware: ``` export const handleHttpRequest = { name: 'my-rate-limiter', before: 'authentication', // run this handler before authentication async handleRequest(request) { ... }, }; export const handleAuthenticatedRequests = { name: 'my-data-handler', after: 'authentication', // only runs after authentication has completed async handleRequest(request) { ... }, }; ``` The `name` field makes a handler addressable so that others can reference it in their `before`/`after` declarations. Cyclic ordering is detected at startup and logged as a warning. These options apply uniformly to HTTP handlers (`handleHttpRequest`), WebSocket handlers (`onWebSocket`), and protocol upgrade handlers (`onUpgrade`). ### Node.js middleware adapter `request.getNodeRequestResponse()` returns a `{ nodeRequest, nodeResponse, response }` triple that bridges Harper's W3C-style Request/Response to Node.js's `IncomingMessage`/`ServerResponse` API. This makes it possible to integrate third-party Node.js middleware (Express, Koa, Passport, etc.) directly inside a Harper application handler without wrapping the entire server. ``` import someMiddleware from 'some-package'; export async function handleRequest(request) { const { nodeRequest, nodeResponse, response } = request.getNodeRequestResponse(); someMiddleware(nodeRequest, nodeResponse, () => {}); return response; } ``` `nodeRequest` mirrors the current request state (including any mutations from earlier middleware), while `nodeResponse` captures headers and body written by the consumer and resolves `response` once headers are available. ## Deployment Tracking `deploy_component` now records a full audit trail for every deployment in the `system.hdb_deployment` system table. Each deployment gets a `deployment_id` and tracks phases (prepare → load → replicate → restart → success), per-node outcomes, and a bounded event log capturing install output. The response from `deploy_component` now includes a `deployment_id`: ``` { "deployment_id": "a3f8c2...", "message": "Component deployed successfully" } ``` New operations provide access to deployment history: * `list_deployments` — query deployment history with filters * `get_deployment` — fetch a single deployment record; supports live SSE streaming for in-progress deploys * `get_deployment_payload` — retrieve the stored tarball for a deployment * `delete_deployment_payload` — free storage by removing the payload blob after deployment See [Deployment Operations](#deployment-operations) in the Operations API reference for details. ## HNSW int8 Quantization HNSW vector indexes now support `int8` quantization, reducing index storage by approximately 3× and improving search throughput approximately 5× with around 1% recall loss at recall\@10: ``` type Document @table { embedding: [Float] @indexed(type: "HNSW", quantization: "int8") } ``` Search uses asymmetric scoring: queries use full-precision float vectors while the index graph uses int8, and results are reranked against full-precision vectors before returning. The full-precision vector is always stored on the record itself. Per-query `ef` can be overridden at query time for applications that need to tune the recall/latency tradeoff dynamically. `int8` quantization is on by default for new HNSW indexes in 5.1. Existing indexes can be reindexed to take advantage of it. ## Replication Improvements Several replication reliability improvements are included in 5.1: * **Resumable bulk clone** — interrupted full-table copies resume from where they left off rather than restarting from the beginning * **Client-side receive watchdog** — dead WebSocket connections are now detected and recovered without waiting for a server-side timeout * **Wedge detection and recovery** — stalled replication streams are detected and re-subscribed automatically * **`isLeader` flag on `add_node`** — explicitly request a full-table copy when joining a cluster, independent of the normal subscription logic * **`replication.pingInterval` / `replication.pingTimeout`** — configurable keepalive intervals for replication connections (values in milliseconds) * **Reconnect backoff cap** — reconnect retry backoff is now capped at 30 seconds; previously it could grow unbounded and leave a peer effectively unable to reconnect * **Boot replay bounded** — the startup transaction-log replay is now bounded so it cannot stall indefinitely on corrupted or out-of-order entries; corrupt frames are skipped, and aged transaction logs are purged before replay begins ## HTTP Caching Harper 5.1 improves HTTP caching behavior in the `CacheOfHttp` resource. Responses are now cached based on RFC 9111 cacheable status codes rather than a hardcoded list. The `allowStaleWhileRevalidate` option enables serving stale cached content while refreshing in the background, reducing perceived latency for read-heavy workloads. The `sourcedFrom` configuration field identifies the upstream source for a cache table and drives the eviction and revalidation logic. ## `LOCAL_ONLY` Writes A new `LOCAL_ONLY` metadata flag allows writes that are persisted locally but never replicated to cluster peers: ``` await table.put({ id: 'node-local-key', value: 42 }, { metadata: { LOCAL_ONLY: true } }); ``` This is useful for node-local state — per-node counters, cache metadata, or analytics that don't belong in the replicated dataset. ## Configuration ### `HARPER_CONFIG` environment variable `HARPER_CONFIG` is now the recommended way to specify the configuration file location: ``` HARPER_CONFIG=/etc/harper/harper-config.yaml harper ``` Previously only `HARPERDB_SETTINGS` and CLI flags were available for this purpose. ### RocksDB memory configuration `storage.rocks.blockCacheSize` explicitly sets the RocksDB block cache size. When not set, Harper auto-sizes it based on available memory. The WriteBufferManager is now on by default at 1/3 of the block cache, which improves memory pressure handling under write-heavy workloads. ### `migrateOnStart` Setting `storage.migrateOnStart: true` automatically migrates LMDB databases to RocksDB on the next startup. This provides a path to migrate existing LMDB-backed instances without manual tooling, at the cost of a longer first startup time. ## v4 to v5 Upgrade Improvements Several reliability fixes landed for in-place v4→v5 upgrades: * The node `hostname` field no longer defaults to `localhost` during an in-place upgrade. Previously, all nodes in a cluster would end up with `hostname: localhost` after a v4→v5 upgrade, causing silent split-brain where nodes couldn't locate each other correctly. * The `__dbis__` structure dictionary is now correctly persisted to RocksDB during the LMDB→RocksDB migration. Without this, cold restarts after migration would fail to decode records. * Blob file references, expiry metadata (`expiresAt`), and residency flags are preserved during migration rather than being dropped. These fixes make the in-place upgrade path substantially more reliable for clustered deployments. See the [migration guide](/release-notes/v5-lincoln/v5-migration.md) for the recommended upgrade procedure. Please see the [migration guide](/release-notes/v5-lincoln/v5-migration.md) for suggestions on how to migrate from v4 to v5. --- # 5.2 Release Notes ### Patch Releases All patch release notes for 5.2.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.2\&expanded=true). ## Querying ### Filtered Vector Search (Predicate-Aware HNSW Traversal) Vector searches combined with filters now evaluate the filter during HNSW graph traversal, so the query keeps exploring until it has enough matching nearest neighbors instead of post-filtering a fixed candidate set (which under-filled results under selective filters). Filters can come from query conditions, a JS-API `vectorFilter` function, or a record-scoped `allowRead` override — overriding `allowRead` on a table now makes it a row-level access check, evaluated per record with `this` bound to the record (closing the gap where a collection scan could return rows a single-record GET would deny). With it, a restricted user's vector search returns the k nearest records they are allowed to see. Very selective conditions automatically use an exact scan instead of graph traversal, and a `filterExpansion` visit budget bounds traversal cost. See [Vector Indexing](/reference/v5/database/schema.md#vector-indexing). ## Configuration ### Replicated `set_configuration` The `set_configuration` operation now accepts `"replicated": true` to apply a configuration change to all cluster nodes in a single Operations API call, with per-node outcomes reported in the response's `replicated` array. Only cluster-appropriate parameters should be replicated — see [Configuration Operations](/reference/v5/configuration/operations.md#set-configuration). --- # Harper Lincoln (Version 5) In honor of Lincoln. ![Lincoln](/assets/images/lincoln-259fea192347546b651d50c04b721dd9.jpeg) Lincoln was a shepherd mutt rescued from a New Mexico reservation in 2021. He lived an extremely active and full life in Summit County, Colorado with his owner Ethan Arrowood (Head of Open Source Engineering here at Harper). Notably, Lincoln summitted seven Colorado 14'ers (hiking mountain summits over 14,000 feet). His favorite activity was playing fetch and tromping through snow. *Whenever it is snowing, know it is Lincoln saying hello!* ❄️🏔️ --- # v5-migration Harper version 5.0 includes many updates to provide a cleaner, more consistent and secure environment. However, there are some breaking changes, and users should review the migration guide for details on how to update their applications. Note that applications that have race conditions that are prone to timing or rely on undocumented features or bugs are always prone to breakage at any point, including major version upgrades. This document describes the important changes to make for applications correctly built in documented APIs. ## Naming Changes HarperDB now uses the name Harper, not HarperDB. And this change is reflected in the package name. So Harper should now be launched with: The open source edition can be run with: `npm i -g harper` And the pro edition can be run with: `npm i -g @harperfast/harper-pro` Application code should import from the `harper` package instead of `harperdb`: ``` import { tables } from 'harper'; ``` ## Package Installation Install Scripts By default, Harper now uses the `--ignore-scripts` flag when installing packages to prevent against accidental execution of scripts, which can be a significant security risk. If you are installing applications that require installation scripts to be executed (sometimes necessary for installing additional binaries for execution), use the `allowInstallScripts` option when deploying. ## `Table.get` return value The return value of `Table.get` has been changed to return a record object instead of an instance of the table class (previously this behavior only occured in classes that had set `static loadAsInstance=false`). This means that the returned object will not have all the table instance methods available. Most functionality is still available through the `Table` class. One notable method that had been commonly used is `wasLoadedFromSource`. Information about whether the request was fulfilled from cache or origin is now available on the request `target` object. For example, if you have existing code like: ``` const record = await Table.get(id); // old method: if (record.wasLoadedFromSource()) { // record was loaded from origin (not cache) } ``` You should update this code to: ``` const target = new RequestTarget(); // note that this is passed in if you are overriding the `get` method target.id = id; const record = await Table.get(target); // new way of checking if it was loaded from source: if (target.loadedFromSource) { // record was loaded from origin (not cache) } ``` The record objects do have `getUpdatedTime` and `getExpiresAt` methods available. ### Frozen Records The record object is also frozen. This means that you cannot add or remove properties from the record object, and if you want a modified version of the record, you must create or copy a new one. For example if you had code: ``` const record = await Table.get(id); record.property = 'changed'; ``` You would need to change this to: ``` let record = await Table.get(id); record = { ...record, property: 'changed' }; ``` ## Transactions and Context With RocksDB, transactions are now fully supported through the storage engine, providing a consistent ability to read and query data that has been written to the transaction. This does result in behavioral changes if code had previously not expected written data to be visible in queries until after a commit. Harper v5 now uses asynchronous context tracking to automatically preserve context and the current transaction across calls and asynchronous operations. Context is used to track the current transaction. Previously, transactions were only applied to calls to other tables if they were explicitly included in the arguments. Now context is implicitly and automatically carried to other calls (this was also behavior in v4.x with `static loadAsInstance=false`). Previous code may have omitted context to another table call to exclude it from a transaction. Code should be updated to explicitly commit/finish a transaction to see new visible data or start a new transaction. For example, if you had a function that polled to determine when a record was updated: ``` import { setTimeout as delay } from 'node:timers/promises'; class MyResource { static async get(target) { // this function is within a transaction, with a consistent snapshot of data that won't change, but previous code could // call Table.get without a context, it would not use the current transaction and would instead get the latest data while ((await Table.get(target)).status !== 'ready') { delay(100); } return Table.get(target); } } ``` Now the internal `Table.get` will automatically use the current transaction, which will never change and won't receive updated data. So we should explicitly commit the transaction to see the updated data and/or start a new transaction for each get request to see the latest data: ``` import { setTimeout as delay } from 'node:timers/promises'; import { getContext } from 'harper'; class MyResource { static async get(target) { // this function is still within a transaction, with a consistent snapshot of data that won't change, but we should // explicitly commit the transaction to see the updated data await getContext().transaction.commit(); // now we can call Table.get and it will read the latest data. // we could also explicitly start a new transaction here for each get: while ((await transaction(() => Table.get(target))).status !== 'ready') { delay(100); } return Table.get(target); } } ``` Automatic context tracking can greatly simplify code and automatically handling transactions, but there are subtle shifts in logic and explicitly committing/finishing transactions is important if you are executing code outside the context of a Harper request. ## Spawning new processes (via `node:child_process`) The ability to spawn new processes is a dangerous pathway for exploitation and security vulnerabilities. Additionally, spawning processes from multiple threads presents unique challenges and hazards. In Harper version 5, spawning new processes (through node's `child_process` module) is more tightly controlled and managed. First, any `spawn`, `exec`, or `execFile` may only spawn executables or commands that have been registered in the `applications.allowedSpawnCommands` configuration. This provides a much more secure evironment, preventing malicious intrusions. Second, it is common to attempt to use spawn child processes with the expectations of code that is written to run in a single thread for an indefinite period of time. However, Harper runs multiple threads that may frequently be restarted. When attempting to start/run a supporting process, spawning every time a module loads leads multiplication of processes and orphaned processes. Harper now manages the spawning process to ensure a single process is spawned. To ensure that only a single process is started, the `spawn`, `exec`, etc. functions require a `name` property in the `options` argument, to create a named process that other threads can check and omit starting a new process if one is already started. If you really want to start a separate process from a previously started process, a new `name` must be provided. ## Response Objects Harper has expanded support for using standard Response-like objects in the API. In particular, if you return an object from a REST method with a `headers` property, this will be used as the response headers. ## `blob.save()` removed The `blob.save()` method has been removed. Please use the `saveBeforeCommit` flag in the options to the `Blob` constructor instead. ## VM Module Loader Harper v5 loads application modules through Node.js's VM module API, giving each application its own module cache and a `harper` module scoped to that application — the `logger` it exports is tagged with the application name, and `config` reflects that application's own configuration. By default (`vm-current-context`), applications share JavaScript intrinsics (`Object`, `Array`, `Promise`, and so on) with Harper. Sharing intrinsics avoids the compatibility problems that separate per-application intrinsics can cause — most commonly `instanceof` and other identity checks failing for values that cross the application/Harper boundary. All module loading behavior is controlled by the `applications` section in `harperdb-config.yaml`: ``` applications: lockdown: freeze-after-load # default; see below moduleLoader: vm-current-context # vm-current-context (default) | vm | native | compartment dependencyLoader: auto # auto (default) | app | native allowedDirectory: app # app (default) | any allowedSpawnCommands: # see "Spawning new processes" above - npm - node # allowedBuiltinModules: [] # if omitted, all Node.js built-ins are allowed ``` ### Module Loader Modes The `moduleLoader` setting selects how application modules are loaded: * `vm-current-context` (default) — the VM module loader running in Harper's own context. Applications share intrinsics with Harper, which gives the best compatibility with packages that perform `instanceof` or other identity checks across the boundary. Application-specific values (`logger`, `config`, `server`, and the rest of the `harper` API) are provided through `import ... from 'harper'` (or `require('harper')` in CommonJS). * `vm` — the VM module loader running in a separate context per application, with its own intrinsics and a custom global object. This provides stronger isolation between applications, but the separate intrinsics are a common source of subtle incompatibilities (cross-context `instanceof`, frozen-prototype mismatches, and similar). * `native` — standard Node.js `import()` with no VM loader. Application-specific context (tagged logging, per-app `config`) is not available. * `compartment` — SES Compartment-based loading. Advanced and considerably heavier; only needed for specialized sandboxing requirements. For most applications the default is the right choice. Choose `vm` only if you specifically need separate per-application intrinsics, and `native` if the VM loader causes compatibility problems you cannot otherwise resolve. > Under `lockdown: ses`, the constrained (https-only) `fetch` is applied only in `vm` mode, which gives each application its own globals. In `vm-current-context` and `native` modes application code uses the standard global `fetch`; choose `vm` mode if you require the constrained `fetch`. ### Intrinsic Lockdown The default lockdown mode (`freeze-after-load`) freezes JavaScript intrinsics (`Object`, `Array`, `Promise`, `Map`, `Set`, and others) after all application code has loaded. This prevents prototype pollution attacks. If application code or a dependency modifies intrinsic prototypes at runtime (after startup), it will throw a TypeError. Available lockdown modes: * `freeze-after-load` — freeze intrinsics after all components have loaded (default) * `freeze` — freeze intrinsics before loading any application code * `ses` — full SES lockdown via the `ses` package (strictest; most likely to break packages that mutate built-ins) * `none` — no lockdown If a dependency modifies intrinsic prototypes and you need a temporary workaround, set `lockdown: none`. ### Allowed Directory In production, applications can only load modules from within their own directory tree (`allowedDirectory: app`). Attempting to load a module from outside that directory will throw. Dev mode installs default to `allowedDirectory: any`, so local development is typically unaffected. If your application legitimately needs to load files from outside its own directory in production, set: ``` applications: allowedDirectory: any ``` ### Allowed Built-in Modules By default all Node.js built-in modules are accessible. To restrict which built-ins applications may import, set an explicit allowlist: ``` applications: allowedBuiltinModules: - fs - path - http ``` ### Dependency Loading By default (`dependencyLoader: auto`), npm packages that do not declare `harper` as a dependency are loaded with the native Node.js loader. Packages that do depend on `harper` are loaded through the VM loader so they receive application context. Set `dependencyLoader: app` to always use the VM loader for dependencies, or `native` to always use the native loader for packages. ### Disabling the VM Loader If the VM loader is causing compatibility issues with existing code, it can be disabled entirely: ``` applications: moduleLoader: native ``` This restores pre-v5 behavior where modules are loaded with a standard `import()`. Application-specific context (tagged logging, per-app `config`) will not be available in native mode. If the goal is only to fix package compatibility while keeping application context for first-party code, `dependencyLoader: native` is a narrower option. It uses native loading only for npm packages while keeping the VM loader for application source files. # Recommend Changes The migration information above highlights necessary changes to make to existing applications, if they have used any of these patterns or features. However, we also have new recommended best practices for applications. These are not necessary, but can help to ensure that your application is using the best patterns. * We recommend using the `static` methods on Resources/Tables to implement endpoints. This should be used in conjunction with accessing request information from the request `target` argument (or the request itself via `getContext`). See the Resources API for more information. * Context does not need to be explicitly passed to every call, and can be accessed through the `getContext` function available as an export from the `harper` package. * Harper functions/APIs should be accessed through the `harper` package rather than through the global variables. ---