4.2.0

HarperDB 4.2.0

HarperDB 4.2 introduces a new interface to accessing our core database engine with faster access, well-typed idiomatic JavaScript interfaces, ergonomic object mapping, and real-time data subscriptions. 4.2 also had adopted a new component architecture for building extensions to deliver customized external data sources, authentication, file handlers, content types, and more. These architectural upgrades lead to several key new HarperDB capabilities including a new REST interface, advanced caching, real-time messaging and publish/subscribe functionality through MQTT, WebSockets, and Server-Sent Events.

4.2 also introduces configurable database schemas, using GraphQL Schema syntax. The new component structure is also configuration-driven, providing easy, low-code paths to building applications. Check out our new getting starting guide to see how easy it is to get started with HarperDB apps.

Resource API

The Resource API is the new interface for accessing data in HarperDB. It utilizes a uniform interface for accessing data in HarperDB database/tables and is designed to easily be implemented or extended for defining customized application logic for table access or defining custom external data sources. This API has support for connecting resources together for caching and delivering data change and message notifications in real-time. The Resource API documentation details this interface.

Component Architecture

HarperDB's custom functions have evolved towards a full component architecture; our internal functionality is defined as components, and this can be used in a modular way in conjunction with user components. These can all easily be configured and loaded through configuration files, and there is now a well-defined interface for creating your own components. Components can easily be deployed/installed into HarperDB using NPM and Github references as well.

Configurable Database Schemas

HarperDB applications or components support schema definitions using GraphQL schema syntax. This makes it easy to define your table and attribute structure and gives you control over which attributes should be indexed and what types they should be. With schemas in configuration, these schemas can be bundled with an application and deployed together with application code.

REST Interface

HarperDB 4.2 introduces a new REST interface for accessing data through best-practice HTTP APIs using intuitive paths and standards-based methods and headers that directly map to our Resource API. This new interface provides fast and easy access to data via queries through GET requests, modifications of data through PUTs, customized actions through POSTs and more. With standards-based header support built-in, this works seamlessly with external caches (including browser caches) for accelerated performance and reduced network transfers.

Real-Time

HarperDB 4.2 now provides standard interfaces for subscribing to data changes and receiving notifications of changes and messages in real-time. Using these new real-time messaging capabilities with structured data provides a powerful integrated platform for both database style data updates and querying along with message delivery. Real-time messaging of data is available through several protocols:

MQTT

4.2 now includes MQTT support which is a publish and subscribe messaging protocol, designed for efficiency (designed to be efficient enough for even small Internet of Things devices). This allows clients to connect to HarperDB and publish messages through our data center and subscribe to messages and data for real-time delivery. 4.2 implements support for QoS 0 and 1, along with durable sessions.

WebSockets

HarperDB now also supports WebSockets. This can be used as a transport for MQTT or as a connection for custom connection handling.

Server-Sent Events

HarperDB also includes support for Server-Sent Events. This is a very easy-to-use browser API that allows web sites/applications to connect to HarperDB and subscribe to data changes with minimal effort over standard HTTP.

Database Structure

HarperDB databases contain a collection of tables, and these tables are now contained in a single transactionally-consistent database file. This means reads and writes can be performed transactionally and atomically across tables (as long as they are in the same database). Multi-table transactions are replicated as single atomic transactions as well. Audit logs are also maintained in the same database with atomic consistency as well.

Databases are now entirely encapsulated in a file, which means they can be moved/copied to another database without requiring any separate metadata updates in the system tables.

Clone Node

HarperDB includes new functionality for adding new HarperDB nodes in a cluster. New instances can be configured to clone from a leader node, performing and copying a database snapshot from a leader node, and self-configuring from the leader node as well, to facilitate accelerated deployment of new nodes for fast horizontal scaling to meet demand needs. See the documentation on Clone Node for more information.

Operations API terminology updates

Any operation that used the schema property was updated to make this property optional and alternately support database as the property for specifying the database (formerly 'schema'). If both schema and database are absent, operation defaults to using the data database. Term 'primary key' now used in place of 'hash'. noSQL operation search_by_hash updated to search_by_id.

Support was added for defining a table with primary_key instead of hash_attribute.

Configuration

There have been significant changes to harperdb-config.yaml, however none of these changes should affect pre-4.2 versions. If you upgrade to 4.2 any existing configuration should be backwards compatible and will not need to be updated.

harperdb-config.yaml has had some configuration values added, removed, renamed and defaults changed. Please refer to harperdb-config.yaml for the most current configuration parameters.

• The http element has been expanded.

  • compressionThreshold was added.

  • All customFunction configuration now lives here, except for the tls section.

• threads has moved out of the http element and now is its own top level element.

• authentication section was moved out of the operationsApi section and is now its own top level element/section.

• analytics.aggregatePeriod was added.

• Default logging level was changed to warn.

• Default clustering log level was changed to info.

• clustering.republishMessages now defaults to false.

• operationsApi.foreground was removed. To start HarperDB in the foreground, from the CLI run harperdb.

• Made operationsApi configuration optional. Any config not defined here will default to the http section.

• Added a securePort parameter to operationsApi and http used for setting the https port.

• Added a new top level tls section.

• Removed customFunctions.enabled, customFunctions.network.https, operationsApi.network.https and operationsApi.nodeEnv.

• Added an element called componentRoot which replaces customFunctions.root.

• Updated custom pathing to use databases instead of schemas.

• Added logging.auditAuthEvents.logFailed and logging.auditAuthEvents.logSuccessful for enabling logging of auth events.

• A new mqtt section was added.

Socket Management

HarperDB now uses socket sharing to distribute incoming connections to different threads (SO_REUSEPORT). This is considered to be the most performant mechanism available for multi-threaded socket handling. This does mean that we have deprecated session-affinity based socket delegation.

HarperDB now also supports more flexible port configurations: application endpoints and WebSockets run on 9926 by default, but these can be separated, or application endpoints can be configured to run on the same port as the operations API for a single port configuration.

Sessions

HarperDB now supports cookie-based sessions for authentication for web clients. This can be used with the standard authentication mechanisms to login, and then cookies can be used to preserve the authenticated session. This is generally a more secure way of maintaining authentication in browsers, without having to rely on storing credentials.

Dev Mode

HarperDB can now directly run a HarperDB application from any location using harperdb run /path/to/app or harperdb dev /path/to/app. The latter starts in dev mode, with logging directly to the console, debugging enabled, and auto-restarting with any changes in your application files. Dev mode is recommended for local application and component development.