4.3.0

HarperDB 4.3.0, Tucker Release

3/19/2024

Relationships and Joins

HarperDB now supports defining relationships between tables. These relationships can be defined as one-to-many, many-to-one, or many-to-many, and use a foreign key to record the relationship between records from different tables. An example of how to use this to define a many-to-one and one-to-many relationships between a product and brand table:

type Product @table {
	id: ID @primaryKey
	name: String @indexed
	# foreign key used to reference a brand
	brandId: ID @indexed
	# many-to-one relationship to brand
	brand: Related @relation(from: "brandId") 
}
type Brand @table {
	id: ID @primaryKey
	name: String @indexed
	# one-to-many relationship of brand to products of that brand
	products: Product @relation(to: "brandId")
}

This relationships model can be used in queries and selects, which will automatically "join" the data from the tables. For example, you could search for products by brand name:

/Product?brand.name=Microsoft

HarperDB also now supports querying with a sort order. Multiple sort orders can be provided breaking ties. Nested select have also been added, which also utilizes joins when related records are referenced. For example:

/Product?brand.name=Microsoft&sort(price)&select(name,brand{name,size})

See the schema definition documentation for more information on defining relationships, and the REST documentation for more information on queries.

OpenAPI Specification

A new default endpoint GET /openapi was added for describing endpoints configured through a GraphQL schema.

Query Optimizations

HarperDB has also made numerous improvements to query planning and execution for high performance query results with a broader range of queries.

Indexing Nulls

New tables and indexes now support indexing null values, enabling queries by null (as well as queries for non-null values). For example, you can query by nulls with the REST interface:

GET /Table/?attribute=null

Note, that existing indexes will remain without null value indexing, and can only support indexing/querying by nulls if they are rebuilt (removed and re-added).

CLI Expansion

The HarperDB now supports an expansive set of commands that execute operations from the operations API. For example, you can list users from the command line:

harperdb list_users

BigInt Support

HarperDB now supports BigInt attributes/values with integers (with full precision) up to 1000 bits (or 10^301). These can be used as primary keys or standard attributes, and can be used in queries or other operations. Within JSON documents, you can simply use standard JSON integer numbers with up to 300 digits, and large BigInt integers will be returned as standard JSON numbers.

Local Studio Upgrade

HarperDB has upgraded the local studio to match the same version that is offered on http://studio.harperdb.io. The local studio now has the full robust feature set of the online version.

MQTT

mTLS Support

HarperDB now supports mTLS based authentication for HTTP, WebSockets, and MQTT. See the configuration documentation for more information.

Single-Level Wildcards

HarperDB's MQTT service now supports single-level wildcards (+), which facilitates a great range of subscriptions.

Retain handling

HarperDB's MQTT now supports the retain handling flags for subscriptions that are made using MQTT v5.

CRDT

HarperDB now supports basic conflict-free data type (CRDT) updates that allow properties to be individually updated and merged when separate properties are updated on different threads or nodes. Individual property CRDT updates are automatically performed when you update individual properties through the resource API. Individual property CRDT updates are used when making PATCH requests through the REST API.

The CRDT functionality also supports explicit incrementation to merge multiple parallel incrementation requests with proper summing. See the Resource API for more information.

Configuration Improvements

The configuration has improved support for detecting port conflicts, handling paths for fastify routes, and now includes support for specifying a heap limit and TLS ciphers. See the configuration documentation for more information.

Balanced Audit Log Cleanup

Audit log cleanup has been improved to reduce resource consumption during scheduled cleanups.

export_* support for search_by_conditions

The export_local and export_to_s3 operations now support search_by_conditions as one of the allowed search operators.

Storage Performance Improvements

Significant improvements were made to handling of free-space to decrease free-space fragmentation and improve performance of reusing free-space for new data. This includes prioritizing reuse of recently released free-space for more better memory/caching utilization.

Compact Database

In addition to storage improvements, HarperDB now includes functionality for compacting a database (while offline), which can be used to eliminate all free-space to reset any fragmentation.

Compression

Compression is now enabled by default for all records over 4KB.