LogoLogo
Studio
4.3
4.3
  • HarperDB Docs
  • Getting Started
  • Developers
    • Applications
      • Caching
      • Defining Schemas
      • Debugging Applications
      • Define Fastify Routes
      • Example Projects
    • Components
      • Installing
      • Writing Extensions
      • Operations
      • Google Data Studio
      • SDKs
      • Drivers
    • REST
    • Operations API
      • Quick Start Examples
      • Databases and Tables
      • NoSQL Operations
      • Bulk Operations
      • Users and Roles
      • Clustering
      • Custom Functions
      • Components
      • Registration
      • Jobs
      • Logs
      • Utilities
      • Token Authentication
      • SQL Operations
      • Advanced JSON SQL Examples
    • Real-Time
    • Clustering
      • Requirements and Definitions
      • Creating A Cluster User
      • Naming A Node
      • Enabling Clustering
      • Establishing Routes
      • Subscription Overview
      • Managing Subscriptions
      • Things Worth Knowing
      • Certificate Management
    • Security
      • JWT Authentication
      • Basic Authentication
      • mTLS Authentication
      • Configuration
      • Users & Roles
      • Certificate Management
    • SQL Guide
      • SQL Features Matrix
      • SQL Date Functions
      • SQL Reserved Word
      • SQL Functions
      • SQL JSON Search
      • SQL Geospatial Functions
  • Administration
    • Best Practices and Recommendations
    • Logging
      • Standard Logging
      • Audit Logging
      • Transaction Logging
    • Clone Node
    • Compact
    • Jobs
    • HarperDB Studio
      • Create an Account
      • Log In & Password Reset
      • Organizations
      • Instances
      • Query Instance Data
      • Manage Databases / Browse Data
      • Manage Charts
      • Manage Clustering
      • Manage Instance Users
      • Manage Instance Roles
      • Manage Applications
      • Instance Metrics
      • Instance Configuration
      • Enable Mixed Content
  • Deployments
    • Configuration File
    • HarperDB CLI
    • Install HarperDB
      • On Linux
    • Upgrade a HarperDB Instance
    • HarperDB Cloud
      • IOPS Impact on Performance
      • Instance Size Hardware Specs
      • Alarms
      • Verizon 5G Wavelength
  • Technical Details
    • Reference
      • Analytics
      • Architecture
      • Content Types
      • Data Types
      • Dynamic Schema
      • HarperDB Headers
      • HarperDB Limits
      • Globals
      • Resource Class
      • Transactions
      • Storage Algorithm
    • Release Notes
      • HarperDB Tucker (Version 4)
        • 4.3.36
        • 4.3.35
        • 4.3.34
        • 4.3.33
        • 4.3.32
        • 4.3.31
        • 4.3.30
        • 4.3.29
        • 4.3.28
        • 4.3.27
        • 4.3.26
        • 4.3.25
        • 4.3.24
        • 4.3.23
        • 4.3.22
        • 4.3.21
        • 4.3.20
        • 4.3.19
        • 4.3.18
        • 4.3.17
        • 4.3.16
        • 4.3.15
        • 4.3.14
        • 4.3.13
        • 4.3.12
        • 4.3.11
        • 4.3.10
        • 4.3.9
        • 4.3.8
        • 4.3.7
        • 4.3.6
        • 4.3.5
        • 4.3.4
        • 4.3.3
        • 4.3.2
        • 4.3.1
        • 4.3.0
        • 4.2.8
        • 4.2.7
        • 4.2.6
        • 4.2.5
        • 4.2.4
        • 4.2.3
        • 4.2.2
        • 4.2.1
        • 4.2.0
        • 4.1.2
        • 4.1.1
        • 4.1.0
        • 4.0.7
        • 4.0.6
        • 4.0.5
        • 4.0.4
        • 4.0.3
        • 4.0.2
        • 4.0.1
        • 4.0.0
        • HarperDB Monkey (Version 3)
        • 3.3.0
        • 3.2.1
        • 3.2.0
        • 3.1.5
        • 3.1.4
        • 3.1.3
        • 3.1.2
        • 3.1.1
        • 3.1.0
        • 3.0.0
        • HarperDB Penny (Version 2)
        • 2.3.1
        • 2.3.0
        • 2.2.3
        • 2.2.2
        • 2.2.0
        • 2.1.1
        • HarperDB Alby (Version 1)
        • 1.3.1
        • 1.3.0
        • 1.2.0
        • 1.1.0
  • More Help
    • Support
    • Slack
    • Contact Us
Powered by GitBook
On this page
  • Query Language Agnostic
  • ACID Compliant
  • Universally Indexed
  • Additional LMDB Benefits
  • HarperDB Indexing Example (Single Table)
  1. Technical Details
  2. Reference

Storage Algorithm

PreviousTransactionsNextRelease Notes

Last updated 1 year ago

The HarperDB storage algorithm is fundamental to the HarperDB core functionality, enabling the and all other user-facing functionality. HarperDB is built on top of Lightning Memory-Mapped Database (LMDB), a key-value store offering industry leading performance and functionality, which allows for our storage algorithm to store data in tables as rows/objects. This document will provide additional details on how data is stored within HarperDB.

Query Language Agnostic

The HarperDB storage algorithm was designed to abstract the data storage from any individual query language. HarperDB currently supports both SQL and NoSQL on top of this storage algorithm, with the ability to add additional query languages in the future. This means data can be inserted via NoSQL and read via SQL while hitting the same underlying data storage.

ACID Compliant

Utilizing Multi-Version Concurrency Control (MVCC) through LMDB, HarperDB offers ACID compliance independently on each node. Readers and writers operate independently of each other, meaning readers don’t block writers and writers don’t block readers. Each HarperDB table has a single writer process, avoiding deadlocks and assuring that writes are executed in the order in which they were received. HarperDB tables can have multiple reader processes operating at the same time for consistent, high scale reads.

Universally Indexed

All top level attributes are automatically indexed immediately upon ingestion. The reflexively creates both the attribute and index reflexively as new schema metadata comes in. Indexes are agnostic of datatype, honoring the following order: booleans, numbers ordered naturally, strings ordered lexically. Within the LMDB implementation, table records are grouped together into a single LMDB environment file, where each attribute index is a sub-database (dbi) inside said environment file. An example of the indexing scheme can be seen below.

Additional LMDB Benefits

HarperDB inherits both functional and performance benefits by implementing LMDB as the underlying key-value store. Data is memory-mapped, which enables quick data access without data duplication. All writers are fully serialized, making writes deadlock-free. LMDB is built to maximize operating system features and functionality, fully exploiting buffer cache and built to run in CPU cache. To learn more about LMDB, visit their documentation.

HarperDB Indexing Example (Single Table)

Dynamic Schema
HarperDB Dynamic Schema