Getting started with HarperDB is easy and fast.

The quickest way to get up and running with HarperDB is with HarperDB Cloud, our database-as-a-service offering, which this guide will utilize.

Set Up a HarperDB Instance

Before you can start using HarperDB you need to set up an instance. Note, if you would prefer to install HarperDB locally, check out the installation guides including Linux, Mac, and many other options.

1. Sign up for the HarperDB Studio

2. Create a new HarperDB Cloud instance

HarperDB Cloud instance provisioning typically takes 5-15 minutes. You will receive an email notification when your instance is ready.

Using the HarperDB Studio

Now that you have a HarperDB instance, you can do pretty much everything you’d like through the Studio. This section links to appropriate articles to get you started interacting with your data.

1. Create a schema

2. Create a table

3. Add a record

4. Load CSV data (Here’s a sample CSV of the HarperDB team’s dogs)

5. Query data via SQL

Using the HarperDB API

Complete HarperDB API documentation is available at api.harperdb.io. The HarperDB Studio features an example code builder that generates API calls in the programming language of your choice. For example purposes, a basic cURL command is shown below to create a schema called dev.

curl --location --request POST 'https://instance-subdomain.harperdbcloud.com' \
--header 'Authorization: Basic YourBase64EncodedInstanceUser:Pass' \
--header 'Content-Type: application/json' \
--data-raw '{
"operation": "create_schema",
"schema": "dev"
}'

Breaking it down, there are only a few requirements for interacting with HarperDB:

• Using the HTTP POST method.

• Providing the URL of the HarperDB instance.

• Providing the Authorization header (more on using Basic authentication).

• Providing the Content-Type header.

• Providing a JSON body with the desired operation and any additional operation properties (shown in the --data-raw parameter). This is the only parameter that needs to be changed to execute alternative operations on HarperDB.

Video Tutorials

HarperDB video tutorials are available within the HarperDB Studio. HarperDB and the HarperDB Studio are constantly changing, as such, there may be small discrepancies in UI/UX.

Install HarperDB

This documentation contains information for installing HarperDB locally. Note that if you’d like to get up and running quickly, you can try a managed instance with HarperDB Cloud. HarperDB is a cross-platform database; we recommend Linux for production use, but HarperDB can run on Windows and Mac as well, for development purposes. Installation is usually very simple and just takes a few steps, but there are a few different options documented here.

HarperDB runs on Node.js, so if you do not have it installed, you need to do that first (if you have installed, you can skip to installing HarperDB, itself). Node.js can be downloaded and installed from their site. For Linux and Mac, we recommend installing and managing Node versions with NVM, which has instructions for installation, but generally NVM can be installed with:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash

And then logout and login, and then install Node.js using nvm. We recommend using LTS, but support all currently maintained Node versions (which is currently version 14 and newer, and make sure to always uses latest minor/patch for the major version):

nvm install 18

Install and Start HarperDB

Then you can install HarperDB with NPM and start it:

npm install -g harperdb
harperdb

HarperDB will automatically start after installation.

If you are setting up a production server on Linux, we have much more extensive documentation on how to configure volumes for database storage, set up a systemd script, configure your operating system for use a database server in our linux installation guide.

With Docker

If you would like to run HarperDB in Docker, install Docker Desktop on your Mac or Windows computer. Otherwise, install the Docker Engine on your Linux server.

Once Docker Desktop or Docker Engine is installed, visit our Docker Hub page for information and examples on how to run a HarperDB container.

Offline Install

If you need to install HarperDB on a device that doesn't have an Internet connection, you can choose your version and download the npm package and install it directly (you’ll still need Node.js and NPM):

Download Install Package

Once you’ve downloaded the .tgz file, run the following command from the directory where you’ve placed it:

npm install -g harperdb-X.X.X.tgz harperdb install

For more information visit the HarperDB Command Line Interface guide.

Installation on Less Common Platforms

HarperDB comes with binaries for standard AMD64/x64 or ARM64 CPU architectures on Linux, Windows (x64 only), and Mac (including Apple Silicon). However, if you are installing on a less common platform (Alpine, for example), you will need to ensure that you have build tools installed for the installation process to compile the binaries (this is handled automatically), including:

• Go: version 1.19.1

• GCC

• Make

• Python v3.7, v3.8, v3.9, or v3.10

HarperDB's documentation covers installation, getting started, APIs, security, and much more. Browse the topics at left, or choose one of the commonly used documentation sections below.

• Install HarperDB Locally

• Getting Started

• HarperDB Operations API

• HarperDB Studio

• HarperDB Cloud

• Developer Project Examples

• Support

Start at the HarperDB Studio sign up page.

1. Provide the following information:

   • First Name

   • Last Name

   • Email Address

   • Subdomain

     Part of the URL that will be used to identify your HarperDB Cloud Instances. For example, with subdomain “demo” and instance name “c1” the instance URL would be: https://c1-demo.harperdbcloud.com.

   • Coupon Code (optional)

2. Review the Privacy Policy and Terms of Service.

3. Click the sign up for free button.

4. You will be taken to a new screen to add an account password. Enter your password. Passwords must be a minimum of 8 characters with at least 1 lower case character, 1 upper case character, 1 number, and 1 special character.

5. Click the add account password button.

You will receive a Studio welcome email confirming your registration.

Note: Your email address will be used as your username and cannot be changed.

HarperDB Studio is the web-based GUI for HarperDB. Studio enables you to administer, navigate, and monitor all of your HarperDB instances in a simple, user friendly interface without any knowledge of the underlying HarperDB API. It’s free to sign up, get started today!

Sign up for free!

How does Studio Work?

While HarperDB Studio is web based and hosted by us, all database interactions are performed on the HarperDB instance the studio is connected to. The HarperDB Studio loads in your browser, at which point you login to your HarperDB instances. Credentials are stored in your browser cache and are not transmitted back to HarperDB. All database interactions are made via the HarperDB Operations API directly from your browser to your instance.

What type of instances can I manage?

HarperDB Studio enables users to manage both HarperDB Cloud instances and privately hosted instances all from a single UI. All HarperDB instances feature identical behavior whether they are hosted by us or by you.

Resources (Marketplace, Drivers, Tutorials, & Example Code)

HarperDB Studio resources are available regardless of whether or not you are logged in.

HarperDB Marketplace

The HarperDB Marketplace is a collection of SDKs and connectors that enable developers to expand upon HarperDB for quick and easy solution development. Extensions are built and supported by the HarperDB Community. Each extension is hosted on the appropriate package manager or host.

To download a Marketplace extension:

1. Navigate to the HarperDB Marketplace page.

2. Identity the extension you would like to use.

3. Either click the link to the package.

4. Follow the extension’s instructions to proceed.

You can submit your rating for each extension by clicking on the stars.

HarperDB Drivers

HarperDB offers standard drivers to connect real-time HarperDB data with BI, analytics, reporting and data visualization technologies. Drivers are built and maintained by CData Software.

To download a driver:

1. Navigate to the HarperDB Drivers page.

2. Identity the driver you would like to use.

3. Click the download link.

4. For additional instructions, visit the support link on the driver card.

Video Tutorials

HarperDB offers video tutorials available in the Studio on the HarperDB Tutorials page as well as our YouTube channel. The HarperDB Studio is changing all the time, as a result these, the videos may not include all of the current Studio features.

Example Code

The code examples page offers example code for many different programming languages. These samples will include a placeholder for your authorization token. Full code examples with the authorization token prepopulated are available within individual instance pages.

Log In to Your HarperDB Studio Account

To log into your existing HarperDB Studio account:

1. Navigate to the HarperDB Studio.

2. Enter your email address.

3. Enter your password.

4. Click sign in.

Reset a Forgotten Password

To reset a forgotten password:

1. Navigate to the HarperDB Studio password reset page.

2. Enter your email address.

3. Click send password reset email.

4. If the account exists, you will receive an email with a temporary password.

5. Navigate back to the HarperDB Studio login page.

6. Enter your email address.

7. Enter your temporary password.

8. Click sign in.

9. You will be taken to a new screen to reset your account password. Enter your new password. Passwords must be a minimum of 8 characters with at least 1 lower case character, 1 upper case character, 1 number, and 1 special character.

10. Click the add account password button.

Change Your Password

If you are already logged into the Studio, you can change your password though the user interface.

1. Navigate to the HarperDB Studio profile page.

2. In the password section, enter:

   • Current password.

   • New password.

   • New password again (for verification).

3. Click the Update Password button.

If you wish to install locally or already have a configured server, see the basic Installation Guide

The following is a recommended way to configure Linux and install HarperDB. These instructions should work reasonably well for any public cloud or on-premises Linux instance.

These instructions assume that the following has already been completed:

1. Linux is installed

2. Basic networking is configured

3. A non-root user account dedicated to HarperDB with sudo privileges exists

4. An additional volume for storing HarperDB files is attached to the Linux instance

5. Traffic to ports 9925 (HarperDB Operations API,) 9926 (HarperDB Custom Functions,) and 9932 (HarperDB Clustering) is permitted

For this example, we will use an AWS Ubuntu Server 22.04 LTS m5.large EC2 Instance with an additional General Purpose SSD EBS volume and the default “ubuntu” user account.

(Optional) LVM Configuration

Logical Volume Manager (LVM) can be used to stripe multiple disks together to form a single logical volume. If striping disks together is not a requirement, skip these steps.

Find disk that already has a partition

used_disk=$(lsblk -P -I 259 | grep "nvme.n1.*part" | grep -o "nvme.n1")

Create array of free disks

declare -a free_disks
mapfile -t free_disks < <(lsblk -P -I 259 | grep "nvme.n1.*disk" | grep -o "nvme.n1" | grep -v "$used_disk")

Get quantity of free disks

free_disks_qty=${#free_disks[@]}

Construct pvcreate command

cmd_string=""
for i in "${free_disks[@]}"
do
cmd_string="$cmd_string /dev/$i"
done

Initialize disks for use by LVM

pvcreate_cmd="pvcreate $cmd_string"
sudo $pvcreate_cmd

Create volume group

vgcreate_cmd="vgcreate hdb_vg $cmd_string"
sudo $vgcreate_cmd

Create logical volume

sudo lvcreate -n hdb_lv -i $free_disks_qty -l 100%FREE hdb_vg

Configure Data Volume

Run lsblk and note the device name of the additional volume

lsblk

Create an ext4 filesystem on the volume (The below commands assume the device name is nvme1n1. If you used LVM to create logical volume, replace /dev/nvme1n1 with /dev/hdb_vg/hdb_lv)

sudo mkfs.ext4 -L hdb_data /dev/nvme1n1

Mount the file system and set the correct permissions for the directory

mkdir /home/ubuntu/hdb
sudo mount -t ext4 /dev/nvme1n1 /home/ubuntu/hdb
sudo chown -R ubuntu:ubuntu /home/ubuntu/hdb
sudo chmod 775 /home/ubuntu/hdb

Create a fstab entry to mount the filesystem on boot

echo "LABEL=hdb_data /home/ubuntu/hdb ext4 defaults,noatime 0 1" | sudo tee -a /etc/fstab

Configure Linux and Install Prerequisites

If a swap file or partition does not already exist, create and enable a 2GB swap file

sudo dd if=/dev/zero of=/swapfile bs=128M count=16
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo "/swapfile swap swap defaults 0 0" | sudo tee -a /etc/fstab

Increase the open file limits for the ubuntu user

echo "ubuntu soft nofile 500000" | sudo tee -a /etc/security/limits.conf
echo "ubuntu hard nofile 1000000" | sudo tee -a /etc/security/limits.conf

Install Node Version Manager (nvm)

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash

Load nvm (or logout and then login)

. ~/.nvm/nvm.sh

Install Node.js using nvm (read more about specific Node version requirements)

nvm install <the node version>

Install and Start HarperDB

Here is an example of installing HarperDB with minimal configuration.

npm install -g harperdb
harperdb start \
  --TC_AGREEMENT "yes" \
  --ROOTPATH "/home/ubuntu/hdb" \
  --OPERATIONSAPI_NETWORK_PORT "9925" \
  --HDB_ADMIN_USERNAME "HDB_ADMIN" \
  --HDB_ADMIN_PASSWORD "password"

Here is an example of installing HarperDB with commonly used additional configuration.

npm install -g harperdb
harperdb start \
  --TC_AGREEMENT "yes" \
  --ROOTPATH "/home/ubuntu/hdb" \
  --OPERATIONSAPI_NETWORK_PORT "9925" \
  --HDB_ADMIN_USERNAME "HDB_ADMIN" \
  --HDB_ADMIN_PASSWORD "password" \
  --OPERATIONSAPI_NETWORK_HTTPS "true" \
  --CUSTOMFUNCTIONS_NETWORK_HTTPS "true" \
  --CLUSTERING_ENABLED "true" \
  --CLUSTERING_USER "cluster_user" \
  --CLUSTERING_PASSWORD "password" \
  --CLUSTERING_NODENAME "hdb1"

HarperDB will automatically start after installation. If you wish HarperDB to start when the OS boots, you have two options

You can set up a crontab:

(crontab -l 2>/dev/null; echo "@reboot PATH=\"/home/ubuntu/.nvm/versions/node/v18.15.0/bin:$PATH\" && harperdb start") | crontab -

Or you can create a systemd script at /etc/systemd/system/harperdb.service

Pasting the following contents into the file:

[Unit]
Description=HarperDB

[Service]
Type=simple
Restart=always
User=ubuntu
Group=ubuntu
WorkingDirectory=/home/ubuntu
ExecStart=/bin/bash -c 'PATH="/home/ubuntu/.nvm/versions/node/v18.15.0/bin:$PATH"; harperdb'

[Install]
WantedBy=multi-user.target

And then running the following:

systemctl daemon-reload
systemctl enable harperdb

For more information visit the HarperDB Command Line Interface guide and the HarperDB Configuration File guide.

Manage instance schemas/tables and browse data in tabular format with the following instructions:

1. Navigate to the HarperDB Studio Organizations page.

2. Click the appropriate organization that the instance belongs to.

3. Select your desired instance.

4. Click browse in the instance control bar.

Once on the instance browse page you can view data, manage schemas and tables, add new data, and more.

Manage Schemas and Tables

Create a Schema

1. Click the plus icon at the top right of the schemas section.

2. Enter the schema name.

3. Click the green check mark.

Delete a Schema

Deleting a schema is permanent and irreversible. Deleting a schema removes all tables and data within it.

1. Click the minus icon at the top right of the schemas section.

2. Identify the appropriate schema to delete and click the red minus sign in the same row.

3. Click the red check mark to confirm deletion.

Create a Table

1. Select the desired schema from the schemas section.

2. Click the plus icon at the top right of the tables section.

3. Enter the table name.

4. Enter the primary key.

   The primary key is also often referred to as the hash attribute in the studio, and it defines the unique identifier for each row in your table.

5. Click the green check mark.

Delete a Table

Deleting a table is permanent and irreversible. Deleting a table removes all data within it.

1. Select the desired schema from the schemas section.

2. Click the minus icon at the top right of the tables section.

3. Identify the appropriate table to delete and click the red minus sign in the same row.

4. Click the red check mark to confirm deletion.

Manage Table Data

The following section assumes you have selected the appropriate table from the schema/table browser.

Filter Table Data

1. Click the magnifying glass icon at the top right of the table browser.

2. This expands the search filters.

3. The results will be filtered appropriately.

Load CSV Data

1. Click the data icon at the top right of the table browser. You will be directed to the CSV upload page where you can choose to import a CSV by URL or upload a CSV file.

2. To import a CSV by URL:

   1. Enter the URL in the CSV file URL textbox.

   2. Click Import From URL.

   3. The CSV will load, and you will be redirected back to browse table data.

3. To upload a CSV file:

   1. Click Click or Drag to select a .csv file (or drag your CSV file from your file browser).

   2. Navigate to your desired CSV file and select it.

   3. Click Insert X Records, where X is the number of records in your CSV.

   4. The CSV will load, and you will be redirected back to browse table data.

Add a Record

1. Click the plus icon at the top right of the table browser.

2. The Studio will pre-populate existing table attributes in JSON format.

   The primary key is not included, but you can add it in and set it to your desired value. Auto-maintained fields are not included and cannot be manually set. You may enter a JSON array to insert multiple records in a single transaction.

3. Enter values to be added to the record.

   You may add new attributes to the JSON; they will be reflexively added to the table.

4. Click the Add New button.

Edit a Record

1. Click the record/row you would like to edit.

2. Modify the desired values.

   You may add new attributes to the JSON; they will be reflexively added to the table.

3. Click the save icon.

Delete a Record

Deleting a record is permanent and irreversible. If transaction logging is turned on, the delete transaction will be recorded as well as the data that was deleted.

1. Click the record/row you would like to delete.

2. Click the delete icon.

3. Confirm deletion by clicking the check icon.

Browse Table Data

The following section assumes you have selected the appropriate table from the schema/table browser.

Browse Table Data

The first page of table data is automatically loaded on table selection. Paging controls are at the bottom of the table. Here you can:

• Page left and right using the arrows.

• Type in the desired page.

• Change the page size (the amount of records displayed in the table).

Refresh Table Data

Click the refresh icon at the top right of the table browser.

Automatically Refresh Table Data

Toggle the auto switch at the top right of the table browser. The table data will now automatically refresh every 15 seconds. Filters and pages will remain set for refreshed data.

HarperDB Studio organizations provide the ability to group HarperDB Cloud Instances. Organization behavior is as follows:

• Billing occurs at the organization level to a single credit card.

• Organizations retain their own unique HarperDB Cloud subdomain.

• Cloud instances reside within an organization.

• Studio users can be invited to organizations to share instances.

An organization is automatically created for you when you sign up for HarperDB Studio. If you only have one organization, the Studio will automatically bring you to your organization’s page.

List Organizations

A summary view of all organizations your user belongs to can be viewed on the page. You can navigate to this page at any time by clicking the all organizations link at the top of the HarperDB Studio.

Create a New Organization

A new organization can be created as follows:

1. Navigate to the page.

2. Click the Create a New Organization card.

3. Fill out new organization details

   • Enter Organization Name This is used for descriptive purposes only.

   • Enter Organization Subdomain Part of the URL that will be used to identify your HarperDB Cloud Instances. For example, with subdomain “demo” and instance name “c1” the instance URL would be: https://c1-demo.harperdbcloud.com.

4. Click Create Organization.

Delete an Organization

An organization cannot be deleted until all instances have been removed. An organization can be deleted as follows:

1. Navigate to the HarperDB Studio Organizations page.

2. Identify the proper organization card and click the trash can icon.

3. Enter the organization name into the text box.

   This is done for confirmation purposes to ensure you do not accidentally delete an organization.

4. Click the Do It button.

Manage Users

HarperDB Studio organization owners can manage users including inviting new users, removing users, and toggling ownership.

Inviting a User

A new user can be invited to an organization as follows:

2. Click the appropriate organization card.

3. Click users at the top of the screen.

4. In the add user box, enter the new user’s email address.

5. Click Add User.

Users may or may not already be HarperDB Studio users when adding them to an organization. If the HarperDB Studio account already exists, the user will receive an email notification alerting them to the organization invitation. If the user does not have a HarperDB Studio account, they will receive an email welcoming them to HarperDB Studio.

Toggle a User’s Organization Owner Status

Organization owners have full access to the organization including the ability to manage organization users, create, modify, and delete instances, and delete the organization. Users must have accepted their invitation prior to being promoted to an owner. A user’s organization owner status can be toggled owner as follows:

1. Navigate to the HarperDB Studio Organizations page.

2. Click the appropriate organization card.

3. Click users at the top of the screen.

4. Click the appropriate user from the existing users section.

5. Toggle the Is Owner switch to the desired status.

Remove a User from an Organization

Users may be removed from an organization at any time. Removing a user from an organization will not delete their HarperDB Studio account, it will only remove their access to the specified organization. A user can be removed from an organization as follows:

2. Click the appropriate organization card.

3. Click users at the top of the screen.

4. Click the appropriate user from the existing users section.

5. Type DELETE in the text box in the Delete User row.

   This is done for confirmation purposes to ensure you do not accidentally delete a user.

6. Click Delete User.

Manage Billing

Billing is configured per organization and will be billed to the stored credit card at appropriate intervals (monthly or annually depending on the registered instance). Billing settings can be configured as follows:

2. Click the appropriate organization card.

3. Click billing at the top of the screen.

Here organization owners can view invoices, manage coupons, and manage the associated credit card.

HarperDB billing and payments are managed via Stripe.

Add a Coupon

Coupons are applicable towards any paid tier or user-installed instance and you can change your subscription at any time. Coupons can be added to your Organization as follows:

1. In the coupons panel of the billing page, enter your coupon code.

2. Click Add Coupon.

3. The coupon will then be available and displayed in the coupons panel.

Example code prepopulated with the instance URL and authorization token for the logged in database user can be found on the example code page of the HarperDB Studio. Code samples are generated based on the HarperDB API Documentation Postman collection. Code samples accessed with the following instructions:

1. Navigate to the page.

2. Click the appropriate organization that the instance belongs to.

3. Select your desired instance.

4. Click example code in the instance control bar.

5. Select the appropriate category from the left navigation.

6. Select the appropriate operation from the left navigation.

7. Select your desired language/variant from the Choose Programming Language dropdown.

8. Copy code from the sample code panel using the copy icon.

Supported Languages

Sample code uses two identifiers: language and variant.

• language is the programming language that the sample code is generated in.

• variant is the methodology or library used by the language to send HarperDB requests.

The list of available language/variants are as follows:

SQL queries can be executed directly through the HarperDB Studio with the following instructions:

1. Navigate to the page.

2. Click the appropriate organization that the instance belongs to.

3. Select your desired instance.

4. Click query in the instance control bar.

5. Enter your SQL query in the SQL query window.

6. Click Execute.

Please note, the Studio will execute the query exactly as entered. For example, if you attempt to SELECT * from a table with millions of rows, you will most likely crash your browser.

Browse Query Results Set

Browse Results Set Data

The first page of results set data is automatically loaded on query execution. Paging controls are at the bottom of the table. Here you can:

• Page left and right using the arrows.

• Type in the desired page.

• Change the page size (the amount of records displayed in the table).

Refresh Results Set

Click the refresh icon at the top right of the results set table.

Automatically Refresh Results Set

Toggle the auto switch at the top right of the results set table. The results set will now automatically refresh every 15 seconds. Filters and pages will remain set for refreshed data.

Query History

Query history is stored in your local browser cache. Executed queries are listed with the most recent at the top in the query history section.

Rerun Previous Query

• Identify the query from the query history list.

• Click the appropriate query. It will be loaded into the sql query input box.

• Click Execute.

Clear Query History

Click the trash can icon at the top right of the query history section.

Create Charts

The HarperDB Studio includes a charting feature where you can build charts based on your specified queries. Visit the Charts documentation for more information.

The HarperDB Studio includes a charting feature within an instance. They are generated in real time based on your existing data and automatically refreshed every 15 seconds. Instance charts can be accessed with the following instructions:

1. Navigate to the page.

2. Click the appropriate organization that the instance belongs to.

3. Select your desired instance.

4. Click charts in the instance control bar.

Creating a New Chart

Charts are generated based on SQL queries, therefore to build a new chart you first need to build a query. Instructions as follows (starting on the charts page described above):

1. Click query in the instance control bar.

2. Enter the SQL query you would like to generate a chart from.

   For example, using the dog demo data from the API Docs, we can get the average dog age per owner with the following query: SELECT AVG(age) as avg_age, owner_name FROM dev.dog GROUP BY owner_name.

3. Click Execute.

4. Click create chart at the top right of the results table.

5. Configure your chart.

   1. Choose chart type.

      HarperDB Studio offers many standard charting options like line, bar, etc.

   2. Choose a data column.

      This column will be used to plot the data point. Typically, this is the values being calculated in the SELECT statement. Depending on the chart type, you can select multiple data columns to display on a single chart.

   3. Depending on the chart type, you will need to select a grouping.

      This could be labeled as x-axis, label, etc. This will be used to group the data, typically this is what you used in your GROUP BY clause.

   4. Enter a chart name.

      Used for identification purposes and will be displayed at the top of the chart.

   5. Choose visible to all org users toggle.

      Leaving this option off will limit chart visibility to just your HarperDB Studio user. Toggling it on will enable all users with this Organization to view this chart.

   6. Click Add Chart.

   7. The chart will now be visible on the charts page.

The example query above, configured as a bar chart, results in the following chart:

Downloading Charts

HarperDB Studio charts can be downloaded in SVG, PNG, and CSV format. Instructions as follows (starting on the charts page described above):

1. Identify the chart you would like to export.

2. Click the three bars icon.

3. Select the appropriate download option.

4. The Studio will generate the export and begin downloading immediately.

Delete a Chart

Delete a chart as follows (starting on the charts page described above):

1. Identify the chart you would like to delete.

2. Click the X icon.

3. Click the confirm delete chart button.

4. The chart will be deleted.

Deleting a chart that is visible to all Organization users will delete it for all users.

HarperDB instance clustering and replication can be configured directly through the HarperDB Studio. It is recommended to read through the clustering documentation first to gain a strong understanding of HarperDB clustering behavior.

All clustering configuration is handled through the cluster page of the HarperDB Studio, accessed with the following instructions:

1. Navigate to the page.

2. Click the appropriate organization that the instance belongs to.

3. Select your desired instance.

4. Click cluster in the instance control bar.

Note, the cluster page will only be available to super users.

Initial Configuration

HarperDB instances do not have clustering configured by default. The HarperDB Studio will walk you through the initial configuration. Upon entering the cluster screen for the first time you will need to complete the following configuration. Configurations are set in the enable clustering panel on the left while actions are described in the middle of the screen.

1. Create a cluster user, read more about this here: Clustering Users and Roles.

   • Enter username.

   • Enter password.

   • Click Create Cluster User.

2. Click Set Cluster Node Name.

3. Click Enable Instance Clustering.

At this point the Studio will restart your HarperDB Instance, required for the configuration changes to take effect.

Manage Clustering

Once initial clustering configuration is completed you a presented with a clustering management screen with the following properties:

• connected instances

  Displays all instances within the Studio Organization that this instance manages a connection with.

• unconnected instances

  Displays all instances within the Studio Organization that this instance does not manage a connection with.

• unregistered instances

  Displays all instances outside of the Studio Organization that this instance manages a connection with.

• manage clustering

  Once instances are connected, this will display clustering management options for all connected instances and all schemas and tables.

Connect an Instance

HarperDB Instances can be clustered together with the following instructions.

1. Ensure clustering has been configured on both instances and a cluster user with identical credentials exists on both.

2. Identify the instance you would like to connect from the unconnected instances panel.

3. Click the plus icon next the appropriate instance.

4. If configurations are correct, all schemas will sync across the cluster, then appear in the manage clustering panel. If there is a configuration issue, a red exclamation icon will appear, click it to learn more about what could be causing the issue.

Disconnect an Instance

HarperDB Instances can be disconnected with the following instructions.

1. Identify the instance you would like to disconnect from the connected instances panel.

2. Click the minus icon next the appropriate instance.

Manage Replication

Subscriptions must be configured in order to move data between connected instances. Read more about subscriptions here: Creating A Subscription. The manage clustering panel displays a table with each row representing an channel per instance. Cells are bolded to indicate a change in the column. Publish and subscribe replication can be configured per table with the following instructions:

1. Identify the instance, schema, and table for replication to be configured.

2. For publish, click the toggle switch in the publish column.

3. For subscribe, click the toggle switch in the subscribe column.

The HarperDB Studio allows you to administer all of your HarperDB instances in one place. HarperDB currently offers the following instance types:

• HarperDB Cloud Instance Managed installations of HarperDB, what we call .

• 5G Wavelength Instance Managed installations of HarperDB running on the Verizon network through AWS Wavelength, what we call . Note, these instances are only accessible via the Verizon network.

• User-Installed Instance Any HarperDB installation that is managed by you. These include instances hosted within your cloud provider accounts (for example, from the AWS or Digital Ocean Marketplaces), privately hosted instances, or instances installed locally.

All interactions between the Studio and your instances take place directly from your browser. HarperDB stores metadata about your instances, which enables the Studio to display these instances when you log in. Beyond that, all traffic is routed from your browser to the HarperDB instances using the standard .

Organization Instance List

A summary view of all instances within an organization can be viewed by clicking on the appropriate organization from the page. Each instance gets their own card. HarperDB Cloud and user-installed instances are listed together.

Create a New Instance

1. Navigate to the page.

2. Click the appropriate organization for the instance to be created under.

3. Click the Create New HarperDB Cloud Instance + Register User-Installed Instance card.

4. Select your desired Instance Type.

5. For a HarperDB Cloud Instance or a HarperDB 5G Wavelength Instance, click Create HarperDB Cloud Instance.

   1. Fill out Instance Info.

      1. Enter Instance Name

         This will be used to build your instance URL. For example, with subdomain “demo” and instance name “c1” the instance URL would be: https://c1-demo.harperdbcloud.com. The Instance URL will be previewed below.

      2. Enter Instance Username

         This is the username of the initial HarperDB instance super user.

      3. Enter Instance Password

         This is the password of the initial HarperDB instance super user.

   2. Click Instance Details to move to the next page.

   3. Select Instance Specs

      1. Select Instance RAM

         HarperDB Cloud Instances are billed based on Instance RAM, this will select the size of your provisioned instance. .

      2. Select Storage Size

         Each instance has a mounted storage volume where your HarperDB data will reside. Storage is provisioned based on space and IOPS. .

      3. Select Instance Region

         The geographic area where your instance will be provisioned.

   4. Click Confirm Instance Details to move to the next page.

   5. Review your Instance Details, if there is an error, use the back button to correct it.

   6. Review the and , if you agree, click the I agree radio button to confirm.

   7. Click Add Instance.

   8. Your HarperDB Cloud instance will be provisioned in the background. Provisioning typically takes 5-15 minutes. You will receive an email notification when your instance is ready.

Register User-Installed Instance

2. Click the appropriate organization for the instance to be created under.

3. Click the Create New HarperDB Cloud Instance + Register User-Installed Instance card.

4. Select Register User-Installed Instance.

   1. Fill out Instance Info.

      1. Enter Instance Name

         This is used for descriptive purposes only.

      2. Enter Instance Username

         The username of a HarperDB super user that is already configured in your HarperDB installation.

      3. Enter Instance Password

         The password of a HarperDB super user that is already configured in your HarperDB installation.

      4. Enter Host

         The host to access the HarperDB instance. For example, harperdb.myhost.com or localhost.

      5. Enter Port

         The port to access the HarperDB instance. HarperDB defaults 9925.

      6. Select SSL

         If your instance is running over SSL, select the SSL checkbox. If not, you will need to enable mixed content in your browser to allow the HTTPS Studio to access the HTTP instance. If there are issues connecting to the instance, the Studio will display a red error message.

   2. Click Instance Details to move to the next page.

   3. Select Instance Specs

      1. Select Instance RAM

         HarperDB instances are billed based on Instance RAM. Selecting additional RAM will enable the ability for faster and more complex queries.

   4. Click Confirm Instance Details to move to the next page.

   5. Review your Instance Details, if there is an error, use the back button to correct it.

   7. Click Add Instance.

   8. The HarperDB Studio will register your instance and restart it for the registration to take effect. Your instance will be immediately available after this is complete.

Delete an Instance

Instance deletion has two different behaviors depending on the instance type.

• HarperDB Cloud Instance This instance will be permanently deleted, including all data. This process is irreversible and cannot be undone.

• User-Installed Instance The instance will be removed from the HarperDB Studio only. This does not uninstall HarperDB from your system and your data will remain intact.

An instance can be deleted as follows:

2. Click the appropriate organization that the instance belongs to.

3. Identify the proper instance card and click the trash can icon.

4. Enter the instance name into the text box.

   This is done for confirmation purposes to ensure you do not accidentally delete an instance.

5. Click the Do It button.

Upgrade an Instance

Instance Log In/Log Out

The Studio enables users to log in and out of different database users from the instance control panel. To log out of an instance:

2. Click the appropriate organization that the instance belongs to.

3. Identify the proper instance card and click the lock icon.

4. You will immediately be logged out of the instance.

To log in to an instance:

2. Click the appropriate organization that the instance belongs to.

3. Identify the proper instance card, it will have an unlocked icon and a status reading PLEASE LOG IN, and click the center of the card.

4. Enter the database username.

   The username of a HarperDB user that is already configured in your HarperDB instance.

5. Enter the database password.

   The password of a HarperDB user that is already configured in your HarperDB instance.

6. Click Log In.

HarperDB users can be managed directly through the HarperDB Studio. It is recommended to read through the users & roles documentation to gain a strong understanding of how they operate.

Instance role configuration is handled through the roles page of the HarperDB Studio, accessed with the following instructions:

1. Navigate to the HarperDB Studio Organizations page.

2. Click the appropriate organization that the instance belongs to.

3. Select your desired instance.

4. Click rules in the instance control bar.

Note, the roles page will only be available to super users.

The roles management screen consists of the following panels:

• super users

  Displays all super user roles for this instance.

• cluster users

  Displays all cluster user roles for this instance.

• standard roles

  Displays all standard roles for this instance.

• role permission editing

  Once a role is selected for editing, permissions will be displayed here in JSON format.

Note, when new tables are added that are not configured, the Studio will generate configuration values with permissions defaulting to false.

Role Management

Create a Role

1. Click the plus icon at the top right of the appropriate role section.

2. Enter the role name.

3. Click the green check mark.

4. Configure the role permissions in the role permission editing panel.

   Note, to have the Studio generate attribute permissions JSON, toggle show all attributes at the top right of the role permission editing panel.

5. Click Update Role Permissions.

Modify a Role

1. Click the appropriate role from the appropriate role section.

2. Modify the role permissions in the role permission editing panel.

   Note, to have the Studio generate attribute permissions JSON, toggle show all attributes at the top right of the role permission editing panel.

3. Click Update Role Permissions.

Delete a Role

Deleting a role is permanent and irreversible. A role cannot be remove if users are associated with it.

1. Click the minus icon at the top right of the schemas section.

2. Identify the appropriate role to delete and click the red minus sign in the same row.

3. Click the red check mark to confirm deletion.

Enabling mixed content is required in cases where you would like to connect the HarperDB Studio to HarperDB Instances via HTTP. This should not be used for production systems, but may be convenient for development and testing purposes. Doing so will allow your browser to reach HTTP traffic, which is considered insecure, through an HTTPS site like the Studio.

A comprehensive guide is provided by Adobe here.

The HarperDB Studio display instance status and metrics on the instance status page, which can be accessed with the following instructions:

1. Navigate to the HarperDB Studio Organizations page.

2. Click the appropriate organization that the instance belongs to.

3. Select your desired instance.

4. Click status in the instance control bar.

Once on the instance browse page you can view host system information, HarperDB logs, and HarperDB Cloud alarms (if it is a cloud instance).

Note, the status page will only be available to super users.

HarperDB instance clustering and replication can be configured directly through the HarperDB Studio. It is recommended to read through the clustering documentation first to gain a strong understanding of HarperDB clustering behavior.

Instance user configuration is handled through the users page of the HarperDB Studio, accessed with the following instructions:

1. Navigate to the HarperDB Studio Organizations page.

2. Click the appropriate organization that the instance belongs to.

3. Select your desired instance.

4. Click users in the instance control bar.

Note, the users page will only be available to super users.

Add a User

HarperDB instance users can be added with the following instructions.

1. In the add user panel on the left enter:

   • New user username.

   • New user password.

   • Select a role.

     Learn more about role management here: Manage Instance Roles.

2. Click Add User.

Edit a User

HarperDB instance users can be modified with the following instructions.

1. In the existing users panel, click the row of the user you would like to edit.

2. To change a user’s password:

   1. In the Change user password section, enter the new password.

   2. Click Update Password.

3. To change a user’s role:

   1. In the Change user role section, select the new role.

   2. Click Update Role.

4. To delete a user:

   1. In the Delete User section, type the username into the textbox.

      This is done for confirmation purposes.

   2. Click Delete User.

HarperDB Custom Functions are enabled by default and can be configured further through the HarperDB Studio. It is recommended to read through the Custom Functions documentation first to gain a strong understanding of HarperDB Custom Functions behavior.

All Custom Functions configuration is handled through the functions page of the HarperDB Studio, accessed with the following instructions:

1. Navigate to the HarperDB Studio Organizations page.

2. Click the appropriate organization that the instance belongs to.

3. Select your desired instance.

4. Click functions in the instance control bar.

Note, the functions page will only be available to super users.

Manage Projects

On the functions page of the HarperDB Studio you are presented with a functions management screen with the following properties:

• projects

  Displays a list of Custom Functions projects residing on this instance.

• /project_name/routes

  Only displayed if there is an existing project. Displays the routes files contained within the selected project.

• /project_name/helpers

  Only displayed if there is an existing project. Displays the helper files contained within the selected project.

• /project_name/static

  Only displayed if there is an existing project. Displays the static file count and a link to the static files contained within the selected project. Note, static files cannot currently be deployed through the Studio and must be deployed via the HarperDB API or manually to the server (not applicable with HarperDB Cloud).

• Root File Directory

  Displays the root file directory where the Custom Functions projects reside on this instance.

• Custom Functions Server URL

  Displays the base URL in which all Custom Functions are accessed for this instance.

Create a Project

HarperDB Custom Functions Projects can be initialized with the following instructions.

1. If this is your first project, skip this step. Click the plus icon next to the projects heading.

2. Enter the project name in the text box located under the projects heading.

3. Click the check mark icon next the appropriate instance.

4. The Studio will take a few moments to provision a new project based on the Custom Functions template.

5. The Custom Functions project is now created and ready to modify.

Modify a Project

Custom Functions routes and helper functions can be modified directly through the Studio. From the functions page:

1. Select the appropriate project.

2. Select the appropriate route or helper.

3. Modify the code with your desired changes.

4. Click the save icon at the bottom right of the screen.

   Note, saving modifications will restart the Custom Functions server on your HarperDB instance and may result in up to 60 seconds of downtime for all Custom Functions.

Create Additional Routes/Helpers

To create an additional route to your Custom Functions project. From the functions page:

1. Select the appropriate Custom Functions project.

2. Click the plus icon to the right of the routes header.

3. Enter the name of the new route in the textbox that appears.

4. Click the check icon to create the new route.

   Note, adding a route will restart the Custom Functions server on your HarperDB instance and may result in up to 60 seconds of downtime for all Custom Functions.

To create an additional helper to your Custom Functions project. From the functions page:

1. Select the appropriate Custom Functions project.

2. Click the plus icon to the right of the helpers header.

3. Enter the name of the new helper in the textbox that appears.

4. Click the check icon to create the new helper.

   Note, adding a helper will restart the Custom Functions server on your HarperDB instance and may result in up to 60 seconds of downtime for all Custom Functions.

Delete a Project/Route/Helper

To delete a Custom Functions project from the functions page:

1. Click the minus icon to the right of the projects header.

2. Click the red minus icon to the right of the Custom Functions project you would like to delete.

3. Confirm deletion by clicking the red check icon.

   Note, deleting a project will restart the Custom Functions server on your HarperDB instance and may result in up to 60 seconds of downtime for all Custom Functions.

To delete a Custom Functions project route from the functions page:

1. Select the appropriate Custom Functions project.

2. Click the minus icon to the right of the routes header.

3. Click the red minus icon to the right of the Custom Functions route you would like to delete.

4. Confirm deletion by clicking the red check icon.

   Note, deleting a route will restart the Custom Functions server on your HarperDB instance and may result in up to 60 seconds of downtime for all Custom Functions.

To delete a Custom Functions project helper from the functions page:

1. Select the appropriate Custom Functions project.

2. Click the minus icon to the right of the helper header.

3. Click the red minus icon to the right of the Custom Functions header you would like to delete.

4. Confirm deletion by clicking the red check icon.

   Note, deleting a header will restart the Custom Functions server on your HarperDB instance and may result in up to 60 seconds of downtime for all Custom Functions.

Deploy Custom Functions Project to Other Instances

The HarperDB Studio provides the ability to deploy Custom Functions projects to additional HarperDB instances within the same Studio Organization. To deploy Custom Functions projects to additional instances, starting from the functions page:

1. Select the project you would like to deploy.

2. Click the deploy button at the top right.

3. A list of instances (excluding the current instance) within the organization will be displayed in tabular with the following information:

   • Instance Name: The name used to describe the instance.

   • Instance URL: The URL used to access the instance.

   • CF Capable: Describes if the instance version supports Custom Functions (yes/no).

   • CF Enabled: Describes if Custom Functions are configured and enabled on the instance (yes/no).

   • Has Project: Describes if the selected Custom Functions project has been previously deployed to the instance (yes/no).

   • Deploy: Button used to deploy the project to the instance.

   • Remote: Button used to remove the project from the instance. Note, this will only be visible if the project has been previously deployed to the instance.

4. In the appropriate instance row, click the deploy button.

   Note, deploying a project will restart the Custom Functions server on the HarperDB instance receiving the deployment and may result in up to 60 seconds of downtime for all Custom Functions.

HarperDB instance configuration can be viewed and managed directly through the HarperDB Studio. HarperDB Cloud instances can be resized in two different ways via this page, either by modifying machine RAM or by increasing drive storage. User-installed instances can have their licenses modified by modifying licensed RAM.

All instance configuration is handled through the config page of the HarperDB Studio, accessed with the following instructions:

1. Navigate to the page.

2. Click the appropriate organization that the instance belongs to.

3. Select your desired instance.

4. Click config in the instance control bar.

Note, the config page will only be available to super users and certain items are restricted to Studio organization owners.

Instance Overview

The instance overview panel displays the following instance specifications:

• Instance URL

• Instance Node Name (for clustering)

• Instance API Auth Header (this user)

  The Basic authentication header used for the logged in HarperDB database user

• Created Date (HarperDB Cloud only)

• Region (HarperDB Cloud only)

  The geographic region where the instance is hosted.

• Total Price

• RAM

• Storage (HarperDB Cloud only)

• Disk IOPS (HarperDB Cloud only)

Update Instance RAM

HarperDB Cloud instance size and user-installed instance licenses can be modified with the following instructions. This option is only available to Studio organization owners.

Note: For HarperDB Cloud instances, upgrading RAM may add additional CPUs to your instance as well. Click here to see how many CPUs are provisioned for each instance size.

1. In the update ram panel at the bottom left:

   • Select the new instance size.

   • If you do not have a credit card associated with your account, an Add Credit Card To Account button will appear. Click that to be taken to the billing screen where you can enter your credit card information before returning to the config tab to proceed with the upgrade.

   • If you do have a credit card associated, you will be presented with the updated billing information.

   • Click Upgrade.

2. The instance will shut down and begin reprovisioning/relicensing itself. The instance will not be available during this time. You will be returned to the instance dashboard and the instance status will show UPDATING INSTANCE.

3. Once your instance upgrade is complete, it will appear on the instance dashboard as status OK with your newly selected instance size.

Note, if HarperDB Cloud instance reprovisioning takes longer than 20 minutes, please submit a support ticket here: https://harperdbhelp.zendesk.com/hc/en-us/requests/new.

Update Instance Storage

The HarperDB Cloud instance storage size can be increased with the following instructions. This option is only available to Studio organization owners.

Note: Instance storage can only be upgraded once every 6 hours.

1. In the update storage panel at the bottom left:

   • Select the new instance storage size.

   • If you do not have a credit card associated with your account, an Add Credit Card To Account button will appear. Click that to be taken to the billing screen where you can enter your credit card information before returning to the config tab to proceed with the upgrade.

   • If you do have a credit card associated, you will be presented with the updated billing information.

   • Click Upgrade.

2. The instance will shut down and begin reprovisioning itself. The instance will not be available during this time. You will be returned to the instance dashboard and the instance status will show UPDATING INSTANCE.

3. Once your instance upgrade is complete, it will appear on the instance dashboard as status OK with your newly selected instance size.

Note, if this process takes longer than 20 minutes, please submit a support ticket here: https://harperdbhelp.zendesk.com/hc/en-us/requests/new.

Remove Instance

The HarperDB instance can be deleted/removed from the Studio with the following instructions. Once this operation is started it cannot be undone. This option is only available to Studio organization owners.

1. In the remove instance panel at the bottom left:

   • Enter the instance name in the text box.

   • The Studio will present you with a warning.

   • Click Remove.

2. The instance will begin deleting immediately.

Restart Instance

The HarperDB Cloud instance can be restarted with the following instructions.

1. In the restart instance panel at the bottom right:

   • Enter the instance name in the text box.

   • The Studio will present you with a warning.

   • Click Restart.

2. The instance will begin restarting immediately.

HarperDB Cloud instance alarms are triggered when certain conditions are met. Once alarms are triggered organization owners will immediately receive an email alert and the alert will be available on the page. The below table describes each alert and their evaluation metrics.

Heading Definitions

• Alarm: Title of the alarm.

• Threshold: Definition of the alarm threshold.

• Intervals: The number of occurrences before an alarm is triggered and the period that the metric is evaluated over.

• Proposed Remedy: Recommended solution to avoid the alert in the future.

Clustering does not run by default; it needs to be enabled.

To enable clustering the clustering.enabled configuration element in the harperdb-config.yaml file must be set to true.

There are multiple ways to update this element, they are:

1. Directly editing the harperdb-config.yaml file and setting enabled to true

clustering:
  enabled: true

Note: When making any changes to the harperdb-config.yaml file HarperDB must be restarted for the changes to take effect.

1. Calling set_configuration through the operations API

{
    "operation": "set_configuration",
    "clustering_enabled": true
}

Note: When making any changes to HarperDB configuration HarperDB must be restarted for the changes to take effect.

1. Using command line variables.

harperdb --CLUSTERING_ENABLED true

1. Using environment variables.

CLUSTERING_ENABLED=true

An efficient way to install HarperDB, create the cluster user, set the node name and enable clustering in one operation is to combine the steps using command line and/or environment variables. Here is an example using command line variables.

harperdb install --CLUSTERING_ENABLED true --CLUSTERING_NODENAME Node1 --CLUSTERING_USER cluster_account --CLUSTERING_PASSWORD letsCluster123!

HarperDB Cloud is the easiest way to test drive HarperDB, it’s HarperDB-as-a-Service. Cloud handles deployment and management of your instances in just a few clicks. HarperDB Cloud is currently powered by AWS with additional cloud providers on our roadmap for the future.

These instances are only accessible from the Verizon network. When accessing your HarperDB instance please ensure you are connected to the Verizon network, examples include Verizon 5G Internet, Verizon Hotspots, or Verizon mobile devices.

HarperDB on Verizon 5G Wavelength brings HarperDB closer to the end user exclusively on the Verizon network resulting in as little as single-digit millisecond response time from HarperDB to the client.

Instances are built via AWS Wavelength. You can read more about AWS Wavelength here.

HarperDB 5G Wavelength Instance Specs While HarperDB 5G Wavelength bills by RAM, each instance has other specifications associated with the RAM selection. The following table describes each instance size in detail*.

*Specifications are subject to change. For the most up to date information, please refer to AWS documentation.

HarperDB 5G Wavelength Storage

HarperDB 5G Wavelength utilizes AWS Elastic Block Storage (EBS) General Purpose SSD (gp2) volumes. This is the most common storage type used in AWS, as it provides reasonable performance for most workloads, at a reasonable price.

AWS EBS gp2 volumes have a baseline performance level, which determines the number of IOPS it can perform indefinitely. The larger the volume, the higher it’s baseline performance. Additionally, smaller gp2 volumes are able to burst to a higher number of IOPS for periods of time.

Smaller gp2 volumes are perfect for trying out the functionality of HarperDB, and might also work well for applications that don’t perform many database transactions. For applications that perform a moderate or high number of transactions, we recommend that you use a larger HarperDB volume. Learn more about the impact of IOPS on performance here.

You can read more about AWS EBS gp2 volume IOPS here.

HarperDB, like any database, can place a tremendous load on its storage resources. Storage, not CPU or memory, will more often be the bottleneck of server, virtual machine, or a container running HarperDB. Understanding how storage works, and how much storage performance your workload requires, is key to ensuring that HarperDB performs as expected.

IOPS Overview

The primary measure of storage performance is the number of input/output operations per second (IOPS) that a storage device can perform. Different storage devices can have dramatically different performance profiles. A hard drive (HDD) might only perform a hundred or so IOPS, while a solid state drive (SSD) might be able to perform tens or hundreds of thousands of IOPS.

Cloud providers like AWS, which powers HarperDB Cloud, don’t typically attach individual disks to a virtual machine or container. Instead, they combine large numbers of storage drives to create very high performance storage servers. Chunks (volumes) of that storage is then carved out and presented to many different virtual machines and containers. Due to the shared nature of this type of storage, the cloud provider places configurable limits on the number of IOPS that a volume can perform. The same way that cloud providers charge more for larger capacity volumes, they also charge more for volumes with more IOPS.

HarperDB Cloud Storage

HarperDB Cloud utilizes AWS Elastic Block Storage (EBS) General Purpose SSD (gp3) volumes. This is the most common storage type used in AWS, as it provides reasonable performance for most workloads, at a reasonable price.

AWS EBS gp3 volumes have a baseline performance level of 3,000 IOPS, as a result, all HarperDB Cloud storage options will offer 3,000 IOPS. We plan to offer scalable IOPS as an option in the future.

You can read more about AWS EBS volume IOPS here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html.

Estimating IOPS for HarperDB Instance

The number of IOPS required for a particular workload is influenced by many factors. Testing your particular application is the best way to determine the number of IOPS required. A reliable method is to estimate about two IOPS for every index, including the primary key itself. So if a table has two indices besides primary key, estimate that an insert or update will require about six IOPS. Note that that can often be closer to one IOPS per index under load due to internal batching of writes, and sometimes even better when doing sequential inserts. Again it is best to test to verify this with application specific data and write patterns.

For assistance in estimating IOPS requirements feel free to contact HarperDB Support or join our Community Slack Channel.

Example Use Case IOPS Requirements

• Sensor Data Collection

  In case of IoT sensors where data collection will be sustained high IOPS are required. While there are not typically large queries going on in this case, there is a high volume of data being ingested. This implies that IOPS will be sustained at a high level. For example, if you are collection 100 records per second you would expect to need roughly 3,000 IOPS just to handle the data inserts.

• Data Analytics/BI Server

  Providing a server for analytics purposes typically requires a larger machine. Typically these cases involve large scale SQL joins and aggregations, which puts a large strain on reads. HarperDB utilizes an in-memory cache, which provides a significant performance boost on machines with large amounts of memory. However, if disparate datasets are constantly being queried and/or new data is frequently being loaded, you will find that the system still needs to have high IOPS to meet performance demand.

• Web Services

  Typical web service implementations with discrete reads and writes often do not need high IOPS to perform as expected. This is often the case is more transactional systems without the requirement for high performance load. A good rule to follow is that any HarperDB operation that requires a data scan will be IOPS intensive, but if these are not frequent then the EBS boost will suffice. Queries utilizing equals operations in either SQL or NoSQL do not require a scan due to HarperDB’s native indexing.

• High Performance Database

  Ultimately, if performance is your top priority, HarperDB should be run on bare metal hardware. Cloud providers offer these options at a higher cost, but they come with obvious performance improvements.

While HarperDB Cloud bills by RAM, each instance has other specifications associated with the RAM selection. The following table describes each instance size in detail*.

*Specifications are subject to change. For the most up to date information, please refer to AWS documentation: https://aws.amazon.com/ec2/instance-types/.

HarperDB uses role-based, attribute-level security to ensure that users can only gain access to the data they’re supposed to be able to access. Our granular permissions allow for unparalleled flexibility and control, and can actually lower the total cost of ownership compared to other database solutions, since you no longer have to replicate subsets of your data to isolate use cases.

• JWT Authentication

• Basic Authentication

• Configuration

• Users and Roles

Node name is the name given to a node. It is how nodes are identified within the cluster and must be unique to the cluster.

The name cannot contain any of the following characters: .,*> . Dot, comma, asterisk, greater than, or whitespace.

The name is set in the harperdb-config.yaml file using the clustering.nodeName configuration element.

Note: If you want to change the node name make sure there are no subscriptions in place before doing so. After the name has been changed a full restart is required.

There are multiple ways to update this element, they are:

1. Directly editing the harperdb-config.yaml file.

clustering:
  nodeName: Node1

Note: When making any changes to the harperdb-config.yaml file HarperDB must be restarted for the changes to take effect.

1. Calling set_configuration through the operations API

{
    "operation": "set_configuration",
    "clustering_nodeName":"Node1"
}

1. Using command line variables.

harperdb --CLUSTERING_NODENAME Node1

1. Using environment variables.

CLUSTERING_NODENAME=Node1

HarperDB uses token based authentication with JSON Web Tokens, JWTs.

This consists of two primary operations create_authentication_tokens and refresh_operation_token. These generate two types of tokens, as follows:

• The operation_token which is used to authenticate all HarperDB operations in the Bearer Token Authorization Header. The default expiry is one day.

• The refresh_token which is used to generate a new operation_token upon expiry. This token is used in the Bearer Token Authorization Header for the refresh_operation_token operation only. The default expiry is thirty days.

The create_authentication_tokens operation can be used at any time to refresh both tokens in the event that both have expired or been lost.

Create Authentication Tokens

Users must initially create tokens using their HarperDB credentials. The following POST body is sent to HarperDB. No headers are required for this POST operation.

{
    "operation": "create_authentication_tokens",
    "username": "username",
    "password": "password"
}

A full cURL example can be seen here:

curl --location --request POST 'http://localhost:9925' \
--header 'Content-Type: application/json' \
--data-raw '{
    "operation": "create_authentication_tokens",
    "username": "username",
    "password": "password"
}'

An example expected return object is:

{
    "operation_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6InVzZXJuYW1lIiwiaWF0IjoxNjA0OTc4MjAwLCJleHAiOjE2MDUwNjQ2MDAsInN1YiI6Im9wZXJhdGlvbiJ9.MpQA-9CMjA-mn-7mHyUXSuSC_-kqMqJXp_NDiKLFtbtMRbodCuY3DzH401rvy_4vb0yCELf0B5EapLVY1545sv80nxSl6FoZFxQaDWYXycoia6zHpiveR8hKlmA6_XTWHJbY2FM1HAFrdtt3yUTiF-ylkdNbPG7u7fRjTmHfsZ78gd2MNWIDkHoqWuFxIyqk8XydQpsjULf2Uacirt9FmHfkMZ-Jr_rRpcIEW0FZyLInbm6uxLfseFt87wA0TbZ0ofImjAuaW_3mYs-3H48CxP152UJ0jByPb0kHsk1QKP7YHWx1-Wce9NgNADfG5rfgMHANL85zvkv8sJmIGZIoSpMuU3CIqD2rgYnMY-L5dQN1fgfROrPMuAtlYCRK7r-IpjvMDQtRmCiNG45nGsM4DTzsa5GyDrkGssd5OBhl9gr9z9Bb5HQVYhSKIOiy72dK5dQNBklD4eGLMmo-u322zBITmE0lKaBcwYGJw2mmkYcrjDOmsDseU6Bf_zVUd9WF3FqwNkhg4D7nrfNSC_flalkxPHckU5EC_79cqoUIX2ogufBW5XgYbU4WfLloKcIpb51YTZlZfwBHlHPSyaq_guaXFaeCUXKq39_i1n0HRF_mRaxNru0cNDFT9Fm3eD7V8axFijSVAMDyQs_JR7SY483YDKUfN4l-vw-EVynImr4",
    "refresh_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6InVzZXJuYW1lIiwiaWF0IjoxNjA0OTc4MjAwLCJleHAiOjE2MDc1NzAyMDAsInN1YiI6InJlZnJlc2gifQ.acaCsk-CJWIMLGDZdGnsthyZsJfQ8ihXLyE8mTji8PgGkpbwhs7e1O0uitMgP_pGjHq2tey1BHSwoeCL49b18WyMIB10hK-q2BXGKQkykltjTrQbg7VsdFi0h57mGfO0IqAwYd55_hzHZNnyJMh4b0iPQFDwU7iTD7x9doHhZAvzElpkWbc_NKVw5_Mw3znjntSzbuPN105zlp4Niurin-_5BnukwvoJWLEJ-ZlF6hE4wKhaMB1pWTJjMvJQJE8khTTvlUN8tGxmzoaDYoe1aCGNxmDEQnx8Y5gKzVd89sylhqi54d2nQrJ2-ElfEDsMoXpR01Ps6fNDFtLTuPTp7ixj8LvgL2nCjAg996Ga3PtdvXJAZPDYCqqvaBkZZcsiqOgqLV0vGo3VVlfrcgJXQImMYRr_Inu0FCe47A93IAWuQTs-KplM1KdGJsHSnNBV6oe6QEkROJT5qZME-8xhvBYvOXqp9Znwg39bmiBCMxk26Ce66_vw06MNgoa3D5AlXPWemfdVKPZDnj_aLVjZSs0gAfFElcVn7l9yjWJOaT2Muk26U8bJl-2BEq_DSclqKHODuYM5kkPKIdE4NFrsqsDYuGxcA25rlNETFyl0q-UXj1aoz_joy5Hdnr4mFELmjnoo4jYQuakufP9xeGPsj1skaodKl0mmoGcCD6v1F60"
}

Using JWT Authentication Tokens

The operation_token value is used to authenticate all operations in place of our standard Basic auth. In order to pass the token you will need to create an Bearer Token Authorization Header like the following request:

curl --location --request POST 'http://localhost:9925' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6InVzZXJuYW1lIiwiaWF0IjoxNjA0OTc4MjAwLCJleHAiOjE2MDUwNjQ2MDAsInN1YiI6Im9wZXJhdGlvbiJ9.MpQA-9CMjA-mn-7mHyUXSuSC_-kqMqJXp_NDiKLFtbtMRbodCuY3DzH401rvy_4vb0yCELf0B5EapLVY1545sv80nxSl6FoZFxQaDWYXycoia6zHpiveR8hKlmA6_XTWHJbY2FM1HAFrdtt3yUTiF-ylkdNbPG7u7fRjTmHfsZ78gd2MNWIDkHoqWuFxIyqk8XydQpsjULf2Uacirt9FmHfkMZ-Jr_rRpcIEW0FZyLInbm6uxLfseFt87wA0TbZ0ofImjAuaW_3mYs-3H48CxP152UJ0jByPb0kHsk1QKP7YHWx1-Wce9NgNADfG5rfgMHANL85zvkv8sJmIGZIoSpMuU3CIqD2rgYnMY-L5dQN1fgfROrPMuAtlYCRK7r-IpjvMDQtRmCiNG45nGsM4DTzsa5GyDrkGssd5OBhl9gr9z9Bb5HQVYhSKIOiy72dK5dQNBklD4eGLMmo-u322zBITmE0lKaBcwYGJw2mmkYcrjDOmsDseU6Bf_zVUd9WF3FqwNkhg4D7nrfNSC_flalkxPHckU5EC_79cqoUIX2ogufBW5XgYbU4WfLloKcIpb51YTZlZfwBHlHPSyaq_guaXFaeCUXKq39_i1n0HRF_mRaxNru0cNDFT9Fm3eD7V8axFijSVAMDyQs_JR7SY483YDKUfN4l-vw-EVynImr4' \
--data-raw '{
    "operation":"search_by_hash",
    "schema":"dev",
    "table":"dog",
    "hash_values":[1],
    "get_attributes": ["*"]
}'

Token Expiration

operation_token expires at a set interval. Once it expires it will no longer be accepted by HarperDB. This duration defaults to one day, and is configurable in harperdb-config.yaml. To generate a new operation_token, the refresh_operation_token operation is used, passing the refresh_token in the Bearer Token Authorization Header. A full cURL example can be seen here:

curl --location --request POST 'http://localhost:9925' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6InVzZXJuYW1lIiwiaWF0IjoxNjA0OTc4MjAwLCJleHAiOjE2MDc1NzAyMDAsInN1YiI6InJlZnJlc2gifQ.acaCsk-CJWIMLGDZdGnsthyZsJfQ8ihXLyE8mTji8PgGkpbwhs7e1O0uitMgP_pGjHq2tey1BHSwoeCL49b18WyMIB10hK-q2BXGKQkykltjTrQbg7VsdFi0h57mGfO0IqAwYd55_hzHZNnyJMh4b0iPQFDwU7iTD7x9doHhZAvzElpkWbc_NKVw5_Mw3znjntSzbuPN105zlp4Niurin-_5BnukwvoJWLEJ-ZlF6hE4wKhaMB1pWTJjMvJQJE8khTTvlUN8tGxmzoaDYoe1aCGNxmDEQnx8Y5gKzVd89sylhqi54d2nQrJ2-ElfEDsMoXpR01Ps6fNDFtLTuPTp7ixj8LvgL2nCjAg996Ga3PtdvXJAZPDYCqqvaBkZZcsiqOgqLV0vGo3VVlfrcgJXQImMYRr_Inu0FCe47A93IAWuQTs-KplM1KdGJsHSnNBV6oe6QEkROJT5qZME-8xhvBYvOXqp9Znwg39bmiBCMxk26Ce66_vw06MNgoa3D5AlXPWemfdVKPZDnj_aLVjZSs0gAfFElcVn7l9yjWJOaT2Muk26U8bJl-2BEq_DSclqKHODuYM5kkPKIdE4NFrsqsDYuGxcA25rlNETFyl0q-UXj1aoz_joy5Hdnr4mFELmjnoo4jYQuakufP9xeGPsj1skaodKl0mmoGcCD6v1F60' \
--data-raw '{
  "operation":"refresh_operation_token"
}'

This will return a new operation_token. An example expected return object is:

{
  "operation_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6eyJfX2NyZWF0ZWR0aW1lX18iOjE2MDQ5NzgxODkxNTEsIl9fdXBkYXRlZHRpbWVfXyI6MTYwNDk3ODE4OTE1MSwiYWN0aXZlIjp0cnVlLCJyb2xlIjp7Il9fY3JlYXRlZHRpbWVfXyI6MTYwNDk0NDE1MTM0NywiX191cGRhdGVkdGltZV9fIjoxNjA0OTQ0MTUxMzQ3LCJpZCI6IjdiNDNlNzM1LTkzYzctNDQzYi05NGY3LWQwMzY3Njg5NDc4YSIsInBlcm1pc3Npb24iOnsic3VwZXJfdXNlciI6dHJ1ZSwic3lzdGVtIjp7InRhYmxlcyI6eyJoZGJfdGFibGUiOnsicmVhZCI6dHJ1ZSwiaW5zZXJ0IjpmYWxzZSwidXBkYXRlIjpmYWxzZSwiZGVsZXRlIjpmYWxzZSwiYXR0cmlidXRlX3Blcm1pc3Npb25zIjpbXX0sImhkYl9hdHRyaWJ1dGUiOnsicmVhZCI6dHJ1ZSwiaW5zZXJ0IjpmYWxzZSwidXBkYXRlIjpmYWxzZSwiZGVsZXRlIjpmYWxzZSwiYXR0cmlidXRlX3Blcm1pc3Npb25zIjpbXX0sImhkYl9zY2hlbWEiOnsicmVhZCI6dHJ1ZSwiaW5zZXJ0IjpmYWxzZSwidXBkYXRlIjpmYWxzZSwiZGVsZXRlIjpmYWxzZSwiYXR0cmlidXRlX3Blcm1pc3Npb25zIjpbXX0sImhkYl91c2VyIjp7InJlYWQiOnRydWUsImluc2VydCI6ZmFsc2UsInVwZGF0ZSI6ZmFsc2UsImRlbGV0ZSI6ZmFsc2UsImF0dHJpYnV0ZV9wZXJtaXNzaW9ucyI6W119LCJoZGJfcm9sZSI6eyJyZWFkIjp0cnVlLCJpbnNlcnQiOmZhbHNlLCJ1cGRhdGUiOmZhbHNlLCJkZWxldGUiOmZhbHNlLCJhdHRyaWJ1dGVfcGVybWlzc2lvbnMiOltdfSwiaGRiX2pvYiI6eyJyZWFkIjp0cnVlLCJpbnNlcnQiOmZhbHNlLCJ1cGRhdGUiOmZhbHNlLCJkZWxldGUiOmZhbHNlLCJhdHRyaWJ1dGVfcGVybWlzc2lvbnMiOltdfSwiaGRiX2xpY2Vuc2UiOnsicmVhZCI6dHJ1ZSwiaW5zZXJ0IjpmYWxzZSwidXBkYXRlIjpmYWxzZSwiZGVsZXRlIjpmYWxzZSwiYXR0cmlidXRlX3Blcm1pc3Npb25zIjpbXX0sImhkYl9pbmZvIjp7InJlYWQiOnRydWUsImluc2VydCI6ZmFsc2UsInVwZGF0ZSI6ZmFsc2UsImRlbGV0ZSI6ZmFsc2UsImF0dHJpYnV0ZV9wZXJtaXNzaW9ucyI6W119LCJoZGJfbm9kZXMiOnsicmVhZCI6dHJ1ZSwiaW5zZXJ0IjpmYWxzZSwidXBkYXRlIjpmYWxzZSwiZGVsZXRlIjpmYWxzZSwiYXR0cmlidXRlX3Blcm1pc3Npb25zIjpbXX0sImhkYl90ZW1wIjp7InJlYWQiOnRydWUsImluc2VydCI6ZmFsc2UsInVwZGF0ZSI6ZmFsc2UsImRlbGV0ZSI6ZmFsc2UsImF0dHJpYnV0ZV9wZXJtaXNzaW9ucyI6W119fX19LCJyb2xlIjoic3VwZXJfdXNlciJ9LCJ1c2VybmFtZSI6InVzZXJuYW1lIn0sImlhdCI6MTYwNDk3ODcxMywiZXhwIjoxNjA1MDY1MTEzLCJzdWIiOiJvcGVyYXRpb24ifQ.qB4FS7fzryCO5epQlFCQe4mQcUEhzXjfsXRFPgauXrGZwSeSr2o2a1tE1xjiI3qjK0r3f2bdi2xpFlDR1thdY-m0mOpHTICNOae4KdKzp7cyzRaOFurQnVYmkWjuV_Ww4PJgr6P3XDgXs5_B2d7ZVBR-BaAimYhVRIIShfpWk-4iN1XDk96TwloCkYx01BuN87o-VOvAnOG-K_EISA9RuEBpSkfUEuvHx8IU4VgfywdbhNMh6WXM0VP7ZzSpshgsS07MGjysGtZHNTVExEvFh14lyfjfqKjDoIJbo2msQwD2FvrTTb0iaQry1-Wwz9QJjVAUtid7tJuP8aBeNqvKyMIXRVnl5viFUr-Gs-Zl_WtyVvKlYWw0_rUn3ucmurK8tTy6iHyJ6XdUf4pYQebpEkIvi2rd__e_Z60V84MPvIYs6F_8CAy78aaYmUg5pihUEehIvGRj1RUZgdfaXElw90-m-M5hMOTI04LrzzVnBu7DcMYg4UC1W-WDrrj4zUq7y8_LczDA-yBC2-bkvWwLVtHLgV5yIEuIx2zAN74RQ4eCy1ffWDrVxYJBau4yiIyCc68dsatwHHH6bMK0uI9ib6Y9lsxCYjh-7MFcbP-4UBhgoDDXN9xoUToDLRqR9FTHqAHrGHp7BCdF5d6TQTVL5fmmg61MrLucOo-LZBXs1NY"
}

The refresh_token also expires at a set interval, but a longer interval. Once it expires it will no longer be accepted by HarperDB. This duration defaults to thirty days, and is configurable in harperdb-config.yaml. To generate a new operation_token and a new refresh_token the create_authentication_tokensoperation is called.

Configuration

Token timeouts are configurable in harperdb-config.yaml with the following parameters:

• operationsApi.authentication.operationTokenTimeout: Defines the length of time until the operation_token expires (default 1d).

• operationsApi.authentication.refreshTokenTimeout: Defines the length of time until the refresh_token expires (default 30d).

A full list of valid values for both parameters can be found here.

HarperDB clustering is the process of connecting multiple HarperDB databases together to create a database mesh network that enables users to define data replication patterns.

HarperDB’s clustering engine replicates data between instances of HarperDB using a highly performant, bi-directional pub/sub model on a per-table basis. Data replicates asynchronously with eventual consistency across the cluster following the defined pub/sub configuration. Individual transactions are sent in the order in which they were transacted, once received by the destination instance, they are processed in an ACID-compliant manor. Conflict resolution follows a last writer wins model based on recorded transaction time on the transaction and the timestamp on the record on the node.

Common Use Case

A common use case is an edge application collecting and analyzing sensor data that creates an alert if a sensor value exceeds a given threshold:

• The edge application should not be making outbound http requests for security purposes.

• There may not be a reliable network connection.

• Not all sensor data will be sent to the cloud--either because of the unreliable network connection, or maybe it’s just a pain to store it.

• The edge node should be inaccessible from outside the firewall.

• The edge node will send alerts to the cloud with a snippet of sensor data containing the offending sensor readings.

HarperDB simplifies the architecture of such an application with its bi-directional, table-level replication:

• The edge instance subscribes to a “thresholds” table on the cloud instance, so the application only makes localhost calls to get the thresholds.

• The application continually pushes sensor data into a “sensor_data” table via the localhost API, comparing it to the threshold values as it does so.

• When a threshold violation occurs, the application adds a record to the “alerts” table.

• The application appends to that record array “sensor_data” entries for the 60 seconds (or minutes, or days) leading up to the threshold violation.

• The edge instance publishes the “alerts” table up to the cloud instance.

By letting HarperDB focus on the fault-tolerant logistics of transporting your data, you get to write less code. By moving data only when and where it’s needed, you lower storage and bandwidth costs. And by restricting your app to only making local calls to HarperDB, you reduce the overall exposure of your application to outside forces.

To create a cluster you must have two or more nodes* (aka instances) of HarperDB running.

*A node is a single instance/installation of HarperDB. A node of HarperDB can operate independently with clustering on or off.

On the following pages we'll walk you through the steps required, in order, to set up a HarperDB cluster.

HarperDB uses Basic Auth and JSON Web Tokens (JWTs) to secure our HTTP requests. In the context of an HTTP transaction, basic access authentication is a method for an HTTP user agent to provide a user name and password when making a request.

** You do not need to log in separately. Basic Auth is added to each HTTP request like create_schema, create_table, insert etc… via headers. **

A header is added to each HTTP request. The header key is “Authorization” the header value is “Basic <<your username and password buffer token>>”

Authentication in HarperDB Studio

In the below code sample, you can see where we add the authorization header to the request. This needs to be added for each and every HTTP request for HarperDB.

Note: This function uses btoa. Learn about btoa here.

function callHarperDB(call_object, operation, callback){

    const options = {
        "method": "POST",
        "hostname": call_object.endpoint_url,
        "port": call_object.endpoint_port,
        "path": "/",
        "headers": {
            "content-type": "application/json",
            "authorization": "Basic " + btoa(call_object.username + ':' + call_object.password),
            "cache-control": "no-cache"

        }
    };

    const http_req = http.request(options, function (hdb_res) {
        let chunks = [];

        hdb_res.on("data", function (chunk) {
            chunks.push(chunk);
        });

        hdb_res.on("end", function () {
            const body = Buffer.concat(chunks);
            if (isJson(body)) {
                return callback(null, JSON.parse(body));
            } else {
                return callback(body, null);

            }

        });
    });

    http_req.on("error", function (chunk) {
        return callback("Failed to connect", null);
    });

    http_req.write(JSON.stringify(operation));
    http_req.end();

}

HarperDB was set up to require very minimal configuration to work out of the box. There are, however, some best practices we encourage for anyone building an app with HarperDB.

CORS

HarperDB allows for managing cross-origin HTTP requests. By default, HarperDB enables CORS for all domains if you need to disable CORS completely or set up an access list of domains you can do the following:

1. Open the harperdb-config.yaml file this can be found in <ROOTPATH>, the location you specified during install.

2. In harperdb-config.yaml there should be 2 entries under operationsApi.network: cors and corsAccessList.

   • cors

     1. To turn off, change to: cors: false

     2. To turn on, change to: cors: true

   • corsAccessList

     1. The corsAccessList will only be recognized by the system when cors is true

     2. To create an access list you set corsAccessList to a comma-separated list of domains.

        i.e. corsAccessList is http://harperdb.io,http://products.harperdb.io

     3. To clear out the access list and allow all domains: corsAccessList is [null]

SSL

HarperDB provides the option to use an HTTP or HTTPS and HTTP/2 interface. The default port for the server is 9925.

These default ports can be changed by updating the operationsApi.network.port value in <ROOTPATH>/harperdb-config.yaml

By default, HTTPS is turned off and HTTP is turned on. It is recommended that you never directly expose HarperDB's HTTP interface through a publicly available port. HTTP is intended for local or private network use.

You can toggle HTTPS and HTTP in the settings file. By setting operationsApi.network.https to true/false. When https is set to false, the server will use HTTP (version 1.1). Enabling HTTPS will enable both HTTPS/1.1 and HTTPS/2.

HarperDB automatically generates a certificate (certificate.pem), a certificate authority (ca.pem) and a private key file (privateKey.pem) which live at <ROOTPATH>/keys/.

You can replace these with your own certificates and key.

Changes to these settings require a restart. Use operation harperdb restart from HarperDB Operations API.

Additional information that will help you define your clustering topology.

Transactions

Transactions that are replicated across the cluster are:

• Insert

• Update

• Upsert

• Delete

• Bulk loads

  • CSV data load

  • CSV file load

  • CSV URL load

  • Import from S3

When adding or updating a node any schemas and tables in the subscription that don’t exist on the remote node will be automatically created.

Destructive schema operations do not replicate across a cluster. Those operations include drop_schema, drop_table, and drop_attribute. If the desired outcome is to drop schema information from any nodes then the operation(s) will need to be run on each node independently.

Users and roles are not replicated across the cluster.

Queueing

HarperDB has built-in resiliency for when network connectivity is lost within a subscription. When connections are reestablished, a catchup routine is executed to ensure data that was missed, specific to the subscription, is sent/received as defined.

Topologies

HarperDB clustering creates a mesh network between nodes giving end users the ability to create an infinite number of topologies. subscription topologies can be simple or as complex as needed.

Subscriptions can be added, updated, or removed through the API.

Note: The schema and tables in the subscription must exist on either the local or the remote node. Any schema and tables that do not exist on one particular node, for example, the local node, will be automatically created on the local node.

To add a single node and create one or more subscriptions use add_node.

This is an example of adding Node2 to your local node. Subscriptions are created for two tables, dog and chicken.

To update one or more subscriptions with a single node use update_node.

This call will update the subscription with the dog table. Any other subscriptions with Node2 will not change.

To add or update subscriptions with one or more nodes in one API call use configure_cluster.

Note: configure_cluster will override any and all existing subscriptions defined on the local node. This means that before going through the connections in the request and adding the subscriptions, it will first go through all existing subscriptions the local node has and remove them. To get all existing subscriptions use cluster_status.

Start time

There is an optional property called start_time that can be passed in the subscription. This property accepts an ISO formatted UTC date.

start_time can be used to set from what time you would like to source transactions from a table when creating or updating a subscription.

This example will get all transactions on Node2’s dog table starting from 2022-09-02T20:06:35.993Z and replicate them locally on the dog table.

If no start time is passed it defaults to the current time.

Note: start time utilizes clustering to back source transactions. For this reason it can only source transactions that occurred when clustering was enabled.

Remove node

To remove a node and all its subscriptions use remove_node.

Cluster status

To get the status of all connected nodes and see their subscriptions use cluster_status.

HarperDB utilizes a Role-Based Access Control (RBAC) framework to manage access to HarperDB instances. A user is assigned a role that determines the user’s permissions to access database resources and run core operations.

Roles in HarperDB

Role permissions in HarperDB are broken into two categories – permissions around database manipulation and permissions around database definition.

Database Manipulation: A role defines CRUD (create, read, update, delete) permissions against database resources (i.e. data) in a HarperDB instance.

1. At the table-level access, permissions must be explicitly defined when adding or altering a role – i.e. HarperDB will assume CRUD access to be FALSE if not explicitly provided in the permissions JSON passed to the add_role and/or alter_role API operations.

2. At the attribute-level, permissions for attributes in all tables included in the permissions set will be assigned based on either the specific attribute-level permissions defined in the table’s permission set or, if there are no attribute-level permissions defined, permissions will be based on the table’s CRUD set.

Database Definition: Permissions related to managing schemas, tables, roles, users, and other system settings and operations are restricted to the built-in super_user role.

Built-In Roles

There are three built-in roles within HarperDB. See full breakdown of operations restricted to only super_user roles .

• super_user - This role provides full access to all operations and methods within a HarperDB instance, this can be considered the admin role.

  • This role provides full access to all Database Definition operations and the ability to run Database Manipulation operations across the entire database schema with no restrictions.

• cluster_user - This role is an internal system role type that is managed internally to allow clustered instances to communicate with one another.

  • This role is an internally managed role to facilitate communication between clustered instances.

• structure_user - This role provides specific access for creation and deletion of data.

  • When defining this role type you can either assign a value of true which will allow the role to create and drop schemas & tables. Alternatively the role type can be assigned a string array. The values in this array are schemas and allows the role to only create and drop tables in the designated schemas.

User-Defined Roles

In addition to built-in roles, admins (i.e. users assigned to the super_user role) can create customized roles for other users to interact with and manipulate the data within explicitly defined tables and attributes.

• Unless the user-defined role is given super_user permissions, permissions must be defined explicitly within the request body JSON.

• Describe operations will return metadata for all schemas, tables, and attributes that a user-defined role has CRUD permissions for.

Role Permissions

When creating a new, user-defined role in a HarperDB instance, you must provide a role name and the permissions to assign to that role. Reminder, only super users can create and manage roles.

• role name used to easily identify the role assigned to individual users.

  Roles can be altered/dropped based on the role name used in and returned from a successful add_role , alter_role, or list_roles operation.

• permissions used to explicitly defined CRUD access to existing table data.

Example JSON for add_role request

Setting Role Permissions

There are two parts to a permissions set:

• super_user – boolean value indicating if role should be provided super_user access.

  If super_user is set to true, there should be no additional schema-specific permissions values included since the role will have access to the entire database schema. If permissions are included in the body of the operation, they will stored within HarperDB, but ignored, as super_users have full access to the database.

• permissions: Schema tables that a role should have specific CRUD access to should be included in the final, schema-specific permissions JSON.

  For user-defined roles (i.e. non-super_user roles, blank permissions will result in the user being restricted from accessing any of the database schema.

Table Permissions JSON

Each table that a role should be given some level of CRUD permissions to must be included in the tables array for its schema in the roles permissions JSON passed to the API (see example above).

Important Notes About Table Permissions

1. If a schema and/or any of its tables are not included in the permissions JSON, the role will not have any CRUD access to the schema and/or tables.

2. If a table-level CRUD permission is set to false, any attribute-level with that same CRUD permission set to true will return an error.

Important Notes About Attribute Permissions

1. If there are attribute-specific CRUD permissions that need to be enforced on a table, those need to be explicitly described in the attribute_permissions array.

2. If a non-hash attribute is given some level of CRUD access, that same access will be assigned to the table’s hash_attribute, even if it is not explicitly defined in the permissions JSON.

   See table_name1’s permission set for an example of this – even though the table’s hash attribute is not specifically defined in the attribute_permissions array, because the role has CRUD access to ‘attribute1’, the role will have the same access to the table’s hash attribute.

3. If attribute-level permissions are set – i.e. attribute_permissions.length > 0 – any table attribute not explicitly included will be assumed to have not CRUD access (with the exception of the hash_attribute described in #2).

   See table_name1’s permission set for an example of this – in this scenario, the role will have the ability to create, insert and update ‘attribute1’ and the table’s hash attribute but no other attributes on that table.

4. If an attribute_permissions array is empty, the role’s access to a table’s attributes will be based on the table-level CRUD permissions.

   See table_name2’s permission set for an example of this.

5. The __createdtime__ and __updatedtime__ attributes that HarperDB manages internally can have read perms set but, if set, all other attribute-level permissions will be ignored.

6. Please note that DELETE permissions are not included as a part of an individual attribute-level permission set. That is because it is not possible to delete individual attributes from a row, rows must be deleted in full.

   • If a role needs the ability to delete rows from a table, that permission should be set on the table-level.

   • The practical approach to deleting an individual attribute of a row would be to set that attribute to null via an update statement.

Role-Based Operation Restrictions

The table below includes all API operations available in HarperDB and indicates whether or not the operation is restricted to super_user roles.

Keep in mind that non-super_user roles will also be restricted within the operations they do have access to by the schema-level CRUD permissions set for the roles.

Error: Must execute as User

You may have gotten an error like, Error: Must execute as <<username>>.

This means that you installed HarperDB as <<user>>. Because HarperDB stores files natively on the operating system, we only allow the HarperDB executable to be run by a single user. This prevents permissions issues on files.

For example if you installed as user_a, but later wanted to run as user_b. User_b may not have access to the hdb files HarperDB needs. This also keeps HarperDB more secure as it allows you to lock files down to a specific user and prevents other users from accessing your files.

Custom functions are a key part of building a complete HarperDB application. It is highly recommended that you use Custom Functions as the primary mechanism for your application to access your HarperDB database. Using Custom Functions gives you complete control over the accessible endpoints, how users are authenticated and authorized, what data is accessed from the database, and how it is aggregated and returned to users.

• Add your own API endpoints to a standalone API server inside HarperDB

• Use HarperDB Core methods to interact with your data at lightning speed

• Custom Functions are powered by Fastify, so they’re extremely flexible

• Manage in HarperDB Studio, or use your own IDE and Version Management System

• Distribute your Custom Functions to all your HarperDB instances with a single click

A route is a connection between two nodes. It is how the clustering network is established.

Routes do not need to cross connect all nodes in the cluster. You can select a leader node or a few leaders and all nodes connect to them, you can chain, etc… As long as there is one route connecting a node to the cluster all other nodes should be able to reach that node.

Using routes the clustering servers will create a mesh network between nodes. This mesh network ensures that if a node drops out all other nodes can still communicate with each other. That being said, we recommend designing your routing with failover in mind, this means not storing all your routes on one node but dispersing them throughout the network.

A simple route example is a two node topology, if Node1 adds a route to connect it to Node2, Node2 does not need to add a route to Node1. That one route configuration is all that’s needed to establish a bidirectional connection between the nodes.

A route consists of a port and a host.

port - the clustering port of the remote instance you are creating the connection with. This is going to be the clustering.hubServer.cluster.network.port in the HarperDB configuration on the node you are connecting with.

host - the host of the remote instance you are creating the connection with.This can be an IP address or a URL.

Routes are set in the harperdb-config.yaml file using the clustering.hubServer.cluster.network.routes element, which expects an object array, where each object has two properties, port and host.

This diagram shows one way of using routes to connect a network of nodes. Node2 and Node3 do not reference any routes in their config. Node1 contains routes for Node2 and Node3, which is enough to establish a network between all three nodes.

There are multiple ways to set routes, they are:

1. Directly editing the harperdb-config.yaml file (refer to code snippet above).

2. Calling cluster_set_routes through the API.

Note: When making any changes to HarperDB configuration HarperDB must be restarted for the changes to take effect.

1. From the command line.

1. Using environment variables.

The API also has cluster_get_routes for getting all routes in the config and cluster_delete_routes for deleting routes.

HarperDB’s Custom Functions is built on top of , so our route definitions follow their specifications. Below is a very simple example of a route declaration.

Route URLs are resolved in the following manner:

• [Instance URL]:[Custom Functions Port]/[Project Name]/[Route URL]

• The route below, within the dogs project, with a route of breeds would be available at http://localhost:9926/dogs/breeds.

In effect, this route is just a pass-through to HarperDB. The same result could have been achieved by hitting the core HarperDB API, since it uses hdbCore.preValidation and hdbCore.request, which are defined in the “helper methods” section, below.

Custom Handlers

For endpoints where you want to execute multiple operations against HarperDB, or perform additional processing (like an ML classification, or an aggregation, or a call to a 3rd party API), you can define your own logic in the handler. The function below will execute a query against the dogs table, and filter the results to only return those dogs over 4 years in age.

IMPORTANT: This route has NO preValidation and uses hdbCore.requestWithoutAuthentication, which- as the name implies- bypasses all user authentication. See the security concerns and mitigations in the “helper methods” section, below.

Custom preValidation Hooks

The simple example above was just a pass-through to HarperDB- the exact same result could have been achieved by hitting the core HarperDB API. But for many applications, you may want to authenticate the user using custom logic you write, or by conferring with a 3rd party service. Custom preValidation hooks let you do just that.

Below is an example of a route that uses a custom validation hook:

Helper Methods

When declaring routes, you are given access to 2 helper methods: hdbCore and logger.

hdbCore

hdbCore contains three functions that allow you to authenticate an inbound request, and execute operations against HarperDB directly, by passing the standard Operations API.

• preValidation

  This takes the authorization header from the inbound request and executes the same authentication as the standard HarperDB Operations API. It will determine if the user exists, and if they are allowed to perform this operation. If you use the request method, you have to use preValidation to get the authenticated user.

• request

  This will execute a request with HarperDB using the operations API. The request.body should contain a standard HarperDB operation and must also include the hdb_user property that was in request.body provided in the callback.

• requestWithoutAuthentication

  Executes a request against HarperDB without any security checks around whether the inbound user is allowed to make this request. For security purposes, you should always take the following precautions when using this method:

  • Properly handle user-submitted values, including url params. User-submitted values should only be used for search_value and for defining values in records. Special care should be taken to properly escape any values if user-submitted values are used for SQL.

logger

This helper allows you to write directly to the Custom Functions log file, custom_functions.log. It’s useful for debugging during development, although you may also use the console logger. There are 5 functions contained within logger, each of which pertains to a different logging.level configuration in your harperdb-config.yaml file.

• logger.trace(‘Starting the handler for /dogs’)

• logger.debug(‘This should only fire once’)

• logger.warn(‘This should never ever fire’)

• logger.error(‘This did not go well’)

• logger.fatal(‘This did not go very well at all’)

Inter-node authentication takes place via HarperDB users. There is a special role type called cluster_user that exists by default and limits the user to only clustering functionality.

A cluster_user must be created and added to the harperdb-config.yaml file for clustering to be enabled.

All nodes that are intended to be clustered together need to share the same cluster_user credentials (i.e. username and password).

There are multiple ways a cluster_user can be created, they are:

1. Through the operations API by calling add_user

When using the API to create a cluster user the harperdb-config.yaml file must be updated with the username of the new cluster user.

This can be done through the API by calling set_configuration or by editing the harperdb-config.yaml file.

In the harperdb-config.yaml file under the top-level clustering element there will be a user element. Set this to the name of the cluster user.

Note: When making any changes to the harperdb-config.yaml file, HarperDB must be restarted for the changes to take effect.

1. Upon installation using command line variables. This will automatically set the user in the harperdb-config.yaml file.

Note: Using command line or environment variables for setting the cluster user only works on install.

1. Upon installation using environment variables. This will automatically set the user in the harperdb-config.yaml file.

A subscription defines how data should move between two nodes. They are exclusively table level and operate independently. They connect a table on one node to a table on another node, the subscription will apply to a matching schema name and table name on both nodes.

Note: ‘local’ and ‘remote’ will often be referred to. In the context of these docs ‘local’ is the node that is receiving the API request to create/update a subscription and remote is the other node that is referred to in the request, the node on the other end of the subscription.

A subscription consists of:

schema - the name of the schema that the table you are creating the subscription for belongs to.

table - the name of the table the subscription will apply to.

publish - a boolean which determines if transactions on the local table should be replicated on the remote table.

subscribe - a boolean which determines if transactions on the remote table should be replicated on the local table.

Publish subscription

This diagram is an example of a publish subscription from the perspective of Node1.

The record with id 2 has been inserted in the dog table on Node1, after it has completed that insert it is sent to Node 2 and inserted in the dog table there.

Subscribe subscription

This diagram is an example of a subscribe subscription from the perspective of Node1.

The record with id 3 has been inserted in the dog table on Node2, after it has completed that insert it is sent to Node1 and inserted there.

Subscribe and Publish

This diagram shows both subscribe and publish but publish is set to false. You can see that because subscribe is true the insert on Node2 is being replicated on Node1 but because publish is set to false the insert on Node1 is not being replicated on Node2.

This shows both subscribe and publish set to true. The insert on Node1 is replicated on Node2 and the update on Node2 is replicated on Node1.